DE102024206379B3 - Procedure for calibrating a camera system - Google Patents

Abstract

To calibrate a camera system with at least three cameras (5 ₁ , 5 ₂ , 5 ₃ ) with known focal lengths (f), which are arranged next to one another in the area of a common camera arrangement plane (xy) along a camera arrangement coordinate (x) and whose object-side aperture angle is each at most 45°, at least three calibration markings are first selected. The calibration markings serve to specify a calibration marking point (P _A , P _B , P _C ), wherein the calibration marking points (P _A , P _B , P _C ) are arranged at different distances (d _A , d _B , d _C ) from the camera arrangement plane (xy). The calibration marking points (P _A , P _B , P _C ) are imaged into a camera image plane of the respective camera (5 ₁ , 5 ₂ , 5 ₃ ) using the cameras (5 1 , 5 2 , 5 ₃ ) to capture calibration marking image points (P _{A '} , P _{B '} , P _C '). For each pairing of two of the cameras (5 ₁ , 5 ₂ , 5 ₃ ), a disparity of the image of the respective calibration marking image point (P _A ', P _B ', P _C ') in the respective camera image plane is measured. From the at least nine measured disparities and the camera focal lengths (f), distances (x _i - x _k ) between the cameras (5 _i , 5 _k ) of a camera pair along the camera arrangement coordinate (x) are determined, as well as position errors of each camera pair (5 _i , 5 _k ) resulting from arrangement deviations of the cameras (5 ₁ , 5 ₂ , 5 ₃ ) relative to the camera arrangement plane (xy). This results in a calibration procedure that ensures precise and reliable operation of the camera system.

Description

The invention relates to a method for calibrating a camera system comprising at least three cameras with known focal lengths. Furthermore, the invention relates to a camera system calibrated according to such a method and to a method for operating such a camera system.

Camera systems are known from the WO 2022/069 424 A1 , the WO 2022/069 425 A2 , the WO 2022/179 998 A1 and the DE 10 2011 080 702 B3 . The US 2019/0158813 A1 discloses a method for recalibrating stereo cameras in real time.

It is an object of the present invention to ensure precise and reliable operation of a camera system with multiple cameras, which can in particular represent components of stereo cameras for triangulatory object distance determination within an image-capturing environment.

This object is achieved according to the invention by a calibration method having the features specified in claim 1.

According to the invention, it was recognized that with the help of such a calibration method, even cameras with a comparatively small object-side aperture angle can be calibrated reliably and precisely. After calibration, these cameras can then be used in the camera system, for example, as components of stereo cameras. The calibration method takes advantage of the fact that the cameras of the camera system to be calibrated, which are arranged side by side along the camera arrangement coordinate, see the selected calibration marker points with defined, different disparities, so that even small position or disparity errors can be determined.

The at least three calibration markers selected during the calibration process can be natural landmarks, particularly natural features of a scene. Alternatively or additionally, markers can also be provided as part of the calibration process.

The calibration marking points are staggered in depth.

The maximum aperture angle of the cameras in the camera system toward the object is 45°. Depending on the camera system design, the aperture angle of the cameras in the camera system can also be smaller, for example, 40°, 35°, or 30°. The maximum aperture angle of the cameras in the camera system toward the object can also be even smaller. This aperture angle is usually greater than 5°.

The calibration marking points depicted during the calibration procedure are all located in the aperture angles of all cameras in the camera system to be calibrated.

In principle, the calibration procedure can also be used with a camera system with at least one camera with an aperture angle greater than 45°. In particular, the calibration procedure can also be used with a camera system with fisheye cameras or fisheye lenses.

The camera system to be calibrated can have more than three cameras, for example, four cameras, five cameras, or even more. The cameras of the camera system to be calibrated do not have to be arranged exactly in one and the same plane. The distance of a particular camera from the nearest common camera arrangement plane is usually significantly smaller than the typical distance between two adjacent cameras of the camera system to be calibrated.

Relative distance differences according to claim 2 improve a calibration result. Distances between the calibration marking points can differ from each other by more than 20%, for example by at least 25%, by at least 50%, or even by at least 100%.

Specifying more than three calibration measurement points according to claim 3 improves calibration precision. Alternatively or additionally, it is possible to specify more than one calibration marking point per selected or provided calibration marking.

The advantages of a camera system according to claim 4 correspond to those already explained above with reference to the calibration method. Such a camera system, designed for spatial image capture, can be well adapted to the practice, particularly for image capture to ensure assisted or autonomous driving. The respective telephoto lens camera can have a focal length in the range between 6 mm and 100 mm, for example, in the range between 8 mm and 50 mm.

With a pixel pitch of the respective camera of 5 µm per pixel and a focal length of 10 mm, the relative focal length (Focal length/pixel pitch) of 2000 pixel pitches (short: 2000 pixels). The pixel pitch can also be smaller, for example, 3.45 µm.

A wide-angle lens with a 2.1mm focal length and 3.45µm per pixel results in a relative focal length of 609 pixels (2.1/3.45 * 1000). Depending on the camera system, all cameras in the system can also be designed as telephoto lenses.

When the camera pairs are designed as stereo cameras according to claim 5, the advantages of calibration are particularly effective.

In the camera system according to claim 6, the functions of at least one stereo camera and an image capture system using a time-of-flight illumination method advantageously complement each other. Both the stereo camera and the components enabling a time-of-flight illumination method, namely the flash source and the sensor array, allow the determination of the distance between objects within the environment.

The camera system can be used for spatial image capture of an environment moving relative to the camera system. The at least one stereo camera can be used for triangulatory object distance determination within the captured environment in the forward direction of a relative movement of the camera system to the environment. This can be used to improve the reliability of environmental detection, particularly in connection with driver-assisted or autonomous driving. The camera system can be part of a vehicle, for example, a car or truck.

The control device can, in particular, specify a delay period between the start of the flash illumination of the at least one flash light source and the start of detection of the respective sensor array of the cameras of the at least one stereo camera. The distance between the cameras of the stereo camera results in a length of a baseline, i.e., a distance between the centers of the entrance pupils of the cameras of the at least one stereo camera. The camera system can combine the principles of a TOF (Time of Flight) camera with a triangulatory distance-measuring stereo camera. A corresponding TOF image acquisition is described, for example, in US 2019/056 498 A1 The flash light source can be designed in a manner similar to that used in TOF cameras or LIDAR systems. The same applies to the camera sensor arrays. A sensor array used in TOF image acquisition can be constructed as a single photon avalanche diode (SPAD) array. Such a SPAD array and its application in connection with a TOF measurement are described in a technical article by T. Swedish et al.: "Beyond the Line of Sight? What's New in Optical Perception", AUVSI XPONENTIAL 2022-SWEDISH (Conference Handout), pp. 1-8, https://www.xponential.org/xponential2022/Custom/Handout/Speaker5158 4_Session4220_1.pdf and from the US 2022/0174255 A1 .

For triangulatory object distance determination, the two cameras of the stereo camera are synchronized with each other via a synchronization unit. This synchronization enables, in particular, an exactly simultaneous capture of the respective object via the cameras of the stereo camera. This distinguishes a stereo camera for triangulatory object distance determination from a stereoscopic camera, which exclusively creates a spatial impression of an image capture and which is described, for example, in the US 2015/296 200 A1 The use of a SPAD array can reduce motion blur and can be used to enhance the contrast of an image capture.

Synchronization and, in particular, simultaneous object detection in triangulatory object distance determination makes it possible to precisely detect the distance of even externally moving objects, since the influence of an additional relative speed due to the external movement of the object does not affect the distance determination.

A focal length f of the respective camera of a stereo camera, the baseline b of the stereo camera and a disparity D, i.e. a difference between image coordinates of imaging positions of the same point on the sensor arrays of the cameras of the stereo camera, are coordinated in such a way that, taking into account the relationship $$\text{d}=\text{f b}/\text{D}$$ For the object distance value d to be recorded, the following applies: $$3\text{ m}\le \text{d}\le 300\text{ m}$$

In addition, the camera system can be designed so that a corresponding object distance range is captured via the TOF image capture functionality of the camera system.

The flash light source can be a controlled flash lamp or a time-controlled LED or laser light source. Flash light sources can be used. which are known for use in LIDAR devices, for example.

A flash illumination duration of the flash light source can be in the range between 10 ns and 200 ns, for example in the range of 100 ns.
The detection duration of the sensor arrays of at least one stereo camera is usually set to twice the flash illumination duration of the flash light source. The detection duration can be in the range between 20 ns and 400 ns, for example, in the range of 200 ns.

Both the flash illumination duration and the detection duration can be set via the control device.

The synchronization unit can be part of the control device. A memory unit, particularly in the form of a flash memory, can be part of the control device. The synchronization unit can be part of a synchronization device for synchronizing the flash light source and the sensor arrays of the cameras of the at least one stereo camera. The synchronization device can, in turn, be part of the control device of the camera system.

Synchronization that can be provided by the synchronization device can be better than 5 ns, can be better than 2 ns, can be better than 1 ns, and can be even better.

The stereo camera's cameras can be spaced apart laterally relative to the forward direction. Depending on the cameras' detection angle, the application, and the stereo camera's primary detection direction, other camera arrangements are also possible.

The cameras of at least one stereo camera can be equipped with a telephoto lens for capturing objects at distances between 30 m and 300 m. Depending on the application, the stereo camera cameras can also be equipped with lenses of other focal lengths. These camera lenses are designed to cover an entire specified distance range, particularly between 3 m and 300 m. In principle, even shorter distances can be covered.

The distance ranges of triangulatory object detection on the one hand and TOF distance detection on the other are coordinated and can, for example, be identical. Partial overlap of these distance ranges is also possible, so that both "triangulatory" and "TOF" distance determination principles are available in one distance range, and one of these principles is available in other distance ranges.

Using the cameras of the at least one stereo camera, a correspondence analysis can be performed to identify objects that differ from a background and are present in a field of view of interest. Methods for such correspondence analysis are known from WO 2022/069 424 A1 and the WO 2022/069 425 A2 In preparation for operation of the camera system, a stereo calibration of the cameras of at least one stereo camera can be carried out, which is generally known from the WO 2022/069 424 A1 and the DE 10 2011 080 702 B3 .

The at least one stereo camera can have more than two cameras spaced apart from each other, which in particular allows a change between different baselines, which is also possible, for example, in the WO 2022/069 424 A1 described.

The camera system can comprise at least two stereo cameras with a relatively large baseline and at least one further stereo camera with a significantly smaller baseline, whereby the smaller baseline can be at most 50%, at most 25%, or at most 10% of the larger baseline. The camera system can also comprise multiple stereo cameras with the smaller baseline.

Further object information, in particular object distance information, can be obtained by evaluating an object's shadow, which is cast by an object within the captured environment as a shadow of the flash illumination and is imaged differently by the two spaced-apart cameras of the at least one stereo camera. In particular, the height of the object can be determined because an image of an object's shadow depends on the object's elevation and lateral offset.

The camera system can be used to capture spatial images of the environment, whether moving or stationary relative to the camera system, which, for example, enables a driver-assisting function or even the function of autonomous driving when the camera system is used as part of a vehicle.

The flash light source of the camera system or another light source of the camera system for ambient lighting can have a measuring light beam that has a textu rized or structured. Such texturing/structuring of the measuring light beam can be achieved by constructing it as a plurality of individual beams that capture the environment in the form of a grid. Such a light source for generating a textured/structured measuring light beam can be designed as a laser. Examples of a detection device that operates using such a textured/structured measuring light beam can be found in DE 10 2018 213 976 A1 and the references given therein, in particular in the WO 2012/095 258 A1 , the US 9,451,237 B2 and the WO 2013/020 872 A1 Using such textured/structured illumination, optically unstructured, particularly diffuse, scattering media within the encompassed environment can be measured with respect to their distance. Such diffuse scattering media can be, for example, fog, smoke, or dust.

The camera system can achieve form redundancy and/or function redundancy, particularly when used for spatial image capture. Form redundancy is achieved when different technologies are used to determine one and the same measured variable. Functional redundancy exists when multiple devices of the same technology are used to measure one and the same measured variable. The terms "form redundancy" and "functional redundancy" are primarily used in the aviation sector. There, a combination of form and function redundancy is required to prevent the multiple effects of a fault. Such redundancies may be required by law, depending on the device's application.

At least one additional flash source and/or at least one additional stereo camera according to claim 7 enable additional functional redundancy of the camera system. By providing at least one additional flash source, for example, if the initially used flash source fails, switching to the additional flash source is possible, thus avoiding a prolonged downtime of the camera system. By using at least one additional stereo camera, additional redundancy in object distance determination can be achieved, for example, during parallel triangulation measurements using two stereo cameras.

The cameras of the additional stereo camera can each have a sensor array with a controlled preset detection start and duration, so that the additional stereo camera can essentially replace the original stereo camera in its function. Redundant TOF determination of object distances is also possible using two corresponding stereo cameras with controlled preset detection start and duration, which provides additional reliability in object distance determination and helps detect synchronization problems, for example.

When using multiple stereo cameras, it is also possible to assign cameras from different stereo cameras to a new "virtual" stereo camera with a correspondingly modified baseline, which provides further redundancy and reliability in object distance determination. For example, the camera pair that proves to be particularly favorable for triangulation determination with respect to a specific surrounding object can be selected as the baseline.

The camera system can in principle also comprise at least one fisheye stereo camera. Such a fisheye stereo camera can comprise at least two fisheye cameras, wherein each of the fisheye cameras is oriented to capture the environment both in a main capture direction and longitudinally thereto in a lateral direction, wherein capture areas of the fisheye cameras overlap in the main capture direction in an overlap area. At least one such fisheye stereo camera enables the use of additional triangulation baselines, which can be used to capture additional environmental objects and/or to check object distance results of the other stereo cameras and/or to calibrate the other stereo cameras, in particular stereo cameras implemented with telephoto lenses. The main capture direction can be the forward direction of a relative movement of the camera system to the environment. The use of fisheye cameras is known from the above-mentioned references as well as from the WO 2022/179 998 A1 A fisheye camera, mounted as rigidly as possible to a telephoto camera, allows for more precise calibration of the stereo cameras relative to each other due to the fisheye camera's larger aperture angle. An inertial measurement unit (IMU) in the respective stereo camera's cameras allows for further optimization of the relative calibration. The IMU can be used, in particular, to determine a rotation rate—that is, a relative tilt of the respective camera around a defined tilt axis.

The camera system can include multiple fisheye stereo cameras, allowing for additional redundancy. The cameras of at least one fisheye stereo camera can also each contain sensor arrays with controlled presets. a detection start and a detection duration, so that a TOF determination of object distances is also possible. In principle, the at least one fisheye stereo camera can also be designed without a TOF function. If the device has multiple fisheye stereo cameras, it is also possible for at least one of the fisheye stereo cameras to be designed with a TOF function and at least one other fisheye stereo camera to be designed without a TOF function.

In principle, the device according to the above claims can be used in such a way that precisely one distance range is detected by means of the TOF components, for example in the distance range between 30 m and 300 m, in particular between 100 m and 200 m, from the device.

The advantages of an operating method according to claim 9 correspond to those already explained above with reference to the camera system. Such an operating method enables real-time operation of the device, so that, for example, autonomous driving is possible when the device is used as part of a vehicle.

In the operating method, a covered total distance range can be divided into a plurality of adjacent or partially overlapping distance ranges. For example, three to 100 such distance ranges can be specified using corresponding image acquisition parameters, for example, 50 distance ranges that lie within the total distance range. During operation of the camera system, the total distance range can be acquired across the specified individual distance ranges several times per second, for example at a rate of 10 Hertz. During operation, the acquisition can begin with the most distant distance range within the total distance range. This ensures that when images acquired in a previous acquisition step are overwritten with a most recently acquired image, nearby objects overwrite more distant objects in non-filtered image areas of the most recently acquired image, so that the result of the operating method, namely the most recently updated, output image, always represents the objects closest to the camera system, to which the system can then react, for example, by intervening in a movement of the camera system. Buffering of images captured in previous acquisition steps can then be eliminated, which increases the speed of the process.

The last updated output image is a 2D image with unfiltered objects in all distance ranges captured within the overall distance range. Information from previously captured distance ranges within the overall distance range that was overwritten during the operating procedure can also be retained for independent spatial (3D) analysis.

For the non-filtered objects assigned to the respective detection step, the method produces a distance range in which the object is located relative to the camera system. This resulting object distance information, determined using TOF image acquisition, in conjunction with triangulatory distance detection enabled by at least one stereo camera, enables form redundancy of the operating method.

The distance information can be output as the smallest distance of the detected object or as a distance per object pixel of the respective sensor array.

The overall result is both safe and predictive image capture, which is well adapted to the requirements of driver assistance or autonomous driving in particular.

When digitally filtering to filter out structureless image areas, a stereo correspondence determination can be used, which is described, for example, in the WO 2022/069 424 A1 .

At least two of the consecutive detection steps can overlap in time. Such a temporal overlap enables the operating method to run quickly. The temporal overlap can be such that at least two distance ranges with simultaneous detection start and the same detection duration are detected simultaneously with the sensor arrays, whereby these detection ranges are specified by different specifications of the respective flash illumination start and/or the respective flash illumination duration with regard to their image acquisition parameters. It is also possible to design the repetition sequence of the operating method such that a detection actually occurs at exactly one time period, i.e. at a specified detection start and with a specified detection duration, for all specified distance ranges. This simultaneous detection of at least two detection ranges can reduce image processing effort and accelerate the entire method.

The distance extensions of the distance ranges can differ from one another. Such differing distance extensions of the distance ranges make it possible, in particular, to specify a distance sensitivity depending on the distance, which also helps to shorten the duration of the operating procedure. As the distance increases, for example, the distance extension of the respective distance range can be increased, in particular, progressively increased.

After capturing the respective distance range, a triangulation determination of the distance to objects in non-filtered image areas can be performed using the at least one stereo camera. Such an additional triangulation determination enables a comparison of object distance values determined, on the one hand, via distance-resolved detection using the TOF image acquisition parameters and, on the other hand, via triangulation.

Before acquisition, a region of interest (ROI) can be specified as part of the image field captured by the stereo cameras. Such a ROI specification allows for a reduction in the amount of image data to be processed, which can also shorten the process. The specified ROI can be a section of the road in the forward direction.

An operating method according to claim 10 can be used alternatively or in addition to the operating method explained above. Such an operating method enables redundant distance determination of a respective object distance. This can, in particular, provide form redundancy, i.e., redundancy provided by using different distance detection technologies.

The TOF distance can be used as an input variable for triangulation detection. Using the measured TOF distance as an input variable for triangulation detection can facilitate correspondence analysis during triangulation detection. In particular, correspondence analysis can be used to narrow down the disparity range to be covered. Conversely, the triangulation-detected distance can also be used as an input variable for TOF detection. The use of a measured object distance can be used in the operation of the camera system to track detected objects. Precisely one such object can be tracked, or multiple such objects can be tracked simultaneously. Object tracking steps can alternate with detection steps of an entire specified distance range. For object tracking, image acquisition parameters of the TOF distance detection and/or the triangulation distance detection can be adapted to the object distance expected during tracking.

The illumination intensity of the flash light source can be adjusted to the distance range to be captured. Such flash intensity/distance range adjustment makes it possible, in particular, to draw conclusions about diffuse scattering media present in the flash light and/or camera detection path. Safe device operation is then possible even in foggy, dusty, or smoky environments.

The flash intensity can depend quadratically on a distance within the distance range to be detected. Such a quadratic dependence of the flash intensity on the distance to be detected enables adjustment such that, with identical ambient backscatter and negligible scattering along a flash path and along a detection path between source and object, a constant intensity illumination of the at least one sensor array is controlled. This can be used to determine diffuse scattering media in the detected distance range.

The pixel sensitivity of a sensor pixel of the sensor array can be controlled as a function of the incident light intensity of the flash light source. Such control of the pixel sensitivity can, in particular, not be linear with the incident light intensity. In particular, gamma correction (γ) can be performed. The sensitivity of the at least one sensor array can then be specifically adapted to the lighting and/or detection conditions prevailing in the application.

A selection of object distance ranges to be captured can be adapted to a movement mode of the camera system relative to the environment. Such an adapted distance range/movement mode selection enables efficient utilization of the camera system's computing power, in particular, and can also increase operational reliability.

When evaluating with triangulatory stereoscopy in binocular vision, two images are compared so that any changes in brightness Changes due to changing density distributions can be neglected in a first approximation. When comparing images with trained data (neural networks, AI) in monocular vision, a homogeneous brightness distribution over distance increases detection performance, whereas changing density distributions make reliable object recognition more difficult.

• 1 schematisch eine Aufsicht auf eine Vorrichtung zur räumlichen Bilderfassung einer relativ zur Vorrichtung bewegten Umgebung, wobei ein Umgebungs-Objekt in Vorwärtsrichtung einer Relativbewegung der Vorrichtung zur Umgebung beispielhaft und perspektivisch veranschaulicht ist;
• 2 ein Zeitdiagramm zur Verdeutlichung von Bilderfassungsparametern beim Betrieb der Vorrichtung nach 1;
• 3 in einer zur 1 ähnlichen Darstellung eine weitere Ausführung einer Vorrichtung zur räumlichen Bilderfassung einer relativ zur Vorrichtung bewegten Umgebung, wobei Umgebungs-Objekte in Vorwärtsrichtung einer Relativbewegung der Vorrichtung zur Umgebung beispielhaft und perspektivisch veranschaulicht sind;
• 4 in einer zur 2 ähnlichen Darstellung einen zeitlichen Ablauf eines Betriebsverfahrens der Vorrichtung nach 3;
• 5 in einer Seitenansicht Hauptkomponenten einer Ausführung der Vorrichtung bei der räumlichen Bilderfassung einer Umgebung innerhalb von drei beispielhaften Entfernungsparametern, wobei innerhalb der erfassten Umgebung ein wolkenförmiges diffuses Streumedium vorliegt; und
• 6 schematisch Parameterverhältnisse bei einer Kalibriersituation eines Kalibrierverfahrens eines Kamerasystems mit drei Kameras.

Embodiments of the invention are explained in more detail below with reference to the drawings, in which:

• 1 schematically shows a plan view of a device for spatial image capture of an environment moving relative to the device, wherein an environmental object is illustrated by way of example and in perspective in the forward direction of a relative movement of the device to the environment;
• 2 a timing diagram to illustrate image acquisition parameters during operation of the device according to 1 ;
• 3 in a 1 similar representation, a further embodiment of a device for spatial image capture of an environment moving relative to the device, wherein environmental objects in the forward direction of a relative movement of the device to the environment are illustrated by way of example and in perspective;
• 4 in a 2 similar representation a chronological sequence of an operating method of the device according to 3 ;
• 5 in a side view, main components of an embodiment of the device in the spatial image capture of an environment within three exemplary distance parameters, wherein a cloud-shaped diffuse scattering medium is present within the captured environment; and
• 6 schematic parameter relationships in a calibration situation of a calibration procedure of a camera system with three cameras.

1 shows a top view of a device 1 for spatial image capture of an environment moving relative to the device 1, which is illustrated by an environmental object 2 in the form of a cylindrical body. The device 1 represents a camera system.

The device 1 can be part of a vehicle 3, for example a car or a truck. A forward direction of the relative movement of the device 1 to the environment is shown in the 1 illustrated by an arrow 4.

The device 1 has several stereo cameras for triangulatory object distance measurement of objects within the detected environment in the forward direction 4, i.e. in particular for determining the distance of the object 2.

A first stereo camera 5 of these stereo cameras has two cameras 5 ₁ , 5 ₂ spaced apart from one another. A distance A between these cameras 5 ₁ , 5 ₂ of the stereo camera 5 results in a length of a baseline of the stereo camera 5, i.e. a length of a distance from the centers of entrance pupils of the cameras 5 ₁ , 5 ₂ of the stereo camera. The cameras 5 ₁ , 5 ₂ of the stereo camera 5 each have a sensor array 6 ₁ , 6 ₂ with controlled specification of a start of detection for the respective camera 5 ₁ and 5 ₂ and a detection duration for the respective camera 5 ₁ , 5 ₂ . The sensor arrays 6 ₁ , 6 ₂ are designed like corresponding sensors of time-of-flight (TOF) cameras and can realize a detection duration in the range between 1 ns and 500 ns. A typical detection time is in the range between 50 ns and 400 ns, for example between 150 ns and 250 ns.

The sensor arrays 6 ₁ , 6 ₂ can each have 100 x 100 pixels or 200 x 200 pixels. A higher number of pixels is also possible, in particular 640 x 480 or 1440 x 1080 pixels.

The sensor arrays 6 ₁ , 6 ₂ can be designed as CMOS arrays.

The cameras 5 ₁ , 5 ₂ have a telephoto lens for detecting objects at distances between 50 m and 300 m.

The device 1 also includes a flash light source 7, which can also be designed like the light source of a TOF camera and allows for controlled setting of a flash illumination start time and flash illumination duration for successive illumination flashes. A flash illumination duration can be in the range between 5 ns and 150 ns, for example, in the range of 100 ns.

2 illustrates a temporal sequence during the operation of the device 1 with the stereo camera 5 and the flash light source 7. At a time t = 0, an illumination flash of the flash light source 7 begins. t = 0 thus represents the illumination flash start of a 2 schematically shown as a signal box. The flash duration of the lighting flash 8 is shown in the 2 indicated by Δt _flash .

Also at time t = 0, a delay time Δt _lag starts, which is also called delay time. This delay time Δt _lag lies in the range between 300 ns and 2 ms and can, for example, be in the range between 500 ns and 1 ms, for example in the range between 700 ns and 900 ns, for example 800 ns.

After the delay period Δt _lag has elapsed, a detection 9 by the respective sensor array 6 ₁ or 6 ₂ of the stereo camera 5 begins at time t = t _E. t _E thus represents the start of detection. The respective sensor array 6 ₁ , 6 ₂ then detects incident light from the flash light source 7 during a detection period Δt _exp .

The detection 9 by the respective sensor array 6 ₁ , 6 ₂ of the stereo camera 5 is in the 2 again illustrated by a time box.

The image acquisition parameters explained above, namely the flash illumination start (t = 0), the flash illumination duration Δt _flash , the detection start (t = t _E ), and the detection duration Δt _{exp ,} are specified in a time-synchronized manner by a control device 10 of the device 1. A synchronization unit of a synchronization device of the control device 10 can be used for this purpose.

The control device 10 has an image processor for real-time data processing. This can be a processor with a field-programmable gate array (FPGA). The image processor can be configured for ultrafast image grabbing of the individual images captured by the sensor arrays _6i , triggered by the control device 10.

Due to the synchronized specification of the start of the flash illumination and the start of detection, the control device 10 necessarily also specifies the delay period Δt _lag between the start of the illumination flash (t = 0) and the start of detection (t = t _E ) of the device 1.

Correlating with the speed of light, this results in a distance range 11 in the forward direction 4 of the device 1, captured by the stereo camera 5. The object 2 is at a distance from the device 1 along the forward direction 4 that lies within this distance range 11, so that the object 2 is captured by the stereo camera 5. The captured distance range 11 is approximately 100 m to 150 m from the device 1 and has an extension of, for example, 30 m along the forward direction 4.

This distance is in the 1 at 12 and is not to scale compared to distance range 11.

Due to the acquisition via the two cameras 5 ₁ , 5 ₂ of the stereo camera 5, a spatial image of the object 2 is captured. A reference distance determination of the object 2 to the device 1 can then be performed triangulatorily, redundantly to the TOF distance measurement over the delay period Δt _lag . The device 1 is therefore essentially a TOF camera with additional triangulation.

The stereo camera 5 can also perform a correspondence analysis of individual camera images recorded with the two cameras 5 ₁ , 5 ₂ in order to separate stereoscopically captured objects from background noise that is only present in one of the two cameras 5 ₁ , 5 _2. Such a correspondence analysis is described, for example, in WO 2022/069424 A1 and in the WO 2022/069425 A2 .

Initially, a stereo calibration of the stereo camera 5, in particular of the alignments and positions of the cameras 5 ₁ , 5 ₂ of the stereo camera 5, can be carried out using a method which is also described in the WO 2022/069424 A1 . Regarding calibration, reference is also made to the DE 10 2011 080 702 B3 referred to.

During spatial image acquisition using device 1, a TOF distance of at least one detected object, acquired using the TOF principle, can be compared with a triangulatory distance of the object, acquired using the triangulatory principle by the at least one stereo camera. The TOF distance can be used, for example, as an input variable for the triangulatory acquisition. This can facilitate correspondence analysis during the triangulatory acquisition, for example, by limiting a disparity range to be covered by the correspondence analysis.

Using the images of the object 2 illuminated by the flash light source 7 captured by the cameras 5 ₁ , 5 ₂ of the stereo camera 5, a shadow cast by the object 2 can also be captured for the illumination light of the flash light source 7. By means of such a shadow casting analysis of the different shadows cast by the object 2, recorded by the two cameras 5 ₁ , 5 ₂ of the stereo camera 5, an additional redundant determination of the distance of the object 2 from the device 1 is possible.

The device 1 has a further flash light source 13, the function of which corresponds to that of the flash light source 7, which is also used for the controlled setting of a flash illumination start and a flash illumination duration of successive Illumination flashes. The additional flash light source 13 can be used redundantly to the initially described flash light source 7, for example, in the event of a failure of the initially described flash light source 7. During operation of the additional flash light source 13, the initially described flash light source 7 can then be replaced, for example, so that continuous operation of the device 1 is possible, for example, during a monitored journey of the car or truck on which the device 1 is mounted.

The device 1 has a further stereo camera 14 with cameras 14 ₁ , 14 ₂ , the structure and function of which correspond to what was already explained above with reference to the stereo camera 5. The stereo camera 14 ₁ is arranged directly adjacent to the stereo camera 5 ₁ and covers practically the same field of view in the forward direction 4 as the camera 5 ₁ . The camera 14 ₂ is in turn arranged directly adjacent to the camera 5 ₂ and covers the same field of view in the forward direction 4 as the camera 5 ₂ .

The stereo camera 14 is in turn redundant to the stereo camera 5. The same applies here as described for the flash light sources 7 and 13.

The device 1 further comprises a fisheye stereo camera 15 with two pairs 15 ₁ , 15 ₂ on the one hand and 15 ₃ , 15 ₄ on the other hand. Each of these fisheye cameras 15 _i (i = 1 to 4) serves to capture the environment around the device 1 both in the forward direction 4 and in the lateral direction (cf. arrow 16 in 1 ). The two in the 1 The fisheye cameras 15 ₁ and 15 ₂ arranged on the left side of the vehicle 3 have a main detection direction 17 which, like the main detection directions of the cameras 5 _i and 6 _i, is horizontal, but in the case of the fisheye cameras 15 ₁ and 15 ₂ , forms an angle of 45 ° to the forward direction 4. These fisheye cameras 15 ₁ , 15 ₂ therefore cover both the forward direction 4 and the lateral direction opposite to the lateral direction 16 (left in the 1 ) of the vehicle 3 on which the device 1 is mounted. Accordingly, the two additional fisheye cameras 15 ₃ , 15 ₄ , whose main detection directions 18 also form an angle of 45° to the forward direction 4, cover not only the forward direction 4 but also a right-hand area in the lateral direction 16.

The fisheye cameras 15 _i have a detection angle around the main detection directions 17, 18 of 180°. The limit angles of the detection ranges of these fisheye cameras 15 _i are shown in the 1 These opening areas overlap in an overlap area 19, which is shown in the 1 indicated by hatching. Object 2, in particular, lies in this overlap area 19.

The illumination light source 7 emits flash light of the illumination flashes along the forward direction in an illumination cone, the edge rays of which are in the 1 at 8a are illustrated. The object 2 and also the overlap area 19 lie within the illumination cone 8a of the flash light source 7.

The fisheye stereo camera 15 can optionally perform a triangulatory object distance determination using at least two of the four fisheye cameras 15 _i , whereby it can work with short baselines (distances between the fisheye cameras 15 ₁ , 15 ₂ on the one hand, or 15 ₃ , 15 ₄ on the other hand), or with long baselines (distances between the fisheye cameras 15 ₁ and 15 ₃ or 15 ₁ and 15 ₄ or 15 ₂ and 15 ₃ or 15 ₂ and 15 ₄ ). This, in turn, allows for a redundant triangulatory object distance determination.

The use of such additional triangulation baselines is described, for example, in the WO 2022/069424 A1 and in the WO 2022/179998 A1 .

The device 1 also has a further fisheye stereo camera 20 with fisheye cameras 20 ₁ to 20 ₄ , which correspond in their structure, orientation, and paired arrangement to the fisheye cameras 15 ₁ to 15 ₄ . The fisheye stereo camera 20 can be used redundantly to the fisheye stereo camera 15. The above statements regarding the flash light sources 7 and 13 and the stereo cameras 5 and 14 apply accordingly.

The cameras 15 _i of the fisheye stereo camera 15 in turn have sensor arrays similar to the sensor arrays 6 _i described above, and can therefore again be used as part of a TOF detection, as already explained above.

The control device 10 is in signal communication with the stereo cameras 5, 14, 15, and 20 and with the flash light sources 7 and 13. Furthermore, the control device 10 can be in signal communication with components of the vehicle 3, for example, with a drive and/or with a braking system and/or with vehicle display devices, for example in a dashboard of the vehicle 3.

In real-time operation, the device 1 can record images at, for example, ten images per second.

3 shows a further embodiment of a device 21 that can be used instead of device 1 for spatial image capture of an environment moving relative to the device. Components and functions of device 21 that correspond to those of device 1 bear the same reference numerals and will not be discussed in detail again.

The device 21 is equipped with two stereo cameras 5, 14, which capture the surroundings of the device 21, which is in turn mounted on a vehicle 3, in the forward direction 4. Three surrounding objects 22, 23, and 24 are illustrated within this captured environment.

With the device 21, in particular, z-buffered operation in the forward direction 4, which is also referred to as the z-direction, is possible, in which a total distance range 25 along the forward direction 4, within which the objects 22 to 24 lie, is divided into a plurality of distance ranges 22 ₁ to 22 _max , i.e. into distance ranges 22 _i (i = 1 to max). The maximum number max of these distance ranges 22 _i can be in the range between 3 and 50, for example in the range of 10. These distance ranges 22 _i are also referred to as gate zones gz.

During the corresponding Z-buffering operation of the device 21, the image acquisition parameters illumination flash start (t = 0), illumination flash duration (Δt _flash , _gz-1 ), a detection start (t = t _E ) and a detection duration (Δt _exp , _gz-1 ) are initially specified.

This image acquisition parameter specification is again carried out by means of the control device 10.

The distance range gz-1, for example the distance range 22 _max , is then recorded using the image acquisition parameters specified in this way.

Further image acquisition parameters are specified via the control device 10, namely a further flash start (t = t _flash ), a further flash duration (Δ _flash , _gz ), a further detection start (t = t _E1 ) and a detection duration (Δt _exp , _gz ). This specification is selected such that a delay period (Δt _{lag, gz} ) with these further specified image acquisition parameters is somewhat shorter than the initially specified delay period Δt _lag , _gz-1 . This means that when capturing with the last specified image acquisition parameters, a further detection range gz is captured, which is adjacent to the initially captured distance range gz-1, i.e. the distance range 22 _max , at shorter distances from the device 21.

The further distance range gz borders on or partially overlaps the previous distance range gz-1 along the forward direction 4 in a detection progress direction +z.

Object 24 is located in this now recorded detection area gz.

The last acquired image in the distance range gz is now digitally filtered to filter out structureless image areas, leaving only the object 24. Subsequently, the image acquired in the acquisition range preceding the acquisition of the distance range gz (acquisition of the distance range gz-1) is overwritten with the last acquired image (distance range gz) in the unfiltered image areas, i.e., in the area of the object 24, of this last acquired image, so that an updated image is stored in the control device 10, which contains the information from the two distance ranges gz-1 and gz.

During operation, the control device 10 specifies further image acquisition parameters so that a further distance range gz+1 is acquired, which in turn adjoins the distance range gz opposite to the forward direction 4 or partially overlaps with it.

Now the last described steps of capturing, filtering, overwriting and specifying image capture parameters are repeated for this distance range gz+1 as well as further distance ranges 22 ₄ , 22 ₃ , 22 ₂ and 22 ₁ until the specified total distance range 25 is covered, which results from the superposition of the captured distance ranges 22 ₁ to 22 _max . In the course of these sequences, the image areas at the location of the further objects 23 (distance range 22 ₂ ) and 22 (distance range 22 ₁ ) are overwritten in the image updated for the distance range 22 _i , so that in the last updated image all three objects 22 to 24 are output with their object distances to the device 1 determined via the assignment to the respective distance range 22 _i .

The number i (i = 1 to max) of distance ranges 22 _i that are acquired during the operating procedure can be, for example, 50. The operating procedure with scanning of all distance ranges 22 _max to 22 ₁ can take place in 10 Hertz operation in real time. The result of such a complete scanning process of the distance ranges 22 _max to 22 ₁ is a 2D image with the non-filtered objects 22, 23 and 24 in all distance ranges 22 _i acquired within the total distance range 25. Written information of the distance ranges 22 _max to 22 ₂ initially recorded within the total distance range 25 can be retained in the last updated output image after recording the distance range 22 ₁ for an independent spatial (3D) evaluation.

As the 4 As illustrated, successive detection steps for distance ranges gz-1 and gz can overlap in time. More than two such detection steps can also overlap in time, so that, for example, during a delay period in the detection of a first distance range gz, a flash of at least one further detection of another distance range gz+1, gz+2 begins. A detection period of a previous detection can also overlap, for example, with a flash period of a subsequent detection.

The distance extensions Δz of the various distance ranges 22 _i , 22 _j can differ from each other. Therefore, the detection times can differ for the various acquisition steps of the operating procedure.

After capturing a respective distance range 22 _i, a triangulation determination of the distance of the objects 22 to 24 in the non-filtered image areas can additionally be performed using the at least one stereo camera 5, 14. In this case, a stereo correspondence determination can again take place, as already explained above.

Before a respective detection step of the operating method, a region of interest (ROI) can be specified as part of an image field that can be captured by the respective camera 5 _i , 14 _i , for example a road area in the forward direction 4 identified by appropriate object filtering.

5 shows a side view of another embodiment of the image capture device 1. Components and functions corresponding to those described above with reference to the 1 to 4 already explained, have the same reference numbers and will not be discussed in detail again.

Based on the 5 The use of the device 1 for determining and detecting a diffuse scattering medium along a flash light path 28 between the flash light source 7 and an exemplary environmental object 2 is explained. 5 For example, camera 5 ₁ of stereo camera 5, which enables not only triangulatory distance determination but also TOF distance determination.

In the example according to 5 the device 1 detects a total of three distance ranges 22 ₁ , 22 ₂ , 22 ₃ by specifying corresponding TOF image acquisition parameters. The first, closest distance range 22 ₁ extends by a first distance d ₁ . The next, further distance range 22 ₂ extends by a second distance d ₂ , for which the following applies: d ₂ = 2 x _{d 1} . The next, third distance range 22 ₃ extends by a third distance d ₃ , for which the following applies: d ₃ = 3 x _{d 1} . The specification of the image acquisition parameters for the three distance ranges 22 ₁ to 22 ₃ includes, in addition to the specification of the TOF image acquisition parameters “start of illumination”, “duration of illumination”, “start of detection” and “duration of detection”, a specification of a flash intensity I _i for illuminating the respective distance range 22 _i . The flash intensity I ₂ for illuminating the distance range 22 ₂ is four times as large as the flash intensity I ₁ for illuminating the first, nearest distance range 22 ₁ . The flash intensity I ₃ for illuminating the distance range 22 1 in the example 5 The third, most distant distance range 22 ₃ is nine times as large as the flash intensity I ₁ . The flash intensity I _i therefore depends quadratically on a distance d _i within a total distance range 25 to be recorded.

Assuming a constant reflectivity or a constant scattering power of an ambient surface 29, this quadratic dependence of the illumination intensity I _i on the distance d _i results in the sensor array 6 ₁ of the camera 5 ₁ experiencing a constant intensity illumination when capturing images of the three distance ranges 22 ₁ , 22 ₂ and 22 ₃ . This constant intensity illumination of the sensor array 6 ₁ only occurs when there is no significant scattering of the illumination flash 8 by the diffuse scattering medium 27 along flash light paths 31 ₁ , 31 ₂ , 31 ₃ between the flash light source 7 and the environment in the distance ranges 22 ₁ to 22 ₃ . In other words: Inequalities in the intensity illumination of the sensor array 6 ₁ along the detection paths 30 _i with the given quadratic dependence of the illumination intensity I _i on the distance d _i within the distance range 22 _i to be detected are an indication of the presence of the diffuse scattering medium 27.

1. a) Das diffuse Streumedium 27 liegt innerhalb des aktuell erfassten Entfernungsbereichs, im in der 5 dargestellten Fall also innerhalb des Entfernungsbereichs 22₂, vor. In diesem Fall wird das diffuse Streumedium 27 längs des Beleuchtungsweges 31₂ direkt beleuchtet und eine entsprechende Streuung des Beleuchtungslichts 8 durch das diffuse Streumedium 27 in das Sensor-Array 6₁ bei der TOF-Erfassung gemessen, was zu einer Erhöhung der Intensitätsausleuchtung längs des Erfassungsweges 30₂ führt.
2. b) Das diffuse Streumedium 27 liegt zwischen der Blitzlichtquelle 7 und dem Sensor-Array 6₁ einerseits und dem aktuell erfassten Entfernungsbereich, beispielsweise dem Entfernungsbereich 22₃, andererseits. In diesem Fall erfolgt eine Schwächung des Beleuchtungslichts längs des Blitzlichtweges 31₃ beim Durchtreten des diffusen Streumediums 27. Eine weitere Schwächung erfährt das vom Erfassungsbereich 22₃ in Richtung des Sensor-Arrays 6₁ längs des Erfassungsweges 30₃ rückgestreute Beleuchtungslicht aufgrund einer nochmaligen Streuung durch das diffuse Streumedium 27. Im Ergebnis ist die Intensitätsausleuchtung des Sensor-Arrays 6₁ bei der Erfassung des Entfernungsbereichs 22₃ aufgrund der im zwischenliegenden Entfernungsbereich 22₂ vorliegenden Wolke des diffusen Streumediums 27 geschwächt.

The following cases can be distinguished:

1. a) The diffuse scattering medium 27 lies within the currently recorded distance range, in the 5 In the case shown, the distance range is 22 ₂ . In this case, the diffuse scattering medium 27 is the illumination path 31 ₂ is directly illuminated and a corresponding scattering of the illumination light 8 by the diffuse scattering medium 27 into the sensor array 6 ₁ is measured during the TOF detection, which leads to an increase in the intensity illumination along the detection path 30 ₂ .
2. b) The diffuse scattering medium 27 lies between the flash light source 7 and the sensor array 6 ₁ on the one hand, and the currently detected distance range, for example the distance range 22 ₃ , on the other hand. In this case, the illuminating light is attenuated along the flash light path 31 ₃ when passing through the diffuse scattering medium 27. The illuminating light backscattered from the detection range 22 ₃ in the direction of the sensor array 6 ₁ along the detection path 30 ₃ experiences a further attenuation due to repeated scattering by the diffuse scattering medium 27. As a result, the intensity illumination of the sensor array 6 ₁ when detecting the distance range 22 ₃ is attenuated due to the cloud of the diffuse scattering medium 27 present in the intermediate distance range 22 ₂ .

A combination of an increased intensity illumination of the sensor array 6 ₁ when detecting the distance range 22 ₂ compared to the expected constant intensity illumination and a reduction in the intensity illumination of the sensor array 6 ₁ when detecting the distance range 22 ₃ thus enables the identification of the diffuse scattering medium 27 within the distance range 22 ₂ .

In the embodiments of the image capture device 1 or 21 explained above, a pixel sensitivity of sensor pixels of the various sensor arrays 6 ₁ , ... can be controlled as a function of an incident light intensity, i.e. as a function of an intensity illumination, of the sensor array 6 ₁ , ... In particular, a γ correction can be carried out during image processing using the respective sensor array 6 ₁ , .... The exponent γ used for the γ correction can vary in the range between, for example, 0.2 and 5. Depending on the value of γ, a pixel sensitivity for adapting a high dynamic range can then be achieved either at high or low intensity illumination of the sensor array.

A selection of object distance ranges 22 _i to be detected can be adapted to a movement mode of the device 1 or 21 relative to the environment in certain operating modes of the detection device 1 or 21. In particular, the total distance range 25 can be adapted to a relative speed of the device 1, 21 to the environment. Adaptation to a reverse or cornering mode is also possible.

• Über eine entsprechende Zuordnung insbesondere der Blitzbeleuchtungsdauer und der Detektionsdauer wird mithilfe der jeweiligen Vorrichtung 1, 21 mittels des TOF-Prinzips eine TOF-Entfernung des mindestens einen Objekts 2 beziehungsweise 22 bis 24 erfasst. Parallel oder sequenziell hierzu wird mittels der einen Stereokamera 5, ... der Vorrichtung 1, 21 mittels des triangulatorischen Prinzips eine triangulatorische Entfernung des jeweiligen Objekts 2 beziehungsweise 22 bis 24 erfasst. Die Erfassungswerte „TOF-Entfernung“ und „triangulatorische Entfernung“ werden dann bei Betrieb der Vorrichtung 1, 21 abgeglichen.

Alternatively or in addition to the above using the 3 and 4 explained operating procedure, the respective device 1 or 21 can also be operated as follows:

• By correspondingly assigning, in particular, the flash illumination duration and the detection duration, a TOF distance of the at least one object 2 or 22 to 24 is detected using the TOF principle with the aid of the respective device 1, 21. In parallel or sequentially, a triangulatory distance of the respective object 2 or 22 to 24 is detected using the triangulatory principle using the one stereo camera 5, ... of the device 1, 21. The detection values "TOF distance" and "triangulatory distance" are then compared during operation of the device 1, 21.

In particular, it is possible to use the determined TOF distance as an input variable for the triangulation acquisition. This can facilitate correspondence analysis during triangulation acquisition. In particular, a disparity range to be covered by the correspondence analysis can be limited. The reverse case is also possible, i.e., the triangulation-determined distance is used as an input variable for distance acquisition using the TOF principle. This can be used to determine or divide a given total distance range into segmented distance ranges 22 _i, as well as to specify the TOF image acquisition parameters based on the distance.

An illumination intensity by the flash light source 7 can be specified in particular such that a common and in particular scalar brightness control of all images of the Z-buffering of the first 3 and 4 in particular with constant intensity illumination of the respective sensor arrays 6 ₁ , ... This brightness control can be used in the form of a control loop in the device 1, 21.

A calibration of a dependence of the illumination intensity of the flash light source on the distance, i.e. an illumination intensity function, can be carried out using a calibration setup using a scattering medium with a homogeneous scattering particle distribution.

The pixel sensitivity as a function of the incident light intensity can be expressed in the form of a proportional tional function or in the form of a non-linear function.

Based on the 6 A method for calibrating a camera system with three cameras 5 ₁ , 5 ₂ and 5 ₃ is explained below, which can be three cameras of the device 1 or the device 21.

The cameras each have a sensor array 6 ₁ , 6 ₂ , 6 ₃ corresponding to what was already explained above in connection with the embodiments of devices 1 and 21. The sensor array 6 _i can also be a CCD array.

To illustrate positional relationships, the following is used in connection with the 6 an xyz coordinate system is used. The x-direction runs in the 6 to the right. The y-direction is perpendicular to the plane of the 6 into it. The z-direction runs in the 6 up.

The cameras 5 ₁ to 5 ₃ are in the area of a common camera arrangement plane, namely the xy plane of the 6 , arranged side by side along a camera arrangement coordinate x.

The cameras 5 _i are designed as cameras with an object-side aperture angle of maximum 30°. The edge limits 35 ₁ , 35 ₂ , 35 ₃ of these aperture angles of the respective cameras 5 ₁ , 5 ₂ and 5 ₃ are shown in the 6 marked.

The 5 _i cameras are designed as telephoto lens cameras with a focal length in the range of 10 mm.

To calibrate cameras ₅₁ to _53, three calibration markers are selected to specify a calibration marker point _PA , _PB, and _PC, respectively. These calibration marker points _PA , _PB , and _PC are arranged at different distances _dA , _dB, and _dC from the camera arrangement plane xy. The calibration marker points _PA , _PB, and _PC can be natural, depth-graded marker points of a scene or artificially provided, depth-graded marker points.

The calibration marking points P _A , P _B and P _C lie within all three aperture angles, i.e. within the respective aperture angle limits 35 ₁ , 35 ₂ and 35 ₃ of the cameras 5 ₁ , 5 ₂ and 5 ₃ .

With regard in particular to the relationship between the distances d _A , d _B , d _C and the other distances in the 6 length parameters shown is the 6 not to scale. The distances d _A , d _B and d _C are in the 6 exaggeratedly small.

The calibration marker points _PA , _PB, and _PC are all located at the same x- and y-coordinates. The calibration procedure does not require any specific accuracy regarding the alignment of the x- and y-coordinates of the calibration marker points _PA , _PB , and _PC . The depth gradation is important.

For example, the distances d _A , d _B and d _C can be: d _A = 50 m, d _B = 100 m, d _C = 200 m.

This distance depth gradation corresponds to a typical street scene in front of a vehicle carrying the camera system to be calibrated.

The x-distance between the cameras can, for example, be 2 m (b = 2 m). The focal length of the respective camera 5 _i can be in the range of 10 mm. With a typical camera pixel size, which corresponds to a camera pixel pitch, in the range between 3 µm and 10 µm, the result is a disparity of, for example, 2 mm, which corresponds, for example, to a distance covered by 200 pixels of the respective camera 5 _i . The relative disparity is then 200 pixels. At a distance d _B of 100 m, the result is a relative disparity of 100 pixels, and at a distance d _C = 200 m, the result is a relative disparity of 50 pixels.

The distances d _A , d _B and d _C differ from each other by at least 20 %. 6 In the illustrated embodiment, this difference in distances is at least 50%.

Depending on the calibration procedure, more than three calibration marking points P _i can be specified, e.g., four, five, six, eight, ten, or even more calibration marking points. This can improve the accuracy of the calibration.

A respective focal length f of the camera 5 _i is in the 6 compared to the distances d to the marking points P. When executed according to 6 the cameras 5 ₁ , 5 ₂ and 5 ₃ each have the same focal length f. Depending on the design of the camera system, the focal lengths of the cameras 5 _i may also differ from one another.

When arranging according to 6 It is true that the distances d are each more than four times the focal length f. Other ratios between the focal lengths f of the cameras 5 _i and the distances d are possible.

Entrance pupils of the cameras 5 _i , which are in the 6 at O ₁ , O ₂ and O ₃ have baseline lengths corresponding to the x-distances of these entrance pupils O _i to each other. An x-coordinate of the respective camera entrance pupil O _i is shown in the 6 , assigned to the respective cameras 5 _i , designated x ₁ , x ₂ and x ₃ .

As part of the calibration process, the cameras 5 _i are used to image the calibration marking points P _A , P _B and P _C and to record them in a camera image plane of the respective camera 5 _i , i.e. in the xy plane. This is shown in the 6 in the case of the camera 5 ₃ is illustrated by a schematic imaging construction in which the images P' _{A, 3} , P' _{B, 3} and P' _{C, 3} , i.e. the calibration marking pixels, are illustrated on the sensor arrays 6 _i as intersection points between rays which run from the entrance pupils O _i to the respective calibration marking points P _A , P _B , P _C .

The differences x _i - x _k between the respective x-coordinates of the entrance pupils of the cameras 5 _i , 5 _j are the baselines corresponding to the stereo cameras to which these two cameras 5 _i , 5 _j belong. This difference x _i - x _k is also referred to as the _baseline .

One and the same calibration marking point, e.g. the marking point P _A , is imaged shifted by a disparity shift u A,i depending on the distance d _A , depending on the focal length f and depending on the relation of the x-coordinates of this marking point P _A and the entrance pupil of the camera 5 _i compared to this x-coordinate x _i by a disparity shift u _A,i . These disparity shifts for the marking points P _A , P _B and P _C are exemplary for the camera 5 ₃ in the 6 highlighted at u _A,3 , u _B,3 and u _C,3 .

A difference between the disparity shifts when imaging each of the marking points by two cameras 5 _i and 5 _k of a camera pair is thus measured, here as an example for the marking point P _A , as the disparity difference du _A,ik = u _A,i - u _{A, k} = A. Corresponding disparities B and C result when imaging the marking points P _B , P _C with the respective camera 5 ₁ , 5 ₂ , 5 ₃ . These disparity differences are also simply referred to as disparities.

When performing the calibration procedure _, the disparity difference du _A,ik (= u _A,i - u _{A,k ), du B,ik and du C,ik of the image of the respective calibration marker pixel in the camera image plane xy is measured for each pairing i and k of two of the cameras, 5 i , 5 k . This results} in a total of nine measured disparities du for the three marker points P _A , P _B , P _C and the three possible camera pairings 5 _ik , namely 5 ₁₂ , 5 ₁₃ and 5 ₂₃ .

Since the camera 5 ₂ is arranged at exactly the same x-coordinate x ₂ as the three calibration marking points P _A , P _B , P _C , the imaging of these calibration marking points P _A , P _B , P _C by the camera 5 ₂ results in there being no x-distances of the calibration marking pixels from this x-coordinate x ₂ , so that the camera 5 ₂ has, by definition, disparity shift values u _{A, 2} , u _B , ₂ and u _C , ₂ = 0.

The stereo imaging formula results in: $$\text{d}=\text{f blen}/\text{du}$$

Solving for the disparity du, this results, for example, for the calibration marker point P _A and the cameras 5 _i and 5 _k : $${\text{du}}_{\text{A}\text{,ik}}={\text{u}}_{\text{A}\text{,i}}-{\text{u}}_{\text{A}\text{,k}}=\text{f}\left({\text{x}}_{\text{i}}-{\text{x}}_{\text{k}}\right)/{\text{d}}_{\text{A}}$$

The three cameras 5 ₁ , 5 _{2 ,} and 5 ₃ are not perfectly aligned within the camera alignment plane xy; rather, there are deviations from this, which are the basis for calibration. These deviations can be described, analogous to the description of a ship's motion, by a pitch angle, a roll angle, and a yaw angle of the camera system with cameras 5 ₁ to 6 ₃ relative to the alignment plane xy, or by a corresponding pitch, roll, and yaw of each of these cameras 5 ₁ to 5 ₃ individually.

A corresponding position error can be taken into account in the stereo imaging description of the above equations by including a tilt error δu _i - δu _k normalized to the focal length, i.e., a relative tilt between cameras 5 _i , 5 _k of each of the camera pairs. From the above equation (II), again as an example for the calibration marker point P _A and the cameras 5 _i and S _k , the following results: $$\left({\text{u}}_{\text{A}\text{,i}}-{\text{u}}_{\text{A}\text{,k}}\right)/\text{f}=\left({\text{x}}_{\text{i}}-{\text{x}}_{\text{k}}\right)/{\text{d}}_{\text{A}}-\left(\mathrm{\delta }{\text{u}}_{\text{i}}-\mathrm{\delta }{\text{u}}_{\text{k}}\right)$$

The left side of equation (III) above is known by measurement for disparity and by pre-calibration for camera focal length.

For the three calibration marking points P _A , P _B and P _C and the three possible camera pairings 5 ₁₂ , 5 ₁₃ and 5 _{23 ,} a total of nine equations of the above type (III) with nine unknowns result, namely the respective x-coordinates x ₁ , x ₂ , x ₃ of the entrance pupils O ₁ , O ₂ and O ₃ of the three cameras 5 ₁ , 5 ₂ , 5 ₃ , the respective rotation errors ler δu ₁ , δu ₂ and δu ₃ and the z-distances d _A , d _B , d _C of the calibration marking points P _A , P _B and P _C .

After measuring the disparities, it is possible to determine these x-coordinates, the rotation errors and the distances of the calibration marking points using the additionally known camera focal lengths f.

During operation of the device 1 or 21, after the cameras have been calibrated, selected objects 2 or 22 to 24 can be tracked or tracked. As soon as one of the objects 2 or 22 to 24 has been detected as part of the triangulatory distance detection and/or as part of the TOF distance detection, the TOF image detection parameters in particular, but also the triangulatory image detection parameters (for example, the length of the baseline of a respectively selected stereo camera, a lens focal length of the cameras of the respective stereo camera, or a disparity range to be monitored) can be specified such that the object detected once is also detected again in subsequent image acquisitions. In this case, a relative movement of the respectively detected object to the device 1 or 21 can be taken into account, so that, for example, if it is expected that the object and the device will approach each other, the distance of the respectively selected distance range 22 _i is reduced during tracking in the TOF detection. The light intensity of the flash light source can be adjusted and, in particular, controlled to a distance range of the object to be tracked.

Depending on the operating method of the device 1, 21, exactly one object can be tracked or several objects can be tracked simultaneously.

If at least one object is tracked, a tracking detection of the at least one tracked object and a scan over the entire distance range 25 can be carried out intermittently, which increases the overall detection reliability with a reasonable detection effort.

The image acquisition parameters can be adjusted for TOF detection of multiple distance ranges 22 _i , 22 _i+1 such that detection via the respective camera occurs during exactly one detection period (same detection start, same detection duration) for these distance ranges 22 _i , 22 _i+1 . In extreme cases, this can apply to all distance ranges 22 _i within the total distance range 25.

When tracking objects, the extent of the respective distance range 22 _i in which the object is expected can be determined by adjusting the flash illumination duration accordingly. For more precise object tracking, shorter flash illumination durations and shorter detection durations can be specified, resulting in a corresponding increase in distance accuracy for TOF distance determination.

The device 1 can contain a neural network or an AI module, which can be used to train the respective operating procedure and to improve image recognition accordingly.

Claims (15)

Method for calibrating a camera system with at least three cameras (5 ₁ , 5 ₂ , 5 ₃ ) with known focal lengths (f), which are arranged next to one another in the area of a common camera arrangement plane (xy) along a camera arrangement coordinate (x) and whose object-side aperture angle is each at most 45°, with the following steps: - selecting at least three calibration markings for specifying a calibration marking point (P _A , P _B , P _C ), wherein the calibration marking points (P _A , P _B , P _C ) are arranged at different distances (d _A , d _B , d _C ) from the camera arrangement plane (xy), - imaging the calibration marking points (P _A , P _B , P _C ) with the cameras (5 ₁ , 5 ₂ , 5 ₃ ) for capturing calibration marking image points (P _A ', P _B ', P _C ') into a camera image plane of the respective camera (5 ₁ , 5 ₂ , 5 ₃ ), - measuring for each pairing of two of the cameras (5 ₁ , 5 ₂ , 5 ₃ ), a disparity (A, B, C) of the image of the respective calibration marking image point (P _A ', P _B ', P _C ') in the respective camera image plane, - determining, from the at least nine measured disparities (A _ik , B _ik , C _ik ) and the camera focal lengths (f), both -- distances (x _i - x _k ) between the respective cameras (5 _j , 5 _k ) of a camera pair along the camera arrangement coordinate (x) and -- position errors (δu _{i )} resulting from arrangement deviations of the cameras (5 ₁ , 5 ₂ , 5 ₃ ) - δu _k ) of each camera pair (5 _i , 5 _k ). Procedure according to Claim 1 , characterized in that the distances (d _A , d _B , dc) of the calibration marking points (P _A , P _B , P _C ) differ from one another by at least 20%. Procedure according to Claim 1 or 2 , characterized in that more than three calibration marks ations for specifying more than three calibration marking points (P _A , P _B , P _C ,...) arranged at different distances (d _A , d _B , de,...) to the camera arrangement plane (xy). Camera system calibrated according to a method according to one of the Claims 1 until 3 , wherein at least one of the cameras (5 ₁ , 5 ₂ , 5 ₃ ) is designed as a telephoto lens camera. Camera system according to Claim 4 , wherein the camera pairs (5 _i , 5 _k ) are designed as stereo cameras for triangulatory object distance determination within a spatially imaged environment. Camera system according to Claim 5 , - with at least one flash light source (7, 13) for ambient illumination with controlled specification of a flash illumination start (t = 0, t = t _flash ) and a flash illumination duration (Δt _flash ) of successive illumination flashes, - wherein the cameras (5 ₁ , 5 ₂ , 14 ₁ , 14 ₂ , 15 ₁ , 15 ₂ , 15 ₃ , 15 4 , 20 ₁ , 20 ₂ , 20 ₃ , 20 ₄ ) of the stereo camera (5, ₁₄ , 15, 20; 5, 14) each have a sensor array (6) with controlled specification of a detection start (t = t _E , t = t _E1 ) and a detection duration (Δt _exp ), - with a control device (10) which is in signal connection with the cameras (5 ₁ , 5 ₂ , 14 ₁ , 14 ₂ , 15 ₁ , 15 ₂ , 15 ₃ , 15 ₄ , 20 ₁ , 20 ₂ , 20 ₃ , 20 ₄ ) and the flash light source (7, 13), for the time-synchronized specification of the following image acquisition parameters: - the start of the flash illumination (t = 0, t = t _flash ) and the flash illumination duration (Δt _flash ) of the flash light source (7, 13) and - the start of detection (t = t _E , t = t _E1 ) and the detection duration (Δt _exp ) of the cameras (5 ₁ , 5 ₂ , 14 ₁ , 14 ₂ , 15 ₁ , 15 ₂ , 15 ₃ , 15 ₄ , 20 ₁ , 20 ₂ , 20 ₃ , 20 ₄ ). Camera system according to Claim 6 - with at least one further flash light source (13) with controlled specification of a flash illumination start (t = 0, t = t _flash ) and a flash illumination duration (Δt _flash ) of successive illumination flashes (8, 8 ₁ ) and/or - with at least one further stereo camera (14) with two further cameras (14 ₁ , 14 ₂ ) spaced apart from one another for triangulatory object distance determination within the enclosed environment. Camera system according to Claim 6 or 7 , characterized in that the control device (10) has an image processor for real-time processing. Method for operating a camera system according to one of the Claims 6 until 8 , with the following steps: - specifying the image acquisition parameters such that a first distance range (gz-1) is acquired, - capturing the distance range (gz-1) with the specified image acquisition parameters using the cameras of the at least one stereo camera, - specifying the image acquisition parameters such that a further distance range (gz) is acquired, which borders on or partially overlaps the previous distance range (gz-1) along the forward direction (4) in a capture progress direction (+z), - capturing the further distance range (gz) with the last specified image acquisition parameters using the cameras of the at least one stereo camera, - digitally filtering the last acquired image to filter out structureless image areas, - overwriting the image acquired in the acquisition step preceding the last acquisition with the last acquired image in non-filtered image areas of the last acquired image to generate an updated image, - specifying the image acquisition parameters such that a further Distance range (gz+1) is detected which borders on or partially overlaps the previous distance range (gz) along the forward direction (4) in the detection progress direction (-z), - repeating the last steps of detection, digital filtering, overwriting and specification for further distance ranges (22 ₄ to 22 ₁ ) until a predetermined total distance range (25) is covered, which results from a superposition of the detected distance ranges (22 ₁ to 22 _max ), - outputting the last updated image. Method for operating a camera system according to one of the Claims 6 until 8 , with the following steps: - detecting a time of flight (TOF) distance of at least one object (2; 22 to 24) via a corresponding assignment of the flash illumination duration and the detection duration using the TOF principle, - detecting a triangulatory distance of the object (2; 22 to 24) by means of the at least one stereo camera using the triangulatory principle, - comparing the TOF distance and the triangulatory distance. Procedure according to Claim 10 , characterized in that the TOF distance is used as input for the triangulatory detection. Method according to one of the Claims 9 until 11 , characterized in that an illumination intensity of the flash light source (7, 13) is set to a distance range to be detected (22 _i ). Procedure according to Claim 12 , characterized in that the flash intensity depends quadratically on a distance (d _i ) within the distance range (22 _i ) to be detected. Method according to one of the Claims 9 until 13 , characterized in that a pixel sensitivity of a sensor pixel of the sensor array (6) is controlled as a function of an incident light intensity of the flash light source (7, 13). Method according to one of the Claims 9 until 14 , characterized in that a selection of object distance ranges (22 _i ) to be detected is adapted to a movement mode of the camera system (1; 21) relative to the environment.