WO2026008709A1 - Method for calibrating a camera system - Google Patents

Abstract

In order to calibrate a camera system having at least three cameras (5₁, 5₂, 5₃) with known focal lengths (f), which are arranged next to one another in the region of a common camera arrangement plane (xy) along a camera arrangement coordinate (x) and the object-side opening angle of which is in each case at most 45°, at least three calibration markings are first selected. The calibration markings are used to specify a respective calibration marking point P_A, P_B, P_C, wherein the calibration marking points (P_A, P_B, P_C) are arranged at different distances (d_A, d_B, d_C) from the camera arrangement plane (xy). The calibration marker points (P_A, P_B, P_C) are mapped into a camera image plane of the respective camera (5₁, 5₂, 5₃) using the cameras (5₁, 5₂, 5₃) to capture calibration marker image points (P_A', P_B', P_C'). For each pairing of two of the cameras (5₁, 5₂, 5₃), a disparity of the image of the respective calibration marker image point (P_A', P_B', P_C') is measured in the respective camera image plane. From the at least nine measured disparities and the camera focal lengths (f), the distances (x_i - x_k) between the cameras (5_i, 5_k) of a camera pair along the camera arrangement coordinate (x) as well as position errors of each camera pair (5_i, 5_k) are determined, which result from arrangement deviations of the cameras (5₁, 5₂, 5₃) relative to the camera arrangement plane (xy). This results in a calibration method that ensures precise and reliable operation of the camera system.

Description

Method for calibrating a camera system

The present patent application claims priority from German patent application DE 10 2024 206 379.7, the contents of which are incorporated herein by reference.

The invention relates to a method for calibrating a camera system with at least three cameras with known focal lengths. The invention further relates to a camera system calibrated according to such a method, and to methods for operating such a camera system.

Camera systems are known from WO 2022/069 424 Al, WO 2022/069 425 A2, WO 2022/179 998 Al and DE 10 2011 080 702 B3. US 2019/0158813 Al discloses a method for recalibrating stereo cameras in real time.

It is an object of the present invention to ensure precise and safe operation of a camera system with multiple cameras, which in particular can be components of stereo cameras for triangulatory object distance determination within an image-capturing environment.

This problem is solved according to the invention by a calibration method with the features specified in claim 1.

According to the invention, it was recognized that with the aid of such a calibration method, even cameras with a comparatively small object-side opening angle can be calibrated safely and precisely, so that these cameras can be used to achieve optimal performance. After calibration, these components can then be used in the camera system, for example as components of stereo cameras. The calibration process utilizes the fact that the cameras to be calibrated, arranged side-by-side along the camera arrangement coordinate, see the selected calibration marker points with defined, different disparities, allowing even small positional or disparity errors to be determined.

The at least three calibration markers selected during the calibration process can be natural marker points, i.e., natural features of a scene. Alternatively or additionally, markers can also be provided as part of the calibration process.

The calibration marker points are arranged at different depths.

The object-side opening angle of the cameras in the camera system is a maximum of 45°. Depending on the camera system design, the opening angle of the cameras can also be smaller, for example, a maximum of 40°, 35°, or even 30°. The maximum object-side opening angle of the cameras in the camera system can also be smaller. This opening angle is typically greater than 5°.

The calibration marker points shown in the calibration procedure are all located in the opening angles of all cameras of the camera system to be calibrated. In principle, the calibration procedure can also be used with a camera system with at least one camera whose field of view is greater than 45°. In particular, the calibration procedure can also be used with a camera system with fisheye cameras or fisheye lenses.

The camera system to be calibrated can contain more than three cameras, for example, four, five, or even more. The cameras in the system to be calibrated do not need to be arranged exactly in the same plane. The distance of each camera from the nearest common camera arrangement plane is typically much smaller than the typical distance between two adjacent cameras in the system to be calibrated.

Distance-relative differences according to claim 2 improve a calibration result. The distances between the calibration marker points can differ from each other by more than 20%, for example by at least 25%, at least 50%, or even at least 100%.

Specifying more than three calibration measurement points according to claim 3 improves calibration precision. Alternatively or additionally, it is possible to specify more than one calibration marker point for each selected or provided calibration marker.

The advantages of a camera system according to claim 4 correspond to those already explained above with reference to the calibration method. Such a camera system, designed for spatial image acquisition, can be well adapted to practical applications, particularly image acquisition for ensuring assistive or autonomous driving. Each telephoto lens camera can have a focal length in the range between 6 mm and 100 mm, for example in the range between 8 mm and 50 mm.

With a pixel pitch of 5 pm per pixel for the respective camera and a focal length of 10 mm, the relative focal length (focal length/pixel pitch) is 2000 pixel pitches (in short: 2000 pixels). The pixel pitch can also be smaller and, for example, 3.45 pm.

With a wide-angle lens with a 2.1mm focal length and 3.45pm per pixel, the relative focal length is 609 pixels (2.1/3.45*1000). Depending on the camera system design, all cameras in the system can also be configured as corresponding telephoto lenses.

When the camera pairs are designed as stereo cameras according to claim 5, the advantages of calibration are particularly evident.

In the camera system according to claim 6, the functions of at least one stereo camera and, on the other hand, an image acquisition system using a time-of-flight method are advantageously complementary. Both the stereo camera and the components enabling a time-of-flight method—the flash light source and the sensor array, via which time-of-flight method image acquisition parameters can be specified—allow the determination of the distance to objects within the environment.

The camera system can be used for spatial image capture of an environment moving relative to the camera system. The at least one stereo camera can be used for triangulation of object distances within the captured environment in the forward direction of a relative movement. Camera systems are used to monitor the surroundings. This can improve the safety of environmental perception, particularly in connection with driver assistance or autonomous driving. The camera system can be integrated into a vehicle, such as a car or truck.

The control unit can, in particular, specify a delay period between the start of the flash illumination from the at least one flash light source and the start of the detection period for the respective sensor array of the cameras of the at least one stereo camera. The distance between the cameras of the stereo camera defines the length of a baseline, i.e., the distance between the centers of the entrance pupils of the cameras of the at least one stereo camera. The camera system combines the principles of a Time-of-Flight (TOF) camera with a triangulatory distance-measuring stereo camera. A corresponding TOF image acquisition is described, for example, in US 2019/056 498 AL. The flash light source can be designed in the same way as is generally known from TOF cameras or LiDAR systems. The same applies to the camera sensor arrays. A sensor array used in TOF image acquisition can be configured as a Single Photon Avalanche Diode (SPAD) array. Such a SPAD array and its application in connection with a TOF measurement are described in a technical article by T. Swedish et al.: “Beyond the Line of Sight? What's New in Optical Perception”, AUVSI XPONENTIAL 2022-SWEDISH (conference handout), pp. 1-8. https://www.xponential.org/xponential2022/Custom/Handout/Speaker5158 4_Session4220_l.pdf and from US 2022/0174255 AL For triangulatory object distance determination, the two cameras of the stereo camera are synchronized with each other via a synchronization unit. This synchronization enables, in particular, precisely simultaneous acquisition of the respective object by the cameras of the stereo camera. This distinguishes a stereo camera for triangulatory object distance determination from a stereoscopic camera, which only creates a spatial impression of an image and is described, for example, in US 2015/296 200 1A. The use of a SPAD array can reduce motion blur and can be used to improve image contrast.

Synchronization, and in particular simultaneous object detection in triangulatory object distance determination, makes it possible to precisely determine the distance to objects that are moving independently, since the influence of an additional relative velocity due to the object's movement does not affect the distance determination.

A focal length f of the respective camera of a stereo camera, the baseline b of the stereo camera, and a disparity D, i.e., a difference between image coordinates of imaging positions of the same point on the sensor arrays of the cameras of the stereo camera, are coordinated such that, taking into account the relationship d = fb/D for the object distance value d to be captured, the following applies:

3 m < d < 300 m Additionally, the camera system can be designed so that a corresponding object distance range is captured via the TOF image capture functionality of the camera system.

The flash source can be a controlled flash lamp or a time-controlled LED or laser light source. Flash sources known to be used, for example, in LID AR devices, can be employed.

The flash illumination duration of the flash light source can be in the range between 10 ns and 200 ns, for example in the range of 100 ns.

The detection duration of the sensor arrays of at least one stereo camera is typically set to be twice the flash duration of the flash source. The detection duration can range from 20 ns to 400 ns, for example, around 200 ns.

Both the flash illumination duration and the detection duration can be set and preset via the control unit.

The synchronization unit can be part of the control unit. A storage unit, particularly in the form of flash memory, can be part of the control unit. The synchronization unit can be part of a synchronization device for synchronizing the flash source and the sensor arrays of the cameras of at least one stereo camera. The synchronization device, in turn, can be part of the control unit of the camera system. Synchronization that can be provided via the synchronization device can be better than 5 ns, can be better than 2 ns, can be better than 1 ns, and can be even better.

The cameras of the stereo camera can be spaced apart laterally to the forward direction. Depending on the camera's detection angle, the intended use, and the main detection direction, other camera arrangements are also possible.

The cameras of at least one stereo camera can have a telephoto lens for capturing objects at distances between 30 m and 300 m. Depending on the application, the cameras of the stereo camera can also be equipped with lenses of other focal lengths. These camera lenses are designed to cover an entire predefined distance range, particularly between 3 m and 300 m. In principle, even shorter distances can be covered.

The distance ranges of triangulatory object detection on the one hand and TOF distance detection on the other are coordinated and can, for example, be the same. Partial overlap of these distance ranges is also possible, so that both distance determination principles, "triangulatory" and "TOF," are available in one distance range, and one of these principles is available in other distance ranges.

Using the cameras of at least one stereo camera, a correspondence analysis can be performed to identify objects that differ from a background and are present in a field of interest. Methods for a corresponding correspondence analysis are known from WO 2022/069 424 A1 and WO 2022/069 425 A2. To prepare for operation of the camera system, a stereo calibration of the cameras of the at least one stereo camera can be carried out, which is generally known from WO 2022/069 424 A1 and DE 10 2011 080 702 B3.

The at least one stereo camera can have more than two cameras spaced apart from each other, which in particular allows switching between different baselines, as is also described, for example, in WO 2022/069 424 Al.

The camera system can include at least two stereo cameras with a relatively large baseline and at least one additional stereo camera with a significantly smaller baseline, where the smaller baseline can be at most 50%, 25%, or 10% of the larger baseline. The camera system can also include multiple stereo cameras with the smaller baseline.

Further object information, in particular object distance information, can be obtained by analyzing an object shadow cast by an object within the recorded environment as a shadow of the flash illumination, and rendered differently by the two cameras of the at least one stereo camera, which are spaced apart from each other. In particular, the object's height can be determined because the image of an object shadow depends on the object's height and lateral position. The camera system enables spatial image capture of the environment, whether moving or stationary relative to the camera system, which, for example when the camera system is used as part of a vehicle, allows for driver assistance functions or even autonomous driving.

The flash light source of the camera system, or another light source of the camera system for ambient illumination, can have a measuring light beam that is textured or structured across its beam cross-section. Such texturing/structuring of the measuring light beam can be achieved by constructing it as a multitude of individual beams that capture the environment in the form of a grid. Such a light source for generating a textured/structured measuring light beam can be a laser. Examples of a detection device that operates using such a textured/structured measuring light beam can be found in DE 10 2018 213 976 A1 and the references cited therein, in particular in WO 2012/095 258 A1, US 9,451,237 B2, and WO 2013/020 872 A1. Using such textured/structured illumination, optically unstructured, especially diffuse, scattering media within the encompassed environment can be measured with respect to their distance. Such diffuse scattering media can include, for example, fog, smoke, or dust.

The camera system can achieve form redundancy and/or functional redundancy, particularly when used for spatial image acquisition. Form redundancy is achieved when different technologies are used to determine one and the same measurement. Functional redundancy exists when multiple devices of the same technology are used to measure one and the same quantity. The terms "form redundancy" and "functional redundancy" are primarily used in the aerospace industry. There, a combination of form and functional redundancy is required to prevent the multiple effects of a failure. Depending on the application of the device, such redundancies may be legally mandated.

At least one additional flash source and/or at least one additional stereo camera according to claim 7 enable additional functional redundancy of the camera system. For example, if at least one additional flash source is provided, the system can switch to the additional flash source after the failure of the initially used flash source, thus avoiding a prolonged downtime of the camera system. If at least one additional stereo camera is used, for example, during parallel triangulation measurements using two stereo cameras, further redundancy in object distance determination can be achieved.

The cameras of the additional stereo camera can each feature a sensor array with controlled specification of a detection start and duration, so that the additional stereo camera can essentially replace the originally used stereo camera in its function. Redundant time-of-flight (TOF) determination of object distances is also possible using two corresponding stereo cameras with controlled specification of the detection start and duration, which provides additional reliability in object distance determination and helps, for example, to detect synchronization problems. When using multiple stereo cameras, it is also possible to assign cameras from different stereo cameras to a new "virtual" stereo camera with a correspondingly modified baseline, thus enabling further redundancy and reliability in determining object distances. For example, the camera pair that proves particularly advantageous for triangulation of a specific environmental object can be selected as the baseline setting.

The camera system can, in principle, also include at least one fisheye stereo camera. Such a fisheye stereo camera can include at least two fisheye cameras, each of which is oriented to capture the environment both in a main detection direction and parallel to it in a lateral direction, with the detection areas of the fisheye cameras overlapping in the main detection direction. At least one such fisheye stereo camera enables the use of additional triangulation baselines, which can be used to capture further environmental objects and/or to check object distance results from the other stereo cameras and/or to calibrate the other stereo cameras, in particular those equipped with telephoto lenses. The main detection direction can be the forward direction of a relative movement of the camera system to the environment. The use of fisheye cameras is known from the references mentioned above and from WO 2022/179 998 A1. A fisheye camera, mounted as rigidly as possible to a telephoto camera, allows for more precise calibration of the stereo cameras relative to each other due to the wider field of view of the fisheye camera. An inertial measuring instrument The inertial measurement unit (IMU) in the cameras of each stereo camera allows for further optimization of the relative calibration. In particular, the IMU can be used to determine the rotation rate, i.e., the relative tilt of the respective camera around a defined tilt axis.

The camera system can include multiple fisheye stereo cameras, providing additional redundancy. The cameras of the at least one fisheye stereo camera can each have sensor arrays with controlled predefined detection start and duration, enabling time-of-flight (TOF) determination of object distances. The at least one fisheye stereo camera can also be configured without TOF functionality. If the device includes multiple fisheye stereo cameras, it is also possible for at least one of the fisheye stereo cameras to have TOF functionality and at least one other fisheye stereo camera to lack it.

In principle, the device according to the above claims can be used in such a way that exactly a distance range is detected by means of the TOF components, for example in the distance range between 30 m and 300 m, in particular between 100 m and 200 m, to the device.

The advantages of an operating method according to claim 9 correspond to those already explained above with reference to the camera system. Such an operating method enables real-time operation of the device, so that, for example, autonomous driving is possible when the device is used as part of a vehicle. In this operating procedure, a total covered distance area can be subdivided into a plurality of adjacent or partially overlapping distance areas. For example, three to 100 such distance areas can be defined via corresponding image acquisition parameters, such as 50 distance areas located within the total distance area. During operation of the camera system, the total distance area can be scanned across the defined individual distance areas several times per second, for example, at a rate of 10 Hz. The scan can begin with the furthest distance area within the total distance area. This ensures that when images captured in a previous acquisition step are overwritten with the most recently captured image, nearby objects overwrite more distant objects in unfiltered areas of the most recently captured image. This means that the result of the operating procedure—namely, the most recently updated, output image—always represents the objects closest to the camera system. This allows for adjustments, such as intervening in the movement of the camera system. Intermediate storage of images captured in previous acquisition steps is then unnecessary, thus increasing the speed of the process.

The most recently updated, output image is a 2D image with unfiltered objects in all distance areas captured within the overall range. Information from previously captured distance areas within the overall range, even if overwritten during operation, can be retained for independent spatial (3D) analysis. For each object not filtered out and assigned to the respective acquisition step, the process yields a distance range within which the object is located relative to the camera system. This resulting object distance information, determined using time-of-flight (TOF) image acquisition, in conjunction with triangulatory distance acquisition enabled by at least one stereo camera, allows for form redundancy in the operating procedure.

Distance information can be output as either the smallest distance of the detected object or a distance per object pixel of the respective sensor array.

Overall, the result is a safe and forward-looking image acquisition system that is well adapted to the requirements of driver assistance or autonomous driving in particular.

When digitally filtering to extract featureless image areas, a stereo correspondence determination can be used, which is described, for example, in WO 2022/069 424 AE.

At least two of the successive detection steps can overlap in time. Such a temporal overlap enables a rapid operational procedure. The temporal overlap can be such that at least two detection ranges with simultaneous detection start and duration are detected simultaneously by the sensor arrays, whereby these detection ranges are distinguished by different specifications for the respective flash illumination start times. and/or the respective flash duration with regard to their image acquisition parameters. It is also possible to design the repetition sequence of the operating procedure in such a way that detection actually occurs at precisely one time period, i.e., at a predetermined detection start and with a predetermined detection duration, for all predetermined distance ranges. This simultaneous detection of at least two detection areas can reduce image processing effort and accelerate the overall process.

The distance extents of the detection zones can differ from one another. Such differing distance extents of the detection zones make it possible, in particular, to define a distance sensitivity depending on the distance, which also helps to shorten the operating time of the process. For example, with increasing distance, the distance extent of the respective detection zone can be increased, especially progressively.

After capturing the respective distance range, a triangulation can be performed to determine the distance to objects in unfiltered image areas using at least one stereo camera. This additional triangulation allows for a comparison of object distance values determined on the one hand by distance-range-resolved acquisition using the TOF image acquisition parameters and on the other hand by triangulation. Before acquisition, a Region of Interest (ROI) can be defined as part of the image field captured by the stereo cameras. Such an ROI definition reduces the amount of image data to be processed, which can also shorten the procedure. The defined ROI could be a section of road in the direction of travel.

An operating method according to claim 10 can be used as an alternative or in addition to the operating method described above. Such an operating method enables redundant distance determination of a given object's position. In particular, this can provide form redundancy, i.e., redundancy achieved by using different technologies for distance detection.

The time-of-flight (TOF) distance can be used as an input for triangulation. Using the measured TOF distance as an input for triangulation can facilitate correspondence analysis during triangulation. In particular, it can help define the disparity range to be covered during correspondence analysis. Conversely, the triangulated distance can also be used as an input for TOF measurement. The use of a measured object distance can be employed in the operation of the camera system to track or monitor detected objects. It is possible to track exactly one such object or to track multiple objects simultaneously. Object tracking steps can be combined with measurement steps of an entire predefined distance range. The field of view alternates. For object tracking, image acquisition parameters of TOF distance detection and/or triangulatory distance detection can be adjusted to the object distance expected during tracking.

The flash intensity can be adjusted to the distance being captured. Such a flash intensity/distance range adjustment makes it possible, in particular, to infer the presence of diffuse scattering media in the flash and/or camera's field of view. This enables reliable operation of the device even in foggy, dusty, or smoky environments.

The flash intensity can depend quadratically on the distance within the detection range. Such a quadratic dependence of the flash intensity on the detection range allows, particularly when the illumination intensity increases quadratically with increasing distance, for adjustment such that, with identical ambient backscatter and negligible scattering along a flash path and along a detection path between source and object, a constant intensity illumination of the at least one sensor array can be achieved. This can be used to determine diffuse scattering media within the detected detection range.

The pixel sensitivity of a sensor pixel in the sensor array can be controlled as a function of the incident light intensity from the flash source. Such control of pixel sensitivity cannot, in particular, be linear to the incident light intensity. Specifically, gamma correction (γ) can be applied. The sensitivity of the At least one sensor array can then be specifically adapted to the lighting and/or detection conditions present in the application.

A selection of object distance ranges to be captured can be adapted to a movement mode of the camera system relative to the environment. Such an adapted distance range-movement mode selection enables efficient utilization of the camera system's computing power and can also increase operational reliability.

When evaluating images using triangulatory stereoscopy with binocular vision, two images are compared, so that any changes in brightness due to altered density distributions can be neglected to a first approximation. When comparing the images with trained data (neural networks, AI) with monocular vision, a homogeneous brightness distribution over distance increases detection performance, while changing density distributions make reliable object recognition more difficult.

Exemplary embodiments of the invention are explained in more detail below with reference to the drawing. This drawing shows:

Fig. 1 schematically shows a top view of a device for spatial image acquisition of an environment moving relative to the device, wherein an environment object is illustrated by way of example and perspective in the forward direction of a relative movement of the device to the environment; Fig. 2 is a time diagram to illustrate image acquisition parameters during operation of the device according to Fig. 1;

Fig. 3, in a representation similar to Fig. 1, shows a further embodiment of a device for spatial image acquisition of an environment moving relative to the device, wherein environment objects are illustrated by way of example and perspective in the forward direction of a relative movement of the device to the environment;

Fig. 4 shows a temporal sequence of an operating procedure of the device according to Fig. 3 in a representation similar to Fig. 2;

Fig. 5 shows a side view of the main components of an embodiment of the device for spatial image acquisition of an environment within three exemplary distance parameters, wherein a cloud-like diffuse scattering medium is present within the acquired environment; and

Fig. 6 schematically shows parameter relationships in a calibration situation of a calibration procedure for a camera system with three cameras.

Fig. 1 shows a top view of a device 1 for spatial image acquisition of an environment moving relative to the device 1, illustrated by an environmental object 2 in the form of a cylindrical body. The device 1 represents a camera system. The device 1 can be part of a vehicle 3, for example a car or a truck. A forward direction of the relative movement of the device 1 to its surroundings is illustrated in Fig. 1 by an arrow 4.

The device 1 has several stereo cameras for triangulatory object distance measurement of objects within the detected environment in the forward direction 4, i.e. in particular for determining the distance of the object 2.

A first stereo camera 5 of these stereo cameras has two cameras 51, 52 spaced apart from each other. A distance A between these cameras 51, 52 of the stereo camera 5 results in a baseline length of the stereo camera 5, i.e., a distance between the centers of the entrance pupils of cameras 51, 52 of the stereo camera. The cameras 5i, 52 of the stereo camera 5 each have a sensor array 61, 62 with controlled specification of a detection start time and a detection duration for the respective camera 5i and 52. The sensor arrays 61, 62 are designed like corresponding sensors of Time-of-Flight (TOF) cameras and can achieve a detection duration in the range between 1 ns and 500 ns. A typical detection time is in the range between 50 ns and 400 ns, for example between 150 ns and 250 ns.

The sensor arrays 61 and 62 can each have 100 x 100 pixels or 200 x 200 pixels. In principle, a higher pixel count is also possible, in particular 640 x 480 or 1440 x 1080 pixels.

The sensor arrays 61, 62 can be implemented as a CMOS array. The 5i and 52 cameras have a telephoto lens for capturing objects at distances between 50 m and 300 m.

Device 1 also includes a flash light source 7, which can also be designed like the light source of a TOF camera and allows controlled specification of the start time and duration of successive flashes. The flash duration can be in the range between 5 ns and 150 ns, for example, in the range of 100 ns.

Fig. 2 illustrates the time sequence during the operation of the device 1 with the stereo camera 5 and the flash source 7. At time t = 0, an illumination flash from the flash source 7 begins. t = 0 thus represents the start of an illumination flash 8, schematically depicted as a signal box in Fig. 2. The duration of the illumination flash 8 is indicated in Fig. 2 by the letter Atfiash.

Also at time t = 0, a delay time Atiag starts, which is also called the delay time. This delay time Atiag is in the range between 300 ns and 2 ms and can, for example, be in the range between 500 ns and 1 ms, for example in the range between 700 ns and 900 ns, for example at 800 ns.

After the delay period Atiag has elapsed, a detection 9 begins at time t = tE by the respective sensor array 6i or 62 of the stereo camera 5. tE thus represents the start of the detection. The respective sensor array 61, 62 then detects incident light from the flash source 7 for a detection duration At _exp . Detection 9 by the respective sensor array 6i, 62 of the stereo camera

Figure 5 is again illustrated by a time box in Fig. 2.

The image acquisition parameters described above, namely the flash illumination start (t = 0), the flash illumination duration Atfiash, the detection start (t = 1E) and the detection duration At _exp , are specified in a time-synchronized manner by a control unit 10 of the device 1. A synchronization unit of a synchronization device of the control unit 10 can be used for this purpose.

The control unit 10 has an image processor for real-time data processing. This can be a processor with a field-programmable gate array (FPGA). The image processor can be configured for ultra-fast image grabbing of the individual images captured by the sensor arrays 6i, triggered by the control unit 10.

Due to the synchronized specification of the flash illumination start and the detection start, the control unit 10 inevitably also specifies the delay period Ati _ag between the illumination flash start (t = 0) and the detection start (t = 1E) of the device 1.

Correlated with the speed of light, a distance range 11 is detected by the stereo camera 5 in the forward direction 4 of the device 1. The object 2 is at a distance from the device 1 along the forward direction 4 that lies within this distance range 11, so that the object 2 is detected by the stereo camera 5. The detected distance- The area 11 is located approximately 100 m to 150 m away from the device 1 and has an extent of, for example, 30 m along the forward direction 4.

This distance is shown at 12 in Fig. 1 and is not to scale compared to the distance range 11.

Based on the acquisition by the two cameras 5i and 52 of the stereo camera 5, a spatial image of object 2 is obtained. Triangulation can then be used redundantly to determine the reference distance of object 2 to device 1 over the delay period Atiag, in addition to the TOF (time-of-flight) position measurement. Device 1 is therefore essentially a TOF camera with added triangulation capabilities.

The stereo camera 5 can also perform a correspondence analysis of individual camera images captured by the two cameras 5i and 52 in order to separate stereoscopically captured objects from background noise that is present only in one of the two cameras 51 and 52. Such a correspondence analysis is described, for example, in WO 2022/069424 A1 and WO 2022/069425 A2.

Initially, a stereo calibration of the stereo camera 5, in particular of the orientations and positions of the cameras 5i, 52 of the stereo camera 5, can be carried out using a method that is also described in WO 2022/069424 AL. With regard to the calibration, reference is also made to DE 10 2011 080 702 B3.

In spatial image acquisition using the device 1, a TOF distance of at least one captured object can be detected using the TOF principle. The object is compared to a triangulatory distance, which was acquired using at least one stereo camera based on the triangulatory principle. The time-of-flight (TOF) distance can, for example, be used as an input for the triangulatory acquisition. This can facilitate correspondence analysis during triangulatory acquisition, for example, by defining the disparity range to be covered in the correspondence analysis.

Using the images of object 2, illuminated by flash source 7 and captured by cameras 5i and 52 of the stereo camera 5, it is also possible to detect the shadows cast by object 2 for the illumination light of the flash source 7. By evaluating the different shadows cast by object 2, as recorded by the two cameras 5i and 52 of the stereo camera 5, an additional redundant determination of the distance between object 2 and device 1 is possible.

The device 1 has an additional flash light source 13, whose function corresponds to that of the flash light source 7, i.e., it also serves to control the start and duration of successive flashes. The additional flash light source 13 can be used redundantly to the flash light source 7 described above, for example, in the event of a failure of the flash light source 7 described above. While the additional flash light source 13 is operating, the flash light source 7 described above can then be replaced, for example, so that uninterrupted operation of the device 1 is possible, for example, during a monitored journey of the car or truck on which the device 1 is mounted. The device 1 has a further stereo camera 14 with cameras 14i and 142, whose construction and function correspond to what has already been explained above with reference to the stereo camera 5. The stereo camera 14i is arranged directly adjacent to the stereo camera 5i and covers practically the same field of view in the forward direction 4 as the camera 5i. The camera 142 is in turn arranged directly adjacent to the camera 52 and covers the same field of view in the forward direction 4 as the camera 52.

Stereo camera 14 is redundant to stereo camera 5. The same applies here as was stated for flash sources 7 and 13.

The device 1 further comprises a fisheye stereo camera 15 with two pairs 15i, 152 on the one hand and 15s, 154 on the other. Each of these fisheye cameras 15i (i = 1 to 4) serves to capture the environment around the device 1 both in the forward direction 4 and in a lateral direction (see arrow 16 in Fig. 1). The two fisheye cameras 151 and 152 arranged on the left side of the vehicle 3 in Fig. 1 have a main detection direction 17, which, like the main detection directions of cameras 5i and 6i, is horizontal, but in the case of fisheye cameras 151 and 152 forms an angle of 45° to the forward direction 4. These fisheye cameras 151, 152 thus cover both the forward direction 4 and the lateral direction opposite to the lateral direction 16 (left in Fig. 1) of the vehicle 3 on which the device 1 is mounted. Correspondingly, the two further fisheye cameras 153, 154, whose main detection directions 18 also form an angle of 45° to the forward direction 4, cover not only the forward direction 4 but also a right-hand area in the lateral direction 16. The fisheye cameras 15i have a detection angle of 180° around the main detection directions 17, 18. The limiting angles of the detection ranges of these fisheye cameras 15i are indicated by dashed lines in Fig. 1. These aperture ranges overlap in an overlap area 19, which is indicated by hatching in Fig. 1. Object 2 is located in this overlap area 19.

The illumination light source 7 emits flashes of light along the forward direction in an illumination cone, the marginal rays of which are illustrated at 8a in Fig. 1. The object 2 and also the overlap area 19 lie within the illumination cone 8a of the flash light source 7.

The fisheye stereo camera 15 can optionally perform triangulatory object distance determination using at least two of the four fisheye cameras 15i, employing either short baselines (distances between fisheye cameras 15i and 152 on the one hand, and 15s and 154 on the other) or long baselines (distances between fisheye cameras 15i and 153, 151 and 154, 152 and 15s, or 152 and 154). This allows for redundant triangulatory object distance determination.

The use of such additional triangulation baselines is described, for example, in WO 2022/069424 Al and in WO 2022/179998 AE.

The device 1 also has another fisheye stereo camera 20 with

Fisheye cameras 201 to 204, which in their construction, their orientation and the paired arrangement corresponds to the fisheye cameras 15i to 154. The fisheye stereo camera 20 can be used redundantly with the fisheye stereo camera 15. The same applies here as described above for the flash sources 7 and 13 and the stereo cameras 5 and 14, respectively.

The cameras 15i of the fisheye stereo camera 15 in turn have sensor arrays of the type of the sensor arrays 6i described above, and can therefore again be used as part of a TOF recording, as already explained above.

The control unit 10 is in signal communication with the stereo cameras 5, 14, 15 and 20 and with the flash sources 7 and 13. Furthermore, the control unit 10 can be in signal communication with components of the vehicle 3, for example with a drive system and/or with a braking system and/or with vehicle display devices, for example in a dashboard of the vehicle 3.

In real-time operation, the device 1 can take an image at, for example, ten images per second.

Fig. 3 shows a further embodiment of a device 21, which can be used instead of the device 1 for spatial image acquisition of an environment moving relative to the device. Components and functions of the device 21 that correspond to those of the device 1 bear, in particular, the same reference numerals and are not discussed again in detail. The device 21 is equipped with two stereo cameras 5, 14, which capture the surroundings of the device 21, which is mounted on a vehicle 3, in the forward direction 4. Three environmental objects 22, 23 and 24 are illustrated within this captured environment.

The device 21 enables, in particular, z-buffered operation in the forward direction 4, also referred to as the z-direction, in which a total distance range 25 along the forward direction 4, within which the objects 22 to 24 lie, is subdivided into a plurality of distance ranges 22i to _22max , i.e., distance zones 22i (i = 1 to max). The maximum number max of these distance zones 22i can be in the range between 3 and 50, for example, in the range of 10. These distance zones 22i are also referred to as gate zones gz.

During the corresponding Z-buffering operation of the device 21, the image acquisition parameters illumination flash start (t = 0), illumination flash duration (Atflash, gz-i), a detection start (t = 1E) and a detection duration (Atexp, gz-i) are initially specified.

This image acquisition parameter specification is again made using control unit 10.

The distance range gz-1, for example the distance range 22max, is then captured using the image acquisition parameters specified in this way.

Further image acquisition parameters are specified via the control unit 10, namely a further flash start (t = tfiash), a further Flash duration (Afiash, _gz ), a further detection start (t = tEi), and a detection duration ( _Atexp , _gz ). This specification is chosen such that a delay period (Atiag, gz) for these further specified image acquisition parameters is slightly shorter than the initially specified delay period Atiag, gz-i. This means that when acquiring with the last specified image acquisition parameters, a further detection area gz is captured, which is adjacent to the initially captured distance area gz-1, i.e., the distance area 22 _max , at shorter distances to the device 21.

The further distance range gz borders on or partially overlaps the preceding distance range gz-1 along the forward direction 4 in a detection progress direction +z.

Object 24 is located within this now recorded recording area gz.

The last captured image in the distance range gz is now digitally filtered to remove unstructured image areas, leaving object 24. Subsequently, the image captured in the preceding capture range (capture of distance range gz-1) is overwritten with the last captured image (distance range gz) in the unfiltered image areas, i.e., in the area of object 24, so that an updated image containing the information from both distance ranges gz-1 and gz is stored in the control unit 10.

During operation, the control unit 10 specifies further image acquisition parameters, so that a further distance range gz+1 is captured, which again connects to the distance range gz in the opposite direction to the forward direction 4 or partially overlaps with it. The previously described steps of capturing, filtering, overwriting, and specifying image capture parameters for this distance range gz+1, as well as further distance ranges 224, 223, 222, and 221, are now repeated until the specified total distance range 25 is covered, which results from the superposition of the captured distance ranges 22i to 22max. During these sequences, the image areas at the location of the further objects 23 (distance range 222) and 22 (distance range 22i) are overwritten in the image updated for distance range 22i, so that in the last updated image, all three objects 22 to 24 are displayed with their object distances to device 1, determined by their assignment to the respective distance range 22i.

The number i (i = 1 to max) of distance ranges 22i acquired during the operating procedure can, for example, be 50. The operating procedure with scanning of all distance ranges 22 _max to 22i can take place in real time at 10 Hz. The result of such a complete scanning process of the distance ranges 22 _max to 22i is a 2D image with the unfiltered objects 22, 23, and 24 in all distance ranges 22i acquired within the total distance range 25. Information overwritten by the distance ranges 22 _max to 222 initially acquired within the total distance range 25 can also be retained in the last updated, output image after the acquisition of distance range 22i for independent spatial (3D) evaluation.

As illustrated in Fig. 4, successive detection steps for distance ranges gz-1 and gz can overlap in time. Furthermore, more than two such detection steps can overlap in time, so that, for example, during a delay period when detecting a first distance range gz, a flash begins at least one further detection of a further distance range gz+1, gz+2. A detection period of a previous detection can also overlap with a flash period of a subsequent detection.

The distance extents Az of the different distance ranges 22i, 22j can differ from each other. Therefore, the detection durations for the various acquisition steps of the operating procedure can differ.

After capturing a respective distance range 22i, a triangulation determination of the distance of objects 22 to 24 in the unfiltered image areas can additionally be carried out using the at least one stereo camera 5, 14. Here, a stereo correspondence determination can again take place, as already explained above.

Prior to each acquisition step of the operational procedure, a Region of Interest (ROI) can be specified as part of an image field that can be acquired via the respective camera 5i, 14i, for example a road area identified by appropriate object filtering in the forward direction 4.

Fig. 5 shows a side view of another embodiment of the image acquisition device 1. Components and functions that correspond to those already described above with reference to Figures 1 to 4. The points that have already been explained bear the same reference numbers and will not be discussed again in detail.

The use of the device 1 for determining and detecting a diffuse scattering medium along a flash path 28 between the flash source 7 and an exemplary surrounding object 2 is explained with reference to Fig. 5. The camera 5i of the stereo camera 5 is shown as an example in Fig. 5, which enables both triangulatory distance determination and time-of-flight (TOF) distance determination.

In the acquisition example shown in Fig. 5, the device 1 acquires a total of three distance ranges 22i, 222, 223 by specifying corresponding TOF image acquisition parameters. The first, nearest distance range 22i extends by a first distance di. The next further distance range 222 extends by a second distance d2, for which d2 = 2 x di. The next furthest, third distance range 22s extends by a third distance di, for which di = 3 x di.

The specification of image acquisition parameters for the three distance ranges 22i to 22s includes, in addition to the specification of the TOF image acquisition parameters "illumination start", "illumination duration", "detection start" and "detection duration", a specification of a flash intensity li for illuminating the respective distance range 22i. The flash intensity I2 for illuminating distance range 222 is four times greater than the flash intensity li for illuminating the first, nearest distance range 22i. The flash intensity I3 for illuminating the third, most distant distance range 223 in the example according to Fig. 5 is nine times greater than the flash intensity li. The flash intensity li therefore depends quadratically on a distance di within a total distance range 25 to be acquired. Assuming a constant reflectivity or scattering power of an ambient surface 29, the quadratic dependence of the illumination intensity L on the distance di results in the sensor array 6i of the camera 5i experiencing constant intensity illumination during image acquisition in the three distance ranges 22i, 222, and 223. This constant intensity illumination of the sensor array 6i only occurs if, along flash paths 3h, 3h, 313 between the flash source 7 and the environment in the distance ranges 22i to 22s, there is no significant scattering of the illumination flash 8 by the diffuse scattering medium 27. In other words: Inequalities in the intensity illumination of the sensor array 61 along the detection paths 3 Oi with the given quadratic dependence of the illumination intensity L on the distance di within the detection range 22i are an indication of the presence of the diffuse scattering medium 27.

The following cases can be distinguished: a) The diffuse scattering medium 27 is located within the currently detected distance range, i.e., within the distance range 222 in the case shown in Fig. 5. In this case, the diffuse scattering medium 27 is directly illuminated along the illumination path 3 h, and a corresponding scattering of the illumination light 8 by the diffuse scattering medium 27 into the sensor array 61 is measured during TOF detection, which leads to an increase in the intensity illumination along the detection path 302. b) The diffuse scattering medium 27 lies between the flash light source 7 and the sensor array 6i on the one hand, and the currently detected distance range, for example, the distance range 223, on the other. In this case, the illumination light is attenuated along the flash path 3h as it passes through the diffuse scattering medium 27. The illumination light backscattered from the detection range 22s towards the sensor array 61 along the detection path 30s is further attenuated due to additional scattering by the diffuse scattering medium 27. As a result, the intensity of the illumination of the sensor array 61 when detecting the distance range 223 is attenuated due to the cloud of diffuse scattering medium 27 present in the intermediate distance range 222.

A combination of an increased intensity illumination of the sensor array 61 when detecting the distance range 222, compared to the expected constant intensity illumination, and a reduction in the intensity illumination of the sensor array 61 when detecting the distance range 223 thus enables the identification of the diffuse scattering medium 27 within the distance range 222.

In the image acquisition devices 1 and 21 described above, the pixel sensitivity of sensor pixels of the various sensor arrays 61, ... can be controlled as a function of the incident light intensity, i.e., as a function of the intensity illumination, of the sensor array 61, ... . In particular, y-correction can be applied during image processing using the respective sensor array 61, ... . The exponent y used in the y-correction can vary in the range between, for example, 0.2 and 5. Depending on the value of y, a pixel sensitivity can then be adjusted to achieve a high dynamic range. This can be achieved either with high or low intensity illumination of the sensor array.

A selection of object distance ranges 22i to be detected can be adapted to a movement mode of the detection device 1 or 21 relative to the environment in certain operating modes of the detection device 1 or 21. In particular, the total distance range 25 can be adapted to a relative speed of the device 1, 21 to the environment. Adaptation to a reverse or cornering mode is also possible.

Alternatively or in addition to the operating procedure described above with reference to Figures 3 and 4, the respective device 1 or 21 can also be operated as follows:

By appropriately assigning, in particular, the flash illumination duration and the detection duration, the respective device 1, 21 uses the time-of-flight (TOF) principle to detect the time-of-flight (TOF) distance of at least one object 2 or 22 to 24. Parallel or sequentially, the stereo camera 5, ... of the device 1, 21 uses the triangulatory principle to detect the triangulatory distance of the respective object 2 or 22 to 24. The detection values "TOF distance" and "triangulatory distance" are then compared during operation of the device 1, 21.

In particular, it is possible to use the determined TOF distance as an input parameter for triangulation. This can especially improve correspondence analysis during triangulation. easy. In particular, a disparity range to be covered in the correspondence analysis can be narrowed down. The reverse case is also possible, i.e., that the triangulated distance is used as input for distance acquisition using the TOF principle. This can be used to determine or divide a given total distance range into segmented distance ranges 22i and to specify the TOF image acquisition parameters according to the distance.

The illumination intensity from the flash source 7 can be specified in such a way that a common and, in particular, scalar brightness control of all images of the Z-buffering of the first figures 3 and 4 is carried out, with a constant intensity illumination of the respective sensor arrays 6i, ... This brightness control can be used in the form of a control loop in the device 1, 21.

Calibration of the dependence of the illumination intensity of the flash light source on the distance, i.e., an illumination intensity function, can be carried out using a calibration setup with a scattering medium with a homogeneous scattering particle distribution.

The pixel sensitivity as a function of the incident light intensity can be set in the form of a proportional function or in the form of a non-linear function.

With reference to Fig. 6, a method for calibrating a camera system with three cameras 5i, 52 and 5s, which can be three cameras of the device 1 or of the device 21, is explained below. The cameras each have a sensor array 6i, 62, 63, as already explained above in connection with the descriptions of devices 1 and 21. The sensor array 6i can also be a CCD array.

To illustrate spatial relationships, an xyz coordinate system is used below in connection with Figure 6. The x-direction in Figure 6 runs to the right. The y-direction runs perpendicular to the plane of Figure 6 and into it. The z-direction in Figure 6 runs upwards.

The cameras 5i to 5s are arranged side by side in the area of a common camera arrangement plane, namely the xy-plane of figure 6, along a camera arrangement coordinate x.

The cameras 5i are designed as cameras with an object-side opening angle s of at most 30°. Edge-side limits 351, 352, 35s of this opening angle s of the respective cameras 5i, 52 and 5s are shown in Figure 6.

The 5i cameras are specifically designed as telephoto lens cameras with a focal length in the range of 10 mm.

To calibrate cameras 5i to 5s, three calibration markers are selected to define calibration marker points PA, PB, and Pc, respectively. These calibration marker points PA, PB, and Pc are positioned at different distances dA, dß, and dc from the camera's xy plane. The calibration marker points PA, PB, and Pc can be natural, depth-staggered marker points. a scene or artificially provided, staggered marker points in depth.

The calibration marker points PA, PB and Pc lie within all three opening angles, i.e. within the respective opening angle limits 351, 352 and 35s of the cameras 5i, 52 and 5s.

With regard to the relationship of the distances dA, dß, dc to the other length parameters shown in Figure 6, Figure 6 is not to scale. The distances dA, dß, and dc are depicted as excessively small in Figure 6.

The calibration marker points PA, PB, and Pc are all positioned at the same x- and y-coordinates. The calibration procedure does not require a specific degree of accuracy regarding the alignment of the x- and y-coordinates of these marker points. The depth of alignment is what matters.

For example, the distances dA, dß and dc can be: dA= 50 m, dß = 100 m, dc= 200 m.

This distance depth layering corresponds to a typical street scene in front of a vehicle carrying the camera system to be calibrated. The x-distances between the cameras can be, for example, 2 m (b = 2 m). The focal length of each camera 5i can be in the range of 10 mm. With a typical camera pixel size, corresponding to a camera pixel pitch in the range of 3 pm to 10 pm, the following results: A disparity of, for example, 2 mm, which corresponds to a distance that spans 200 pixels of the respective camera 5i. The relative disparity is then 200 pixels. At a distance dß of 100 m, the relative disparity is 100 pixels, and at a distance dc = 200 m, the relative disparity is 50 pixels.

The distances dA, dß, and dc each differ from each other by at least 20%. In the embodiment shown in Figure 6, this difference between the distances is even at least 50%.

Depending on the specific calibration procedure, more than three calibration marker points (Pi) can be specified, e.g., four, five, six, eight, ten, or even more. This can improve the accuracy of the calibration.

The focal length f of each camera 5i is illustrated in Figure 6 in comparison to the distances d to the marker points P. In the embodiment shown in Figure 6, cameras 5i, 52, and 5s each have the same focal length f. Depending on the design of the camera system, the focal lengths of cameras 5i can also differ from one another.

In the arrangement shown in Figure 6, the distances d are each more than four times the focal length f. Other ratios between the focal lengths f of the cameras 5i and the distances d are also possible.

The entrance pupils of cameras 5i, illustrated in Figure 6 at Oi, O2, and O3, have baseline lengths corresponding to the x-distances of these entrance pupils Oi to each other. An x-coordinate of the respective The camera entrance pupil Oi is assigned to the respective cameras 5i in Figure 6, labelled xi , X2 and X3.

During the calibration procedure, the cameras 5i image the calibration marker points PA, PB, and Pc and capture them in a camera image plane of the respective camera 5i, i.e., in the xy-plane. This is illustrated in Figure 6 for camera 5s by a schematic image construction, in which the images P'A, 3, P'B, 3, and P'c, 3, i.e., the calibration marker image points, are shown on the sensor arrays 6i as intersection points between rays that extend from the entrance pupils Oi to the respective calibration marker points PA, PB, and PC.

The differences Xi - Xk between the respective x-coordinates of the entrance pupils of cameras 5i and 5j are the baselines corresponding to the stereo cameras to which these two cameras 5i and 5j belong. This difference Xi - xk is also referred to as bienik.

One and the same calibration marker point, e.g., marker point PA, is depicted shifted by a disparity shift UA, depending on the distance dA, the focal length f, and the relationship between the x-coordinates of this marker point PA and the entrance pupil of camera 5i compared to this x-coordinate Xi. These disparity shifts for marker points PA, PB, and Pc are highlighted as examples for camera 53 in Figure 6 at UA,3, UB,3, and uc,3.

A difference between the disparity shifts in the mapping of each of the marker points by two cameras 5i and 5k of a The disparity difference between the camera pair is thus measured, using marker point PA as an example, as duA, ik = UA, i - UA, k = A. Corresponding disparities B and C result when imaging marker points PB, PC with the respective cameras 5i, 52, 5s. These disparity differences are also simply referred to as disparities.

During the calibration procedure, for each pairing i and k of two cameras, 5i, 5k, the disparity difference duA,ik (= UA,i - UA,k), duB.ii,- and duc.ii, of the image of the respective calibration marker point in the camera image plane xy, is measured. A total of nine measured disparities du result for the three marker points PA, PB, Pc and the three possible camera pairings 5ik, namely 512, 5 and 523.

Since the camera 52 is positioned at exactly the same x-coordinate X2 as the three calibration marker points PA, PB, PC, when imaging these calibration marker points PA, PB, PC by the camera 52, there are no x-distances of the calibration marker image points from this x-coordinate X2, so that by definition the camera 52 has disparity shift values UA, 2, UB, 2 and uc, 2 = 0.

The stereo imaging formula yields: d = fblen/du (I)

Solving for the disparity du, this yields, for example, for the calibration marker point PA and the cameras 5i and 5k: dUA,ik = UA,i - UA,k = f (Xi - Xk)/dA (II) The three cameras 51, 52, and 5s are not perfectly aligned with the camera arrangement plane xy; there are deviations from this plane, which is why calibration is performed. These deviations can be described analogously to the description of a ship's movement by the pitch angle, roll angle, and yaw angle of the camera system with cameras 5i to 63 compared to the arrangement plane xy, or by the corresponding pitch, roll, and yaw of each of these cameras 5i to 5s individually.

A corresponding positional error can be accounted for in the stereo imaging description of the equations above by including a tilting error 5ui - 5uk normalized to the focal length, i.e., a relative tilt between cameras 5i and 5k, each of the camera pairs. From equation (II) above, the following results, again using the calibration marker PA and cameras 5i and 5k as examples:

(UA,i - UA,k)/f = (Xi - Xk)/dA - (5ui - 5uk) (III)

The left side of the above equation (III) is known, with respect to the disparity, by measurement and, with respect to the camera focal length, by pre-calibration.

For the three calibration marker points PA, PB and Pc and the three possible camera pairings 512, 513 and 523, a total of nine equations of the above type (III) with nine unknowns result, namely the respective x-coordinates xi, X2, X3 of the entrance pupils Oi, O2 and O3 of the three cameras 5i, 52, 53, and the respective rotational errors 5ui, 5u2. and Sus and the z-distances dA, dß, dc of the calibration marker points PA, PB and Pc.

After measuring the disparities, it is therefore possible to determine these x-coordinates, the rotation errors and the distances of the calibration marker points using the additionally known camera focal lengths f.

During operation of device 1 or 21, selected objects 2 or 22 to 24 can be tracked after the cameras have been calibrated. Once one of the objects 2 or 22 to 24 has been detected using triangulatory distance detection and/or time-of-flight (TOF) distance detection, the TOF image acquisition parameters, in particular, but also the triangulatory image acquisition parameters (for example, the baseline length of a selected stereo camera, the focal length of the lenses of the respective stereo camera, or a disparity range to be monitored), can be preset so that the object, once detected, is also detected in subsequent image acquisitions. Relative movement of the detected object to device 1 or 21 can be taken into account, so that, for example, if it is expected that the object and device will approach each other, the distance of the selected distance range 22i is reduced during TOF detection. The light intensity of the flash source can be adjusted and, in particular, regulated according to the distance range of the object being tracked. Depending on the operating procedure of the device 1, 21, exactly one object can be tracked or several objects can be tracked simultaneously.

Provided that at least one object is tracked, intermittent tracking of the at least one tracked object and a scan over the entire distance range 25 can be performed, which increases the overall recording reliability with reasonable recording effort.

The image acquisition parameters for TOF acquisition of multiple distance ranges 22i, 22i+i can be adjusted so that detection via the respective camera occurs for exactly one detection period (same detection start time, same detection duration) for these distance ranges 22i, 22i+i. In extreme cases, this can apply to all distance ranges 22i within the total distance range 25.

When tracking objects, the extent of the respective distance range 22i, in which the object is expected, can be adjusted by appropriately modifying the flash illumination duration. For more precise object tracking, shorter flash illumination durations and shorter detection durations can be specified, with a corresponding increase in distance accuracy during time-of-flight (TOF) distance determination.

Device 1 can include a neural network or an AI module, which can be used to train the respective operating procedure and to improve image germination accordingly. ...

Claims

Patent claims 1. Method for calibrating a camera system with at least three cameras (5 i, 52, 5s) with known focal lengths (f), which are arranged next to each other in the area of a common camera arrangement plane (xy) along a camera arrangement coordinate (x) and whose object-side opening s angle is at most 45°, comprising the following steps: Selecting at least three calibration markers to specify one calibration marker point (PA, PB, PC), wherein the calibration marker points (PA, PB, PC) are arranged at different distances (dA, dß, dc) from the camera arrangement plane (xy), Imaging the calibration marker points (PA, PB, PC) with the cameras (5i, 52, 5s) to capture calibration marker image points (PA , PB', PC') in a camera image plane of the respective camera (5i, 52, 5s), For each pairing of two cameras (5i, 52, 5s), measure a disparity (A, B, C) of the image of the respective calibration marker pixel (PA , PB', PC') in the respective camera image plane, Determine, from the at least nine measured disparities (Aik, Bik, Cik) and the camera focal lengths (f), both — of distances (xi - Xk) between each of the cameras (5i, 5k) of a camera pair along the camera arrangement coordinate (x) as well as — due to positional errors (5ui - 5uk) of each camera pair (5 5k) resulting from deviations in the arrangement of the cameras (5i, 52, 5s). 2. Method according to claim 1, characterized in that the distances (A, dß, dc) of the calibration marker points (PA, PB, PC) differ from each other by at least 20%. 3. Method according to claim 1 or 2, characterized in that more than three calibration marks are provided for specifying more than three calibration mark points (PA, PB, PC,. ..) which are arranged at different distances ( A, dß, dc,. . .) to the camera arrangement plane (xy). 4. Camera system calibrated according to a method according to one of claims 1 to 3, wherein at least one of the cameras (5i, 52, 5s) is designed as a telephoto lens camera. 5. Camera system according to claim 4, wherein the camera pairs (5i, 5k) are designed as stereo cameras for triangulatory object distance determination within a spatially imaged environment. 6. Camera system according to claim 5, - with at least one flash light source (7, 13) for ambient illumination with controlled specification of a flash illumination start (t = 0, t = Uash) and a flash illumination duration (Atfiash) of successive illumination flashes, - where the cameras (5i, 52, 14i, 142, 15i, 152, 15s, 154, 20i, 202, 2O3, 2O4) of the stereo camera (5, 14, 15, 20; 5, 14) each have a sensor array (6) with controlled specification of a detection start (t = tß, t = tEi) and a detection duration (At _exp ) - with a control unit (10) that is connected to the cameras (5i, 52, 14i, 142, 15i, 152, 153, 154, 20i, 202, 2O3, 2O4) and the flash source (7, 13) are in signal connection, for the time-synchronized specification of the following image acquisition parameters: the flash illumination start (t = 0, t = tiiash) and the flash illumination duration (Atiiash) of the flash source (7, 13) as well as the detection start (t = tE, t = tEi) and the detection duration (Atexp) of the cameras (5i, 52, 14i, M2, 15i, 152, 153, 154, 20i, 202, 203, ₂₀₄ ). 7. Camera system according to claim 6 - with at least one additional flash light source (13) with controlled Specification of a flash illumination start time (t = 0, t = tfiash) and a flash illumination duration (Atfiash) of successive illumination flashes (8, 81) and/or - with at least one additional stereo camera (14) with two additional cameras spaced apart from each other (14i, M2) for triangulatory object distance determination within the encompassed environment. 8. Camera system according to claim 6 or 7, characterized in that the control unit (10) has an image processor for real-time processing. 9. Method for operating a camera system according to any one of claims 6 to 8, comprising the following steps: - Specifying the image acquisition parameters such that a first distance range (gz-1) is captured, - Capturing the distance range (gz-1) with the specified image capture parameters using the cameras of at least one stereo camera, - Specifying the image acquisition parameters such that a further distance area (gz) is captured which borders on or partially overlaps the previous distance area (gz-1) along the forward direction (4) in a capture progress direction (+z), - Capturing the further distance range (gz) with the last specified image capture parameters using the cameras of at least one stereo camera, - digital filtering of the last captured image to remove unstructured image areas, - Overwriting the image captured in the previous capture step with the most recently captured image in unfiltered image areas of the most recently captured image to generate an updated image, - Specifying the image acquisition parameters such that a further distance range (gz+1) is captured, which borders on or partially overlaps the previous distance range (gz) along the forward direction (4) in the acquisition progress direction (-z), - Repeat the last steps: capture, digital filter, overwrite and specify further distance ranges (224 to 22i) until a specified total distance range (25) is covered, which results from an overlay of the captured distance ranges (22i to 22max), - Displaying the most recently updated image. 10. Method for operating a camera system according to any one of claims 6 to 8, comprising the following steps: - Recording a Time of Flight (TOF) distance of at least one Object (2; 22 to 24) via a corresponding assignment of the flash illumination duration and the detection duration using the TOF principle, - Determining a triangulatory distance of the object (2; 22 to 24) using at least one stereo camera using the triangulatory principle, - Comparison of TOF distance and triangulatory distance. 11. Method according to claim 10, characterized in that the TOF distance is used as an input variable for the triangulatory acquisition. 12. Method according to one of claims 9 to 11, characterized in that the illumination intensity of the flash light source (7, 13) is adjusted to a distance range (220) to be detected. 13. Method according to claim 12, characterized in that the flash intensity depends quadratically on a distance (di) within the distance range (220) to be detected. 14. Method according to one of claims 9 to 13, characterized in that a pixel sensitivity of a sensor pixel of the sensor array (6) is controlled as a function of an incident light intensity of the flash light source (7, 13). 15. A method according to any one of claims 9 to 14, characterized in that a selection of object distance ranges (22i) to be detected is adapted to a movement mode of the camera system (1; 21) relative to the environment. ```