DE102023130929B3 - Method for fusing camera images and vehicle - Google Patents

Abstract

The invention relates to a method for fusing camera images (B) from a plurality of cameras (4, 5) that are at least partially movable relative to one another, wherein each independently movable camera (4, 5) or each group of jointly movable cameras (4, 5) is assigned at least one inertial measuring unit (6) and is movable together with it, wherein in a learning phase acceleration data (A) of the inertial measuring units (6) are recorded together with camera images (B) of the assigned cameras (4, 5), a homography matrix is calculated and stored, a look-up table is created from the homography matrix, wherein in an online phase the look-up table is called up for certain acceleration values (A) determined by the inertial measuring units (6) and camera images (B) captured by the assigned cameras (4, 5) and provides homography matrices in order to calculate fused images.

Description

The invention relates to a method for fusing camera images according to the preamble of claim 1 and a vehicle according to the preamble of claim 3.

The fusion (stitching) of multiple camera images from vehicle cameras is complex if the camera positions may shift relative to each other over time.

US 2021/0179172 A1 describes methods and apparatus for determining a trailer hitch articulation angle, including a camera for capturing a first image and a second image, a steering sensor configured to detect a steering angle, a speed sensor configured to detect a vehicle speed, a processor configured to generate a bird's eye view of the first image and the second image through a perspective transformation, generating a trailer hitch model from the first image in response to the steering angle and vehicle speed indicating that a vehicle is traveling straight, and generating a current hitch model from the second image, determining a trailer hitch articulation angle in response to an angle difference between the trailer hitch model and the current hitch model, and a vehicle controller that controls the vehicle in response to the trailer hitch articulation angle.

DE 10 2019 202 269 A1 describes a method for calibrating a mobile camera unit of a camera system, wherein static position parameters (pus, pos, prs, pls) are fused with dynamic position parameters (pud, pod, prd, pld) of at least two edges and local coordinates (x, y, z) of the camera unit are calculated therefrom.

DE 10 2013 209 156 A1 describes a camera with a housing for recording the surroundings of a motor vehicle for use in a motor vehicle surround view system, wherein image data of the surroundings of the motor vehicle can be generated by means of the camera and a first interface is present via which the image data of the camera can be forwarded for further processing outside the housing. It is provided that the camera has an inertial sensor which is arranged inside the housing and the data of the inertial sensor can be forwarded via the first interface or a second interface for further processing outside the housing.

US 2017/0341583 A1 describes systems and methods for a tow vehicle towing a trailer having at least one imaging device. A method includes: receiving a first image stream having a plurality of first images from a first imaging device connected to the trailer; receiving a second image stream having a plurality of second images from a second imaging device connected to the vehicle; determining at least one common feature between a first image of the first images and a second image of the second images; determining a first distance from the first imaging device to the at least one common feature and a second distance from the second imaging device to the at least one common feature; and determining a position of the first imaging device relative to the vehicle based on the first distance, the second distance, and a known position and posture of the second imaging device.

The invention is based on the object of specifying a novel method for merging camera images and a novel vehicle.

The object is achieved according to the invention by a method for fusing camera images with the features of claim 1 and by a vehicle with the features of claim 3.

Advantageous embodiments of the invention are the subject of the subclaims.

A method is proposed for merging camera images from several cameras that are at least partially movable relative to one another. This can be understood as two or more cameras that are movable relative to one another. Likewise, two or more cameras can be combined into a group, with the cameras within a group being immobile relative to one another but movable relative to another camera or another group of cameras.

According to the invention, at least one inertial measuring unit is assigned to each independently movable camera or to each group of jointly movable cameras and is jointly movable with this, wherein in a learning phase acceleration data of the inertial measuring units are recorded together with camera images of the assigned cameras, a homography matrix is calculated and stored, wherein a look-up table is created from the homography matrix, wherein in an online phase the look-up table is called up for certain acceleration values determined by the inertial measuring units and camera images recorded by the assigned cameras and homographs fiematrices to calculate fused images.

In one embodiment, at least one of the cameras is designed as a mirror replacement camera on a cabin of a vehicle, wherein at least one further of the cameras is designed as a rear view camera on a chassis of the vehicle, wherein the cabin is movable relative to the chassis.

According to one aspect of the present invention, a vehicle is proposed, comprising a chassis and a cabin that is movable relative to the chassis, wherein at least one mirror replacement camera and at least one inertial measuring unit are arranged on the cabin, wherein at least one rear view camera and at least one inertial measuring unit are arranged on the chassis. The vehicle has a control unit that is configured to carry out the method described above.

In one embodiment, the vehicle is designed as a commercial vehicle or bus.

Furthermore, the vehicle can be designed as an autonomous or semi-autonomous vehicle.

According to the, by taking into account acceleration data determined by at least one inertial measurement unit and by knowing the influence of certain accelerations on the ratio of the overlap of the camera images, the calculation for merging the camera images can be accelerated.

The method according to the invention eliminates the need for online image processing and calculation to dynamically fuse images. Instead, historical data from the value range is used.

Embodiments of the invention are explained in more detail below with reference to drawings.

• 1 eine schematische Ansicht eines Nutzfahrzeugs mit einer Zugmaschine und einem Auflieger oder Anhänger, und
• 2 eine schematische Ansicht eines Verfahrens zum Fusionieren von Kamerabildern mehrerer Kameras

The following show:

• 1 a schematic view of a commercial vehicle with a tractor and a semitrailer or trailer, and
• 2 a schematic view of a method for merging camera images from multiple cameras

Corresponding parts are provided with the same reference numerals in all figures.

1 is a schematic view of a vehicle 1, in particular a commercial vehicle 1 with a tractor 2 and a semitrailer 3 or trailer. The tractor 2 has at least one mirror replacement camera 4 instead of an outside mirror. In particular, the tractor 2 has a mirror replacement camera 4 on each side instead of a respective outside mirror. The tractor 2 also has at least one reversing camera 5, which, like the at least one mirror replacement camera 4, is directed against a direction of travel. The tractor 2 also has at least one inertial measuring unit 6. For example, an inertial measuring unit 6 is arranged in or on a cabin 7 and a further inertial measuring unit 6 is arranged on a chassis 8 of the tractor 2. The reversing camera 5 is also arranged on the chassis 8 and is therefore movable relative to the mirror replacement camera 4 arranged on the air-sprung cabin 7.

2 is a schematic view of a method for fusing camera images B from several cameras 4, 5, for example the at least one mirror replacement camera 4 and the at least one rear view camera 5.

For merging several camera images B with overlapping image sections, a one-time calculation of the superposition and homography matrix is necessary if the ratio of the positions of cameras 4, 5 is static.

In a moving system with dynamic positions of the cameras 4, 5 (for example the position of a mirror replacement camera 4 on an air-sprung cabin 7 relative to the position of a rear view camera 5 on the chassis 8), no general calculation can be used. When the vehicle 1 moves, the cabin 7 can move differently in relation to the chassis 8 due to the accelerations. Overlapping image sections between the camera images B of the cameras 4, 5 are therefore also dynamic and increased computing effort is therefore required for the fusion.

According to the present invention, it is proposed to accelerate the calculation for merging the camera images B by taking into account acceleration data A determined by at least one inertial measuring unit 6 and based on the knowledge of the influence of certain accelerations on the ratio of the overlap of the camera images B.

The fusion (stitching) of images is done by determining interest points in several images. These different interest points are compared and placed on top of each other, and a transformation is then calculated (see e.g. https://courses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdf ). If you determine in a learning phase how the images will roughly overlap, then individual sections of the images can be viewed later to perform a faster online process. If the transformations (homography matrices) are already known for different camera positions, then these can be reused when the exact same camera positions are present again. If this is roughly known, then a faster and more accurate search algorithm can be achieved, since it only has to search for and compare interest points in a relatively small area.

In a learning phase in a step S1, historical data of the acceleration data A of the inertial measurement units 6 can be recorded together with the camera images B or image sections, in particular over the entire value range. From this, a homography matrix can be calculated once in a step S2 and saved in a step S3. In a step S4, a look-up table is created which can be called up in an online phase in a step S5 for certain acceleration values A and camera images B determined by the inertial measurement units 6 and provides homography matrices in order to calculate the fused images in a step S6. Two images are merged into a single image after a transformation. The computationally intensive part of the image processing thus takes place once in the learning phase and can be transferred to many vehicles. For example, steps S1 to S4 can take place during the development of vehicle 1. This also has the advantage that blurring can be reduced by taking the acceleration data A into account at an early stage.

The value range mentioned above results from possible extreme camera positions, e.g. with the driver's cab completely compressed and with the driver's cab completely extended. In this value range, an optimal homography matrix is calculated using optimizations. If it is known where the cameras are in relation to each other, then an optimization can be carried out in a limited value range. If the exact position is known, then a homography matrix that is already known for this position can be used directly without optimization.

The look-up table has an input and an output, where the input is formed by the position (or change in the relative position) of the two cameras based on acceleration data (integrated). The output is the corresponding homography matrix.

Claims (4)

Method for fusing camera images (B) from a plurality of cameras (4, 5) which are at least partially movable relative to one another, characterized in that at least one inertial measuring unit (6) is assigned to each independently movable camera (4, 5) or to each group of jointly movable cameras (4, 5) and is movable together with this, wherein in a learning phase acceleration data (A) of the inertial measuring units (6) are recorded together with camera images (B) of the assigned cameras (4, 5), a homography matrix is calculated and stored, a look-up table is created from the homography matrix, wherein in an online phase the look-up table is called up for certain acceleration values (A) determined by the inertial measuring units (6) and camera images (B) recorded by the assigned cameras (4, 5) and provides homography matrices in order to calculate fused images. procedure according to claim 1 , characterized in that at least one of the cameras (4) is designed as a mirror replacement camera (4) on a cabin (7) of a vehicle (1), wherein at least one further of the cameras (5) is designed as a rear view camera (5) on a chassis (8) of the vehicle (1), wherein the cabin (7) is movable relative to the chassis (8). Vehicle (1) comprising a chassis (8) and a cabin (7) movable relative to the chassis (8), wherein at least one mirror replacement camera (4) and at least one inertial measuring unit (6) are arranged on the cabin (7), wherein at least one rear view camera (5) and at least one inertial measuring unit (6) are arranged on the chassis (8), characterized in that the vehicle (1) has a control unit which is used to carry out the method according to claim 1 or 2 is configured. Vehicle (1) to claim 3 , characterized in that the vehicle (1) is designed as a commercial vehicle (1).

Images