DE102018203003A1 - Method and device for operating a camera monitor system for a motor vehicle - Google Patents

Abstract

A method for operating a camera monitor system (100) for a motor vehicle (101), wherein the camera monitor system (100) comprises two cameras (102, 103), which share a common side (106, 107) of the Each being adapted to provide an image (201, 202) of an environment (105) of the motor vehicle (101), the imaged environment (105) partially overlapping the images (201, 202) comprising:
Providing a default for a first image (201) of the first camera (102),
Transforming the first image (201) to a first transformed image (301) such that the provided specification is fulfilled in the transformed image (301),
Determining at least one picture element (303) on a plane (309) in the first transformed image (301),
Determining from a corresponding picture element (304) in at least one second picture (202) of the second camera (103) which corresponds to the at least one picture element (303),
- transforming the second image (202) in dependence on the determined image elements (303, 304) of the first transformed image (301) and the second image (202), so that another image element (305) in an edge region (308) of a transformed second image (302) connects to a corresponding further picture element (306) in an edge region (307) of the transformed first image (301).

Description

The invention relates to a method for operating a camera-monitor system for a motor vehicle, in particular for a truck. The invention further relates to a device which is designed to carry out the method.

Motor vehicles, such as trucks and buses, have conventional exterior mirrors. There are also mirror replacement systems are known in which by means of a camera and a monitor, the viewing areas of the usual exterior mirrors are displayed.

It is desirable to provide a method of operating a camera-monitor system for a motor vehicle that enables reliable operation. It is further desirable to provide an apparatus for operating a camera-monitor system that enables reliable operation.

The invention is characterized by a method for operating a camera monitor system for a motor vehicle and by a corresponding device which is designed to carry out the method.

In accordance with at least one embodiment, the camera-monitor system has two cameras which are assigned to a common side of the motor vehicle. The cameras are each designed to provide an image of an environment of the motor vehicle. The imaged environment of the images overlaps at least partially, in particular in an edge area.

A default for a first image of the first camera is provided. The first image is transformed to a first transformed image so that the provided default is met in the transformed image. The first image is transformed in particular depending on the provided specification. At least one pixel is detected at a level in the first transformed image. In particular, the plane is a plane in the 3D environment of the motor vehicle. A corresponding picture element is determined in a second picture of the second camera. The corresponding picture element corresponds to the at least one picture element of the first transformed picture. The second image is transformed in dependence on the determined image elements of the first transformed image and the second image, such that another image element in an edge region of a transformed second image adjoins a corresponding further image element in an edge region of the transformed first image.

The first image and the second image are each transformed in order to be able to be attached to one another by means of a so-called stitching in such a way that the further image element is represented contiguously in the edge regions. The transformed first image serves as a reference image. The second image is transformed, for example by means of homography, also called projective transformation. The reference positions in the two images whose match is used as an optimality criterion are obtained, for example, by feature recognition, feature comparison, for example SIFT, feature sorting, for example RANSAC, and / or by the least squares method. For this purpose, in particular four pixels and four corresponding pixels are determined.

By means of the method it is possible to display the recordings of the two different cameras on the common page in such a way that the represented environment can be intuitively detected by a user. Although the first image was taken by a different camera than the lower image, the two transformed images seamlessly adjoin each other in their peripheral areas.

For example, the first camera is a camera with a low viewing angle and a high focal length, for example with a Telelinse. By way of example, the second camera is a camera with a short focal length and a large viewing angle, for example with a wide-angle lens. For example, the first camera serves to the legal field of view classes 2 and 4 to represent on a monitor. For example, the second camera is used to the legal field of view classes 5 and partly 4 display. In particular, it is possible with the method to display larger areas of the environment than the statutory field of view classes provide.

The method makes it possible to implement the provision provided in the first image and, in addition, to connect the two transformed images so that the edge regions adjoin one another seamlessly. Thus, neither the first image nor the second image must be defined as a reference image. Various specifications for the first image and / or the second image or for the first transformed image and the second transformed image can be realized. In addition, the method can also be reliably carried out if the first image and / or the second image have a comparatively poor quality, for example in the dark. In addition, merging of the two transformed images is possible despite the different characteristics of the cameras used.

In accordance with at least one embodiment, the first image is lens-corrected before Transform. Alternatively or additionally, the second image is lens-corrected before transforming. For example, the input images of the two cameras are equalized so that known distortions due to the lenses used are compensated.

In accordance with at least one embodiment, the at least one picture element is determined on a ground plane assigned to the motor vehicle. For example, this is realized by a fixed restriction of the image region or by a detection from the vehicle movement while driving. This ensures that no object that is on the ground in the border area of the two images is lost due to the transformation. Objects standing on the ground are always displayed in the transformed images. On the ground standing objects always remain partly in the field of vision of the driver. This increases safety.

For example, the provided default for the transformed first image is a horizontal representation of a horizon. Alternatively or additionally, the provision provided is a representation of a bottom edge of the motor vehicle straight. Alternatively or additionally, the provision provided is a representation of an outer edge of the motor vehicle straight. Alternatively or additionally, the provision provided is a representation of a rear edge of a motor vehicle vertically. Alternatively or additionally, the provision provided is a representation of a predetermined element of the motor vehicle in the first transformed pixel. For example, the top corner of the trailer trailing edge should still be visible in the first transformed image. Other specifications are also possible, which result, for example, from legal requirements and / or user comfort analyzes, from settings of a user or depending on the driving situation.

In accordance with at least one embodiment, the first transformed image and / or the second transformed image are each further transformed in order to fulfill a further provision provided. For example, further transforming includes zooming, stretching, distorting or the like to display a desired area of the environment. The further transformations are combined according to embodiments with the previous transformations and thus executed simultaneously.

According to at least one embodiment, the transformation of the first image by means of homography. Alternatively or additionally, the transformation of the second image by means of homography. This allows a reliable and fast enough transformation.

In accordance with at least one embodiment, the second image is transformed in dependence on the determined picture elements in such a way that the determined picture elements in the transformed second picture and in the transformed first picture match. The coordinates of the pixels match in the transformed first image and in the transformed second image. As a result, a seamless connection of the two edge regions to each other is possible.

According to at least one embodiment, transforming the second image comprises transforming the detected image elements such that a line in the edge region of the second transformed image directly adjoins a corresponding line in the edge region of the first transformed image, in particular a bottom-level line. Thus, the two transformed images seamlessly connect to each other. In particular, lines remain connected at the ground level in the peripheral areas.

In accordance with at least one embodiment, the first transformed image and the second transformed image are displayed as an overall image on a common monitor of the camera-monitor system. For example, the first transformed image is displayed as the upper image of the overall image. The second transformed image is displayed as a lower image of the overall image. The method allows the overall image to be intuitively captured by a user, although the first transformed image is based on a different camera than the second transformed image.

The device for the motor vehicle, which is designed to carry out the method according to at least one embodiment, is for example part of a control unit for the motor vehicle (English: ECU, electronic control unit).

Advantages, features and developments described for the method also apply to the device and vice versa.

Further advantages, features and developments emerge from the following, explained in conjunction with the figures examples. Identical, similar and equally acting elements can be provided across the figures with the same reference numerals.

• 1 eine schematische Darstellung eines Kraftfahrzeugs mit einem Kamera-Monitor-System gemäß einem Ausführungsbeispiel,
• 2 eine schematische Darstellung eines Bildes einer ersten Kamera gemäß einem Ausführungsbeispiel,
• 3 eine schematische Darstellung eines Bildes einer zweiten Kamera gemäß einem Ausführungsbeispiel,
• 4 eine schematische Darstellung eines Gesamtbildes gemäß einem Ausführungsbeispiel, und
• 5 ein Ablaufdiagramm eines Verfahrens gemäß einem Ausführungsbeispiel.

Show it:

• 1 a schematic representation of a motor vehicle with a camera-monitor system according to an embodiment,
• 2 1 is a schematic representation of an image of a first camera according to an exemplary embodiment,
• 3 a schematic representation of an image of a second camera according to an embodiment,
• 4 a schematic representation of an overall image according to an embodiment, and
• 5 a flowchart of a method according to an embodiment.

1 shows a schematic representation of a motor vehicle 101 according to an embodiment from above. The car 101 For example, a truck with a trailer, a bus and / or a passenger car.

The car 101 has a camera monitor system 100 as a mirror replacement system. The camera monitor system 100 shows each side 106 . 107 of the motor vehicle 101 two cameras 102 . 103 on. The cameras 102 . 103 serve an environment 105 the respective assigned page 106 . 107 take.

According to further embodiments are only on one side of the pages 106 . 107 the two cameras 102 . 103 arranged.

The camera monitor system 100 has two monitors 104 on, each one of the pages 106 . 107 assigned. In operation, a section of the environment 105 on the website 106 , of the cameras arranged there 102 . 103 is recorded on the associated monitor 104 as a whole picture 400 shown. On the monitor 104 , the side 107 is assigned a section of the environment 105 as a whole picture 400 represented by the cameras 102 . 103 recorded is the side 107 assigned. Thus, the overall picture is different 400 on the two monitors 104 ,

The overall picture 400 on the monitor 104 is in particular adapted to display a mirror image of a conventional exterior gazebo and in particular further excerpts from the surroundings 105 represent, which are not mapped with conventional mirrors.

A device 110 is provided with the cameras 102 . 103 and the monitors 104 signal technology are coupled. The device 110 is set up, both the cameras 102 . 103 as well as the monitors 104 to control. For example, the device 110 a control unit or part of a control unit of the motor vehicle 101 ,

2 shows a first picture 201 For example, on the page 106 by means of the first camera 102 is recorded. The first camera 102 is for example a camera with a low viewing angle and a high focal length. The camera 102 takes a section of the environment 105 back on. For example, due to the optics used, there is a horizon 310 in the picture 201 shown curved. A bottom edge 311 of the motor vehicle 101 is for example in the picture 201 shown curved, although this runs straight on the real motor vehicle. For example, a tail edge 312 of the motor vehicle 101 curved or obliquely represented, although this runs vertically straight on the real motor vehicle.

3 shows a second picture 202 For example, on the page 106 by means of the second camera 103 is recorded. The second picture 202 shows you the first picture 201 different section of the environment 105 , The represented areas of the environment 105 but at least partially overlap.

After carrying out the following related to 5 described method is the overall picture 400 that on the two pictures 201 . 202 based on the monitor 104 represented as in 4 shown by way of example.

The method starts according to the embodiment of 5 with step 501 , In step 501 become the pictures 201 and 202 each lens-corrected. Thus, distortions due to the optics are digitally compensated.

The following will be in step 502 on one of the two pictures 201 . 202 applied a first transformation. In the described embodiment is on the first image 201 applied a transformation. The transformation can be represented in particular by a homography and / or does not change the lower image edge to be supported. The transformation in step 502 is performed to meet presets for the presentation of the first image. For example, the presets are at least one: a representation of the horizon 310 horizontal, a representation of the bottom edge 311 straight, a representation of the tail edge 312 vertical and a representation of a given element 313 ( 4 ), for example the upper corner of the trailer trailing edge, in a first transformed image 301 , The first transformed picture 301 becomes by means of the transformation in step 502 from the first picture 201 generated. The transformation in step 502 by means of Homography allows a natural-looking representation of the first image 201 ,

For example, the transformation is done manually, that is, by moving and changing the transformed image 301 until a desired display is achieved. alternatively or additionally, the transformation is carried out automatically with the aid of calibration patterns, for example four patterns. The calibration patterns are shown in the picture 201 recognized and then transformed to defined locations.

In step 503 become in the transformed first picture 301 four or more picture elements 303 annotated. It is also possible to have less than four picture elements 303 to anotieren if more information about the picture elements are available. In the not yet transformed picture 202 the second camera 103 become corresponding picture elements 304 ( 3 ) anotiert.

For example, annotation is done manually by selecting the pixels 301 and the corresponding picture elements 304 , Alternatively or additionally, the picture elements become 303 and the corresponding picture elements 304 automatically detected.

The picture elements 303 and the corresponding picture elements 304 lie in a plane 309 , in particular at ground level with respect to the motor vehicle 101 , In a calibration setup, it is possible that the plane 309 a wall or ceiling is when the cameras 102 . 103 are fastened differently than later in operation in the motor vehicle 101 ,

For example, the anotation is done semi automatically by means of one or more calibration patterns that are laid out on the floor. For example, the calibration patterns contain a total of at least four clearly and automatically assigned to each other and pairs as far as possible from each other distant points.

Also, a fully automated Anotation means of Aussortieralgorithmus is possible, which allows only to each other as suitably recognized feature points, in the plane 309 lie. This can be done, for example, by a previously defined restriction of the image region or by recognition from the vehicle movement while driving.

In step 504 becomes the second picture 202 to a transformed second image 302 transformed. This is done in particular by means of a homography transformation. This is done in particular on the basis of the picture elements 303 and the corresponding picture elements 304 calculated a transformation matrix. This will be on the second picture 202 applied to the transformed second image 302 to obtain. The transformation is done so that the second image 202 is so distorted that it to the already transformed first picture 301 followed. The transformed first picture 301 and the transformed second image 302 connect to each other in such a way that in particular all lines at ground level remain connected. Especially in a border area 307 of the transformed first image 301 and in an immediately adjacent border area 308 of the transformed second image 302 remains another picture element 305 , for example a line, and a corresponding further picture element 306 of the transformed first image 301 in the overall picture 400 connected ( 4 ). For example, the determined transformation matrix directly becomes directly on the second image 202 applied.

In an optional step 505 carried out according to embodiments further transformations. These can be combined with the transformation in step 504 or subsequently. The further transformations, for example bilinear and / or general polynomial, are applied to further specifications for the overall picture 400 and / or the transformed first image 301 and / or the transformed second image 302 to fulfill. The transformations in the step 505 are done so that the pixels in the border areas 307 and 308 stay in the same position, which adjoin one another. For example, the second picture becomes 202 so compressed that a larger section of the environment 105 in the overall picture 400 is pictured. Other further requirements are possible.

The initial homography, in particular after a lens equalization assumed to be ideal, ensures, for example, that on the one hand the given constraints and specifications for the transformed first image 301 be fulfilled and on the other hand the second picture 202 still through a homography in ground level 309 can be exactly gestitched.

The use of the picture elements 303 . 304 for the stitching, which is on the ground level 309 In particular, ensures that the two transformed images 301 . 302 at ground level 309 connect exactly to each other. At ground level, adjacent straight lines are still straight. As a result, objects standing on the ground always remain at least partially in the field of vision of the driver whenever possible. Thus, the security is guaranteed. The use of homograph transformations as transformations ensures that, on the one hand, no pixels are lost discontinuously and thus unintuitively and, on the other hand, straight lines remain straight in order to accurately estimate the direction of travel when maneuvering, for example. The homography ensures in particular that, in particular in one plane, for example the ground level 309 straight edges are also displayed straight.

The advantage that straight edges remain straight and do not appear kinked, according to embodiments, in particular only before step 505 that is, before the further transformation. After the step 505 This can, for example, depending on the transformation even for straight lines in the direction of travel, for example, for the median stripe guaranteed, but in particular not generally for all lines at ground level.

The camera monitor system 100 in particular also allows incorporation of further camera images, for example in the overall picture 400 to the legal field of view class 6 map.

To get the overall picture 400 be on the vehicle 101 first in the device 110 incoming pictures 201 . 202 the cameras 102 . 103 polynomial radially equalized by a barrel lens equalization with the parameters A, B, C and D.

On the whole picture 400 above to be displayed first image 201 the camera 102 Homography is used, for example, to ensure that the tail edge 312 vertical stands, the horizon 310 oriented horizontally, the upper corner 313 the trailer trailing edge is just visible, not too much of the trailer is visible in the picture and the perspective distortion is not too extreme. This homography is determined, for example, by four manually determined displacements of four manually defined points in the image, for example the upper corner of the trailer trailing edge, the motor vehicle 101 in the lower part of the picture, the horizon 310 at the outer end of the image and a point in the lower, relatively far outer region of the image to finally establish the amount of perspective distortion. For this example, a calibration test image is used, which was recorded in advance.

For example, below are the four pixels 303 and the corresponding pixels 304 in the two calibration test pictures 201 . 202 annotated. The pixels 303 and the corresponding pixels 304 agree. For example, four test points are placed on the floor as far away from each other as possible, so that they are still in both pictures 201 . 202 are visible.

On the whole picture 400 image to be displayed below 202 a homography transformation is applied so that the anotized pixels 303 and 304 exactly match.

For example, on the second picture 202 or the transformed second image 302 applied a vertical compression, where the upper row of pixels in the edge area 308 stays fixed.

The definition of the picture elements 303 and 304 is also possible automatically or semi-automatically as an alternative to manual anotation. For semi-automatic anotation, calibration patterns are designed to be taken by the device 110 or a software of the device 110 automatically in the pictures 101 and 102 be recognized. For fully automatic anotation no extra calibration points are necessary. The device 110 or the software of the device 110 is trained in the pictures 201 and 202 suitable elements, especially in the plane 309 to determine, by means of which the homography transformation can be performed. Such elements may be, for example, markings on the roadway, Gullideckel or the like. The procedure allows in the overall picture 400 exactly adjacent image edges and lines between all in the monitor 104 shown camera images 301 . 302 in the case of an indication of a section of each camera picture corresponding to particular criteria and specifications 201 . 202 ,

LIST OF REFERENCE NUMBERS

100

Camera Monitor System

101

motor vehicle

102

103 camera

104

monitor

105

Surroundings

106, 107

page

110

device

201, 202

image

301, 302

transformed picture

303

picture element

304

corresponding picture element

305

another picture element

306

corresponding additional picture element

307

Edge region of the transformed first image

308

Edge region of the transformed second image

309

level

310

horizon

311

bottom edge

312

rear edge

313

given element

400

overall picture

501 - 505

steps

Claims (10)

Method for operating a camera monitor system (100) for a motor vehicle (101), in which the camera monitor system (100) has two cameras (102, 103) which are a common side (106, 107) of the motor vehicle and each being adapted to provide an image (201, 202) of an environment (105) of the motor vehicle (101), the imaged environment (105) partially overlapping the images (201, 202), comprising: Providing a default for a first image (201) of the first camera (102), Transforming the first image (201) to a first transformed image (301) such that the provided specification is fulfilled in the transformed image (301), Determining at least one picture element (303) on a plane (309) in the first transformed image (301), Determining from a corresponding picture element (304) in at least one second picture (202) of the second camera (103) which corresponds to the at least one picture element (303), - transforming the second image (202) in dependence on the determined image elements (303, 304) of the first transformed image (301) and the second image (202), so that another image element (305) in an edge region (308) of a transformed second image (302) connects to a corresponding further picture element (306) in an edge region (307) of the transformed first image (301). Method according to Claim 1 comprising: lens dicing the first image (201) and / or the second image (202) before transforming. Method according to Claim 1 or 2 comprising: determining the at least one picture element (303) on a ground plane assigned to the motor vehicle (101). Method according to one of Claims 1 to 3 in which the provided specifications comprise at least one of: - representing a horizon (310) horizontally, - presenting a bottom edge (311) of the motor vehicle (101) straight, - presenting a tail edge (312) of the motor vehicle (101) vertically, and Representing a predetermined element (313) of the motor vehicle (101) in the first transformed image (301). Method according to one of Claims 1 to 4 comprising: further transforming the first transformed image (301) and / or the second transformed image (302) so that another provided default is met. Method according to one of Claims 1 to 5 in which at least one transform comprises homograph transforming. Method according to one of Claims 1 to 6 wherein transforming the second image (202) comprises: - transforming the second image (202) in response to the determined pixels (303, 304) such that the detected pixels (303, 304) in the transformed second image (302) and in the transformed first image (301). Method according to one of Claims 1 to 7 wherein transforming the second image (202) comprises: - transforming the second image (202) in response to the determined pixels (303, 304) so that a line in the edge region (308) of the second transformed image (302) is applied to a corresponding line in the border area (307) of the first transformed image (301) immediately followed. Method according to one of Claims 1 to 8th comprising: displaying the first transformed image (301) and the second transformed image (302) as an overall image (400) on a common monitor (104) of the camera monitor system (100). Device for a motor vehicle, which is designed, a method according to one of Claims 1 to 9 perform.

Images