DE102024001342A1 - Correction of a synthetic vehicle surround view using corneal reflex image - Google Patents

Abstract

The invention relates to a method for generating a correction formula for creating a corrected synthetic panoramic image of the environment around a vehicle, comprising the steps of: capturing (S1) images by means of surrounding cameras (1) of the vehicle directed in different directions into the environment; capturing (S2) an image of a corneal area of an eye of an occupant of the vehicle by means of an interior camera (3), wherein the image of the corneal area has a corneal reflection image in which the environment of the vehicle is reflected; generating (S3) a reference image by rectifying and extracting the corneal reflection image; generating (S4) the correction formula by determining a mapping from the images of the surrounding cameras (1) or from a panoramic image generated from the images of the surrounding cameras (1) onto the reference image.

Description

The invention relates to a method for generating a correction instruction for creating a corrected synthetic all-round view image of the environment around a vehicle, and a vehicle comprising a plurality of environment cameras directed in different directions into the environment of the vehicle.

It is known in the prior art to display a virtual view of a vehicle's surroundings to the driver. This allows for the synthetic generation of a 360° view that can be grasped practically at a glance, without the driver having to frequently turn their head to gain an overall impression of the environment around the vehicle's entire horizon.

Often, so-called fisheye cameras are used in a camera-based environment monitoring system, often called SVS system (for "Surround View System"), to map the surroundings onto a camera image as completely as possible around the vehicle, in order to make it easier for the driver of the vehicle to perform maneuvers at slow speeds between obstacles, as is the case, for example, when parking or pulling out of a parking space.

The DE 10 2021 214 952 A1 In this context, this concerns a method for displaying a virtual view of a vehicle's surroundings, in particular a 360° or panoramic view, comprising at least the following steps: capturing a camera image of a part of the surroundings using a camera with a wide-angle lens, determining image information depending on the captured camera image, wherein the captured camera image is geometrically corrected, and displaying the virtual view by projecting the determined image information onto a virtual projection plane, wherein, in determining the image information, the resolution of the geometrically corrected camera image is increased in a first sub-area, in particular by means of a trained machine recognition method.

Typically, such cameras are positioned relatively low on the vehicle and sometimes require a low viewing angle to capture the vehicle's contours, for example, to create a realistic top-down view of the vehicle through artificial but consistent image stitching. This so-called "image stitching" results in images where objects close to the camera appear disproportionately large, while more distant objects are only faintly represented.

The object of the invention is to generate a synthetic environment image which does not exhibit such a strong overrepresentation of an object located near a given environment camera.

The invention is defined by the features of the independent claims. Advantageous further developments and embodiments are the subject of the dependent claims.

• - Aufnehmen von Bildern durch in verschiedene Richtungen in das Umfeld gerichtete Umfeldkameras des Fahrzeugs;
• - Aufnehmen eines Bilds eines Korneabereichs eines Auges eines Insassen des Fahrzeugs durch eine Innenraumkamera, wobei das Bild des Korneabereichs ein Hornhautreflexbild aufweist, in welchem sich das Umfeld des Fahrzeugs spiegelt;
• - Erzeugen eines Referenzbildes durch Entzerren und Extrahieren des Hornhautreflexbilds;
• - Erzeugen der Korrekturvorschrift, indem eine Abbildung von den Bildern der Umfeldkameras oder von einem aus den Bildern der Umfeldkameras erzeugten Rundumsichtbild auf das Referenzbild ermittelt wird.

A first aspect of the invention relates to a method for generating a correction instruction for creating a corrected synthetic all-around view image of the environment around a vehicle, comprising the steps of:

• - Capturing images using the vehicle's surround-view cameras, which are directed in different directions into the surroundings;
• - Taking an image of a corneal area of an occupant of the vehicle by an interior camera, wherein the image of the corneal area has a corneal reflection image in which the environment of the vehicle is reflected;
• - Generating a reference image by correcting and extracting the corneal reflection image;
• - Generating the correction formula by determining a mapping from the images of the surrounding cameras or from a panoramic image generated from the images of the surrounding cameras onto the reference image.

To create a 360° view, the individual images from the vehicle's surround-view cameras are stitched together using a process called "stitching" or "image stitching." However, the images from these cameras are typically distorted, for example, due to the use of wide-angle cameras, particularly so-called "fisheye" cameras. The goal is to generate a synthetic 360° view from these camera images and display it to the driver as if they were positioned above their vehicle in a bird's-eye view. This virtual view can be advantageously rendered in perspective, for example, for parking, so that the view of the vehicle includes not only a vertical but also a horizontal component.

The surround view assists the driver with slow maneuvers such as parking or exiting a parking space, as it provides a view from a third party in addition to their own perspective from inside the vehicle. This third party is not actually equipped with a camera. Therefore, it is a synthetic surround view, as it is based solely on images from the surrounding cameras. However, it does not display this direct and unmodified image itself. The panoramic image is generated as a corrected synthetic panoramic image through the process steps explained above and below, whereby the corrections compensate for distortions in the images from the surrounding cameras.

There are essentially two different approaches: Firstly, the correction can be applied to the images from the surround-view cameras, and the synthetic, and therefore also corrected, 360-degree image can be generated from these corrected images. Secondly, the images from the surround-view cameras can be combined to create a synthetic, uncorrected 360-degree image, which can then be corrected using the correction formula. The result of both alternatives should be essentially the same, thus allowing for some flexibility in the specific implementation.

This correction rule, however, must first be generated using a separate process. Ideally, all surrounding cameras are used to generate their respective images. These images are stored, at least temporarily, and simultaneously, an image of the corneal area of an occupant's eye is captured by an interior camera. For this purpose, the interior camera's field of view can be directed, for example, towards the occupant's head. Depending on the occupant's viewing direction, the interior camera image includes not only sections of the head or the occupant's head itself, particularly the eyes, but also a reflection on the cornea, a so-called corneal reflection. This reflection, depending on the occupant's viewing direction, mirrors the vehicle's surroundings. Thus, at least in certain areas, image information about the surroundings is obtained twice: once from the images of the surrounding cameras and again from the reflection on the occupant's cornea.

This advantageously utilizes the curvature of the cornea of a human eye and its property of reflecting ambient light over a wide angular range. The light reflected from surrounding objects passes through windows or open areas of the vehicle and reaches the cornea of the occupant, particularly the vehicle itself, where it is reflected in such a way that a spherically curved image is, in principle, visible as a reflection from the outside.

This reflection, which manifests as the corneal reflex image, is extracted and corrected to be as comparable as possible with the similarly corrected images from the peripheral cameras. This modified corneal reflex image then provides a reference point for determining a correction formula to correct, and in particular correct, the peripheral camera images or a panoramic image already generated from these images. The modified corneal reflex image can thus serve as a reference image for determining the corrective formula.

Naturally, only small areas of the total images from the surrounding cameras, or of the resulting panoramic image, overlap with the reference image. Therefore, image registration is necessary, preferably implemented using classic computer vision algorithms or image analysis algorithms.

The correction formula is determined by modifying the images from the surrounding cameras, or the resulting panoramic image, to achieve the desired reference image. This correction formula can be identified in various ways, such as using classical image processing algorithms. However, unlike a simple analytical solution, an iterative solution or search algorithm will be necessary, which is why the use of a machine learning model, such as an artificial neural network, is a viable option.

According to an advantageous embodiment, the correction rule is generated by training a machine learning model by using the reference image as a predetermined output variable and the images of the surrounding cameras or the 360° view generated from the images of the surrounding cameras as a predetermined input variable of the machine learning model.

Preferably, an artificial neural network is used as the machine learning model. This network is trained to obtain the desired correction formula as described above and below. For this purpose, input and output variables are specified and adjusted according to the parameters of the machine learning model, so that a mapping from the input variables to the output variables is obtained, as it corresponds to the specified training data.

According to a further advantageous embodiment, a spatial environment model is generated using sensor data from an environmental sensor unit of the vehicle, wherein the spatial environment model is transformed into a range image or a depth map, and the range image or the depth map is used as a predefined input variable of the machine learning model.

The environmental sensor unit is an independent sensor unit, separate from the interior and exterior cameras, and can be based on technologies such as lidar, radar, vision, or ultrasonic distance sensors. It generates a spatial model of the environment, which is then projected from 3D onto a depth image (also called a depth map) or a range image. This projection is aligned with the images from the exterior cameras and the corneal reflex image or reference image, based on their registration. For example, a transformer-based image generator or style transfer can be used to generate a corresponding image from the depth values in the depth image. The range image or depth map can then be used as a further predefined input for the machine learning model.

According to another advantageous embodiment, the range image or depth map is generated from the spatial environment model by a convolutional neural network.

According to a further advantageous embodiment, the reference image and/or the range image or the depth map are registered and cropped onto the corneal reflection image.

According to a further advantageous embodiment, the 360° image and/or the range image or the depth map and/or the corneal reflection image are projected onto a common projection surface.

According to another advantageous embodiment, the projection surface has at least partially the surface of a sphere.

Once the correction formula has been identified, it can be applied to generate a corrected synthetic surround-view image from images continuously captured by the vehicle's surround-view cameras or from a previously generated 360° view image. This corrected synthetic surround-view image can then be displayed to the driver, providing a realistic representation of the vehicle's surroundings from a virtual camera position. This approach is advantageously independent of ambient light and weather conditions and is based on the driver's current perception of their surroundings.

Another aspect of the invention relates to a vehicle comprising a plurality of surround-view cameras directed in various directions towards the vehicle's surroundings, an interior camera, and a processing unit, wherein the processing unit is configured to generate a synthetic surround-view image from the images of the surround-view cameras, to extract and rectify a corneal reflection image, in which the vehicle's surroundings are reflected, from an image of the interior camera which captures a corneal area of an occupant's eye, and to apply a pre-stored correction formula to the images of the surround-view camera, wherein the correction formula comprises a machine learning model which is pre-trained with rectified and extracted corneal reflection images as the respective predetermined output and with images of the surround-view cameras or with surround-view images generated from the images of the surround-view cameras as the respective predetermined input, further comprising a display unit configured to display a synthetic surround-view image corrected by means of the correction formula.

The correction formula is not necessarily implemented statically; it can also be dynamically adapted during the vehicle's product lifecycle. Recurring online recalibration can be implemented by repeatedly determining the generation level. If a similarity value between the synthetic surround-view image corrected by the correction formula, or the images from the surround-view cameras corrected by the correction formula, and an extracted and rectified corneal reflection image or reference image falls below a predefined range or threshold, delta learning can subsequently be implemented by retraining weights with an updated set of input and output variables as described above and below.

According to a further advantageous embodiment, the computing unit is designed to determine a similarity value between the synthetic all-round view image corrected with the correction rule or the images of the surrounding cameras corrected with the correction rule and an extracted and rectified corneal reflection image, and if the similarity value is outside a predetermined range, to recalibrate the machine learning model in operation of the vehicle by retraining.

According to a further advantageous embodiment, the computing unit is configured to use a reference image as a predefined output variable and the images from the surrounding cameras or the 360° view image generated from the images from the surrounding cameras and/or a range image or a depth map as predefined input variables for the machine learning model for retraining the machine learning model, and to generate the reference image by rectifying and extracting the corneal reflex image and the To generate the range image or depth map by creating a spatial environment model using sensor data from an environmental sensor unit of the vehicle and by transforming the spatial environment model into the range image or depth map.

Advantages and preferred further developments of the proposed vehicle result from an analogous and substantive transfer of the above statements made in connection with the proposed procedure.

Further advantages, features and details will become apparent from the following description, in which - possibly with reference to the drawing - at least one embodiment is described in detail.

• 1: Ein Verfahren zum Erzeugen einer Korrekturvorschrift für ein Rundumsichtbild gemäß einem Ausführungsbeispiel der Erfindung.
• 2: Ein Fahrzeug, an welchem die Korrekturvorschrift erzeugt wird und angewendet wird gemäß einem Ausführungsbeispiel der Erfindung.
• 3: Eine beispielhafte Korrektur für ein Rundumsichtbild gemäß einem Ausführungsbeispiel der Erfindung.

They show:

• 1 : A method for generating a correction instruction for a 360° view image according to an embodiment of the invention.
• 2 : A vehicle on which the correction instruction is generated and applied according to an embodiment of the invention.
• 3 An exemplary correction for a 360° view image according to an embodiment of the invention.

The representations in the figures are schematic and not to scale.

• - Aufnehmen S1 von Bildern durch in verschiedene Richtungen in das Umfeld gerichtete Umfeldkameras 1 des Fahrzeugs;
• - Aufnehmen S2 eines Bilds eines Korneabereichs eines Auges eines Insassen des Fahrzeugs durch eine Innenraumkamera 3, wobei das Bild des Korneabereichs ein Hornhautreflexbild aufweist, in welchem sich das Umfeld des Fahrzeugs spiegelt;
• - Erzeugen S3 eines Referenzbildes durch Entzerren und Extrahieren des Hornhautreflexbilds;
• - Erzeugen S4 der Korrekturvorschrift, indem eine Abbildung von den Bildern der Umfeldkameras 1 oder von einem aus den Bildern der Umfeldkameras 1 erzeugten Rundumsichtbild auf das Referenzbild ermittelt wird.

1 This document describes a method for generating a correction rule for creating a corrected synthetic 360° image of the area around a vehicle. The following steps are performed, with steps S1 and S2 preferably occurring simultaneously:

• - Recording S1 of images by the vehicle's surround-view cameras 1, which are directed in different directions into the environment;
• - Recording S2 of an image of a corneal area of an eye of an occupant of the vehicle by an interior camera 3, wherein the image of the corneal area has a corneal reflection image in which the environment of the vehicle is reflected;
• - Generating S3 of a reference image by rectifying and extracting the corneal reflex image;
• - Generating S4 of the correction rule by determining a mapping from the images of the surrounding cameras 1 or from a panoramic image generated from the images of the surrounding cameras 1 onto the reference image.

2 Figure 1 shows an example vehicle from a bird's-eye view. Such a view, with corresponding objects in the surroundings, can be displayed to the driver as a 360° view when parking. However, an inclined view of the pure bird's-eye perspective can also be used, particularly one with a horizontal component towards the vehicle itself. The vehicle has a multitude of surround-view cameras 1 distributed around its perimeter. These are wide-angle cameras designed to collectively capture a complete view of the surroundings. By fusing these individual images from the surround-view cameras 1, a continuous image of the surroundings can be generated. To generate the correction rule, as described in the procedure steps of... 1 As explained, an interior camera 3 is also used to capture a corneal area of an occupant's eye and extract a corneal reflection image from it, which includes a reflection of the surroundings, and to generate a reference image through appropriate rectification. Overlapping areas of the reference image and the images from the surrounding cameras 1 can be compared by registration and corresponding transformations. Such a correction formula, which is generated to correct the images from the surrounding cameras 1 or a previously generated surround view image, can be executed when the vehicle is put into operation, but can also be executed repeatedly, preferably especially when it is found that a currently used correction formula leads to corrections that bear too little resemblance to a currently generated reference image. The correction formula can also be updated for different viewing directions of the driver. For this purpose, a machine learning model in the form of an artificial neural network is preferably used, the parameters of which define the correction formula.

3 Figure 1 shows an example transformation using a previously determined correction formula. An image from one of the surround-view cameras 1 is shown in sub-image (A). This image is highly distorted, unnaturally bringing nearby objects into the foreground, while more distant objects or parts of objects appear smaller. By applying a pre-trained machine learning model to implement the correction formula, the original image from surround-view camera 1 can be rectified and corrected. The result is shown in sub-image (B). This is suitable for comparison with a reference image and for display on a display unit inside the vehicle for the driver. The driver thus receives a realistic representation and rendering of objects. in the vehicle's surroundings in the corrected synthetic 360° view image from a virtual camera perspective, which is created by correcting the stitched surrounding images, or as in the example of the 3 , is created by combining the individually corrected environment images.

Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for instance, the scope of protection, the possible applications, or the configuration of the invention. Rather, the preceding description and the description of the figures enable a person skilled in the art to implement the exemplary embodiments in concrete terms. With knowledge of the disclosed inventive concept, a person skilled in the art can make numerous modifications, for example, regarding the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

List of reference symbols

1

Surround camera

3

Interior camera

S1

To record

S2

To record

S3

Generate

S4

Generate

QUOTES CONTAINED IN THE DESCRIPTION

This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• DE 10 2021 214 952 A1 [0004]

Claims (10)

A method for generating a correction formula for creating a corrected synthetic surround-view image of the environment around a vehicle, comprising the steps of: - Capturing (S1) images by surround-view cameras (1) of the vehicle directed in various directions into the environment; - Capturing (S2) an image of a corneal area of an eye of an occupant of the vehicle by an interior camera (3), wherein the image of the corneal area includes a corneal reflection image in which the environment of the vehicle is reflected; - Generating (S3) a reference image by rectifying and extracting the corneal reflection image; - Generating (S4) the correction formula by mapping the images of the surround-view cameras (1) or a surround-view image generated from the images of the surround-view cameras (1) onto the reference image. Procedure according to Claim 1 , wherein the correction rule is generated by training a machine learning model by using the reference image as a given output variable and the images of the surrounding cameras (1) or the 360° view image generated from the images of the surrounding cameras (1) as a given input variable of the machine learning model. Procedure according to Claim 2 , wherein a spatial environment model is generated using sensor data from an environmental sensor unit of the vehicle, wherein the spatial environment model is transformed into a range image or a depth map, and the range image or depth map is used as a predefined input variable of the machine learning model. Procedure according to Claim 3 , where the range image or depth map is generated from the spatial environment model by a convolutional neural network. Method according to one of the preceding claims, wherein the reference image and/or the range image or the depth map is registered and cropped onto the corneal reflection image. Method according to one of the preceding claims, wherein the 360° image and/or the range image or the depth map and/or the corneal reflection image are projected onto a common projection surface. Procedure according to Claim 6 , where the projection surface has at least partially the surface of a sphere. The vehicle comprises a plurality of surround-view cameras (1) directed in various directions towards the vehicle's surroundings, an interior camera (3), and a processing unit. The processing unit is configured to generate a synthetic surround-view image from the images of the surround-view cameras (1), to extract and rectify a corneal reflection image, in which the vehicle's surroundings are reflected, from an image of the interior camera (3) capturing a corneal area of an eye of a vehicle occupant, and to apply a pre-stored correction formula to the images of the surround-view camera (1) based on a currently extracted and rectified corneal reflection image. The correction formula comprises a machine learning model pre-trained with rectified and extracted corneal reflection images as the respective predefined output and with images of the surround-view cameras (1) or with surround-view images generated from the images of the surround-view cameras (1) as the respective predefined input. The vehicle further comprises a display unit for displaying a synthetic image corrected by the correction formula. It is a 360° view image. Vehicle after Claim 8 , wherein the computing unit is configured to determine a similarity value between the synthetic surround view image corrected by the correction rule or the images of the surround view cameras (1) corrected by the correction rule and an extracted and rectified corneal reflection image, and if the similarity value is outside a specified range, to recalibrate the machine learning model during operation of the vehicle by retraining. Vehicle after Claim 9 , wherein the computing unit is configured to use a reference image as a predefined output variable and the images of the environment cameras (1) or the surround view image generated from the images of the environment cameras (1) and/or a range image or a depth map as predefined input variables of the machine learning model for retraining the machine learning model, and to generate the reference image by rectifying and extracting the corneal reflection image and to generate the range image or the depth map by generating a spatial environment model using sensor data from an environment sensor unit of the vehicle and by transforming the spatial environment model into the range image or the depth map.