FR3161049A1 - Method and device for training a depth prediction model to predict a depth for any distance traveled between the acquisition of two images. - Google Patents

Abstract

A method or device for learning a depth prediction model associated with a monocular vision system. The method comprises a first learning process (32) of the depth prediction model from first pairs of images, each first pair of images being formed from images of the same sequence of images acquired by a camera mounted in a vehicle and moving at a speed lower than a threshold speed in a three-dimensional scene, and a second learning process (33) of the depth prediction model from second pairs of images, each second pair of images being formed from images of the same sequence of images acquired by the camera mounted in the vehicle and moving at a speed higher than the threshold speed in the three-dimensional scene. Abstract: Figure 3

Description

Method and device for training a depth prediction model to predict a depth for any distance traveled between the acquisition of two images.

The present invention relates to methods and devices for learning a depth prediction model associated with a vision system on board a vehicle, for example in a motor vehicle. The present invention also relates to a method and a device for determining a depth and/or measuring a distance separating an object from a vehicle carrying a vision system.

Technological background

Many modern vehicles are equipped with so-called ADAS (Advanced Driver Assistance System). ADAS are passive and active safety systems designed to eliminate human error in the operation of all types of vehicles. ADAS uses advanced technologies to assist the driver while driving and thus improve their performance. ADAS uses a combination of sensor technologies to perceive the environment around a vehicle, then provides information to the driver or influences certain vehicle systems.

There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control, and automatic parking systems.

ADAS systems embedded in a vehicle are powered by data obtained from one or more on-board sensors such as, for example, cameras. These cameras are used to detect and locate other road users or possible obstacles around a vehicle in order to, for example:
- to adapt the vehicle's lighting according to the presence of other users;
- to automatically regulate the vehicle speed;
- to act on the braking system in the event of a risk of impact with an object.

A position of another user or an obstacle is, for example, determined by a vision system comprising a model for predicting a depth associated with a pixel or a distance separating the vision system from an object in a three-dimensional scene. Such a model is for example implemented by a convolutional neural network and learned using images, these images being obtained from a universal database, for example Kitti® or Sceneflow®. Kitti® for example provides images of a road environment in a city center, but such a database does not, however, include all the road environments in which a vehicle can move. Above all, the sequences of images acquired by the same moving camera, a sequence of images constituting in particular a video, are obtained when this camera makes movements of small amplitude, in particular of the order of a few centimeters. Indeed, the image sequences available in Kitti® are for example representative of images acquired by a moving camera, each image being acquired from a viewpoint approximately 15cm (fifteen centimeters) from the viewpoint of the previously acquired image, which corresponds to a vehicle moving at a speed close to 18km/h (eighteen kilometers per hour), the vehicle carrying the camera acquiring the images of the image sequence at a frequency of 30Hz (thirty hertz). Such images are useful for training a prediction model for a vehicle moving at a low speed in an urban environment, but are not suitable for a vehicle moving in another type of environment such as a fast lane or a motorway, the speed of the vehicle being significantly higher, for example of the order of 110 (one hundred and ten) or 130 km/h (one hundred and thirty kilometers per hour), and the other environment of the vehicle being also different. The training data is then unsuitable for training the prediction model for a vehicle traveling in this other type of road environment.

However, the quality of the training of the depth or distance prediction model is very important, the depths or distances predicted by the depth prediction model representing distances to which other users or obstacles present in the road environment of the vehicle carrying the vision system and ADAS are located. Indeed, the proper functioning of the driving assistance devices using this data depends on the quality of the data emitted by the vision system.

A vision system comprising a single camera, hereinafter called a monocular vision system, uses several consecutive images to predict depths associated with pixels of one of the images acquired by the camera using the depth prediction model. It is then important to train this depth prediction model with images or sequences of images acquired in conditions similar to those encountered when the vehicle is moving, in particular on the road network and at different speeds of movement of the vehicle carrying the vision system.

Summary of the present invention

An object of the present invention is to solve at least one of the problems of the technological background described above.

Another object of the present invention is to improve the learning phase of a depth prediction model from sequences of images acquired by a camera from viewpoints located at different distances from each other.

Another object of the present invention is to improve road safety, in particular by improving the operational safety of ADAS systems supplied by data obtained from a camera of a vision system.

According to a first aspect, the present invention relates to a method for learning a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, the vision system comprising a camera arranged so as to acquire an image of a three-dimensional scene of an environment outside the vehicle,
the method being implemented by at least one processor, and being characterized in that it comprises the following steps:
- reception of image sequences, each image sequence comprising a set of images acquired by the camera at different consecutive acquisition time instants separated by the same time interval, each image sequence being classified according to a speed of the vehicle during the acquisition of the images of the sequence in a first class when the speed of the vehicle is lower than a threshold speed and in a second class when the speed of the vehicle is higher than the threshold speed;
- for each sequence of images included in the first class, first training of the depth prediction model from first pairs of images, each first pair of images comprising a first image and a second image selected from each sequence of images included in the first class;
- for each sequence of images included in the second class, second training of the depth prediction model from second pairs of images, each second pair of images comprising a third image and a fourth image selected from each sequence of images included in the second class.

The method advantageously makes it possible to learn the depth prediction model in two stages. Firstly, the depth prediction model is learned from image sequences whose images are acquired from a camera on board a vehicle moving at a speed lower than the threshold speed, i.e. at a reduced speed. Secondly, the depth prediction model is learned from image sequences whose images are acquired from a camera on board a vehicle moving at a speed higher than the threshold speed, i.e. at a high speed. Thus, the prediction model is learned from image sequences representative of different vehicle movement speeds; it is then able to accurately predict depths or distances when the vehicle is moving at a speed whether low or high. In addition, the image sequences used during learning are, for example, acquired by the vehicle itself and are therefore perfectly representative of real driving conditions of the vehicle, and therefore of three-dimensional scenes actually observed by the camera on board the vehicle.

According to a variant of the method, the most recent image in each sequence of images is assigned in each first pair of images as being the first image and each image acquired prior to the first image is assigned in a first pair of images as being the second image, each second image being different in each first pair of images.

The first pairs of images thus obtained comprise images acquired by a camera positioned at viewpoints positioned at a different distance for each first pair of images obtained from the same image sequence. The prediction model is then learned from pairs of images acquired from viewpoints positioned at a variable distance.

According to another variant of the method, the first learning comprises several iterations, each iteration using a different first pair of images corresponding to the first pair of images for which a time interval separating the acquisition of the first image and the second image is the smallest among the first pairs not used during a previous iteration, the depth prediction model being learned from the first pair of images used.

At each iteration, the distance separating the image acquisition viewpoints of a first pair of images is increasing, the prediction model is thus first learned for a small distance separating the image acquisition viewpoints then for increasingly large distances, until it is capable of predicting reliable depths for a large distance, this large distance being of the order, for example, of a multiple of the small distance, the multiple being equal to the number of iterations.

According to a further variant of the method, the first pairs are generated by selecting the first image as corresponding to the most recent image in each sequence of images and the second image as corresponding to an image prior to the first image in each sequence of images, the second image being selected according to a number of time intervals separating the acquisition of the first image from the acquisition of the second image, the number of intervals being determined according to a determined probability law.

The probability law thus makes it possible to control an occurrence of use of the first pairs of images, in particular according to the number of time intervals between the acquisition times of the images of the first pair of images. It is then possible to favor or not the first pairs of images comprising images acquired at few time intervals or on the contrary at many time intervals. The sensitivity of the depth prediction model to the number of time intervals and therefore to the distance separating the acquisition viewpoints of the images of the first pairs of images is thus adjusted.

According to yet another variant of the process, the probability law is obtained by the following function:

with :
• a probability associated with a noted interval ,
• a standard deviation.

Such a probability distribution is representative of a normal distribution whose mean is zero. This probability distribution favors the use of first pairs of images comprising images acquired at short time intervals. The variation of the standard deviation also allows for adjustment. Indeed, increasing the standard deviation then makes it possible to reduce the occurrence of the first pairs of images comprising images acquired at short time intervals and, conversely, to increase the occurrence of the first pairs of images comprising images acquired at long time intervals.

According to yet another variant of the method, the threshold speed is determined as a function of a target distance and a frequency of image acquisition by the camera. This target distance makes it possible, for example, to distinguish between image sequences comprising images acquired when the vehicle is moving at low speed, for example in an urban environment, and image sequences comprising images acquired when the vehicle is moving at higher speed, for example in an extra-urban environment, on a road with restricted access and on a motorway.

According to an additional variant of the method, the depth prediction model is learned by minimizing a loss error determined from each pair of images of a plurality of pairs of images by comparing the images of each pair of images to images reconstructed from the images of each pair of images and depths predicted by the depth prediction model and associated with pixels of the images of said each pair of images, the plurality of pairs of images corresponding to the first pairs of images during the first learning and to the second pairs of images during the second learning.

Determining this loss error allows self-supervised training, i.e., using only images acquired by the on-board camera and without requiring the use of annotated data coming, for example, from another on-board system such as a LiDAR®. The reconstruction of the two images is called bilateral reconstruction. It allows both the depth prediction model to be trained, but also, if necessary, a model for predicting the movement of the camera in the three-dimensional scene between the two time instants of image acquisition of the first pair of images.

According to a second aspect, the present invention relates to a device configured to learn a depth prediction model by a vision system on board a vehicle, the device comprising a memory associated with at least one processor configured to implement the steps of the method according to the first aspect of the present invention.

According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

Such a computer program may use any programming language and be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium may include a storage medium, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording medium or a hard disk.

Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from a network such as the Internet.

Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question.

Brief description of the figures

Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 7, in which:

FIG. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention;

FIG. 2 illustrates a flowchart of the different steps of a method for determining the depth of a pixel in an image using a depth prediction model associated with a vision system embedded in the vehicle of the FIG. 1 , according to a particular and non-limiting embodiment of the present invention;

FIG. 3 illustrates a flowchart of the different steps of a process for learning the depth prediction model used in the process of the FIG. 2 , according to a particular and non-limiting embodiment of the present invention; and

FIG. 4 schematically illustrates a device configured to learn a depth prediction model by an on-board vision system in the vehicle of the FIG. 1 , according to a particular and non-limiting embodiment of the present invention;

FIG. 5 schematically illustrates the generation of first pairs of images from a sequence of images acquired by the camera on board the vehicle of the FIG. 1 , according to a particular and non-limiting embodiment of the present invention;

FIG. 6 illustrates a flowchart of different steps of a method of training the depth prediction model from a pair of images used in the method of training the depth prediction model of the FIG. 3 , according to a particular and non-limiting embodiment of the present invention; and

FIG. 7 illustrates a graph representing a distance traveled by the vehicle of the FIG. 1 depending on its speed and a number of time intervals between the acquisition of two images, according to a particular and non-limiting exemplary embodiment of the present invention.

Description of examples of implementation

A method and a device for learning a depth prediction model implemented by a convolutional neural network associated with an on-board vision system in a vehicle will now be described in what follows with joint reference to figures 1 to 7. The same elements are identified with the same reference signs throughout the description which follows.

The terms “first(s)”, “second(s)” (or “first(s)”, “second(s)”), etc. are used in this document by arbitrary convention to identify and distinguish different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

For the entire description, reception of an image or a sequence of images means the reception of data representative of an image or respectively a sequence of images. Similarly, for the determination of a depth or an error, we mean the determination of data representative of a depth or an error. These shortcuts are only intended to simplify the description, however, since the methods described below are implemented by one or more processors, it is obvious that the input and output data of the different steps of a method are computer data.

According to a particular and non-limiting example of embodiment of the present invention, the depth prediction model is learned in a learning phase comprising a first and a second learning phase.

The first training of the depth prediction model is carried out from first pairs of images, each first pair of images being formed from images of the same sequence of images acquired by a camera on board a vehicle and moving at a speed lower than a threshold speed in a three-dimensional scene.

The second training of the depth prediction model is carried out from second pairs of images, each second pair of images being formed from images of the same sequence of images acquired by the camera on board the vehicle and moving at a speed greater than the threshold speed in the three-dimensional scene.

There FIG. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

The vehicle 10 is located in an environment 1 corresponding, for example, to a road environment formed of a network of roads accessible to the vehicle 10.

In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, an electric motor(s) or a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as a car, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

The vehicle 10 advantageously comprises at least one on-board camera 11, configured to acquire images of a three-dimensional scene taking place in the environment of the vehicle 10 from a current observation position. The camera 11 forms a monocular vision system if it is used alone as illustrated in the FIG. 1 . The present invention is however not limited to a monocular vision system comprising a single camera but extends to any vision system comprising at least one camera, for example 1, 2, 3 or 5 cameras.

Camera 11 has intrinsic parameters, including:
- a focal length,
- distortions which are due to imperfections in the optical system of the camera 11,
- a direction of the optical axis of the camera 11, and
- a resolution.

The intrinsic parameters characterize the transformation which associates, for an image point, subsequently called a “point”, its three-dimensional coordinates in the frame of reference of the camera 11 with the pixel coordinates in an image acquired by the camera 11. These parameters do not change if the camera is moved.

Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account, that their correction is processed at the time of image acquisition or at the time of calibration.

The camera 11 is arranged so as to acquire an image of a three-dimensional scene according to a defined point of view, the point of view is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10 as illustrated in the FIG. 1 The point of view of the camera 11 is then fixed in the frame of reference of the vehicle 10, and when the vehicle 10 moves, the point of view of the camera 11 also moves in the frame of reference of the observed three-dimensional scene which is located in the environment of the vehicle 10.

The camera 11 for example acquires images of a three-dimensional scene located in front of the vehicle 10, the first camera 11 covering an acquisition field 12. An object 13 is placed in the acquisition field 12 of the camera 11, defining an occlusion field 14 for the vision system, an object present in the occlusion field 14 not being observable by the camera 11 from its current observation position.

It is obvious that it is possible to use such a vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently, the invention not being limited to the observation of a three-dimensional scene taking place in front of the vehicle 10 carrying the camera 11.

According to a particular embodiment, the camera 11 is of the “wide-angle” type, a wide-angle camera being, for example, equipped with a lens designed to acquire an image representative of a three-dimensional scene perceived according to a wider field of vision than that of a standard camera, also sometimes called a panoramic lens. In other words, a wide-angle lens makes it possible to capture a larger portion of the three-dimensional scene taking place in front of or around the camera, which is particularly useful in situations where it is necessary to include more elements in the frame of the image acquired by this camera. The angle α of the field of vision of the camera 11 is, for example, equal to 120°, 145°, 180° or 360°, whereas a standard camera offers, for example, an open field of vision following an angle of 45° or less. Such a camera 11 corresponds, for example, to a camera equipped with mirrors or to a “fisheye” camera (in French “fish eye”). Wide-angle lenses have a shorter focal length than standard lenses, making them suitable for capturing images of landscapes, architecture, road intersections, or any other subject requiring a broad perspective. Wide-angle cameras are, for example, used to capture immersive and dynamic images with an extended depth of field.

An image acquired by the camera 11 at an acquisition time instant is presented in the form of data representing pixels characterized by:
- coordinates in the image; and
- data relating to the colors and brightness of objects in the observed scene in the form, for example, of RGB colorimetric values (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness).

Each pixel of the acquired image is representative of an object in the three-dimensional scene present in the field of vision of the camera 11. Indeed, a pixel of the acquired image is the smallest visible unit and corresponds to a luminous point resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When the light strikes this object, photons are emitted or reflected, captured by a photosensitive sensor of the camera 11 after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image faithfully representing the physical object observed by the camera 11. An image point previously presented is thus a point on a surface of an object in the three-dimensional scene.

When the vehicle 10 is in motion, then two images acquired by the camera 11 at two distinct time instants represent views of the same three-dimensional scene taken from different viewpoints or observation positions, the observation positions of the camera 11 being distinct. On this three-dimensional scene are for example:
- buildings;
- road infrastructure;
- other users or stationary objects, for example a parked vehicle; and/or
- other users or moving objects, for example another vehicle, a cyclist or a moving pedestrian.

The distance separating the different acquisition viewpoints depends in particular on the speed of the vehicle 10 and the frequency of image acquisition by the camera 11. Note that if the vehicle moves at a constant speed and the camera 11 regularly acquires images, that is to say at a constant frequency, then the distance separating viewpoints associated with consecutively acquired images is constant. If the speed of the vehicle varies, this distance also varies proportionally. Thus, when the vehicle 10 moves at low speed, then the distance separating the viewpoints during acquisition of two consecutive images by the camera 11 is small, and conversely, when the vehicle 10 moves at high speed or high speed, then the distance separating the viewpoints for acquisition of two consecutive images by the camera 11 is high.

According to a particular exemplary embodiment, an image acquired by the camera 11 comprises a distortion equal to 0.5%, 0.8% or greater than 1%. The measurement of such a distortion corresponds to the determination of a ratio between:
- the maximum spacing of a pixel of the image from a straight line of the first three-dimensional scene whose image is a line touching the longest edge of the first image, either at the center of the edge of the image, or at the corners of the edge of the image, and
- the length of this edge.

Commonly, distortion is considered, in the world of photography, as:
• negligible if it is less than 0.3%,
• not very sensitive if it is between 0.3% or 0.4%,
• sensitive if it is between 0.5% and 0.6%,
• very sensitive if it is between 0.7% and 0.9%, and
• annoying if it is greater than or equal to 1% or more.

A barrel distortion is characterized by a positive percentage, while a crescent distortion is characterized by a negative percentage.

Each image acquired by the camera 11 is for example sent to a processor, for example a computer of a device equipping the vehicle 10, or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10. It is then used during the implementation of a method for determining a depth of one of its pixels and/or during the implementation of a method for learning a depth prediction model associated with this camera of the vision system.

There FIG. 2 illustrates a flowchart of the different steps of a method 2 for determining the depth of a pixel of an image by a depth prediction model implemented by a convolutional neural network associated with a vision system on board a vehicle, for example in the vehicle 10 of the FIG. 1 , according to a particular and non-limiting embodiment of the present invention. The method 2 is for example implemented by a device of the vision system on board the vehicle 10 or by the device 4 of the FIG. 4 .

In a step 21, two images acquired by the camera 11 are received.

In a step 22, depths associated with a set of pixels of one of the two received images are determined by the depth prediction model from the two received images.

Each determined depth then corresponds to a distance separating the vehicle 10 or a part of the vehicle 10 from an object of the three-dimensional scene with which a pixel is associated, the determination of a depth of a pixel then corresponding to a measurement of a distance separating an object from the vehicle carrying the vision system.

If the ADAS uses these depths or distances as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases significantly, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident.

There FIG. 3 illustrates a flowchart of the different steps of a method for learning the depth prediction model used in a method for determining the depth of a pixel of an image, for example in method 2 of the FIG. 2 , according to a particular and non-limiting embodiment of the present invention.

The learning method 3 is for example implemented by the device on board the vehicle 10 implementing the method for determining a depth for the vision system on board a vehicle 10 or by the device 4 of the FIG. 4 .

In a step 31, image sequences are received. Each image sequence comprises a set of images acquired by the camera 11 at different consecutive acquisition time instants separated by the same time interval. For example, each received image sequence comprises six images acquired by the camera 11.

According to a particular embodiment, the acquired images are filtered beforehand so that no image includes a pixel corresponding to a dynamic object, i.e. to an object in motion in the three-dimensional scene observed by the camera at the time instant of acquisition of an image. Indeed, the presence of pixels corresponding to a dynamic object in the three-dimensional scene deteriorates the learning, a monocular vision system having difficulty estimating a depth for such a pixel because a disparity or a determined optical flow between pixels of two images associated with this same object then includes two components, a first component resulting from a movement of the camera 11 in the three-dimensional scene, i.e. of the vehicle 10, and a second component resulting from a specific movement of the dynamic object in the observed three-dimensional scene.

In another example, the camera is always moving between two consecutive acquired images. Indeed, to predict depths, the monocular vision system needs to be moving in order to observe the three-dimensional scene from different points of view.

Note that in the case where a sequence includes images comprising pixels linked to dynamic objects or images acquired from the same point of view, it is possible to filter them with a filtering process or with the help of a technician. Indeed, sorting such a sequence does not require an analysis of each image in the image sequence but an analysis of the image sequence itself, a sequence of images corresponding to a video.

For the remainder of the description, the acquired images are considered to be of the same definition. Indeed, being acquired by the same camera 11, their definition does not have to be different. However, in the case where the definition of images in the same sequence is different, it is possible to add an additional step consisting of resizing or cropping certain images to obtain a sequence of images comprising only images of the same definition, that is to say comprising the same number of pixels according to their height and according to their width.

Each image sequence is classified according to a vehicle speed 10 during the acquisition of the images of the sequence:
• in a first class when said speed of vehicle 10 is lower than a threshold speed, and
• in a second class when said speed of vehicle 10 is higher than the threshold speed.

The speed of the vehicle 10 is for example associated with each image or each sequence of images, the speed being for example obtained from systems embedded in the vehicle 10, for example from an odometer or a geolocation system.

According to a particular exemplary embodiment, the threshold speed is determined as a function of a target distance and an image acquisition frequency by the camera 11. The threshold speed is then determined, for example, by the following function:
v _threshold = d _target / f _acquisition , with:
• v _threshold the threshold speed,
• d _targets the target distance, and
• f _acquisition the frequency of image acquisition by the camera 11.

The image sequences classified in the first class thus correspond to image sequences acquired at low speed, while the image sequences classified in the second class correspond to image sequences acquired at high speed.

The aim of this learning method 3 is then to learn the depth prediction model from image sequences acquired at low speed, i.e. images acquired from viewpoints that are not very distant, then progressively to learn the depth prediction model from image sequences acquired at higher speed, i.e. images acquired from more distant viewpoints. Learning method 3 is thus divided into a first learning process, then into a second learning process.

For each sequence of images included in the first class, in a step 32, the depth prediction model is learned in a first training from first pairs of images, each first pair of images comprising a first image and a second image selected in each sequence of images.

Learning a depth prediction model from images is known to those skilled in the art. An example of a method 6 for learning the depth prediction model from a pair of images is notably described below with regard to the FIG. 6 .

According to a first variant of the learning method 3, the most recent image in each sequence of images is assigned in each first pair of images as being the first image and each image acquired prior to the first image is assigned in a first pair of images as being the second image, each second image being different in each first pair of images.

According to a particular embodiment of the first variant, the first learning comprises several iterations, each iteration using a different first pair of images corresponding to the first pair of images for which a time interval separating the acquisition of the first image and the second image is the smallest among the first pairs not used during a previous iteration, the depth prediction model being learned from the first pair of images used.

There FIG. 5 schematically illustrates the generation of first pairs of images from a sequence of images acquired by the camera 11 on board the vehicle 10, according to a particular and non-limiting exemplary embodiment of the present invention.

An image sequence 500 comprises, according to this particular embodiment, seven images acquired temporally in a specific order, the camera 11 having acquired each of the images of the image sequence 500 in the following order: image 51, image 52, image 53, image 54, image 55, image 56 and lastly image 57. Image 57 is therefore the most recent image of the image sequence 500, i.e. the last image acquired temporally in the image sequence 500, while image 51 is the oldest image acquired in the image sequence 500, i.e. the first image acquired temporally in the image sequence 500.

During the different iterations, the prediction model is learned from first pairs of images.
• During the first iteration, the first image pair 510 comprises as first image the most recent image corresponding to image 57 and the second image for which a single time interval separates its acquisition from the acquisition of the first image, here image 57, the second image then corresponding to image 56.
• During the second iteration, the first pair of images 520 comprises as first image the most recent image still corresponding to image 57 and the second image for which the smallest or minimum time interval separates its acquisition from the acquisition of the first image among the images of the sequence of images 500 not previously used, the second image then corresponding to image 55. Indeed, two time intervals separate the instant of acquisition of image 57 from the instant of acquisition of image 55 and image 56 was used during the previous iteration.
• During the third iteration, a first pair of images 530 is formed and includes image 57 as the first image and image 54 as the second image.
• During the fourth iteration, a first pair of images 540 is formed and includes image 57 as the first image and image 53 as the second image.
• During the fifth iteration, a first pair of images 550 is formed and includes image 57 as the first image and image 52 as the second image.
• During the sixth and final iteration, a first pair of images 560 is formed and includes image 57 as the first image and image 51 as the second image.

The vehicle 10 traveling at a certain speed at the time of acquisition of the images of a sequence of images, the latter and therefore the camera 11 on board the vehicle 10, has traveled a determined distance between each point of view of acquisition of an image of the same sequence of images.

There FIG. 7 illustrates a graph representing a distance traveled 'd' by the vehicle 10 as a function of its speed V ₁₀ and a number 'i' of time intervals between the acquisition of two images.

According to the example illustrated, the camera 11 acquires images at a frequency of 30Hz (thirty Hertz), or thirty images per second. When the vehicle 10 travels at a speed of 15km/h (fifteen kilometers per hour), or at a speed of 4.2m/s (four point two meters per second), it then travels 14cm (fourteen centimeters) between the point of view of acquisition of one image and the point of view of acquisition of a subsequent consecutive image. Similarly, it travels 14cm between the point of view of acquisition of two other images. The vehicle has therefore traveled 28cm (twenty-eight centimeters) between the points of view of acquisition of three consecutive images, or during two time intervals. Learning the depth prediction model from images separated by two time intervals is equivalent to learning the depth prediction model from images acquired consecutively, i.e. at one time interval, while vehicle 10 is traveling twice as fast.

Using a similar reasoning, the vehicle travels 42cm (forty-two centimeters) between the acquisition viewpoints of images acquired at three time intervals. This distance of 42cm is close to a distance traveled equal to 46cm (forty-six centimeters) between two acquisition viewpoints of two consecutive images at a speed of 50km/h (fifty kilometers per hour). Learning the depth prediction model with these images temporally separated by three time intervals is equivalent to learning the depth prediction model for a vehicle traveling three times faster, i.e. 50km/h simulated versus 15km/h in reality.

Similarly, the distance traveled by the vehicle 10 between the acquisition of images acquired at two time intervals at a speed of 50 km/h is close to a distance traveled by the vehicle 10 between the acquisition of two consecutive images at a speed of 100 km/h. A speed of 100 km/h is then simulated from images acquired at a speed of 50 km/h.

Thus, during each iteration, the depth prediction model is learned with a distance separating the image acquisition viewpoints of the first pair of images that increases. The depth prediction model is thus learned for a small distance separating the acquisition viewpoints from the acquired images, then, when the depth prediction model is sufficiently accurate, a new iteration is implemented and the previously learned depth prediction model continues its learning with a greater distance separating the image acquisition viewpoints. Increasing this distance then amounts to simulating a greater speed of the vehicle 10. Indeed, the distance traveled between the image acquisition viewpoints is doubled between the first iteration and the second iteration, then tripled between the first iteration and the third iteration, and so on. It is then possible to learn the depth prediction model by successive distance steps and thus to carry out progressive learning with an increasing simulated speed. In this way, the learning of the depth prediction model is controlled and consistent for large speed ranges of the vehicle 10.

According to a second variant of the learning method 3, the first pairs are generated by selecting the first image as corresponding to the most recent image in each sequence of images and the second image as corresponding to an image prior to the first image in each sequence of images, the second image being selected according to a number of time intervals separating the acquisition of the first image from the acquisition of the second image, the number of intervals being determined according to a determined probability law.

The probability law is, for example, obtained by the following function:

With :
• a probability associated with a noted interval ,
• a standard deviation.

The standard deviation varies for example at each iteration of the first learning, so as to define depending on iteration i, for example with . If the number of images in an image sequence is seven, at the first iteration while in the last iteration, .

A function generating a random result according to the probability law is for example defined in Python® language by the numpy.random.normal function, the Python® language being particularly used in the field of image processing.

Depending on a result of this function, a first pair of images is selected depending on the number of time intervals separating the acquisition times of the first and second images of the first pair of images, this selection being determined by the following function:

With :
• the number of time intervals calculated, and
• the result of a function generating a random result according to the law of probability.

The function presented above is obviously suitable for a sequence comprising seven images, and therefore up to six time intervals.

Such a probability distribution corresponds to a normal distribution centered on a zero value which makes it possible to obtain a greater probability of using a first pair of images when the first and second images of the first pair of images are acquired at a low number of time intervals, and a lower probability of using a first pair of images when the first and second images of the first pair of images are acquired at a high number of time intervals. Thus, compared to the pairs illustrated in the FIG. 5 , the first pair of images 510 has a greater chance of being used for training the depth prediction model than the first pair 520, which itself has a greater chance of being used than the first pair 530. The first pair 560 is thus the one with the lowest chance of being used.

Unlike the first variant, in this second variant not all possible pairs of images are used and the increase in the distance separating the acquisition viewpoints of the images of a second pair of images is not progressive.

The first training ends when the depth prediction model is sufficiently reliable. This step 32 ends when the depth prediction model reaches, for example, the same performance as in the publication “Digging Into Self-Supervised Monocular Depth Estimation” by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow, published in August 2019. The performance is to be evaluated by a qualitative means such as by a comparison of depth maps generated for the same image, a relative error being for example less than 25%.

For each sequence of images included in the second class, in a step 33, the depth prediction model is learned in a second learning from second pairs of images, each second pair of images comprising a third image and a fourth image selected in each sequence of images.

The second learning is thus similar to the first learning but with as input data second pairs of images corresponding to images acquired at higher speed. High speed is therefore no longer simulated as previously, the depth prediction model thus continues its learning with real and representative data, for example from extra-urban environments where the vehicle 10 actually travels quickly. Indeed, during the first learning, the images are for example representative of a city center but the simulated speed is greater than 100 km/h, which does not correspond to a truly observable driving situation.

The first and second learning correspond to the determination of input parameters of the convolutional neural network implementing the depth prediction model. FIG. 6 illustrates a flowchart of different steps of such a method 6 for learning the depth prediction model from a pair of images used in the method 3 for learning the depth prediction model, according to a particular and non-limiting exemplary embodiment of the present invention.

According to the particular embodiment example illustrated in the FIG. 6 , the depth prediction model is learned by minimizing a loss error determined from each pair of images of a plurality of pairs of images by comparing the images of each pair of images to images reconstructed from the images of each pair of images and depths predicted by the depth prediction model and associated with pixels of the images of each pair of images, the plurality of pairs of images corresponding to the first pairs of images during said first learning and to the second pairs of images during said second learning.

In a step 61, a pair of images comprising a fifth and a sixth image is received. The fifth image then corresponds to a first image during a first learning or to a third image during a second learning, the sixth image corresponds to a second image during a first learning or to a fourth image during a second learning.

In a step 62, depths are predicted by the depth prediction model for pixels of the fifth and sixth images from the fifth and sixth images.

Determining the depth of a pixel of an image acquired by a moving camera from a depth prediction model implemented by a convolutional neural network is known to those skilled in the art, for example with monodepth2® described in the document “Digging Into Self-Supervised Monocular Depth Estimation” written by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow and published in August 2019 or with an algorithm called NRS described in the document “Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion” written by Igor Vasiljevic, Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Wolfram Burgard, Greg Shakhnarovich and Adrien Gaidon, published in August 2020.

In a step 63, seventh and eighth images are generated. The seventh image is generated from the fifth image, depths associated with the pixels of the fifth image and extrinsic parameters of the moving monocular vision system while the eighth image is generated from the sixth image, depths associated with the pixels of the sixth image and extrinsic parameters of the moving monocular vision system. This step 63 is commonly referred to as bilateral image reconstruction.

The generation of a seventh image from a fifth image acquired by the camera 11 consists of reprojecting a pixel of the fifth image acquired from a first viewpoint at the time instant of acquisition of the fifth image in the form of a point, then projecting this point into the image plane of the camera 11 at its position corresponding to a second viewpoint at the time instant of acquisition of the sixth image, so as to obtain an image corresponding to a view of the three-dimensional scene from the second viewpoint of the camera 11. The image plane of the camera 11 corresponds to a plane defined in the frame of reference of the camera 11, normal to the optical axis of the camera 11 and located at the focal length of the camera 11. Thus, the seventh image generated from the fifth image is comparable to the sixth image. Similarly, the eighth image generated from the sixth image is comparable to the fifth image.

The first and second positions of the camera 11 at the time instants of acquisition of the fifth and sixth images respectively not being confused when the vehicle 10 is in motion and objects being able to mask other objects in the scene, the generated images are not identical to the images acquired by the camera 11. In addition, the prediction of depths and/or the prediction of a movement of the camera 11 between the first and second points of view are not free from errors. Thus, the comparison of a generated image with an acquired image makes it possible to evaluate the accuracy of the different models used: the depth prediction model and/or the movement prediction model of the camera 11.

According to a particular embodiment, the seventh and eighth images are respectively generated by the following function:

With :
• the coordinates of a pixel of the seventh image, respectively of the eighth image,
• a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector,
• a direction prediction model associated with camera 11,
• a matrix including the extrinsic parameters, and
• a projection function into the three-dimensional scene of a pixel depending on its coordinates in the fifth image, respectively sixth image, and the depth which is associated with it.

According to one variant, the direction prediction model corresponds to an intrinsic matrix of the camera 11. This variant is particularly applicable to pinhole and calibrated cameras.

According to another variant, the direction prediction model corresponds to a learned model implementing a second neural network. Such a direction prediction model is known to those skilled in the art; it is notably presented in the document “Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion”. This other variant is suitable for wide-angle or “fish-eye” type cameras or even for uncalibrated cameras. According to this other variant, a direction prediction step associated with the pixels of the fifth image and the pixels of the sixth image is necessary.

The matrix comprising the extrinsic parameters of the moving camera 11 can be obtained in several ways. For example, from data emitted by a driving assistance system of the vehicle 10, called an ADAS system, or from sensors embedded in the vehicle 10. According to another example, the extrinsic parameters corresponding to the movement of the camera 11 between the time instants of acquisition of the fifth and sixth images are determined in an additional step by a motion prediction model, for example by that presented in the document “Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion”.

In a step 64, the depth prediction model is learned by minimizing a loss error.

The loss error includes, for example, a first photometric error determined by comparing the fifth and eighth images and a second photometric error determined by comparing the sixth and seventh images.

If the depth prediction model is accurate, then a first or second photometric error is small. Conversely, if the depth prediction model is inaccurate, for example because its training is not complete, then a first and/or second photometric error is not negligible.

According to a first particular embodiment, the first and second photometric errors are respectively determined by the following function:

With :
• the first photometric error noted , respectively the second photometric error noted , being a pixel defined by its two-dimensional coordinates,
• a pixel value in the fifth image, respectively in the sixth image,
• a pixel value in the eighth image, respectively in the seventh image,
• SSIM a function that takes into account a local structure, and
• a weighting factor depending in particular on the type of environment in which the vehicle operates 10.

According to a second particular embodiment, the first and second photometric errors comprise a determination of a reconstruction error of a generated image determined by the following function:

With :
• the reconstruction error for a pixel p of the seventh image, respectively the construction error for a pixel p of the eighth image;
• is a depth associated with the pixel in the seventh image, respectively in the eighth image;
• is a parameter matrix;
• is the order of a smoothing gradient;
• an L1 norm of second-order depth gradients is calculated with =1, and =2 ;
• And are the dimensions of the seventh and eighth;
• is a hyperparameter dependent on the environment in which the vehicle 10 operates; and
• is a colorimetric value of the pixel in the seventh image, respectively eighth image.

This second function is generally used to deal with discontinuity at the edge of objects (in English “edge aware smoothness”).

The photometric error is thus defined, for example, from the previously defined photometric errors and reconstruction errors.

According to the first particular embodiment, the photometric error associated with each first pair of blocks is determined by the following function:

With :
• corresponding to the loss error,
• the first photometric error determined for a pixel in the fifth image, the pixel being defined by its two-dimensional coordinates, and
• the second photometric error determined for the pixel in the sixth image.

According to the second particular embodiment, the photometric error associated with each first pair of blocks is determined by the following function:

With :
• corresponding to the loss error,
• the first photometric error determined for a pixel in the fifth image, the pixel being defined by its two-dimensional coordinates, and
• the second photometric error determined for the pixel in the eighth image,
• the reconstruction error for a pixel p of the seventh image, and
• the reconstruction error for a pixel p of the eighth image.

Training the depth prediction model consists of adjusting input parameters of the convolutional neural network to minimize the previously calculated loss error.

An object visible in the field of vision of the camera 11 at the time instant of acquisition of the fifth image and masked in the field of vision of the camera 11 at the time instant of acquisition of the sixth image or vice versa does not impact the loss error thanks to the min function, this method is therefore insensitive to occlusions.

Thus, the depth prediction model used for the depth prediction of a pixel of an image acquired by the camera 11 is made reliable thanks to this learning process.

This learning is carried out from data acquired by the on-board vision system and therefore does not require data annotated by another on-board system or storage of a library of training images. In addition, the learning data is representative of the data received when the system is in operation or in production, in fact the learning data is representative of real environments in which the vehicle carrying the monocular vision system evolves or moves, this learning data is therefore particularly relevant.

ThereFIG. 4schematically illustrates a device 4 configured to learn a depth prediction model associated with a vision system on board a vehicle and/or to predict a depth associated with a pixel of an image acquired by a camera from the depth prediction model associated with it, according to a particular and non-limiting exemplary embodiment of the present invention. The device 4 corresponds for example to a device on board the first vehicle 10, for example a computer associated with the stereoscopic vision system.

The device 4 is for example configured for the implementation of the operations described with regard to figures 1 and 4 and/or steps described with regard to figures 2 and 3. Examples of such a device 4 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 4, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and/or in discrete components. The device 4 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the method and/or for executing the instructions of the software(s) embedded in the device 4. The processor 40 may include integrated memory, an input/output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41 corresponding for example to a volatile and/or non-volatile memory and/or comprises a memory storage device which may comprise volatile and/or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored in memory 41.

According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and/or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input/output ports.

According to a particular and non-limiting exemplary embodiment, the device 4 comprises a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 comprise one or more of the following interfaces:
- RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced;
- USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French);
HDMI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French);
- LIN interface (from the English “Local Interconnect Network”).

According to another particular and non-limiting exemplary embodiment, the device 4 comprises a communication interface 43 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 430. The communication interface 43 corresponds for example to a transmitter configured to transmit and receive information and/or data via the communication channel 430. The communication interface 43 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO/IEC 802-3 standard).

According to a particular and non-limiting exemplary embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch-sensitive or not, one or more speakers 450 and/or other peripherals 460 via the output interfaces 44, 45, 46 respectively. According to a variant, one or other of the external devices is integrated into the device 4.

Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for determining the depth of a pixel of an image acquired by a vision system, and/or for measuring a distance separating an object from a vehicle carrying a vision system, the depth and/or the distance being predicted and/or measured via a depth prediction model learned according to the learning method described above, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 4 of the FIG. 4 .

Claims (10)

Method for learning a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle (10), the vision system comprising a camera (11) arranged so as to acquire an image of a three-dimensional scene of an environment outside the vehicle (10),
said method being implemented by at least one processor, and being characterized in that it comprises the following steps:
- reception (31) of image sequences, each image sequence comprising a set of images acquired by the camera (11) at different consecutive acquisition time instants separated by the same time interval, each image sequence being classified according to a speed of the vehicle (10) during the acquisition of the images of said sequence in a first class when said speed of the vehicle (10) is lower than a threshold speed and in a second class when said speed of the vehicle (10) is higher than the threshold speed;
- for each sequence of images included in the first class, first training (32) of the depth prediction model from first pairs of images, each first pair of images comprising a first image and a second image selected in said each sequence of images;
- for each sequence of images included in the second class, second training (33) of the depth prediction model from second pairs of images, each second pair of images comprising a third image and a fourth image selected in said each sequence of images. The method of claim 1, wherein the most recent image in said each sequence of images is assigned in each first pair of images as the first image and each image acquired prior to the first image is assigned in a first pair of images as the second image, each second image being different in each first pair of images. Method according to claim 2, for which the first learning comprises several iterations, each iteration using a different first pair of images corresponding to the first pair of images for which a time interval separating the acquisition of the first image and the second image is the smallest among the first pairs not used during a previous iteration, the depth prediction model being learned from said first pair of images used. Method according to claim 1, for which said first pairs are generated by selecting said first image as corresponding to the most recent image in said each sequence of images and said second image as corresponding to an image prior to the first image in said each sequence of images, the second image being selected according to a number of time intervals separating the acquisition of the first image from the acquisition of the second image, said number of intervals being determined according to a determined probability law. Method according to claim 4, for which the probability law is obtained by the following function:

with :
• a probability associated with a noted interval ,
• a standard deviation. Method according to one of claims 1 to 5, for which the threshold speed is determined as a function of a target distance and a frequency of image acquisition by the camera (11). Method according to one of claims 1 to 5, for which the depth prediction model is learned by minimizing a loss error determined from each pair of images of a plurality of pairs of images by comparing the images of said each pair of images to images reconstructed from the images of said each pair of images and depths predicted by the depth prediction model and associated with pixels of the images of said each pair of images, said plurality of pairs of images corresponding to said first pairs of images during said first learning and to said second pairs of images during said second learning. Computer program comprising instructions for implementing the method according to any one of the preceding claims, when these instructions are executed by a processor. Device (4) configured to learn a depth prediction model by a vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured to implement the steps of the method according to any one of claims 1 to 7. Vehicle (10) comprising the device (4) according to claim 9.