GB2559753A

GB2559753A - Fusion of images from drone and vehicle

Info

Publication number: GB2559753A
Application number: GB1702515.6A
Authority: GB
Inventors: Kalyazin Nikita; Riley Tom; Pandey Abhinav; Plowman Robin; Sun Duo; Jin Yonggang
Original assignee: Continental Automotive GmbH
Current assignee: Continental Automotive GmbH
Priority date: 2017-02-16
Filing date: 2017-02-16
Publication date: 2018-08-22
Also published as: GB201702515D0

Abstract

A driver assistance system (100) comprises a drone (110), which communicates wirelessly with a communication unit (120), and a control unit (130) which is configured to provide information to a driver of a vehicle based on information acquired by the drone about a vicinity of the vehicle. The system may comprise a camera (140, 150) for each of the drone and vehicle, and the control unit may generate a fused surround view image using the image data obtained by these cameras (Fig. 2). The fused image may be generated in real time and as the drone flies. The image data may be combined to give a three-dimensional image. The control unit may be configured to change the perspective of the fused image, or may be configured to select a sub-set of the image data received from the drone for data fusion.

Description

(54) Title of the Invention: Fusion of images from drone and vehicle

Abstract Title: A driver assistance system using a drone to transmit information about the vicinity of a vehicle, and which may fuse image data from the drone and the vehicle (57) A driver assistance system (100) comprises a drone (110), which communicates wirelessly with a communication unit (120), and a control unit (130) which is configured to provide information to a driver of a vehicle based on information acquired by the drone about a vicinity of the vehicle. The system may comprise a camera (140, 150) for each of the drone and vehicle, and the control unit may generate a fused surround view image using the image data obtained by these cameras (Fig. 2). The fused image may be generated in real time and as the drone flies.

The image data may be combined to give a three-dimensional image. The control unit may be configured to change the perspective of the fused image, or may be configured to select a sub-set of the image data received from the drone for data fusion.

Fig. 1

100

130

Application No. GB1702515.6

RTM

Date :9 August 2017

Intellectual

Property

Office

The following terms are registered trade marks and should be read as such wherever they occur in this document:

WiFi

Zigbee

Bluetooth

WiMax

UMTS

FTE

Galileo

Intellectual Property Office is an operating name of the Patent Office www.gov.uk/ipo

Fusion of images from drone and vehicle

The invention relates to a driver assistance system for acquiring vicinity information of a vehicle by a drone, a vehicle comprising the driver assistance system, a method, a program element, and a computer-readable medium.

Since the beginning of the development of motor vehicles, systems for assisting a driver have been developed. Some important milestones thereof are the ABS, the parking assist and the cruise control. Especially in the last 10 years numerous driver assistance systems have been developed. During the parking procedure, driving assistance systems can support the driver, so that the vehicle does not get damaged or the vehicle parks completely autonomous. One of the systems can generate a surround view of the car, so that the driver can see the vicinity of the vehicle. Therefore, the driver may also see obstacles which he would not have seen by looking through the windows of the vehicle.

It would be advantageous to receive detailed information about the vicinity of a vehicle.

The object of the present invention is solved with the subject matter of the independent claims, wherein further embodiments are incorporated in the dependent claims.

A first aspect of the invention is related to a driver assistance system for a vehicle. The driver assistance system comprises a drone configured to acquire information about a vicinity of the vehicle, a communication unit configured to receive the information from the drone via a wireless communication link and a control unit configured to provide assistance information to the driver of the vehicle based on the information acquired by the drone .

Thus, the driver assistance system includes a separate, external device, i.e. a drone. The drone may be configured to fly over, above, in front of or next to the vehicle. Further, the drone may be configured to acquire vicinity information of the vehicle. The vicinity information may comprise, for example, images, infrared images, radar images, distances, parking spots, obstacles and / or traffic information. The drone may be able to send the acquired information to the vehicle. Furthermore, the drone may send the acquired information to several different vehicles simultaneously. A communication unit of the driver assistance system may be configured to receive the acquired vicinity information from the drone via a wireless communication link. The communication unit may be located in the vehicle and may be connected to a control unit of the driver assistance system. The control unit may be configured to provide assistance information to the driver of the vehicle based on the vicinity information acquired by the drone. The control unit may be further configured to fuse information acquired by the drone and information acquired by a sensor and / or several sensors of the vehicle. According to another embodiment of the invention the drone may be configured to receive the information acquired by the sensors of the vehicle. Further, the drone may be configured to fuse the information received from the vehicle with the information acquired by the drone. It may also be possible that the control unit of the vehicle fuses the information acquired by the drone and acquired by the sensors of the vehicle and send this fused information to the drone. The control unit of the driver assistance system and the communication unit may be configured to receive vicinity information from several different drones. Thus, the control unit may be able to provide a surround view image based on vicinity information acquired by more than one drone.

Furthermore, the drone may be configured to navigate and determine its location based on the acquired vicinity information of the vehicle. This may be advantageous if the positioning over a global positioning system (GPS) is not accurate. Also the positioning of the vehicle may be improved by the information acquired by the drone. The control unit may be configured to control the flight of the drone, such that the control unit directs the drone to fly to a certain area, i.e. to find a parking spot, to determine the length and / or the cause of a traffic jam or to fly to a specific obstacle to acquire more information about this specific obstacle.

A drone, in a technological context, is an unmanned aircraft. Drones are more formally known as unmanned aerial vehicles (UAVs) or unmanned aircraft systems (UASes). Essentially, a drone is a flying robot. The aircrafts may be remotely controlled or can fly autonomously through software-controlled flight plans in their embedded systems working in conjunction with onboard sensors and GPS .

The wireless transmission of the vicinity information may be carried out via Bluetooth, WiFi (e.g. WiFi 802.11a / b / g / n or WiFi 802.lip), ZigBee or WiMax or cellular radio systems such as GPRS, UMTS or LTE . It is also possible to use other transmission protocols. The mentioned protocols offer the advantage of standardization.

It should be noted that, in the context of the present invention, GPS is representative of all Global Navigation Satellite Systems (GNSS), e.g. GPS, Galileo, GLONASS (Russia), Compass (China) or IRNSS (India).

According to an embodiment of the invention the driver assistance system further comprises at least one camera for the vehicle configured to acquire first image data and at least one camera for the drone, wherein the information acquired by the drone comprises second image data. The control unit is configured to fuse the first image data with the second image data, to receive a fused surround view image.

Thus, according to this embodiment the drone may comprise a camera, which may provide image data. The vehicle may also comprise a camera to acquire image data. The images from the camera of the vehicle may be titled first image data and the images form the camera of the drone may be titled second image data. The control unit may be configured to fuse the first image data with the second image data to receive a fused surround view image of the vehicle. The vehicle may comprise more than one camera, for example four cameras, so that the vehicle can generate a surround view image. The control unit may be configured to analyse the second image data from the drone and enlarge the first image data with the second image data or parts of the second image data. Further, the control unit may be configured to use specific details of the second image data for fusing with the first image data .

In an embodiment of the invention the control unit may be configured to use image data and other vicinity information acquired by the drone to provide assistance information to the driver of the vehicle, i.e. use the second image data and distance information for a specific obstacle.

According to an embodiment of the invention the control unit is configured to generate the fused surround view image during the flight of the drone, and may even be configured to do this in real time .

Real time is the operation of a computer system in which programs for data processing are constantly ready for operation, such that the processing results are available within a predetermined period of time, for example 1, 10, 20 or 50 ms. Depending on the application, the data may be generated according to a temporally random distribution or at predetermined times.

It may be advantageous, if the control unit could provide a surround view image in real time. Further, the control device should create the surround view image based on the second image data acquired by the drone also during the flight of the drone. The control unit could also be configured to generate a surround view image, if the drone is not flying or use not real time image data acquired by the drone, for example, the drone flies to the beginning of a traffic jam to measure the length of the traffic jam. Than the drone flies back to the vehicle with the information about the length of the traffic jam. And this information is provided to the driver of the vehicle.

According to another embodiment of the invention the control unit is configured to generate a 3D image from the combination of the first image data and the second image data.

Thus, the control unit may be configured to generate a 3D image for the driver of the vehicle. To create the 3D image, the control device may fuse the first image data acquired by the camera of the vehicle and the second image data acquired by the camera of the drone. For an accurate 3D image generation, the control unit may acquire information about the position of the drone and the camera of the drone relative to the camera of the vehicle. A 3D image may be more realistic for the driver, so that the driver can handle the information shown by the driver assistance system in a better way. Therefore, the driver assistance system is more useful and accepted by the driver of the vehicle.

According to an embodiment of the invention the control unit is configured to change the perspective of the fused image.

In another embodiment of the invention the control unit may be configured to change the perspective of the fused surround view image based on the first image data and the second image data. In other words, the content can be shown from the view of an virtual camera location. So the content can be shown in a more natural way to the driver. Thus, the driver can handle the shown content of the driver assistance system in a better way. The change of the perspective may also be dynamically changed based on the situation and the preferences of the driver.

According to another embodiment of the invention the control unit is configured to select a sub-set of the second image data for data fusion.

Thus, the control unit may be configured to analyse the content of the second image data acquired by the camera of the drone . Based on this analysis, the control unit may augment the first image data with specific details from the second image data, for example, one obstacle is only present in the second image data because this obstacle is not in the field of view of the cameras of the vehicle. Thus, the control unit may add this specific obstacle to the first image data. Further, the first image data with the added obstacle from the second image data could be displayed to the driver of the vehicle.

According to a second aspect of the invention a vehicle comprising a driving assistance system is provided.

The vehicle may, for example, be a motor vehicle, such as a car, bus, truck or motorcycle, a rail vehicle, a ship or an aircraft, such as a helicopter or an airplane.

According to a third aspect of the invention a method for fusion a surround view image is provided, comprising:

Acquiring vicinity information of a vehicle by a drone; Transmitting the vicinity information from the drone via a wireless communication link to a communication unit of the vehicle;

Providing assistance information to the driver of the vehicle by a control unit based on the information acquired by the drone .

According to another aspect, there is provided a computer program element for controlling the driver assistance system as previously described which, if the computer program element is executed by a processing unit, is adapted to perform the method steps as previously described.

There is also provided a computer readable medium having stored the computer element as previously described.

Advantageously, the benefits provided by any of the above aspects equally apply to all of the other aspects and vice versa.

The above aspects and examples will become apparent from and be elucidated with reference to the embodiments described hereinafter .

Exemplary embodiments will be described in the following with reference to the following drawings:

Fig. 1 shows a schematic set up of the driver assistance system with a drone, according to an embodiment of the invention.

Fig. 2 shows a schematic set up of the field of view of the cameras of the vehicle and of the drone.

Fig. 3 shows a flow chart for a method for a driver assistance system according to an embodiment of the invention.

Fig. 1 shows the driver assistance system 100. The driver assistance system 100 comprises a drone 110, a camera 140 on the drone 110, a control unit 130, a camera 150 on the vehicle 200 and a communication unit 120. The communication unit 120 exchanges vicinity data from the drone 110 and the vehicle 200. The drone 110 is configured to fly over / above, in front of or next to the vehicle 200. Further, the drone 110 is configured to acquire vicinity information of the vehicle 200. These vicinity information may comprise image data, distances, parking spots and/or traffic information. The drone 110 is further configured to send the acquired vicinity information via a wireless communication link to the communication unit 120 of the driver assistance system 100. The wireless communication link may be, for example, WiFi, LTE, UMTS, 3G or Bluetooth. The control unit 130 of the driver assistance system 100 is configured to provide assistance information to the driver of the vehicle 200 based on the information acquired by the drone 110. The information acquired by the drone 110 may help the driver of the vehicle 200 while driving the vehicle 200, i.e. the driver can focus on the street or traffic and the vehicle 200 could be protected of damages. The drone 110 is able to change the height and the position during the flight and therefore, the drone 110 is able to change the perspective and the scope of the image data acquired by the camera 150 of the drone 110. According to an embodiment of the invention the drone 110 is configured to fly ahead of the vehicle 200 and therefore sees what is in front of the vehicle 200, i.e. if a traffic jam is present and the length of it. In another embodiment of the invention the drone 110 is configured to fly near to or around the vehicle 200 and espy a parking space for the vehicle 200. An advantage of the drone 110 may be that the drone 110 can see over walls and around corners,

i.e. flies to a position or change the field of view to get another perspective. Further, the done 110 may also acquire vicinity information when the vehicle 200 stand still or is blocked in traffic .

The control unit 130 is configured to process the vicinity information acquired by the drone 110 and based on these information provide assistance information to the driver of the vehicle 200. Further, the control unit 130 is configured to use data from the vehicle 200 and its sensors and use data acquired by the drone 110 and based on these data providing assistance information to the driver. The control unit 130 may also be configured to fuse the data acquired by the vehicle 200 and the data acquired by the drone 110. In an embodiment of the invention, the vicinity information acquired by the drone 110 comprise image data. With this image data and image data acquired by a camera 150 of the vehicle 200, the control unit 130 of the driver assistance system 100 is configured to generate a surround view image. This surround view image can be generated due to fusion of the image data acquired by the camera 140 of the drone 110 with image data acquired by the camera 150 of the vehicle 200. The control unit 130 is further configured to generate the surround view image in real time and while the drone 110 is flying. The control unit 130 is also configured to generate a 3D image based on image data from the camera 150 of the vehicle 200 and on image data from the camera 140 of the drone 110. The control unit 130 is further configured to change dynamically the perspective of the resulting image data shown to the driver, based on image data acquired by the camera 150 of the vehicle 200 and on the image data acquired by the camera 140 of the drone 110 and vice versa.

According to an embodiment of the invention the drone 110 can be located, positioned and controlled based in the image data acquired by the drone 110. In other words, the position control of the drone 110 can be improved by imply the image data captured by the drone and not only by the use of GPS information. The control unit 130 may control the drone 110 by leading the drone 110 to a specific location and / or area.

In Fig. 2 image data from the camera 140 of the drone 110 and image data from the camera 150 from the vehicle 200 is shown. Especially, the field of view of the cameras 140, 150 is shown. The image data acquired by the vehicle 200 is generated by four cameras, symbolised by the rectangle within the vehicle 200 in the middle. The image data acquired by the camera 140 of the drone 110 is symbolised by the big rectangle. The control unit 130 is configured to fuse both image data to one fused surround view image. Further, the control unit 130 is configured to display the resulting fused surround view image to the driver of the vehicle 200. The advantage of the fusion of the image data acquired by the camera 140 of the drone 110 with the image data acquired by the camera 150 of the vehicle 200 may be that the driver can be provided with additional information and also vicinity information that can only be acquired with another perspective. Especially, a perspective more elevated, i.e. while flying in a certain height.

Fig. 3 shows a schematic flow chart for a method for a driver assistance system. In step 301 vicinity information of a vehicle is acquired by a drone. The acquired information is transmitted in step 302 from the drone to the control unit of the driver assistance system. In step 303 assistance information is provided to the driver of the vehicle. The assistance information is based on the vicinity information acquired by the drone.

Claims

1. Driver assistance system (100) for a vehicle, comprising: a drone (110) configured to acquire information about a vicinity of the vehicle;

a communication unit (120) configured to receive the information from the drone (110) via a wireless communication link;

a control unit (130) configured to provide assistance information to a driver of the vehicle based on the information acquired by the drone (110) .

2. Driver assistance system (100) according to claim 1, further comprising:

at least one camera (150) for the vehicle configured to acquire first image data;

at least one camera (140) for the drone (110) configured to acquire second image data;

wherein the control unit (130) is configured to fuse the first image data with the second image data, to receive a fused surround view image.

3. Driver assistance system (100) according to claim 2, wherein the control unit (130) is configured to generate the fused surround view image in real time and during the flight of the drone (110).

4. Driver assistance system (100) according any one of the claims 2 or 3, wherein the control unit (130) is configured to generate a 3D image from the combination of the first image data and the second image data.

5. Driver assistance system according (100) any one of the preceding claims, wherein the control unit (130) is configured to change the perspective of the fused image.

6. Driver assistance system (100) according any one of the claims 2 to 5, wherein the control unit (130) is configured to select a sub-set of the second image data for data fusion.

7. Vehicle (200) with a driver assistance system (100) according to any one of the preceding claims.

8. Method for a driver assistance system, comprising:

Acquiring (301) vicinity information of a vehicle by a drone;

Transmitting (302) the vicinity information from the drone via a wireless communication link to a communication unit of the vehicle;

Providing (303) assistance information to the driver of the vehicle by a control unit based on the information acquired by the drone.

9. Computer program element for a driver assistance system according to any one of claims 1 to 6 and/or a vehicle according to claim 7, which when executed by a processor causes the driver assistance system to carry out the method according to claim 8.

10. Computer readable medium, on which a computer program element according to claim 9 is stored.

Intellectual

Property

Office

Application No: GB1702515.6 Examiner: Dr Maurice Blount