WO2018050465A1

WO2018050465A1 - Method and system for determining a saliency value for prominent roi in an image

Info

Publication number: WO2018050465A1
Application number: PCT/EP2017/072078
Authority: WO
Inventors: Abhishek MURTHY; Dan ZIEMBIENSKI; Talmai Brandão DE OLIVEIRA
Original assignee: Philips Lighting Holding BV
Current assignee: Signify Holding BV
Priority date: 2016-09-14
Filing date: 2017-09-04
Publication date: 2018-03-22
Anticipated expiration: 2019-03-14

Abstract

A device, system, and method evaluate an image to assist in night-time drivability. The method performed at a vehicle device associated with a vehicle traveling on a road includes receiving an image from a point of view (PoV) of a driver of the vehicle. The method includes identifying a region of interest (RoI) in the image, the RoI including an object that affects a drivability on the road. The method includes determining a prominence of the RoI from the PoV of the driver. The method includes determining a saliency value for the RoI based on the prominence.

Description

METHOD AND SYSTEM FOR DETERMINING A SALIENCY VALUE FOR PROMINENT ROI IN AN IMAGE

BACKGROUND INFORMATION

A vehicle may be driven by a driver along a road to reach a destination. While the vehicle is driven during the day with natural ambient light present, a point of view of the driver may capture various objects present on the road and adjust a tracking of the vehicle accordingly. However, while the vehicle is driven during night-time with little to no natural ambient light present, the point of view of the driver may miss otherwise noticeable objects and fail to adjust the tracking of the vehicle. To accommodate for the lack of natural ambient light, the vehicle and/or surrounding areas on the road may include an artificial light source to illuminate a field of view. The manner in which the light sources provide the illumination is critical for safety in night-time driving. With appropriate illumination, the driver may observe the road and lane curvature, locate defects on the road to maneuver accordingly, and minimize interference with other drivers in other vehicles. However, the manner in which the light sources provide the illumination do not utilize a systematic approach in evaluating the impact of lighting the road for night-time drivability in a quantitative and scalable fashion.

SUMMARY

The exemplary embodiments are directed to a method, comprising: at a vehicle device associated with a vehicle traveling on a road: receiving an image from a point of view (PoV) of a driver of the vehicle; identifying a region of interest (Rol) in the image, the Rol including an object that affects a drivability on the road; determining a prominence of the Rol from the PoV of the driver; and determining a saliency value for the Rol based on the prominence.

The exemplary embodiments are directed to a vehicle device associated with a vehicle traveling on a road, comprising: an imager generating an image from a point of view (PoV) of a driver of the vehicle; and a processor receiving the image, the processor identifying a region of interest (Rol) in the image, the Rol including an object that affects a drivability on the road, the processor determining a prominence of the Rol from the PoV of the driver, the processor determining a saliency value for the Rol based on the prominence. The exemplary embodiments are directed to a method, comprising: at a lighting server; receiving data from a vehicle located on a road, the data including a region of interest (Rol) based on an image taken from a point of view (PoV) of a driver of the vehicle and a saliency value for the Rol; and adjusting a lighting source on the road based on the Rol and the saliency value.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 shows a system according to the exemplary embodiments. Fig. 2 shows a vehicle device of Fig. 1 according to the exemplary embodiments.

Fig. 3 shows a method for evaluating images to assist in night-time drivability according to the exemplary embodiments.

Figs. 4A-C show views including salient factors according to the exemplary embodiments.

DETAILED DESCRIPTION

The exemplary embodiments may be further understood with reference to the following description and the related appended drawings, wherein like elements are provided with the same reference numerals. The exemplary embodiments are related to a device, a system, and a method for evaluating images to assist in night-time drivability. The images may be captured from a perspective of a driver driving a vehicle (e.g., from a driver's point of view (PoV)). The exemplary embodiments provide a mechanism in which the images are analyzed to identify regions of interest (Rol) in the images and determine saliency values of the Rols. Accordingly, the mechanism according to the exemplary embodiments may provide a further feature directly or indirectly to the driver driving the vehicle based on the Rols and the corresponding saliency values.

A driver driving a vehicle during night-time has to compensate for utilizing artificial light sources directed in finite fields of view in contrast to a natural light source that is omnipresent. Thus, outdoor lighting via the artificial light sources is critical for safe night- time driving. The artificial light sources may include an incorporated light source on the vehicle (e.g., headlights) and/or roadway light sources (e.g., street lamps). Therefore, the manner in which the artificial light sources illuminate the road has several safety-critical implications on night-time drivability. Currently, a manual approach is used for evaluating light sources on the vehicle in which the driver selects a level of illumination (e.g., no lights, fog lights, normal illumination, high beams) and a static approach is used for evaluating light sources on the road (e.g., streetlamps) in which a constant illumination level is used.

Despite the correlation of lighting a road and safety in night-time driving, there are no systematic approaches that evaluate the impact of this lighting on night-time drivability in a quantitative and scalable fashion. Such approaches assist in articulating the efficacy of solutions in lighting a road. Quantitative evaluations of night-time drivability may also aid in comparing different solutions in lighting a road. For example, a lighting project may be justified on a particular road by quantifying an improvement to night-time driving conditions.

To address the lack of systematic approaches, the exemplary embodiments provide a mechanism that utilizes a quantitative analysis of various factors associated with night-time drivability. The mechanism according to the exemplary embodiments includes a system configured to capture an image that represents a PoV of the driver driving the vehicle during night-time driving. The system also analyzes the image to identify Rols in the image. The system further subjects the image to a modified saliency detection that measures a conspicuousness of different parts of the PoV of the driver which considers an effect of lighting on various objects exterior to the vehicle through representation in saliency values. When the saliency values are determined for the Rols, the system may provide direct and indirect features. As will be described below, the direct features may include alerts associated with the Rols provided to the user and/or graphics displayed to the user indicative of the

Rols. The indirect features may include a feature provided by a separate entity that affects the lighting condition for the driver and the vehicle. The indirect features may further include an amalgamated feature in which information from a plurality of vehicles are collected for a particular section of road.

Fig. 1 shows a system 100 according to the exemplary embodiments. The system 100 relates to a vehicle 105 traveling along a road 125. Depending on the features to be provided by the exemplary embodiments, the functionalities of the system 100 may be performed wholly within the vehicle 105. The functionalities of the system 100 may also be performed with exterior components. Specifically, the system 100 may include the vehicle 105 including a vehicle device 110, a light source 115, and an imager 120, the road 125, a communications network 130, and a lighting server 130.

The vehicle 105 may represent any motorized vehicle in which a driver is capable of controlling a speed and a direction of movement. For example, the vehicle 105 may be a car, a truck, a semi, a bus, a motorcycle, etc. The vehicle 105 may be driven by the driver along the road 125 at any time, particularly during night-time. For purposes of describing the exemplary embodiments, the vehicle 105 may include the vehicle device 110, the light source 115, and the imager 120. However, those skilled in the art will understand that there may be additional components that may be included in the vehicle 105 to perform the operations and options associated with the vehicle 105. Some examples of additional devices will be provided below. The vehicle device 110 will be described in further detail below with regard to Fig. 2.

The light source 115 on the vehicle 105 may be a lighting arrangement oriented on and/or at least partially incorporated in the vehicle 105. Specifically, the light source 115 may include a headlight mounted in a front section of the vehicle 105 such that light emitted from the light source 115 shines on an area in front of the vehicle 105.

Depending on the type of vehicle, the light source 115 may be mounted at various heights and in various locations to sufficiently illuminate the area in front of the vehicle 105 for the benefit of the driver. For example, a car may have the light source 115 mounted

approximately two feet above the road 125, whereas a semi may have the light source 115 mounted approximately four feet above the road 125. The light source 115 may also be configured to provide different levels of illumination based upon the types of light emitting components and associated power provided thereto.

It is noted that the light source 115 may be a lighting arrangement oriented in a plurality of positions around the vehicle 105. For example, the light source 115 may include a tail light in a rear section of the vehicle 105 such that light emitted from the light source 115 shines on an area behind the vehicle 105. In another example, the light source 115 may include a spot light mounted near a side mirror of the vehicle 105 or a row of spot lights mounted on a roof of the vehicle 105 to further illuminate the area in front of the vehicle 105. In a further example, the light source 115 may include an interior light such that light emitted from the light source 115 shines on an area inside the vehicle 105. The exemplary

embodiments may be configured to be utilized with any one or any combination of artificial light sources that may be associated with the vehicle 105. The light source 115 may also include fog lights or any other type of light source that is typically associated with vehicles. The light source 115 may include all of the various lighting elements included on the vehicle 105.

The imager 120 on the vehicle 105 may be an image capturing component. The imager 120 may be positioned in various different locations such as in substantially similar locations as the light source 115. Specifically, the imager 120 may be positioned in a strategic location such that images captured by the imager 120 represent a PoV of the driver or at least includes the PoV of the driver. For example, the imager 120 may be positioned on an inner surface of the roof of the vehicle 105 and aligned with a center of the driver's seat. The imager 120 may also be positioned such that the image capturing components are directed toward the windshield of the vehicle 105 such that the imager 120 captures an area in front of the vehicle 105. The imager 120 may capture the images at predetermined time intervals or may continuously capture the images in a video sequence.

It is noted that the imager 120 may also be an imaging arrangement oriented in a plurality of positions around the vehicle 105. For example, the imager 120 may include a conventional camera such as a rear camera that provides video while the vehicle 105 is in a reverse operation. In another example, the imager 120 may include an interior camera that captures images of the driver. The images of the driver may provide information about the PoV, the field of view, and areas being viewed by the driver at a particular time. For example, the images of the driver captured by the interior camera may be used to adjust the positioning, orientation, and/or angle of the imager 120 capturing images of the PoV of the driver. For example, a height of the driver may affect the PoV and therefore, the imager 120 may be adjusted to correct for the variations in height.

It is noted that the vehicle 105 may include further components. Specifically, the vehicle 105 may include a location determining device such as a global positioning system (GPS) device. For example, the driver may utilize the GPS device by entering a destination and being provided a map and directions to reach the destination. The GPS device may also provide the location of the vehicle. The location of the vehicle 105 may be included on the map to show where the driver is currently located relative to the directions. The location of the vehicle 105 may also be used for further purposes independent of the driver.

The road 125 may be any surface upon which the vehicle 105 may travel. For example, the road 125 may be an asphalt road, a dirt road, a bridge, a tunnel, etc. The road 125 may include various features to aid the driver in driving the vehicle 105. For example, the road 125 may include multiple lanes that are marked with lane dividers. In another example, the road 125 may include street lamps to illuminate an area. In a further example, the road 125 may include other light related components such as reflectors that reflect light from other sources (e.g., the light source 115).

Although the road 125 may provide the surface and these various other features, those skilled in the art will understand that the road 125 may provide sub-optimal, even detrimental, conditions for the driver to drive the vehicle 105, particularly during night- time driving. For example, the road 125 may include damage on the surface such as potholes, raised or lowered surfaces, etc. In another example, the road 125 may be undergoing maintenance which introduces intended defects on the road such as raised manholes. In a further example, the street lamps on the road 125 may not be operating or providing an illumination that is too low (e.g., not enough illumination) or too high (generating a glare). As will be described in detail below, the exemplary embodiments may be configured to generate data associated with any and all of these factors, particularly to improve safety in night-time drivability.

As described above, the mechanism according to the exemplary embodiments may be performed entirely within the vehicle 105. Thus, the features provided by the exemplary embodiments may not require any exterior component to be involved. However, in another manner of utilizing the mechanism according to the exemplary embodiments, exterior components may be associated with providing different features. Thus, the vehicle 105 may include a wireless data exchange capability such that data may be transmitted to and/or received from the exterior components. Specifically, the data exchange may be performed via the communications network 130.

The communications network 130 may be configured to communicatively connect the various components of the system 100 to exchange data. Specifically, the vehicle 105 may exchange data with the lighting server 135. The communications network 130 may represent any single or plurality of networks used by the components of the system 100 to communicate with one another. For example, if data generated by the vehicle 105 is associated with a proprietary organization, the communications network 110 may include a private or proprietary network in which the vehicle 105 may initially connect. The private network may connect to a network of an Internet service provider to connect to the Internet. Subsequently, through the Internet, a connection may be established to other electronic devices. It should be noted that the communications network 130 and all networks that may be included therein may be any type of network. For example, the communications network 110 may be a local area network (LAN), a wide area network (WAN), a virtual LAN

(VLAN), a WiFi network, a HotSpot, a cellular network (e.g., 3G, 4G, Long Term Evolution (LTE), etc.), a cloud network, a wired form of these networks, a wireless form of these networks, a combined wired/wireless form of these networks, etc.

The lighting server 135 may be an exterior component that receives data from the vehicle 105. The lighting server 135 may be configured to provide features related to safety in night-time drivability. For example, the lighting server 135 may have control over objects on the road 125 including any street lamps. Thus, based on the data received from the vehicle 105, the lighting server 135 may adjust the manner in which the street lamps operate. The lighting server 135 may also provide the above noted amalgamated feature in which data is received from a plurality of vehicles. For example, the lighting server 135 may be part of an Intelligent Transportation System (ITS).

As described above, the vehicle device 110 may be a component associated with the vehicle 105. Specifically, the vehicle device 110 may perform functionalities associated with analyzing the images captured by the imager 120. Fig. 2 shows the vehicle device 110 of Fig. 1 according to the exemplary embodiments. The vehicle device 105 may provide various functionalities in analyzing the images. The vehicle device 110 is described as a computing component incorporated in the vehicle 105. For example, the vehicle device 110 may be a separate computing component from other computing components of the vehicle. In another example, the vehicle device 110 may include functionalities that are performed by an onboard computing component of the vehicle 105. As shown in Fig 2, the vehicle device 110 may include a processor 205, a memory arrangement 210, a display device 215, an input and output (I/O) device 220, a transceiver 225, and other components 230.

It should be noted that when the features that are to be provided by the exemplary embodiments may be determined entirely within the vehicle 105, the vehicle device 110 may be incorporated within the vehicle 105, as shown in Fig. 1. However, the incorporation of the vehicle device 110 with the vehicle 105 is only exemplary. According to another exemplary embodiment, the vehicle device 110 may be a remote computing component that receives data from the vehicle 105 (e.g., via the communications network 130). The vehicle device 110 may accordingly provide a response to the vehicle 105 or generate data for the lighting server 135. For illustrative purposes, the description herein relates to when the vehicle device 110 is incorporated in the vehicle 105.

The processor 205 may be configured to execute a plurality of applications of the vehicle device 110. As will be described in further detail below, the processor 205 may utilize a visibility application 250 that analyzes the images captured by the imager 120 and determine saliency values to Rols within an image. It should be noted that the visibility application 250 being an application (e.g., a program) executed by the processor 205 is only exemplary. The functionality associated with the visibility application 250 may also be represented as components of one or more multifunctional programs, a separate incorporated component of the vehicle device 110 or may be a modular component coupled to the vehicle device 110, e.g., an integrated circuit with or without firmware.

The memory 210 may be a hardware component configured to store data related to operations performed by the vehicle device 110. Specifically, the memory 210 may store the images received from the imager 110 and data extracted from the images. The display device 215 may be a hardware component configured to show data to a user while the I/O device 220 may be a hardware component that enables the user to enter inputs. It should be noted that the display device 215 and the I/O device 220 may be separate components or integrated together such as a touchscreen. As those skilled in the art will understand, the vehicle 105 may already include a display device and an I/O device. Thus, the vehicle device 110 including a separate version of these components is only exemplary and the vehicle device 110 may be configured to utilize already existing components in the vehicle 105. The transceiver 225 may be a hardware component configured to transmit and/or receive data via the communications network 130. In this manner, the vehicle 105 may be configured to exchange data via the communications network 130.

According to the exemplary embodiments, the visibility application 250 may analyze images captured by the imager 120 to generate data used in improving safety of night-time drivability. As described above, the imager 120 may generate images that are from a PoV of the driver. The visibility application 250 may receive the images and annotate Rols therein such as lanes, potholes, look-ahead regions, street lamps, etc. The Rols can be predefined objects/areas that affect the drivability of the vehicle on the road or may also be on/near the vehicle, such as glare on a vehicle window or insects in or near the vehicle. The visibility application 250 may utilize image processing techniques such as machine learning to automate the annotation operation. In this manner, the visibility application 250 may identify the various objects/conditions that are present in the image according to the PoV of the driver.

Figs. 4A-C show views 400, 425, 450, respectively, of images including salient factors according to the exemplary embodiments. Specifically, the views 400, 425, 450 represent an image captured by the imager 120 and annotated by the visibility application 250. That is, the views 400, 425, 450 illustrate an exemplary Rol within the images. In the view 400 of Fig. 4A, the visibility application 250 may determine where lanes 405 are located. For example, the lanes 405 may be a first lane indicating a first width end of the road 125, a parking area indicating lane, driving area lanes, a shoulder area indicating lane, and a second lane indicating a second width end of the road 125. In the view 425 of Fig. 4B, the visibility application 250 may determine where road defects 430 are located. For example, the road defects 430 may be potholes or manholes present on the road 125. In the view 450 of Fig. 4C, the visibility application 250 may determine where there is a visibility obstruction 455. For example, the visibility obstruction 455 may be a glare from a street lamp that is emitting a significantly bright illumination. Another manner in which the visibility application 250 may be utilized is determining the Rol for a look-ahead evaluation.

Specifically, the Rol may be a general area in which the vehicle 105 is heading through which the vehicle 105 is likely to pass. It should be noted that throughout this description, the Rol may be described as being or including an "object." It should be clear from these example that the "object" does not need to be a physical object. For example, the glare in Fig. 4C may be an object of the Rol, but is not a physical object.

Accordingly, the visibility application 250 may be configured to annotate the Rols in the image captured by the imager 120. It is noted that the visibility application 250 and/or the imager 120 may be configured to annotate the captured images with other information. Specifically, the images may be annotated with geo-location information based on, for example, GPS data. Thus, each image may be associated with a particular section of the road 125 and identified with this association.

The visibility application 250 may be further configured to utilize a saliency detection operation. The saliency detection operation identifies areas on the images that represent a most prominent part of the PoV of the driver. The saliency detection operation may identify the parts of the image that attracts the attention of the driver in a pre-cognitive phase. Specifically, the saliency detection operation may be capable of simulating what the driver sees during an initial time period of looking at the PoV. For example, the saliency detection operation may be for the first 3 to 5 seconds of looking. The saliency detection operation may be used to detect the salient aspects of the image in the PoV of the driver during this initial time period.

As those skilled in the art will understand, the human visual system takes in a large amount of visual information in the PoV at any given time. However, the human visual system expends the bulk of its resources processing only a small fraction of this information, typically where the viewer is fixated. That is, portions within the visual field may go unnoticed from focusing on the small fraction in which is being fixated. Although the majority of the visual field is not being attended to, this majority of the visual field serves as an important part of the visual experience. The human visual system uses the information in the periphery of the visual field to monitor regions that might be of interest to the driver (e.g., the regions that attract visual attention). If the early perceptual properties (e.g., color, motion, contrast etc.) are engaging, the human visual system will move its fixation to that location to gather more visual information.

Given the manner in which the human visual system operates, the saliency detection operation may utilize image-processing techniques to automatically detect the most prominent parts of an image in the PoV of the driver. Any associated bias (e.g., age, gender, experience, etc.) may be minimal as driver-to-driver variability may be negligible in the initial time period of being exposed to an image.

The saliency detection operation may also be used to identify undesirable objects from the PoV of the driver. The undesirable objects may relate to any object or condition that represents a potential safety hazard, particularly when related to night-time drivability. The undesirable objects may include those in the road 125 (e.g., potholes, manholes, etc.) or surrounding the road 125 (e.g., glare from a street lamp, distracting billboard, etc.).

Using the above mechanism, the visibility application 250 may identify the

Rols in the image that are "hot spots" based on the saliency detection operation. The visibility application 250 may then determine a saliency value associated with each Rol. The saliency value may indicate a prominence of the Rol that measures a saliency used in quantifying an impact of the Rol on safety to night-time drivability. For example, the impact may be to the lighting conditions for night-time drivability. The visibility application 250 may utilize a "before-vs-after" analysis that allows for a quantitative comparison of impact on the night-time drivability. Thus, significantly relevant areas may be scored to quantify the prominence of the corresponding Rols in which safety-critical features (e.g., lane markings, road defects, etc.) may be annotated in the images and analyzed for discernibility.

Specifically, a Rol that has a high saliency value may indicate that the driver is likely to recognize the object/condition in the Rol while a Rol that has a low saliency value may indicate that the driver is likely to miss the object/condition in the Rol.

The saliency values of the Rols may further determine a ranking of the different Rols. In a first example, the Rols may be ranked in order of saliency values. In this manner, the visibility application 250 may select a predetermined number of Rols (e.g., the top five Rols with the lowest saliency values) that are to be used in the further features. In a second example, the Rols may again be ranked in order of saliency values. However, the visibility application 250 may determine a predetermined threshold saliency value that indicates whether the Rols are to be used in the further features. Specifically, the Rols having a saliency value below the predetermined threshold saliency value may be selected.

Through incorporation of the saliency values to the Rols according to the PoV of the driver and the safety nature of each of the Rols, the visibility application 250 may gauge whether safety-critical aspects for night-time drivability are indeed prominent to the driver (i.e., noticed or noticeable to the driver). For objects and/or conditions that are safety- critical but have a low saliency value, the visibility application 250 may be configured to emphasize such a condition so that the potentially missed object/condition will be recognized. In contrast, objects and/or conditions that are safety-critical but have a high saliency value are likely to be noticed by the driver who may adapt for the object/condition accordingly without any further intervention by the system 100.

As noted above, the identified Rols may be utilized in further features that may be used in a variety of different ways. The visibility application 250 may provide the further features based on the types of Rols and the corresponding saliency values. Thus, the examples described herein may be used for select types of Rols having certain saliency values. Therefore, when the image includes different types of Rols, any one or any combination of the further features may be utilized.

In a first example, the further features may be provided to the driver as information presented on the display device 215. As noted above, the display device 215 may be an existing component of the vehicle 105 such as a portion of the dashboard directly in front of the driver or a touchscreen on the dashboard between the driver-side and the passenger-side (e.g., where a GPS device may be utilized). The visibility application 250 may accordingly generate a text alert or provide driver assistance that is displayed on the display device 215. For example, road defects may be presented in the information.

In a second example, the further features may be provided to the driver as a graphic representation presented on the display device 215 or as an overlay. When the graphic representation is presented on the display device 215, the captured image or captured video that is updated with more recent captured images may be shown with portions therein emphasized (e.g., circling objects such as potholes, highlighting objects such as lane markings, etc.). When the graphic representation is presented as an overlay, the display device 215 may be configured with a projector that shows the overlay on the windshield using a non-distracting light or projection. Accordingly, the emphasized aspects may be shown as a projection which coincides with objects being viewed by the driver through the windshield from the PoV of the driver. For example, road defects or lane markings may be shown in the graphic representation.

In a third example, the further features may be provided to the driver in an automated manner. When the Rols and the corresponding saliency values indicate accordingly, the visibility application 250 may generate commands to control how the light source 115 operates. For example, when the visibility application 250 analyzes an image using a look-ahead evaluation in which very low light conditions are detected, the visibility application 250 may generate a command to automatically activate a high beam setting. In another example, the light source 115 may also include a backlight illumination for the dashboard (e.g., speedometer, odometer, engine temperature gauge, gas gauge, etc.). When the visibility application 250 analyzes an image in which very low light conditions are detected, the visibility application 250 may generate a command to lower the level on the backlight illumination. In a substantially similar manner, any interior light may be deactivated which may interfere with exterior visibility.

In a fourth example, the vehicle device 110 may transmit data to an exterior component such as the lighting server 135. The data may correspond to the Rols in the captured image along with corresponding saliency values. In transmitting the data to the lighting server 135, the lighting server 135 may communicate with lighting networks that control artificial light sources on the road 125. For example, as illustrated in Fig. 4C, the street lamp may generate a glare from the PoV of the driver in the vehicle 105 indicating a level of illumination for the street lamp is too high. The lighting network may be an

Intelligent Transportation System (ITS) that may adjust the lighting conditions produced by the street lamp to assist in the night-time drivability for the user. In another example, the ITS may utilize the street lamps or other means of communication to provide alerts to the vehicle 105 in real time such as upcoming traffic, construction, accidents, etc.

In a fifth example, the vehicle device 110 of the vehicle 105 and further vehicle devices in further vehicles may transmit respective data to the exterior component such as the lighting server 135. Through gathering volumes of data, the lighting server 135 may characterize night-time drivability for the road 125 in a specific sense as well as for a general area including the road 125 in a city-wide scale. When related to a single road 125, the characterization based upon the received data may provide indications of conditions for night-time drivability. For example, if the majority of the data from vehicles indicate that a particular street lamp generates a glare, the ITS may consider reducing the illumination of this street lamp. Contrarily, if the majority of the data from vehicles indicate that a particular stretch of the road has little to no assistance in night-time drivability, the ITS may consider adding lighting components such as a street lamp or reflectors. When related to roads of an area, the data may be amassed to reflect driving safety. The data may then be converted to visualize the scores in the form of an overlay on a geographic map (e.g., a heat-map overly).

Fig. 3 shows a method 300 for evaluating images to assist in night-time drivability according to the exemplary embodiments. Specifically, the method 300 may relate to the mechanism of the exemplary embodiments in which the images are analyzed to identify objects and/or conditions that are critical to safety for night-time drivability and has a probability of going unnoticed by a driver. The method 300 will be described from the perspective of the vehicle device 110. The method 300 will also be described with regard to the system 100 of Fig. 1 and the vehicle device 110 of Fig. 2.

It may be assumed that the vehicle 105 is on the road 125 either in motion or stationary (e.g., preparing to move) while the ambient, natural light is low or none. It may be further assumed that the imager 120 is properly set to capture images from the PoV of the driver. For example, a camera may take images of the driver and determine the field of view of the driver to determine the PoV of the driver.

In step 305, the vehicle device 110 receives the image captured by the imager 120. As described above, the image may include a variety of different objects and/or conditions such as those described in Figs. 4A-C. For example, lane markings, road defects, and lighting conditions may be included in the image. Thus, in step 310, the visibility application 250 of the vehicle device 110 may define Rols in the image that include the objects/conditions. That is, step 310 may relate to utilizing machine learning and other image analysis techniques for object identification.

In performing the objection identification, according to an exemplary embodiment, this process may be used for automatically annotating the image captured by the imager 120 with bounding boxes that signify the Rols in the image. With regard to an image captured from the PoV of the driver and an object of interest, the process of object identification may entail identifying a group of pixels in the image that correspond to the object of interest. In a first example, the process of object identification may be performed using pre-defined templates of different objects such as lane markings, pavements, pedestrian crossings, roadway signage, etc. In a second example, the process of object identification may be performed using machine- learning based techniques that learn Rols in an image. In a particular example, the machine- learning based technique may be used to detect potholes in the road. As potholes are not standard objects from which a template may be created, potholes represent anomalies in the roadway structure. Thus, the machine-learning based techniques may be trained to detect anomalies and identify them as Rols for saliency detection and further processing.

In step 315, the visibility application 250 of the vehicle device 110 determines saliency values for each Rol. As described above, the visibility application 250 may utilize a saliency detection operation to determine a prominence of objects/conditions in the Rols according to the PoV of the driver. Thus, a Rol with a high prominence may have a high saliency value associated therewith whereas a Rol with a low prominence may have a low saliency value associated therewith.

According to an exemplary embodiment, the saliency values may be determined using a variety of different methods. For example, the exemplary embodiments may utilize an entropy-based method that uses information gain as a driving metric to learn salient regions in an image. In another example, the exemplary embodiments may utilize a hierarchical technique that incrementally builds layers of saliency in an image. According to the exemplary embodiments, in a further example, the entropy-based method and/or the hierarchical technique may be trained in a context of analyzing night-time drivability to ensure maximal accuracy and minimize false detections. Furthermore, the entropy-based method and/or the hierarchical technique may take into account a psychological aspect of manually driving to focus on a pre-attention phase of human vision to enable the exemplar embodiments to focus on parts of the image that correspond to the regions in the PoV that receive maximum attention within the first few seconds of view. Those skilled in the art will understand that these few seconds may be critical when the driver has to react to emergency situations.

In a specific example of how Rols are detected and a saliency value is associated therewith, an image may include Rols that are at least one of an identified object or of a particular illumination level. That is, the Rols may be an object fitting a template or a learned object shape. One manner of assigning the saliency values to identified objects is based on distance. For example, the image may include Rols that represent first and second pedestrian crossing lane marking in which the first pedestrian crossing lane marking is closer in proximity to the driver than the second pedestrian crossing lane marking. Based on the distance from the driver, the saliency value assigned to the pedestrian crossing lane marking may be determined in which the first, closer pedestrian crossing lane marking has a highest saliency value (e.g., 87%) than the second, further pedestrian crossing lane marking (e.g., 72%). The Rols may also be an area in which an illumination level is different from the surrounding areas. That is, the Rols may be an area in which a light source is providing illumination or an area in which light is relatively absent. One manner of assigning the saliency values to areas of a particular illumination level is through comparison to an overall average illumination level in the image or an average illumination level in the image where artificial light sources are absent. For example, the image may include Rols that represent a first area having a first illumination level above an overall average illumination level and a second area having a second illumination level above the overall average illumination level but lower than the first illumination level. Based on the illumination level that is above the overall average illumination level, the first area may be given a first saliency value (e.g., 98%) while the second area may be given a second saliency value (e.g., 77%). The relative size of the Rols may also be considered in assigning the saliency value such as a larger high illumination level area having a highest saliency value while a smaller low illumination level having a lower saliency value.

It is noted that the image may also include a Rol that is a combination of the two examples described above with objection identification and illumination level. For example, the image may include a third area having a third illumination level that is lower than the overall average illumination level. However, this third area may be identified with the object identification as possibly matching a particular template. However, given the low illumination level, the saliency value that is assigned may be relatively low (e.g., 13%).

Based on the Rols and the corresponding saliency values, the visibility application 250 may perform at least one further operation associated with a respective further feature. As described above, the visibility application 250 may generate a display in step 320, perform an adjustment in step 325, and/or transmit a report in step 330. In generating a display, the display may be an information display on the display device 215 or a graphical representation on the display device 215 or an overlay on the windshield. In performing an adjustment, the light source 115 may be adjusted based on the Rols and the corresponding saliency values. In transmitting a report, the lighting server 135 may receive the Rols and the corresponding saliency values such that an adjustment may be provided (e.g., changing an illumination of a street lamp) or an amalgamated feature may be determined for drivability conditions on the road 125 or city- wide.

The exemplary embodiments have been described above with regard to nighttime drivability. However, it should be noted that night-time drivability is only exemplary and the exemplary embodiments may be utilized for any driving condition in which safety- critical objects/conditions are emphasized to a driver. For example, weather conditions during the day (e.g., precipitation, fog, etc.) may also affect the prominence of safety-critical objects/conditions resulting in low saliency values. The exemplary embodiments may be modified for such as use as well.

The exemplary embodiments provide a device, system, and method of evaluating images to assist in night-time drivability. The image may be captured from the PoV of the driver of a vehicle. The image may be analyzed to identify objects and/or conditions in the image. The image may be further analyzed to determine the prominence of the objects and/or conditions based on the PoV of the driver. For objects/conditions that are not as prominent may be emphasized so that these objects/conditions are also recognized and the driver may adjust accordingly.

Those skilled in the art will understand that the above-described exemplary embodiments may be implemented in any suitable software or hardware configuration or combination thereof. An exemplary hardware platform for implementing the exemplary embodiments may include, for example, an Intel x86 based platform with compatible operating system, a Windows platform, a Mac platform and MAC OS, a mobile device having an operating system such as iOS, Android, etc. In a further example, the exemplary embodiments of the above described method may be embodied as a computer program product containing lines of code stored on a computer readable storage medium that may be executed on a processor or microprocessor. The storage medium may be, for example, a local or remote data repository compatible or formatted for use with the above noted operating systems using any storage operation.

It will be apparent to those skilled in the art that various modifications may be made in the present disclosure, without departing from the spirit or the scope of the disclosure. Thus, it is intended that the present disclosure cover modifications and variations of this disclosure provided they come within the scope of the appended claims and their equivalent.

Claims

CLAIMS:

1. A method, comprising:

at a vehicle device associated with a vehicle traveling on a road: processing a received image to determine a point of view (PoV) of a user of the vehicle;

identifying a plurality of region of interests (Rols) in the image, the Rols including a predefined object/area of interest on the road or on/near the vehicle;

processing the Rols with a saliency detection operation to identify prominent Rols from the plurality of Rols, based on the user's saliency of the processed image ; and determining a saliency value for at least the prominent Rols based predetermined safety-critical features.

2. The method of claim 1, wherein, when the saliency value is below a threshold saliency value, the method further comprising:

displaying at least one of a textual information or a graphic representation of the Rol to the driver.

3. The method of claim 1, wherein, when the saliency value is below a threshold saliency value, the method further comprising:

generating an overlay of the Rol to be shown to the driver.

4. The method of claim 1, wherein, when the saliency value is below a threshold saliency value, the method further comprising:

performing an automatic adjustment to a light source of the vehicle.

5. The method of claim 4, wherein the automatic adjustment is one of a change to a level of illumination produced by the light source, a turning on of an element of the light source or a turning off of an element of the light source.

6. The method of claim 1, wherein, when the saliency value is below a threshold saliency value, the method further comprising:

transmitting data of the prominent Rols and the saliency values to a lighting server.

7. The method of claim 1, wherein the object comprises a lane marking, a road defect, an illumination condition, and a combination thereof.

8. The method of claim 1, wherein the drivability is a night-time drivability.

9. A vehicle device associated with a vehicle traveling on a road, comprising:

an imager generating an image from a point of view (PoV) of a user of the vehicle; and

a processor configured to receive the image, identify regions of interest (Rols) in the image, the Rols including an object that affects a drivability on the road, processing the image with a saliency detection operation to identify prominent Rol, based on the user's saliency of the image, and determine a saliency value for at least the prominent Rols based on predetermined safety-critical features.

10. The vehicle device of claim 9, wherein, when the saliency value is below a threshold saliency value, the processor further displays at least one of a textual information or a graphic representation of the Rol for the driver.

11. The vehicle device of claim 9, wherein, when the saliency value is below a threshold saliency value, the processor further generates an overlay of the Rol to be shown to the driver.

12. The vehicle device of claim 9, wherein, when the saliency value is below a threshold saliency value, the processor further performs an automatic adjustment to a light source of the vehicle.

13. The vehicle device of claim 12, wherein the automatic adjustment is a change to a level of illumination produced by the light source.

14. The vehicle device of claim 9, wherein, when the saliency value is below a threshold saliency value, the processor further transmits data of the prominent Rols and the saliency values to a lighting server.

15. The vehicle device of claim 9, wherein the object comprises a lane marking, a road defect, an illumination condition, and a combination thereof.