WO2019042728A2 - RECOGNIZING TRANSPORT PARTICIPANTS ON A TRANSPORT ROUTE - Google Patents

Abstract

The invention relates to a method (10) for detecting road users (12) on a traffic route (14) in an image, comprising: - generating (42) a plurality of range proposals (18) for possible objects shown in the illustration (16) recorded by applying a range proposal generator; - providing object detection (72) for all area proposals (18) to detect the traffic route (14) and / or the road users (12) by classification taking into account a predefined confidence level; Outputting detection data received by the object detection; and providing filtering (48) for the range suggestions (18) before the step of providing object detection, the filtering being performed based on respective filter data based on a relevance of the range suggestions (18) associated with the traffic participants (12). and / or the traffic route (14).

Description

DESCRIPTION

The invention relates to a method for detecting road users on a traffic route in a map provided by a camera taking the image, the method generating a plurality of area proposals for possible objects, which in of the map by applying a scoped generator to the map, providing object detection for all scoped proposals by applying an object scouting device to all scoped proposals to detect the traffic route and / or the road users by classifying taking into account a predetermined confidence level, and outputting of detection data received from the object detection for the detected traffic route and / or the detected road users. The invention also relates to a device for detecting road users on a traffic route in an image, wherein the device comprises at least one camera that records the image of the traffic route, and a device that is configured to provide a plurality of range proposals for possible objects, the in the figure, by applying a range suggestion generator to the mapping, to provide object detection for all range suggestions to detect the traffic route and / or the road users by classifying taking into account a predetermined confidence level, and to output detection data obtained from the object detection received for the detected traffic route and / or the detected road users. Moreover, the invention also relates to a method of providing traffic guidance comprising detecting traffic participants on a traffic route in a sequence of successive images provided by a camera comprising the sequence of images of the traffic route, determining a used capacity of the traffic route provided by the road users, identifying individual actual speed and / or position of each of the road users to determine respective individual actual tracks, determining at least one respective nominal track for each of the road users, and communicating the specific nominal lanes to the respective road users. Further, the invention also relates to a traffic guidance system for providing traffic guidance comprising at least one device for detecting traffic participants on a traffic route in a sequence of successive images, and a device configured to detect traffic participants on the traffic route in the sequence of successive images to determine a used capacity of the traffic route provided by the road users, to identify individual actual speed and / or position of each of the road users, to determine respective individual actual tracks, to determine at least one respective nominal track for each of the road users, and certain nominal lanes to the respective road users to communicate. Finally, the invention also relates to a computer program product having a program for a processing device.

Current methods and devices are commonly applied to camera-based monitoring and control of traffic junctions. This may include automatic determination and assignment of an adaptive time window and / or a lane of a traffic route or a complete route to each of the vehicles on the traffic route, as road users.

So far, monitoring and control of traffic junctions have usually been conventional

Control systems with static light, operated by the vehicle Traffic light control systems, respective roundabout, extra lanes provided during peak times, and the like. The invention thus relates to the use of data from at least one camera, in particular video data in an outdoor environment for traffic monitoring.

The growth and scale of vehicles makes traffic management more difficult. Existing and conventional traffic management control systems are based on timing mechanisms, such as traffic lights, which typically provide equal time windows for each intersection track, resulting in inherent inefficiency due to non-uniform traffic flow. In addition, at peak times, roundabouts and extra tracks may be provided which typically require greater land area or require demolition of adjacent structures for their construction.

A more efficient approach is based on providing adaptive time windows for each lane of the road based on real-time traffic density data through advanced communication systems of a corresponding set of sensor technology for observing the traffic scenario.

Autonomous vehicles or self-driving automobiles are becoming increasingly feasible, and the intercommunication between multiple vehicles and local intersection control devices allow automatic interaction.

Although current vehicle coordination techniques are designed for driving on the open road to work with human drivers, the concept of autonomous vehicles allows the application of overall road control devices that interact directly with the individual vehicles and identify the most efficient and safest navigation path in both urban traffic scenarios and Allow heavy traffic hubs. In the case of automobiles driven by humans, the service road and intersection control device may provide at least navigation information about optimal lane and stripe selection along with the most useful speed recommendations.

The conventional traffic control system is based on stop panels, traffic lights or the insertion of recessed loop detectors for vehicle detection and automatic

Speed control displays to inform drivers. For traffic flow control, traditional systems employ intersection control mechanisms that usually assign equal or preprogrammed time slots for each intersection lane, with the temporal switchover pattern being limited to establishing uniform vehicle flows for man-driven non-automated automobiles.

In view of the prior art, there remains a need to improve traffic control in order to improve traffic flow and reduce congestion on the traffic route.

In order to cope with the above-mentioned subject matter, the invention proposes methods, devices and computer program products according to the independent claims.

Further improvements may be achieved by features of the dependent claims. In particular, with regard to a generic method for detecting road users on a traffic route, the invention teaches providing the range suggestions prior to the step of providing object detection, wherein filtering is performed based on respective filter data based on a relevance of the range suggestions the road user and / or the traffic route.

In particular, in connection with a respective generic device for detecting traffic participants on a traffic route in a sequence of consecutive maps, the invention teaches that the device is further configured to provide area range suggestions before providing object detection, wherein the filtering is based on corresponding filter data is estimated, which are estimated based on a relevance of the range proposals in connection with the road users and / or the traffic route.

With regard to a generic method for providing traffic guidance, the invention particularly teaches the detection of road users on the traffic route by applying a method according to the invention for detecting

Road users on a traffic route. The invention also teaches in particular for a traffic control system of the generic type that the device for detecting road users is configured according to the invention.

The invention is based on the finding that proper detection of road users on a traffic route can be a good concept to reduce the overall vehicle latency through camera-based traffic monitoring associated with each autonomous and automated automobile or vehicle. This allows optimizing the traffic flow and the roadway efficiency or

Transport path efficiency, in particular in view of the concept of autonomous road users, such as vehicles, automobiles, trucks and the like. Autonomous vehicles are equipped with mutual communication infrastructure, the means of vehicle coordination and traffic flow control without provides the limitations for significant acceleration or slowing or useless stopping.

In particular, the invention allows road users or vehicles to be connected to a central intersection control system that employs video infrastructure to check the recommended lanes for the individual vehicles. The video infrastructure can be provided by one or more cameras. This makes it possible to replace conventional traffic lights by the intersection control device.

The crossing control device, occasionally too

Called intersection management control device, has both the ability to detect and for communication and control, and may additionally be associated with preferably each of the approaching road users or vehicles to coordinate their individual trajectories, allowing a more adaptive and smarter traffic flow control. In addition, a time window-based intersection control mechanism having adequate temporal resolution additionally allows coordinating a preferably uniform vehicle flow, wherein the idling stop or the complete stopping of the vehicle at traffic lights can be smoothed or avoided, or remaining anticipatory movements, such as Adjusting a speed of a particular vehicle to the trajectory of a predetermined vehicle to provide substantially stationary traffic.

The invention results in that substantially all vehicles move at the same time and faster, so that the crossing efficiency can be significantly increased. With autonomous and automated road users or vehicles, traffic lights for managing traffic intersections are therefore unnecessary. The innovative concept can also be applied to feeder for motorways, for the merging of lanes in general, and the like.

In a specific embodiment, the detection of road users on a traffic route, in particular the automatic vehicle detection framework of the traffic

Junction control system, based on Deep Convolutional Neural Networks. This allows to readily consider that vehicles usually appear in pictures taken by the camera because of their variable distance from the camera at different scales. In order to detect the road users or vehicles of varying sizes, the device must in each case search for objects in several scales in the images or images. However, the search for multiple scales entails a high latency and could result in lower detection precision. In this regard, the invention allows to carry out self-commissioning. It automatically estimates and uses scene layout information such as floor space or the like.

In addition, the invention also allows a scale of the floor area to be estimated from a long-term observation of the appearing vehicles having different scales. This may be based on self-consistent analysis or regression, and the like. For this purpose, the acquisition of scaling information and / or

Land surface information from a long-term traffic observation can be provided, which can also help to speed up the detection. For example, the scaling information may be automatically included in the RPN.

Starting the inventive method or apparatus requires no knowledge of the present scene. The invention begins with detecting objects in all scales. After a sufficient number of objects, namely, road users, preferably vehicles, has been detected at several different areas in the image or image, the invention is capable of estimating a layout of the floor area of the scene. This makes the invention more robust and faster, since the number of scales for searching for an item at different positions in the image or image can be reduced.

Self-startup can therefore also be helpful if the camera position is reset, for example, during regular maintenance, service work that is usually provided from time to time, and thus there is no need to provide additional effort, such as technical assistance or startup ,

In general, the term "road user" can be applied to any user who participates in the traffic, and thus a road user can be a vehicle, a bicycle, a pedestrian, etc. In this regard, the term "traffic route" corresponds to any one on land based area that is provided so that road users can move on it. A traffic route can thus be a highway, a road, an avenue, a highway, but also a sidewalk and the like. It should be noted, however, that the invention is not limited to outdoor applications. It may also be applied to indoor applications such as warehousing in a warehouse where autonomous vehicles can transport goods to and from predefined storage locations and the like.

In this context, a "traffic route" may be formed by a predefined area for the autonomous vehicles on which the autonomous vehicles can be moved The predefined area may be a specific section of a floor intended for transportation, in particular within the warehouse , The camera is used to capture images related to a certain predetermined view. The camera is configured to take more than one image or only one image, in particular to capture a sequence of successive images of the same view. In this regard, the camera may preferably consist of a video camera. The camera provides image data that is provided to an image processing device that is preferably capable of real-time pre-processing a large amount of image data to reduce the volume of data to be transmitted to the intersection controller. The image processing unit may be formed of on-board processing, edge computing, and the like.

In this context, the image processing unit is preferably part of the camera. However, it may be provided by a separate unit communicatively connected to the camera. Real-time pre-processing means real-time detection and monitoring of road users, such as vehicles, bicycles, pedestrians, and the like, including lane calculation and simulation. The camera and the image processing unit may be part of an infrastructure unit that may include a street lamp mast, a building, and the like. The infrastructure unit may include the camera. It may additionally comprise a FLIR, an LPR, a RADAR and the like. The infrastructure unit can form a device according to the invention.

In order to carry out the invention, a detection frame may be provided which enables the inventive method to be carried out and to constitute the inventive device. In this regard, the coverage framework may be able to Scene and street layout data, for example, related to the floor area or the like may be used as additional knowledge to improve the efficiency and robustness of a detection algorithm by automatically cropping the object scales. This means that knowing the layout of the floor space allows the camera to relate the internal scales to absolute real scales. The detection frame therefore makes it possible to create a virtual three-dimensional scene of the objects recorded in the images. In particular, it may be the size of the visible 2D object that is useful for object recognition. Preferably, the detection frame is capable of performing self-commissioning. That is, in an initial phase, the detection frame may gradually acquire data about the scene layout, data from a long-term observation by analyzing appearing quantities and scales of the detected road users and / or the detected traffic route. After the detection frame has detected a sufficient number of road users, preferably at several different areas in the image or image, it may be able to estimate the layout of the floor area of the scene. The layout of the floor space can thus be gradually tuned and improved over time. The new setting of the camera, in particular the camera position, can therefore be considered and automatically adjusted. The capture frame may also allow an operator to directly input certain data associated with the scene, particularly the bottom surface of the scene. This can be accomplished by providing certain parameters of visible floor space which can be provided in advance as commissioning. In this context, manual commissioning can be provided. if the Knowledge of the scene layout is available, a search space for the scale during the tracking of road users can be reduced. According to the invention, a first image of the sequence of successive images is optionally selected and defined as the image to be processed. The first image does not need to be the first mapping of the sequence of images. In general, it is possible that this map may be any of the sequence of maps. Preferably, however, it is the image that has an earliest timestamp. For purposes of selection, a particular selection unit may be provided which selects the first image from a database in which the sequence of images is stored, at least in part, for example, temporarily stored. However, it may be provided that an image provided by the camera is directly defined as the image to be processed and subjected to the inventive process.

Then, a range proposal generator generates a plurality of range suggestions to process possible objects recorded in the image. The

A range suggestion generator may be a computing unit, such as a computer having a microprocessor, a Digital Signal Processing (DSP) or the like, controlled by a particular computer program to generate the range suggestions.

Object detection is preferably provided for all area proposals by applying an object detection apparatus to all area proposals to detect the traffic route and / or road users while considering a predetermined level of confidence. The object detection leads to the generation of acquisition data for the detected traffic route and / or the detected road users who are issued for further processing or reporting.

Then, the method may be continued by selecting another mapping of the sequence of consecutive mappings than the image to be processed.

According to one aspect of the invention, filtering of the range proposals is provided before performing the step of providing object detection, wherein the filtering is performed based on respective filter data estimated based on a relevance of the range suggestions associated with the road users and / or the traffic route become. This makes it possible to find the bottom surface of the scene, based on this finding, the cost of object detection can be reduced because the number of range suggestions can be reduced. In particular, if the floor area or the traffic route is available, the generated area suggestions may be selected such that only the area suggestions that may result in relevant detected objects are taken into account during the object detection.

Filtering the range suggestions therefore results in a reduction in the number of range suggestions produced by the range suggestion generator. The filtering is performed based on respective filter data that is estimated based on a relevance of the range suggestions in connection with the road users and / or the traffic route, in particular the floor area of the scene.

According to an exemplary improvement

Provided scale information prior to the step of performing object detection, wherein scale information is acquired from long-term observation of the detected vehicles on the ground surface. In particular, a floor surface scaling off the self-commissioning phase are derived. This makes it possible to improve the invention and to reduce the effort, in particular in connection with the execution of the object detection.

Preferably, the filter data is estimated based on a scene captured by the image to be processed, in particular the traffic route included in the scene. This makes it possible to provide the scaling so that the object detection can be improved. In particular, if it is assumed that predetermined classes of objects are detected, the scaling can be additionally taken into account in order to reduce the effort for object detection. Over time, the scaling data can become more precise.

In addition, it is proposed that the filter data provide an observation area for an object that is classified by the object detection as a road user. The observation area may be derived by identifying one or more areas within the image to be processed where predefined object classes may or may not appear. In particular, if the road user is a vehicle, a bicycle, a pedestrian, the classification provided by the object detection can be improved. The invention may also allow filter data to be updated in dependence on the data of the object detection. It is therefore possible to improve existing filter data by further carrying out the invention. Over time, the filter data can become more precise.

The observation area may be, for example, a horizon derived from the image to be processed. Usually, objects such as vehicles, bicycles, pedestrians, and the like may not appear above the horizon. Range proposals above the horizon can thus be eliminated from further processing.

In addition, the size of the range suggestions may be more precise by using the filtering. As far as the vehicles are concerned, the size can be selected by considering the scene and the position at which a particular range suggestion should appear. Preferably, deep learning methods and / or artificial intelligence can be used. For example, the infrastructure unit may employ deep learning techniques and / or artificial intelligence along with the inventive concept to perform road user detection, such as vehicle detection, vehicle identification, vehicle tracking, and the like.

The infrastructure unit may also employ deep learning techniques and / or artificial intelligence to learn from long-term observations a relationship between visible object scaling associated with the location of the object on the floor surface of the intersection, particularly in a particular image to be processed. It may also learn long-term observations relating to the area and area of a field containing valid observations that may be provided by the horizon line or the like. The

The infrastructure unit may also employ the deep learning method and artificial intelligence, and may apply the knowledge gained about scales related to the detected objects, taking into account a location in the map to be processed, to optimize the performance parameters of the object recognition procedure, such as Example latency, reliability and distance range, where objects beyond the horizon line need not be taken into account.

Furthermore, it is proposed by way of example that at least one of the road users requests a desired train, which causes the determination of a specific nominal track for the road user and the communication of the determined nominal track to the road user. This allows to consider certain wishes of the road user. The

For example, the infrastructure unit may check the request and determine the nominal lane by taking into account lanes of all road users. The nominal track can then be announced to the road user, so that the road user can drive on the nominal track. This can also result in a better flow of traffic as the infrastructure unit can accommodate most, especially all lanes of current road users.

By way of example, it is also proposed that the testing of a specific nominal track by simulation be included for the road user. In particular, an S / W based check of the webs may be provided prior to shipment.

In another example enhancement, retrieving an actual speed and / or position of at least one of the road users is included to determine its actual trajectory.

In a further exemplary improvement, the speed of the road users is automatically adjusted as a function of the used capacity of the traffic route.

In another refinement, automatically adjusting the speed of at least one of the road users is included as the road user approaches a predetermined transit pattern for the traffic route. These Improvement may relate in particular to autonomous driving, preferably indoors as well as outdoors.

Each of the infrastructure units, in particular inventive devices, may be capable of bidirectionally communicating the road users, in particular detected vehicles, to receive inquiries from the road users and respective intersection control devices and dispatch instructions to the road users.

The infrastructure unit may also be able to receive requests and command data from the road users and the intersection control devices, as well as resend dispatch of processed object information.

Preferably, all of the infrastructure units are preferably interconnected and may form a mesh network topology for bidirectional communication, vehicle requests, and commands from the intersection controller to avoid single fault locations and increase the area of the intersection area by applying refreshes and iterations.

More preferably, the infrastructure unit may communicate with preferably all of the approaching road users, particularly vehicles that reach the communication area, and may query the coordinates of requested destinations and forward the requests to the intersection controller.

At least one of the infrastructure units located at the intersection or access may also host a central intersection control system which may preferably collect traffic data from all of the infrastructure units, including requests from the road users, particularly from the vehicles. The intersection control device may coordinate a plurality of self-propelled autonomous or automated road users or vehicles requesting access to a respective exclusive trajectory on the traffic route, in particular one or more lanes to the intersection. In this way, a specific passage for a particular road user can be reserved. Further, the intersection control device may be the

Reservation request from the road user, in particular the vehicle received and can calculate a trajectory that can be simulated directly in relation to the actual traffic data that are received in particular from external sensors, together with the path data of other vehicles in advance. The intersection controller may handle requests from the road user, particularly vehicles, and provide uninterrupted scheduling and control. In addition, the intersection control device may assign each of the road users, in particular vehicles, a specific time slot or lane in the prevailing transit pattern. The communication between the intersection control device and the road users or vehicles can be carried out by the infrastructure units.

In addition, the intersection control device may interrogate any precise location and speed of preference of each of the individual road users, particularly the vehicles, and may calculate their individual lanes for precisely controlling the position of each of the road users at a particular time. In addition, the intersection control device can both a conventional coordination mechanism in the form of a Deterministic state machine and artificial intelligence based on a coordination mechanism in the form of monitored and rule-based machine learning techniques.

The intersection controller may decide to grant or deny the request in response to the prevailing traffic flow and to the basic intersection control policy.

Preferably, the intersection controller may apply on-board simulation analysis to test the functionality of proposed vehicle lanes. Uninterrupted and ongoing intersection simulation can help to avoid collisions.

If the request of an individual road user can not be granted, the road user must be slowed down in order to wait for a later reservation permit. In the worst case, the road user can be stopped completely before he gets permission to enter the intersection.

Preferably, if the vehicle is automatically driven or automatically controlled, the speed of the individual vehicle may be automatically adjusted as it approaches a four-scene pass pattern so that the vehicle enters the projected trajectory at the correct time and time slot Overall pattern can flow continuously undisturbed uninterrupted.

In heavy traffic conditions, any one of the intersection controllers organizes, preferably all

Junction control devices, preferably the approaching vehicles into virtual batches, rather than coordinating the overall traffic flow based on the individual vehicles. The Heavy Traffic Cooperative Truth Control may cause the vehicle to follow another to share joint acceleration maneuvers on a feed forward control path through vehicle-to-vehicle communication.

Preferably, the intersection control device is preferably connected to at least one camera-based surveillance sensor (CCTV camera) to control video data of the prevailing traffic scenario.

Preferably, the intersection control device may host an on-board image processing system capable of identifying the individual road users, especially vehicles, on the basis of the appearance to temporarily draw an individual identity index allowing fully automated tracking and re-identification in the case where the vehicle reappears in the field of view of adjacent camera sensors downstream of the traffic route.

For this purpose, the intersection control apparatus may host on-board image processing capable of identifying the individual road users, particularly vehicles, based on automatic number plate recognition by means of Optical Character Recognition (OCR) or the like.

In addition, the intersection control device may host an on-board vision system capable of identifying and tracking pedestrians, especially pedestrians, outside of crosswalks.

The onboard vision system may also be configured to identify and track pedestrians. Preferably, the onboard image processing system of

Intersection control device configured to be pedestrian outside the crosswalk and to follow it on the traffic route, in particular, if the traffic route is a road, or at the intersection, to trigger the execution of a special collision avoidance procedure, offering new road users to the road users, especially nearby vehicles. and assign route data.

Preferably, all the infrastructure units with the crossing control unit, which can be replaced by a

Junction control device is provided to communicate.

The teachings of the present invention may be readily understood, and at least some additional specific details will become apparent upon consideration of the following detailed description of at least one exemplary embodiment, taken in conjunction with the accompanying drawings, in which: FIG Have a variety of range proposals for the detection of objects and three exemplary classified objects;

 Figure 2 shows in a schematic three-dimensional sketch a scaling function according to the invention for filtering proposals using estimated scaled filters;

Figure 3 is a schematic two-dimensional sketch which is a projection of Figure 2 in a plane used to estimate the scaling functions;

FIG. 4 shows the figure to be processed according to FIG. 1, wherein the range proposals are filtered according to the invention, and additionally shows that FIG Range proposals get smaller as they approach the horizon; shows a schematic flowchart showing a method for detecting road users on a traffic route according to the invention; shows a schematic sketch scale compared to recognition for VGG-M networks, where bins are determined by uniformly distributed patterns; Figure 10 shows a schematic occlusion sketch as compared to recognition for VGG-M-Net, where bins are determined by uniformly distributed patterns, and with a maximum occlusion ratio set to 0.5.

The detailed embodiments described below are concerned with how the invention is particularly subjected to autonomous driving. However, the invention is not limited to outdoor applications but can also be applied in indoor applications, such as warehouse applications, particularly with regard to warehousing and the like.

Autonomous driving still remains a major challenge in that the environment as picked up by one or more cameras set up to observe images of vehicles as traffic participant changes that can occur quickly and unexpectedly. Vehicles, for example, can be parked on the roadside, various initiatives and events in the city center can affect the traffic, and the like. In particular, the presence of more people may result in a higher chance of someone crossing the street. In this context, infrastructure-based mapping has the potential to complement the single vehicle point of view and accelerate the deployment of fully autonomous vehicles. More particularly, the invention relates to the detection and detection of road users, such as vehicles, from the foresight of surveillance cameras, which substantially impart to the vehicles a perception of the vehicle ahead of them and behind a corner.

The cameras may be a component of one or more devices according to the invention. There is great potential in using the static view of a surveillance camera, which can provide better and faster capture.

In particular, the invention relates to range proposals that form an important feature of modern detection algorithms. In this regard, the invention proposes a simple extension of the R-CNN (Regional-based Convolutional Neural Network) and shows that ranking proposals in relation to a scene geometry can result in less false positive results by reducing suggestions in overloaded areas where respective algorithms are usually prone to error.

In addition, the invention gives less false negative results since it increases recognition by containing more suggestions where they are most needed, for example for small vehicles at the distance. In connection with such an embodiment, experimentation is made with the UA DETRAC data set, which can improve on the Vanilla Faster R-CNN (VGG-16) by more than 19%. This improvement can be largely maintained when switching to a Faster VGG-M network. According to the invention, it is proposed to generate 3D object proposals by using scene geometry using calibrated monocular or stereo camera arrangement. An automatic estimate of the approximate scene geometry in terms of a true-to-scale layout is proposed.

This information is incorporated into a detector to produce range suggestions. The inventive approach takes into account that the scene is largely static but usually does not require camera calibration information.

The importance of high-resolution functions is also addressed in the semantic segmentation literature, where the goal is to determine precise object boundaries and high-quality semantics. For dense prediction tasks, de-convolutions are typically used. However, a simpler alternative has been proposed, namely the removal of max-pooling layers for dense function maps, and the use of extended convolutions to maximize contact size without increasing the number of parameters. Removing pooling layers can also affect capturing small-format pedestrians. However, they do not take into account extended folds.

The invention extends Faster R-CNN in conjunction with object detection to incorporate proposed geometric suggestions. Geometric proposals encode the scene layout of a static camera in a simple and effective way. In general, the Faster R-CNN detection can work in at least two stages. In a first stage, a full Region Proposal Network (RPN) is provided which can take the complete image or image that is to be processed at the input and generate class-agnostic object suggestions. The second stage is based on a classification network that classifies the incoming proposals into given object classes. The convolutional layers are shared for both tasks, that is, generating proposals and classifying them.

Geometric extension to RPN Typically, an image or image that is to be processed may potentially contain a few large objects and a few or more small objects. However, this is usually not considered in the Faster R-CNN algorithm, particularly due to the RPN which suggests an equal amount of objects across scales. A relationship between the range suggestions and the scene geometry is retrieved using an object scale estimate. First, the safest objects are captured. Then, a pixel-by-pixel scale estimate is estimated as a proxy for the actual scene geometry. Finally, the RPN proposals were curtailed.

Initial observations The safest (with high results)

Road user surveys are used to automatically estimate this layout, shown in FIG. An initial seconds view of a video sequence, such as data from 10s of reliable acquisitions, is enough for a truly reliable scale estimation. It should be noted that the safest detections are generally not occluded and not truncated.

Pixel-wise scale estimation Then, the scale layout for the image to be processed or the image to be processed is estimated, that is, an image scale function describing the scale of the object in view of its position in the image. In this case, this corresponds to the expected size of a road user or vehicle in a certain image position.

Suppose an initial set of acquisitions at positions {x} i ^N , where x is the center of capture bounding rectangles and N is the number of initial acquisitions. It is aimed at estimating a scale function (x) that presents the size of the object bounding rectangle in pixel ² at each pixel coordinate. A second rank polynomial is assumed to approximate the function and to fit it using the least squares method:

(1) where p ₂ , Pi and po are parameters to be estimated. Notes on polynomial adaptation

The second order polynomial suffices to represent a flat scene layout from a homographic projection. This assumption is plausible in most street scenes, as traffic routes visible from the surveillance camera are mostly flat. The approximation of the scale layout compensates for the size variations of the objects or vehicles, for example automobiles of different sizes and the like, in view of the sufficient number of acquisitions. This is illustrated by a test estimation according to FIG. The Maßstablayout automatically provides a horizon estimate, which is also indicated in Figures 2 to 4 by the reference numeral 50. proposal pruning

Within the RPN module of the proposed GP-FRCNN, object proposals are clipped as follows:

II s () - b II

 <σ

 s ()

 (2)

Here, s (x) is the scale estimate of the object at position x as described in equation (1), and b is the object's actual bounding rectangle size, o represents the acceptable deviation of the default size from the scale function Value set to 0.3 for all different embodiments based on the observed variants in the training data.

Beyond an object class

For the DETARC Challenge, only vehicles are considered as target acquisition for the estimation of the scale layout. It should be noted, however, that equations (1), (2) can also apply to the estimation of other object sizes.

With a given single correct scale observation of another object, for example a bus, as a certain vehicle, one can adjust the scale estimator s (x) by simply scaling it with the factor - ^ -, where s (x) s (x)

the original scale function estimate is the pixel position, and b 'is the size of the bus sense.

Fine tuning for Faster R-CNN

The use of geometric suggestions is a simple extension to the Vanilla Faster R-CNN, but its ease of integration can degrade performance. Below is Adaptation proposed to improve the model or process. The adjustments may apply to different model settings, for example the selection of the network.

Specific anchor scales

During training, Faster R-CNN can separate the object bounding rectangles into anchor scales and expected ratios. By default, the scale set of anchor rectangles can be {8, 16, 32}. This may be appropriate for most acquisition benchmarks, such as PASCAL VOC. However, applying the Faster R-CNN to the standard anchorage standards on the UA-DETRAC standard can be seriously lower than expected because most vehicles are much smaller than the smallest standard scale. The range suggestions corresponding to the smallest anchor rectangles must therefore serve for any object smaller than its adjusted size in contrast to the actual concept of anchor scales. This problem can be remedied by extending the set with smaller scales in the sequence, that is {1, 2, 4}, for which results may be shown in FIG. Figure 6 shows in a schematic sketch the scale compared to recognition for a VGG-M network, where bins have been determined to be uniformly distributing patterns. FIG. 6 shows a sketch 52 in which an ordinate 54 is associated with recognition in%, and an abscissa 56 is assigned to the average vehicle size (pixel ² ). The tuples of bins concern bin 58 corresponding to FRCNN, bin 60 corresponding to FRCNN + GP, bin 62 corresponding to FRCNN + BW, and bin 64 corresponding to FRCNN + BW + PG. In an alternative embodiment, quantized scales may also be used in connection with the training data be experimented. However, it has been found that both techniques result in similar performances, so below only the simple extension to the scale set of the anchors in RPN is considered.

Function maps with higher resolution

A second limitation of the Faster R-CNN to smaller objects can be given by the quasar resolution of its CNN function block. This issue has been identified several times in semantic segmentation, which allows quasar granularity to limit pixel-by-pixel resolution.

Therefore, before the proposal of the Faster R-CNN and the classification system, a Finder function map is proposed. In more detail, the functional distance is reduced from 16 to 18 by removing the last max-pooling layer from the base function networks on all the models experimented.

This can effectively result in increasing the number of digits on the image or map to search for the object, resulting in a significant gain in recognition for the small vehicles, as seen in FIG.

It should also be noted that this may result in a smaller respective field on the input image or on the input image being processed. Although this may not affect all of the small objects that the model needs to view a larger area in the object, this can reduce recognition for the larger vehicles in the experiments for which the context becomes too scarce. This effect may be more apparent in smaller functional models, such as FGG-M, while larger models seem more robust and potentially maintain sufficiently large respective fields.

Multi-stage training

Learning the parameters for all convolutional layers of the acquisition task is not easy, so in the standard strategy for training the Faster R-CNN model, the parameters are preferably initialized with the pre-trained ImageNet model, and the learning of the first f 4 convolutional layers can be skipped. This means that the low-level functions in the basic Faster R-CNN model can still be those that are actually trained only for the image net classification tasks. Of course, that does not need to be an optimal setting. However, a multi-level training approach may be used, and these initial convolutional layers skipped in the standard training stage of the Faster R-CNN may also be learned. In the first stage, the default strategy can be maintained and the parameters of the intimal convolution layers of the networks are not learned. At the second stage, the training policy may continue on the full network after the original convolution layers are also unlocked. Alternatively, one could also study a strategy similar to warm-up training, with very small learning raids.

Experiments and Results The details of the experiments and the results of the inventive approaches on the UA-DETRAC are provided, which is a very comprehensive record for surveillance scenarios. The dataset consists of 100 video sequences (60 for training, 40 for testing) that present real traffic scenes in different weather conditions. Network structures

Two different variants of the VGG network structures are used. The first is VGG_CNN_M_1024 with 5 convolutional and 3 fully connected layers. Below this network is called VGG-M. The second is VGG-16 with 13 convolutional and 3 fully connected layers. training strategy

The following strategy is described in the original report of the UA DETRAC dataset for selecting round-truth

Vehicle notes for training the models described. This means that only vehicles with less than or equal to 50% occlusion and 50% truncation are included. To make the models robust and avoid excessive adaptation to DETRAC scenarios, PASCAL VOC 2007 and 2012 Trainval image sets were used along with the DETRAC images to train the models. Training is created for all 20 standard classes of the PASCAL VOC record. The ratings show that training for all 20 classes in general can be slightly better than training for vehicles only. For all experiments, the end-to-end approach of the Faster R-CNN is used for model training, which trains both RPN and the classification network simultaneously.

The selection of the NMS threshold is quite critical for typical object detectors. Since only vehicles with less than or equal to 50% occlusion are included in the validation rate, it may theoretically make sense to use an NMS threshold of 0.5. In addition, a stricter value for the Faster R-CNN parameter FG_THRESH, that is, 0.7 instead of 0.5, may be used for detection to be considered as a positive class during training of the classification network. This value is for everyone optimal results since the online UA DETRAC rating uses IoU of 0.7 to count a detection as correct. validation

It is assumed that the distributions of the vehicle data in the training and test sets are similar, so it makes sense to remove a validation set for comprehensive assessments. In the present embodiment, 36 videos are selected that have different viewpoints and weather conditions in a validation set. The remaining 24 videos are used to train the models in this validation phase.

To train the model, all frames from the selected 24 videos of the training set are used. For testing, every tenth frame from the 36 sequences of the validation set is considered. This allows us to quickly evaluate multiple approaches, including enhancements and refinements to the Faster R-CN framework. In Table 1, it is noted that a significant improvement in average precision (AP) can be achieved after recording PASCAL VOC data sets. The proposed method can be evaluated in detail using a smaller VGG-M network in conjunction with different aspects, including the ability to handle scale changes and different levels of occlusion.

Procedure AP (%) speed

 (Fps)

 Faster RCNN 58, 9 12

 Faster RCNN 64, 1 11

+ Extra anchor 68, 8 11 + high-resolution 72, 3 8

 function cards

 + GP GP-FRCNN 78,7 8

 + Multistage80, 9 8

 training

 Table 1

Scale Invariance Here it is reported that the final model is capable of enhancing the scale invariance characteristic of the original detector. It is shown that the fusion of geometric proposals and a modified version of the Faster R-CNN is able to significantly improve detector recognition regardless of the scale of the object.

The results are reported according to FIG. It is noted that the basic approach significantly misses performance for capturing smaller scale vehicles. The addition of the Geometric Proposals (GP) by using the inventive proposed scale layout (Figures 1 to 4) may not help at all. This indicates that basic models do not have enough ability to handle smaller scale objects as discussed above.

However, in the context of small objects, an improvement can be achieved by introducing barbs and whistles (BW) into the network, for example extra anchors aimed at smaller objects, and reducing the functional distance to allow higher resolution function maps. Although higher recognition may be achieved for the smaller vehicles, these changes may have a negative impact on capturing larger vehicles. In FIG. 6 it can be seen that recognition consistently decreases to BW. This shows that increasing the resolution of the feature cards may not always be helpful. As this can reduce the effective receptive field on the image or image being processed, capturing larger objects for the network may become difficult to handle. However, the proposed Geometrical Proposals (GPs), which may have previously been ineffective, can provide a substantial gain in recognition for smaller and medium vehicles, and also work in synchronism with the line model for the larger objects. It is an impressive statement that geometrically consistent proposals can significantly increase the detector's capacity for smaller objects without degrading performance for the larger objects.

Better occlusion handling

Here it can be assessed how good the models are in terms of handling different occlusion levels.

Interestingly, one notes a similar tendency that the introduction of the Beils and Whistles (BW) together with the Geometric Proposals (GP) may be able to master the classification failures of the RPN and provide the best results.

According to FIG. 7, the recognition of all models in connection with different occlusion levels is demonstrated. Figure 7 shows a schematic sketch 66 having an ordinate as Figure 6 and an abscissa 68 assigned to a major occlusion ratio. In the sketch 66, the tuple of the bins 58 to 64 corresponding to the bins according to FIG. 6 are shown. Recognition improves only slightly when the geometric proposals are applied to the Vanilla Faster R-CNN, demonstrating the limited capability of the model. It can be seen, however, that the Faster R-CNN with Beils and Whistles function significantly worse as the occlusion ratio increases, and in fact does not improve for any occlusion ratio level. This result explains the importance of a larger receptive field that provides greater context for the object on the image or image being processed, thus allowing better occlusion manipulation.

The above findings can also be made in terms of average precision. These results are provided in Table 1. In the case of the UA-DTERAC, the ratio of the object size to the image size is rather small by comparison, which is observed in a typical image of a PASCAL VOC data set. The task becomes easier if more suitable anchor measures are allowed to learn the regression parameters. It can be seen that adding additional scales for the anchor rectangles to detect smaller vehicles significantly improves the AP to 68.8%. As described above, increasing the resolution of the function maps may allow more suggestions and may further aid in capturing smaller objects. Of course this can increase the computing time. Furthermore, it can be noted that the proposed extension to the RPN by incorporating scale layout, which offers geometric suggestions, raises the AP by more than 6%. This result summarizes the gain that can be seen in Figures 6 and 7 for different object scales and occlusion levels. Finally, multi-level training, as described above, can further improve the AP to a remarkable 89.9%.

Finally, the proposed approach may be evaluated using the larger VGG-16 network. These results are shown in Table 2. Overall, slightly better results can be achieved but lose half the frame rate. Cooperatively it can be observed that smaller Networks (VGG-M) benefit significantly more from the geometric proposals. It can be noted that the scale changes make the larger model (VGG-16) function significantly better than the smaller one (VGG-M). However, the proposed geometric proposals reduce the power column again.

 Table 2 The UA DETRAC Challenge

The results for the UA-DETRAC Acquisition Challenge are reported below, and at the time of writing, all results currently available on the website are better in terms of average precision.

To train the models, the complete UA DETRAC train set (60 video sequences or pictures) is used. PASCAL VOC 2007 and 2012 Trainval image sets are also included, as is done in the validation phase. It can be seen that some of the traffic scenarios in the UA-DETRAC test satellite are relatively denser and more crowded than, comparatively, the video sequences in the training set. However, most of the results obtained for the online challenge are consistent with these scores during the validation phase. Overall, it can be improved by an impressive 19.5% in terms of AP compared to the Vanilla Faster R-CNN means from 57, 08% to 67, 57%. It can be noted that the effect of adding geometric suggestions is not as strong as observed during validation. It is believed that this is due to the fact that a large number of small scale objects are ignored during online evaluation. This may be due to the marked ignored detections in the image or image being processed. These results are shown in Table 3.

 Table 3

Figure 1 shows in a schematic view an image 16 to be processed, with all range suggestions 18 generated by a range suggestion generator. The figure 16 to be processed further shows a traffic route 14 having a plurality of lanes on which vehicles 12 drive as road users.

As can be seen in FIG. 1, three rectangles 20, 22, 24 are provided. The rectangle 20 is located in an upper portion of the image 16 to be processed so that this rectangle is too large because the road users to be captured in this portion of the image 16 to be processed are much smaller. The rectangle 24 in the lower section of the On the other hand, Figure 16 to be processed is too small to contain a road user or a vehicle. In contrast, a rectangle 22 in the lower portion of the image 16 to be processed is adapted to contain a road user or a vehicle. The size of the rectangle coincides with the vehicle 12. The image 16 to be processed is a single image of a video stream of a camera, not shown. Figure 2 shows a three-dimensional sketch showing filter suggestions using the estimated scale filters and the image scale function, respectively. A vertical axis 28 corresponds to the size of the object in units of pixels ² . The axes 30, 32 refer to positions. A plane 26 defines an area where surface points 28 may be located. Points 28 refer to positions of reliable detections in the image or image 16 that is to be processed, which can be used to estimate the scale functions. As can be seen, a line defines the horizon 50. Figure 3 shows a projection of the plane 26 in the direction of the axis 28 from above. As can be seen, the horizon 50 is parallel to the axis 32 which intersects the axis 30 at the value 100. The points 28 are located only below the horizon 50, that is, at values greater than 100 of the axis 30. In the area of the horizon 50, there is a small scale, wherein in the area of the axis 32 is a large scale.

Figure 4 shows an effect of the invention, namely that range suggestions are much more precise. First, they are deployed only below the horizon 50. In addition, the size of the range suggestions 34 is better adapted to the scale. This results in more accurate and faster detection of vehicles 12 by the rectangles 36. The reason for this is that the rectangles 36 are dimensioned to vary depending on a scale and position in the vehicle Figure 16 are better adapted to sizes of vehicles 12. Object suggestions become smaller as they approach horizon 50. FIG. 5 shows in an exemplary embodiment a schematic flow diagram of a method 10 for detecting road users 12, here vehicles, on a traffic route 14 in a sequence of successive images provided by a camera recording the sequence of images of the traffic route 14. The sequence of images is currently provided by a video stream.

At step 40, a first map of the sequence of consecutive images is selected and the first image is defined as an image 16 to be processed. Then, at step 42, a

Range Suggestion Generator is applied to Figure 16, which is to be processed, which may be provided by selective RP (Faster R-CNN) search, sliding windows, and the like. The range proposal generator generates a plurality of range suggestions 18 for possible objects recorded in the image 16 to be processed. Then, at step 44, it is checked if filters are available. If not, the method continues with step 46 by presenting all range suggestions 18 of the range suggestion generator also shown in FIG. Then, the method continues with step 72, wherein object detection is provided for all area proposals 18 to detect road users 12, taking into account a pre-defined level of confidence. In method step 74, reliable detections (with a high result) are collected. At step 76, it is checked whether enough detections have been made to allow a filter estimate. If not (n), the method continues with step 40 and selects another mapping of the sequence of consecutive mappings to improve the number of acquisitions achieved.

If sufficient detections have been made in step 76 (y), the method continues with step 78. At step 78, filters are estimated, such as scale filters, aspect ratio filters, a horizon, and the like. The estimated filters are provided to control suggestion filtering, as discussed below. If the filters are estimated in step 78, the method continues to step 40, as described above.

If it is determined at step 44 that filters are available (y), the method continues to step 48 using suggestion filters as discussed above. The method then proceeds to step 70 by applying filtered suggestions to the image 16 to be processed, which is also shown in FIG. Then, as discussed above, the method continues to step 73.

Optionally, at step 80, camera calibration information may be received and provided to step 78 to enhance the filter estimation.

As demonstrated by this disclosure, the proposed GP-FRCNN approach has the potential to overcome the classification failures of the basic RPN and, as a result, can achieve more or less similar performance regardless of the scale of the object. The inventors' findings also suggest that one can not simply accommodate the geometric layout to reclassify proposals and then expect desired improvements, but instead a number of scale changes are preferably provided. If desired, the various functions and embodiments discussed herein may be performed in a different, different order and / or concurrently with each other in various ways. Further, one or more of the functions and / or embodiments described above may be optional, or preferably combined in an arbitrary manner, as desired.

Although various aspects of the invention are set forth in the independent claims, other aspects of the invention encompass other combinations of features from the described embodiments and / or the dependent claims, with the features of the independent claims, and only the combination set forth in the claims ,

It is also noted that although the above describes exemplary embodiments of the invention, this description should not be considered as limiting the scope of protection. Rather, there are several variations and changes that can be made without departing from the scope of the present invention as protected by the dependent claims.

REFERENCE SIGNS Procedure

vehicle

Street

Illustration

range proposal

rectangle

rectangle

rectangle

level

axis

axis

axis

range proposal

rectangle

Points

step

step

step

step

step

horizon

sketch

ordinate

abscissa

am

am

am

am

 sketch

abscissa

 step

 step

 step

step 76 step

78 step

80 step

Y a no

Claims

Claims: A method (10) for capturing road users (12) on a traffic route (14) in a map provided by a camera that captures the map, the method comprising:  Generating (42) a plurality of possible object range estimates (18) recorded in the image (16) by applying a range suggestion generator to the image (16);  Providing object detection (72) for all range suggestions (18) by applying a Object detection apparatus on all area proposals (18) to detect the traffic route (14) and / or road users (12) by staging taking into account a predetermined level of confidence; and  Outputting acquisition data received from the object detection for the detected traffic route (14) and / or the detected road users (12), marked by:  Providing a range suggestion filter (48) prior to the step of providing object detection, wherein the filtering is performed based on respective filter data based on a relevance of the range suggestions (18) associated with the Road users (12) and / or the traffic route (14). 2. The method of claim 1, wherein scale information is provided prior to the step of performing the object detection, wherein scale information is acquired from long-term observation of the vehicles detected on the ground surface. 3. The method of claim 1 or 2, wherein the filter data is estimated based on a scene defined by the Figure (16) to be processed is recorded in particular the traffic route (14) included in the scene. 4. The method according to any one of claims 1 to 3, wherein the filter data provide an observation area for possible objects, which are classified by the object detection as road users (12). 5. The method according to any one of claims 1 to 4, wherein the filter data is updated in dependence on data of the object detection. 6. The method according to any one of claims 1 to 5, wherein deep learning method and / or artificial intelligence is used. The method of any one of claims 1 to 6, wherein the output data from at least two mappings of the sequence of mappings are processed to determine at least one trajectory for a particular one of the detected road users (12). The method of any of claims 1 to 7, wherein the filter data is estimated by taking into account camera calibration data. 9. An apparatus for detecting road users (12) on a traffic route (14) in a sequence of successive images, the device comprising: at least one camera which records the sequence of images of the traffic route (14), and  a device that is configured to select (40) a first mapping of the sequence of consecutive maps and define the first map as an image (16) to be processed; generating a plurality of area suggestions (18) for possible objects recorded in the image to be processed (16) by applying a range proposal generator to the image (16) to be processed (42);  Object detection (72) for all area proposals (18) to provide to detect the traffic route (14) and / or the road users (12) by classification taking into account a predefined confidence level;  Output acquisition data received from the object detection for the detected traffic route (14) and / or the detected road users (12); and continue by selecting a different map of the sequence of consecutive images than the map (16) to be processed; characterized in that  the device is further configured to  providing filtering (48) for the range suggestions (18) prior to the step of providing object detection (72), wherein the filtering (48) is performed based on respective filter data based on a relevance of the range suggestions (18) associated with the Road users (12) and / or the traffic route (14). 10. A method of providing traffic guidance, comprising:  Detecting road users (12) on a traffic route (14) in a sequence of successive images provided by a camera that captures the sequence of images of the traffic route (14);  Determining a used capacity of the traffic route (14) provided by detecting the road users (12); Identify individual actual speed and / or position of each of the road users (12) to respective individual actual orbits of the To determine road users (12);  Determining at least one respective nominal track for each of the road users (12); and - communicating the determined nominal lanes to the respective road users (12); characterized in that  the detection of traffic participants (12) on the traffic route (14) by a method according to one of claims 1 to 8 is provided. 11. The method of claim 10, wherein at least one of the road users (12) is at least partially driven automatically. The method of claim 10 or 11, wherein at least one of the road users (12) requests a desired orbit, causing:  Determining a particular nominal track for the road user (12); and  Communicating the determined nominal trajectory to the road user (12). 13. The method according to any one of claims 10 to 12, which comprises testing a particular nominal track by simulation for the Road users (12). A method according to any one of claims 10 to 13, comprising querying an actual speed and / or position of at least one of the road users (12) to determine its actual trajectory. 15. The method according to any one of claims 10 to 14, wherein, depending on the used capacity of the traffic route (14), the speed of the road users (12) is automatically adjusted. - l - The method of any of claims 10 to 15, comprising automatically adjusting the speed of at least one of the road users (12) as the road user (12) approaches a predetermined transit pattern for the traffic route (14). 17. Traffic guidance system for providing traffic guidance, comprising:  at least one device for detecting road users (12) on a traffic route (14) in a sequence of successive images, and  a device that is configured to  To detect road users (12) on the traffic route (14) in the sequence of successive images; to determine a used capacity of the traffic route (14) provided by the road users (12); identify individual actual speed and / or position of each of the road users (12) to determine respective individual actual lanes; - to determine at least one respective nominal track for each of the road users (12); and  communicate the designated nominal lanes to the respective road users (12); characterized in that the device for detecting road users (12) is configured on a traffic route (14) according to claim 9. A computer program product comprising a program for a processing device comprising software code portions of a program for performing the steps of a method according to any one of claims 1 to 8 and / or for performing the steps of a method according to any one of claims 10 to 16 when the program is performed on the processing device.