CN119445538B - Vehicle-mounted camera-based driving environment recognition method and system - Google Patents
Vehicle-mounted camera-based driving environment recognition method and system Download PDFInfo
- Publication number
- CN119445538B CN119445538B CN202510031345.1A CN202510031345A CN119445538B CN 119445538 B CN119445538 B CN 119445538B CN 202510031345 A CN202510031345 A CN 202510031345A CN 119445538 B CN119445538 B CN 119445538B
- Authority
- CN
- China
- Prior art keywords
- target
- detection
- sub
- image data
- vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
本发明公开了一种基于车载摄像头的行车环境识别方法和系统，方法包括：获取检测图像数据并进行预处理，得到预处理图像数据；将预处理图像数据输入至预设行车环境识别模型，确定行车环境识别数据、识别目标和识别目标的风险等级；将检测区域划分为多个子检测区域，基于注意力机制调整子检测区域的第一注意力得分，根据第一注意力得分对下一检测图像数据进行分析；对相邻采集时间的检测图像数据进行对比，确定新增目标和消失目标；提取消失目标的局部图像特征，调整子检测区域的第二注意力得分，根据第二注意力得分对下一检测图像数据进行分析。本发明通过调整每个子识别区域的注意力得分，提高对行车环境内识别目标的识别效率。
The present invention discloses a driving environment recognition method and system based on a vehicle-mounted camera, the method comprising: acquiring detection image data and preprocessing to obtain preprocessed image data; inputting the preprocessed image data into a preset driving environment recognition model to determine driving environment recognition data, recognition targets and risk levels of recognition targets; dividing a detection area into multiple sub-detection areas, adjusting the first attention score of the sub-detection area based on an attention mechanism, and analyzing the next detection image data according to the first attention score; comparing the detection image data of adjacent acquisition times to determine newly added targets and disappeared targets; extracting local image features of disappeared targets, adjusting the second attention score of the sub-detection area, and analyzing the next detection image data according to the second attention score. The present invention improves the recognition efficiency of the recognition target in the driving environment by adjusting the attention score of each sub-recognition area.
Description
Technical Field
The application relates to the technical field of intelligent recognition, in particular to a vehicle-mounted camera-based driving environment recognition method and system.
Background
Along with the rapid development of automatic driving technology, accurate identification of driving environment becomes a key for ensuring driving safety. The vehicle-mounted camera is used as one of important sensors for automatically driving an automobile, and can capture rich road information. But the identification effect is easily affected by factors such as illumination conditions, weather changes, camera pollution and the like. When dealing with complex and changeable driving environments, the problems of low recognition accuracy, low adaptability and the like still exist. In addition, in the prior art, the image data acquired by the vehicle-mounted camera is generally integrally identified, the driving environment identification data is determined, the identification of movable targets such as pedestrians and vehicles is not preferentially carried out, the uncontrollability of the movable targets such as pedestrians and vehicles on driving safety is considered, the situation that the vehicle-mounted camera is left for a central control system is short, corresponding obstacle avoidance processing cannot be carried out in time, and certain automatic driving risks exist.
Therefore, the prior art has defects, and improvement is needed.
Disclosure of Invention
In view of the above problems, an object of the present invention is to provide a vehicle-mounted camera-based driving environment recognition method and system, which divide a recognition area into a plurality of sub-recognition areas, and improve recognition efficiency of recognizing objects in a driving environment by adjusting attention score of each sub-recognition area.
The first aspect of the invention provides a vehicle-mounted camera-based driving environment identification method, which comprises the following steps:
Acquiring detection image data of a detection area;
preprocessing the detection image data to obtain preprocessed image data;
Inputting the preprocessed image data into a preset driving environment recognition model, wherein the driving environment recognition model performs feature extraction and classification on the preprocessed image data, outputs driving environment recognition data and determines recognition targets, wherein the driving environment recognition data comprises but is not limited to lane line detection, traffic sign recognition, pedestrian detection and vehicle detection, and the recognition targets comprise but are not limited to pedestrian targets and vehicle targets;
calculating a first coordinate distance between the identification target and the current vehicle to determine the risk level of the identification target;
Dividing the detection area into a plurality of sub-detection areas, dividing the detection image data into a plurality of sub-detection image data based on the plurality of sub-detection areas;
Determining a vehicle moving route of the current vehicle and a target moving route of the recognition target based on the detected image data of the continuous multiframes;
Adjusting a first attention score of the sub-detection area based on an attention mechanism, and analyzing a plurality of sub-detection image data of the next detection image data according to the order of the first attention score from big to small;
determining a newly added target and a disappeared target by comparing the detected image data of adjacent acquisition time;
When the vanishing target exists, extracting local image characteristics of the vanishing target, adjusting second attention scores of the sub-detection areas at each data acquisition time according to the vehicle moving route and the target moving route of the vanishing target, and analyzing a plurality of sub-detection image data of the next detection image data according to the order of the second attention scores from large to small.
In this scheme, the preprocessing the detected image data to obtain preprocessed image data includes:
performing data enhancement on the detected image data;
And carrying out image preprocessing on the detected image data, wherein the image preprocessing at least comprises image denoising and color correction.
In this scheme, still include:
Acquiring history detection image data;
and analyzing according to the historical detection image data, and establishing a preset driving environment recognition model.
In this aspect, the determining the vehicle moving route of the current vehicle and the target moving route of the recognition target based on the detected image data of the continuous multiframe includes:
Analyzing the detection image data of the continuous multiframes to judge whether the identification target is a static target or a moving target;
when the identification target is a moving target, drawing a target moving route of the identification target based on the detection image data of the continuous multiframes;
Acquiring a vehicle moving route of a current vehicle;
Analyzing according to the vehicle moving route of the current vehicle and the target moving route of the identification target, and determining the minimum coordinate distance between the current vehicle and the identification target in a preset time interval;
and when the minimum coordinate distance is smaller than a preset distance threshold value, early warning is carried out on the identification target.
In this aspect, the adjusting the first attention score of the sub-detection area based on the attention mechanism analyzes a plurality of sub-detection image data of the next detection image data according to the order of the first attention score from big to small, including:
analyzing according to the vehicle moving route of the current vehicle and the target moving route of the identification target, predicting the coordinate position of the identification target in a detection area in the next detection image data, and determining a first sub-detection area occupied by the identification target;
Increasing a first attention score of the first sub-detection area based on an attention mechanism and decreasing the first attention scores of other sub-detection areas outside the first sub-detection area;
analyzing the sub-detection image data corresponding to each sub-detection area sequentially from the left to the right and from the top to the bottom based on the first attention score from the large to the small and the sub-detection area, and determining the coordinate position of the identification target after moving through feature comparison;
and correcting the first sub-detection area occupied by the identification target according to the coordinate position of the identification target after moving, and adjusting the first sub-detection area and other sub-detection areas which are not subjected to image analysis.
In this scheme, through comparing the detection image data of adjacent acquisition time, confirm newly increased target and disappearance target, include:
Comparing the current detection image data with the last detection image data, and carrying out pairwise matching on the identification target of the current detection image data and the identification target of the last detection image data according to the image characteristics to determine a new target and a disappearance target;
And determining the new reason of the new target and the disappearing reason of the disappearing target based on the target moving route respectively.
In this solution, when there is a vanishing target, extracting a local image feature of the vanishing target, adjusting a second attention score of a sub-detection area at each data acquisition time according to a vehicle moving route and a target moving route of the vanishing target, and analyzing a plurality of sub-detection image data of next detection image data according to a sequence from big to small of the second attention score, including:
Extracting one or more local image features of the vanishing target in the previous detected image data P n-1, and determining one or more first coordinates and corresponding feature scores corresponding to the one or more local image features;
Determining one or more second coordinates corresponding to the one or more local image features in the detected image data P n+m according to the vehicle moving route and the target moving route of the vanishing target, wherein m is a positive integer greater than 1;
Analyzing each second coordinate in turn, and marking the sub-detection areas in the preset range of the second coordinates according to the local image characteristics corresponding to the second coordinates to determine second sub-detection areas;
Determining the influence weight of the feature marks of the second sub-detection area according to the second coordinate distance from the center coordinates of the second sub-detection area to the corresponding second coordinates;
weighting calculation is carried out on the feature scores of the feature marks of the second sub-detection areas and the corresponding influence weights, and the second attention score and the average attention score of the second sub-detection areas are determined;
filtering a second sub-detection area with the average attention score lower than a preset score threshold value;
Analyzing the sub-detection image data of the detection image data P n+m in each of the second sub-detection areas sequentially from the top to the bottom in the second attention score, determining that the vanishing target reappears when the image features of the sub-detection image data match with the local image features of the vanishing target,
And selecting the vanishing target in a frame mode according to the coordinate data of the sub-detection image data and the historical identification data of the vanishing target, and determining the identification data of the vanishing target.
In this scheme, still include:
After the identification data of the vanishing target is determined, the feature marks of the local feature data of the vanishing target in each sub-detection area are cleared, the second attention score and the average attention score of each second sub-detection area are recalculated, and the identification of other vanishing targets is continued.
The second aspect of the invention provides a vehicle-mounted camera-based driving environment recognition system, which comprises:
the data acquisition module is used for acquiring detection image data of the detection area;
The preprocessing module is used for preprocessing the detection image data to obtain preprocessed image data;
the vehicle environment recognition module is used for inputting the preprocessed image data into a preset vehicle environment recognition model, wherein the vehicle environment recognition model is used for carrying out feature extraction and classification on the preprocessed image data, outputting vehicle environment recognition data and determining recognition targets, wherein the vehicle environment recognition data comprise lane line detection, traffic sign recognition, pedestrian detection and vehicle detection;
the detection image processing device comprises a region segmentation module, a detection image processing module and a detection image processing module, wherein the region segmentation module is used for dividing a detection region into a plurality of sub-detection regions and dividing detection image data into a plurality of sub-detection image data based on the plurality of sub-detection regions;
the system comprises an identification target monitoring module, a detection target detection module and a target detection module, wherein the identification target monitoring module is used for determining a vehicle moving route of a current vehicle and a target moving route of an identification target based on detection image data of a plurality of continuous frames;
And the vanishing target monitoring module is used for extracting local image characteristics of the vanishing target when the vanishing target exists, adjusting second attention scores of the sub-detection areas at each data acquisition time according to the vehicle moving route and the target moving route of the vanishing target, and analyzing a plurality of sub-detection image data of the next detection image data according to the order of the second attention scores from large to small.
The third aspect of the present invention provides a computer readable storage medium, where the computer readable storage medium includes a vehicle-mounted camera-based driving environment recognition method program, where the vehicle-mounted camera-based driving environment recognition method program, when executed by a processor, implements the steps of the vehicle-mounted camera-based driving environment recognition method described above.
The invention discloses a vehicle-mounted camera-based driving environment recognition method and system, wherein the method comprises the steps of obtaining detection image data and preprocessing to obtain preprocessed image data; the method comprises the steps of inputting preprocessed image data into a preset driving environment recognition model, determining driving environment recognition data, recognition targets and risk levels of the recognition targets, dividing a detection area into a plurality of sub-detection areas, adjusting first attention scores of the sub-detection areas based on an attention mechanism, analyzing next detection image data according to the first attention scores, comparing detection image data of adjacent acquisition time to determine a new target and a disappearing target, extracting local image characteristics of the disappearing target, adjusting second attention scores of the sub-detection areas, and analyzing the next detection image data according to the second attention scores. According to the invention, the attention score of each sub-recognition area is adjusted, so that the recognition efficiency of recognizing the target in the driving environment is improved.
Drawings
Fig. 1 shows a flow chart of a driving environment recognition method based on a vehicle-mounted camera;
FIG. 2 is a flow chart of a first attention score adjustment method for a sub-detection area provided by the present invention;
FIG. 3 shows a flow chart of the method for determining the new targets and the disappeared targets provided by the invention;
Fig. 4 shows a block diagram of a driving environment recognition system based on a vehicle-mounted camera.
Detailed Description
In order that the above-recited objects, features and advantages of the present application will be more clearly understood, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description. It should be noted that, without conflict, the embodiments of the present application and features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those described herein, and therefore the scope of the present invention is not limited to the specific embodiments disclosed below.
Fig. 1 shows a flow chart of a driving environment recognition method based on a vehicle-mounted camera.
As shown in fig. 1, the invention discloses a vehicle driving environment recognition method based on a vehicle-mounted camera, which comprises the following steps:
s102, acquiring detection image data of a detection area;
s104, preprocessing the detected image data to obtain preprocessed image data;
s106, inputting the preprocessed image data into a preset driving environment recognition model, wherein the driving environment recognition model performs feature extraction and classification on the preprocessed image data, outputs driving environment recognition data, and determines recognition targets, wherein the driving environment recognition data comprises but is not limited to lane line detection, traffic sign recognition, pedestrian detection and vehicle detection;
S108, calculating a first coordinate distance between the identification target and the current vehicle to determine the risk level of the identification target;
S110, dividing a detection area into a plurality of sub-detection areas, and dividing detection image data into a plurality of sub-detection image data based on the plurality of sub-detection areas;
S112, determining a vehicle moving route of the current vehicle and a target moving route of the identification target based on the detection image data of the continuous multi-frame;
S114, adjusting a first attention score of the sub-detection area based on an attention mechanism, and analyzing a plurality of sub-detection image data of the next detection image data according to the order of the first attention score from large to small;
s116, determining a newly added target and a disappeared target by comparing the detected image data of adjacent acquisition time;
And S118, when the vanishing target exists, extracting local image characteristics of the vanishing target, adjusting second attention scores of the sub-detection areas at each data acquisition time according to the vehicle moving route and the target moving route of the vanishing target, and analyzing a plurality of sub-detection image data of the next detection image data according to the order of the second attention scores from large to small.
According to the embodiment of the invention, the detection image data corresponding to the vehicle-mounted camera is acquired by extracting frames from the image data acquired by the vehicle-mounted camera according to the system set time interval, and the acquired detection image data is subjected to corresponding data enhancement and image preprocessing steps to obtain preprocessed image data, so that the definition and detail of the detection image data are improved, and the accuracy of identifying the driving environment is improved. The preprocessing image data is input into a preset driving environment recognition model, targets such as lane lines, traffic signs, pedestrians and vehicles are recognized through image feature extraction, comparison and classification, driving environment recognition data are output, and targets with possible movement such as pedestrians and vehicles are determined to be recognition targets. The system sets a corresponding risk identification strategy for each identification target based on the type of the identification target, adjusts the risk identification strategy of each identification target according to the real-time speed of the current vehicle, determines the risk level of each identification target through the risk identification strategy of each identification target, and sets and dynamically adjusts the number of the risk levels and the first coordinate distance interval corresponding to each risk level by the system. If the current vehicle runs at a speed below 30km/h, the pre-warning distance of the pedestrian target is 0.5-2 m, when the first coordinate distance between the pedestrian target and the current vehicle is less than 2m, the pedestrian target enters a pre-warning state, the risk level is determined to be the lowest risk level, and when the first coordinate distance between the pedestrian target and the current vehicle is less than 0.5m, the risk level of the pedestrian target is determined to be the highest risk level. When the speed of the current vehicle is adjusted to 60km/h, the pre-warning distance of the pedestrian target is adjusted to 2-8 meters.
When the driving environment is identified, the identification target is preferentially identified due to uncertainty of the identification target. Dividing the detection area into a plurality of sub-detection areas according to a preset specification of the system (the length and the width of each sub-detection area are the same), drawing a vehicle moving route of the current vehicle according to the running data of the current vehicle, drawing a target moving route of the recognition target according to the coordinate data of the recognition target in continuous multi-frame detection image data, predicting the corresponding position of the recognition target in the detection image data acquired at the next acquisition time by combining a set time interval of the system, determining a first sub-detection area corresponding to the recognition target, improving the first attention score of the first sub-detection area, reducing the first attention score of other sub-detection areas, and analyzing the sub-detection image data possibly having the recognition target preferentially. Wherein the initial first attention score for each sub-detection zone is the same. When the first attention score of the sub-detection areas is the same, sub-detection image data corresponding to each sub-detection area is analyzed sequentially from left to right and from top to bottom.
After the vanishing target is determined, the coordinate position of the local image feature of the vanishing target in the next detection image data to be analyzed is predicted by extracting the local image feature of the vanishing target and combining the vehicle moving route of the current vehicle and the target moving route of the vanishing target, and a plurality of second sub-detection areas where the local image feature possibly appears are determined based on a preset range. And the priority of the second sub-detection area is continuously improved on the basis of the first sub-detection area, the sub-detection area where the message target possibly appears is preferentially identified, and the situation that the identification target suddenly appears and early warning cannot be timely carried out is avoided.
According to an embodiment of the present invention, preprocessing detected image data to obtain preprocessed image data includes:
Performing data enhancement on the detected image data;
And performing image preprocessing on the detected image data, wherein the image preprocessing at least comprises image denoising and color correction.
The vehicle-mounted camera is arranged outside the vehicle, so that the definition of a shot image is reduced when the vehicle-mounted camera is affected by the environment (such as when the vehicle-mounted camera is covered by pollutants such as mud, and the like), and the real road condition information is difficult to reflect. Image enhancement is carried out on the detected image data through technologies such as self-adaptive histogram equalization, defogging algorithm and the like, meanwhile, a proper image denoising method (such as Gaussian filtering, median filtering and the like) is selected to carry out noise removal on the detected image data, and color correction is carried out through adjusting parameters such as brightness, contrast, saturation and the like of an image so as to improve color balance and improve image data quality.
According to an embodiment of the present invention, further comprising:
Acquiring history detection image data;
and analyzing according to the historical detection image data, and establishing a preset driving environment recognition model.
The method comprises the steps of marking lane lines, traffic signboards, pedestrians, vehicles and other targets in the historical detection image data in a manual marking mode and the like, inputting the marked historical detection image data into a model for training, extracting image features corresponding to different targets, and establishing a preset driving environment recognition model.
According to an embodiment of the present invention, determining a vehicle moving route of a current vehicle and a target moving route of an identification target based on detected image data of consecutive frames includes:
judging whether the identification target is a static target or a moving target by analyzing the detection image data of the continuous multiframes;
When the identification target is a moving target, drawing a target moving route of the identification target based on the detection image data of the continuous multiframes;
Acquiring a vehicle moving route of a current vehicle;
Analyzing according to a vehicle moving route of the current vehicle and a target moving route of the identification target, and determining the minimum coordinate distance between the current vehicle and the identification target in a preset time interval;
And when the minimum coordinate distance is smaller than a preset distance threshold value, early warning is carried out on the identification target.
Since the image data in the detection area changes with the running of the vehicle, there is a difference in the coordinate data of even the stationary object in each detection image data. By analyzing the detected image data of the continuous multiframes, a known stationary target (such as a traffic sign) is used as a reference object, and whether the recognition target moves or not is judged by comparing the relative distance between the recognition target and the reference object, so that the recognition target is judged to be the stationary target or the moving target. The method mainly aims at analyzing targets in moving and stationary states of pedestrians, vehicles and the like. After determining the state of the recognition target, analyzing the moving target, drawing a target moving route of the moving target according to the relative coordinate data of the moving target and the reference object in each detection image data, and determining the target moving route of the static target as a fixed coordinate point. The coordinate distance is calculated by analyzing the vehicle moving route of the current vehicle and the target moving route of the recognition target by means of vehicle navigation and the like or the vehicle moving route of the current vehicle, determining the coordinate data of the current vehicle and the recognition target at each acquisition time. And selecting the minimum coordinate distance between the current vehicle and the recognition target in the preset time interval, comparing the minimum coordinate distance with a preset distance threshold value, and judging whether the early warning level of the recognition target in the preset time interval is changed to the highest early warning level, so that early warning is carried out in advance, and a driver is reminded to carry out obstacle avoidance treatment on the recognition target in advance.
Wherein the preset time interval and the preset distance threshold are set by a person skilled in the art according to actual requirements.
Fig. 2 is a flowchart of a first attention score adjustment method of a sub-detection area provided by the present invention.
As shown in fig. 2, according to an embodiment of the present invention, a first attention score of a sub-detection area is adjusted based on an attention mechanism, and a plurality of sub-detection image data of next detection image data is analyzed in order of the first attention score from large to small, including:
S202, analyzing according to a vehicle moving route of a current vehicle and a target moving route of an identification target, predicting a coordinate position of the identification target in a detection area in next detection image data, and determining a first sub-detection area occupied by the identification target;
S204, the first attention score of the first sub-detection area is improved based on the attention mechanism, and the first attention score of other sub-detection areas except the first sub-detection area is reduced;
S206, analyzing the sub-detection image data corresponding to each sub-detection area in turn from left to right and from top to bottom based on the first attention score from large to small and the sub-detection area, and determining the coordinate position of the identification target after moving through feature comparison;
S208, correcting the first sub-detection area occupied by the recognition target according to the coordinate position of the recognition target after moving, and adjusting the first sub-detection area and other sub-detection areas which are not subjected to image analysis.
It should be noted that, the coordinate position in the next detection image data acquired by the recognition target at the next acquisition time may be predicted by the vehicle moving route of the current vehicle and the target moving route of the recognition target, and the sub-detection area occupied by the recognition target (i.e., the sub-detection area including the recognition target) is determined as the first sub-detection area. The first sub-detection area is a sub-detection area with high probability of existence of the recognition target, which is affected by actual movement and predicted movement errors. The first attention scores of the first sub-detection area and other sub-detection areas are adjusted through an attention mechanism, in the process of analyzing the detection image data, the first sub-detection area with higher first attention score is preferentially analyzed, and whether the whole or part of image features of the identification target exist in the first sub-detection area is judged through comparing the image features of the sub-detection image data of the first sub-detection area with the image features of the identification target, so that the coordinate position of the identification target after movement is determined.
After determining the coordinate position of the moving recognition target, correcting the first sub-detection area according to the sub-detection area actually occupied by the recognition target, for example, reducing the first attention score of the sub-detection area which is actually occupied by the recognition target and is not analyzed, and adjusting the first sub-detection area which is not matched with the actually occupied sub-detection area to be other sub-detection areas. In addition, the first attention score of the edge sub-detection area can be improved, so that whether the newly added recognition target exists or not can be preferentially recognized.
Fig. 3 shows a flowchart of the method for determining the new target and the disappeared target provided by the present invention.
As shown in fig. 3, according to an embodiment of the present invention, determining a newly added target and a disappeared target by comparing detected image data of adjacent acquisition times includes:
S302, comparing the current detection image data with the last detection image data, and carrying out pairwise matching on the identification target of the current detection image data and the identification target of the last detection image data according to image characteristics to determine a new target and a disappearance target;
s304, determining a new reason for the new object and a disappearing reason for the disappearing object based on the object moving route.
The identification target that is present in the current detection image data and that is not present in the previous detection image data is determined as a new target, and the identification target that is not present in the current detection image data and that is present in the previous detection image data is determined as a disappearing target. The new reasons of the new targets comprise new and shielding reproduction, and the disappearing reasons of the disappearing targets comprise detachment from the detection area and shielding. And preferentially analyzing the newly-added targets, determining the newly-added reasons of the newly-added targets which meet the condition that the history coordinate data or the coordinate data are not in the edge sub-detection area as newly-added targets, and determining the newly-added reasons of the newly-added targets which meet any one of the newly-added targets which have the history coordinate data or the coordinate data and are not in the edge sub-detection area as occlusion reproduction. Then, the historical target moving route of the target is analyzed, the disappearing reason of the disappearing target moving towards the direction of the departure detection area is determined as the departure detection area, and the disappearing reason of the disappearing target moving towards other directions is determined as the target is blocked.
According to an embodiment of the present invention, when there is a vanishing target, local image features of the vanishing target are extracted, second attention scores of sub-detection areas at each data acquisition time are adjusted according to a vehicle moving route and a target moving route of the vanishing target, and a plurality of sub-detection image data of next detection image data are analyzed in order of the second attention scores from large to small, including:
Extracting one or more local image features of the vanishing target in the previous detected image data P n-1, and determining one or more first coordinates and corresponding feature scores corresponding to the one or more local image features;
determining one or more second coordinates corresponding to one or more local image features in the detected image data P n+m according to the vehicle moving route and the target moving route of the vanishing target, wherein m is a positive integer greater than 1;
analyzing each second coordinate in turn, and marking the sub-detection areas in the preset range of the second coordinates according to the local image characteristics corresponding to the second coordinates to determine second sub-detection areas;
Determining the influence weight of the feature marks of the second sub-detection area according to the second coordinate distance from the center coordinates of the second sub-detection area to the corresponding second coordinates;
Weighting calculation is carried out on the feature scores of the feature marks of the second sub-detection areas and the corresponding influence weights, and the second attention score and the average attention score of the second sub-detection areas are determined;
filtering a second sub-detection area with the average attention score lower than a preset score threshold value;
Analyzing the sub-detection image data of the detection image data P n+m in each of the second sub-detection areas sequentially from the top to the bottom in the second attention score, determining that the vanishing target reappears when the image features of the sub-detection image data match with the local image features of the vanishing target,
And carrying out frame selection on the vanishing target according to the coordinate data of the sub-detection image data and the historical identifying data of the vanishing target, and determining the identifying data of the vanishing target.
It should be noted that the local image features refer to local structure or texture information of the vanishing target having uniqueness, stability and distinguishing property in the detected image data, and reflect the local characteristics of the vanishing target on the image, such as line intersection, brightness change and other structural information. The method comprises the steps of identifying a target to be detected through local image features, determining a first coordinate of each local image feature in current detection image data, and giving a corresponding feature score to each local image feature according to the correlation degree of each local image feature and a vanishing image, wherein the value range of the feature score is 0-1, and the greater the feature score is, the higher the accuracy of specially identifying the vanishing image through the corresponding local image is.
When the next detected image data P n+1 is analyzed, the first coordinates are translated according to the vehicle moving route and the target moving route of the disappearing target, the second coordinates corresponding to the local image features in the next detected image data P n+1 are determined, feature marks are carried out on the sub-detection areas in the second coordinates preset range according to the system preset range, feature scores of the local image features are recorded in the feature marks, the sub-detection areas with the feature marks are determined as second sub-detection areas, the second coordinate distances from each second sub-detection area to the corresponding second coordinates are calculated respectively, the influence weight of the feature marks in each second sub-detection area is determined, the value range of the influence weight is 0-1, and the smaller the second coordinate distance from the second sub-detection area to the corresponding second coordinates is, the higher the corresponding influence weight is. After all the local feature analysis is completed, each second sub-detection area is analyzed in sequence, the feature score of each feature mark in the second sub-detection area is multiplied by the corresponding influence weight respectively, the calculation results are accumulated, the second attention score of the second sub-detection area is determined, the ratio of the second attention score to the number of feature marks is calculated, and the average attention score detected by the second sub-detection area is determined. Similarly, when analyzing the detected image data P n+m, the second coordinates of the local image features of the vanishing target in the detected image data P n+m are calculated according to the above steps, the second sub-detection areas are determined, the second attention score and the average attention score of each second sub-detection area are calculated, the second sub-detection areas with the average attention score lower than the preset score threshold are filtered, and the sub-detected image data of the detected image data P n+m in each second sub-detection area are sequentially analyzed according to the second attention score from the top to the bottom. By analyzing the second sub-detection area with the possible disappeared target preferentially, the priority of disappeared target identification is improved, and the situation that the identification target suddenly appears and early warning cannot be performed in time is avoided.
In addition, the maximum recognition times of the vanishing target can be set through the system, when the vanishing target does not appear in the detection image data of the maximum recognition times, the vanishing target is judged to have left the detection area, and the sub-detection area where the vanishing target possibly appears is not preferentially recognized.
Wherein the preset score threshold is set by a person skilled in the art according to actual requirements.
According to the embodiment of the invention, after the identification data of the vanishing target is determined, the feature marks of the local feature data of the vanishing target in each sub-detection area are cleared, the second attention score and the average attention score of each second sub-detection area are recalculated, and the identification of other vanishing targets is continued.
When the identification data of the vanishing target, that is, the coordinate data of the vanishing target reappearance is determined, the local feature data of the vanishing target is not required to be identified, the feature marks corresponding to the local feature data of the vanishing target in the second detection areas are canceled, the second attention score and the average attention score of each second sub-detection area are recalculated according to the feature marks existing in each second detection area, the second sub-detection areas with the average attention score lower than the preset score threshold are filtered, the sub-detection image data corresponding to the remaining second sub-detection areas which are not analyzed are sequentially analyzed according to the second attention score from the big to the small, and other vanishing targets are identified.
After the second detection areas are all identified, the remaining sub-detection areas which are not analyzed are analyzed according to the first attention score.
Fig. 4 shows a block diagram of a driving environment recognition system based on a vehicle-mounted camera.
As shown in fig. 4, a second aspect of the present invention provides a vehicle-mounted camera-based driving environment recognition system, including:
the data acquisition module is used for acquiring detection image data of the detection area;
The preprocessing module is used for preprocessing the detection image data to obtain preprocessed image data;
The vehicle environment recognition module is used for inputting the preprocessed image data into a preset vehicle environment recognition model, extracting and classifying the characteristics of the preprocessed image data by the vehicle environment recognition model, outputting vehicle environment recognition data, and determining recognition targets, wherein the vehicle environment recognition data comprise, but are not limited to, lane line detection, traffic sign recognition, pedestrian detection and vehicle detection;
the region segmentation module is used for dividing the detection region into a plurality of sub-detection regions and dividing the detection image data into a plurality of sub-detection image data based on the plurality of sub-detection regions;
the system comprises an identification target monitoring module, a detection target detection module and a target detection module, wherein the identification target monitoring module is used for determining a vehicle moving route of a current vehicle and a target moving route of an identification target based on detection image data of a plurality of continuous frames;
And the vanishing target monitoring module is used for extracting local image characteristics of the vanishing target when the vanishing target exists, adjusting second attention scores of the sub-detection areas at each data acquisition time according to the vehicle moving route and the target moving route of the vanishing target, and analyzing a plurality of sub-detection image data of the next detection image data according to the order of the second attention scores from large to small.
The third aspect of the present invention provides a computer readable storage medium, where the computer readable storage medium includes a vehicle-mounted camera-based driving environment recognition method program, and when the vehicle-mounted camera-based driving environment recognition method program is executed by a processor, the steps of the vehicle-mounted camera-based driving environment recognition method are implemented.
Information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, presented data, etc.), and signals (including but not limited to signals transmitted between a user terminal and other devices, etc.) referred to by the present application are all user-authorized or fully authorized by parties, and the collection, use, and processing of relevant data is required to comply with relevant laws and regulations and standards of relevant countries and regions. For example, reference in the present disclosure to "detected image data", "history detected image data", and the like are all acquired with sufficient authorization.
The invention discloses a vehicle-mounted camera-based driving environment recognition method and system, wherein the method comprises the steps of obtaining detection image data and preprocessing to obtain preprocessed image data; the method comprises the steps of inputting preprocessed image data into a preset driving environment recognition model, determining driving environment recognition data, recognition targets and risk levels of the recognition targets, dividing a detection area into a plurality of sub-detection areas, adjusting first attention scores of the sub-detection areas based on an attention mechanism, analyzing next detection image data according to the first attention scores, comparing detection image data of adjacent acquisition time to determine a new target and a disappearing target, extracting local image characteristics of the disappearing target, adjusting second attention scores of the sub-detection areas, and analyzing the next detection image data according to the second attention scores. According to the invention, the attention score of each sub-recognition area is adjusted, so that the recognition efficiency of recognizing the target in the driving environment is improved.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is merely a logical function division, and there may be additional divisions of actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The units described as separate components may or may not be physically separate, and components displayed as units may or may not be physical units, may be located in one place or distributed on a plurality of network units, and may select some or all of the units according to actual needs to achieve the purpose of the embodiment.
In addition, each functional unit in each embodiment of the present invention may be integrated in one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of hardware plus a form of software functional unit.
It will be appreciated by those of ordinary skill in the art that implementing all or part of the steps of the above method embodiments may be implemented by hardware associated with program instructions, where the above program may be stored in a computer readable storage medium, where the program when executed performs the steps comprising the above method embodiments, where the above storage medium includes a mobile storage device, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic or optical disk, or other various media that may store program code.
Or the above-described integrated units of the invention may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solutions of the embodiments of the present invention may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present invention. The storage medium includes various media capable of storing program codes such as a removable storage device, a ROM, a RAM, a magnetic disk or an optical disk.
Claims (9)
1. The vehicle driving environment recognition method based on the vehicle-mounted camera is characterized by comprising the following steps of:
Acquiring detection image data of a detection area;
preprocessing the detection image data to obtain preprocessed image data;
The method comprises the steps of inputting preprocessing image data into a preset driving environment recognition model, wherein the driving environment recognition model carries out feature extraction and classification on the preprocessing image data, outputting driving environment recognition data and determining a recognition target, wherein the driving environment recognition data comprises lane line detection, traffic sign recognition, pedestrian detection and vehicle detection, and the recognition target comprises a pedestrian target and a vehicle target;
calculating a first coordinate distance between the identification target and the current vehicle to determine the risk level of the identification target;
Dividing the detection area into a plurality of sub-detection areas, dividing the detection image data into a plurality of sub-detection image data based on the plurality of sub-detection areas;
Determining a vehicle moving route of the current vehicle and a target moving route of the recognition target based on the detected image data of the continuous multiframes;
Adjusting a first attention score of the sub-detection area based on an attention mechanism, and analyzing a plurality of sub-detection image data of the next detection image data according to the order of the first attention score from big to small;
determining a newly added target and a disappeared target by comparing the detected image data of adjacent acquisition time;
When a vanishing target exists, extracting local image characteristics of the vanishing target, adjusting second attention scores of sub-detection areas at each data acquisition time according to a vehicle moving route and a target moving route of the vanishing target, and analyzing a plurality of sub-detection image data of next detection image data according to the sequence of the second attention scores from large to small;
the adjusting the first attention score of the sub-detection area based on the attention mechanism, analyzing the plurality of sub-detection image data of the next detection image data according to the order of the first attention score from big to small, includes:
analyzing according to the vehicle moving route of the current vehicle and the target moving route of the identification target, predicting the coordinate position of the identification target in a detection area in the next detection image data, and determining a first sub-detection area occupied by the identification target;
Increasing a first attention score of the first sub-detection area based on an attention mechanism and decreasing the first attention scores of other sub-detection areas outside the first sub-detection area;
analyzing the sub-detection image data corresponding to each sub-detection area sequentially from the left to the right and from the top to the bottom based on the first attention score from the large to the small and the sub-detection area, and determining the coordinate position of the identification target after moving through feature comparison;
and correcting the first sub-detection area occupied by the identification target according to the coordinate position of the identification target after moving, and adjusting the first sub-detection area and other sub-detection areas which are not subjected to image analysis.
2. The vehicle-mounted camera-based driving environment recognition method according to claim 1, wherein the preprocessing the detection image data to obtain preprocessed image data comprises:
performing data enhancement on the detected image data;
And carrying out image preprocessing on the detected image data, wherein the image preprocessing at least comprises image denoising and color correction.
3. The vehicle-mounted camera-based driving environment recognition method according to claim 1, further comprising:
Acquiring history detection image data;
and analyzing according to the historical detection image data, and establishing a preset driving environment recognition model.
4. The vehicle-mounted camera-based driving environment recognition method according to claim 1, wherein the determining the vehicle moving route of the current vehicle and the target moving route of the recognition target based on the detection image data of the continuous multiframe includes:
Analyzing the detection image data of the continuous multiframes to judge whether the identification target is a static target or a moving target;
when the identification target is a moving target, drawing a target moving route of the identification target based on the detection image data of the continuous multiframes;
Acquiring a vehicle moving route of a current vehicle;
Analyzing according to the vehicle moving route of the current vehicle and the target moving route of the identification target, and determining the minimum coordinate distance between the current vehicle and the identification target in a preset time interval;
and when the minimum coordinate distance is smaller than a preset distance threshold value, early warning is carried out on the identification target.
5. The vehicle-mounted camera-based driving environment recognition method according to claim 1, wherein the determining the newly added target and the disappeared target by comparing the detected image data of the adjacent acquisition time comprises:
Comparing the current detection image data with the last detection image data, and carrying out pairwise matching on the identification target of the current detection image data and the identification target of the last detection image data according to the image characteristics to determine a new target and a disappearance target;
And determining the new reason of the new target and the disappearing reason of the disappearing target based on the target moving route respectively.
6. The vehicle-mounted camera-based driving environment recognition method according to claim 1, wherein when a vanishing target exists, extracting local image features of the vanishing target, adjusting a second attention score of a sub-detection area at each data acquisition time according to a vehicle moving route and a target moving route of the vanishing target, and analyzing a plurality of sub-detection image data of the next detection image data in order of the second attention score from large to small, comprising:
Extracting one or more local image features of the vanishing target in the previous detected image data P n-1, and determining one or more first coordinates and corresponding feature scores corresponding to the one or more local image features;
Determining one or more second coordinates corresponding to the one or more local image features in the detected image data P n+m according to the vehicle moving route and the target moving route of the vanishing target, wherein m is a positive integer greater than 1;
Analyzing each second coordinate in turn, and marking the sub-detection areas in the preset range of the second coordinates according to the local image characteristics corresponding to the second coordinates to determine second sub-detection areas;
Determining the influence weight of the feature marks of the second sub-detection area according to the second coordinate distance from the center coordinates of the second sub-detection area to the corresponding second coordinates;
weighting calculation is carried out on the feature scores of the feature marks of the second sub-detection areas and the corresponding influence weights, and the second attention score and the average attention score of the second sub-detection areas are determined;
filtering a second sub-detection area with the average attention score lower than a preset score threshold value;
Analyzing the sub-detection image data of the detection image data P n+m in each of the second sub-detection areas sequentially from the top to the bottom in the second attention score, determining that the vanishing target reappears when the image features of the sub-detection image data match with the local image features of the vanishing target,
And selecting the vanishing target in a frame mode according to the coordinate data of the sub-detection image data and the historical identification data of the vanishing target, and determining the identification data of the vanishing target.
7. The vehicle-mounted camera-based driving environment recognition method according to claim 6, further comprising:
After the identification data of the vanishing target is determined, the feature marks of the local feature data of the vanishing target in each sub-detection area are cleared, the second attention score and the average attention score of each second sub-detection area are recalculated, and the identification of other vanishing targets is continued.
8. A vehicle-mounted camera-based driving environment recognition system for implementing the vehicle-mounted camera-based driving environment recognition method according to any one of claims 1 to 7, comprising:
the data acquisition module is used for acquiring detection image data of the detection area;
The preprocessing module is used for preprocessing the detection image data to obtain preprocessed image data;
The vehicle environment recognition module is used for inputting the preprocessed image data into a preset vehicle environment recognition model, wherein the vehicle environment recognition model is used for carrying out feature extraction and classification on the preprocessed image data, outputting vehicle environment recognition data and determining recognition targets, wherein the vehicle environment recognition data comprise lane line detection, traffic sign recognition, pedestrian detection and vehicle detection;
the detection image processing device comprises a region segmentation module, a detection image processing module and a detection image processing module, wherein the region segmentation module is used for dividing a detection region into a plurality of sub-detection regions and dividing detection image data into a plurality of sub-detection image data based on the plurality of sub-detection regions;
the system comprises an identification target monitoring module, a detection target detection module and a target detection module, wherein the identification target monitoring module is used for determining a vehicle moving route of a current vehicle and a target moving route of an identification target based on detection image data of a plurality of continuous frames;
The vanishing target monitoring module is used for extracting local image characteristics of the vanishing target when the vanishing target exists, adjusting second attention scores of the sub-detection areas at each data acquisition time according to the vehicle moving route and the target moving route of the vanishing target, and analyzing a plurality of sub-detection image data of the next detection image data according to the sequence of the second attention scores from large to small;
the adjusting the first attention score of the sub-detection area based on the attention mechanism, analyzing the plurality of sub-detection image data of the next detection image data according to the order of the first attention score from big to small, includes:
analyzing according to the vehicle moving route of the current vehicle and the target moving route of the identification target, predicting the coordinate position of the identification target in a detection area in the next detection image data, and determining a first sub-detection area occupied by the identification target;
Increasing a first attention score of the first sub-detection area based on an attention mechanism and decreasing the first attention scores of other sub-detection areas outside the first sub-detection area;
analyzing the sub-detection image data corresponding to each sub-detection area sequentially from the left to the right and from the top to the bottom based on the first attention score from the large to the small and the sub-detection area, and determining the coordinate position of the identification target after moving through feature comparison;
and correcting the first sub-detection area occupied by the identification target according to the coordinate position of the identification target after moving, and adjusting the first sub-detection area and other sub-detection areas which are not subjected to image analysis.
9. The vehicle-mounted camera-based driving environment recognition system according to claim 8, wherein the adjusting the first attention score of the sub-detection area based on the attention mechanism analyzes a plurality of sub-detection image data of the next detection image data in order of the first attention score from the top to the bottom, comprises:
analyzing according to the vehicle moving route of the current vehicle and the target moving route of the identification target, predicting the coordinate position of the identification target in a detection area in the next detection image data, and determining a first sub-detection area occupied by the identification target;
Increasing a first attention score of the first sub-detection area based on an attention mechanism and decreasing the first attention scores of other sub-detection areas outside the first sub-detection area;
analyzing the sub-detection image data corresponding to each sub-detection area sequentially from the left to the right and from the top to the bottom based on the first attention score from the large to the small and the sub-detection area, and determining the coordinate position of the identification target after moving through feature comparison;
and correcting the first sub-detection area occupied by the identification target according to the coordinate position of the identification target after moving, and adjusting the first sub-detection area and other sub-detection areas which are not subjected to image analysis.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510031345.1A CN119445538B (en) | 2025-01-09 | 2025-01-09 | Vehicle-mounted camera-based driving environment recognition method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510031345.1A CN119445538B (en) | 2025-01-09 | 2025-01-09 | Vehicle-mounted camera-based driving environment recognition method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN119445538A CN119445538A (en) | 2025-02-14 |
| CN119445538B true CN119445538B (en) | 2025-04-01 |
Family
ID=94510033
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202510031345.1A Active CN119445538B (en) | 2025-01-09 | 2025-01-09 | Vehicle-mounted camera-based driving environment recognition method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119445538B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120327531B (en) * | 2025-06-20 | 2025-08-22 | 南京博晟宇网络科技有限公司 | Machine learning-based user high-risk behavior recognition method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115471526A (en) * | 2022-09-14 | 2022-12-13 | 西安电子科技大学杭州研究院 | Automatic driving target detection and tracking method based on multi-source heterogeneous information fusion |
| CN116110028A (en) * | 2023-02-01 | 2023-05-12 | 苏州轻棹科技有限公司 | Processing method for matching new detection target by combining disappeared target |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102099265B1 (en) * | 2017-11-27 | 2020-05-15 | 고려대학교 산학협력단 | System and method for pedestrian-vehicle collision warning based on pedestrian state |
| US20230410459A1 (en) * | 2020-11-06 | 2023-12-21 | Hitachi Astemo, Ltd. | Outside environment recognition device and outside environment recognition method |
| AU2023443525A1 (en) * | 2022-06-22 | 2025-01-09 | Image Insight Inc. | Devices, systems, and methods for automatic object detection, identification, and tracking |
-
2025
- 2025-01-09 CN CN202510031345.1A patent/CN119445538B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115471526A (en) * | 2022-09-14 | 2022-12-13 | 西安电子科技大学杭州研究院 | Automatic driving target detection and tracking method based on multi-source heterogeneous information fusion |
| CN116110028A (en) * | 2023-02-01 | 2023-05-12 | 苏州轻棹科技有限公司 | Processing method for matching new detection target by combining disappeared target |
Also Published As
| Publication number | Publication date |
|---|---|
| CN119445538A (en) | 2025-02-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111815959B (en) | Vehicle violation detection method and device and computer readable storage medium | |
| CN106203398B (en) | A kind of method, apparatus and equipment detecting lane boundary | |
| CN106652468B (en) | The detection and from vehicle violation early warning alarm set and method in violation of rules and regulations of road vehicle front truck | |
| CN106022243B (en) | An image processing-based method for identifying wrong-way vehicles in motor vehicle lanes | |
| CN106647776B (en) | Method and device for judging lane changing trend of vehicle and computer storage medium | |
| CN103324930B (en) | A license plate character segmentation method based on gray histogram binarization | |
| CN107590470B (en) | Lane line detection method and device | |
| CN119445538B (en) | Vehicle-mounted camera-based driving environment recognition method and system | |
| CN110178167A (en) | Crossing video frequency identifying method violating the regulations based on video camera collaboration relay | |
| CN111047874B (en) | Intelligent traffic violation management method and related product | |
| CN107909012B (en) | Real-time vehicle tracking detection method and device based on disparity map | |
| CN109948552B (en) | A method of lane line detection in complex traffic environment | |
| CN103927548B (en) | Novel vehicle collision avoiding brake behavior detection method | |
| CN111753749A (en) | A Lane Line Detection Method Based on Feature Matching | |
| CN113033352B (en) | Real-time mobile traffic violation detection method based on combination of improved target semantic segmentation and target detection model | |
| CN119132101B (en) | Intelligent parking space state recognition system and method based on video detection | |
| CN108520528B (en) | Mobile vehicle tracking method based on improved difference threshold and displacement matching model | |
| CN101369312B (en) | Method and equipment for detecting intersection in image | |
| CN104915642A (en) | Method and apparatus for measurement of distance to vehicle ahead | |
| Bulla | Traffic sign detection and recognition based on convolutional neural network | |
| Coronado et al. | Detection and classification of road signs for automatic inventory systems using computer vision | |
| CN117437608B (en) | All-terrain pavement type identification method and system | |
| Phu et al. | Traffic sign recognition system using feature points | |
| CN111611942B (en) | Method for extracting and building database by perspective self-adaptive lane skeleton | |
| CN119469192B (en) | A method and system for intelligent driving path planning based on reinforcement learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |