CN115409873A - Obstacle detection method, system and electronic equipment based on optical flow feature fusion - Google Patents

Abstract

The invention provides an obstacle detection method, system and electronic equipment based on optical flow feature fusion. The method includes: collecting continuous images in the parking process, and preprocessing the collected images; dividing the preprocessed images into a plurality of grids Grid, using the optical flow tracking algorithm to track the position changes of each grid in adjacent frames, get the grid containing the moving object, and fuse the grid containing the moving object to get the range of the moving object; according to the preset aspect ratio Threshold, divide the range of the moving object into regions, perform texture calculation on each region separately, and obtain the region where the moving object is located. The invention solves the problem that the detection of moving objects in the process of automatic parking relies on ultrasonic radar and AI detection algorithms. Only the pictures recorded by the camera of the vehicle can be used to detect whether there are moving objects in the pictures after calculation. The cost is low and the hardware requirements are low. Low, small amount of calculation, fast speed, high detection accuracy.

Description

Obstacle detection method, system and electronic equipment based on optical flow feature fusion

technical field

The present invention relates to the technical field of intelligent driving, and more specifically, to an obstacle detection method, system and electronic equipment based on fusion of optical flow features.

Background technique

With the development of intelligent driving, automatic parking has become a configuration selected by more and more models. The purpose of automatic parking is to improve the convenience of parking, provide drivers with safe, comfortable and fast parking services, and reduce the difficulty of parking for drivers. The moving object detection of the pure vision algorithm in the automatic parking process mainly relies on the change of the optical flow feature in the image to find the approximate range of the moving object, and then finds the position of the moving object through fine feature distinction.

At present, the detection of moving objects in the process of automatic parking mainly relies on ultrasonic radar and AI detection algorithms. However, the equipment and installation costs of ultrasonic radar are relatively high, while the AI detection algorithm finds suspicious targets first and then performs position screening and matching. The calculation is huge, and the hardware requirements are high, and it is also difficult to reduce the cost.

Contents of the invention

Aiming at the technical problems existing in the prior art, the present invention provides an obstacle detection method, system and electronic equipment based on optical flow feature fusion, which solves the problem that the detection of moving objects in the automatic parking process relies on ultrasonic radar and AI detection algorithms with high costs The problem.

According to the first aspect of the present invention, an obstacle detection method based on optical flow feature fusion is provided, including:

Collect continuous images during the parking process, and preprocess the collected images;

The preprocessed image is divided into multiple grids, and the optical flow tracking algorithm is used to track the position change of each grid in adjacent frames to obtain the grid containing the moving object, and the grid containing the moving object is fused to obtain range of moving objects;

According to the preset aspect ratio threshold, the range of the moving object is divided into regions, and the texture calculation is performed on each region to obtain the region where the moving object is located.

On the basis of the above technical solution, the present invention can also make the following improvements.

Optionally, the preprocessing of the collected images includes:

According to the current vehicle speed, thinning processing is performed on the collected continuous frame images during the parking process to obtain a collection of continuous images with intervals of n frames, where n is a natural number greater than 1.

Optionally, the preprocessed image is divided into multiple grids, and the optical flow tracking algorithm is used to track the position change of each grid in adjacent frames to obtain a grid containing moving objects, including:

Divide each frame of image obtained by preprocessing into multiple grids, and obtain the coordinates of the center point of each grid;

The center point is tracked from the previous frame image to the next frame image using the optical flow tracking algorithm to obtain the corresponding tracking point; the optical flow tracking algorithm is used to track the tracking point from the next frame picture to the previous frame picture, Get the coordinates of the corresponding anti-tracking point;

Judging whether the center point and its corresponding anti-tracking point coincide according to the coordinate value: if the judgment result is coincidence, it is determined that the grid where the current center point is located has moved; grid still.

Optionally, in the process of dividing the image into multiple grids, the edge area of the image is first excluded according to the number of pixels, and then the remaining area is divided into multiple grids.

Optionally, the merging of the grids containing the moving objects is performed to obtain the range of the moving objects, including:

After dilation and erosion operations are performed on the mesh containing the moving object, the mesh containing the moving object is screened again, and the screened new mesh containing the moving object is fused to obtain the range of the moving object.

Optionally, the range of the moving object is divided into regions according to the preset aspect ratio threshold, and texture calculation is performed on each region to obtain the region where the moving object is located, including:

Calculating and judging whether the range of the moving object conforms to the aspect ratio of a normal moving object according to a preset first aspect ratio threshold;

If the range of the moving object conforms to the aspect ratio of a normal moving object, then use the range of the moving object as the area where the moving object is located;

If the range of the moving object does not conform to the aspect ratio of a normal moving object, the range of the moving object is divided into k regions according to the preset second aspect ratio threshold range, where k is a natural number greater than 1; The horizontal texture, vertical texture and oblique texture are calculated in each area, and the area that best matches the aspect ratio of the normal moving object is selected according to the texture calculation results of each area, and this area is used as the area where the moving object is located.

Optionally, the second aspect ratio threshold range includes multiple levels of aspect ratio threshold ranges, and each level of aspect ratio threshold range corresponds to a number of area divisions.

According to a second aspect of the present invention, an obstacle detection system based on optical flow feature fusion is provided, including:

The collection and preprocessing module is used to collect continuous images during the parking process and preprocess the collected images;

The optical flow tracking detection module is used to divide the preprocessed image into multiple grids, use the optical flow tracking algorithm to track the position change of each grid in adjacent frames, and obtain a grid containing moving objects, which will contain moving The grid of the object is fused to obtain the range of the moving object;

The fine feature recognition module is used to divide the range of the moving object into regions according to the preset aspect ratio threshold, and perform texture calculation on each region to obtain the region where the moving object is located.

According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, and the processor is used to implement the steps of an obstacle detection method based on optical flow feature fusion when executing a computer management program stored in the memory.

According to the fourth aspect of the present invention, there is provided a computer-readable storage medium, on which a computer management program is stored, and when the computer management program is executed by a processor, an obstacle detection method based on optical flow feature fusion is implemented. step.

The present invention provides an obstacle detection method, system, electronic equipment and storage medium based on optical flow feature fusion, which solves the problem that the detection of moving objects in the automatic parking process relies on ultrasonic radar and AI detection algorithms, and only uses the vehicle's own The pictures recorded by the camera can detect whether there is a moving object in the picture after calculation, with low cost, low hardware requirements, small amount of calculation, fast speed and high detection accuracy.

Description of drawings

Fig. 1 is the flow chart of the obstacle detection method based on optical flow feature fusion provided by the present invention;

Fig. 2 is the flow chart of optical flow tracking and anti-tracking of the method provided by the present invention;

Fig. 3 is the schematic diagram that image is carried out grid division in the embodiment;

FIG. 4 is a schematic diagram of a grid including a moving object in an embodiment;

FIG. 5 is a schematic diagram of the region of the fused moving object in the embodiment;

Fig. 6 is a schematic diagram of an area conforming to the aspect ratio of a normal moving object in the embodiment;

Fig. 7 is a schematic diagram of dividing the range of the moving object into three regions in the embodiment;

Fig. 8 is a schematic diagram of dividing the range of the moving object into two regions in the embodiment;

Fig. 9 is a schematic diagram 1 of selecting the region most suitable for the moving target in the embodiment;

FIG. 10 is a schematic diagram 2 of the region selected to be the most suitable for the moving target in the embodiment;

Fig. 11 is a structural diagram of an obstacle detection system based on optical flow feature fusion provided by the present invention;

FIG. 12 is a schematic diagram of a hardware structure of a possible electronic device provided by the present invention;

FIG. 13 is a schematic diagram of a hardware structure of a possible computer-readable storage medium provided by the present invention.

Detailed ways

The specific implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

Fig. 1 is a flow chart of an obstacle detection method based on optical flow feature fusion provided by the present invention. As shown in Fig. 1, the method includes:

101. Collect continuous images during the parking process, and preprocess the collected images;

102. Divide the preprocessed image into multiple grids, use the optical flow tracking algorithm to track the position change of each grid in adjacent frames, obtain a grid containing moving objects, and fuse the grids containing moving objects , get the moving object range;

103. Divide the range of the moving object into regions according to the preset aspect ratio threshold, perform texture calculation on each region respectively, and obtain the region where the moving object is located.

It can be understood that, based on the defects in the background technology, the embodiment of the present invention proposes an obstacle detection method based on optical flow feature fusion. This method solves the problem that the detection of moving objects in the automatic parking process relies on ultrasonic radar and AI detection algorithms. It does not need to install radar on the car for moving object detection, and only uses the camera (such as a monocular camera or a monocular fisheye) that comes with the vehicle. The pictures recorded by the camera) can be calculated to detect whether there are moving objects in the picture, which is low in cost and easy to use; low in hardware requirements, small in calculation amount, fast in speed, and high in detection accuracy.

In a possible embodiment, the preprocessing of the collected images includes:

According to the current vehicle speed, thinning processing is performed on the collected continuous frame images during the parking process to obtain a collection of continuous images with intervals of n frames, where n is a natural number greater than 1.

It is understandable that because the speed of the vehicle during the automatic parking process is generally between 5km/h and 10km/h, that is, the moving distance of the vehicle in 1s is between 1.39m and 2.78m, and the monocular camera records about 30 frames in 1s. That is, the vehicle has moved 0.046m to 0.092m in the two adjacent frames of pictures. If the interval between the two frames of pictures is 8 frames, the vehicle moves 0.368m to 0.736m. The pictures taken by this moving distance are the most suitable for processing the optical flow changes on the pictures. charactermatic. Therefore, the image collection collected by the camera is thinned out, and an image is extracted every few frames to form a new image collection for subsequent optical flow tracking. After the image is thinned out, the calculation amount of optical flow tracking calculation is reduced on the basis of ensuring the accuracy of subsequent detection, and the image processing speed is improved.

In a possible embodiment, as shown in the flow chart of Figure 2, the preprocessed image is divided into multiple grids, and the optical flow tracking algorithm is used to track the position of each grid in adjacent frames Change, obtain the grid that contains moving object, comprise step 201,202 and 203, wherein:

201. Divide each frame of image obtained by preprocessing into multiple grids, and obtain the coordinates of the center point of each grid.

It can be understood that, in this step, each preprocessed image is divided into multiple small grids, and the center point of each grid is taken. For example, as shown in the image in Figure 3, the size of the image captured by the camera is 1920*1080, and there will be loss of optical flow tracking at the edge of the image. Therefore, in this step, the edge area of the image is first excluded according to the number of pixels, and then the remaining The area is divided into multiple grids. For example, the area 100 pixels away from the left and right edges is regarded as the edge area, and optical flow tracking is not performed, and the area 50 pixels away from the upper and lower edges is regarded as the edge area, and optical flow tracking is not performed, and then the remaining area is divided into 172*98 Small grids, record the center point coordinates of each small grid at the same time.

202. Use the optical flow tracking algorithm to track the central point from the previous frame image to the next frame image to obtain the corresponding tracking point; use the optical flow tracking algorithm to perform the reverse image from the next frame image to the previous frame image on the tracking point Track to get the coordinates of the corresponding anti-tracking point.

It can be understood that in this step, the coordinates of the anti-tracking point corresponding to the center point are obtained through optical flow tracking and anti-tracking, and the coordinates of the anti-tracking point can be used as a basis for subsequent judgment on whether the object in the grid moves.

203. Determine whether the center point is coincident with its corresponding anti-tracking point according to the coordinate value: if the judgment result is coincidence, it is determined that the grid where the current center point is located has moved; if the judgment result is non-coincident, then it is determined that the current center point is located grid is static.

It can be understood that in this step, the optical flow tracking result is verified. Through steps 201 and 202, the grid center point, corresponding tracking point, and corresponding anti-tracking point can be obtained. If the grid center point and the corresponding anti-tracking point basically coincide, it means that the center point is a static point, that is, in In the actual situation, this point does not move; on the contrary, if the grid center point and the corresponding anti-tracking point cannot coincide, it means that the center point is moving in the actual situation. If the center point is stationary, it is estimated that the points in the entire small grid are also stationary; if the center point is moving, it is estimated that the points in the entire small grid are also moving, and the resulting grid diagram containing moving objects As shown in Figure 4.

In a possible embodiment, the fusion of the grids containing the moving objects to obtain the range of the moving objects includes:

After dilation and erosion operations are performed on the mesh containing the moving object, the mesh containing the moving object is screened again, and the screened new mesh containing the moving object is fused to obtain the range of the moving object.

It can be understood that, through step 102 or steps 201-203, which grids on the current picture contain moving objects can be obtained. Due to the loss and false detection of optical flow tracking, sometimes the obtained moving areas are not complete and ideal. It is necessary to perform erosion and expansion operations on these areas, and then perform screening and fusion operations, as shown in Figure 4. The grid area containing the moving object extracted after optical flow tracking is shown in Figure 5 as the area of the fused moving object.

In a possible embodiment, the range of the moving object is divided into regions according to the preset aspect ratio threshold, and texture calculation is performed on each region to obtain the region where the moving object is located, including:

Calculating and judging whether the range of the moving object conforms to the aspect ratio of a normal moving object according to a preset first aspect ratio threshold;

If the range of the moving object conforms to the aspect ratio of a normal moving object, then use the range of the moving object as the area where the moving object is located;

If the range of the moving object does not conform to the aspect ratio of a normal moving object, the range of the moving object is divided into k regions according to the preset second aspect ratio threshold range, where k is a natural number greater than 1; The horizontal texture, vertical texture and oblique texture are calculated in each area, and the area that best matches the aspect ratio of the normal moving object is selected according to the texture calculation results of each area, and this area is used as the area where the moving object is located.

It can be understood that, in this embodiment, the fine feature recognition is performed for the case where the range of the moving object is too large. A first aspect ratio threshold range is set in advance for judging whether the obtained moving object range belongs to the normal moving object range. After comparing the aspect ratio of the moving object range obtained in step 102 with the first aspect ratio threshold range, it is determined that the obtained moving object range is the aspect ratio of a normal moving object, then the area of the moving object can be directly obtained, such as Figure 6 shows.

The approximate range of the moving object is obtained through step 102, but sometimes due to a sudden change in vehicle speed or a sudden change in light, the range of the moving object detected by optical flow tracking is too large, which does not conform to the normal aspect ratio of the moving object. At this time, it is necessary to re-divide this range, and calculate whether the characteristics of each divided area conform to the characteristics of the moving object.

As shown in Figure 7 and Figure 8, according to the difference in the aspect ratio of the moving object range, set the second aspect ratio threshold range, and divide the moving object range into left and right regions according to the second aspect ratio threshold range Or the three areas of left, middle and right. In order to increase the accuracy of division, optionally, the second aspect ratio threshold range includes multiple levels of aspect ratio threshold ranges, and each level of aspect ratio threshold range corresponds to the number of area divisions.

For example, if the aspect ratio ranges from 1.2 to 2.4, divide the range into left and right regions; if the aspect ratio ranges from 2.4 to 3.6, divide the range into left, middle and right regions. Then calculate the horizontal texture, vertical texture and oblique texture for each divided area, so as to filter out the area that best matches the moving target, as shown in Figure 9 and Figure 10, the dotted line box represents the candidate area of the moving object, and the solid line box Represents the finally found moving object area.

FIG. 11 is a structural diagram of an obstacle detection system based on optical flow feature fusion provided by an embodiment of the present invention. As shown in FIG. 11 , an obstacle detection system based on optical flow feature fusion includes an acquisition and preprocessing module 1101 , optical flow tracking detection module 1102 and fine feature recognition module 1106, wherein:

The collection and preprocessing module 1101 is used to collect continuous images during the parking process and preprocess the collected images;

The optical flow tracking detection module 1102 is used to divide the preprocessed image into multiple grids, use the optical flow tracking algorithm to track the position change of each grid in adjacent frames, and obtain a grid containing moving objects, which will contain The grid of the moving object is fused to obtain the range of the moving object;

The fine feature recognition module 1103 is configured to divide the range of the moving object into regions according to a preset aspect ratio threshold, and perform texture calculation on each region to obtain the region where the moving object is located.

It can be understood that the obstacle detection system based on optical flow feature fusion provided by the present invention corresponds to the obstacle detection method based on optical flow feature fusion provided in the foregoing embodiments, and the obstacle detection based on optical flow feature fusion For the relevant technical features of the system, please refer to the relevant technical features of the obstacle detection method based on optical flow feature fusion, which will not be repeated here.

Please refer to FIG. 12 . FIG. 12 is a schematic diagram of an embodiment of an electronic device provided by an embodiment of the present invention. As shown in Figure 12, the embodiment of the present invention provides an electronic device, including a memory 1210, a processor 1220, and a computer program 1211 stored in the memory 1210 and operable on the processor 1220, and the processor 1220 executes the computer program 1211 When performing the following steps:

Collect continuous images during the parking process, and preprocess the collected images;

The preprocessed image is divided into multiple grids, and the optical flow tracking algorithm is used to track the position change of each grid in adjacent frames to obtain the grid containing the moving object, and the grid containing the moving object is fused to obtain range of moving objects;

According to the preset aspect ratio threshold, the range of the moving object is divided into regions, and the texture calculation is performed on each region to obtain the region where the moving object is located.

Please refer to FIG. 13 , which is a schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. As shown in FIG. 13 , this embodiment provides a computer-readable storage medium 1300 on which a computer program 1311 is stored. When the computer program 1311 is executed by a processor, the following steps are implemented:

Collect continuous images during the parking process, and preprocess the collected images;

The preprocessed image is divided into multiple grids, and the optical flow tracking algorithm is used to track the position change of each grid in adjacent frames to obtain the grid containing the moving object, and the grid containing the moving object is fused to obtain range of moving objects;

According to the preset aspect ratio threshold, the range of the moving object is divided into regions, and the texture calculation is performed on each region to obtain the region where the moving object is located.

The embodiment of the present invention provides an obstacle detection method, system, electronic device and storage medium based on optical flow feature fusion, which solves the problem that the detection of moving objects in the automatic parking process relies on ultrasonic radar and AI detection algorithms, and only uses the vehicle The pictures recorded by the built-in camera can detect whether there are moving objects in the pictures after calculation, with low cost, low hardware requirements, small amount of calculation, fast speed and high detection accuracy.

It should be noted that, in the foregoing embodiments, descriptions of each embodiment have their own emphases, and for parts that are not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce a machine for A device for realizing the functions specified in one or more procedures of a flowchart and/or one or more blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

While preferred embodiments of the invention have been described, additional changes and modifications to these embodiments can be made by those skilled in the art once the basic inventive concept is understood. Therefore, it is intended that the appended claims be construed to cover the preferred embodiment as well as all changes and modifications which fall within the scope of the invention.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and equivalent technologies thereof, the present invention also intends to include these modifications and variations.

Claims (10)

1. The obstacle detection method based on optical flow feature fusion, is characterized in that, comprises: Collect continuous images during the parking process, and preprocess the collected images; The preprocessed image is divided into multiple grids, and the optical flow tracking algorithm is used to track the position change of each grid in adjacent frames to obtain the grid containing the moving object, and the grid containing the moving object is fused to obtain range of moving objects; According to the preset aspect ratio threshold, the range of the moving object is divided into regions, and the texture calculation is performed on each region to obtain the region where the moving object is located. 2. The method according to claim 1, wherein the preprocessing of the collected image comprises: According to the current vehicle speed, thinning processing is performed on the collected continuous frame images during the parking process to obtain a collection of continuous images with intervals of n frames, where n is a natural number greater than 1. 3. The method according to claim 1 or 2, wherein the preprocessed image is divided into a plurality of grids, and an optical flow tracking algorithm is used to track the position changes of each grid in adjacent frames, Get a mesh with moving objects, including: Divide each frame of image obtained by preprocessing into multiple grids, and obtain the coordinates of the center point of each grid; The center point is tracked from the previous frame image to the next frame image using the optical flow tracking algorithm to obtain the corresponding tracking point; the optical flow tracking algorithm is used to track the tracking point from the next frame picture to the previous frame picture, Get the coordinates of the corresponding anti-tracking point; Judging whether the center point and its corresponding anti-tracking point coincide according to the coordinate value: if the judgment result is coincidence, it is determined that the grid where the current center point is located has moved; grid still. 4. The method according to claim 3, wherein, in the process of dividing the image into a plurality of grids, the edge regions of the image are first excluded according to the number of pixels, and then the remaining areas are divided into a plurality of grids. 5. The method according to claim 1, wherein said merging the grid containing the moving object to obtain the range of the moving object comprises: After dilation and erosion operations are performed on the mesh containing the moving object, the mesh containing the moving object is screened again, and the screened new mesh containing the moving object is fused to obtain the range of the moving object. 6. The method according to claim 1, wherein the range of the moving object is divided into regions according to a preset aspect ratio threshold, and texture calculation is performed on each region to obtain the region where the moving object is located, include: Calculating and judging whether the range of the moving object conforms to the aspect ratio of a normal moving object according to a preset first aspect ratio threshold; If the range of the moving object conforms to the aspect ratio of a normal moving object, then use the range of the moving object as the area where the moving object is located; If the range of the moving object does not conform to the aspect ratio of a normal moving object, the range of the moving object is divided into k regions according to the preset second aspect ratio threshold range, where k is a natural number greater than 1; The horizontal texture, vertical texture and oblique texture are calculated in each area, and the area that best matches the aspect ratio of the normal moving object is selected according to the texture calculation results of each area, and this area is used as the area where the moving object is located. 7 . The method according to claim 6 , wherein the second aspect ratio threshold range includes multiple levels of aspect ratio threshold ranges, and each level of aspect ratio threshold range corresponds to the number of area divisions. 8. An obstacle detection system based on optical flow feature fusion, characterized in that, comprising: The collection and preprocessing module is used to collect continuous images during the parking process and preprocess the collected images; The optical flow tracking detection module is used to divide the preprocessed image into multiple grids, use the optical flow tracking algorithm to track the position change of each grid in adjacent frames, and obtain a grid containing moving objects, which will contain moving The grid of the object is fused to obtain the range of the moving object; The fine feature recognition module is used to divide the range of the moving object into regions according to the preset aspect ratio threshold, and perform texture calculation on each region to obtain the region where the moving object is located. 9. An electronic device, characterized in that it includes a memory and a processor, and the processor is used to implement the optical flow-based feature fusion as described in any one of claims 1-7 when executing a computer management program stored in the memory The steps of the obstacle detection method. 10. A computer-readable storage medium, characterized in that a computer management program is stored thereon, and when the computer management program is executed by a processor, the optical flow-based Steps of a feature fusion obstacle detection method.

Images