JP2021117988A - Positioning element detection method, positioning element detection device, electronic equipment, non-transitory computer-readable storage medium, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for detecting a positioning element which can be applied when positioning a vehicle in a scene where the vehicle is actively parked.
A positioning element detection method includes a step of acquiring a surround view composite image around a vehicle, and detecting the surround view composite image, and at least one positioning element and a surround view composite image existing on the ground around the vehicle. This includes a step of determining the meaning type to which each pixel point belongs, and a step of matching and fusing at least one positioning element using the meaning type and acquiring a detection result of the positioning element.
[Selection diagram] Fig. 1

Description

The present application relates to the field of visual positioning technology, particularly to target detection technology, specifically to positioning element detection methods, positioning element detection devices, electronic devices, non-temporary computer-readable storage media and computer programs.

Visual positioning systems are becoming more and more widely used in fields such as unmanned driving, and the function of the visual positioning system is to solve the position and orientation of the unmanned vehicle in real time based on the information acquired by the camera, which is also an unmanned vehicle. Is an important premise for voluntary exercise.

Conventionally, there are mainly two types of methods for acquiring visual information with a visual positioning system.
The first method is the visual SLAM plan, that is, the environment sensing work is completed based on the camera image, the key points in the image are extracted by the conventional image algorithm, and the matching relationship of the key points in the image of multiple frames is obtained. It is used to calculate its own positioning information. However, this method can be applied only to static scenes with sufficient lighting and clear environmental texture features, and the detection robustness of conventional visual features is low, making it difficult to obtain stable and highly accurate detection results. Naru,
The second method is to identify a manually customized positioning indicator, i.e., by detecting a specific indicator in the camera image, the exact three-dimensional position and orientation of the positioning indicator corresponding to the camera, etc. It can be calculated quickly. Achieving an algorithmic level in this way is relatively easy, but many specific markers need to be customized and placed in large numbers in the scene, and late maintenance costs are high, mass production that can be used in any scene. Cannot be used as a plan.

The present application provides a positioning element detection method, a positioning element detection device, an electronic device, a non-temporary computer-readable storage medium and a computer program, and the accuracy of detection when the positioning indicator does not need to be manually customized. And improve robustness.

In the first aspect of the present application, there is a step of acquiring a surround view composite image around the vehicle, and at least one positioning element existing on the ground around the vehicle and the surround view composite by detecting the surround view composite image. A positioning element detection method including a step of determining a word meaning type to which each pixel point of an image belongs and a step of matching and fusing at least one of the positioning elements using the word meaning type and acquiring a detection result of the positioning element. offer.

In this embodiment, a surround view composite image is input, a target naturally existing on the ground around the vehicle is detected as a positioning element, and the positioning elements are matched and fused based on the meaning information of the pixel points, and a normal sensor is used. It has the advantage or beneficial effect of not only avoiding the problem of limited field of view, but also improving the accuracy and robustness of the detection of positioning elements without the need for modification or placement on the parking side. ..

In the above aspect, the positioning element may include at least one of a parking space, a parking space number, a lane, a ground arrow, a deceleration zone and a sidewalk.

This allows these natural positioning elements to exist naturally and with the positioning elements without modifying the site, as compared to the method of using common ground markings as positioning elements and manually customizing the positioning markers. It can have the advantage or beneficial effect of playing the same role as a manually customized positioning element.

Further, in the above aspect, the surround view composite image is detected to determine the meaning type to which at least one positioning element existing on the ground around the vehicle and each pixel point of the surround view composite image belongs. The step detects the surround view composite image using a pre-trained deep neural network model, divides the meaning of each pixel point of the surround view composite image, and exists on the ground around the vehicle. The information of at least one positioning element to be used, and a step of determining the meaning type to which each of the pixel points of the surround view composite image belongs may be included.

As a result, the positioning element in the surround view composite image is detected using the deep neural network model, and the feature points based on the meaning of the word are detected. The feature points of the image in the conventional technique are unstable and are affected by environmental factors. Avoids vulnerable problems and has the advantage or beneficial effect of better robustness.

Further, in the above aspect, the information of the positioning element may include at least the type and position information of the positioning element and the type and position information of the key point of the positioning element.

Further, in the above aspect, the step of matching and fusing at least one of the positioning elements using the meaning type is combined with the position information of the key point and the pixel position of each of the pixel points of the surround view composite image. A step of matching the type of the key point at the same position and the meaning type of the pixel point, and a step of proofreading the position information of the key point of each of the positioning elements based on the matching result and a preset fusion policy. And may be included.

Thereby, in combination with the result of the word meaning division of the pixel points, the position of the key point of the positioning element is further proofread, the detection accuracy of the positioning element is improved, and the effect of the word meaning division is effective for the incomplete positioning element. It also has the advantage or beneficial effect of being robust.

Further, in the above aspect, the deep neural network model includes a positioning element detection branch and a key point detection branch, wherein the positioning element detection branch includes target classification and the key point for the positioning element. The key point detection branch is used to detect the key point, and correspondingly, the position information of the key point of the positioning element is acquired by returning the position. It may be determined by fusing the key point and the key point acquired by detecting the key point.

As a result, by providing a key point detection branch in the network in combination with the key point detection technology, accurate matching is performed for the returned key point position, and the key point position information with higher accuracy is determined by fusing. Has the advantage or beneficial effect.

Further, in the above aspect, before the step of acquiring the detection result of the positioning element, when the type of the positioning element is the parking space number, the parking space number detection frame is extracted from the deep neural network model. The angle between the step and the line connecting the corner points of the two parking spaces close to the parking space number in the parking space to which the parking space number detection frame belongs and the horizontal axis of the image coordinate system of the surround view composite image. The calculation in the deep neural network model so that the corresponding parking space number after rotation is horizontal in the image coordinate system based on the calculation step and the center point and the angle of the parking space number detection frame. A step of rotating the feature diagram of the parking space number corresponding to the parking space number detection frame and a step of identifying the parking space number with respect to the feature diagram of the parking space number after rotation using the character classifier. It may be further included.

Thereby, in the ground positioning element, the parking space number is very important information and is the only positioning element having a global ID. Therefore, the parking space number is also detected as a positioning element to identify the parking space number. This has the advantage or beneficial effect of positioning the absolute position of the vehicle on the map and improving the positioning accuracy.

Further, in the above aspect, before the step of detecting the reflection region existing in the surround view composite image using the deep neural network model and the step of acquiring the detection result of the positioning element correspondingly, the step is described. In combination with the detection result of the reflection region, the position information may further include a step of filtering the positioning element in the reflection region.

This has the advantage that the detection of the positioning element with respect to the reflection region is not accurate and therefore the accuracy of the detection of the positioning element is further improved by detecting the reflection region and filtering the positioning element using the detection result. Has a beneficial effect.

Further, in the above aspect, the step of acquiring the surround view composite image around the vehicle is the step of acquiring the image collected by the fisheye camera located around the vehicle, and the step of synthesizing the image. The step of acquiring the surround view composite image may be included.

As a result, the surround view composite image is formed by synthesizing the images collected by the four fisheyes around the vehicle, covering the 360-degree viewing angle around the vehicle body, and the field of view is wider. In addition, when one of the fisheyes expires, the other three fisheye images can also be detected by synthesizing the figures, the detection function is not invalidated, and the robustness is strong. Then, the fish eye of the vehicle body is basically attached toward the ground, and an image of the ground around the vehicle body can be obtained well, which is particularly suitable for detecting a natural positioning element of the ground. It also has the advantage or beneficial effect of having good generalization by removing the effects of fisheye distortion, internal and external parameters, and mounting position based on the detection of surround view composite images.

In the second aspect of the present application, an image acquisition module that acquires a surround view composite image around the vehicle, at least one positioning element that detects the surround view composite image and exists on the ground around the vehicle, and the surround. A positioning element detection module that determines the word meaning type to which each pixel point of the view composite image belongs, and a matching fusion module that matches and fuses at least one of the positioning elements using the word meaning type and acquires the detection result of the positioning element. Provided is a positioning element detection device comprising.

In a third aspect of the present application, the electronic device is communicably connected to a fisheye camera located around the electronic device for collecting images, at least one processor, and at least one of the processors. An instruction that can be executed by at least one processor is stored in the memory so that the at least one processor executes the positioning element detection method. Further provided are electronic devices executed by at least one of the processors.

In a fourth aspect of the present application, a non-temporary computer-readable storage medium in which computer instructions are stored, the computer instructions being used to cause the computer to perform the positioning element detection method. Provide a non-temporary computer-readable storage medium.
In a fifth aspect of the present application, there is further provided a computer program that causes the computer to execute the positioning element detection method when operating on the computer.

In this application, it has the following advantages or beneficial effects. By inputting a surround view composite image, a target that naturally exists on the ground around the vehicle is detected as a positioning element, and the positioning element is matched and fused based on the meaning information of the pixel point, and the field of view of a normal sensor is limited. Not only does it avoid the problem of The feature points of the image in the conventional technology are unstable, avoid the problem that is easily affected by environmental factors, and the robustness is better. Moreover, in combination with the result of the word meaning division of the pixel points, the position of the key point of the positioning element can be further proofread, and the detection accuracy of the positioning element can be further improved. The detection of the parking space number can further determine the absolute position of the vehicle and improve the positioning accuracy. Finally, detection of the reflection area also further improves the accuracy of detection of the positioning element.

Other effects of the selectable method will be described below in combination with specific examples.

The drawings are used to better understand the proposed technology and are not intended to limit the application.
It is a schematic flowchart of the positioning element detection method which concerns on 1st Example of this application. It is a schematic flowchart of the positioning element detection method which concerns on 2nd Example of this application. It is a schematic flowchart of the positioning element detection method which concerns on 3rd Example of this application. It is a schematic block diagram of the positioning element detection apparatus which concerns on 4th Embodiment of this application. It is a block diagram of the electronic device of the positioning element detection method which can carry out the Example of this application.

Hereinafter, exemplary examples of the present application are described in combination with the drawings, which include various details of the examples of the present application for ease of understanding, which are merely exemplary. Should be considered as a thing. It should be appreciated that one of ordinary skill in the art can therefore make various changes and amendments to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for clarity and brevity, the following description omits the description of well-known functions and structures.

FIG. 1 is a schematic flowchart of a positioning element detection method according to the first embodiment of the present application. This embodiment can be applied to the case where the positioning element is detected and the vehicle is positioned in a scene where the vehicle is actively parked. For example, the scene is a scene in which the vehicle is actively parked in an indoor parking lot. .. The positioning element detection method can be carried out by a positioning element detection device, which can be implemented by adopting software and / or hardware methods, preferably electronic devices such as automated guided vehicles. Or place it in an intelligent driving vehicle. As shown in FIG. 1, the positioning element detection method specifically includes the following steps S101 to S103.

Step S101 acquires a surround view composite image around the vehicle.

Specifically, a fisheye camera can be installed around the vehicle. For example, one fisheye camera for collecting images around the vehicle in real time is installed in front of, behind, left, and right of the vehicle, and then the surround view composite image is obtained by synthesizing the images. Can be obtained.

The combined surround view composite image covers a 360 degree viewing angle around the vehicle body, has a wider field of view, and if one of those fisheyes expires, the other three fisheye images. The figure can be synthesized and detected, the detection function is not revoked, and the robustness is strong. Moreover, the fish eye of the vehicle body is basically attached toward the ground, and a good image of the ground around the vehicle body can be obtained, which is particularly suitable for detecting a natural positioning element of the ground. It also has good generalization by removing fisheye distortion, internal and external parameters, and installation position based on the detection of surround view composite images.

Step S102 detects the surround view composite image and determines the meaning type to which at least one positioning element existing on the ground around the vehicle and each pixel point of the surround view composite image belongs.

In order to avoid the waste of labor costs and the inability to mass-produce due to large-scale remodeling of the scene and manual customization of positioning markers, the examples of this application naturally exist on the ground around the vehicle. For example, the positioning element may include at least one of a parking space, a parking space number, a lane, a ground arrow, a deceleration zone and a sidewalk. These natural positioning elements are naturally present and can be used as positioning elements without modifying the site and play the same role as manually customized positioning elements.

Of course, the present application is not limited to the above types of positioning elements in the examples of the present application, and other natural elements existing on the ground can also be identified as detection as the positioning elements in the examples of the present application. In the embodiment of, the present invention is not limited.

Detection of positioning elements can be performed based on a deep neural network model. Therefore, as a modification, step S102 of detecting the surround view composite image and determining the meaning type to which at least one positioning element existing on the ground around the vehicle and each pixel point of the surround view composite image belongs is pre-trained. The surround view composite image is detected using the deep neural network model, the meaning of each pixel point of the surround view composite image is divided, the information of at least one positioning element existing on the ground around the vehicle, and the surround It may include a step of determining the meaning type to which each pixel point of the view composite image belongs.

Here, the model may use a one-stage (one-stage) anchor-free (no candidate box) multitasking deep neural network algorithm in target detection, and multiple targets can be calculated in one network model. It can be output at the same time. Then, for different sized positioning elements in the surround view composite image, the model employs predictions on different scale sidewalk maps, with smaller sidewalk maps having wider receptive areas, such as parking spaces, etc. Suitable for predicting larger target objects such as sidewalks, larger feature maps have more detailed features and are suitable for predicting small objects such as lanes, ground arrows and details such as key points and edges of objects. Thereby, the detection effect of the multitasking combination is achieved by performing the target detection of different sizes in the feature map of different scales.

In addition, the model can realize the word meaning division, that is, the word meaning division is performed for each pixel point of the surround view composite image, and the word meaning type to which each pixel point belongs is identified, for example, whether it is a foreground or a background. , Which type of foreground it belongs to, whether it is a parking space or parking space number, etc. Thereby, the information of at least one positioning element existing on the ground around the vehicle and the meaning type to which each pixel point of the surround view composite image belongs can be acquired.

In step S103, at least one positioning element is matched and fused using the meaning type, and the detection result of the positioning element is acquired.

In order to improve the accuracy of the detection of the positioning element, the embodiment of the present application detects the positioning element on the one hand, performs the word meaning division on the other side, and uses the result of the word meaning division to use the positioning element and the pixel at the same position. Matching the meaning types of points, if the types do not match, the detection results are shown to be different and need to be fused, and the fused results are more accurate. For example, the result of the word meaning division can be selected as the final detection result, the result of the detection and the result of the word meaning division are weighted fusion as the final result, and the embodiment of the present application does not limit the method of fusion.

It should be noted that the matching fusion based on the meaning type avoids the problem of accuracy when the model is used alone, the accuracy of the detection after the fusion becomes high, and the influence of noise can be avoided. And for positioning elements that are incompletely detected in the model, the effect of word meaning division is more robust, and therefore the matching fusion of word meaning types makes the detection of positioning elements more complete.

According to the technical proposal of the embodiment of the present application, the surround view composite image is input, the target naturally existing on the ground around the vehicle is detected as the positioning element, and the positioning element is determined based on the meaning information of the pixel points. Not only does the matching fusion avoid the problem of limiting the field of view of a normal sensor, but it does not require modification or placement on the parking lot side, and at the same time, the matching fusion also improves the accuracy and robustness of the detection of positioning elements. Improve.

FIG. 2 is a schematic flowchart of a positioning element detection method according to a second embodiment of the present application, and this embodiment is further optimized based on the above embodiment. As shown in FIG. 2, the positioning element detection method specifically includes the following steps S201 to S204.

Step S201 acquires a surround view composite image around the vehicle.

Step S202 detects the surround view composite image using a pre-trained deep neural network model, divides the meaning of each pixel point of the surround view composite image, and at least one present on the ground around the vehicle. The information of the positioning element and the meaning type to which each pixel point of the surround view composite image belongs is determined.

Here, the information of the positioning element includes at least the type and position information of the positioning element and the type and position information of the key point of the positioning element.

Positioning element types include, for example, parking spaces, parking space numbers, lanes, ground arrows, deceleration zones and sidewalks, position information includes, for example, the position of the positioning element detection frame, and key points are models. It can be pre-determined prior to training and represents the feature points of different types of positioning elements, such as the corner point of the parking space, the left apex of the parking space number or the end line point of the lane. .. In the examples of the present application, the selection of key points for different types of positioning elements is not limited.

Detection of positioning elements and key points can be realized by a deep neural network algorithm model of multitasking coupling. Selectably, based on the model's target classification and repositioning, the embodiments of the present application add a detection network branch of the target key point to further improve the accuracy of repositioning in post-processing. That is, the deep neural network model includes a positioning element detection branch and a key point detection branch, where the positioning element detection branch is used to perform target classification and key point position return for the positioning element. , The key point detection branch is used to detect the key point.

Specifically, the positioning element detection branch can determine the detection frame of the positioning element by detection and identify the type of the positioning element, and the position of at least one key point of the positioning element by the position return in the detection frame. To get. Since the position accuracy detected at the key point is always higher than the position accuracy of the key point calculated by the position return, the embodiment of the present application is made so as to improve the position accuracy of the finally determined key point. The position information of the key points of the positioning element is obtained by detecting the key points using the key point detection branch, and then fusing the acquired key points by returning to the position and the key points acquired by detecting the key points. Is finally decided.

Step S203 matches the type of the key point at the same position and the word meaning type of the pixel point in combination with the position information of the key point and the pixel position of each pixel point of the surround view composite image.

In step S204, the position information of the key point of each positioning element is calibrated based on the matching result and the preset fusion policy, and the detection result of the positioning element is acquired.

Here, if the type of the key point at the same position and the word meaning type of the pixel point do not match, it is shown that the detected and acquired positioning element and key point are not accurate. Therefore, a fusion policy based on the result of the word meaning division can be selected to proofread the position information of the key points of the positioning element, and the same type of the same position obtained by the word meaning division method and the model detection method, respectively. You may choose to calibrate the position information of the key points by weighted fusion of the position results. As a result, the accurate key point position is finally acquired by the matching fusion method, and the more accurate detection result of the positioning element is acquired.

According to the technical proposal of the embodiment of the present application, the surround view composite image is input, the target naturally existing on the ground around the vehicle is detected as the positioning element, and the positioning element is determined based on the meaning information of the pixel points. Not only does it avoid the problem of limited field of view of the normal sensor by matching fusion, but it does not require modification or placement on the parking lot side, and at the same time, it is combined with the result of word meaning division of pixel points of the positioning element. The position of the key point can be further calibrated to improve the accuracy of detecting the positioning element.

FIG. 3 is a schematic flowchart of the positioning element detection method according to the third embodiment of the present application, and this embodiment is further optimized based on the above embodiment. As shown in FIG. 3, the positioning element detection method specifically includes the following steps S301 to S307.

Step S301 acquires a surround view composite image around the vehicle.

Step S302 detects the surround view composite image using a pre-trained deep neural network model, divides the meaning of each pixel point of the surround view composite image, and at least one existing on the ground around the vehicle. The information of the positioning element and the meaning type to which each pixel point of the surround view composite image belongs is determined. Here, the information of the positioning element includes at least the type and position information of the positioning element and the type and position information of the key point of the positioning element.

Step S303 extracts a parking space number detection frame from the deep neural network model when the type of positioning element is parking space number.

Step S304 calculates the angle between the line connecting the corner points of the two parking spaces close to the parking space number in the parking space to which the parking space number detection frame belongs and the horizontal axis of the image coordinate system of the surround view composite image.

Step S305 corresponds to the parking space number detection frame in the deep neural network model so that the corresponding parking space number after rotation is horizontal in the image coordinate system based on the center point and the angle of the parking space number detection frame. Rotate the feature diagram of the parking space number to be used.

Step S306 identifies the parking space number with respect to the characteristic diagram of the parking space number after rotation using the character classifier.

In step S307, at least one positioning element is matched and fused using the meaning type, and the detection result of the positioning element is acquired.

In the operations of S301 to S306, the identification of the parking space number was realized. In the ground positioning element, the parking space number is very important information, but it is the only positioning element having a global ID. Therefore, the parking space number is also detected as a positioning element to identify the parking space number. , Position the absolute position of the vehicle on the map and improve the positioning accuracy.

Specifically, the parking space number detection frame is first extracted, and then the line connecting the corner points of the two parking spaces close to the parking space number in the parking space to which the detection frame belongs and the side of the image coordinate system of the surround view composite image. Determine the angle between the axes, then rotate the feature diagram of the parking space number corresponding to the detection frame in the model according to the angle, level the corresponding parking space number in the image coordinate system, and park after the last rotation. The parking space number is identified by inputting the feature diagram of the space number into the character classifier.

In real life, the parking space number is usually located at the front end of the parking space, that is, the end that the vehicle passed through when it first entered the parking lot, that is, the two parking spaces that are close to the parking space number in the parking space. The deep neural network model is a parking space in a parking space because it is one end of a line connecting the corner points of and the parking space number is usually marked from left to right when people face the parking space. By identifying the corner points of the two parking spaces, left and right, which are close to the numbers, the line connecting the two corner points can be determined to determine the direction of rotation, and the corresponding parking space number in the rotated feature diagram. Is horizontal, but avoids the phenomenon of inverting the characters in the parking space number.

Further, in the embodiment of the present application, the deep neural network model further includes a branch used particularly for detecting the reflection region, and here, the detection algorithm of the reflection region is limited in the embodiment of the present application. Instead, any algorithm in the conventional technique may be adopted.

That is, the positioning element detection method according to the present embodiment further includes a step of detecting a reflection region existing in the surround view composite image using a deep neural network model, and correspondingly, a step of acquiring a detection result of the positioning element. Previously, it may further include a step of filtering the positioning elements whose position information is in the reflection region in combination with the detection result of the reflection region.

In indoor and outdoor environments, especially indoor environments, reflections typically occur and detection results for positioning elements in the reflection area are not accurate. Therefore, the accuracy of detecting the positioning element is further improved by detecting the reflection region, filtering the positioning element using the detection result, and removing these positioning elements displayed in the reflection region.

As a modification, for a deep neural network model, its output includes at least positioning element information, key point detection information, word meaning division information of each pixel point, and reflection region information. You can stay. When these outputs are sent to the post-processing module, accurate matching between the key point detection information and the key points of the positioning element can be achieved, thereby improving the accuracy of the key point detection. Further, it is possible to realize a matching fusion of the accurately matched key points and the information of the meaning division, and the purpose is to further improve the accuracy as well. It is also possible to identify parking space numbers and perform fusion filtering of positioning elements using reflective areas, and of course, to perform lane point screening with point clusters for detected lanes. Can improve the accuracy of lane detection. The execution order of each operation in the post-processing module is not limited in the examples of the present application.

According to the technical proposal of the embodiment of the present application, the surround view composite image is input, the target naturally existing on the ground around the vehicle is detected as the positioning element, and the positioning element is determined based on the word meaning information of the pixel points. Not only does it match and fuse to avoid the problem of limiting the field of view of a normal sensor, it does not require modification or placement on the parking lot side, and at the same time it uses a deep neural network model for positioning in surround view composite images. It detects elements, realizes detection of feature points based on word meaning, avoids the problem that feature points of images are unstable and susceptible to environmental factors in the conventional technology, and has better robustness. The detection of the parking space number can further determine the absolute position of the vehicle and improve the positioning accuracy. Finally, detection of the reflection area also further improves the accuracy of detection of the positioning element.

FIG. 4 is a schematic configuration diagram of the positioning element detection device according to the fourth embodiment of the present application, and is a case where the present embodiment is applicable. The positioning element detection device can realize the positioning element detection method described by any embodiment of the present application.
As shown in FIG. 4, specifically, the positioning element detection device 400 detects an image acquisition module 401 that acquires a surround view composite image around the vehicle and a surround view composite image, and puts the surround view composite image on the ground around the vehicle. At least one positioning element existing and the positioning element detection module 402 that determines the word meaning type to which each pixel point of the surround view composite image belongs and at least one positioning element are matched and fused using the word meaning type, and the detection result of the positioning element The matching fusion module 403 and the matching fusion module 403 are provided.

As a variant, the positioning element may include at least one of a parking space, a parking space number, a lane, a ground arrow, a deceleration zone and a sidewalk.

As a modification, specifically, the positioning element detection module 402 detects the surround view composite image using a deep neural network model trained in advance, and divides the meaning of each pixel point of the surround view composite image. , Information on at least one positioning element present on the ground around the vehicle, and used to determine the meaning type to which each pixel point of the surround view composite image belongs.

As a modification, the information on the positioning element may include at least the type and position information of the positioning element and the type and position information of the key points of the positioning element.

As a modification, the matching fusion module 403 is combined with the key point position information and the pixel position of each pixel point of the surround view composite image to match the key point type at the same position and the pixel point word meaning type with a matching unit. , A calibration unit for calibrating the position information of the key points of each positioning element based on the matching result and the preset fusion policy may be provided.

As a modification, the deep neural network model includes a positioning element detection branch and a key point detection branch, wherein the positioning element detection branch performs target classification and key point position return for the positioning element. Used, the key point detection branch is used to detect the key point, and correspondingly, the position information of the key point of the positioning element is acquired by detecting the key point and the key point acquired by returning to the position. It may be determined by fusing with the key points that have been made.

As a variant, the positioning element detector further comprises a parking space number detection module, the parking space number detection module, specifically, the type of positioning element before the matching fusion module obtains the detection result of the positioning element. If it is a parking space number, the parking space number detection frame is extracted from the deep neural network model, and the line connecting the corner points of the two parking spaces close to the parking space number in the parking space to which the parking space number detection frame belongs and the surround view. Calculate the angle between the horizontal axis of the image coordinate system of the composite image and make the corresponding parking space number after rotation horizontal in the image coordinate system based on the center point and angle of the parking space number detection frame. In addition, the feature diagram of the parking space number corresponding to the parking space number detection frame in the deep neural network model is rotated, and the parking space number is identified for the feature diagram of the parking space number after the rotation using the character classifier. It may be used to execute the operation.

As a variant, the positioning element detection module 402 is further used to detect the reflection region present in the surround view composite image using a deep neural network model, and the positioning element detection device accordingly detects the positioning element. Before acquiring the result, a positioning element filtering module for filtering the positioning element whose position information is in the reflection region may be further provided in combination with the detection result of the reflection region.

As a modification, the image acquisition module 401 includes an image acquisition unit that acquires images collected by fisheye cameras located around the vehicle, and an image composition unit that synthesizes images to acquire a surround view composite image. You may have it.

The positioning element detection device 400 disclosed by the embodiments of the present application can execute the positioning element detection method provided by any embodiment of the present application, and can provide a functional module corresponding to the execution method and a beneficial effect. Be prepared. For content not described in detail in this example, the description of an example of any method of the present application can be referred to.

According to the embodiments of the present application, the present application further provides electronic devices and readable storage media.

As shown in FIG. 5, it is a block diagram of an electronic device of a positioning element detection method according to an embodiment of the present application. Electronic devices are intended to represent various types of digital computers such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices can also represent various types of mobile devices such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the description and / or the required realization of this application.

As shown in FIG. 5, the electronic device includes at least one processor 501, a memory 502, and an interface for connecting each component including a high-speed interface and a low-speed interface. Each component can be interconnected on different buses and mounted on a common motherboard, or otherwise mounted as needed. The processor processes instructions executed within an electronic device, including instructions stored in memory to display GUI graphic information on an external input / output device (such as a display device coupled to an interface). Can be done. In other embodiments, a plurality of processors and / or a plurality of buses can be used with the plurality of memories and the plurality of memories, if necessary. Similarly, a plurality of electronic devices can be connected, and each electronic device can provide a partial necessary operation (for example, a server array, a blade server, or a multiprocessor system). In FIG. 5, one processor 501 is taken as an example.

Memory 502 is a non-temporary computer-readable storage medium provided by this application. Here, the memory 502 stores an instruction that can be executed by at least one processor so that at least one processor executes the above-mentioned positioning element detection method. The non-temporary computer-readable storage medium of the present application stores computer instructions for causing a computer to execute the above-mentioned positioning element detection method.

The memory 502 is a non-temporary computer-readable storage medium, and is a program instruction / module (for example, the image acquisition module 401 shown in FIG. 4 and the positioning element detection module) corresponding to the positioning element detection method according to the embodiment of the present application. It is used to store non-temporary software programs, non-temporary computer-executable programs and modules such as 402, and Matching Fusion Module 403). The processor 501 executes various functional applications and data processing of the server by executing non-temporary software programs, instructions and modules stored in the memory 502, that is, realizing the positioning element detection method described above. ..

The memory 502 can include a program storage area and a data storage area, where the program storage area can store an operating system, an application program required for at least one function, and the data storage area is a data storage area. Data generated by the use of electronic devices based on the realization of the positioning element detection method according to the embodiment of the present application can be stored. The memory 502 can also include fast random storage memory and can further include non-temporary memory, eg, at least one disk storage device, flash memory device, or other non-temporary solid state. It is a state storage device. In some embodiments, the memory 502 can selectively include memory located remotely to the processor 501, which remote memory detects positioning elements of the embodiments of the present application over a network. It can be connected to an electronic device that realizes the method. Examples of the above networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device that realizes the positioning element detection method according to the embodiment of the present application may further include an input device 503 and an output device 504. The processor 501, the memory 502, the input device 503, and the output device 504 can be connected via a bus or other method, and in FIG. 5, the connection by the bus is taken as an example.

The input device 503 can receive the input numerical or character information, and generates a key signal input relating to user setting and function control of the electronic device that realizes the positioning element detection method according to the embodiment of the present application. It is an input device such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, at least one mouse button, a trackball, a joystick, and the like. The output device 504 can include a display device, an auxiliary lighting device (eg, LED), a haptic feedback device (eg, a vibration motor), and the like. Display devices can include, but are not limited to, liquid crystal displays (LCDs), light emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

Various embodiments of the systems and techniques described herein include digital electronic circuit systems, integrated circuit systems, application specific ASICs (ASICs), computer hardware, firmware, software, and / or them. It can be realized by the combination of. These various embodiments may include: It can include being implemented in at least one computer program, the at least one computer program can be run and / or interpreted in a programmable system that includes at least one programmable processor, and the programmable processor is specific. It may be a general purpose or general purpose programmable processor, receiving data and instructions from a storage system, at least one input device, and at least one output device, and sending the data and instructions to the storage system, at least one input device, and It can be transmitted to at least one output device.

These computing programs (also referred to as programs, software, software applications, or code) may include programmable processor machine instructions in high-level process and / or object-oriented programming languages and / or assembly / machine languages. These computing programs can be implemented. As used herein, the terms "machine readable medium" and "computer readable medium" are any computer programs used to provide machine instructions and / or data to programmable processors. Refers to products, devices, and / or devices (eg, magnetic disks, optical disks, memories, programmable logic devices (PLDs)), including machine-readable media that receive machine commands, which are machine-readable signals. The term "machine readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

To provide interaction with the user, the systems and techniques described herein can be implemented on a computer, which is a display device for displaying information to the user (eg, a CRT (cathode tube)). Alternatively, it has an LCD (liquid crystal display) monitor) and a keyboard and a pointing device (for example, a mouse or a track ball), and the user can provide input to the computer by the keyboard and the pointing device. Other types of devices can also be used to provide interaction with the user, eg, the feedback provided to the user is any form of sensing feedback (eg, visual feedback, auditory feedback, or tactile feedback). ), And the input from the user can be received in any format (including acoustic input, voice input, and tactile input).

The systems and techniques described herein are computing systems that include back-end components (eg, data servers), or computing systems that include middleware components (eg, application servers), or computing that includes front-end components. With a system (eg, a user computer having a graphical user interface or web browser, the user interacts with embodiments of systems and techniques described herein by the graphical user interface or web browser), or such back-end components. It can be implemented in computing systems that include any combination of middleware components and front-end components. The components of the system can be interconnected by digital data communication of any form or medium (eg, a communication network). An example of a communication network includes a local area network (LAN), a wide area network (WAN), and the Internet.

A computer system can include a client and a server. Clients and servers are generally separated from each other and typically interact over a communication network. A client-server relationship is created by a computer program that runs on the corresponding computer and has a client-server relationship with each other.

According to the technical proposal of the embodiment of the present application, the surround view composite image is input, the target naturally existing on the ground around the vehicle is detected as the positioning element, and the positioning element is determined based on the word meaning information of the pixel points. Not only does it match and fuse to avoid the problem of limiting the field of view of a normal sensor, it does not require modification or placement on the parking lot side, and at the same time it uses a deep neural network model for positioning in surround view composite images. It detects elements, realizes detection of feature points based on word meaning, avoids the problem that the feature points of images in the conventional technique are unstable and susceptible to environmental factors, and has better robustness. Moreover, in combination with the result of the word meaning division of the pixel points, the position of the key point of the positioning element can be further proofread, and the detection accuracy of the positioning element can be further improved. The detection of the parking space number can further determine the absolute position of the vehicle and improve the positioning accuracy. Finally, detection of the reflection area also further improves the accuracy of detection of the positioning element.

It should be understood that steps can be rearranged, added, or deleted using the various forms of flow shown above. For example, the steps described in this application may be performed in parallel, sequentially, or in a different order, but the techniques disclosed in this application. The present specification is not limited as long as the desired result of the proposal can be achieved.

The specific embodiments described above do not limit the scope of protection for this application. One of ordinary skill in the art can make various modifications, combinations, subcombinations, and alternatives based on design requirements and other factors. Any amendments, equivalent replacements, and improvements made within the spirit and principles of this application should be within the scope of this application's protection.

Claims (21)

It is a positioning element detection method
Steps to get a surround view composite image around the vehicle,
A step of detecting the surround view composite image and determining the meaning type to which at least one positioning element existing on the ground around the vehicle and each pixel point of the surround view composite image belongs.
A positioning element detection method including a step of matching and fusing at least one of the positioning elements using the meaning type and acquiring a detection result of the positioning element. The positioning element detection method according to claim 1, wherein the positioning element includes at least one of a parking space, a parking space number, a lane, a ground arrow, a deceleration zone, and a sidewalk. The step of detecting the surround view composite image and determining the meaning type to which at least one of the positioning elements present on the ground around the vehicle and each of the pixel points of the surround view composite image belongs.
The surround view composite image is detected using a pre-trained deep neural network model, word meaning division is performed for each pixel point of the surround view composite image, and at least one existing on the ground around the vehicle. The positioning element detection method according to claim 1 or 2, further comprising a step of determining the meaning type to which each of the pixel points of the surround view composite image belongs and the information of the positioning element. The positioning element detection method according to claim 3, wherein the positioning element information includes at least the type and position information of the positioning element and the key point type and position information of the positioning element. The step of matching and fusing at least one of the positioning elements using the word meaning type is
A step of matching the type of the key point at the same position and the meaning type of the pixel point in combination with the position information of the key point and the pixel position of each pixel point of the surround view composite image.
The positioning element detection method according to claim 4, further comprising a step of calibrating the position information of the key point of each of the positioning elements based on a matching result and a preset fusion policy. The deep neural network model includes a positioning element detection branch and a key point detection branch, wherein the positioning element detection branch performs target classification and position return of the key point for the positioning element. Used, the keypoint detection branch is used to detect the keypoint.
Correspondingly, the position information of the key point of the positioning element is determined by claim 4 which is determined by fusing the key point acquired by returning to the position and the key point acquired by detecting the key point. Positioning element detection method. Before the step of acquiring the detection result of the positioning element,
A step of extracting a parking space number detection frame from the deep neural network model when the type of the positioning element is a parking space number, and
A step of calculating the angle between the line connecting the corner points of the two parking spaces close to the parking space number in the parking space to which the parking space number detection frame belongs and the horizontal axis of the image coordinate system of the surround view composite image. When,
The parking space number detection frame in the deep neural network model so that the corresponding parking space number after rotation is horizontal in the image coordinate system based on the center point and the angle of the parking space number detection frame. The step of rotating the feature diagram of the parking space number corresponding to
The positioning element detection method according to claim 4, further comprising a step of identifying the parking space number with respect to the feature diagram of the parking space number after rotation using the character classifier. A step of detecting a reflection region existing in the surround view composite image using the deep neural network model, and
Correspondingly, the method according to claim 4, further comprising a step of filtering the positioning element whose position information is in the reflection region in combination with the detection result of the reflection region before acquiring the detection result of the positioning element. .. The step of acquiring the surround view composite image around the vehicle is
The step of acquiring the images collected by the fisheye cameras located around the vehicle, and
The positioning element detection method according to claim 1, further comprising a step of synthesizing the images and acquiring the surround view composite image. An image acquisition module that acquires surround view composite images around the vehicle,
A positioning element detection module that detects the surround view composite image and determines the meaning type to which at least one positioning element existing on the ground around the vehicle and each pixel point of the surround view composite image belongs.
A positioning element detection device including a matching fusion module that matches and fuses at least one of the positioning elements using the meaning type and acquires a detection result of the positioning element. The positioning element detecting device according to claim 10, wherein the positioning element includes at least one of a parking space, a parking space number, a lane, a ground arrow, a deceleration zone, and a sidewalk. Specifically, the positioning element detection module is
The surround view composite image is detected using a pre-trained deep neural network model, word meaning division is performed for each pixel point of the surround view composite image, and at least one existing on the ground around the vehicle. The positioning element detection device according to claim 10 or 11, which is used to determine the information of the positioning element and the meaning type to which each of the pixel points of the surround view composite image belongs. The positioning element detection device according to claim 12, wherein the information on the positioning element includes at least the type and position information of the positioning element and the type and position information of the key points of the positioning element. The matching fusion module
A matching unit that matches the type of the key point at the same position and the meaning type of the pixel point in combination with the position information of the key point and the pixel position of each pixel point of the surround view composite image.
The positioning element detection device according to claim 13, further comprising a calibration unit that calibrates the position information of the key point of each of the positioning elements based on a matching result and a preset fusion policy. The deep neural network model includes a positioning element detection branch and a key point detection branch, wherein the positioning element detection branch performs target classification and position return of the key point for the positioning element. Used, the keypoint detection branch is used to detect the keypoint.
Correspondingly, claim 13 determines that the position information of the key point of the positioning element is determined by fusing the key point acquired by returning to the position and the key point acquired by detecting the key point. The positioning element detection device according to the description. Equipped with a parking space number detection module
Specifically, the module is used before the matching fusion module acquires the detection result of the positioning element.
When the type of the positioning element is the parking space number, the parking space number detection frame is extracted from the deep neural network model, and the parking space number detection frame is extracted.
The angle between the line connecting the corner points of the two parking spaces close to the parking space number in the parking space to which the parking space number detection frame belongs and the horizontal axis of the image coordinate system of the surround view composite image is calculated.
The parking space number detection frame in the deep neural network model so that the corresponding parking space number after rotation is horizontal in the image coordinate system based on the center point and the angle of the parking space number detection frame. Rotate the feature diagram of the parking space number corresponding to
The positioning element detection device according to claim 13, which is used for performing an operation of identifying a parking space number with respect to a characteristic diagram of the parking space number after being rotated by using a character classifier. The positioning element detection module 402 is used to detect a reflection region existing in the surround view composite image using the deep neural network model.
30. Positioning element detector. The image acquisition module
An image acquisition unit that acquires images collected by fisheye cameras located around the vehicle, and an image acquisition unit.
The positioning element detection device according to claim 10, further comprising an image composition unit that synthesizes the images and acquires the surround view composite image. It ’s an electronic device,
A fisheye camera located around the electronic device to collect images,
With at least one processor
It comprises at least one processor and communicably connected memory.
An instruction that can be executed by at least one processor is stored in the memory, and when the instruction is executed by at least one processor, the instruction is executed by at least one processor according to claims 1 to 9. An electronic device that performs the positioning element detection method described in any of the above. A non-temporary computer-readable storage medium that stores computer instructions.
The computer instruction is a non-temporary computer-readable storage medium used to cause the computer to execute the positioning element detection method according to any one of claims 1 to 9. A computer program that causes the computer to execute the positioning element detection method according to any one of claims 1 to 9, while operating on the computer.

Images