CN113963188B - Method, system, device and medium for combining map information visual positioning - Google Patents
Method, system, device and medium for combining map information visual positioning Download PDFInfo
- Publication number
- CN113963188B CN113963188B CN202111087441.6A CN202111087441A CN113963188B CN 113963188 B CN113963188 B CN 113963188B CN 202111087441 A CN202111087441 A CN 202111087441A CN 113963188 B CN113963188 B CN 113963188B
- Authority
- CN
- China
- Prior art keywords
- image
- pose
- images
- matching
- distance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The application relates to a method, a system, a device and a medium for visual positioning by combining map information, wherein the method comprises the following steps: acquiring an image to be positioned, extracting feature vectors, key points and descriptors, matching the feature vectors with the images in a map database to obtain candidate place image sets, sorting the sets, selecting two sets closest to each other, selecting a first close set from the two sets according to a self-defined algorithm, and matching the key point descriptors of the image to be positioned with the first close set to obtain matching point pairs; according to the matching point pairs or the 2D-3D matching pairs, carrying out relevance ranking on the images in the first proximity set, estimating the initial value of the pose of the image to be positioned, and calculating the final pose of the image to be positioned through a PnP algorithm; and (3) checking the final pose by comparing the distance and the included angle between the final pose and the initial pose value, and returning a positioning result. According to the application, the calculation speed and the positioning accuracy are improved.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a method, system, apparatus, and medium for visual positioning in conjunction with map information.
Background
The 3D visual positioning refers to calculating and obtaining the position and the posture of a camera when shooting a certain image according to a 3D model and related information which are constructed in advance. This is a very important technique for 3D vision that can be used to help achieve personnel positioning and navigation. The conventional method of visual localization is to input image information, and output the 6DoF pose of the obtained image through feature extraction, feature matching, pnP calculation and the like.
However, in the related art, in the case of visually locating an image, on one hand, if a scene and an image to be located have a plurality of similar features, and a correct position cannot be resolved, an incorrect locating result is usually obtained, or a locating result cannot be calculated. On the other hand, when the features are matched, all the scenes are matched, and as the database is too large, the matching speed is low, and excessive noise is easy to introduce; or there may be some false images that are misled, and feature matching is performed on the false images, which not only wastes time, but also easily introduces excessive noise points to cause deviation or incapability of solving the result.
At present, aiming at the problems of inaccurate positioning, low matching efficiency and high time cost when the image is visually positioned in the related technology, no effective solution is proposed yet.
Disclosure of Invention
The embodiment of the application provides a method, a system, a device and a medium for visual positioning by combining map information, which at least solve the problems of inaccurate positioning, low matching efficiency and high time cost in the visual positioning of images in the related technology.
In a first aspect, an embodiment of the present application provides a method for visual positioning in conjunction with map information, where the method includes:
acquiring an image to be positioned, extracting feature vectors, key points and key point descriptors of the image to be positioned, and matching the feature vectors with image feature vectors in a map database to obtain a candidate place image set;
Sorting the candidate place image sets according to the image pose, selecting two sets closest to the feature vector of the image to be positioned, selecting a first close set from the two sets according to a self-defining algorithm, and performing feature matching on the key point descriptors of the image to be positioned and the key point descriptors of the image in the first close set to obtain key point matching point pairs;
According to the key point matching point pairs or the 2D-3D matching pairs between the image to be positioned and the map, carrying out relevance ranking on the images in the first proximity set, estimating to obtain the pose initial value of the image to be positioned, and calculating to obtain the final 6DoF pose of the image to be positioned through a PnP algorithm;
and comparing the distance and the included angle between the final 6DoF pose and the initial value of the pose, and checking the final 6DoF pose to obtain a positioning result.
In some embodiments, before matching the feature vector with an image feature vector in a map database to obtain a candidate location image set, the method includes:
and acquiring the map database, wherein the map database comprises a map scene image, feature vectors of the image, image key points and key point descriptors, 6DoF pose information of the image, information of map 3D points and association information between the image key points and the map 3D points.
In some embodiments, the matching the feature vector with an image feature vector in a map database to obtain a candidate location image set includes:
Calculating Euclidean distance between the image feature vector to be positioned and the image feature vector in the map database, sorting according to the distance, and selecting the first N images closest to the map database as candidate place images;
and clustering the candidate place images according to the image pose to obtain a plurality of candidate place image sets.
In some embodiments, the selecting the first proximity set from the two sets according to a custom algorithm includes:
Judging whether the image to be positioned has ambiguity according to a distance judging formula, wherein the distance judging formula is as follows: d1> αd2, α is a scaling factor of 0-1, d1 is an average distance of the image feature vector to be localized to the first close set, and d2 is an average distance of the image feature vector to be localized to the second close set;
If the distance discrimination formula is established, the image to be positioned has ambiguity, and returns a conclusion and terminates the positioning process;
and under the condition that the distance discrimination formula is not established, the image to be localized is not ambiguous, and the first proximity set is selected.
In some of these embodiments, before feature matching the keypoint descriptor of the image to be located with the keypoint descriptors of the images in the first proximity set, the method includes:
and sequencing the images in the first proximity set according to the feature vector distance, and selecting M images, wherein the selection mode comprises but is not limited to scene characteristics, image textures or distance.
In some embodiments, the performing relevance ranking on the images in the first proximity set, and estimating the pose initial value of the image to be positioned includes:
P candidate images with highest association degree are selected, and the pose initial value of the image to be positioned is estimated and obtained through a coarse positioning estimation method according to the pose of the candidate images.
In some of these embodiments, said verifying said final 6DoF pose by comparing a distance and an angle between said final 6DoF pose and said pose initial value comprises:
Presetting a distance deviation threshold and an included angle deviation threshold, wherein when the distance deviation between the final 6DoF pose and the pose initial value exceeds the distance deviation threshold or the included angle deviation between the final 6DoF pose and the pose initial value exceeds the included angle deviation threshold, the final 6DoF pose is checked and not passed, the final 6DoF pose is discarded and the positioning process is terminated, otherwise, checking is passed, and the final 6DoF pose is returned as a positioning result.
In a second aspect, embodiments of the present application provide a system for visual localization in conjunction with map information, the system comprising:
the acquisition module is used for acquiring an image to be positioned and extracting feature vectors, key points and key point descriptors of the image to be positioned;
the retrieval module is used for matching the feature vector with the image feature vector in the map database to obtain a candidate place image set;
the matching module is used for sorting the candidate place image sets according to the image pose, selecting two sets closest to the feature vector of the image to be positioned, selecting a first close set from the two sets according to a self-defining algorithm, and performing feature matching on the key point descriptors of the image to be positioned and the key point descriptors of the images in the first close set to obtain key point matching point pairs;
The positioning module is used for sorting the association degree of the images in the first approaching set according to the key point matching point pairs or the 2D-3D matching pairs between the images to be positioned and the map, estimating to obtain the pose initial value of the images to be positioned, and calculating to obtain the final 6DoF pose of the images to be positioned through a PnP algorithm;
And the verification module is used for verifying the final 6DoF pose by comparing the distance and the included angle between the final 6DoF pose and the initial value of the pose, so as to obtain a positioning result.
In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the method for visual positioning in combination with map information according to the first aspect described above when the processor executes the computer program.
In a fourth aspect, an embodiment of the present application provides a storage medium having stored thereon a computer program which, when executed by a processor, implements a method of visual localization in combination with map information as described in the first aspect above.
Compared with the related art, the method for combining map information for visual positioning provided by the embodiment of the application has the advantages that the image to be positioned is obtained, the feature vector, the key point and the key point descriptor of the image to be positioned are extracted, and the feature vector is matched with the image feature vector in the map database to obtain the candidate place image set; then, sorting the candidate place image sets according to the image pose, selecting two sets closest to the feature vector of the image to be positioned, selecting a first close set from the two sets according to a self-defining algorithm, and performing feature matching on the key point descriptors of the image to be positioned and the key point descriptors of the image in the first close set to obtain key point matching point pairs; then, according to the key point matching point pairs or the 2D-3D matching pairs between the image to be positioned and the map, carrying out relevance ranking on the images in the first proximity set, estimating to obtain the pose initial value of the image to be positioned, and calculating to obtain the final 6DoF pose of the image to be positioned through a PnP algorithm; and finally, comparing the distance and the included angle between the final 6DoF pose and the initial pose value, and checking the final 6DoF pose to obtain a positioning result.
The application has the beneficial effects that: 1. by means of clustering the candidate site images, the search range is locked in a small area, computer resource expenditure of the following key point matching steps is reduced, and calculation speed is improved. 2. The application eliminates more error interference through multistage verification and similarity comparison, not only can improve the matching efficiency, ensure that the finally solved positioning result can be stably output, but also can ensure the accuracy of the positioning result, and particularly can effectively improve the positioning precision under the condition of repeated textures and similar scenes. The problems of inaccurate positioning, low matching efficiency and high time cost existing in the visual positioning of the image in the related technology are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
FIG. 1 is a schematic view of an application environment of a method for visual localization in conjunction with map information according to an embodiment of the present application;
FIG. 2 is a flow chart of a method of visual localization in conjunction with map information in accordance with an embodiment of the present application;
FIG. 3 is a block diagram of a system for visual localization in conjunction with map information in accordance with an embodiment of the present application;
fig. 4 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application.
Detailed Description
The present application will be described and illustrated with reference to the accompanying drawings and examples in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application. All other embodiments, which can be made by a person of ordinary skill in the art based on the embodiments provided by the present application without making any inventive effort, are intended to fall within the scope of the present application. Moreover, it should be appreciated that while such a development effort might be complex and lengthy, it would nevertheless be a routine undertaking of design, fabrication, or manufacture for those of ordinary skill having the benefit of this disclosure, and thus should not be construed as having the benefit of this disclosure.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly and implicitly understood by those of ordinary skill in the art that the described embodiments of the application can be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," and similar referents in the context of the application are not to be construed as limiting the quantity, but rather as singular or plural. The terms "comprising," "including," "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to only those steps or elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "connected," "coupled," and the like in connection with the present application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as used herein means greater than or equal to two. "and/or" describes an association relationship of an association object, meaning that there may be three relationships, e.g., "a and/or B" may mean: a exists alone, A and B exist together, and B exists alone. The terms "first," "second," "third," and the like, as used herein, are merely distinguishing between similar objects and not representing a particular ordering of objects.
The method for combining the map information with the visual positioning can be applied to an application environment shown in fig. 1, and fig. 1 is a schematic diagram of the application environment of the method for combining the map information with the visual positioning according to the embodiment of the application, as shown in fig. 1. Wherein the terminal 11 communicates with the server 10 via a network. The server 10 acquires an image to be positioned, extracts feature vectors, key points and key point descriptors of the image to be positioned, and matches the feature vectors with image feature vectors in a map database to obtain a candidate place image set; then, sorting the candidate place image sets according to the image pose, selecting two sets closest to the feature vector of the image to be positioned, selecting a first close set from the two sets according to a self-defining algorithm, and performing feature matching on the key point descriptors of the image to be positioned and the key point descriptors of the image in the first close set to obtain key point matching point pairs; then, according to the key point matching point pairs or the 2D-3D matching pairs between the image to be positioned and the map, carrying out relevance ranking on the images in the first proximity set, estimating to obtain the pose initial value of the image to be positioned, and calculating to obtain the final 6DoF pose of the image to be positioned through a PnP algorithm; finally, the final 6DoF pose is checked by comparing the distance and the included angle between the final 6DoF pose and the initial pose value, and a positioning result is obtained and displayed on the terminal 11. The terminal 11 may be, but not limited to, various personal computers, notebook computers, smartphones, tablet computers, and portable wearable devices, and the server 10 may be implemented as a stand-alone server or a server cluster composed of a plurality of servers.
The present embodiment provides a method for combining map information to perform visual positioning, and fig. 2 is a flowchart of a method for combining map information to perform visual positioning according to an embodiment of the present application, as shown in fig. 2, where the flowchart includes the following steps:
Step S201, obtaining an image to be positioned, extracting feature vectors, key points and key point descriptors of the image to be positioned, and matching the feature vectors with image feature vectors in a map database to obtain a candidate place image set;
Preferably, in this embodiment, before matching the feature vector with the image feature vector in the map database to obtain the candidate location image set, map data needs to be obtained from a map database constructed in advance, where the map database includes a map scene image, feature vectors of the image, image keypoints and keypoint descriptors, 6DoF pose information of the image, information of map 3D points, association information between image keypoints and map 3D points, and the like.
Then, the embodiment acquires an image to be positioned, extracts feature vectors, key points and key point descriptors of the image to be positioned, and matches the feature vectors with image feature vectors in a map database to obtain a candidate place image set. Optionally, in this embodiment, first, the euclidean distance between the feature vector of the image to be located and the feature vector of the image in the map database is calculated, then, the images in the map database are ordered according to the distance obtained by calculation, and the top N images closest to the map database are selected as the candidate location images. Wherein, N generally selects values of tens to thousands according to the size of the scene, the density of the image sampling positions in the map database, and the like; finally, clustering the candidate place images according to the image pose, wherein the clustering method comprises the following steps: k-means, DBSCAN, etc., a number of candidate site image sets are obtained, e.g., { S1, S2, … }.
According to the embodiment, the candidate place images close to the image to be positioned are selected from the map database through feature matching, and then the search range is locked in a small area through clustering sets of the candidate place images, so that the computer resource cost in the following key point matching steps is reduced, and the calculation speed is improved;
step S202, sorting candidate place image sets according to image pose, selecting two sets closest to feature vectors of images to be positioned, selecting a first close set from the two sets according to a self-defining algorithm, and performing feature matching on key point descriptors of the images to be positioned and key point descriptors of the images in the first close set to obtain key point matching point pairs;
Preferably, in this embodiment, the candidate location image sets are ranked according to the image pose, and two sets closest to the feature vector of the image to be positioned are selected, and optionally, the method for selecting the sets includes, but is not limited to, 1, selecting the sets according to the image consistency classification, for example, scene features or image textures of the image; 2. selecting a set, such as a regional concentrated interval, according to a custom rule;
Then, further verification and selection are performed on the two selected sets, and optionally, in this embodiment, whether the image to be localized has ambiguity is determined according to a distance discrimination formula, where the distance discrimination formula is: d1> αd2, α is a scaling factor of 0-1, d1 is the average distance of the image feature vector to be localized to the first close set, and d2 is the average distance of the image feature vector to be localized to the second close set; if the distance discrimination formula is established, the image to be positioned has ambiguity, and the conclusion is returned to the corresponding upper logic, and the positioning flow is terminated; under the condition that a distance discrimination formula is not established, the fact that the image to be positioned has no ambiguity is indicated, a set which is close to the feature vector of the image to be positioned for the first time in the two sets is selected, and feature matching is carried out on a key point descriptor of the image to be positioned and a key point descriptor of the image in the first close set, so that a key point matching point pair is obtained;
Preferably, before feature matching is performed on the keypoint descriptors of the image to be positioned and the keypoint descriptors of the images in the first proximity set, the images in the first proximity set are ordered according to the feature vector distance, and M images are selected, wherein the selection mode includes, but is not limited to, image selection according to scene characteristics, image texture richness or distance.
According to the embodiment, through multistage verification and similarity comparison, more error interference factors are eliminated, the matching efficiency is improved, and the positioning result finally solved in the follow-up process can be stably output and is not interfered by excessive noise;
Step S203, according to the key point matching point pairs or the 2D-3D matching pairs between the image to be positioned and the map, carrying out relevance ranking on the images in the first proximity set, estimating to obtain the pose initial value of the image to be positioned, and calculating to obtain the final 6DoF pose of the image to be positioned through a PnP algorithm;
Specifically, in this embodiment, the 2D-3D matching pair between the image to be located and the map is obtained according to the association information between the image key point in the map database and the 3D point in the map, and the association information between the 2D key point in the image to be located and the 3D point in the map is obtained.
Preferably, in this embodiment, the relevance ranking is performed on the M images in the first proximity set obtained in step S202 according to the key point matching point pairs or the 2D-3D matching pairs between the image to be located and the map.
Optionally, in this embodiment, P candidate images with highest association degree in the M images are selected, and according to the pose of the candidate images, the pose initial value of the image to be positioned is estimated by a coarse positioning estimation method. The coarse positioning estimation method comprises the following steps: 1. and 2, calculating an average value, calculating a weighted average value, and 4, directly selecting the pose of the most relevant image.
Further, in this embodiment, the obtained pose initial values are substituted by different PnP algorithms, and the final 6DoF pose of the image to be positioned is obtained through iterative calculation;
And S204, checking the final 6DoF pose by comparing the distance and the included angle between the final 6DoF pose and the initial pose value to obtain a positioning result.
Specifically, the distance deviation threshold and the included angle deviation threshold are preset, the final 6DoF pose is checked by comparing the distance and the included angle between the final 6DoF pose and the initial pose value, and if the distance deviation between the final 6DoF pose and the initial pose value exceeds the distance deviation threshold or the included angle deviation between the final 6DoF pose and the initial pose value exceeds the included angle deviation threshold, the final 6DoF pose is checked not to pass, the final 6DoF pose result is discarded, and the positioning process is terminated; and if the distance and the included angle pass the verification, returning the final 6DoF pose as a positioning result.
According to the embodiment, the final pose is compared and verified to obtain the positioning result, so that the accuracy of the positioning result can be effectively ensured, and particularly, the positioning precision can be effectively improved under the condition of repeated textures and similar scenes.
Through the steps S201 to S204, the embodiment of the application reduces the searching range through the local clustering, improves the calculation speed of key point matching, eliminates more error interference through multi-stage verification and similarity comparison, not only improves the matching efficiency, ensures the stable output of the finally solved positioning result, but also ensures the accuracy of the positioning result, and particularly effectively improves the positioning precision under the condition of repeated textures and similar scenes. The problems of inaccurate positioning, low matching efficiency and high time cost existing in the visual positioning of the image in the related technology are solved.
It should be noted that the steps illustrated in the above-described flow or flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order other than that illustrated herein.
The embodiment also provides a system for combining map information to realize visual positioning, which is used for realizing the above embodiment and the preferred implementation, and the description is omitted. As used below, the terms "module," "unit," "sub-unit," and the like may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
Fig. 3 is a block diagram of a system for visual localization in conjunction with map information according to an embodiment of the present application, and as shown in fig. 3, the system includes an acquisition module 31, a retrieval module 32, a matching module 33, a localization module 34, and a verification module 35:
The acquiring module 31 is configured to acquire an image to be positioned, and extract feature vectors, key points and key point descriptors of the image to be positioned; the retrieval module 32 is configured to match the feature vector with the image feature vector in the map database to obtain a candidate location image set; the matching module 33 is configured to sort the candidate location image sets according to the image pose, select two sets closest to the feature vector of the image to be positioned, select a first close set from the two sets according to a custom algorithm, and perform feature matching on the key point descriptor of the image to be positioned and the key point descriptor of the image in the first close set to obtain a key point matching point pair; the positioning module 34 is configured to perform relevance ranking on the images in the first proximity set according to the key point matching point pairs or the 2D-3D matching pairs between the image to be positioned and the map, estimate to obtain a pose initial value of the image to be positioned, and calculate to obtain a final 6DoF pose of the image to be positioned through PnP algorithm; and the verification module 35 is used for verifying the final 6DoF pose by comparing the distance and the included angle between the final 6DoF pose and the initial pose value, so as to obtain a positioning result.
Through the system, the embodiment of the application reduces the search range through the location clustering, improves the calculation speed of key point matching, and in addition, the matching module 33 and the verification module 35 also remove more error interference through multi-level verification and similarity comparison, so that the matching efficiency can be improved, the finally solved positioning result can be stably output, the accuracy of the positioning result can be ensured, and the positioning precision can be effectively improved especially under the condition of repeated textures and similar scenes. The problems of inaccurate positioning, low matching efficiency and high time cost existing in the visual positioning of the image in the related technology are solved.
It should be noted that, specific examples in this embodiment may refer to examples described in the foregoing embodiments and alternative implementations, and this embodiment is not repeated herein.
The above-described respective modules may be functional modules or program modules, and may be implemented by software or hardware. For modules implemented in hardware, the various modules described above may be located in the same processor; or the above modules may be located in different processors in any combination.
The present embodiment also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, where the transmission device is connected to the processor, and the input/output device is connected to the processor.
In addition, in combination with the method for visual positioning combined with map information in the above embodiment, the embodiment of the application can be implemented by providing a storage medium. The storage medium has a computer program stored thereon; the computer program when executed by a processor implements the method of any of the above embodiments in combination with map information visual localization.
In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of visual localization in conjunction with map information. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
In one embodiment, fig. 4 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application, and as shown in fig. 4, an electronic device, which may be a server, is provided, and an internal structure diagram thereof may be as shown in fig. 4. The electronic device includes a processor, a network interface, an internal memory, and a non-volatile memory connected by an internal bus, where the non-volatile memory stores an operating system, computer programs, and a database. The processor is used for providing computing and control capabilities, the network interface is used for communicating with an external terminal through a network connection, the internal memory is used for providing an environment for the operation of an operating system and a computer program, the computer program is executed by the processor to realize a method for combining map information visual positioning, and the database is used for storing data.
It will be appreciated by those skilled in the art that the structure shown in fig. 4 is merely a block diagram of a portion of the structure associated with the present inventive arrangements and is not limiting of the electronic device to which the present inventive arrangements are applied, and that a particular electronic device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It should be understood by those skilled in the art that the technical features of the above-described embodiments may be combined in any manner, and for brevity, all of the possible combinations of the technical features of the above-described embodiments are not described, however, they should be considered as being within the scope of the description provided herein, as long as there is no contradiction between the combinations of the technical features.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (10)
1. A method of visual localization in conjunction with map information, the method comprising:
acquiring an image to be positioned, extracting feature vectors, key points and key point descriptors of the image to be positioned, and matching the feature vectors with image feature vectors in a map database to obtain a candidate place image set;
Sorting the candidate place image sets according to the image pose, selecting two sets closest to the feature vector of the image to be positioned, selecting a first close set from the two sets according to a self-defining algorithm, and performing feature matching on the key point descriptors of the image to be positioned and the key point descriptors of the image in the first close set to obtain key point matching point pairs;
According to the key point matching point pairs or the 2D-3D matching pairs between the image to be positioned and the map, carrying out relevance ranking on the images in the first proximity set, estimating to obtain the pose initial value of the image to be positioned, and calculating to obtain the final 6DoF pose of the image to be positioned through a PnP algorithm;
and comparing the distance and the included angle between the final 6DoF pose and the initial value of the pose, and checking the final 6DoF pose to obtain a positioning result.
2. The method of claim 1, wherein prior to matching the feature vector with an image feature vector in a map database to obtain a set of candidate location images, the method comprises:
and acquiring the map database, wherein the map database comprises a map scene image, feature vectors of the image, image key points and key point descriptors, 6DoF pose information of the image, information of map 3D points and association information between the image key points and the map 3D points.
3. The method of claim 1, wherein matching the feature vector with an image feature vector in a map database to obtain a set of candidate location images comprises:
Calculating Euclidean distance between the image feature vector to be positioned and the image feature vector in the map database, sorting according to the distance, and selecting the first N images closest to the map database as candidate place images;
and clustering the candidate place images according to the image pose to obtain a plurality of candidate place image sets.
4. The method of claim 1, wherein selecting a first close set from the two sets according to a custom algorithm comprises:
Judging whether the image to be positioned has ambiguity according to a distance judging formula, wherein the distance judging formula is as follows: d1> αd2, α is a scaling factor of 0-1, d1 is an average distance of the image feature vector to be localized to the first close set, and d2 is an average distance of the image feature vector to be localized to the second close set;
If the distance discrimination formula is established, the image to be positioned has ambiguity, and returns a conclusion and terminates the positioning process;
and under the condition that the distance discrimination formula is not established, the image to be localized is not ambiguous, and the first proximity set is selected.
5. The method of claim 1, wherein prior to feature matching the keypoint descriptors of the image to be located with the keypoint descriptors of the images in the first proximity set, the method comprises:
and sequencing the images in the first proximity set according to the feature vector distance, and selecting M images, wherein the selection mode comprises but is not limited to scene characteristics, image textures or distance.
6. The method of claim 1, wherein the ranking the images in the first proximity set according to the association degree, and estimating the pose initial value of the image to be localized comprises:
P candidate images with highest association degree are selected, and the pose initial value of the image to be positioned is estimated and obtained through a coarse positioning estimation method according to the pose of the candidate images.
7. The method of claim 1, wherein said verifying said final 6DoF pose by comparing a distance and an angle between said final 6DoF pose and said pose initial value comprises:
Presetting a distance deviation threshold and an included angle deviation threshold, wherein when the distance deviation between the final 6DoF pose and the pose initial value exceeds the distance deviation threshold or the included angle deviation between the final 6DoF pose and the pose initial value exceeds the included angle deviation threshold, the final 6DoF pose is checked and not passed, the final 6DoF pose is discarded and the positioning process is terminated, otherwise, checking is passed, and the final 6DoF pose is returned as a positioning result.
8. A system for visual localization in conjunction with map information, the system comprising:
the acquisition module is used for acquiring an image to be positioned and extracting feature vectors, key points and key point descriptors of the image to be positioned;
the retrieval module is used for matching the feature vector with the image feature vector in the map database to obtain a candidate place image set;
the matching module is used for sorting the candidate place image sets according to the image pose, selecting two sets closest to the feature vector of the image to be positioned, selecting a first close set from the two sets according to a self-defining algorithm, and performing feature matching on the key point descriptors of the image to be positioned and the key point descriptors of the images in the first close set to obtain key point matching point pairs;
The positioning module is used for sorting the association degree of the images in the first approaching set according to the key point matching point pairs or the 2D-3D matching pairs between the images to be positioned and the map, estimating to obtain the pose initial value of the images to be positioned, and calculating to obtain the final 6DoF pose of the images to be positioned through a PnP algorithm;
And the verification module is used for verifying the final 6DoF pose by comparing the distance and the included angle between the final 6DoF pose and the initial value of the pose, so as to obtain a positioning result.
9. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, the processor being arranged to run the computer program to perform the method of visual localization in combination with map information of any one of claims 1 to 7.
10. A storage medium having stored therein a computer program, wherein the computer program is arranged to, when run, perform the method of combining map information visual localization of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111087441.6A CN113963188B (en) | 2021-09-16 | 2021-09-16 | Method, system, device and medium for combining map information visual positioning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111087441.6A CN113963188B (en) | 2021-09-16 | 2021-09-16 | Method, system, device and medium for combining map information visual positioning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113963188A CN113963188A (en) | 2022-01-21 |
| CN113963188B true CN113963188B (en) | 2024-08-23 |
Family
ID=79461834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111087441.6A Active CN113963188B (en) | 2021-09-16 | 2021-09-16 | Method, system, device and medium for combining map information visual positioning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113963188B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117058230A (en) * | 2023-07-13 | 2023-11-14 | 杭州易现先进科技有限公司 | Object pose estimation method, system and computer readable storage medium |
| CN120339988B (en) * | 2025-06-13 | 2025-08-26 | 深圳库犸科技有限公司 | Image matching method, map construction method, travel control method, apparatus, and medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111724443A (en) * | 2020-06-09 | 2020-09-29 | 中国科学院自动化研究所 | A unified scene visual localization method based on generative adversarial network |
| CN112150551A (en) * | 2020-09-25 | 2020-12-29 | 北京百度网讯科技有限公司 | Object pose acquisition method and device and electronic equipment |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120300020A1 (en) * | 2011-05-27 | 2012-11-29 | Qualcomm Incorporated | Real-time self-localization from panoramic images |
| US8711370B1 (en) * | 2012-10-04 | 2014-04-29 | Gerard Dirk Smits | Scanning optical positioning system with spatially triangulating receivers |
| GB2599947B (en) * | 2020-10-16 | 2022-10-19 | Slamcore Ltd | Visual-inertial localisation in an existing map |
| CN113393515B (en) * | 2021-05-21 | 2023-09-19 | 杭州易现先进科技有限公司 | Visual positioning method and system combining scene annotation information |
-
2021
- 2021-09-16 CN CN202111087441.6A patent/CN113963188B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111724443A (en) * | 2020-06-09 | 2020-09-29 | 中国科学院自动化研究所 | A unified scene visual localization method based on generative adversarial network |
| CN112150551A (en) * | 2020-09-25 | 2020-12-29 | 北京百度网讯科技有限公司 | Object pose acquisition method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113963188A (en) | 2022-01-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8798357B2 (en) | Image-based localization | |
| CN111783770B (en) | Image correction method, device and computer readable storage medium | |
| JP6211407B2 (en) | Image search system, image search device, search server device, image search method, and image search program | |
| CN112581477B (en) | Image processing method, image matching method, device and storage medium | |
| Yang et al. | An improved Bag-of-Words framework for remote sensing image retrieval in large-scale image databases | |
| CN109117854B (en) | Key point matching method and device, electronic equipment and storage medium | |
| EP2668618A1 (en) | Method and system for comparing images | |
| CN113963188B (en) | Method, system, device and medium for combining map information visual positioning | |
| Tzeng et al. | User-driven geolocation of untagged desert imagery using digital elevation models | |
| Gonzalez-Diaz et al. | Neighborhood matching for image retrieval | |
| CN110851639A (en) | Method and equipment for searching picture by picture | |
| JP6017277B2 (en) | Program, apparatus and method for calculating similarity between contents represented by set of feature vectors | |
| Chen et al. | Robust local structure visualization for remote sensing image registration | |
| CN105190689A (en) | Image processing including adjoin feature based object detection, and/or bilateral symmetric object segmentation | |
| CN110083731B (en) | Image retrieval method, device, computer equipment and storage medium | |
| CN113505257B (en) | Image retrieval method, trademark retrieval method, electronic device and storage medium | |
| Jiao et al. | Deep combining of local phase quantization and histogram of oriented gradients for indoor positioning based on smartphone camera | |
| Peng et al. | The knowing camera 2: recognizing and annotating places-of-interest in smartphone photos | |
| Peng et al. | KISS: Knowing camera prototype system for recognizing and annotating places-of-interest | |
| CN114936119A (en) | Page element positioning method and device, computer equipment and storage medium | |
| Bhattacharya et al. | DT-RANSAC: a delaunay triangulation based scheme for improved RANSAC feature matching | |
| CN113392672A (en) | Face retrieval method, device and storage medium | |
| Iwanaga et al. | A Modified Histogram Approach to Trademark Image Retrieval | |
| Li et al. | A robust correspondence-based registration method for large-scale outdoor point cloud | |
| Ma et al. | A correspondence selection method based on same object and same position constraints |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |