CN112288817A - Image-based three-dimensional reconstruction processing method and device - Google Patents

Abstract

The present disclosure particularly relates to an image-based three-dimensional reconstruction method and apparatus, a computer-readable medium, and an electronic device. The method includes: collecting an image sequence and obtaining positioning information; selecting consecutive multiple frames of subsequent images following the first image in the image sequence, and performing feature point matching on the first image and each subsequent image to filter out the images matching the first image. Post-sequence images, generating a first image matching pair and establishing a matching relationship between the first image and the post-sequence image; Feature point matching is performed on each image in the two image sets to screen the second image matching the first image, a second image matching pair is generated, and the matching relationship between the first image and the second image is established; The image matching relationship between the first image matching pair and the second image matching pair constructs a global matching relationship; three-dimensional reconstruction of the image sequence is performed based on the global matching relationship.

Description

Image-based three-dimensional reconstruction processing method and device

technical field

The present disclosure relates to the technical field of image processing, and in particular, to an image-based three-dimensional reconstruction method, an image-based three-dimensional reconstruction apparatus, a computer-readable medium, and an electronic device.

Background technique

In the field of computer vision, 3D reconstruction is a research focus. Generally speaking, commonly used three-dimensional reconstruction algorithms in the prior art include: SFM (Structure from motion, motion recovery structure) technology, Dynamic Fusion (dynamic fusion) algorithm, Bundle Fusion (bundle fusion) algorithm and so on. In the related art, the method of matching and screening based on image features is mostly adopted, and the robustness of the descriptors depending on the image features is compared. However, for locally similar textures, image features cannot effectively solve such scene reconstruction. For example, when the same sign or logo appears in different places, false matches may occur; i.e., correlating pictures that do not belong to the same place, which can seriously reduce the accuracy of 3D reconstruction and even cause the mapping to fail.

It should be noted that the information disclosed in the above Background section is only for enhancement of understanding of the background of the present disclosure, and therefore may contain information that does not form the prior art that is already known to a person of ordinary skill in the art.

SUMMARY OF THE INVENTION

The present disclosure provides an image-based three-dimensional reconstruction method, an image-based three-dimensional reconstruction device, a computer-readable medium, and an electronic device, which can effectively avoid image mismatch and improve the accuracy of reconstructed maps.

Other features and advantages of the present disclosure will become apparent from the following detailed description, or be learned in part by practice of the present disclosure.

According to a first aspect of the present disclosure, there is provided an image-based three-dimensional reconstruction method, comprising:

Collect image sequences, and obtain the positioning information of each image;

In the image sequence, a first image and consecutive frames of subsequent images of the first image are selected, and feature point matching is performed on the first image and each of the subsequent images to filter out the first image and the subsequent images. Image-matched post-sequence images, generating a first image matching pair and establishing a matching relationship between the first image and the post-sequence image; and

Use the positioning information to filter a second image set matching the position of the first image in the image sequence, and perform feature point matching on the first image and each image in the second image set to filter out the second image matched by the first image, generate a second image matching pair and establish a matching relationship between the first image and the second image;

A global matching relationship is constructed based on the image matching relationship between the first image matching pair and the second image matching pair corresponding to the first image; and three-dimensional reconstruction is performed on the image sequence based on the global matching relationship.

According to a second aspect of the present disclosure, there is provided an image-based three-dimensional reconstruction device, comprising:

The data acquisition module is used to collect the image sequence and obtain the positioning information of each image;

The first image matching module is used to select the first image and successive frames of subsequent images of the first image in the image sequence, and perform feature points on the first image and each of the subsequent images. matching to filter subsequent images that match the first image, generate a first image matching pair and establish a matching relationship between the first image and the subsequent image; and

A second image matching module, configured to use the positioning information to filter a second image set matching the position of the first image in the image sequence, and to compare each of the first image and the second image set performing feature point matching on the image to screen the second image matching the first image, generating a second image matching pair and establishing a matching relationship between the first image and the second image;

A reconstruction module, configured to construct a global matching relationship based on the image matching relationship between the first image matching pair and the second image matching pair corresponding to the first image; and perform three-dimensional reconstruction on the image sequence based on the global matching relationship.

According to a third aspect of the present disclosure, there is provided a computer-readable medium on which a computer program is stored, and when the computer program is executed by a processor, implements the above-mentioned image-based three-dimensional reconstruction method.

According to a fourth aspect of the present disclosure, there is provided an electronic device, comprising:

one or more processors;

A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the above-mentioned image-based three-dimensional reconstruction method .

An image-based three-dimensional reconstruction method provided by an embodiment of the present disclosure first collects an image sequence, and marks the positioning information of each image when collecting images; Establish a matching relationship; after establishing the above-mentioned matching relationship, use the positioning information corresponding to each image to screen again the images of the position matching, and calculate the feature matching relationship between the images of the position matching, so that the loopback detection can be realized according to the position information; Only unserialized matching of images attached to the same spatial location is realized, which greatly reduces false loopback matching and improves the accuracy and robustness of mapping.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 schematically shows a schematic flowchart of an image-based three-dimensional reconstruction method in an exemplary embodiment of the present disclosure;

FIG. 2 schematically shows a schematic flowchart of an image acquisition method in an exemplary embodiment of the present disclosure;

FIG. 3 schematically shows a schematic flowchart of an image matching method in an exemplary embodiment of the present disclosure;

FIG. 4 schematically shows a composition diagram of an image-based three-dimensional reconstruction apparatus in an exemplary embodiment of the present disclosure;

FIG. 5 schematically shows a schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the figures are functional entities that do not necessarily necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

In the related technologies of computer vision, SFM technology is usually used to recover the spatial structure of the three-dimensional environment. The traditional SFM algorithm generally includes: feature extraction and matching, calculation of initial matching pairs and point clouds, beam adjustment, and then repeatedly adding new image frame data and then combining certain strategies for beam adjustment and other steps to achieve 3D reconstruction. There are two common methods for serialized image input in feature matching: one is serialized matching, and the other is global brute force matching. Among them, serialized matching will not be able to form a loopback in a large-scale environment due to the excessive accumulation of accumulated errors. In order to ensure that the final map is still consistent with the actual map in a large-scale environment, brute force search is often used to perform feature matching calculation on all images to add some loopback image pairs and reduce the cumulative error. However, this method results in a large amount of computation and a long time-consuming. Based on this, the existing methods are usually based on image features for matching and screening, and rely on the robustness of image feature descriptors, and for locally similar textures, image features cannot effectively solve such scene reconstruction. For example, when the same signboard or logo appears in different places, there may be incorrect matching, that is, pictures that do not belong to the same place are related to each other, which will seriously reduce the accuracy of 3D reconstruction and even lead to the failure of mapping.

In view of the above-mentioned shortcomings and deficiencies of the prior art, this exemplary embodiment provides an image-based three-dimensional reconstruction method. Referring to Fig. 1, the above-mentioned image-based three-dimensional reconstruction method may include the following steps:

S11, collect an image sequence, and obtain the positioning information of each image;

S12: Select a first image and multiple consecutive frames of subsequent images following the first image in the image sequence, and perform feature point matching on the first image and each of the subsequent images to filter out the first image and the subsequent images. A subsequent image of the first image matching, generating a first image matching pair and establishing a matching relationship between the first image and the subsequent image; and

S13: Screen a second image set matching the position of the first image in the image sequence by using the positioning information, and perform feature point matching on the first image and each image in the second image set to Screening a second image matching the first image, generating a second image matching pair and establishing a matching relationship between the first image and the second image;

S14. Build a global matching relationship based on the image matching relationship between the first image matching pair and the second image matching pair corresponding to the first image; and perform three-dimensional reconstruction on the image sequence based on the global matching relationship.

In the three-dimensional reconstruction method provided by this exemplary embodiment, on the one hand, serialized matching is performed, that is, the matching images are screened and a matching relationship is established by using successive multiple frames of subsequent images after each image; After the relationship, the positioning information corresponding to each image is used to screen the position-matched images again, and the feature matching relationship between the position-matched images is calculated, so that the loop closure detection can be realized according to the position information; thus, the traditional image-based image-based detection can be avoided. Feature loopback detection, and only perform non-serialized matching on images near the same spatial position, that is, loopback detection; furthermore, false loopback matching is greatly reduced, and the accuracy and robustness of mapping are improved. Especially for scenes with unreliable image features (including but not limited to many repeated textures, illumination changes, and weak textures), the accuracy of the reconstructed map can be greatly improved.

Hereinafter, each step of the image-based three-dimensional reconstruction method in this exemplary embodiment will be described in more detail with reference to the accompanying drawings and embodiments.

In step S11, an image sequence is collected, and positioning information of each image is obtained.

In this exemplary embodiment, the above method can be applied to terminal devices with a shooting function, such as smart terminal devices such as mobile phones and tablet computers equipped with rear cameras; for example, it can be applied to image and indoor positioning information of indoor environments. collection. Specifically, referring to FIG. 2 , the above-mentioned step S11 may include:

Step S111, in response to the image acquisition instruction, activate the monocular camera to perform RGB image acquisition; and

Step S112: Invoke the UWB driver to acquire the position information when the RGB image is collected, and configure the position information as the positioning information of the RGB image.

For example, the user can trigger an image acquisition instruction in the interactive interface of the application program on the terminal device. After the application obtains the user's image capture instruction, it can call and activate the monocular camera through the command port, so that the monocular camera can collect and shoot RGB images of the current indoor environment or current indoor scene according to certain rules, and obtain the corresponding image sequence. For example, the shooting rules may be the moving speed of the terminal device, the frequency of image shooting, the shooting angle, specific shooting parameters, and so on.

In addition, while calling the monocular camera for RGB image acquisition, the Ultra Wide Band (UWB) driver can also be called synchronously according to the image acquisition instruction, and the positioning information of the UWB can be synchronously collected while collecting the image. For example, a UWB chip can be installed in the above-mentioned terminal equipment; the user can pre-configure the UWB driver to collect the positioning information at the same frequency as the image acquisition frequency; or, according to the requirements of the UWB driver itself frequency. Specifically, for the collected positioning information, the corresponding positioning information collection time may be marked.

In step S12, a first image and successive frames of subsequent images following the first image are selected from the image sequence, and feature point matching is performed on the first image and each of the subsequent images to filter For a subsequent image matched with the first image, a first image matching pair is generated and a matching relationship between the first image and the subsequent image is established.

In this exemplary embodiment, after the collection of images and positioning information is completed, the image sequence may be processed in an offline manner. Specifically, for the image sequence, each frame of image can be selected as the first image in sequence; at the same time, the consecutive k frames of images after the first image are selected as the subsequent images of the first image. Wherein, k is a positive integer, for example, it can be configured as 5, 7, 8, 9, or 10, 11 and other values. The present disclosure does not specifically limit the specific value of k. Then, feature matching can be performed on the first image and the corresponding set of subsequent images. Realize the serialization of each image in the image sequence to calculate the matching relationship between the images.

Specifically, performing feature point matching on the first image and the subsequent images as described in the above step S12 to filter the subsequent images matched by the first image, and establishes the relationship between the first image and the subsequent images. The matching relationship of the subsequent images, as shown in FIG. 3 , may specifically include:

Step S121, performing feature extraction on the first image and subsequent images to obtain two-dimensional feature points and feature descriptors corresponding to each of the two-dimensional feature points;

Step S122, using the feature descriptor to calculate the distance between the feature points of the first image and the subsequent images, and establishing the feature points between the first image and the subsequent images with the feature points whose distance is less than a preset threshold point matching pair;

Step S123, constructing a corresponding basic matrix based on the above-mentioned first image and the subsequent image, and using a random sampling consistency algorithm to screen the feature point matching pairs based on the basic matrix;

Step S124 , if the number of matched feature points after screening is greater than a preset threshold, determine that the first image matches the subsequent image and establish a matching relationship between the first image and the subsequent image.

Specifically, for an image sequence, serialized matching can be performed from the first frame of image. For example, the first frame of images in the image sequence is initially used as the first image, and then six consecutive frames of images in the subsequent sequence are selected as the subsequent images. For the selected first image and multiple frames of subsequent images, the matching relationship between the first image and each frame of subsequent images may be calculated respectively.

Specifically, feature point extraction may be performed on the first image and each subsequent image to calculate corresponding two-dimensional feature points and their feature description sub-information. The feature descriptor information is used to calculate the distance of each feature point between the two images; if the distance before the feature point is smaller than the preset threshold, it can be determined that the two feature points constitute a pair of feature point matching pairs. In this way, multiple pairs of two-dimensional feature point matching pairs can be obtained between every two images.

After obtaining all feature point matching pairs between the two images, the RANSAC algorithm (RandomSample Consensus, random sampling consensus algorithm) is used to filter the feature point matching pairs between the two images, and the mismatched feature point matching pairs are deleted. . Specifically, a fundamental matrix can be constructed first by using the feature point information of the two images, and the matching point pairs between these images can be screened by RANSAC using the fundamental matrix. If the number of feature point matching point pairs between the two images after screening is still greater than a preset threshold, it is determined that the two images match, forming an image matching pair; and the matching relationship between the two images is saved in the database, and records Filtered feature point matching pairs. The random sampling consistency algorithm can complete the selection of feature point matching by conventional methods, and its specific process will not be repeated.

By traversing the image sequence in the above manner, the first matching of each image in the image sequence is completed, and the image matching result and matching relationship corresponding to each image are obtained.

Alternatively, in some exemplary embodiments of the present disclosure, after the image acquisition is completed, feature extraction may be performed on all images in the image sequence, two-dimensional feature points and corresponding feature descriptors of each image are calculated, and stored in the preset database. For example, a SIFT algorithm (Scale-invariant features transform, scale-invariant features transform) can be used for feature extraction.

In step S13, a second image set matching the position of the first image is screened in the image sequence by using the positioning information, and each image in the first image and the second image set is characterized point matching to filter the second image matching the first image, generate a second image matching pair and establish a matching relationship between the first image and the second image

In this exemplary embodiment, while the feature information is used to match the image sequence, the second matching of the images can also be performed by using the positional relationship. For example, positioning information can be used to match images in a sequence of images. Specifically, for the above-mentioned first image, while using the feature information to perform the first matching, or after the first matching is completed, the positioning information can also be used to filter the corresponding second image set.

In this example implementation, specifically, the above-mentioned using the positioning information to filter the second image set matching the position of the first image in the image sequence may include:

Step S131, respectively calculating the distance between the positioning information of the first image and the positioning information of other images in the image sequence;

Step S132, if the distance is less than a preset distance threshold, add the corresponding image to the second image set.

Specifically, due to the pictures collected at the same position, there is a matching relationship even if the pictures are collected at different times during the entire sampling time. For the selected first image, the corresponding positioning information P can be read from the database, and the positioning information Q of other images in the image sequence can be read at the same time, and the difference between the position Q of the first image and the position Q between each image can be calculated respectively. Euclidean distance between .

If the distance between the two frames of images is less than the preset distance threshold, it can be considered that the two frames of images were collected at the same location, which may constitute an image matching pair, and the image is added to the second image set corresponding to the first image. . If the distance between the two frames of images is large, it is considered that there is no corresponding matching relationship. By repeating the above steps and traversing the image sequence, the second image set corresponding to each image can be obtained.

In this exemplary embodiment, after the second image set corresponding to each image is obtained, the above-mentioned methods of steps S121 to S124 can be used to perform feature information on each image in the first image and the second image set again. For matching, the corresponding matching relationship is calculated; that is, firstly, the matching pairs of feature points between two images are calculated, and then the matching pairs of feature points are screened to obtain the matching result of the image pairs.

If a matching relationship is established after re-matching, it is considered that a loopback match has occurred. So far, the first match and the second match are completed.

Alternatively, in other exemplary embodiments of the present disclosure, the second matching may also be performed by using the preceding image of the first image. Specifically, after the first image matching pair is generated and the matching relationship between the first image and the subsequent image is established, the method further includes:

Step S21, select each frame image of the pre-order of the first image, and use the positioning information to screen the pre-order images whose distance from the first image is less than a preset distance threshold to construct a pre-order image set; Feature point matching is performed between an image and each of the preceding images in the set of preceding images to screen the preceding images matched by the first image, generate a third image matching pair, and establish the relationship between the first image and the preceding image. The matching relationship of the preceding images;

Step S22, traverse the image sequence to obtain the matching relationship between the pre-order images and/or the post-order images corresponding to each of the images, so as to construct a global matching relationship; and perform three-dimensional reconstruction on the image sequence based on the global matching relationship .

Specifically, for the first image selected in the image sequence, each frame of the previous sequence of the first image can be selected; the position information is used to calculate the Euclidean distance between the first image and the corresponding frames of the previous image. . In this case, except for the first frame of the image sequence.

Specifically, for each frame of images and all the corresponding pre-order images, the Euclidean distance between the position information of the images can be calculated first by using the positioning information, and the Euclidean distance between the calculation results of the Euclidean distance and the preset distance threshold can be used for the first screening to obtain A third image matching pair of possible matching relationships. After that, use the feature information of each image to perform a second screening on the initially matched third image matching pair, as in the above-mentioned steps S121-S124 method, use the feature information to perform feature information on each image in the third image matching step again. For matching, the corresponding matching relationship is calculated; that is, firstly, the matching pairs of feature points between two images are calculated, and then the matching pairs of feature points are screened to obtain the matching result of the image pairs. If the matching is successful and a matching relationship is established, it is considered that a loopback matching occurs. So far, the first match and the second match are completed.

For each frame of image in the image sequence, by using the consecutive multiple frames of images in the sequence after each frame of image to perform the first matching, and using each frame of images in the sequence before each frame of image to perform the second matching, it can effectively reduce the Resource consumption in the image matching process, and maximize the accuracy of the matching relationship to ensure the effectiveness of loopback matching.

In step S14, a global matching relationship is constructed based on the image matching relationship between the first image matching pair and the second image matching pair corresponding to the first image; and three-dimensional reconstruction is performed on the image sequence based on the global matching relationship.

In this exemplary embodiment, the above-mentioned methods of steps S12 and S13 may be used to traverse the image sequence, so as to obtain the matching relationship corresponding to each frame of images. Based on the matching relationships, a global matching relationship of the image sequence can be constructed, and three-dimensional reconstruction can be performed using the global matching relationship and the image sequence. For example, three-dimensional reconstruction can be performed using the SFM (Structure from motion) algorithm. Generally speaking, when using the SFM algorithm for three-dimensional reconstruction, the input can be a two-dimensional image sequence; various parameters of the camera can be inferred through the above-mentioned global matching relationship. For example, the process of the SFM algorithm may include: first extracting the focal length information from the image (needed to initialize BA), then using feature extraction algorithms such as SIFT to extract image features, and using the kd-tree model to calculate the distance between the feature points of the two images. The Euclidean distance is used to match the feature points, so as to find the image pairs with the required matching number of feature points. For each image matching pair, the epipolar geometry is calculated, the F matrix is estimated and the matching pair is refined by ransac algorithm optimization. In this way, if there are feature points that can be chained in such matching pairs and are always detected, then a trajectory can be formed. After entering the structure-from-motion part, the key first step is to select a good image pair to initialize the entire BA process. First, the first BA is performed on the two images selected for initialization, and then new images are added cyclically for new BA. Finally, the BA ends until there are no suitable images that can be added. Get camera estimation parameters and scene geometry information, i.e. sparse 3D point cloud. The bundleadjust between the two pictures uses the sba software package of the sparse beam adjustment method. The sparse beam adjustment method is a nonlinear least squares optimization objective function algorithm.

Of course, in other exemplary embodiments of the present disclosure, after the global matching relationship is constructed, other algorithms may also be used to perform three-dimensional reconstruction. For example, deep learning-based depth estimation and structure reconstruction algorithms, etc.

Based on the above content, in other exemplary embodiments of the present disclosure, when the image sequence is acquired, the above method may further include:

Step S31, analyzing the matching feature points between the images to obtain attitude information and three-dimensional coordinates of the feature points;

Step S32 , performing posture correction on the camera based on the matched posture information corresponding to the image and the three-dimensional coordinates of the feature points.

Specifically, after the global matching relationship is constructed, or, in the process of collecting the image sequence, the feature point matching between any two adjacent images in the image sequence can be performed, and the matched feature points can be analyzed. Thereby, the position and attitude information of the camera are calculated. And, solve the position coordinates of the two-dimensional feature points in the image in the three-dimensional space. Based on the information, correction information for the camera posture can be generated, so that the posture of the camera is corrected in real time before the shooting of the subsequent images, thereby ensuring that the attitude information of the subsequent shooting images is consistent.

Based on the above content, in other exemplary embodiments of the present disclosure, after the global matching relationship is constructed, the above method may further include: verifying the global matching relationship according to the acquisition time of each image, so as to delete the wrong matching relationship .

Specifically, when the monocular camera is used for RGB image acquisition, the acquisition time of each frame of image can also be marked. After the global matching relationship is obtained, for each image matching pair, each image matching pair can be checked again according to the image acquisition time, and if the image acquisition time is close, it can be judged again whether the positioning information is close. Therefore, the image matching pairs in the global image matching result can be verified according to the acquisition time and position information of the images.

The image-based three-dimensional reconstruction method provided by the embodiments of the present disclosure can be applied to an offline three-dimensional reconstruction process. It is used in indoor and outdoor positioning and navigation solutions, such as AR navigation. For scenes with unreliable image features, such as scenes with many repeated textures, illumination changes, and weak textures, the map accuracy can be greatly improved. At the same time, because the method of this scheme completely combines the positioning information of the image in the second matching process , to avoid the image mismatch, which can greatly increase the success rate of mapping. In addition, since the loop-back matched images are determined in the data collection stage, global brute force matching during image matching retrieval is avoided, and the computation time can be greatly saved. Different from the traditional method, it makes full use of other sensors to obtain the approximate global position information to directly determine the position of the loopback frame, and obtains a more accurate global image connectivity map, which provides a good input for the subsequent steps of 3D reconstruction. Moreover, the scheme is easy to operate, and can also be used as a supplement to the existing three-dimensional reconstruction scheme, and can be combined with other loop closure screening methods. Compared with other cases, it has a greater use advantage.

It should be noted that the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the chronological order of these processes. In addition, it is also readily understood that these processes may be performed, for example, synchronously or asynchronously in multiple modules.

Further, referring to FIG. 4 , the embodiment of this example also provides an image-based three-dimensional reconstruction device 40 , including: a data acquisition module 401 , a first image matching module 402 , a second image matching module 403 and a reconstruction module 404 . in,

The data collection module 401 can be used to collect image sequences and obtain positioning information of each image.

The first image matching module 402 can be configured to select the first image and successive frames of subsequent images of the first image in the image sequence, and compare the first image and each of the subsequent images. Feature point matching is performed to screen subsequent images matching the first image, a first image matching pair is generated, and a matching relationship between the first image and the subsequent image is established.

The second image matching module 403 can be configured to use the positioning information to filter a second image set matching the position of the first image in the image sequence, and compare the first image and the second image set. Feature point matching is performed on each image in the image to filter the second image matching the first image, a second image matching pair is generated, and a matching relationship between the first image and the second image is established.

The reconstruction module 404 can be configured to construct a global matching relationship based on the image matching relationship between the first image matching pair and the second image matching pair corresponding to the first image; reconstruction.

In an example of the present disclosure, the data acquisition module 401 may include: an image acquisition unit and a positioning information acquisition unit (not shown in the figure). in,

The image acquisition unit may be configured to activate the monocular camera to perform RGB image acquisition in response to the image acquisition instruction.

The positioning information acquisition unit can be used to call the UWB driver to acquire the position information when collecting the RGB image, and configure the position information as the positioning information of the RGB image

In an example of the present disclosure, the first image matching module 402 may also be configured to perform feature extraction on the first image and subsequent images to obtain two-dimensional feature points, and each of the two-dimensional feature points corresponds to The feature descriptor; the feature descriptor is used to calculate the distance between the feature points between the first image and the subsequent images, and the feature points whose distance is less than a preset threshold are established between the first image and the subsequent images. based on the first image and the subsequent image to construct a corresponding basic matrix, and use the random sampling consistency algorithm to screen the feature point matching pairs based on the basic matrix; if the screened feature points If the number of matches is greater than a preset threshold, it is determined that the first image matches the subsequent image, and a matching relationship between the first image and the subsequent image is established.

In an example of the present disclosure, the second image matching module 403 may be further configured to calculate the distance between the positioning information of the first image and the positioning information of other images in the image sequence; If the distance is smaller than the preset distance threshold, the corresponding image is added to the second image set.

In an example of the present disclosure, the apparatus 40 may further include: a third matching module (not shown in the figure).

The third matching module can be configured to select each frame image of the first sequence of the first image after generating the first image matching pair and establish the matching relationship between the first image and the subsequent image, and use the positioning method. The information filters the pre-sequence images whose distance from the first image is less than a preset distance threshold to construct a pre-sequence image set; perform feature points on each of the pre-sequence images in the first image and the pre-sequence image set. matching, to filter the pre-order images matched by the first image, generate a third image matching pair and establish a matching relationship between the first image and the pre-order image; traverse the image sequence to obtain the corresponding images of each image The matching relationship between the pre-sequence images and/or the post-sequence images to construct a global matching relationship; and three-dimensional reconstruction is performed on the image sequence based on the global matching relationship.

In an example of the present disclosure, the apparatus 40 further includes: a first verification module (not shown in the figure).

The first verification module can be used to analyze the matching feature points between the images during image acquisition or after building a global matching relationship to obtain attitude information and three-dimensional coordinates of the feature points; The attitude correction of the camera is performed according to the attitude information corresponding to the image and the three-dimensional coordinates of the feature points.

In an example of the present disclosure, the apparatus 40 may further include: a second verification module (not shown in the figure).

The second verification module may be configured to verify the global matching relationship according to the acquisition time of each image after constructing the global matching relationship, so as to delete the wrong matching relationship.

The specific details of each module in the above-mentioned image-based three-dimensional reconstruction apparatus have been described in detail in the corresponding image-based three-dimensional reconstruction method, and therefore will not be repeated here.

It should be noted that although several modules or units of the apparatus for action performance are mentioned in the above detailed description, this division is not mandatory. Indeed, according to embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into multiple modules or units to be embodied.

Figure 5 shows a schematic diagram of an electronic device suitable for implementing embodiments of the present invention.

It should be noted that the electronic device 500 shown in FIG. 5 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG. 5 , the electronic device 500 includes a central processing unit (Central Processing Unit, CPU) 501, which can be loaded into a random device according to a program stored in a read-only memory (Read-Only Memory, ROM) 502 or from a storage part 508 Various appropriate operations and processes are executed by accessing a program in a memory (Random Access Memory, RAM) 503 . In the RAM 503, various programs and data necessary for system operation are also stored. The CPU 501 , the ROM 502 , and the RAM 503 are connected to each other through a bus 504 . An Input/Output (I/O) interface 505 is also connected to the bus 504 .

The following components are connected to the I/O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc. ; a storage part 508 including a hard disk and the like; and a communication part 509 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

In particular, the processes described below with reference to the flowcharts may be implemented as computer software programs according to embodiments of the present invention. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 509 and/or installed from the removable medium 511 . When the computer program is executed by the central processing unit (CPU) 501, various functions defined in the system of the present application are executed.

Specifically, the above-mentioned electronic device may be an intelligent mobile terminal device such as a mobile phone, a tablet computer, or a notebook computer. Alternatively, the above-mentioned electronic device may also be an intelligent terminal device such as a desktop computer.

It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Erasable Programmable Read Only Memory (EPROM), flash memory, optical fiber, portable Compact Disc Read-Only Memory (CD-ROM), optical storage device, magnetic storage device, or any suitable of the above The combination. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The units involved in the embodiments of the present invention may be implemented in a software manner, or may be implemented in a hardware manner, and the described units may also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.

It should be noted that, as another aspect, the present application also provides a computer-readable medium, and the computer-readable medium may be included in the electronic device described in the foregoing embodiments; assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, causes the electronic device to implement the methods described in the following embodiments. For example, the electronic device can implement the various steps shown in FIG. 1 .

Furthermore, the above-mentioned figures are merely schematic illustrations of the processes included in the methods according to the exemplary embodiments of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the chronological order of these processes. In addition, it is also readily understood that these processes may be performed, for example, synchronously or asynchronously in multiple modules.

Furthermore, the above-mentioned figures are merely schematic illustrations of the processes included in the methods according to the exemplary embodiments of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the chronological order of these processes. In addition, it is also readily understood that these processes may be performed, for example, synchronously or asynchronously in multiple modules.

Other embodiments of the present disclosure will readily suggest themselves to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. An image-based three-dimensional reconstruction method, comprising:

acquiring an image sequence and acquiring positioning information of each image;

selecting a first image and continuous multi-frame subsequent images subsequent to the first image in the image sequence, performing feature point matching on the first image and each subsequent image to screen subsequent images matched with the first image, generating a first image matching pair and establishing a matching relation between the first image and the subsequent images; and

screening a second image set matched with the first image in position in the image sequence by using the positioning information, carrying out feature point matching on each image in the first image and the second image set to screen a second image matched with the first image, generating a second image matching pair and establishing a matching relation between the first image and the second image;

constructing a global matching relation based on the image matching relation of a first image matching pair and a second image matching pair corresponding to the first image; and performing three-dimensional reconstruction on the image sequence based on the global matching relation.

2. The image-based three-dimensional reconstruction method according to claim 1, wherein the acquiring the image sequence and obtaining the positioning information of each image comprises:

responding to an image acquisition instruction, and activating a monocular camera to acquire RGB images; and

and calling an ultra-wideband driving program to acquire the position information when the RGB image is acquired, and configuring the position information as the positioning information of the RGB image.

3. The image-based three-dimensional reconstruction method according to claim 1, wherein the performing feature point matching on the first image and the subsequent images to screen the subsequent images matched with the first image and establish a matching relationship between the first image and the subsequent images comprises:

performing feature extraction on the first image and the subsequent images to obtain two-dimensional feature points and feature descriptors corresponding to the two-dimensional feature points;

calculating the distance of the feature points between the first image and the subsequent image by using the feature descriptors, and establishing a feature point matching pair between the first image and the subsequent image by using the feature points of which the distance is smaller than a preset threshold value;

constructing a corresponding basic matrix based on the first image and the subsequent image, and screening the feature point matching pairs by utilizing a random sampling consistency algorithm based on the basic matrix;

and if the matching quantity of the screened feature points is greater than a preset threshold value, judging that the first image is matched with the subsequent image and establishing a matching relation between the first image and the subsequent image.

4. The method of claim 1, wherein the using the localization information to filter the second set of images in the sequence of images that match the first set of images includes:

respectively calculating distances between the positioning information of the first image and the positioning information of other images in the image sequence;

and if the distance is smaller than a preset distance threshold value, adding the corresponding image to the second image set.

5. The method according to claim 1 or 3, wherein after generating the first image matching pair and establishing the matching relationship between the first image and the subsequent image, the method further comprises:

selecting each frame of image of the first image preamble, and screening the preamble image with the distance between the frame of image and the first image smaller than a preset distance threshold value by utilizing positioning information to construct a preamble image set; performing feature point matching on the first image and each pre-order image in the pre-order image set to screen pre-order images matched with the first image, generating a third image matching pair and establishing a matching relation between the first image and the pre-order images;

traversing the image sequence to obtain a matching relation of a preamble image and/or a subsequent image corresponding to each image so as to construct a global matching relation; and performing three-dimensional reconstruction on the image sequence based on the global matching relation.

6. The method of claim 1, wherein the acquiring of the sequence of images further comprises:

analyzing the matched feature points between the images to obtain attitude information and three-dimensional coordinates of the feature points;

and carrying out attitude correction on the camera based on the matched attitude information corresponding to the image and the three-dimensional coordinates of the characteristic points.

7. The method of claim 1, wherein after the constructing the global matching relationship, the method further comprises:

and verifying the global matching relationship according to the acquisition time of each image so as to delete the wrong matching relationship.

8. An apparatus for image-based three-dimensional reconstruction, comprising:

the data acquisition module is used for acquiring an image sequence and acquiring positioning information of each image;

the first image matching module is used for selecting a first image and continuous multi-frame subsequent images subsequent to the first image in the image sequence, performing feature point matching on the first image and each subsequent image to screen the subsequent images matched with the first image, generating a first image matching pair and establishing a matching relation between the first image and the subsequent images; and

the second image matching module is used for screening a second image set matched with the first image in position in the image sequence by using the positioning information, carrying out feature point matching on each image in the first image set and the second image set so as to screen a second image matched with the first image, generating a second image matching pair and establishing a matching relation between the first image and the second image;

the reconstruction module is used for constructing a global matching relationship based on the image matching relationship of a first image matching pair and a second image matching pair corresponding to the first image; and performing three-dimensional reconstruction on the image sequence based on the global matching relation.

9. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method for image-based three-dimensional reconstruction according to one of claims 1 to 7.

10. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method of image-based three-dimensional reconstruction according to any one of claims 1 to 7.

Images