JP2022178000A - Imaging device and method - Google Patents

Abstract

A three-dimensional image can be easily generated from a captured two-dimensional image. A feature point extracting unit (103) extracts each of a previous image stored in a storage unit (102) by imaging an object (151) with a digital camera (101) and a current image (151) of the object (151) being imaged by the digital camera (101). Extract feature points. The comparison processing unit 104 compares each feature point extracted by the feature point extraction unit 103 to obtain a match state between the previous image and the current image. The display unit 105 displays the matching state obtained by the comparison processing unit 104 . [Selection diagram] Fig. 1

Description

The present invention relates to an imaging device and method for obtaining three-dimensional images.

In recent years, from the perspective of promoting digital transformation (DX), various structures and spatial information are acquired as digital information, and spaces and objects that reproduce the real thing in virtual space are generated to simulate the real world. Various services and services using virtual reality (VR) technology for cognition and perception, and augmented reality (AR) technology for augmenting visual information by superimposing spaces and objects that reproduce real objects in real space. There is a growing movement to use it to improve productivity and quality at production sites.

For example, in improving productivity by BIM (Building Information Modeling) and CIM (Construction Information Modeling/Management) promoted by the government, 3D measurement of current topography, structures, facilities (outdoor/indoor), etc. is indispensable. can't In realizing these, it is important to acquire detailed three-dimensional shapes of objects and positional information (including relative positional relationships between objects).

A simple method is to prepare two-dimensional images obtained by capturing images from multiple viewpoints using a monocular camera or a stereo camera, extract feature points from each image, and connect similar feature points in other images. Therefore, there is a technique for synthesizing three-dimensional information (three-dimensional image) (Patent Document 1). Further, there is known a technique for improving the accuracy of three-dimensional information by adding information from a sensor called a depth sensor that measures the distance between a viewpoint and an object to the two-dimensional image described above (Patent Document 2).

As the above-mentioned two-dimensional image capturing equipment, depth cameras equipped with depth sensors, and capturing equipment using laser distance measurement sensors (Light Detection and Ranging, Laser Imaging Detection and Ranging: Lidar) are known. ing.

JP 2020-159965 A JP 2020-135679 A

In the above-described technique, a three-dimensional image is obtained by photographing an object from a plurality of locations, performing image processing on the photographed image data (two-dimensional images), and reconstructing the data. In reconstructing photographed image information into a three-dimensional image, it is common that the more precise the three-dimensional information is to be obtained, the greater the amount of computational processing required. A certain amount of processing time is required until the data becomes usable.

If the three-dimensional information reconstructed in this way is found to be disturbed or lost in spatial information due to an abnormal photographing, the above-described processing must be performed again, which is time-consuming. In particular, when an operator takes an image while holding the imaging equipment in his or her hand, there is a tendency for the operator to retake an image due to an abnormality in the image, and the above-described problem becomes more pronounced. As described above, conventionally, there has been a problem that it is not easy to generate a three-dimensional image from a photographed two-dimensional image.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and an object of the present invention is to make it possible to more easily generate a three-dimensional image from a photographed two-dimensional image.

An imaging apparatus according to the present invention includes a digital camera, a storage section for storing an image captured by the digital camera, a previous image stored in the storage section by capturing an image of an object by the digital camera, and the object by the digital camera. A feature point extraction unit that extracts each feature point of the current image that is being captured, and a comparison processing unit that compares each feature point extracted by the feature point extraction unit to determine the state of matching between the previous image and the current image. , and a display unit for displaying the matching state obtained by the comparison processing unit.

1. In one configuration example of the imaging apparatus, the digital camera has a depth sensor, and the image captured by the digital camera has depth information.

In one configuration example of the imaging apparatus described above, the comparison processing unit obtains a matching state by comparing descriptors of respective feature points.

The configuration example of the imaging apparatus further includes a storage processing unit that stores the current image in the storage unit when the matching state obtained by the comparison processing unit exceeds a set standard.

One configuration example of the imaging apparatus described above includes an image generation unit that generates a three-dimensional image of the object from a plurality of images stored in the storage unit by imaging the object with a digital camera.

Further, an image capturing method according to the present invention includes a first step of storing an image of an object captured by a digital camera in a storage unit; a second step of extracting each feature point of the current image in which the and a fourth step of displaying the status of the determined match.

In one configuration of the imaging method described above, the image captured by the digital camera comprises depth information, and the third step is to determine a match by comparing the descriptors of each feature point.

The configuration example of the imaging method further includes a fifth step of storing the current image in the storage unit when the matching state obtained in the third step exceeds a set standard.
One configuration example of the imaging method includes a sixth step of generating a three-dimensional image of the object from a plurality of images stored in the storage unit by imaging the object with a digital camera.

As described above, according to the present invention, the feature points extracted in each of the previous image and the current image are compared to determine the matching state between the previous image and the current image. Three-dimensional images can be generated more easily.

FIG. 1 is a configuration diagram showing the configuration of an imaging device according to an embodiment of the present invention. FIG. 2 is a flow chart explaining an imaging method according to an embodiment of the present invention. FIG. 3 is an explanatory diagram for explaining a state in which feature points are extracted from an image. FIG. 4 is an explanatory diagram for explaining comparison of corresponding feature points between images. FIG. 5 is a configuration diagram showing the hardware configuration of the imaging device according to the embodiment of the present invention.

An imaging apparatus according to an embodiment of the present invention will be described below with reference to FIG. This imaging apparatus includes a digital camera 101 , a storage unit 102 , a feature point extraction unit 103 , a comparison processing unit 104 and a display unit 105 .

Digital camera 101 can be, for example, a stereo camera with a depth sensor. The depth sensor can be an infrared-based, laser-based, or other sensor that provides depth information. The digital camera 101 may be monocular, but preferably a stereo camera. An image captured by the digital camera 101 having a depth sensor includes depth information.

The storage unit 102 stores images captured (captured) by the digital camera 101 . For example, first, the lens of the activated digital camera 101 is aimed at the object 151, and the object 151 is set to be within the imaging range using the liquid crystal monitor of the digital camera 101 or the like. An image of a subject received by the imaging element of the digital camera 101 is displayed on the liquid crystal monitor. By pressing the shutter button of the digital camera 101 in this state, the image displayed on the liquid crystal monitor is stored in the storage unit 102 .

The feature point extracting unit 103 extracts feature points of each of the previous image stored in the storage unit 102 by imaging the object 151 with the digital camera 101 and the current image of the object 151 being imaged by the digital camera 101. do. In extracting feature points, a descriptor is obtained for each extracted feature point.

The comparison processing unit 104 compares each feature point extracted by the feature point extraction unit 103 to obtain a matching state between the previous image and the current image. The comparison processing unit 104 compares the descriptors of each feature point to obtain a matching state. The display unit 105 displays the matching state obtained by the comparison processing unit 104 .

The imaging apparatus also includes a storage processing unit 106 that stores the current image in the storage unit 102 when the matching state obtained by the comparison processing unit 104 exceeds a set standard. This imaging apparatus also includes an image generation unit 107 that generates a three-dimensional image of the object 151 from a plurality of images stored in the storage unit 102 by imaging the object 151 with the digital camera 101 .

Next, an imaging method according to an embodiment of the present invention will be described with reference to FIG. First, in step S101, an image (previous image) of the object 151 captured by the digital camera 101 is stored in the storage unit 102 (first step). As described above, the lens of the digital camera 101 is directed toward the object 151, and the object 151 is set in the imaging range using the liquid crystal monitor of the digital camera 101. Then, the shutter button of the digital camera 101 is pressed, and the liquid crystal monitor , is stored in the storage unit 102 .

Next, in step S102, the current image is captured. The lens of the digital camera 101 is directed toward the target object 151 in a direction different from the direction in which the previous image was captured, and the target object 151 is set in the imaging range using the liquid crystal monitor of the digital camera 101 or the like.

Next, in step S103, the feature point extracting unit 103 extracts feature points of each of the previous image stored in the storage unit 102 and the current image of the object 151 captured by the digital camera 101 (first image). 2 steps).

Next, in step S104, the comparison processing unit 104 compares the feature points extracted by the feature point extraction unit 103 to determine the matching state between the previous image and the current image (third step). The images captured by the digital camera 101 are provided with depth information, where the descriptors of each feature point are compared to determine a match. Image features such as SIFT (Scale Invariant Feature Transform) features and SURF (Speeded Up Robust Features) features can be used to detect feature points and feature amounts in each image. Also, the present invention is not limited to these, and it is also possible to carry out a matching search in a brute-force manner based on the descriptor of the feature amount of each feature point. For example, Brute-Force match is known.

Next, the matching state (result) obtained in step S105 is displayed on the display unit 105 (fourth step).

For example, as shown in FIG. 3, in each of the previous image (a) and the current image (b), the descriptors of the extracted feature points are compared and matched (the fields of view are different, but they may be the same place). high) multiple selections. Matching candidates are circled in FIG.

As a display of the matching state, a frame reflecting the relative positional relationship calculated from the parallax and the focal length of the camera for each of the matching points in the current image is displayed so as to be superimposed on the previous image (Fig. 4). . In FIG. 4, the previous image is shown in a solid-line frame, and the current image is shown only in a dotted-line frame as described above.

Dotted-line frames are generated for each of the candidates with matching feature points indicated by circles shown in FIG. If the positions of each of the plurality of feature points are approximately the same, the plurality of dotted line frames are displayed so as to overlap each other, as shown in FIG. 4(a). On the other hand, when the positions of the plurality of feature points do not match, as shown in FIG. 4B, the plurality of dotted line frames are displayed at different positions without overlapping.

If a plurality of dotted-line frames are displayed in such a manner that they overlap each other, it can be determined that the current image has more than a certain number of feature points corresponding to the previous image (the degree of matching is high). On the other hand, if a plurality of dotted-line frames are displayed at different positions without overlapping, it can be determined that the current image does not have a large number of feature points corresponding to the previous image (the degree of matching is low). .

Also, if the overlapping displayed dotted-line frames are displaced from the position of the previous image indicated by the solid-line frame, it can be understood that the previous image and the current image are separated by a certain distance. If the previous image and the current image have the above-described relationship, it can be determined that they are in a data state that can be utilized as a three-dimensional image. Here, it has been found that if the amount of movement of the camera for each imaging exceeds about 1/10 of the field of view of the digital camera 101, the above-described state of high degree of matching cannot be obtained.

Note that the color of the dotted line frame can be changed according to the matching degree of the corresponding points. For example, if the degree of matching is greater than or equal to a certain number, it can be displayed in blue, and if it is less than the certain number, it can be displayed in red. Also, at this time, the state of camera shake is measured by the camera shake correction function of the digital camera 101, and the result of this measurement can be displayed on the display unit 105 to indicate that the camera shake does not exceed a standard level.

Next, in step S106, when the above-described matching state exceeds the set standard (yes in step S106), in step S107, the storage processing unit 106 stores the current image in the storage unit 102 ( 5th step). As a state of matching, it is also possible to make it a condition that the above-described camera shake does not occur beyond the reference level.

When the above-described steps S102 to S107 are performed until the number of images stored in the storage unit 102 reaches the set number of images (yes in step S108), the image generation unit 107 moves the target object 151 in step S109. is captured by the digital camera 101 to generate a three-dimensional image of the object 151 from a plurality of images stored in the storage unit 102 (sixth step). As described above, the three-dimensional images generated in this way guarantee the degree of matching between the images, so that the probability of obtaining detailed three-dimensional spatial information is greatly improved.

It is also possible to search for matching corresponding points relative to two-dimensional image information at the time of imaging, and add depth information (distance information) at the time of three-dimensional spatial reconstruction after imaging.

As shown in FIG. 5, the storage unit, feature point extraction unit, comparison processing unit, and display unit of the image capturing apparatus according to the above-described embodiment are composed of a CPU (Central Processing Unit) 301 and a main memory. A computer device having a device 302, an external storage device 303, a network connection device 304, and the like, and the CPU 301 operates (executes the program) according to a program developed in the main storage device 302, thereby performing each function (imaging method) can also be realized. The program is a program for a computer to execute the imaging method shown in the above embodiment. A network connection device 304 connects to a network 305 . Also, functions may be distributed among multiple computing devices.

As described above, according to the present invention, extracted feature points are compared in each of the previous image and the current image to determine the state of matching between the previous image and the current image. 3D images can be generated more easily.

According to the present invention, since feature point matching is performed between the previous image and the current image at the time of imaging, it is possible to greatly reduce the possibility of discovering problems when reconstructing three-dimensional information after imaging. Therefore, useless work such as re-imaging can be reduced. In addition, by realizing imaging with imaging quality exceeding a certain level, it is possible to obtain precise three-dimensional information without using an expensive stationary imaging apparatus.

It should be noted that the present invention is not limited to the embodiments described above, and many modifications and combinations can be implemented by those skilled in the art within the technical concept of the present invention. It is clear.

DESCRIPTION OF SYMBOLS 101... Digital camera, 102... Storage part, 103... Feature point extraction part, 104... Comparison process part, 105... Display part, 106... Storage process part, 107... Image generation part, 151... Object.

Claims (9)

a digital camera and
a storage unit that stores images captured by the digital camera;
a feature point extracting unit for extracting feature points of each of a previous image stored in the storage unit and a current image of the target object captured by the digital camera;
a comparison processing unit that compares each of the feature points extracted by the feature point extraction unit to obtain a matching state between the previous image and the current image;
an imaging device comprising: a display section for displaying the matching state obtained by the comparison processing section. The imaging device according to claim 1, wherein
An imaging apparatus, wherein the digital camera includes a depth sensor, and an image captured by the digital camera includes depth information. The imaging device according to claim 1 or 2,
The image pickup apparatus, wherein the comparison processing unit obtains a matching state by comparing descriptors of respective feature points. In the imaging device according to any one of claims 1 to 3,
The imaging apparatus, further comprising a storage processing unit that stores the current image in the storage unit when the matching state obtained by the comparison processing unit exceeds a set standard. In the imaging device according to any one of claims 1 to 4,
An imaging apparatus, comprising: an image generation unit that generates a three-dimensional image of the object from a plurality of images stored in the storage unit by imaging the object with the digital camera. a first step of storing an image of an object captured by a digital camera in a storage unit;
a second step of extracting feature points of each of the previous image stored in the storage unit and the current image of the object captured by the digital camera;
a third step of obtaining a matching state between the previous image and the current image by comparing each feature point extracted in the second step;
and a fourth step of displaying the matching state determined in the third step. In the imaging method according to claim 6,
An image capturing method, wherein the image captured by the digital camera comprises depth information, and wherein the third step determines a match state by comparing descriptors of each feature point. In the imaging method according to claim 6 or 7,
The imaging method, further comprising: a fifth step of storing the current image in the storage unit when the matching state obtained in the third step exceeds a set criterion. In the imaging method according to any one of claims 6 to 8,
An imaging method, comprising: a sixth step of generating a three-dimensional image of the object from a plurality of images stored in the storage unit by imaging the object with the digital camera.

Images