JP7710185B2 - Photographing device, photographing operation support method, and photographing operation support program - Google Patents

Description

The present invention relates to an imaging device that captures images of each point of a measurement target location in order to perform three-dimensional measurement processing that generates three-dimensional spatial information of the measurement target location based on images captured at each point of the measurement target location, as well as a method and program for supporting a user in the imaging work using the imaging device.

A three-dimensional measurement technique is known that generates three-dimensional spatial information (map data) about a measurement target location based on images of the measurement target location. In recent years, the SLAM (Simultaneous Localization And Mapping) method has been attracting attention for this type of three-dimensional measurement. In the SLAM method, a camera device is held on a moving object, and based on images of each point in the measurement target location, three-dimensional spatial information and the vehicle's position information can be generated as measurement results. In particular, a user can easily perform three-dimensional measurement by taking images while moving around the measurement target location using a handheld camera device, thereby acquiring images of each point in the measurement target location.

On the other hand, in 3D measurements using SLAM, relative self-location estimation is repeated, and so accumulation of errors is unavoidable. Such accumulation of errors in self-location estimation causes a significant decrease in the accuracy of the 3D measurement results.

As a method for improving the accuracy degradation caused by such accumulated errors, a technology has been known that, if an error occurs during the creation of three-dimensional spatial information (map data), allows the creation of the three-dimensional spatial information to be restarted from the middle, without having to start over from the beginning (see Patent Document 1). This technology also detects the event that caused the error (lost) and presents the user with the period during which the event occurred, allowing the user to know the position where the creation of the three-dimensional spatial information should be restarted.

Furthermore, a loop-closing technique has been known as a method for improving accuracy degradation due to accumulated errors (see Non-Patent Document 1). With this technique, when it is detected that the aircraft's travel path forms a loop, i.e., the aircraft has returned to a location where it was previously photographed, the self-position estimation result (past position) acquired during the previous photographing at that location is regarded as the correct position, and the position on the path from the current position to the past position is corrected, thereby eliminating the accumulated error. With this technique, it is possible to eliminate the trouble of restarting photographing midway, as with the technique disclosed in Patent Document 1.

Patent No. 6639734

SSII2015 Tutorial: Incremental 3D reconstruction from video images using feature point tracking and its applications (lecture given at the Image Sensing Symposium on June 10, 2015)

In 3D measurements using SLAM, loop closing technology is effective in improving accuracy degradation due to accumulated errors, as disclosed in Patent Document 2. Meanwhile, loop closing processing can be performed when the movement path forms a loop, that is, when returning to a point where a previous image was captured. For this reason, it is important to intentionally create a state in which loop closing processing can be performed. It is therefore desirable to clearly present the need for loop closing processing to users who are currently taking images. However, conventional technology does not take such demands into consideration at all.

The main objective of the present invention is to provide an imaging device, imaging operation support method, and imaging operation support program that can improve the accuracy of 3D measurements using SLAM by clearly presenting the need for loop closing processing to a user during imaging work and encouraging the user to perform appropriate imaging work.

The photographing device of the present invention is a photographing device that photographs each point of a measurement target location in order to perform a three-dimensional measurement process that generates three-dimensional spatial information of the measurement target location based on the photographed images of each point of the measurement target location, and is equipped with a device main body held by a user, a photographing unit provided in the device main body that photographs the measurement target location, a display unit that displays support information related to the photographing work in which the user holds the device main body and moves around the measurement target location while causing the photographing unit to sequentially photograph each point of the measurement target location, and a processor that controls the photographing unit and the display unit, and the processor is configured to display, as the support information, a support image on the display unit that visualizes the area in which loop closing processing has already been performed in the three-dimensional measurement processing.

The photography support method of the present invention is a method for supporting photography by a processor that supports a user in photographing a location to be measured by moving around the location to be measured while holding a photography device and having the photography device sequentially photograph each point of the location to be measured, in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured based on images captured at each point of the location to be measured, and is configured to display, on a display unit, a support image that visualizes the area in which loop closing processing has already been performed in the three-dimensional measurement processing, as support information for the photography work.

The photography support program of the present invention is a photography support program that causes a processor to execute a process to support a user's photography work by moving around the measurement target location while holding a photography device and having the photography device sequentially photograph each point of the measurement target location in order to perform a three-dimensional measurement process that generates three-dimensional spatial information of the measurement target location based on images captured at each point of the measurement target location, and is configured to display, on the display unit, a support image that visualizes the area in which loop closing processing has been performed in the three-dimensional measurement processing as support information for the photography work.

According to the present invention, the need for loop closing processing is clearly indicated to a user (operator) performing photography work using support images, and the user is urged to perform appropriate photography work. This allows the user to perform appropriate photography work in response to the support images, thereby improving the accuracy of 3D measurement processing using SLAM.

FIG. 1 is an explanatory diagram showing a situation of a photographing operation using the photographing device according to the present embodiment; Plan view showing the measurement location FIG. 13 is an explanatory diagram showing a support image in which the area already implemented and the area recommended for implementation are visualized on an overhead point cloud image. An explanatory diagram showing a support image in which the completed area and the recommended area are visualized on an overhead point cloud route image. FIG. 13 is an explanatory diagram showing a support image in which the completed area and the recommended area are visualized on an overhead route image. FIG. 13 is an explanatory diagram showing a support image in which the area already treated and the area recommended for treatment are visualized on a captured image. FIG. 1 is a block diagram showing a schematic configuration of an imaging device. FIG. 13 is an explanatory diagram showing the shooting screen displayed on the display. FIG. 13 is an explanatory diagram showing the shooting screen displayed on the display.

The first invention made to solve the above problem is an imaging device that images each point of a measurement target location in order to perform a three-dimensional measurement process that generates three-dimensional spatial information of the measurement target location based on the captured images of each point of the measurement target location, and includes a device main body held by a user, an imaging unit provided in the device main body that images the measurement target location, a display unit that displays support information related to the imaging work in which the user holds the device main body and moves around the measurement target location while causing the imaging unit to sequentially capture each point of the measurement target location, and a processor that controls the imaging unit and the display unit, and the processor is configured to display, as the support information, an support image on the display unit that visualizes the area in which loop closing processing has already been performed in the three-dimensional measurement processing.

This allows the need for loop closing processing to be clearly indicated to the user (worker) during photography work using support images, encouraging the user to perform appropriate photography work. This allows the user to perform appropriate photography work in accordance with the support images, thereby improving the accuracy of 3D measurement processing using SLAM. Note that the range in which loop closing processing has been performed may be visualized by area or by route.

In addition, the second invention is configured such that the processor displays, on the display unit, the support image that visualizes the range in which the loop closing process has been performed in at least one of an overhead image generated from the measurement results of the three-dimensional measurement process and the current captured image.

This makes it possible to present to the user the extent to which the loop closing process has been performed in an easy-to-understand manner. In this case, for example, the extent to which the loop closing process has been performed may be visualized by coloring. Furthermore, the overhead image may be an overhead point cloud image, an overhead point cloud route image, or an overhead route image.

In addition, the third invention is configured such that the processor visualizes in the support image the range in which the loop closing process has been performed in a display mode different from the range in which the loop closing process is recommended to be performed.

This allows the range in which loop closing processing has been performed to be presented to the user in an easy-to-understand manner. In this case, for example, the range in which loop closing processing has been performed may be visualized in a color different from the range in which loop closing processing is recommended.

In addition, the fourth invention is configured such that the processor acquires the number of times the loop closing process has been performed within the range in which it has already been performed.

This makes it possible to present to the user the number of times the loop closing process has been performed within the range in which it has already been performed. In this case, it is possible to obtain the number of times it has been performed for each shooting location (captured image). It is also possible to obtain the number of times it has been performed for each point of the point cloud data as three-dimensional spatial information.

In a fifth aspect of the present invention, the processor is configured to change the display mode of the range in the support image where the loop closing process has been performed, depending on the number of times the process has been performed.

This makes it possible to clearly present to the user the number of times the loop closing process has been performed within the range in which it has already been performed. In this case, for example, the color may be changed as a change in the display mode. For example, a range in which it has been performed once is drawn in green, a range in which it has been performed twice is drawn in blue, and a range in which it has been performed zero times is drawn in red.

The sixth invention is a method for supporting photography by a processor that supports a user in photographing a location to be measured by moving around the location to be measured while holding a photography device and having the photography device photograph each point of the location to be measured in sequence, in order to perform three-dimensional measurement processing to generate three-dimensional spatial information of the location to be measured based on images taken of each point of the location to be measured, and is configured to display, on a display unit, a support image that visualizes the area in which loop closing processing has already been performed in the three-dimensional measurement processing, as support information for the photography task.

As with the first invention, this makes it possible to improve the accuracy of 3D measurements using SLAM by clearly indicating the need for loop closing processing to a user who is taking photographs and encouraging the user to take appropriate photographs.

The seventh invention is a photography operation support program that causes a processor to execute a process to support a user's photography operation by moving within the measurement target location while holding a photography device and having the photography device sequentially photograph each point of the measurement target location in order to perform a three-dimensional measurement process that generates three-dimensional spatial information of the measurement target location based on photographed images of each point of the measurement target location, and is configured to display, on a display unit, a support image that visualizes the area in which loop closing processing has been performed in the three-dimensional measurement process as support information related to the photography operation.

As with the first invention, this makes it possible to improve the accuracy of 3D measurements using SLAM by clearly indicating the need for loop closing processing to a user who is taking photographs and encouraging the user to take appropriate photographs.

The following describes an embodiment of the present invention with reference to the drawings.

Figure 1 is an explanatory diagram showing the situation of a user performing a photographing operation using the photographing device 1 according to this embodiment. Figure 2 is a plan view showing the location to be measured.

The photographing device 1 includes a device body 11 and a sensor unit 12. The sensor unit 12 includes a visible camera 21 (photographing unit). The visible camera 21 is a monocular camera that detects visible light to photograph a subject, and outputs a photographed image, for example, a color image in the RGB format. The photographing device 1 can be configured as a tablet terminal or a notebook PC.

The user (worker) walks through the location to be measured while holding the device body 11 of the imaging device 1. At this time, the imaging device 1 captures the location to be measured with the visible camera 21, and sequentially acquires images of each point in the location to be measured.

In addition, in this embodiment, the imaging device 1 is a three-dimensional measurement device. That is, in the imaging device 1, three-dimensional measurement processing is performed based on the captured images of each point acquired in sequence by the visible camera 21, and three-dimensional spatial information regarding the measurement target location is generated. In the three-dimensional measurement processing, the SLAM method is used to generate point cloud data (environmental map) as three-dimensional spatial information regarding the measurement target location. At this time, in addition to generating the point cloud data, a self-position estimation process is performed, and the self-position for each time, i.e., the position of the imaging point, is estimated.

In addition, when the image capture device 1 detects that its own travel path forms a loop, i.e., that its own device has returned to a location where it previously captured an image, it performs a loop closing process. In the loop closing process, for a previously captured location, the self-location estimation result (past location) acquired during the previous image capture is regarded as the correct location, and the location of each image capture location on the path from the current location to the past location is corrected.

In the example shown in Figure 2, shooting begins from the shooting start point, goes around a part of the area of the measurement target location, and returns to the shooting start point. Shooting then continues, and another area of the measurement target location is photographed. In this case, a loop closing process is performed when returning to the shooting start point. This improves the accuracy of the point cloud data and self-location estimation results for each shooting point included in the loop.

Next, we will explain a support image in which the completed range and recommended range are visualized on an overhead point cloud image. Figure 3 is an explanatory diagram showing a support image in which the completed range and recommended range are visualized on an overhead point cloud image.

In this embodiment, a support image is generated in which the completed range and the recommended range are visualized on the overhead point cloud image, and the support image is presented to the user during the shooting work. The overhead point cloud image is an image (rendering) of each point of the point cloud data as seen from an aerial viewpoint. The completed range represents the range in which loop closing processing has been performed. The recommended range represents the range in which loop closing processing is recommended, i.e., the range in which loop closing processing has not been performed, or the range in which the accuracy of the 3D measurement results is low.

The completed area (range where the action has been taken) and the recommended area (range where the action is recommended) are drawn in different colors. For example, the completed area is drawn in blue, and the recommended area is drawn in red. Specifically, in the overhead point cloud image, each point included in the completed area is drawn in blue, and each point included in the recommended area is drawn in red.

At this time, each point drawn at a predetermined size in the overhead point cloud image is colored in a predetermined color. Also, a sphere of a predetermined diameter centered on each point may be drawn in a predetermined color. Also, the target space may be divided into cubic voxels, and the voxels containing each point may be drawn in a predetermined color.

By viewing the support image, the user can understand the area where photography is required for loop closing processing and then perform the photography work. Specifically, the user performs photography work targeting the recommended implementation area drawn in red. In other words, the user performs photography work so as to pass through the recommended implementation area. Note that the specific content of the photography work, i.e., the travel route, etc., can be appropriately determined by the user depending on the situation at the site, etc.

In this manner, in this embodiment, the support image can be used to clearly present to the user the necessity of loop closing processing. Furthermore, the range in which photographing work is required for loop closing processing can be clearly presented to the user. This can encourage the user to perform appropriate photographing work, and the loop closing processing can be reliably performed, thereby improving the accuracy of the 3D measurement processing.

Note that both the completed area and the recommended area represent the range within the measurement location where photography has been performed. In other words, areas within the measurement location that are not included in either the completed area or the recommended area represent ranges within which photography has not been performed. Therefore, by viewing the support image, the user can also check the ranges within which photography has not been performed.

In addition, in this embodiment, the number of times the loop closing process is performed is counted, and the area is visualized (color-coded) according to the number of times the loop closing process is performed. For example, an area where it has been performed once is drawn in green, an area where it has been performed twice is drawn in blue, and an area where it has been performed zero times is drawn in red.

Next, we will explain a support image in which the completed range and recommended range are visualized on an overhead point cloud route image. Figure 4 is an explanatory diagram showing a support image in which the completed range and recommended range are visualized on an overhead point cloud route image.

In this embodiment, a support image is generated in which the completed range and recommended range are visualized on an overhead point cloud route image, and the support image is presented to the user during shooting work. The overhead point cloud route image is an overhead point cloud image on which a route (movement trajectory) is superimposed. The overhead point cloud image is an image (rendering) of each point of the point cloud data as seen from a viewpoint in the sky. The route is the self-position estimation result based on each captured image, i.e., the shooting points of each capture, drawn on the overhead point cloud image as lines connecting them.

The completed route (area that has been completed) and the recommended route (area that is recommended) are drawn in different colors. For example, the completed route is drawn in blue, and the recommended route is drawn in red.

In addition, a shooting location mark is drawn on the support image. The shooting location mark indicates the shooting location and shooting direction. Specifically, the shooting location mark is drawn as an isosceles triangle, with the apex of the isosceles triangle indicating the shooting location, and the direction from the apex to the base indicating the shooting direction.

By viewing the support image, the user can understand the area where photography is required for the loop closing process and then perform the photography work. Specifically, the user performs the photography work by targeting the recommended implementation route drawn in red. In other words, the user performs the photography work so as to form a loop that includes the recommended implementation route.

In addition, in this embodiment, the number of times the loop closing process is performed is counted, and the route is visualized (color-coded) according to the number of times the loop closing process is performed. For example, a route that has been performed once is drawn in green, a route that has been performed twice is drawn in blue, and a route that has been performed zero times is drawn in red.

Next, we will explain the support image in which the completed range and the recommended range are visualized on the overhead route image. Figure 5 is an explanatory diagram showing the support image in which the completed range and the recommended range are visualized on the overhead route image.

In this embodiment, a support image is generated in which the completed range and the recommended range are visualized on the overhead route image, and the support image is presented to the user during the shooting work. The overhead route image is obtained by removing the overhead point cloud image from the overhead point cloud route image (see FIG. 4), and is a drawing of the self-position estimation results based on the images captured each time, i.e., the lines connecting the shooting points for each time, as viewed from an aerial viewpoint.

Next, we will explain the support image in which the implemented range and the recommended implementation range are visualized on the captured image. Figure 6 is an explanatory diagram showing the support image in which the implemented range and the recommended implementation range are visualized on the captured image.

In this embodiment, a support image is generated in which the already implemented range and the recommended implementation range are visualized on the captured image, and the support image is presented to the user during the shooting work. The captured image is the current captured image, that is, the image captured in real time output from the visible camera 21.

The completed area (range where the procedure has been completed) and the recommended area (range where the procedure is recommended) are drawn in different colors. For example, the completed area is drawn in blue, and the recommended area is drawn in red. Specifically, an image showing the completed area drawn in a specified color and an image showing the recommended area drawn in a specified color are superimposed in a semi-transparent state on the captured image.

In this manner, in this embodiment, the implemented range and the recommended implementation range (unimplemented range) are visualized on the captured image. This allows the user to easily determine whether the area in front of them is an implemented range or a recommended implementation range. The user then performs the capture operation so that the recommended implementation range is included.

Next, we will explain the general configuration of the imaging device 1. Figure 7 is a block diagram showing the general configuration of the imaging device 1.

In addition to the sensor unit 12, the imaging device 1 includes a display 13 (display unit), an input device 14, a memory 15, and a processor 16 (CPU).

In addition to the visible camera 21, the sensor unit 12 also includes a depth camera 22 and an IMU 23 (Inertial Measurement Unit). The depth camera 22 is a stereo camera that detects infrared light to capture an image of a subject, and outputs depth information (distance image) as the detection result. The distance to the subject can be measured based on the detection result of the depth camera 22. The IMU 23 detects three-dimensional angular velocity and acceleration. The amount of movement and rotation of the imaging device 1 can be measured based on the detection result of the IMU 23. Note that the visible camera 21, the depth camera 22, and the IMU 23 may not be integrated into a sensor unit. Also, the depth camera 22 and the IMU 23 may be omitted, and only the visible camera 21 may be provided.

The display 13 presents various information related to the shooting work to the user, and displays the shooting screen 101 (see Figures 8 and 9) and the like. The input device 14 is used by the user to perform input operations. The input device 14 may be a keyboard, a mouse, a touch pad, a touch panel, or the like. Note that when the shooting device 1 is configured as a tablet terminal, a touch panel display is provided in which the touch panel as the input device 14 and the display panel as the display 13 are integrated.

The memory 15 stores programs executed by the processor 16. The memory 15 also stores images captured by the visible camera 21 and the detection results of the depth camera 22 and the IMU 23. The memory 15 also stores measurement results generated by the processor 16.

The processor 16 performs various processes by executing the programs stored in the memory 15. In this embodiment, the processor 16 performs detection information acquisition process, point cloud generation process, first support image generation process, second support image generation process, screen control process, message notification process, and the like.

In the detection information acquisition process, the processor 16 acquires images captured by the visible camera 21. The processor 16 also acquires the detection results of the depth camera 22 and the IMU 23.

In the point cloud generation process (three-dimensional measurement process), the processor 16 uses the SLAM method to generate point cloud data (environmental map) as three-dimensional spatial information about the measurement target location, based on the images of each location captured sequentially by the visible camera 21. In the point cloud generation process, self-position estimation is also performed in addition to the generation of the point cloud data, and the self-position for each time, i.e., the position of the shooting point, is obtained.

The point cloud generation process also includes a loop detection process. In the loop detection process, the processor 16 detects that the route of the aircraft forms a loop, i.e., that the aircraft has returned to a location where an image was previously captured. In this case, the detection of the previously captured location is not performed by comparing the image with a preset specified captured image, but rather, if the current captured image is similar to any of the previously captured images, it is determined that the aircraft has returned to a location where an image was previously captured. Note that the similarity of the captured images may be determined based on features extracted from the captured images.

The point cloud generation process also includes a loop closing process. When a loop is detected in the loop detection process, the loop closing process is carried out. In the loop closing process, the processor 16 regards the self-location estimation result (past position) acquired during the previous image capture as the correct position for a previously captured point, and corrects the position of each image capture point on the path from the current position to the past position.

The point cloud generation process also includes an implementation count acquisition process. In the implementation count acquisition process, the processor 16 counts the number of times the loop closing process has been performed for each shooting location. Implementation count information is added to the captured image (frame) of each shooting location. Implementation count information is also added to each point in the point cloud data. At this time, for example, implementation count information is added to each point included in the point cloud data generated from the captured image (frame) of the shooting location where the loop closing process was performed. The implementation count information also includes information regarding whether or not the loop closing process has been performed; if the implementation count is 0, it has not been performed, and if the implementation count is 1 or more, it has been performed.

The point cloud generation process also includes an accuracy acquisition process. In the accuracy acquisition process, the processor 16 acquires the accuracy of the three-dimensional measurement results (point cloud data and self-position estimation results) at each shooting point. Specifically, the accuracy of each shooting point is acquired based on the amount of distance from the point where the loop closing process has been performed. Specifically, the amount of distance is the moving distance (length of the moving trajectory) from the point where the loop closing process was performed, the elapsed time from the time when the loop closing process was performed, the number of images (number of frames) taken since the loop closing process was performed, etc. Note that the accuracy may be determined by combining the moving distance, elapsed time, and number of frames as the amount of distance. Furthermore, the accuracy may be determined by combining the amount of distance with the moving speed. Accuracy information is added to the captured image (frame) at each shooting point. Also, accuracy information may be added to each point in the point cloud data.

In addition, accuracy information for each point in the point cloud data can be obtained regardless of the implementation status of the loop closing process. For example, if the measurement results (point cloud data) based on the images captured at each shooting point do not match the detection results of the depth camera 22 or IMU 23, it can be determined that the accuracy is low. Accuracy may also be determined by comparing the measurement results based on previous and subsequent captured images (frames). In this case, accuracy information is added to each point in the point cloud data.

In addition, in the point cloud generation process, in addition to the images captured by the visible camera 21, the detection results of the depth camera 22 and the IMU 23 may be used. Specifically, based on the detection results of the depth camera 22, relative position information (distance from the imaging device 1) of each feature point extracted from the images captured by the visible camera 21 with respect to the imaging device 1 can be obtained, and the self-position estimation result based on the images captured by the visible camera 21 can be corrected. Also, based on the detection results of the IMU 23, relative position information of each imaging point can be obtained, and based on the relative position information of the imaging point, the self-position estimation result based on the images captured by the visible camera 21 can be corrected.

In the first support image generation process, the processor 16 generates a support image (first support image) in which the performed range and the recommended range are visualized on an overhead image. Specifically, a support image in which the performed range and the recommended range are visualized on an overhead point cloud image (see FIG. 3), a support image in which the performed range and the recommended range are visualized on an overhead point cloud route image (see FIG. 4), and a support image in which the performed range and the recommended range are visualized on an overhead route image (see FIG. 5) are generated.

When visualizing the performed and recommended implementation ranges on an overhead point cloud image, information on the performed and recommended implementation ranges is obtained based on the number of times performed information added to each point in the point cloud data, and each point on the point cloud image that is included in each of the performed and recommended implementation ranges is colored (point cloud coloring process).

In addition, when visualizing the completed and recommended areas on the overhead point cloud route image or the overhead route image, the image of the route (movement trajectory) connecting the positions of each shooting point with a line is colored based on the number of times the action has been taken information added to the shooting image (frame) of each shooting point (route coloring process).

In addition, in this embodiment, when visualizing the recommended implementation range on the overhead point cloud image, the range with low accuracy in the 3D measurement results is set as the recommended implementation range, and the points with low accuracy are colored with the color of the recommended implementation range based on the accuracy information added to each point in the point cloud data. Note that an area with many points with low accuracy in the point cloud data may be extracted as the recommended implementation range.

In addition, in this embodiment, when visualizing the recommended implementation range on an overhead point cloud route image or an overhead route image, the range with low accuracy on the route connecting each shooting point is set as the recommended implementation range, and information representing the recommended implementation range is obtained based on the accuracy information added to the captured image (frame) of each shooting point, and the image of the route is colored.

In the second support image generation process, the processor 16 generates a support image (second support image) (see FIG. 6) in which the performed area and the recommended area are visualized on the captured image. Specifically, an image representing the performed area drawn in a predetermined color and an image representing the recommended area drawn in a predetermined color are generated (superimposed image generation process), and the images representing the performed area and the recommended area are superimposed in a semi-transparent state on the captured image. Note that in the superimposed image generation process, information representing the range of each of the performed area and the recommended area may be obtained based on the number of times each point in the point cloud data is performed from the same viewpoint as the captured image.

In the screen control process, the processor 16 controls the screen to be displayed on the display 13. In this embodiment, a shooting screen 101 (see Figures 8 and 9) is generated and displayed on the display 13. On the shooting screen 101, a first support image (see Figures 3, 4, and 5) generated in a first support image generation process and a second support image (see Figure 6) generated in a second support image generation process are displayed.

In the message notification process, the processor 16 performs a process of notifying the user of messages regarding guidance for the shooting operation, various warnings, etc. In this embodiment, a message prompting the user to perform shooting operations for the loop closing process is displayed on the shooting screen 101 (see Figures 8 and 9).

At this time, if a specified notification condition is met, a message is displayed. For example, if the amount of movement from the point where the loop closing process was performed exceeds a specified threshold, a message is displayed. Also, if the time elapsed since the loop closing process was performed exceeds a specified threshold, a message is displayed. Here, the amount of movement and the elapsed time are indices for evaluating the deterioration of the accuracy of the 3D measurement results, and if the amount of movement or the elapsed time exceeds the respective threshold, it is determined that the deterioration of accuracy has exceeded the allowable limit, and a message is displayed to improve accuracy. Note that the message may be displayed at all times.

In this embodiment, the imaging device 1 performs the 3D measurement process (point cloud generation process), but the 3D measurement process may be performed by a server device (not shown) that can communicate with the imaging device 1.

Next, we will explain the shooting screen 101 displayed on the display 13. Figures 8 and 9 are explanatory diagrams showing the shooting screen 101.

The shooting screen 101 is provided with a main window 102 (main image display frame) and a sub-window 103 (sub-image display frame). The main window 102 and the sub-window 103 have different display magnifications, with the main window 102 displaying an enlarged image and the sub-window 103 displaying a reduced image.

The main window 102 and the sub-window 103 display a first support image 121 and a second support image 122, respectively. In the shooting screen 101 shown in FIG. 8, the second support image 122 is displayed in an enlarged manner in the main window 102, and the first support image 121 is displayed in a reduced manner in the sub-window 103. In the shooting screen 101 shown in FIG. 9, the first support image 121 is displayed in an enlarged manner in the main window 102, and the second support image 122 is displayed in a reduced manner in the sub-window 103.

The first support image 121 includes a support image in which the completed range and recommended range are visualized on an overhead point cloud image (see FIG. 3), a support image in which the completed range and recommended range are visualized on an overhead point cloud route image (see FIG. 4), and a support image in which the completed range and recommended range are visualized on an overhead route image (see FIG. 5), and one of the images is displayed depending on the settings. In the example shown in FIGS. 8 and 9, a support image in which the completed range and recommended range are visualized on an overhead point cloud image (see FIG. 3) is displayed as the first support image 121.

The second support image 122 is a support image (see FIG. 6) in which the implemented range and the recommended implementation range are visualized on the captured image. The captured image is an image captured in real time and output from the visible camera 21.

In addition, either the first support image 121 or the second support image 122 displayed in the main window 102 or the sub-window 103 may display an overhead image (overhead point cloud image, overhead point cloud route image, overhead route image) or a captured image as is, without visualizing the implemented range and recommended implementation range. The user may be able to set in advance whether or not to visualize the implemented range and recommended implementation range in the main window 102 and the sub-window 103.

The shooting screen 101 also has a "Start shooting" button 105 and a "Check recording" button 106. When the user operates the "Start shooting" button 105, shooting by the visible camera 21 begins, and the captured images are stored in the memory 15. When the user operates the "Check recording" button 106, the display transitions to a mode in which the captured images stored in the memory 15 are played back. This allows the user to check whether the shooting was performed properly.

The shooting screen 101 also has a "screen switch" button 107 and a check box 108 for displaying the sub-window 103. When the user operates the "screen switch" button 107, the state of the shooting screen 101 shown in FIG. 8 can be switched to the state of the shooting screen 101 shown in FIG. 9. That is, the state can be switched between one in which a shot image is displayed in the main window 102 and an overhead map is displayed in the sub-window 103, and one in which an overhead map is displayed in the main window 102 and a shot image is displayed in the sub-window 103. When the user checks the check box 108, the state transitions to one in which the sub-window 103 is not displayed.

The shooting screen 101 also has a message window 110. The message window 110 displays messages to notify the user. Specifically, messages regarding shooting instructions and various warnings are displayed.

In particular, during shooting work, if there is a location where loop closing processing is recommended, i.e., a location where loop closing processing has not been performed, or a location where the accuracy of the 3D measurement results is low, a support message is displayed to encourage the user to perform shooting work for the location where loop closing processing is recommended, i.e., the recommended implementation range drawn in red. In the example shown in Figures 8 and 9, the message "Please take photos so that loop closing is performed in the red area" is displayed.

In this embodiment, the captured image is displayed in real time on the capture screen 101. This allows the user to check the current capture status while capturing images. In addition, the completed range and the recommended range (uncompleted range) are visualized on the captured image, allowing the user to easily determine whether the area in front of them is a completed range or a recommended range.

In this embodiment, there is a method of visualizing (color-coding) areas where loop closing processing has not been performed as a recommended implementation range, and a method of visualizing areas where the accuracy of the 3D measurement results is low as a recommended implementation range. These two methods of visualizing the recommended implementation range can be switched as appropriate. For example, the method of visualizing the recommended implementation range may be set in advance on a setting screen (not shown). Also, the user may be allowed to specify the method of visualizing the recommended implementation range on the shooting screen 101.

As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to this, and can also be applied to embodiments in which modifications, substitutions, additions, omissions, etc. have been made. It is also possible to combine the components described in the above embodiments to create new embodiments.

The imaging device, imaging work support method, and imaging work support program of the present invention have the effect of improving the accuracy of 3D measurement using SLAM by clearly presenting the need for loop closing processing to a user during imaging work and encouraging the user to perform appropriate imaging work, and are useful as an imaging device that captures each point of a measurement target location in order to perform 3D measurement processing that generates 3D spatial information of the measurement target location based on captured images of each point of the measurement target location, as well as an imaging work support method and imaging work support program that support the imaging work of a user using the imaging device.

1: Photographing device 11: Device body 12: Sensor unit 13: Display (display unit)
14: Input device 15: Memory 16: Processor 21: Visible camera (photographing unit)
101: Shooting screen 102: Main window (main image display frame)
103: Sub-window (sub-image display frame)
110: Message window 121: First support image 122: Second support image

Claims (7)

An imaging device that captures images of each point of a measurement target location in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on an image captured at each point of the measurement target location,
A device body held by a user;
An imaging unit provided in the device body for imaging a measurement target location;
a display unit that displays support information for a photographing operation in which a user holds the device body and moves around a measurement target location, and causes the photographing unit to photograph each point of the measurement target location in sequence;
A processor that controls the image capturing unit and the display unit,
The processor,
The imaging device is characterized in that, as the support information, a support image that visualizes an area in which a loop closing process has been performed in the three-dimensional measurement process is displayed on the display unit. The processor,
The photographing device according to claim 1, characterized in that the support image, which visualizes the range in which the loop closing process has been performed in at least one of an overhead image generated from the measurement results of the 3D measurement process and a current photographed image, is displayed on the display unit. The processor,
2. The imaging device according to claim 1, wherein, in the support image, a range in which the loop closing process has already been performed is visualized in a display manner different from a range in which the loop closing process is recommended to be performed. The processor,
The photographing apparatus according to claim 1 , further comprising: a step of acquiring a number of times the loop closing process has been performed within a range in which the loop closing process has already been performed. The processor,
5. The photographing apparatus according to claim 4 , wherein a display mode of the range in which the loop closing process has been performed in the support image is changed according to the number of times the loop closing process has been performed. A photographing operation support method, in which a processor performs a process to support a photographing operation of a user who moves within a measurement target location while holding a photographing device and causes the photographing device to sequentially photograph each point of the measurement target location, in order to perform a three-dimensional measurement process that generates three-dimensional spatial information of the measurement target location based on photographed images of the respective points of the measurement target location, the process comprising:
A method for supporting an imaging operation, comprising: displaying, on a display unit, a support image that visualizes a range in which a loop closing process has been performed in the three-dimensional measurement process, as support information for the imaging operation. A photography operation support program that causes a processor to execute a process for supporting a photography operation of a user who moves within a measurement target location while holding a photography device and causes the photography device to sequentially photograph each point of the measurement target location, in order to perform a three-dimensional measurement process for generating three-dimensional spatial information of the measurement target location based on photographed images of the respective points of the measurement target location, the program comprising:
A photography operation support program, characterized in that a support image that visualizes the range in which loop closing processing has been performed in the three-dimensional measurement processing is displayed on a display unit as support information related to the photography operation.