JP2022129040A - Appropriate image selection system - Google Patents

Abstract

To solve a problem that the prior art has, namely, to provide a proper image selection system that can select images having less occlusion as a pair of stereo images among acquired aerial photographs.SOLUTION: A proper image selection system is a system selecting proper images used in stereoscopy in an object domain comprising image storage means, terrain model storage means, normal vector calculation means, line-of-vision vector calculation means, angle calculation means, and image selection means. The angle calculation means calculates angular differences between a normal vector at a fixation point and line-of-vision vectors regarding images, the image selection means selects proper images suitable for the fixation point based on the angular difference that the angle calculation means has calculated.SELECTED DRAWING: Figure 1

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to photogrammetry, and more specifically, to an appropriate image selection system capable of selecting an image more suitable for stereoscopic viewing of a feature.

When creating a topographic map or creating a three-dimensional topographic model (a topographic model composed of three-dimensional coordinates), topographic measurement is usually performed over a wide range. Photogrammetry, in which aerial photographs taken from an aircraft are used to plot (model) the terrain, is sometimes used to measure the terrain over a wide area. This photogrammetry is performed by preparing a set of two different aerial photographs (so-called stereo pair photographs) of the same location and performing stereoscopic vision (also called stereoscopic vision). This is a method of identifying the object and obtaining the three-dimensional coordinates of the object on the ground by utilizing the difference in the position of the object on the photograph (parallax).

The coordinates of the feature captured in the aerial photograph are obtained by the forward resection method with the camera center point (principal point of photography) as a reference. Therefore, the camera specifications such as the coordinates and direction of the principal point when each stereo pair photograph was taken, the focal length of the camera, the deviation of the position of the principal point, and various distortions (radiative distortion, asymmetrical distortion, etc.) are known. be done. In addition, the camera principal point coordinates and shooting direction at the time of photographing are called "exterior orientation elements", and the camera focal length, principal point position deviation, various distortions, etc. are called "interior orientation elements". The general term for exterior orientation elements and interior orientation elements is orientation elements.

Although it has been explained that the exterior orientation parameters are known in order to obtain the coordinates of ground objects by aerial photogrammetry, these exterior orientation parameters are not necessarily clarified by measurement or the like. For example, if a satellite positioning system (GNSS: Global Navigation Satellite System) or an IMU (Inertial Measurement Unit) is installed in an aircraft, it is possible to directly obtain exterior orientation elements, and if such measurement equipment is not installed, Exterior orientation parameters can be determined by aerial triangulation.

By the way, as shown in FIG. 5, photographing for photogrammetry may be performed by multiple high-lap photography in which the same location is photographed from multiple directions. 5A and 5B are model diagrams schematically showing a situation in which multiple high-lap imaging is performed. FIG. 6A is an image diagram showing a 3×3 multiple-shot image obtained by the multiple high-lap imaging, and FIG. ) is an image diagram showing a 5×5 multiple shot image obtained by multiple high-lap photography.

In this way, when there are a large number of images of the same location, different results may be obtained depending on the selected stereo pair photographs, and in some cases, desired results may not be obtained. Therefore, Japanese Patent Application Laid-Open No. 2002-200002 proposes a technique for automatically selecting suitable stereo pair image candidates.

JP 2017-49883 A

The technology disclosed in Patent Literature 1 automatically selects an appropriate stereo pair image using a "skew vector" based on the view point of the user who performs stereoscopic viewing and an "image vector" based on the shooting principal point. It is. Therefore, according to the technique disclosed in Patent Document 1, the user can preferably perform stereoscopic viewing, but on the other hand, there is room for improvement in terms of avoiding occlusion.

For example, in a case where it is desired to grasp the three-dimensional coordinates of a road edge, occlusion may occur due to roadside trees covering the road edge. If you shoot from the sky (especially directly above), you may get an aerial photo that includes the roadside trees that cover the road edge, and if you adopt such aerial photos as a stereo pair image, you can obtain the appropriate 3D coordinates of the road edge. It is not possible. On the other hand, an aerial photograph taken from an oblique direction may include the road edge, and if this aerial photograph is used as a stereo pair image, appropriate three-dimensional coordinates of the road edge can be obtained. That is, when the problem of occlusion is expected to occur, it is desirable to adopt, as a stereo pair image, those with less occlusion among a large number of aerial photographs obtained by multiple high-lap photography or the like.

An object of the present invention is to solve the problems of the prior art, that is, to provide an appropriate image selection system capable of selecting, as stereo pair images, images with less occlusion from acquired aerial photographs. It is in.

The present invention focuses on selecting an aerial photograph with less occlusion by using the angle difference between the "normal vector" based on the terrain model and the "line-of-sight vector" based on the shooting principal point. It is an invention made based on an idea that did not exist in the past.

The appropriate image selection system of the present invention is a system for selecting an appropriate image to be used for stereoscopic viewing of a target area, and comprises image storage means, terrain model storage means, normal vector calculation means, line-of-sight vector calculation means, and angle calculation means. , and image selection means. Among these, the image storage means is means for storing a plurality of images obtained by photographing the target area while partially overlapping, and the terrain model storage means stores a terrain model representing the target area in three dimensions. It is a means. The normal vector calculation means is a means for calculating a "normal vector" at the gaze point set within the target area based on the terrain model. , and the gaze point. Then, the angle calculation means calculates an angle difference between the normal vector at the point of gaze and the line-of-sight vector related to the image, and the image selection means selects an appropriate image suitable for the point of gaze based on the angle difference calculated by the angle calculation means. to select.

The appropriate image selection system of the present invention can also use a terrain model having a plurality of coordinated grid points and a plurality of divided regions.

The appropriate image selection system of the present invention may further comprise gradient calculating means and area extracting means. The gradient calculating means is means for calculating the gradient of the divided areas of the terrain model, and the area extracting means is means for extracting the divided areas whose gradient exceeds a predetermined threshold. In this case, the three-dimensional coordinate points (lattice points of this divided area, constituent points of feature polygons included in the divided area, etc.) related to the divided area extracted by the area extracting means are set as points of gaze.

The appropriate image selection system of the present invention may further comprise two-dimensional map information storage means and feature extraction means. The two-dimensional map information storage means is means for storing two-dimensional map information in which feature classifications are given to the features of the target area. It is a means for extracting features (for example, road edges, sidewalk edges, and median strip edges) to which object classifications have been assigned. In this case, a three-dimensional coordinate point related to a divided area including a feature extracted by the feature extraction means (for example, a road edge, a sidewalk edge, or an outer edge of a median strip) is set as a gaze point.

The appropriate image selection system of the present invention has the following effects.
(1) Since the three-dimensional coordinates of the object are obtained after selecting an image with little occlusion, it is possible to obtain appropriate results even for features that tend to be shaded, such as road edges.
(2) Since the user's judgment is not required, the user's labor is reduced, human error due to subjectivity and the like is avoided, and efficient and highly accurate results can be obtained.

A model diagram schematically showing a “normal vector” and a “line-of-sight vector”. 1 is a block diagram showing the main configuration of a suitable image selection system according to the present invention; FIG. FIG. 4 is a flow chart showing the flow of main processing when using the appropriate image selection system of the present invention; FIG. 4 is a flow chart showing the flow of main processing for automatically setting a gaze point by a point of view setting means; FIG. 4 is a model diagram schematically showing a situation in which multiple high-lap photography is performed; (a) is an image diagram showing a 3×3 multiple photographed image obtained by multiple high lap photography, and (b) is an image diagram showing a 5×5 multiple photographed image obtained by multiple high lap photography.

An example of implementation of the appropriate image selection system of the present invention will be described with reference to the drawings.

1. Overall Outline One of the technical features of the present invention is to select an aerial photograph with less occlusion by using the angle difference between the "normal vector" and the "line-of-sight vector". Therefore, the normal vector VN and the line-of-sight vector VS will be described with reference to FIG. 1, and the procedure for selecting an aerial photograph will be described.

In calculating the normal vector VN and the line-of-sight vector VS, a gaze point PV is set as will be described later. This gaze point PV is a point such as a road edge for which the user wants to obtain the coordinates, and it is preferable to set a point where occlusion may occur, such as a road edge covered with roadside trees.

The normal vector VN is set in the direction normal to the point of interest PV as shown in FIG. 1, and is a vector of arbitrary length (for example, unit length) starting from the point of interest PV. The normal direction at the gaze point PV can be set by using a three-dimensional terrain model such as DSM (Digital Surface Model) or DTM (Digital Terrain Model). Usually, a three-dimensional terrain model is composed of a plurality of grid points assigned with three-dimensional coordinates and a plurality of divided regions (hereinafter referred to as "mesh"), and includes a gaze point PV (or a gaze point PV is a grid point), and the direction perpendicular to the inclination (orientation) of the mesh can be set as the normal direction at the gaze point PV. Alternatively, the normal vector VN is determined based on the normal direction obtained by various conventional methods, such as obtaining the normal direction using a triangulated irregular network (TIN) formed from random data. can be set.

On the other hand, the line-of-sight vector VS, as shown in FIG. 1, is a vector whose starting point is the photographing principal point PM and whose end point is the gaze point PV. Then, when the normal vector VN and the line-of-sight vector VS are obtained, the included angle between the normal vector VN and the line-of-sight vector VS (hereinafter referred to as the "vector included angle") is obtained. By the way, in principle, only one normal vector VN is set for one gaze point PV, but as many line-of-sight vectors VS as the number of photographing principal points PM can be set for one gaze point PV. That is, a plurality of (the number of shooting principal points PM) vector included angles are obtained for one gazing point PV. Therefore, a line-of-sight vector VS that gives a small vector included angle is selected, and an aerial photograph related to this line-of-sight vector VS is selected. A small vector included angle means that the normal vector VN is generally directed toward the shooting principal point PM and the line-of-sight vector VS is generally directed toward the gaze point PV. can be considered as an image with less occlusion (that is, suitable for stereoscopic viewing).

2. Appropriate Image Selection System Next, the appropriate image selection system of the present invention will be described in detail with reference to the drawings. FIG. 2 is a block diagram showing the main configuration of the appropriate image selection system 100 of the present invention. As shown in this figure, the appropriate image selection system 100 of the present invention includes normal vector calculation means 101, line-of-sight vector calculation means 102, angle calculation means 103, image selection means 104, image storage means 109, and 3D terrain model storage means 110. , and further includes gradient calculation means 105, area extraction means 106, feature extraction means 107, gaze point setting means 108, 2D map information storage means 111, shooting specification storage means 112, etc. can.

Normal vector calculation means 101, line-of-sight vector calculation means 102, angle calculation means 103, image selection means 104, gradient calculation means 105, area extraction means 106, feature extraction means 107, gaze point setting, which constitute an appropriate image selection system 100. Means 108 can be manufactured as a dedicated one, or can utilize a general-purpose computing device. This computer device includes a processor such as a CPU and a memory such as ROM and RAM. Portable terminal devices such as smartphones, personal computers (PCs), servers, and the like can be exemplified.

The image storage means 109, the 3D terrain model storage means 110, the 2D map information storage means 111, and the photographing specification storage means 112 can use the storage device of the computer device, or can be constructed in a database server. can. When building a database server, it can be placed on a local network (LAN: Local Area Network), or it can be a cloud server for storage via the Internet (that is, wireless communication).

Each main element constituting the appropriate image selection system 100 will be described in detail below.

(3D Terrain Model Storage Means and 2D Map Information Storage Means)
The 3D terrain model storage means 110 is means for storing a 3D terrain model (DSM, DTM, etc.) of a target range (hereinafter referred to as "target area"). On the other hand, the 2D map information storage means 111 stores the 2D map information of the target area. Here, the two-dimensional map information is two-dimensional topographic information such as a topographic map without height information, and is map information with information on features (hereinafter referred to as "feature information"). The feature information includes information on the planar position and shape of features such as polygons and polylines (hereinafter referred to as "feature figures") and attribute information on the features. Furthermore, feature attribute information includes roads, road edges, sidewalk edges, outer edges of median strips (hereinafter simply referred to as "median strip edges"), types of features such as buildings, outer edges of buildings, parks, and schools ( hereinafter referred to as “feature classification”). In other words, a feature figure is laid out in a plane within the target area, and attribute information (including feature classification) relating to the feature is associated with the feature figure.

(Image storage means and photographing specification storage means)
The image storage means 109 is a means for storing a plurality of images obtained by photographing a target region while partially overlapping. For example, as shown in FIG. It is something that can be done. Further, the photographing specification storage means 112 stores the photographing principal point PM when each image stored in the image storage means 109 is obtained, and also stores other exterior orientation elements and interior orientation elements. be able to. Note that the photographing principal point PM is stored in the photographing item storage unit 112 in association with (associated with) the image related to the photographing principal point PM.

(Point-of-regard setting means)
The gazing point setting means 108 is means for setting the gazing point PV. This gaze point setting means 108 can be set by the user operating a pointing device or keyboard while viewing the terrain model or image, or can be automatically set without user operation. You can also Processing for automatically setting the point of interest PV by the point of interest setting means 108 will be described below.

As described above, the gazing point PV is a point such as a road edge for which the user wants to obtain the coordinates. Therefore, meshes in which occlusion is predicted (hereinafter referred to as "specific meshes") are extracted from the 3D terrain model. Specifically, the gradient calculation means 105 reads out the 3D terrain model from the 3D terrain model storage means 110, calculates the slope of the mesh (hereinafter referred to as "mesh gradient"), and calculates a predetermined threshold (hereinafter referred to as "gradient threshold ) is extracted as a specific mesh by the region extracting means 106 (FIG. 1). The mesh gradient can be calculated in advance and stored in the 3D terrain model storage means 110. In this case, the mesh gradient can be read out from the 3D terrain model storage means 110 without calculating the mesh gradient. The mesh gradient is calculated as the angle with the horizontal plane, and similarly the gradient threshold is set as the angle with the horizontal plane. Normally, when there is no obstruction such as a street tree, the feature (that is, the mesh) is generally horizontal or has a gentle slope. On the other hand, a mesh that is greatly inclined from the horizontal plane is not in a normal state, that is, the presence of an obstructing object such as a street tree is suspected. When the specific mesh is extracted by the region extracting means 106, the point-of-regard setting means 108 sets the grid point of the specific mesh as the point of interest PV (FIG. 1). Alternatively, in the case where the 3D terrain model storage means 110 stores polygons, polylines, etc. of features separately (independently) from the 3D terrain model, three-dimensional configuration points such as polygons included in a specific mesh (hereinafter referred to as (referred to as a "3D configuration point") can also be set as the gaze point PV. In other words, any point related to a specific mesh (points forming or included) and having three-dimensional coordinates can be set as the gaze point PV.

Also, the point of gaze PV may be set for a feature assigned a specific feature classification (hereinafter referred to as "specific feature"). In this case, the feature extraction means 107 obtains the plane coordinates of the specific feature by referring to the 2D map information storage means 111 (FIG. 1). For example, when the user designates a road edge as a specific feature, the feature extraction means 107 acquires the plane coordinates of the road edge from the 2D map information storage means 111 . Of course, the user can also designate a sidewalk edge, a median strip edge, or the like instead of the road edge as the specific feature. Then, the point-of-regard setting means 108 sets a mesh corresponding to the plane coordinates of a specific feature (for example, a road edge) as a specific mesh, and also sets 3D constituent points such as grid points of the specific mesh and feature polygons included in the specific mesh. is set as the gaze point PV (FIG. 1). At this time, the gradient calculation means 105 calculates the mesh gradient for the mesh corresponding to the plane coordinates of the specific feature acquired by the feature extraction means 107, and then the area extraction means 106 extracts it as the specific mesh. can also

(Normal vector calculation means)
The normal vector calculation means 101 is means for calculating the normal vector VN. More specifically, the normal vector calculation means 101 reads the mesh related to the point of regard PV set by the point of regard setting means 108 from the 3D terrain model storage means 110, and calculates the normal vector VN based on the inclination (orientation) of the mesh. Calculate The normal vector calculation means 101 can be calculated at the timing when the point of interest setting means 108 sets the point of interest PV, or the normal vector calculation means 101 can be calculated in advance for each 3D configuration point such as a grid point of a mesh or a feature polygon. can also

(Line-of-sight vector calculation means)
The line-of-sight vector calculation means 102 is means for calculating the line-of-sight vector VS. More specifically, the line-of-sight vector calculation means 102 reads the shooting principal point PM from the shooting specification storage means 112 and calculates the line-of-sight vector VS based on this shooting principal point PM and the gaze point PV set by the gaze point setting means 108. do. Note that the line-of-sight vector calculation means 102 can calculate the line-of-sight vector VS as many as the number of shooting principal points PM as described above. Of course, it is possible to adopt a specification that calculates the line-of-sight vector VS for all shooting principal points PM (that is, all images). It is also possible to adopt a specification in which the line-of-sight vector VS is calculated after selecting an image that meets the condition of .

(Angle calculation means)
The angle calculation means 103 is means for calculating a vector included angle based on the normal vector VN and the line-of-sight vector VS. The angle calculating means 103 can also calculate an inner product (hereinafter referred to as "vector inner product") based on the normal vector VN and the line-of-sight vector VS instead of (or in addition to) the vector included angle. However, when calculating the vector inner product, one of the normal vector VN and the line-of-sight vector VS is reversed (multiplied by a negative value) before being calculated.

(Image selection means)
The image selection means 104 is a means for selecting an image suitable for stereoscopic viewing (hereinafter referred to as "appropriate image") from the image storage means 109 based on the vector angle (vector inner product). When the determination is based on the vector included angle, the image associated with the vector included angle with a small value (that is, the line-of-sight vector VS) is selected. VS) image is selected. At this time, the top two images of the stereo pair can be selected as appropriate images, or all images whose vector included angles are below a predetermined threshold (above in the case of vector inner products) can be selected as appropriate images. Alternatively, images positioned in the top (eg, 1st to 6th) can be selected as appropriate images.

(Processing flow)
Next, with reference to FIG. 3, the main processing flow when using the appropriate image selection system 100 of the present invention will be described. In the flow diagrams of FIG. 3 and FIG. 4, which will be described later, the middle column shows the action to be performed, the left column shows what is necessary for that action, and the right column shows what results from that action.

First, the gaze point PV is set by the gaze point setting means 108 (Step 210 in FIG. 3). As described above, the point-of-regard setting means 108 can be set by the user operating a pointing device or keyboard while viewing the terrain model or image, or can be set automatically without user's operation. It can also be specified to Hereinafter, automatic setting processing of the point-of-regard PV by the point-of-regard setting means 108 will be described with reference to FIG.

When a specific feature (for example, a road edge) is designated by a user operation, the feature extraction means 107 acquires the plane coordinates of the specific feature from the 2D map information storage means 111 (Step 201 in FIG. 4). Then, the gradient calculation means 105 calculates the mesh gradient for the mesh corresponding to the plane coordinates (or reads the mesh calculated in advance by the gradient calculation means 105) (Step 202 in FIG. 4), and the region extraction means 106 extracts meshes exceeding the gradient threshold as specific meshes (Step 203 in FIG. 4). When the specific mesh is extracted by the region extracting means 106, the gaze point setting means 108 sets the 3D coordinate points (lattice points of the specific mesh, 3D constituent points such as feature polygons included in the specific mesh) related to the specific mesh. It is set as the gaze point PV (Step 210 in FIG. 4).

When the gazing point setting means 108 sets the gazing point PV, the normal vector calculating means 101 calculates the normal vector VN for the gazing point PV (Step 220 in FIG. 3), and the line-of-sight vector calculating means 102 A line-of-sight vector VS is calculated (Step 230 in FIG. 3). Note that when the normal vector VN is calculated in advance for each 3D constituent point such as a grid point of a mesh or a feature polygon, it is only necessary to read the normal vector VN corresponding to the gaze point PV. Then, when the normal vector VN and the line-of-sight vector VS are obtained, the angle calculation means 103 calculates a vector included angle (vector inner product) (Step 240 in FIG. 3). The line-of-sight vector calculation means 102 repeatedly calculates the line-of-sight vector VS while changing the photographing principal point PM, and accordingly the angle calculation means 103 also repeatedly calculates the vector included angle. At this time, it is also possible to calculate the line-of-sight vector VS and the vector included angle for all the photographing principal points PM, or to calculate the line-of-sight vector VS and the vector included angle can also be calculated.

When the vector included angle is calculated by the angle calculation means 103, the image selection means 104 selects an appropriate image from the image storage means 109 based on the vector included angle. By the way, it is conceivable that the proper image selected by the image selection means 104 does not include the gaze point PV. Therefore, the image selected by the image selection means 104 based on the included vector angle may be temporarily set as a "provisional image" (Step 250 in FIG. 3), and it may be determined whether or not the gaze point PV is included in this provisional image ( Step 260 in FIG. 3). That is, if the provisional image contains the gaze point PV (Yes in Step 260 of FIG. 3), this provisional image is determined as a proper image (Step 270 in FIG. 3). If not (No in Step 260 in FIG. 3), the photographing principal point PM is changed and a series of processes (Steps 230 to 260 in FIG. 3) are repeatedly executed. However, in the case of calculating the line-of-sight vector VS and the vector included angle with respect to the shooting principal point PM related to the image including the gaze point PV within the angle of view, provisional image setting processing (Step 250 in FIG. 3) and provisional image The determination process (Step 260 in FIG. 3) can be omitted.

The appropriate image selection system of the present invention can be particularly suitably used as map information used for management of various facilities including road facilities and for automatic driving. In addition, according to the present invention, it is possible to provide step information with high accuracy that is useful for the elderly and wheelchair users, and it can also be effectively used for disaster prevention planning. It is an invention that can be expected not only to be used but also to make a great contribution to society.

100 Appropriate Image Selection System of the Present Invention 101 Normal Vector Calculation Means 102 Line of Sight Vector Calculation Means 103 Angle Calculation Means 104 Image Selection Means 105 Gradient Calculation Means 106 Region Extraction Means 107 Feature Extraction Means 108 Gaze Point Setting Means 109 Image Storage Means 110 3D terrain model storage means 111 2D map information storage means 112 Shooting specification storage means PM Principal point of photography PV Point of regard VN Normal vector VS Line of sight vector

Claims (5)

A system for selecting an appropriate image for stereoscopic viewing of a target region, comprising:
an image storage means for storing a plurality of images obtained by photographing a target area while partially overlapping;
a terrain model storage means for storing a terrain model representing the target area in three dimensions;
a normal vector calculation means for calculating a normal vector at a gaze point set within the target area based on the terrain model;
a line-of-sight vector calculation means for calculating a line-of-sight vector consisting of the principal point of photography when the image was acquired and the point of gaze;
angle calculation means for calculating an angle difference between the normal vector at the gaze point and the line-of-sight vector relating to the image;
an image selection means for selecting the appropriate image suitable for the point of gaze based on the angle difference calculated by the angle calculation means;
A suitable image selection system characterized by: The terrain model comprises a plurality of coordinated lattice points and a plurality of divided regions.
2. The appropriate image selection system according to claim 1, wherein: Gradient calculation means for calculating a gradient of the divided regions;
a region extracting means for extracting the divided regions in which the gradient exceeds a predetermined threshold;
A three-dimensional coordinate point related to the divided area extracted by the area extracting means is set as the gaze point;
3. The appropriate image selection system according to claim 2, wherein: 2D map information storage means for storing 2D map information in which feature classifications are assigned to the features of the target area;
a feature extracting means for extracting the feature to which the specific feature classification is assigned, based on the two-dimensional map information;
The three-dimensional coordinate point related to the divided area containing the feature extracted by the feature extraction means is set as the gaze point;
4. The proper image selection system according to claim 2 or 3, characterized in that: 2D map information storage means for storing 2D map information in which feature classifications are assigned to the features of the target area;
a feature extraction means for extracting a road edge, a sidewalk edge, or a median strip edge based on the two-dimensional map information;
The three-dimensional coordinate point related to the divided area including the road edge, the sidewalk edge, or the median strip edge extracted by the feature extraction means is set as the gaze point.
4. The proper image selection system according to claim 2 or 3, characterized in that:

Images