JP2024061982A - Self-location estimation device, self-location estimation method, and self-location estimation program - Google Patents

Abstract

[Problem] To provide a self-location estimation device, a self-location estimation method, and a self-location estimation program capable of shortening processing time. [Solution] A self-location estimation device 1 includes an image data acquisition unit 11 that acquires image data output from a multi-camera 20 consisting of multiple cameras mounted on a moving body MC at set intervals, a feature point extraction unit 12 that extracts feature points from the image data, a storage unit 13 that stores position information of each feature point in each image data, a flow creation unit 14 that creates flows that are pairs of corresponding feature points in different image data for each time series, a selection unit 15 that uses predetermined constraint conditions including an epipolar constraint calculated based on the predicted motion of the moving body to select flows that satisfy the constraint conditions as correct flows, and a self-location calculation unit 16 that calculates the position and orientation of the moving body using information on the correct flows selected by the selection unit. [Selected Figure] Figure 1

Description

The present disclosure relates to a self-location estimation device, a self-location estimation method, and a self-location estimation program.

Conventionally, as shown in, for example, Patent Document 1, a method of estimating one's own position using SLAM (Simultaneous Localization and Mapping) is known. SLAM is a technology that uses a mobile object equipped with a camera or sensor to simultaneously estimate one's own position and create a three-dimensional map of the environment without relying on a satellite system such as GPS.

JP 2022-62807 A

However, the above-mentioned conventional technology has a problem in that a large amount of processing is required for self-location estimation and the processing time is long because a three-dimensional map is created. The present disclosure has been created in consideration of the above-mentioned points, and its purpose is to provide a self-location estimation device, a self-location estimation method, and a self-location estimation program that can shorten the processing time.

This disclosure can be realized in the following forms:

According to one embodiment of the present disclosure, a self-location estimation device is provided. This self-location estimation device is a self-location estimation device that estimates the self-location of a moving body (MC), and includes an image data acquisition unit (11) that acquires image data output from a multi-camera (20) consisting of a plurality of cameras (21, 22, 23, 24) mounted on the moving body at set intervals, a feature point extraction unit (12) that extracts feature points from the image data acquired by the image data acquisition unit, a storage unit (13) that stores position information of each of the feature points in each of the image data, and a storage unit (14) that stores current feature points that are feature points in the latest image data and feature points in past image data. The system includes a flow creation unit (14) that associates a feature point with a previous feature point that is different for each time series to create a flow that is a pair of corresponding feature points in the image data that differs for each time series, a selection unit (15) that uses predetermined constraint conditions including an epipolar constraint calculated based on the predicted motion of the moving body to select, as a correct flow, a flow that satisfies the constraint conditions from among the flows created by the flow creation unit, and a self-position calculation unit (16) that calculates the position and orientation of the moving body using information on the correct flow selected by the selection unit.

According to this form of self-position estimation device, the self-position of the vehicle can be estimated without creating a three-dimensional map, using image data output from the multi-camera acquired by the image data acquisition unit. Then, the selection unit uses predetermined constraint conditions including epipolar constraints to select flows that satisfy the constraint conditions as correct flows. By using the constraint conditions in this way, it is possible to limit an appropriate search range when identifying an appropriate current feature point in the latest image data, making it possible to predict the current feature point and reducing the search processing time. As a result, the processing time for self-position estimation in the self-position estimation device can be reduced.

1 is a block diagram showing a schematic system configuration of a self-position estimation device according to a first embodiment; 1 is a plan view showing a vehicle on which a self-position estimation device is mounted. 13 is a flowchart showing a procedure for obtaining a feature point list for a current frame. FIG. 13 is a diagram for explaining epipolar constraints. 11 is a diagram showing dynamic vehicle motion information and changes in the positions of the cameras; FIG. 1 is a histogram showing the statistical distribution of flow lengths. 1 is a histogram showing the statistical distribution of flow lengths. 1 is a histogram showing the statistical distribution of flow lengths. 1 is a histogram showing the statistical distribution of flow lengths. 11 is a diagram showing dynamic vehicle motion information and changes in the positions of the cameras; FIG. 1 is a histogram showing the statistical distribution of flow lengths. 1 is a histogram showing the statistical distribution of flow lengths. 1 is a histogram showing the statistical distribution of flow lengths. 1 is a histogram showing the statistical distribution of flow lengths. FIG. 2 illustrates an example of an arbitrary current frame, with feature points and flow illustrated within the frame. FIG. 2 illustrates an example of an arbitrary current frame, with feature points and flow illustrated within the frame.

Hereinafter, an embodiment will be described with reference to FIGS.
A. First embodiment:
A1. Configuration of self-location estimation device 1:
The configuration of a self-location estimation device 1 of the first embodiment will be described with reference to Fig. 1 and Fig. 2. The self-location estimation device 1 of the first embodiment is mounted on an autonomously driven vehicle MC (hereinafter simply referred to as "vehicle MC") as a moving body, and estimates the self-location of the vehicle MC. The vehicle MC can move by autonomous driving based on the self-location estimated by the self-location estimation device 1. Note that "self-location" includes the position and orientation of the moving body, and means the position and attitude.

As shown in FIG. 1, the vehicle MC includes a self-position estimation device 1, a vehicle control unit 10, and a camera 20. The vehicle control unit 10 is a functional unit that controls the vehicle MC, and the camera 20 is electrically connected to the vehicle control unit 10, and image data (hereinafter also referred to as "frames") captured by the camera 20 is notified to the vehicle control unit 10. In addition, a GNSS (Global Navigation Satellite System) sensor (not shown) and the like are electrically connected to the vehicle control unit 10.

As shown in FIG. 2, the camera 20 is configured as a compound-eye multi-camera having a total of four cameras: a front camera 21, a rear camera 22, a right camera 23, and a left camera 24. The front camera 21 captures an image of the surroundings in front of the vehicle MC. The rear camera 22 captures an image of the surroundings behind the vehicle MC.

The right camera 23 captures an image of the surroundings to the right when viewed in the direction of travel of the vehicle MC. The left camera 24 captures an image of the surroundings to the left when viewed in the direction of travel of the vehicle MC. Hereinafter, when there is no need to distinguish between the cameras 21, 22, 23, and 24, they will simply be referred to as "camera 20." Camera 20 is a wide-angle camera that is representative of a fisheye lens, and the combined angle of view of the four cameras covers a 360-degree field of view.

Referring again to FIG. 1 . The self-location estimation device 1 of this embodiment is electrically connected to the vehicle control unit 10 of the vehicle MC. Specifically, since the self-location estimation device 1 of this embodiment is connected to the OBD 2 (On Board Diagnostics 2) of the vehicle MC, the self-location estimation device 1 can receive various information about the vehicle MC from the vehicle control unit 10.

Note that the connection between the self-location estimation device 1 and the vehicle MC is not limited to this, and for example, a connection by wireless communication may be used. Also, the self-location estimation device 1 and the vehicle MC may not be connected, and the self-location estimation device 1 may be provided with a separate camera 20. In this embodiment, power is supplied to the self-location estimation device 1 from the OBD 2, but this is not limited to this, and the self-location estimation device 1 may be provided with a separate battery, and power for the self-location estimation device 1 may be supplied from the cigarette lighter socket of the vehicle MC.

The self-location estimation device 1 includes the following functional modules: an image data acquisition unit 11, a feature point extraction unit 12, a memory unit 13, a flow creation unit 14, a selection unit 15, and a self-location calculation unit 16. The self-location estimation device 1 is configured as a well-known computer including a CPU and memories such as ROM and RAM. Each of the functional modules 11 to 16 is realized by the CPU executing a program stored in the memory.

The image data acquisition unit 11 acquires image data of the surrounding environment of the vehicle MC. Specifically, the image data acquisition unit 11 executes a process of acquiring image data output from the camera 20 from the vehicle MC at set intervals.

The feature point extraction unit 12 executes a process of extracting feature points from the image data acquired by the image acquisition unit 11. Feature points include, for example, edges, corners, color distribution, and the like. The feature point extraction method may be an indirect method such as ORB, SIFT, or SURF, or a direct method such as DSO (Direct Sparse Odometry).

The storage unit 13 executes a process of storing three-dimensional position coordinates as position information of each feature point in each image data. In addition, the storage unit 13 stores various data and calculation results used in a series of processes for estimating the self-position of the vehicle MC.

The flow creation unit 14 executes a process of creating a flow that is a pair of corresponding feature points in different image data for each shooting time. When creating the flow, the flow creation unit 14 performs correspondence between previous feature points, which are feature points in past image data stored in the storage unit 13, and current feature points, which are feature points in the latest image data. Specifically, correspondence between previous feature points in the image data immediately before the latest and current feature points is performed for each frame.

The selection unit 15 executes a process of selecting, from among the flows created by the flow creation unit 14, flows that satisfy the predetermined constraint conditions as correct flows, using the constraint conditions. The constraint conditions include an epipolar constraint calculated based on the predicted motion of the vehicle MC. Details of the constraint conditions will be described in detail together with a control flowchart in the self-location estimation method described later.

The self-position calculation unit 16 executes a process of calculating the position and orientation of the vehicle MC using the flow information selected by the selection unit 15. The calculated position and orientation of the vehicle MC, i.e., the position and orientation, become the estimated result of the current position of the vehicle MC. Here, SLAM processing is used for calculating the self-position. SLAM is an acronym for Simultaneous Localization And Mapping, and is a method of simultaneously estimating the self-position and creating an environmental map. Note that in this embodiment, an environmental map as a three-dimensional map is not created, and the position of the vehicle MC is estimated using only the image from the camera 20.

A2. Self-location estimation method:
Next, a description will be given of a self-location estimation method performed by the self-location estimation device 1. Fig. 3 is a flowchart showing a procedure for obtaining a feature point list of the current frame to be used for current (i.e., latest) self-location calculation. The "current frame" is the latest frame, and below, the "previous frame" is the frame immediately preceding the current frame, and the "previous frame" is the frame immediately preceding the previous frame. The "previous frame" and the "frame before last" both correspond to past image data.

The feature point list includes the information of the above flow. The process shown in FIG. 3 is repeatedly executed by the self-location estimation device 1 at set intervals. The set interval here is the same as the set interval for photographing by the camera 20.

As shown in FIG. 3, in S101, the previous estimated values of the angle change amount and movement amount of the vehicle MC are read. The "previous estimated values" are the angle change amount and movement amount of the vehicle MC between the frame before last and the previous frame, and are calculated in advance and stored in the memory unit 13. Next, in S102, the previous estimated values read in S101 are used to create a fundamental matrix between the previous frame and the current frame, assuming uniform motion. The fundamental matrix is a matrix used when expressing the epipolar constraint in a mathematical formula.

Next, in S103, an epipolar line is created. The epipolar line is created by a known method using the fundamental matrix created in S102, the three-dimensional coordinates of the feature points in the current frame and the focal length of the camera 20, and the three-dimensional coordinates of the feature points in the previous frame and the focal length of the camera 20.

Then, in S104, the upper threshold value due to the epipolar constraint and the reference range of the flow length are determined. The process of S104 is executed by the selection unit 15. The process of S104 may be executed by another functional unit, not limited to the selection unit 15. The "flow length" is the length of one flow connecting a previous feature point and a current feature point corresponding to the previous feature point. The "flow length" is the length of a flow between consecutive image data in a time series, in other words, the length of a flow between one frame. The upper threshold value due to the epipolar constraint and the reference range of the flow length determined in S104 are collectively referred to as "constraint conditions". The constraint condition of satisfying the upper threshold value due to the epipolar constraint is referred to as the "first condition", and the constraint condition of satisfying the reference range of the flow length is referred to as the "second condition".

After the constraint conditions are determined in S104, a judgment based on the constraint conditions is made in S105. That is, it is judged whether the above-mentioned first and second conditions are satisfied. Specifically, if the coordinates of the current feature point are within an upper threshold of a predetermined distance from the epipolar line, and the flow length connecting the current feature point and the previous feature point is within a reference range, it is judged that the constraint conditions are satisfied. If either the first or second condition is not satisfied, it is judged that the constraint conditions are not satisfied. This judgment is made for all feature points in the frame. The determination and judgment of the constraint conditions in S104 and S105 will be described in detail later.

Next, in S106, the feature points that satisfy the constraint conditions are added to a "feature point list". The "feature point list" is a list that stores information on feature points and flows used in calculating the self-location estimation after the end of this control routine. Next, in S107, it is determined whether the number of flows added to the feature point list is equal to or greater than a reference value. The reference value for the number of flows is determined in advance by a prior test as the number required for calculating an appropriate self-location estimation. Then, in S107, if the number of flows is equal to or greater than the reference value (S107: Yes), the process proceeds to S109, where the feature point list is output. The feature point list output here is a list in which information on feature points and flows that do not satisfy the constraint conditions has been thinned out. After the process of S109, this processing routine ends. Then, using the information in the output feature point list, self-location estimation is performed, for example, by the two-point method.

On the other hand, in S107, if the number of flows added to the feature point list is not equal to or greater than the criterion (S107: No), the process proceeds to S108, where the constraint conditions are relaxed. If the number of flows is too small and less than the criterion, the current self-location estimation cannot be calculated or the accuracy drops significantly, so the constraint conditions are relaxed to increase the number of flows in the feature point list.

Specifically, "relaxing the constraints" means increasing the upper threshold of the predetermined distance from the epipolar line, or increasing the reference range of the flow length. After the constraints are relaxed, the process returns to the determination process in S105, and the processes in S108, S105, and S106 are repeated until the number of flows in S107 becomes equal to or greater than the reference.

A3. Details of constraints:
Next, details regarding the determination and judgment of the constraint conditions in S104 and S105 will be described. First, the upper threshold value by the epipolar constraint will be described with reference to FIG. 4. FIG. 4 is a diagram for explaining the epipolar constraint, and shows the geometry of a three-dimensional space photographed from cameras at different positions. In FIG. 4, points e _L and e _R are epipoles, which are positions at which one camera photographs the other camera. In addition, the epipolar line E is a line that is a candidate for "where the object is photographed from the camera O _R " when "an object is photographed at X _L as viewed from the camera O _L ".

As shown in Fig. 4, an object at _XP is captured at X _L when viewed from camera O _L in the previous frame, and is captured at X _R when viewed from camera O _R in the current frame. When all that is known is that "an object is captured at X _L when viewed from camera O _L ," the position at which the object is captured by camera O _R (where X _R is) is limited to "somewhere on the epipolar line E." This is called an epipolar constraint.

In the case of a conventional method for creating a three-dimensional map, the rough position of Xp in the depth direction can be known from two frames, so the search range for searching for feature points in the current frame can be set to a relatively small two-dimensional range. However, in this embodiment, since there is no three-dimensional map, although it is known that the feature point is on an extension of the X-ray, which is a straight line connecting O _-L and X- _L , the position in the depth direction on the X-ray is unknown. That is, in FIG. 4, it may be at X- ₂ , X- ₁ , or X _-0 . Also, although it is known that the feature point is near X- _R , the search range for the feature point is wide and cannot be determined, even if it is near the epipolar line E.

In this embodiment, when searching for a current feature point in the current frame, a range within a predetermined distance from the epipolar line E is set as the search range S. In other words, the search range S is the area in the current frame between two parallel straight lines L1 and L2 at a predetermined distance on both sides from the epipolar line E. In FIG. 4, the "distance from the epipolar line E" is the one-dimensional distance between the epipolar line E and the straight lines L1 and L2 in the current frame as a two-dimensional plane. In other words, in S104, the upper threshold due to the epipolar constraint, i.e., the above-mentioned search range S, is determined based on the created epipolar line and the preset distance.

Then, in S105, the selection unit 15 selects, from among the created flows, those flows whose position coordinates of the current feature point in the current frame are within the search range S as correct flows. The upper threshold due to the epipolar constraint, i.e., a predetermined distance from the epipolar line E, is used as an index for selecting flows. Note that the upper threshold of the distance is appropriately set in advance based on prior test data, taking into account the search processing time and the positional accuracy of the feature points, etc., so that the search range S does not become too wide.

Next, the determination of the reference range of the flow length will be described with reference to Figs. 5 to 14. Fig. 5 dynamically shows the motion information of the vehicle MC, and shows the positional changes of the cameras 21, 22, 23, and 24, and shows the form when turning left in an arbitrary environment. Figs. 6 to 9 are histograms showing the frequency distribution of the flow length. Fig. 6 is a histogram based on data acquired by the front camera 21, and Fig. 7 is a histogram based on data acquired by the rear camera 22. Fig. 8 is a histogram based on data acquired by the right camera 23, and Fig. 9 is a histogram based on data acquired by the left camera 24. In Figs. 6 to 9, the horizontal axis represents the flow length, and the vertical axis represents the occurrence value. The scale of the flow length is a unitless numerical value for the convenience of data processing to see the magnitude of the flow length. The same applies to Figs. 11 to 14 described below.

As shown in Figures 5 to 9, when vehicle MC turns left, the histograms of flow lengths acquired by each of cameras 21, 22, 23, and 24 are different. As shown in Figure 6, in the histogram from front camera 21, the occurrence value for flow length 30 is approximately 90, which is extremely large, followed by the occurrence value for flow length 40, which is approximately 60, and the others are small. As shown in Figure 7, in the histogram from rear camera 22, the occurrence value for flow length 30 is approximately 60, which is extremely large, and the occurrence values for flow lengths 20, 40, and 50 are around 20, which is less than half the occurrence value for flow length 30.

As shown in FIG. 8, in the histogram taken by the right camera 23, the occurrence values for flow lengths of 50 and 60 are particularly large. As shown in FIG. 9, in the histogram taken by the left camera 24, the occurrence value increases as the flow length increases from 10 to 40, with the occurrence value for a flow length of 40 being the largest, at 60. On the other hand, the occurrence values for flow lengths of 50 and 60 are small, at 10 or less.

As described above, it was found that the statistical distribution system of flow length differs for each of the cameras 21 to 24. It was also found that the accuracy is improved when estimation is performed using flows with a large occurrence value, i.e., long flows. Therefore, in this embodiment, these properties are utilized to appropriately set the reference range of flow length for each of the cameras 21 to 24. Furthermore, the reference range of flow length is set according to the driving situation, such as turning right or turning left. Note that driving situations such as turning right or turning left correspond to an example of the "moving state of a moving body."

In this embodiment, for example, for the front camera 21 (see FIG. 6), the reference range for the flow length is set to "25 to 45", near the most frequent values of 30 and 40. For the rear camera 22 (see FIG. 7), the reference range for the flow length is set to "25 to 35", near the most frequent value of 30. For the right camera 23 (see FIG. 8), the reference range for the flow length is set to "45 to 60", near the most frequent values of 50 and 60.

In other words, the reference range of flow length when turning left in the front camera 21 and the rear camera 22 is set based on the most frequent value of the flow length. "Based on the most frequent value" may be in the form of an arbitrary range based on empirical rules, with the most frequent value as the median, as in the above example, or it may be in the form of a value determined in advance by testing only above or below the most frequent value. In other words, it means any form of setting the reference range using the "most frequent value" as an indicator. Hereinafter, the same meaning is used for "based on the average value."

For the left camera 24 (see FIG. 9), the reference range for flow length is set to "25 to 45", close to the average value of 40. In other words, the reference range for flow length for the left camera 24 is set based on the average value of flow length. Whether to use the mode or the average value as an index for the reference range can be selected appropriately for each camera 21, 22, 23, 24 depending on the form of the histogram. In addition, the width of the allowable range from the mode or average value is set in advance based on prior test data, etc. Therefore, the specific numerical values of the above reference range can be changed as appropriate.

Below, an example of turning right will be described. Figure 10 dynamically shows the motion information of the vehicle MC, and is a diagram showing the positional changes of the cameras 21, 22, 23, and 24, and shows the form when turning right in an arbitrary environment. Figures 11 to 14 are histograms showing the statistical distribution of flow length. Figure 11 is a histogram based on data acquired by the front camera 21, and Figure 12 is a histogram based on data acquired by the rear camera 22. Figure 13 is a histogram based on data acquired by the right camera 23, and Figure 14 is a histogram based on data acquired by the left camera 24.

As can be seen by appropriately comparing Figures 6 to 14, the way objects are seen by each camera 21 to 24 differs when turning right and when turning left, so naturally the histograms of flow length are different for each. As shown in Figure 11, in the histogram taken by the front camera 21 when turning right, there is a dispersion of small occurrence values of 50 or less. When the histogram has this type of form, it is preferable to take the average value rather than the most frequent value. In this case, for example, for the front camera 21, the reference range of flow length is set to "15 to 25", close to the average value of 20.

On the other hand, as shown in FIG. 12, in the histogram taken by the rear camera 22, the occurrence value for a flow length of 20 is approximately 260, which is extremely high, and the others are small. When the histogram has this type of form, it is preferable to set the reference range for the flow length to "15 to 25", which is close to the most frequent value of 20. Similarly, as shown in FIG. 13 and FIG. 14, in the histograms taken by the right camera 23 and the left camera 24, it is preferable to determine the reference range for the flow length to be, for example, close to the average value.

As described above, histograms from each camera 21-24 in the current frame are obtained, and the reference range of the flow length is determined whenever necessary when estimating the self-position. In S105, the selection unit 15 selects, from among the created flows, those whose flow length between the current frame and the previous frame is within the reference range as correct flows. In other words, the reference range obtained from the histogram, which is a statistic of the flow lengths, is used as an index for selecting flows. The flow distribution changes depending on the moving speed and turning direction of the vehicle MC. Note that, in general, the higher the moving speed of the vehicle MC, the larger the reference range. Also, there is a tendency for the value in the camera installed opposite the turning direction of the vehicle MC to be larger.

Figures 15 and 16 show an example of an arbitrary current frame, illustrating feature points and flows within the frame. Figure 15 shows a frame captured by the front camera 21 during a left turn. Figure 16 shows a frame captured by the right camera 23 during a left turn. In each figure, the multiple white circles indicate the current feature points C, the black lines drawn from the feature points indicate the erroneous flow Fe, and the gray lines drawn from the feature points indicate the correct flow Fc. An "incorrect flow" is a flow that is determined not to satisfy the constraint conditions based on the constraint conditions determined in S105. A "correct flow" is a flow that is determined to satisfy the constraint conditions based on the constraint conditions determined in S105.

In this way, there is a difference between the correct flow Fc and the incorrect flow Fe. In the first embodiment, instead of using all of the flows drawn using the extracted current feature points for self-location estimation, the self-location estimation of the vehicle MC is calculated using the information of the correct flow Fc shown in Figures 15 and 16.

[effect]
(1) According to the self-position estimation device 1 of the first embodiment, the self-position of the vehicle MC can be estimated only from images captured by the camera 20 as a multi-camera, without using sensors or the like, and without creating a three-dimensional map. This can reduce the processing time in the self-position estimation device 1. In addition, since a three-dimensional map is not created, the memory resources in the self-position estimation device 1 can be reduced.

(2) In addition, the self-location estimation device 1 of the first embodiment does not create a three-dimensional map, and therefore cannot reflect depth information for the image in the current frame. However, in the self-location estimation device 1 of the first embodiment, the range in which feature points are expected to exist is set as a predetermined distance from the epipolar line, and a two-dimensional predicted range based on the epipolar line can be set as the search range S. This makes it possible to predict the range in which feature points exist and shorten the search processing time.

(3) According to the self-position estimation device 1 of the first embodiment described above, an upper threshold value that is predetermined as a predetermined distance from the epipolar line calculated based on the predicted motion of the vehicle MC is included as an index for selecting flows. Then, from among the flows created by the selection unit 15, flows whose position coordinates of the current feature point are within the range of the upper threshold value that is predetermined as a predetermined distance from the epipolar line are selected as correct flows. This makes it possible to improve the accuracy of self-position estimation of the vehicle MC.

(4) Furthermore, according to the self-location estimation device 1 of the first embodiment described above, a reference range obtained from a histogram as a statistic of flow length is included as an index for selecting flows. Then, the selection unit 15 selects, from among the flows created, those whose flow length is within the reference range as correct flows. This makes it possible to improve the accuracy of self-location estimation of the vehicle MC.

In addition, in the first embodiment, only flows that satisfy both the first and second conditions are selected as correct flows and used for self-location estimation, which further improves the accuracy of self-location estimation of the vehicle MC.

(5) According to the self-location estimation device 1 of the first embodiment described above, the second condition is determined individually and appropriately for each of the cameras 21 to 24 depending on the shape of the histogram based on the data acquired by each of the front, rear, left and right cameras 21 to 24. In other words, an appropriate reference range, such as near the mode or near the average value, is determined based on the shape of the histogram. This allows for more accurate flow extraction.

(6) According to the self-location estimation device 1 of the first embodiment described above, in S102, the fundamental matrix is created on the assumption of uniform motion, so that the fundamental matrix can be easily calculated.

(7) According to the self-location estimation device 1 of the first embodiment described above, if the number of flows that satisfy the constraint conditions is less than a reference value (S107) after the constraint conditions are determined (S105), the previous constraint conditions are relaxed and the determination is performed again (S108). This makes it possible to avoid the number of flows used for self-location estimation becoming excessively small.

B. Other embodiments:
(B1) The self-position estimation device 1 in the first embodiment described above is mounted on a vehicle MC, but it may also be mounted on other moving bodies such as ships, drones, autonomous mobile robots, or people.

(B2) In the self-location estimation device 1 of the first embodiment described above, the camera 20 is configured with four cameras on the front, back, left and right, but it is sufficient if it is a compound eye camera with two or more cameras.

(B3) In the self-position estimation device 1 of the first embodiment described above, in S102, a base matrix between the previous frame and the current frame is created on the assumption of uniform motion, but the motion prediction may use steering information from the CAN data of the vehicle MC in addition to the previous calculated value.

(B4) In the self-location estimation device 1 of the first embodiment described above, the mode or average value in the histogram is used to determine the reference range of the flow length, but the median value may also be used. For example, if there are flow lengths in the statistics that deviate significantly, appropriate measures can be taken, such as applying the median value instead of the average value.

(B5) The self-location estimation device 1 and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the self-location estimation device 1 and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the self-location estimation device 1 and the method thereof described in the present disclosure may be realized by one or more dedicated computers configured by combining a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program (self-location estimation program) may be stored in a computer-readable non-transient tangible recording medium as instructions executed by the computer.

The present disclosure is not limited to the above-described embodiments, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in each embodiment that correspond to the technical features in the form described in the Summary of the Invention column can be replaced or combined as appropriate to solve some or all of the above-described problems or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described in this specification as essential, it can be deleted as appropriate.

1... Self-position estimation device, 10... Vehicle control unit, 11... Image data acquisition unit, 12... Feature point extraction unit, 13... Storage unit, 14... Flow creation unit, 15... Selection unit, 16... Self-position calculation unit, 20... Camera, 21... Front camera, 22... Rear camera, 23... Right camera, 24... Left camera, MC... Vehicle

Claims (12)

A self-location estimation device that estimates a self-location of a moving body (MC),
an image data acquisition unit (11) that acquires image data output from a multi-camera (20) consisting of a plurality of cameras (21, 22, 23, 24) mounted on the moving body at set intervals;
a feature point extraction unit (12) that extracts feature points from the image data acquired by the image data acquisition unit;
a storage unit (13) for storing position information of each of the feature points in each of the image data;
a flow creation unit (14) that associates current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, and creates flows that are pairs of corresponding feature points in the image data that differ for each time series;
a selection unit (15) that selects, from among the flows created by the flow creation unit, a flow that satisfies a predetermined constraint condition, including an epipolar constraint calculated based on a predicted motion of the moving object, as a correct flow;
a self-position calculation unit (16) that calculates a position and an orientation of the moving object using information on the correct flow selected by the selection unit;
A self-location estimation device comprising: the constraint condition includes, as an index for selecting the flow, an upper limit threshold value that is a distance from an epipolar line calculated based on a predicted motion of the moving object, and
The self-location estimation device according to claim 1 , wherein the selection unit selects, from the created flows, flows in which position coordinates of the current feature point are within a range of the upper limit threshold from the epipolar line in the latest image data. the constraint condition includes a reference range of a flow length, which is a length of the flow in the image data that is continuous in a time series, as an index for selecting the flow;
The self-location estimation device according to claim 1 , wherein the selection unit selects, from the created flows, flows whose flow lengths are within the reference range. The self-location estimation device according to claim 3, wherein the reference range of the flow length is set individually for each of the multiple cameras. The self-location estimation device according to claim 3, wherein the reference range of the flow length is set according to the moving state of the moving body. The self-location estimation device according to claim 3, wherein the reference range of the flow length is set using statistics of the flow length. The self-location estimation device according to claim 6, wherein the reference range of the flow length is set based on the most frequent value in a frequency distribution of the flow length. The self-location estimation device according to claim 6, wherein the reference range of the flow length is set based on the average value in the frequency distribution of the flow length. The self-location estimation device according to claim 6, wherein the reference range of the flow length is set based on the median in the frequency distribution of the flow length. The self-location estimation device according to claim 1 or 2, wherein, when the number of flows selected as the correct flows is less than a predetermined criterion, the selection unit performs the selection of the flows again using constraint conditions changed from the constraint conditions previously used until the number of the correct flows becomes equal to or greater than the criterion. A self-location estimation method for estimating a self-location of a moving body (MC) using a self-location estimation device, comprising:
The self-position estimation device includes an image data acquisition unit (11), a feature point extraction unit (12), a storage unit (13), a flow creation unit (14), a selection unit (15), and a self-position calculation unit (16),
acquiring image data output from a multi-camera (20) consisting of a plurality of cameras (21, 22, 23, 24) mounted on the moving body at set intervals by the image data acquisition unit;
extracting feature points from the image data acquired by the image data acquisition unit using the feature point extraction unit;
storing position information of each of the feature points in each of the image data by the storage unit;
a step of associating current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, by the flow creation unit, to create a flow, which is a pair of corresponding feature points in the image data that differs for each time series;
a step of selecting, by the selection unit, from among the flows created by the flow creation unit, a flow that satisfies a predetermined constraint condition, the constraint condition including an epipolar constraint calculated based on a predicted motion of the moving object, as a correct flow;
calculating, by the self-location calculation unit, a position and an orientation of the moving object using information on the correct flow selected by the selection unit;
A self-location estimation method comprising: A self-location estimation program for estimating a self-location of a moving object (MC),
A function of acquiring image data output from a multi-camera (20) consisting of a plurality of cameras (21, 22, 23, 24) mounted on the moving body at set intervals;
A function of extracting feature points from the image data;
a function of storing position information of each of the feature points in each of the image data;
a function of associating current feature points, which are feature points in the latest image data, with previous feature points, which are feature points in past image data, to generate a flow, which is a set of corresponding feature points in the image data that differs for each time series;
a function of selecting, from among the created flows, a flow that satisfies predetermined constraint conditions, including an epipolar constraint calculated based on a predicted motion of the moving object, as a correct flow;
A function of calculating a position and an orientation of the moving object using the information of the selected correct flow;
A self-location estimation program that enables a computer to achieve this.

Images