HK1119460B - Imaging position analyzing method - Google Patents

Description

Photographic position analysis method

Technical Field

The present invention relates to a technique for analyzing a photographing position of each frame of an image composed of a plurality of frames, such as a moving image captured while moving.

Background

Various proposals have been made for the use of images captured by a camera mounted on a vehicle. For example, information obtained from these images can be applied to electronic map data for generating a three-dimensional map for providing a scene in which streets are reproduced using a three-dimensional image of a computer graphic. In contrast, japanese patent application laid-open No. 7-37065 (hereinafter referred to as patent document 1) discloses a technique for synthesizing images of frames captured by a camera to generate a wide-area image for use in monitoring the state of a railway line or a power transmission line.

In order to apply the image captured by the camera to the above-described purpose, it is necessary to detect the position where each frame image is captured, for example, the latitude, longitude, and the like, with high accuracy in advance. In this regard, japanese patent No. 2687645 (hereinafter referred to as patent document 2) and japanese patent application laid-open No. 7-71973 (hereinafter referred to as patent document 3) disclose techniques for determining the position of a vehicle in the traveling direction using known distances such as the number of intermittent white lines provided as partition lines of a passing belt and the number of guide rail stays. In addition, a technique has been proposed in which a GPS (Global Positioning System) and other direction sensors such as a gyroscope are used together for position detection.

Patent document 1: japanese laid-open patent publication No. 7-37065

Patent document 2: japanese patent No. 2687645

Patent document 3: japanese laid-open patent publication No. 7-71973

However, in the conventional technique, the accuracy of the imaging position of the image is not necessarily sufficiently high. The imaging spot must be positioned by using at least two-dimensional coordinate values such as latitude and longitude. The techniques described in patent documents 2 and 3 can improve the positional accuracy in the traveling direction of the vehicle, but sufficient consideration is not given to the direction intersecting the traveling direction. Further, since the position detected by the GPS includes an error of about several tens of meters, it cannot be said that the position has sufficient accuracy to be applied to various analyses using image data. The same applies to the position detected by a gyroscope or the like.

Further, since the GPS includes an error in time, even if the position detection error is increased, the GPS cannot be completely synchronized with the captured image, and as a result, the position at the time of capturing cannot be determined with sufficient accuracy. That is, the position specified by the GPS at a certain time may be a position indicating a time different from the certain time. Therefore, when the position is detected by the GPS while moving, the obtained information cannot represent the imaging position with sufficient accuracy even if the position detection accuracy of the GPS is improved.

In an image with low accuracy of the imaging position, even if the resolution of the image itself is increased, analysis cannot be performed that makes full use of the resolution of the image data. For example, if the imaging position is incorrect, the shape and position of a road sign, a representation on a road, a building, or the like imaged on the image cannot be determined with high accuracy, and cannot be reflected in the map data. In addition, in the case of image synthesis as in the technique described in patent document 1, if the positional accuracy is insufficient, the images of a plurality of frames are shifted from each other, and a high-definition synthesized image that utilizes the resolution of the original image sufficiently cannot be obtained.

Such a problem may occur not only when the vehicle is mounted on the vehicle to capture an image, but also when an image captured while moving, such as an image captured while walking, is captured. And not limited to moving images, the same may occur for still images taken at a plurality of different locations. In view of the above problems, an object of the present invention is to accurately specify a photographing position of each frame of an image composed of a plurality of frames photographed while moving.

Disclosure of Invention

The present invention may be configured as a photographing position analyzing device (hereinafter, also simply referred to as "analyzing device") that analyzes a photographing position of each frame with respect to an image composed of a plurality of frames. The imaging position analysis device takes as a processing target an image captured at a known timing while moving in a state in which an attitude angle relative to a ground surface is substantially constant. Examples of such an image include a moving image captured while a vehicle in which the imaging device is fixed at a certain attitude angle moves. The moving image may be captured while walking as long as the posture angle can be maintained substantially constant. It is also not necessary to move the image, and still images taken at a plurality of positions may be used. Each frame of the image includes a predetermined continuum captured together with at least one immediately preceding frame. The continuum is not necessarily common to all frames to be processed. The continuum may be, for example, a section line of a traffic belt on a road.

The analyzing device inputs the image data composed of a plurality of frames. Then, as an initial value of the analysis, an initial trajectory of the movement at the time of photographing is input. Each frame image can be arranged on the initial trajectory assuming that the image is captured while moving on the initial trajectory. In this way, the analysis device can temporarily set the imaging position of each frame along the moving direction of the initial trajectory in accordance with the imaging timing. Since the initial trajectory includes an error with respect to the imaging position, the images of the continuous body shift between frames arranged on the initial trajectory.

The analyzer detects a shift between the captured images of the continuous object captured across a plurality of frames by image processing, and corrects the temporarily set imaging position in the direction intersecting the movement of the initial trajectory based on the shift to analyze the imaging position of each frame. For example, when the position of the continuum is shifted to the right in the next frame compared to the previous frame, the temporarily set imaging position is corrected to be shifted to the left in accordance with the shift amount. On the other hand, when the image is shifted to the left, the temporarily set imaging position is corrected to be shifted to the right. The imaging position analyzing apparatus of the present invention determines an error in a direction crossing a movement with respect to an initial trajectory by analyzing an image of a continuous body to be imaged, and reflects the error, thereby accurately detecting a movement trajectory (hereinafter referred to as an "actual trajectory") at the time of actual imaging or accurately determining an imaging position of each frame. This method uses image analysis for determination of the actual trajectory, and therefore has an advantage that position accuracy in accordance with the resolution of the image can be achieved. Further, if this method is employed, it is assumed that even if the actual trajectory includes a positional error, the matching between the images of a plurality of frames can be sufficiently maintained, and therefore there is an advantage that the accuracy required for analysis such as generation of map data using images of a plurality of frames can be ensured.

The image may include a part of the image that is forward or rearward in the moving direction, and for example, an image captured by a camera installed diagonally forward or diagonally rearward may be used. As long as an image is captured at a sufficiently large wide angle, an image captured by a camera facing the front side with respect to the moving direction may be used. However, in view of being able to detect the deviation in the movement intersecting direction most effectively and with high accuracy, it is preferable that the image be an image of the front or back side in the movement direction. In addition, in order to analyze the imaging position, a part of the lower part may be used instead of the entire image captured in this way. This is because the lower part of the image is considered to be a place relatively close to the imaging position, and therefore, it is advantageous to improve the analysis accuracy of the imaging position.

A continuum for determining the actual trajectory may employ various objects captured across multiple frames. Although a large vehicle such as a bus or a truck passing from the vicinity may be used, it is necessary to ensure that the vehicle does not move in the cross-travel direction. The continuum is preferably an object fixed to a road in view of ensuring that the continuum does not move in the direction of movement crossing, and for example, a guide rail on the road side, an edge of a building, or the like may be regarded as the continuum. It is preferable that the continuum be a section line of a traffic lane of a road if it is possible to recognize an image relatively easily and perform the image with high accuracy.

In the present invention, the configuration of the frame image with respect to the initial trajectory may employ various methods. For example, movement distance information indicating a relationship between a time at the time of photographing and a movement distance along the movement direction may be input, and the photographing position of each frame along the movement direction may be temporarily set based on the movement distance information. Since the imaging time of each frame is known, the imaging position of each frame on the initial trajectory can be determined with high accuracy by using the movement distance information, and the accuracy of the two-dimensional coordinates of the finally obtained imaging position can be improved.

Further, frames photographed at predetermined moving distances may be extracted from the plurality of frame data based on the moving distance information as frames for analyzing the photographing position. In such a mode, since the moving distances between the extracted frames are equal, there is an advantage that image processing such as combining a plurality of frames is easily performed. In such a state, it is desirable that the number of frames shot per unit time (hereinafter referred to as "frame rate") is sufficiently high, whereby it is possible to ensure that frames exist for every prescribed moving distance. The frame rate to be requested is determined by the moving speed at the time of photographing and the moving distance as a reference at the time of frame extraction. For example, when photographing is performed by a photographing device mounted on a vehicle that moves at a speed of about the speed limit of a general road, the above-mentioned requirement can be satisfied if the moving image has a frame rate of about 30 frames/second.

As the travel distance information, image information obtained by imaging a target object whose interval is known, such as a partition line drawn intermittently on a road or a pillar of a guide rail, may be used. When an image is captured by an imaging device mounted on a vehicle, a vehicle speed pulse of the vehicle, that is, a pulse signal output every time the vehicle travels a certain distance may be used as the travel distance information.

When performing imaging, it is preferable to record reference position information indicating the time when a known reference position such as an intersection is reached in advance in association with image data. By doing so, the analysis device can process the imaging position at the time corresponding to the reference position information (hereinafter referred to as "reference position") as a known position. Therefore, at least the position along the moving direction can be initialized based on the reference position information in the analysis process, and the accuracy of estimating the imaging position can be improved.

The reference position information can be utilized in various forms. For example, the frames may be arranged in the order of shooting with the reference position as the starting point. That is, the frames may be arranged in time series along the direction of movement during photographing. Conversely, the frames may be arranged in the order reverse to the order of shooting, starting from the reference position. That is, the frames may be arranged in the reverse order of the time series in the direction opposite to the moving direction at the time of photographing. In any arrangement method, the closer to the start point, the higher the frame arrangement accuracy.

For example, in a navigation system, a situation is considered in which a captured frame image or a graphic generated based on the captured frame image is displayed in accordance with the position of a vehicle. In the latter state, that is, in a state where the reference position is arranged in reverse order of the time series with the reference position as the start point, the closer the vehicle is to the reference position, the higher the positional accuracy of the provided image. In the case of using the intersection as the reference position, if it is considered that the vehicle stops in front of the intersection or turns at the intersection, it is preferable that the position accuracy of the image be higher as the distance from the intersection is closer. In this sense, it can be said that the latter state described above is more useful for generation of data for navigation.

When processing an image obtained by imaging a road on which a plurality of opposite pass belts are provided, it is useful to image only one pass belt (usually, a pass belt that moves during imaging) in a state where frame images and the like are arranged in reverse order of time series starting from a reference position. On the other hand, a method of sequentially arranging frame data in both the moving direction and the reverse direction with the reference position as a starting point for capturing images of the traffic zones on both sides is useful.

The initialization of the imaging position may be performed by the following method. First, the lateral image data of a plurality of frames is photographed in the moving cross direction. As the lateral image data, for example, a photographed image photographed by a camera provided laterally to the traveling direction of the vehicle may be used. It is assumed that the position coordinates of the subject of such lateral image data are already known by referring to the map data. The imaging position analyzing device calculates subject coordinates indicating the position of the subject from the horizontal image data of the plurality of frames. Since the horizontal image data of a plurality of frames corresponds to image data obtained by imaging the object from a plurality of imaging positions, if the moving speed and the imaging time of each frame are known, the distances between the plurality of imaging positions are known, and therefore the position of the object can be determined based on the imaging positions based on the principle of triangulation. The deviation between the coordinates of the object thus obtained and the position coordinates recorded in the map data indicates an error in the imaging position used when the coordinates of the object are obtained. Accordingly, the imaging position can be initialized, that is, error correction can be performed based on the deviation.

The initial trajectory used in the present invention is used as an initial value of analysis using a captured image, and therefore, may be a trajectory that indicates an approximate situation of a captured trajectory. For example, when road network data indicating roads with nodes and links can be referred to, a trajectory that passes during imaging may be specified with nodes and links, and the initial trajectory may be set from the road network data by this method or the like. Even when the road network data includes the height information of the road, the initial trajectory can be three-dimensionally determined.

The initial trajectory may be set using the output of the position detection sensor. The position detection sensor may be a device capable of detecting a movement locus in an image in at least two dimensions within a predetermined error range, such as a gyroscope, a distance meter, and a GPS. Here, the allowable error is preferably within a range capable of correcting an error in a direction intersecting with the initial trajectory by image processing. Thus, the allowable error is preferably a magnitude that can make the deviation between the initial trajectory and the actual trajectory converge within the angle of view of the photographing device.

In the present invention, before the analysis of the imaging position, the image data may be converted into an image in a state where the continuous body is imaged from the front. Various methods such as affine transformation can be used for image transformation. Image data can be distributed to a plurality of regions, and different transform coefficients are used for each region, whereby the accuracy in transformation can be improved. For example, the plurality of regions and the transform coefficients may be set so that an orthographic image of a mesh body having a known shape can be obtained from image data obtained by imaging the mesh body.

The present invention may be configured as an image data acquisition apparatus that generates image data used for the analysis. The image data acquiring device may be a device including, for example, a vehicle moving on the ground surface, a photographing device, and a moving distance information recording unit. The imaging device is attached to the vehicle in a state where the relative attitude angle is substantially constant, and captures an image, a moving image, or a still image formed of a plurality of frames at a known timing. Of course, a plurality of imaging devices may be mounted. The travel distance information recording unit records travel distance information indicating that the vehicle has moved a predetermined distance, in association with the time when the image is captured. As the travel distance information, for example, a vehicle speed pulse generated by the vehicle may be used. In so doing, it is possible to provide the analysis apparatus with image data and movement distance information suitable for analysis.

The present invention does not necessarily have all of the features described above, and some of them may be omitted or appropriately combined. The present invention may be configured as an analysis method for analyzing the imaging position by a computer, in addition to the state of the imaging position analysis device and the image data acquisition device. Further, the present invention may be configured as a computer program for realizing such analysis, or may be configured as a recording medium on which such a computer program is recorded. In this case, various computer-readable recording media such as a flexible disk, a CD-ROM, a magneto-optical disk, an IC card, a read only memory cartridge (ROM cartridge), a punched card, a printed matter on which a symbol such as a barcode is printed, an internal storage device of a computer (a memory such as a RAM or a ROM), and an external storage device can be used as the recording medium.

Drawings

Fig. 1 is an explanatory diagram showing a configuration of an image data processing system as an embodiment.

Fig. 2 is an explanatory diagram showing an example of mounting and connection of each device constituting the image data acquisition device 100.

Fig. 3 is an explanatory diagram showing a configuration of image data and the like.

Fig. 4 is an explanatory diagram illustrating a method of setting the initial trajectory.

Fig. 5 is an explanatory diagram illustrating a method of setting an initial trajectory as a modification.

Fig. 6 is an explanatory diagram illustrating the principle of the feature point tracking process.

Fig. 7 is a process diagram of a transform coefficient setting method.

Fig. 8 is an explanatory diagram showing an example of the feature point tracking process.

Fig. 9 is a flowchart of the imaging position analysis processing.

Fig. 10 is a flowchart of the feature point tracking process.

Fig. 11 is a flowchart of a feature point tracking process as a modification.

FIG. 12 is a flow chart of the tag/label extraction process.

Fig. 13 is an explanatory diagram showing one example of processing of image data.

Fig. 14 is an explanatory diagram showing a second example of processing of image data.

Fig. 15 is an explanatory diagram showing a frame data arrangement method as a modification.

Fig. 16 is an explanatory diagram showing a method of obtaining the reference position using the side image.

Fig. 17 is a flowchart of a reference position calculation process of the modification.

Fig. 18 is a flowchart of the temporal change determination process.

Fig. 19 is a flowchart of the guide plate position coordinate analysis process.

Detailed Description

The examples of the present invention are explained in the following order.

A. Device structure

B. Data structure

C. Principle of analysis of photographic position

C-1. initial trajectory

C-2. characteristic point tracing process

D. Imaging position analysis processing

E. Mark and mark extraction process

F. Example of treatment

G1. Modification example-frame data arrangement method

G2. Modification example side image utilization

G3. Modification example-determination of temporal change

G4. Modification example-analysis of position coordinates of guide plate

A. Device structure

Fig. 1 is an explanatory diagram showing a configuration of an image data processing system as an embodiment. An image data processing system is a processing system for moving images captured while moving on a street or the like. As the processing contents, the 1 st one is a photographing position analysis process of analyzing each frame constituting a moving image to obtain photographing position coordinates such as longitude and latitude. As the data for generating the three-dimensional map data, there is a process (this process is referred to as "image synthesis process") of generating a synthetic image for each frame using the result, and specifying the type and position of a marker on the road surface and a marker at the road boundary from the synthetic image (this process is referred to as "marker/marker extraction process"). The moving image of the photographing position analyzed by means of the 1 st process can also be used for the measurement of the height and the frontal width of buildings at road boundaries.

The image data processing system is constituted by an image data acquisition apparatus 100 for capturing moving images and an image data processing apparatus 200 for processing moving images. In the present embodiment, the two are configured independently, but may be configured as one device. The image data processing apparatus 200 may be a distributed processing system including a plurality of apparatuses.

The image data acquisition apparatus 100 according to the embodiment is configured by mounting various apparatuses on a vehicle. The vehicle is equipped with a camera 120 for taking a front image and a camera 122 for taking a side image. The cameras 120 and 122 are fixed in a state capable of maintaining a certain attitude angle with respect to the vehicle. In order to obtain a wide-range, high-definition image with high efficiency, the cameras 120 and 122 are preferably wide-angle cameras using high definition.

The image of the camera 120 is used for the shooting position analysis processing as described below. From such a viewpoint, it is preferable that the camera 120 is installed along the front-rear axis of the vehicle at an attitude angle parallel to the ground surface during traveling. However, such an attitude angle is not essential. In the shooting position analysis processing, since it is sufficient that there is an image in which a part of the image is shot in the front or rear direction of the vehicle, the camera 120 may be attached to the rear, diagonally front, diagonally rear, or the like, or may be attached to the front side as long as it can shoot at a sufficiently large wide angle.

The side image is used for applications such as identification, extraction of marks, measurement of the height and the front width of a building, and the like. Therefore, the number of cameras 122 may be increased or the installation direction of the cameras may be determined according to the type and purpose of the application.

The vehicle is provided with a hard disk 114 for storing the acquired data as digital image data, and a control unit 110 for controlling the storage and the like. The control unit 110 may be configured by installing a computer program for acquiring and managing image data on a general-purpose computer, for example.

A GPS (Global Positioning System) 102 periodically detects the longitude and latitude, which is the position information of the vehicle at the time of imaging, and outputs the detected longitude and latitude together with the detection time. This data is recorded on various hard disks 114 together with image data. Of course, as described below, the GPS102 may be omitted because the output of the GPS102 is not data necessary for analysis of the imaging position. The vehicle speed sensor 104 outputs a signal called a vehicle speed pulse every time the vehicle moves a certain distance. The vehicle speed pulse is also recorded in the hard disk 114 together with the image data.

In the present embodiment, in order to improve the accuracy of analysis of the imaging position, the time when the imaging position passes through a reference position, that is, a position whose latitude and longitude are known, and the information of the reference position are recorded. In order to perform the recording in accordance with the instruction of the operator, a reference position input unit 112 is provided on the vehicle. The reference position input unit 112 can be configured by installing a computer program for realizing a reference position input function on a general-purpose computer, as in the control unit 110.

In the present embodiment, if the operator clicks a reference position to be recorded on the map 132d displayed on the display with a mouse or the like, information on the time of the click and the reference position can be recorded. The map data required for displaying the map may be recorded in advance on a recording medium such as the hard disk 114 or a CD-ROM, or may be acquired from an external server via a wireless network. The input of the reference position information is not limited to the above-described method, and information such as latitude and longitude may be directly input from a keyboard or the like, and when a reference position to be recorded is set in advance, a code corresponding to each reference position may be input. The input of the reference position information may be omitted at the time of photographing, and only the time when the image passes may be input.

In the present embodiment, the control section 110 and the reference position input section 112 are realized in a software manner by installing a computer program, but these components may be configured in a hardware manner by using a dedicated circuit.

The configuration of the image data processing apparatus 200 will be explained below. The image data processing apparatus 200 is configured by installing a computer program for image data processing on a general-purpose computer. The detachable hard disk 114a is used for data transfer from the image data acquisition apparatus 100 to the image data processing apparatus 200. However, the present invention is not limited to this method, and other recording media such as DVDs may be used, or transmission may be performed via a network.

By installing the computer program, the image data processing apparatus 200 configures various functional blocks shown in the drawing. Of course, at least a part of these functional blocks may be configured in hardware by an ASIC or the like.

The data input unit 206 inputs image data generated by the image data acquisition apparatus 100. As described above, the latitude and longitude detected by the GPS102, the vehicle speed pulse, and the reference position information are also input together with the image data. This information is delivered to the initial trajectory setting unit 204 and the cutout image generation unit 208.

The initial trajectory setting unit 204 generates an initial trajectory for the imaging position analysis process described below. The command input unit 202 inputs a command required for generating an initial trajectory by an operation of a mouse or a keyboard by an operator. Although the command input by the operator is appropriately transferred to a function block other than the initial trajectory setting unit 204, in order to avoid complication of the drawing, only the state of transferring to the initial trajectory setting unit 204 having the highest degree of association is shown in the drawing.

In this embodiment, two generation methods may be adopted for setting the initial trajectory. One method is a method using the latitude and longitude detected by the GPS 102. Another method is a method of generating road network data without using the detection result of the GPS 102. Road network data is data used for route search, and is data representing a road by means of a link representing a passing point of the road as a polygonal line, a node representing an intersection or an end point of the link, and attribute information of the link and the node. In the present embodiment, the information is stored as the network database 220 in the image data processing apparatus 200. The network database 220 may be provided by a recording medium such as a CD-ROM or an external server connected via a network.

The image arrangement unit 210 specifies the position of each frame shot in the image data along the set initial trajectory. Since each frame can be arranged on the initial trajectory by using this position, the process of determining the imaging position is sometimes referred to as "arrangement" in this specification. The position specified here includes an error, and is an initial value of the imaging position analysis processing. The vehicle speed pulse and the reference position information are used in this processing. The contents of the processing will be described below.

In the present embodiment, only a part of the input image data is used in the photographing position analysis processing. To realize this processing, the cutout image generation unit 208 performs a process of cutting out a portion used for the imaging position analysis processing from each frame of the input image. In this case, it is preferable to correct various distortions included in an image captured at a wide angle by affine transformation or the like. The image cutout is not essential, and when the imaging position analysis processing is performed using all of the frame images, the cutout image generation unit 208 may be omitted, or the cutout image generation unit 208 may be caused to perform only the correction of the image distortion.

The trajectory correction unit 212 arranges a plurality of cutout images on the initial trajectory, and corrects the initial trajectory so that continuity is maintained between the cutout images in accordance with image processing performed on the cutout images. The contents of this process will be described below. By correcting the initial trajectory in this manner, the correct imaging position of each frame can be obtained. The image data processing apparatus 200 can also output the image capturing position obtained by the trajectory correction unit 212 and complete the processing of the image data.

The image data processing apparatus 200 of the present embodiment can perform the identification, display extraction processing using the processing result described above. The marker/display extraction process is a process of generating a composite image in which images of a plurality of frames are combined, and determining the shape and position of a marker on a road, a marker at a boundary of the road, and the like from the composite image. This processing is realized by the logo/display extraction unit 214. Examples of the sign or indication for specifying the object include an indication such as a crosswalk or an arrow indicating a restriction on a traveling direction, a signal, a road sign, and a street tree. The approximate shape and color of these markings and indicia are predetermined. The identification and designation database 222 stores its approximate shape and color as a basic pattern.

The marker/logo extracting unit 214 extracts an image corresponding to the basic pattern stored in the marker/logo database 222 from the composite image, deforms the basic pattern, determines an accurate marker/logo shape suitable for the composite image, and determines the position of the marker/logo shape. The image data processing apparatus 200 manages the shape and position thus set as the marker/index position data 224. The identification, index position data 224 may be used in the generation of a three-dimensional map with high authenticity, or the like.

Fig. 2 is an explanatory diagram showing an example of mounting and connection of each device constituting the image data acquisition device 100. Each of the devices shown in the drawings can be detachably mounted to a vehicle. The power supply of each device is used after alternating-current conversion of a direct-current power supply drawn from a battery of a vehicle through a lighting jack 110b by a DC-AC vehicle inverter 110 c. The functions of the control unit 110 and the removable hard disk 114a shown in fig. 1 are realized by a notebook computer 110 a. The notebook computer 110a inputs the detection signal of the GPS 102. The antenna 102a of the GPS102 is provided at a position where it can receive radio waves of the GPS.

In the configuration example in the figure, 3 cameras 120a, 120b, and 120c for photographing the front side are provided. The camera 120b is a dedicated device for guiding the shooting direction. The cameras 120a, 120b, and 120c are connected to the notebook computer 110a via the IEEE interface 200a, and time codes indicating the shooting times of the respective frames are transmitted from the cameras 120a, 120b, and 120c to the notebook computer 110 a. The notebook computer 110a records the time code in association with the vehicle speed pulse and the reference position information in advance for analyzing the imaging position.

The cameras 122R and 122L are cameras for capturing images in the right and left directions of the vehicle. The audio inputs of the cameras 122R, 122L are connected to the audio output of the notebook computer 110 a. When the notebook computer 110a outputs a predetermined sound pulse signal in accordance with the vehicle speed pulse, the sound pulse signal is recorded on the sound tracks of the cameras 122R and 122L, and therefore each frame captured during analysis can be associated with the vehicle speed pulse. In addition to the illustrated connection examples, a connection method using audio input/output may be applied to the cameras 120a, 120b, and 120 c. A connection method using an IEEE interface may be applied to the cameras 122R and 122L.

The pulse generator 104a for detecting the vehicle speed magnetically detects the rotation of the rear wheel of the vehicle and generates a pulse synchronized with the rotation. For example, ND-PG1 (trademark) manufactured by McKer K.K. (パイオニア Co.) can be used. The pulse counter 104b counts the generated pulses and outputs the count result together with the time. For example, TUSB-S01CN1 (trade mark) manufactured by タ - トル, Inc. can be used. In the present embodiment, the pulse generator 104a and the pulse counter 104b are respectively provided in the trunk at the rear of the vehicle.

If the configuration shown in the figure is adopted, the image data acquisition apparatus 100 can be constituted by combining commercially available apparatuses. Moreover, since the device is detachably attached to the vehicle, the device is easily moved. For example, there is an advantage that each device shown in the drawings is transported to a measurement site by a train, an airplane, or the like, and measurement can be easily performed at the measurement site by a vehicle.

B. Data structure

Fig. 3 is an explanatory diagram showing a configuration of image data and the like. This represents the relationship between the frame data constituting the moving image captured by the camera 120 and the vehicle speed pulse and the reference position pulse. The data group T on the upper side of the figure represents a state in which the various data are arranged on a time basis.

As shown in the data group T, the frame data Fr1 to Fr10 are acquired at regular time intervals. In this example 30 Hz. Of course, the frame data may be a set of still images captured at arbitrary timing. In this case, the time interval may also be indefinite.

The vehicle speed pulses P1-P6 are obtained every time the vehicle moves for a certain distance. In the present embodiment, the vehicle speed pulse is taken every about 0.39m of movement. The data group T is arranged on a time basis, and therefore the interval of the vehicle speed pulses varies in accordance with the moving speed of the vehicle. For example, the interval between the pulses P1, P2 is narrow because the moving speed is relatively large. On the other hand, the interval between the pulses P2 and P3 is relatively wide because the moving speed is relatively slow.

The reference position pulse is acquired at a timing when the vehicle passes a predetermined reference position such as a crosswalk. Since the reference position pulse is acquired at a time when the vehicle moves a certain distance, the frequency of acquisition is relatively low compared to the frame data and the vehicle speed pulse. As described below, the reference position pulse is used as an initial position to improve the accuracy of the imaging position analysis processing, and therefore, the frequency may be low.

The lower data group R represents a state in which the respective data shown in the data group T are arranged based on the moving distance at the time of photographing. Since this is the movement distance reference, the vehicle speed pulses P1 to P6 are arranged at equal intervals as shown in the figure. The frame data Fr1 to Fr8 are arranged on the assumption that the vehicle moves at a constant speed between the vehicle speed pulses. As a result, for example, the frame data Fr2 is arranged according to the following rule.

t1∶t2＝r1∶r2

time between a vehicle speed pulse P1 and a frame Fr2 in a data group T

T2.. time between vehicle speed pulse P2 and frame Fr2 in data group T

Distance between vehicle speed pulse P1 and frame Fr2 in data group R

r2.. distance between vehicle speed pulse P2 and frame Fr2 in data group R

The other frame data and reference position pulses are also the same. By doing so, as shown in the data group R, each frame data can be arranged along the trajectory at the time of photographing, that is, the position in the direction along the trajectory can be specified.

The position of the frame data may be determined by various methods other than the illustrated method. For example, when frame data is acquired at a sufficiently high frequency compared to the vehicle speed pulse, frame data synchronized with the vehicle speed pulse may be extracted from the frame data. By doing so, a group of frame data acquired at equal intervals can be generated. In the case of extracting frame data synchronized with the vehicle speed pulse, an error of a predetermined range between the times of the two may be allowed in consideration of the accuracy required for the analysis of the imaging position.

C. Principle of analysis of photographic position

The principle of the imaging position analysis processing will be described below by taking as an example a case where a group of frame data acquired at equal intervals is generated by extraction of frame data. However, the same processing may be used even when the frame data is arranged at an indefinite interval as in the data group R of fig. 3.

In the imaging position analysis processing of the present embodiment, first, an initial trajectory indicating a movement trajectory at the time of imaging is set within a certain error range. Then, the initial trajectory is corrected by image analysis processing using frame data, which is called feature point tracking processing, to obtain the imaging position of each frame data. First, the method of setting the initial trajectory and the sequence of the feature point seek processing will be described.

C-1. initial trajectory

Fig. 4 is an explanatory diagram illustrating a method of setting the initial trajectory. In this example, the initial trajectory is set based on latitude and longitude data acquired using the GPS 102. Assume that a trajectory passing from arrow Ar1 to Ar2 through the intersection point including roads R1 and R2 is used at the time of photography. The black triangles in the figure represent latitude and longitude data obtained by the GPS 102. The latitude and longitude of the GPS102 may be interpolated by using a gyroscope or the like. White dots indicate groups of frame data obtained at equal intervals. The white double dots CP1 and CP2 indicate points at which the reference position pulses are acquired. As described above, in the present embodiment, it is assumed that the basic position pulses are acquired on the crosswalks CW1 and CW 2.

The initial trajectory is set by sequentially connecting the latitude and longitude obtained by the GPS 102. Each frame data is arranged at equal intervals on the initial trajectory with reference to the point at which the reference position pulse is acquired, and the imaging position of each frame data is obtained as an initial value of the imaging position analysis processing as shown in fig. 4. Since the latitude and longitude obtained by the GPS102 include errors, the initial trajectory shown in fig. 4 also includes an error in the moving direction during shooting (hereinafter simply referred to as "moving direction") and an error in the direction intersecting the moving direction (hereinafter simply referred to as "intersecting direction"). However, since the reference position pulse is arranged at equal intervals from the point where the pulse is obtained, the error in the moving direction is considered to be sufficiently small, and mainly includes the error in the intersecting direction.

Fig. 5 is an explanatory diagram illustrating a method of setting an initial trajectory as a modification. This shows an example in which the initial trajectory is set without using the latitude and longitude obtained by the GPS 102. In a modification, road network data is used instead of the latitude and longitude. Broken lines L1, L2 shown in the figure represent links corresponding to roads R1, R2, respectively. The black dots N1 are nodes, and the road network data is data indicating a road by links and nodes as described above. In fig. 5, the links are shown as straight lines, but some roads may be shown as broken lines. The links are specified by the latitude and longitude of each passing point and end point. There are also cases where altitude information is included in addition to the latitude and longitude.

In the modification, a link of a road through which the image is taken is used as an initial trajectory. In the illustrated example, when the route from the road R1 to the road R2 is used for photographing, the links L1 and L2 are used as initial trajectories. Each frame data is arranged at equal intervals on the initial locus with reference to a point corresponding to the reference position pulse. In the method of the modification, although the initial trajectory may be in a state of being cut as shown in the vicinity of the node N1, the trajectory is corrected by the following feature point tracking process to obtain a continuous trajectory, and therefore, there is no problem.

The initial trajectory of the modification also includes an error with respect to the actual imaging position. However, as in the case of the example using the output of the GPS102 (see fig. 4), it is considered that the error in the moving direction is relatively small, and the error mainly includes the error in the crossing direction.

C-2. tracing process of characteristic point

Fig. 6 is an explanatory diagram illustrating the principle of the feature point tracking process. Circles Pt1 to Pt4 of the broken lines shown in the center of the figure indicate positions where frame data are arranged on the initial trajectory. The direction from the circle Pt1 to Pt4 is the moving direction. Images Pic1 to Pic4 indicated by frame data corresponding to respective positions are illustrated on the right side of the figure. These images are images in which a part of the lower side of the captured image is cut out. For example, the image Pic1 is a cut-out image of a lower area indicated by a broken line in the original image Por shown below. The other images Pic2 to Pic4 are also cut-out images from the bottom in the same manner. The image thus cut out from the original image is hereinafter referred to as a cut-out image.

In the feature point tracking process, an error of an initial trajectory is determined based on the positions of the feature points included in the images, and the trajectory is corrected. In the present embodiment, the section line of the vehicle passage zone of the road is used as the feature point. In the example shown in the figure, the solid line shown in the lower center of the original image Por indicates a partition line.

The centers of gravity of the images Pic1 to Pic4 are arranged so as to coincide with the initial trajectories Pt1 to Pt4, respectively. At this time, the feature points are sequentially shifted as indicated by a broken line FP in the figure. If the initial trajectories Pt1 to Pt4 correctly indicate the trajectories during shooting, the characteristic points should not be shifted between the images. That is, the shift of the feature point indicates a case where an error is included in the initial trajectory. For example, when the offset OS between the images Pic1 and Pic2 is considered with reference to the initial trajectory position Pt1, the error in the crossing direction of the position Pt2 is OS. Therefore, if the position Pt2 is moved in the cross direction by "-OS", an accurate trajectory can be obtained. The position thus obtained is a circle Pc2 shown by a solid line in the figure.

Similarly to the other positions Pt3 and Pt4, the accurate positions Pc3 and Pc4 on the trajectory can be obtained by obtaining the shift amount of the feature point between the adjacent images and correcting the position in the intersecting direction based on the shift amount. In the illustration, by this processing, the locus of the solid line passing through the positions Pt1, Pc2 to Pc4 can be obtained. The left side of the figure shows an example in which the centers of gravity of the images Pic1 to Pic4 are arranged to come on the locus of the solid line. As shown, the shift of the feature points between the images is eliminated.

Fig. 6 shows an example of using an image simply cut out from an original image. The distortion of the lower part of the image may be corrected by affine transformation or the like before the feature point tracing processing. In the case of using the image data captured by the wide-angle camera as in the present embodiment, it is preferable to perform correction. Since an image corresponding to the state of the road surface captured from directly above the road can be obtained by such correction, the accuracy of analysis of the captured position obtained by the feature point tracking process can be improved.

In the present embodiment, as described above, the lower portion of the original image is cut out and used for the feature point tracking process. As the feature point, an arbitrary point in the deviation of the position where the error of the initial trajectory appears within the image may be used. In addition to, for example, markings on roads, a guide rail, a building at a road boundary, or the like, a part of a continuum of multiple frames may be captured as a feature point. The clipped image used in the feature point tracing process is not limited to the lower part of the original image, and any part including the feature point may be selected. And the original image itself can be used for the characteristic point tracing processing. However, since the feature point in the lower part of the image includes the position closest to the camera among various parts captured in the image, the accuracy of analysis of the captured position can be improved by using the lower part of the image.

As described above, in the feature point tracking process, distortion correction may be performed on an image by affine transformation or the like. The setting procedure of the transformation coefficient for correcting the distortion is also exemplified.

Fig. 7 is a process diagram of a transform coefficient setting method. First, a grid for correction is arranged on a road surface in front of a vehicle as the image data acquisition device 100 (step S100). The road surface on which the grid is disposed is preferably a flat surface having no inclination or unevenness. The grid is disposed forward with a space d2 that is only enough to capture the vehicle-side end line NS. The width W and length L of the grid can be arbitrarily set, but in order to improve the setting accuracy of the transform coefficient, it is preferable to use a size that can cover the range that can be captured by the camera. In this embodiment, the width W is 15m and the length is 3.5 m. The mesh size d1 of the grid can be set arbitrarily. The finer the mesh size is, the higher the conversion accuracy is, but the storage capacity for storing the conversion coefficients in advance is increased. In the present embodiment, the mesh size d1 is 50 cm.

Next, a transform coefficient is calculated for each mesh of the grid (step S102). The figure shows a method of calculating transform coefficients. The photographed image of the grid disposed in front of the vehicle is distorted into a substantially trapezoidal shape as shown by the solid line in the figure. The transformation coefficients are set for each mesh so that each mesh distorted in this manner can be transformed into an original shape shown by a dotted line in the figure, that is, an image obtained when the mesh is photographed from the front. For example, transform coefficients for imaging the grid G11 as the grid G21 are set. Transform coefficients for imaging the grid G12 as the grid G22 are also set. The transform coefficients of both may also be different.

The transform coefficients thus set are stored as a table (step S104), and used for distortion correction in the feature point tracking process. The figure shows an example of the structure of the table. In the present embodiment, a table is used in which the conversion coefficients Cxy1 and Cxy2.. Cxyn are assigned to each pixel Pxy of the photographic image SCR. For example, the transform coefficient calculated in step S102 in correspondence with the mesh G13 is set for the pixel of the imaging mesh G13 in the image of the imaging grid. By doing so, it is possible to perform highly accurate distortion correction for each pixel of a captured image even in the feature point tracking process. The transform coefficient is not limited to such a setting, and may be a uniform value within the screen of the photographed image, or may be a uniform value for each grid arranged in the x direction.

Fig. 8 is an explanatory diagram showing an example of the feature point tracking process. This illustrates the result of performing the feature point seek processing on the initial trajectory (see fig. 5) shown in the modification. Frame data (white circles in the figure) arranged on the initial trajectory corrects errors in the cross direction by means of the feature point tracking process, respectively. As a result, a track Tr indicated by a thick line in the figure is obtained. The initial trajectory is in a state of being broken in the vicinity of the node N1, but after the position in the intersecting direction is corrected, the trajectory Tr becomes a continuous state as shown in the figure.

Fig. 8 shows an example in which the reference position (double circle in the figure) itself is also corrected in the cross direction. This is because, although the reference position is a point whose latitude and longitude are known, the reference position needs to be temporarily set on the road network data in order to set the initial trajectory using the road network. The reference position is moved to a position with a known latitude and longitude, and then characteristic point tracing processing is performed to obtain a trajectory shown in the figure.

In the present embodiment, it is assumed that the latitude and longitude are known at the reference position, but the feature point tracking process can be used even in a case where either one of the latitude and longitude of the reference position is unclear, that is, in a case where the position in the intersecting direction cannot be specified. In such a case, the imaging position in the intersecting direction may be determined based on, for example, the absolute position of the feature point in the image at the reference position. For example, when the center point of the width of the road in the image coincides with the center point of the lower part of the image at the reference position, the image capturing position is determined as the center of the road. When the midpoint in the lower part of the image is an internal division point dividing the road width by a predetermined ratio, the imaging position is determined as the position of the internal division road at the same ratio.

In the above description, a case where a single reference position is used as a reference in both the moving direction and the intersecting direction has been exemplified. The feature point tracking process may be performed such that the reference position in the moving direction is different from the reference position in the intersecting direction. For example, as shown in fig. 8, the reference position in the moving direction may be a point on the crosswalk, and the reference position in the intersecting direction may be a point Fra where the trajectory Tr intersects the dividing line. The location Fra is a location where the partition line comes to about the center of the image. In this way, various methods can be used to select the reference position and use the coordinates thereof in the feature point tracking process. In the feature point tracing process, only one of the single methods may be adopted, or a plurality of methods may be separately used according to various conditions.

D. Imaging position analysis processing

Fig. 9 is a flowchart of the imaging position analysis processing. This is a process realized by the cooperation of the functional blocks (see fig. 1) of the image data processing apparatus 200, and is a process executed by the CPU of the computer constituting the image data processing apparatus 200 when implemented in a hardware manner.

When the process is started, the CPU inputs image data, reference position information, and a vehicle speed pulse (step S10). As shown in fig. 1, these data are data generated by the image data acquisition apparatus 100, and in the present embodiment, are input to the image data processing apparatus 200 via the removable hard disk 114 a.

Next, the CPU inputs an initial trajectory (step S20). In the present embodiment, either an initial trajectory in the case of using the longitude and latitude detected by the GPS102 (see fig. 4) or an initial trajectory using road network data (see fig. 5) is input according to the selection of the user. Of course, only either one may be used. When the road network data is used, the initial trajectory may be generated in step S20 by receiving a designation of a node or a link in response to a user instruction.

Once the initial trajectory is input, the CPU arranges frame data on the initial trajectory (step S30). This corresponds to the process previously shown in fig. 3. That is, the processing is performed to determine the position along the initial trajectory using the velocity pulse for each input frame data. A process may be employed in which frame data corresponding to the vehicle speed pulse is extracted and the frame data is arranged at equal intervals.

The CPU performs distortion correction by affine transformation on these frame data, and then performs feature point tracing processing on the lower portion of the clipped image (steps S40, S50). Then, the data of the obtained photographing position is stored in association with each frame data (step S60). These correspond to the processing described with reference to fig. 6.

Fig. 10 is a flowchart of the feature point tracking process. This corresponds to the processing of step S50 of fig. 9 described above. In this process, the CPU inputs a cutout image as a processing target (step S51), and specifies the feature point position (step S52). The feature point position is a coordinate in the x-axis direction that is taken in the left-right direction with respect to the cutout image, i.e., the direction intersecting the vehicle movement direction.

The method of finding the position of the feature point is illustrated in the figure. In this example, a certain region Ras in the lower part of the slice image is analyzed to find the position of the feature point. When the position of the division line on the road is set as the feature point, the division line is brighter than other portions because of the white line. Thus, a luminance distribution as shown in the x-axis direction can be obtained for the region. In this case, a range D exceeding a threshold Th set in advance in a range in which a white line can be recognized is obtained, and the center value of the range D is used as the feature point position.

Once the feature point position is obtained in this way, the CPU calculates the amount of shift from the feature point position of the cutout image immediately before the moment of shift (step S53). The term "immediately preceding" refers to data arranged immediately before the instant of target frame data among a plurality of frame data arranged in time series along an initial trajectory. For example, when frame data corresponding to a vehicle speed pulse is extracted from captured frame data and arranged, the extracted frame data may be different from frame data immediately before the captured frame data.

The CPU repeats the above processing until the above processing is completed for all the frame data (step S54), and completes the feature point seek processing.

The above method is merely an example. The feature points may be set according to the building edge, or the like, in addition to the white lines. In the above example, the feature points are found from the luminance distribution, but the feature point positions may be determined in consideration of hue and chroma. As another method, for example, an edge in a cutout image may be extracted by image processing, and a line that is regarded as a segment line may be specified from the edge, so that the feature point position may be obtained.

Fig. 10 shows an example of the process of finding the shift amount of the feature point position from the area Ras in the lower part of the image. However, the shift amount may be determined from the feature point position based on the area above the extracted image for the frame data immediately before. That is, the amount of shift between the position of the feature point on the top of the cutout image immediately before and the position of the feature point on the bottom of the cutout image to be processed is determined. The advantage of using this method is that the 2 cut-out images can be matched with higher accuracy.

Fig. 11 is a flowchart of a feature point tracking process as a modification. In this process, the CPU first inputs 2 cutout images [1], [2] arranged in series (step S54). Then, while the positions of the cut-out images [1] and [2] are relatively shifted, the luminance difference evaluation value Ev of the connected portion is calculated (step S55).

The figure shows a method of calculating the luminance difference evaluation value Ev for the clipped images Pic [1] and [2 ]. Cases a to D each show a state in which the relative position in the intersecting direction of the cut-out image Pic [2] is shifted from left to right by 4 steps with respect to the cut-out image Pic [1 ]. As shown in case a, in the region where the clipped images Pic [1], [2] are connected in this arrangement, the absolute value of the luminance difference or the square value of the luminance difference between the pixels P × 1 and P × 2 having the same coordinates in the x-axis direction is obtained, and the sum of the values in the x-axis direction is used as the luminance difference evaluation value Ev.

The luminance difference evaluation value Ev varies depending on the relative positions of the clip images Pic [1] and [2 ]. The right side of the figure shows the change in the luminance difference evaluation value Ev. As shown in the figure, in the case C of image matching of the clipped images Pic [1] and [2], the luminance difference evaluation value Ev is the smallest. On the other hand, if the relative positional relationship in which the luminance difference evaluation value Ev is the smallest is obtained, the shift amount of the clipped images Pic [1] and [2] can be specified (step S56).

When the offset amount is obtained in this way, the CPU repeatedly processes all the frame data until completion while replacing the cutout image [1] with the cutout image [2 ]. The feature point seek processing can also be realized by the method of the modification. In this process, since the shift amount is determined based on the luminance difference between adjacent cut-out images, there is an advantage that the images can be matched with each other with high accuracy.

E. Mark and mark extraction process

FIG. 12 is a flow chart of the tag/label extraction process. This is processing as an application performed using the imaging position data obtained by the imaging position analysis processing shown in fig. 10 and 9. This corresponds to the processing executed by the logo/logo extraction processing 214 shown in fig. 1, and is the processing executed by the CPU of the image data processing apparatus 200 in terms of hardware.

Upon the start of the processing, the CPU inputs the cutout image and the photographing position data (step S70). The cutout image is affine-transformed as described above, and corresponds to a planar image obtained by imaging the road from above.

The CPU arranges the respective cutout images in accordance with the photographing position data and synthesizes the images (step S71). This configuration means that the cut-out image is pasted on a plane. The figure illustrates a configuration method of cutting out an image. As a result of the imaging position analysis processing, the imaging position of the cutout image is determined by the x and y coordinates, and the movement trajectory of the vehicle is determined as a curve Pass. Further, assume that the barycentric position of the extracted image Pic is Ps and the coordinate axes of the intersecting direction and the moving direction are Lx and Ly. In this case, the extracted image Pic has the center of gravity Ps aligned with the imaging position, and the coordinate axis Ly of the image is arranged in a direction tangential to the movement locus Pass.

In the lower part of the figure, an example of arranging images successively by the above-described method is shown. The left side shows a state in which the cutout images are two-dimensionally arranged on the x-y plane. The cutout images are arranged along the movement trajectory while smoothly changing the direction, and a planar composite image such as an aerial photograph can be obtained. This composite image is an image taken near the road surface in the present embodiment, and therefore can obtain resolution several times that of an aerial photograph.

The right side of the figure shows an example in which images are arranged along a movement trajectory accompanied by a change in the height direction (z direction). Such a movement trajectory can be obtained by using a feature point tracing process (see fig. 10 and 9) for an initial trajectory set by the height information of the road network data, for example. By using the height information in this way, a composite image can be obtained even on an ascending road such as a slope leading to an expressway, and a planar composite image can be obtained even on a portion that cannot be captured by aerial photography such as a road passing under an overhead because an image captured while traveling on a road surface is used.

The CPU identifies the position of the marker by image processing in which the marker and marker pattern data prepared in advance is read (step S72) and the image is synthesized, and identifies a suitable region for the pattern data (step S73). And the shape of the logo can be determined. The marker/marking pattern data includes, for example, a marker drawn on a road such as an arrow indicating a right/left steering restriction, a signal light, a road marker, and the like. The CPU extracts a shape corresponding to the marker from the composite image by image processing, and determines the position thereof. When the shape prepared as the pattern data is different from the mark included in the composite image, the pattern data is enlarged or reduced at an arbitrary ratio in the horizontal or vertical direction, and the shape of the mark or mark is determined. The same applies to the processing of road signs and the like.

F. Example of treatment

Fig. 13 and 14 are explanatory diagrams showing an example of processing of image data. An image Porg shown in fig. 13 is a front image of the vehicle taken by a camera. The cut-out image cut out from the lower part of the image is an image Pco. The figure represents the state before the affine transformation is applied.

The lower image E × 1 in fig. 14 is an example of a cutout image. By performing affine transformation on the cutout image E × 1, an image of a portion surrounded by a rectangular frame below the image E × 2 can be obtained. Similarly, a cutout image is prepared for other frame data, and the cutout image is once arranged along the initial trajectory after affine transformation is performed, thereby obtaining an image E × 2. That is, the image E × 2 corresponds to the image in the state of step S30 in which the imaging position analysis processing (fig. 9) is performed, but the image E × 2 is an image shown for convenience of description of the processing content, and it is not necessary to generate a composite image in which the respective frame data are arranged and synthesized in this manner in the actual processing.

Since the position of the image E × 2 in the direction intersecting the moving direction is not corrected, there is a place where the road surface is not marked with a discontinuity. As can be understood from this, for example, the markers Mk indicating straight-ahead and left-turn lanes are shifted in the left-right direction in the drawing at the boundary SL of the frame data. In image E × 2, a trimming edge is added to mark Mk to make the offset easily recognizable.

The image E × 3 represents a state in which the offset in the cross direction is corrected by the feature point tracking process. This corresponds to the state of step S71 in the marker/logo extraction process (fig. 12) because the synthesized image is obtained after the imaging position of each frame data is obtained. It can be seen from the figure that the deviations of the road markings have been eliminated. By using the image, for example, the positions of the markings indicating the left-turn lane and the straight lane and the crosswalk on the road can be obtained. In the figure, although a part of the marks of the left-turn lane and the straight lane is lost due to the vehicle, the shapes of the marks can be reproduced by the pattern data.

With the image data processing system of the present embodiment described above, the imaging position in the direction intersecting the moving direction can be obtained with high accuracy by the feature point tracking process. Further, the positional accuracy in the moving direction can be improved by using information indicating the moving distance at the time of photographing such as a vehicle speed pulse. As a result, the imaging position of each frame data can be determined with good accuracy, and a high-resolution composite image can be obtained as shown in fig. 14.

By using such a composite image, the shape and position of the road marking on the road surface and the roadside marker can be determined without requiring special measurement as long as the vehicle is photographed by the camera while traveling. Thus, the burden required for generating three-dimensional map data that can accurately reproduce the state of the road surface can be greatly reduced. This is merely an example of an application in which the imaging position of each frame data is determined, and the frame data to which the imaging position is assigned can be used for various purposes such as height estimation of a building.

G1. Modification example-frame data arrangement method:

fig. 15 is an explanatory diagram showing a frame data arrangement method as a modification. This corresponds to a modification of the processing of step S30 in the imaging position analysis processing (fig. 9). As in the example, the example in which the frame data shot while moving from the arrow Ar1 in the direction of Ar2 is arranged on the initial trajectory set by the links L1 and L2 is illustrated.

In the modification, frames are arranged in reverse time series at the time of imaging with reference positions (double circles in the figure) as starting points. For example, on the link L1, frame data is sequentially arranged at a position of a distance corresponding to the backward movement of the vehicle speed pulse, that is, a position moving in the direction of the arrow DL in the figure, with the reference position as the start point. As a result, on the link L1, the position accuracy of the frame image is higher in the region E1 close to the reference position than in the region E2 far from the reference position. Accordingly, if the analysis of the imaging position and the image synthesis, marker and marker extraction processing (see fig. 12 to 12) are performed with such an arrangement, the accuracy of the image, the position accuracy of the extracted marker, and the like become higher as they approach the reference position. The data thus generated can be used, for example, as a guide image of a navigation system mounted on a vehicle, and when moving from arrow Ar1 to Ar2 in fig. 15, the vehicle can be accurately and smoothly guided with higher accuracy as the distance from the intersection corresponding to node N1 increases. Further, automatic running control of the vehicle can be realized.

As still another modification of the above-described embodiment, the frame data may be arranged in order in both the moving direction and the reverse direction with the reference position as a start point. That is, the arrangement method described in the modification (fig. 15) may be used in combination with the arrangement method described in the embodiment (fig. 5). This has the advantage that the positional accuracy of all the frame data can be improved.

In the case of processing an image obtained by imaging a road on which a plurality of opposite pass belts are provided, a configuration in which frame images and the like are arranged in reverse order of time series with a reference position as a starting point is useful for imaging only an image of a one-sided pass belt (usually, a pass belt that moves during imaging). On the other hand, a method of sequentially arranging frame data in both directions of the moving direction and the reverse direction with the reference position as the starting point for capturing images of the traffic zones on both sides is useful.

G2. Modification-use of side images:

in the embodiment, an example in which a crosswalk is used as a reference position is described (see fig. 6). In order to improve the analysis accuracy, it is preferable to use various reference positions in combination. For example, when the crosswalk is blurred to such an extent that the crosswalk cannot be used as the reference position, or when the crosswalk is hidden by another vehicle or the like and cannot be photographed, another reference position can be used. In the modification, as an example of such a reference position, an example of a building appearing on an image captured by the side cameras 122R and 122L (see fig. 2) is illustrated.

Fig. 16 is an explanatory diagram showing a method of obtaining the reference position using the side image. A part of the two-dimensional map is shown enlarged above the figure. Assume that a side image is captured while moving from point P1 to point P2 through the road in the figure. Assume that there are buildings BLD 1-BLD 3 on the road. The position coordinates (latitude, longitude) of the buildings BLD1 through BLD3 are known.

Consider the case where a building is photographed at perspective a1, indicated by a solid line at point P1. As shown in the lower part of the figure, buildings BLD1 and BLD2 are photographed in an image PIC1 at this time. When moving to point P2, the building is photographed at a viewing angle indicated by a dotted line. The images taken at points P1 and P2 are compared to find that the position of the corner CNR of the building BLD1 is relatively shifted. When the angle between the end line of the angle of view and the line connecting the point P1 and the angle CNR is defined as an angle a2, the edge EDG1 corresponding to the angle CNR is captured at the point of the width SC of the intra picture PIC1 in a2/a1 at the point P1. At the point of time P2, the position of edge EDG1 moves to the right in the figure as angle a2 becomes larger. This moving distance is denoted as MB. The distance of movement MA from point P1 to point P2 is known. Accordingly, the distance DIST from the photographing position to the angle CNR can be geometrically determined according to the moving distances MA and MB and the angle of view a 1. The same process can be performed for the other edge BDG2 of the building BLD 1. Also, if the distance DIST is found, the actual distance between the edges EDG1, EDG2, i.e. the width of the building BLD1, can be determined from the view angle a 1.

Although the case where the building to be photographed is assumed to be the building BLD1 has been described above, the above calculation can be smoothly performed even in a state where it is not determined which building is photographed. The image data processing apparatus 200 of the modification searches for a building captured in the image PIC1 based on the above calculation result. In the modification, the positions of the point P1 and the point P2 can be determined within a predetermined error range. Therefore, the position coordinates of the building located at the position distant from DIST can be obtained within a predetermined error range with reference to the point P1. Further, by referring to the map database, it is possible to specify a building having a width corresponding to the value obtained from the edges EDG1 and EDG2, that is, a building captured in the image PIC1, in the vicinity of the position coordinates obtained by the above-described method, and to specify the coordinates thereof. The error between the coordinates obtained from the map database and the coordinates obtained by the calculation is the position error of point P1. Accordingly, the position coordinates of point P1 can be corrected by reflecting the error. The thus corrected imaging position can be used as a reference position.

In the above processing, the building edges EDG1 and EDG2 may be automatically specified by image processing, but in the modification, the operator drags the edges EDG1 and EDG2 with a pointing device such as a mouse while viewing the image to specify the edges with high accuracy.

Fig. 17 is a flowchart of a reference position calculation process of the modification. This is a process in which the image data processing apparatus 200 obtains the reference position according to the method described with reference to fig. 16. The image data processing apparatus 200 first inputs a plurality of target frames, a target building as an analysis target, and specification of edges thereof (step S200). The shooting position of the designated target frame, that is, the position coordinates detected by the GPS are input (step S202).

Next, the image data processing apparatus 200 calculates a movement distance MB of an edge between the target frames and a movement distance MA of the photographing position (see fig. 16) (step S204). Then, the distance DIST between the photographing position and the object building and the width of the object building are calculated from the movement distances MA and MB (step S206). The image data processing apparatus 200 refers to the network database 220 (see fig. 1), and identifies a building suitable for the shooting position (GPS output), the distance DIST, and the width of the target building as the target building (step S208).

When the object building is thus determined, the coordinates of the object building can be obtained from the network database 220. Accordingly, the image data processing apparatus 200 specifies the photographing position based on the distance DIST and the position of the edge in the photographed image with reference to the position of the target building (step S210). This corresponds to a process of reflecting an error between the calculated position coordinates of the target building and the position coordinates obtained from the network database 220 on the imaging position output as the GPS to correct the error of the imaging position. The image data processing apparatus 200 sets the imaging position thus determined as a reference position (step S211), and ends the reference position calculation process. By adopting the processing of the modified example described above, even when the crosswalk cannot be used as the reference position, the error of the imaging position can be eliminated from the building position, and therefore the accuracy of analyzing the imaging position of each frame can be improved.

G3. Modification example-determination of temporal change:

the moving image capturing described in the above embodiment and modification is performed not only when newly generating three-dimensional map data but also in some cases as maintenance of a region where three-dimensional map data has already been generated. In this case, the photographed moving image can be used to judge whether there is a new building, whether there is a demolition, a reconstruction, or the like, with time by comparing it with the three-dimensional map data that has been prepared. The following exemplifies the determination of such a temporal change.

Fig. 18 is a flowchart of the temporal change determination process. In this process, the image data processing apparatus 200 first reads a plurality of target frames and imaging positions (step S150). In this process, the process is performed using the side image. The imaging position uses the output result of the GPS.

The image data processing device 200 generates a two-dimensional image corresponding to the target frame by generating an image of a building or the like viewed from the imaging position using existing 3D graphics data, that is, three-dimensional data of the building or the like for generating three-dimensional map data (step S152). Then, the two-dimensional image thus generated is matched with the target frame, and it is determined whether or not there is a mismatch therebetween (step S154). The matching between images may be performed by a known method such as template matching, DP matching, and eigenspace method. The reference value for determining whether or not there is any inconsistency may be set within a range in which it is possible to detect whether or not there is a large inconsistency corresponding to new building or demolition of a building, among inconsistencies between the two-dimensional image and the target frame.

If the matching results do not match (step S156), the image data processing apparatus 200 uses the target frame in the reference position calculation process described above with reference to fig. 17 (step S200). This is because the reference position calculation processing can be performed with high accuracy and stability by using the photographed image of the building determined not to change with time. The reference position calculation process is not essential and may be omitted. If the matching results do not match (step S156), the image data processing apparatus 200 performs a process of updating the three-dimensional graphics data from the target frame image (step S158). When the building is a new building or a rebuilt building, the processing includes a process of cutting out a structure of the new building or the rebuilt building from the target frame image automatically or by an operation of an operator. When a building is destroyed, a process of deleting data of the corresponding building from existing graphic data is included. In either case, the processing need not be performed in a fully automatic manner, and may be performed in accordance with an operation by an operator. By using the time-varying determination process described above, it is possible to easily determine whether or not there is a change in existing 3D graphics data with time, and thus the burden of maintenance can be reduced.

G4. Modification example-analysis of position coordinates of guide plate:

fig. 19 is a flowchart of the guide plate position coordinate analysis process. This is a process for determining the position coordinates of the guide plate provided on the street from the captured image. The determination of the position coordinates is based on the principle described with reference to fig. 16, that is, a method of calculating the position coordinates of the building with reference to the imaging position.

The image data processing apparatus 200 reads a plurality of target frames used for analysis and the imaging positions of the respective frames (step S300). The subject frame is a side image obtained by photographing the guide plate, and the photographing position is the output result of the GPS or the processing result of the embodiment.

Next, the image data processing apparatus 200 inputs the designation of the post position of the guide plate from the target frame (step S302). Here, a method is adopted in which, in the image of the guide plate SP taken as shown in the drawing, the operator drags the line PSP to the post position with a pointing device to specify the line PSP. The strut position may also be determined automatically using image analysis.

The image data processing apparatus 200 calculates the position coordinates of the support based on the support position and the imaging position specified for each frame according to the principle described with reference to fig. 16 (step S304). The position coordinates thus obtained are then output (step S306), and the analysis processing of the guide plate position coordinates is ended. It is also possible to confirm that the calculated position coordinates are not abnormal values with reference to the network database. For example, when the calculated position coordinates are the center of a road or a sidewalk, it can be determined that the result is an abnormal result.

If the above-described processing is employed, it is relatively easy to generate three-dimensional map data in a state where the guide plate is set in place. The guide plate is generally unable to obtain the position coordinates from existing data such as a network database. The guide plate is usually provided near the intersection, but the three-dimensional map data generated in a state of being provided at a place different from the actual position may be rather confusing for the user. If the above processing is used, the position can be analyzed by simply capturing an image of the moving image of the guide plate, and therefore, the trouble of measuring the position coordinates of the guide plate can be avoided, and there is an advantage that three-dimensional map data conforming to the actual state can be provided while avoiding the above-described problems.

While various embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and it is needless to say that the present invention may take various configurations without departing from the scope of the present invention. For example, in the embodiment, the feature point tracing process is performed according to the segment line of the road. The feature point tracking process may be performed with various objects photographed in a plurality of frames. For example, guide rails, roadside buildings, etc. may also be used.

In the embodiment, the position in the moving direction is specified by the vehicle speed pulse, but information other than the vehicle speed pulse may be used in the processing. As long as it is information that provides a relationship between the moving distance of the vehicle and the time, various kinds of information can be used. For example, the number of lane lines drawn with a broken line on the road, the number of pillars of the guide rail, and the like may be used instead of the vehicle speed pulse.

Industrial applicability

The present invention can be used to analyze the imaging position of each frame of an image composed of a plurality of frames, such as a moving image captured while moving.

Claims (2)

1. A photographing position analyzing apparatus for analyzing a photographing position of each frame with respect to an image, wherein the image is composed of a plurality of frames photographed at a known timing while moving in a state where an attitude angle with respect to a ground surface is constant, and each frame includes a predetermined continuum photographed together with at least one frame immediately before and after the predetermined continuum, the photographing position analyzing apparatus comprising

An input unit that inputs the image data composed of the plurality of frames;

an initial trajectory input unit that inputs an initial trajectory of the movement as an initial value of the analysis; and

and a photographing position analyzing unit that temporarily sets a photographing position of each frame along a moving direction of the initial trajectory based on the photographing time, and corrects the temporarily set photographing position in a moving intersecting direction of the initial trajectory based on a shift between images photographed across the plurality of frames for the predetermined continuous body to analyze the photographing position of each frame.

2. The photographing position analyzing apparatus according to claim 1,

the image is an image of the front or back of the direction of movement,

the input section inputs a part of a lower part of the image.

3. The photographing position analyzing apparatus according to claim 1,

the image is an image taken while moving on a road,

the continuum is a section line of a road traffic belt.

4. The photographing position analyzing apparatus according to claim 1,

the input unit inputs moving distance information indicating a relationship between a time at the time of the photographing and a moving distance along the moving direction,

the imaging position analysis unit temporarily sets the imaging position of each frame along the movement direction based on the movement distance information.

5. The photographing position analyzing apparatus according to claim 1,

the input unit inputs moving distance information indicating a relationship between a time at the time of the photographing and a moving distance along the moving direction,

the imaging position analysis unit extracts frames captured at predetermined movement distances from the movement distance information, and performs the analysis.

6. The photographing position analyzing apparatus according to claim 4,

the image is an image captured by a camera mounted on the vehicle,

the moving distance information is a vehicle speed pulse of the vehicle.

7. The photographing position analyzing apparatus according to claim 1,

the input unit also inputs reference position information indicating a time at which the image data is captured to a known reference position in a manner to be in correspondence with the image data,

the imaging position analysis unit initializes at least a position along the moving direction based on the reference position information during the analysis.

8. The photographing position analyzing apparatus according to claim 1,

the image data further includes lateral image data of a plurality of frames taken in the moving cross direction,

a map data reference unit having map data for referring to the position coordinates of the subject in which the lateral image data is recorded, and

a coordinate calculation unit for calculating subject coordinates indicating the position of the subject from the horizontal image data of the plurality of frames,

the imaging position analysis unit performs initialization based on position coordinates recorded in the map data and coordinates of the object at least for a position along the moving direction during analysis.

9. The photographing position analyzing apparatus according to claim 1,

has a network data reference unit for referring to road network data in which a road is represented by nodes and links,

the initial trajectory input unit sets the initial trajectory based on the road network data.

10. The photographing position analyzing apparatus according to claim 1,

the initial trajectory input unit inputs an output of a position detection sensor that detects a movement trajectory in at least 2 dimensions within a predetermined error range when the image is captured, and sets the initial trajectory.

11. The photographing position analyzing apparatus according to claim 1,

an image conversion processing unit for converting the image data into an image in a state where the continuous body is captured from the front side before the analysis of the imaging position,

the image conversion processing unit distributes the image data to a plurality of regions, and performs the conversion with different conversion coefficients for each region,

the plurality of regions and the conversion coefficient are set so that an orthographic view of a mesh body having a known shape can be obtained from image data obtained by imaging the mesh body.

12. A photographing position analyzing method for analyzing a photographing position of each frame for an image composed of a plurality of frames photographed at a known timing while moving in a state where an attitude angle relative to a ground surface is constant, the frames including a predetermined continuum photographed together with at least one frame immediately before and after the predetermined continuum, the method comprising a step executed by a computer

An input step of inputting the image data composed of a plurality of frames,

An initial trajectory input step of inputting an initial trajectory of the movement as an initial value of the analysis, and

and a photographing position analyzing step of temporarily setting the photographing position of each frame along the moving direction of the initial trajectory based on the photographing time, and correcting the temporarily set photographing position in the moving intersecting direction of the initial trajectory based on the deviation between the images photographed across the plurality of frames for the predetermined continuous body to analyze the photographing position of each frame.