JP2011204118A - Three-dimensional image creation system and three-dimensional image creation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional image creation system capable of obtaining a three-dimensional image with an expanded view field range, and a three-dimensional image creation method.SOLUTION: The three-dimensional image creation system includes: one or more stereoscopic cameras 100 attached to a movable body; a three-dimensional data generation means 101 generating three-dimensional data in time series from images photographed by the stereoscopic cameras; a movement detection means 102 detecting movement of the movable body; and time series data integrating means 104 and 103 integrating the three-dimensional data generated by the three-dimensional data generation means together while performing interpolation based on three-dimensional data in the past and the movement detected by the movement detection means.

Description

  The present invention relates to a 3D image creation system and a 3D image creation method, and more particularly to a 3D image creation system and a 3D image creation method using a stereo camera installed on a moving body.

  In recent years, a driving support system has been developed in which a stereo camera is installed on a vehicle to acquire a three-dimensional image, thereby recognizing the terrain and ensuring safe driving of the vehicle.

  In the following Patent Document 1, image data is acquired in time series by a stereo camera installed in a vehicle, and the first image data, the second image data before that, and the distance data related to these image data, The amount of vehicle movement is detected. Then, the terrain data obtained from the image data is sequentially synthesized in consideration of the movement amount. This describes a technique for accurately recognizing the three-dimensional shape of the ground.

JP 2004-144644 A

  However, in the three-dimensional image obtained by the stereo camera, omission of images is caused by occlusion, but the above-described conventional technology cannot reduce omission of images. For this reason, the prior art has a problem that a three-dimensional image can be acquired only in a limited visual field range.

  In order to solve the above problems, a three-dimensional image creation system according to the present invention includes one or more stereo cameras attached to a moving body and three-dimensional data for generating three-dimensional data in time series from images taken by the stereo cameras. Generating means, movement detecting means for detecting movement of the moving body, and three-dimensional data generated by the three-dimensional data generating means are interpolated based on the past three-dimensional data and the movement detected by the movement detecting means. And time-series data integration means for integrating them.

  In addition, a 3D image creation method according to the present invention includes a 3D data generation stage for generating 3D data in time series from images taken by one or more stereo cameras attached to a mobile body, and detecting the movement of the mobile body And a time series data integration stage for interpolating and integrating the three-dimensional data generated in the three-dimensional data generation stage based on the past three-dimensional data and the movement detected in the movement detection stage. Have.

  According to the three-dimensional image creation system and the three-dimensional image creation method according to the present invention, the three-dimensional data of the image of the stereo camera attached to the moving body is the past three-dimensional data, and the movement detected by the movement detector. , Interpolate and integrate based on As a result, data loss due to occlusion can be reduced, and a three-dimensional image with an expanded visual field range can be obtained.

It is a block diagram which shows the structure of the three-dimensional image creation system which concerns on 1st embodiment of this invention. It is a figure which shows the 1st example of occlusion. It is a figure which shows the 2nd example of occlusion. It is a figure which shows the 3rd example of occlusion. It is a figure which shows the flowchart of the three-dimensional image creation method which concerns on 1st embodiment of this invention. It is a figure which shows the subroutine flowchart of step S505 of the flowchart shown in FIG. It is a figure which shows the subroutine flowchart of step S506 of the flowchart shown in FIG. It is explanatory drawing which illustrated the vector for determining whether an object is a stationary object. It is a figure which shows the flowchart of the three-dimensional image creation method which concerns on 2nd embodiment of this invention. It is a figure which shows the subroutine flowchart of step S901 of the flowchart shown in FIG. It is a figure which shows the subroutine flowchart of step S909 of the flowchart shown in FIG. It is a figure which shows the example of the flowchart regarding holding | maintenance of the time series data in the case of changing the frame rate of the time series data to hold | maintain in 3rd embodiment of this invention. It is a figure which shows the relative positional relationship of the mobile body and stereo camera of the three-dimensional image creation system which concern on 4th embodiment of this invention. It is a block diagram which shows the three-dimensional image creation system which concerns on 4th embodiment of this invention. It is a figure which shows the flowchart of the three-dimensional image creation method which concerns on 4th embodiment of this invention.

  Hereinafter, a three-dimensional image creation system and a three-dimensional image creation method according to an embodiment of the present invention will be described in detail with reference to the drawings.

[First embodiment]
FIG. 1 is a block diagram showing a configuration of a three-dimensional image creation system according to the first embodiment of the present invention.

  A three-dimensional image creation system 10 according to the present embodiment includes a stereo camera 100, a three-dimensional data generation unit (three-dimensional data generation unit, stationary object determination unit) 101, and a movement amount calculation unit (movement detection unit, stationary object determination unit). 102, a time-series data holding unit (time-series data holding unit) 103, a time-series data integration unit (time-series data integration unit) 104, and an output unit (output unit) 105. Some or all of these components are composed of computer hardware and software (computer programs) such as a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), hard disk, and input / output device. May be.

  These components can be mounted on a mobile object. However, components other than the stereo camera 100 may not be mounted on the moving body. Here, as the moving body, any moving body such as a robot in addition to a vehicle can be considered.

  Stereo camera 100 has two cameras: a base camera and a reference camera. Each of the reference camera and the reference camera can be configured by an imaging device such as a CCD device and other optical systems, and the configuration is the same as that of a general stereo camera.

  The stereo camera 100 acquires images with the reference camera and the reference camera, respectively. The standard image acquired by the standard camera and the reference image acquired by the reference camera are converted into digital images, and are output as standard image data and reference image data in frame units or pixel units, respectively.

  The three-dimensional data generation unit 101 receives standard image data and reference image data from the stereo camera 100, and generates three-dimensional data based on these image data. The three-dimensional data calculates the parallax from the standard image data and the reference image data, and the distance from the stereo camera 100 to the object according to the principle of triangulation from the parallax, the base line length of the stereo camera 100, and the focal length. It is obtained by computing. The three-dimensional data includes at least parallax and luminance data for each pixel (or for each of a plurality of pixel groups).

  The three-dimensional data generation unit 101 calculates the reliability of the image (hereinafter referred to as “reliability”). Here, the reliability refers to the clarity of matching between the standard image data and the reference image data. That is, the case where the object image data present in one of the standard image data and the reference image data does not exist in the other is low reliability, and the object image data exists in both the standard image data and the reference image data. High reliability. Here, the phenomenon that the image of the object existing in one of the standard image and the reference image does not exist in the other can be considered as a kind of occlusion. That is, the low-reliability image data is image data in the occlusion area.

  The reliability is a concept that can be quantified, and the level of reliability can be evaluated even when object image data exists in both the standard image data and the reference image data.

  On the other hand, a phenomenon in which an object is hidden by an obstacle in the image is also occlusion. That is, in the image data, the area where the object is hidden by the obstacle is the image data of the occlusion area.

  The movement amount calculation unit 102 receives the base image data and the reference image data from the stereo camera 100, calculates the movement of the image between the base image data and the reference image data, and between the frames, and thereby calculates the image by the correlation calculation. The movement amount and movement direction (hereinafter referred to as “movement vector”) of the object inside are calculated. Furthermore, the movement amount calculation unit 102 calculates the movement vector of the moving object by calculating the correlation between the base image data and the reference image data and between the frames. Here, the correlation calculation may be performed by any of the phase-only correlation method, the sum of absolute differences, the sum of squared differences, and the normalized cross-correlation.

  Note that the movement vector of the moving body may be calculated using the measurement result of the acceleration sensor or the gyro sensor.

  The time-series data holding unit 103 holds the three-dimensional data and reliability received from the three-dimensional data generation unit 101, and the movement vector of the object and the movement vector of the moving body in the image received from the movement amount calculation unit. . The retained data can be read as data at a desired time. Here, holding the three-dimensional data and the reliability in the time-series data holding unit 103 makes it easy for the time-series data integration unit 104 to interpolate the current three-dimensional data with the past three-dimensional data. Because.

  The time-series data integration unit 104 receives the three-dimensional data and the reliability from the three-dimensional data generation unit 101, and the past three-dimensional data and the reliability from the time-series data holding unit 103. If the current three-dimensional data includes the occlusion area data, the data integration unit 104 converts the occlusion area data on the condition that the past three-dimensional data corresponding to the occlusion area satisfies a predetermined condition. Interpolate and integrate with past 3D data.

  The output unit 105 outputs the three-dimensional data integrated by the time series data integration unit 104 to the monitor.

  Occlusion will be described in more detail.

  FIG. 2 is a diagram illustrating a first example of occlusion. 2 indicates the occlusion area 210 at time t1. As shown in FIG. 2, when viewed from the moving body 200a at time t0 (which is a past time with respect to time t1), the occlusion area 210 at time t1 is not obstructed by the obstacle 220, and the occlusion area is Don't be. However, when viewed from the moving body 200b at the time t1, the area 210 is an occlusion area because the visual field is blocked by the obstacle 210.

  FIG. 3 is a diagram illustrating a second example of occlusion. 3 indicate occlusion areas 210a and 210b at time t1. As shown in FIG. 3, when viewed from the moving body 200a at time t0, the occlusion areas 210a and 210b at time t1 are not obstructed by the obstacle 220 and do not become occlusion areas. However, when viewed from the moving body 200b at the time t1, the area 210 is an occlusion area because the visual field is blocked by the obstacle 210.

  FIG. 4 is a diagram illustrating a third example of occlusion. As shown in FIG. 4, when viewed from the reference camera 100b of the stereo camera 100, the object 400 is not obstructed by the obstacle 210, and the image data of the object exists in the reference image data. However, when viewed from the reference camera 100a of the stereo camera 100, the field of view of the object 400 is obstructed by the obstacle 210, and the image data of the object does not exist in the reference image data. Thus, the phenomenon that the object 400 cannot be matched between the standard image data and the reference image data is occlusion due to low reliability, and the image data of the object 400 is image data of the occlusion area.

  FIG. 5 is a diagram illustrating a flowchart of the three-dimensional image creation method according to the present embodiment.

  FIG. 6 shows a subroutine flowchart of step S505 of the flowchart shown in FIG.

  FIG. 7 is a diagram showing a subroutine flowchart of step S506 in the flowchart shown in FIG.

  Hereinafter, the three-dimensional image creation method according to the present embodiment will be described in detail with reference to the flowcharts shown in FIGS. The 3D image creation method according to the present embodiment is performed by the 3D image creation system shown in FIG.

  The three-dimensional data generation unit 101 generates three-dimensional data based on the standard image data and reference image data received from the stereo camera 100 (S500), and the time-series data holding unit 103 is generated by the three-dimensional data generation unit 101. The obtained three-dimensional data is held (S501). Such generation and holding of three-dimensional data is performed for all measurement points (pixels) in one frame (S502, S503). That is, it is determined whether or not the processing of all three-dimensional measurement points has been completed (S502). If it is determined that the processing of all three-dimensional measurement points has not been completed (S502, No), the next measurement point is determined. (S503), three-dimensional data is created (S500), and time-series data is held (S501).

  An output data generation area is set (S504). That is, a region of a 3D image to be output and displayed on the monitor is set. For example, the generation area of output data can be set by designating coordinates of a three-dimensional image displayed on the monitor.

  The occlusion is determined (S505). The occlusion determination is performed according to a subroutine flowchart shown in FIG.

  It is checked whether data exists in the current three-dimensional data (S600). That is, it is determined whether or not there is a missing image in the current three-dimensional data. If there is a missing image in the current three-dimensional data, the missing portion is an occlusion area, so it is determined to be occlusion (S603). If there is no missing image in the current three-dimensional data (S600, Yes), the process proceeds to step S601.

  Whether there is a missing image is determined by comparing current image data with past image data (for example, image data one frame before), and an object existing in the past image data exists in the current image data. It can be judged by whether or not.

  It is determined whether the current three-dimensional data is highly reliable (S601). When it is determined that the current 3D data is not highly reliable (S601, No), it is determined that there is occlusion (S603), and when it is determined that the current 3D data is highly reliable (S601, Yes), it is determined that it is not occlusion (S602).

  Whether the three-dimensional data is highly reliable is determined by comparing the standard image data with the reference image data. That is, for example, when there is no object data in the standard image data and there is object data in the reference image data, it is determined that the three-dimensional data of the object is low reliability (that is, not high reliability). . The comparison between the reference image data and the reference image data is performed by correlation calculation. The correlation calculation can be performed by any one of, for example, a phase-only correlation method, a sum of absolute differences, a sum of squares of differences, and a normalized cross-correlation.

  After determining the occlusion (S505), the time series data is interpolated (S506). Time series data interpolation is performed according to a subroutine flowchart shown in FIG.

  If it is determined that it is not occlusion (No in S700), the data is not interpolated and the current three-dimensional data is used as it is as the current three-dimensional data (S708).

  If it is determined that it is occlusion (S700, Yes), the initial value 1 is assigned to the integer type variable N, and it corresponds to the occlusion area of the current three-dimensional data from the frame specified by N (ie, the first frame). Data to be extracted is extracted (S702). Here, the first frame refers to a past frame immediately before the current frame.

  It is determined whether the extracted data is of a stationary object and has high reliability (S703). If the extracted data is of a stationary object and has high reliability (S703, Yes), the data of the occlusion area of the current three-dimensional data is interpolated with the extracted data (S706). In the interpolation, coordinate conversion is performed on the extracted data in consideration of the movement of the moving body as necessary. At this time, the time-series data integration unit 104 creates three-dimensional data that matches the area specified in step S504. That is, the time-series data integration unit 104 performs coordinate conversion of the three-dimensional data of the pixel before or after the interpolation of the three-dimensional data in order to obtain three-dimensional data that matches the area specified in step S504.

  Note that instead of determining whether or not the extracted data is of a stationary object and has high reliability, it may be determined whether or not the extracted image is an image taken closest.

  FIG. 8 is an explanatory diagram illustrating vectors for determining whether an object is a stationary object.

  As shown in FIG. 8, a vector M indicating the displacement between the position of the moving body 200a at time t0 and the position of the moving body 200b at time t1 after time t0 is the moving vector of the moving bodies 200a and 200b. A vector A01 indicating the displacement between the position of the object 800a at time t0 and the position of the moving body 800b at time t1 is a movement vector of the objects 800a and 800b. If the vector A01 is zero, it can be determined that the objects 800a and 800b are stationary objects.

  Vector A0 is a position vector of object 800a with respect to moving body 200a at time t0, and vector A1 is a position vector of object 800b with respect to moving body 200b at time t1. Vector A0 and vector A1 can be calculated from the three-dimensional data of stereo camera 100 at times t0 and t1. Further, as described above, the vector M, which is the movement vector of the moving bodies 200a and 200b, can be calculated by correlation calculation between frames of three-dimensional data. Therefore, the movement vector A01 of the objects 800a and 800b can be calculated from these vectors A0, A1 and M.

  Note that it is necessary to determine whether the objects 800a and 800b are stationary objects when the current object 800b is image data of an occlusion area. Then, the object 800b cannot be visually recognized from the stereo camera 100 at present. Therefore, strictly speaking, the vector A1 cannot be calculated at present, and therefore it cannot be determined whether or not the current object 800b is a stationary object. Therefore, in practice, it will be determined whether or not the object 800b is a stationary object in the past (for example, the past several frames before) in which the object 800b exists as image data.

  If the extracted data is not a stationary object or is not highly reliable (S703, No), it is determined whether or not the check has been completed for all frames held in the time-series data holding unit 103. (S704). That is, for all past frames held in the time-series data holding unit 103, it is determined whether the extracted data is for a stationary object and has high reliability. If it is determined that the check has been completed for all the frames (S704, Yes), the occlusion area relating to the current three-dimensional data is determined as a measurement impossible point (S707), and data interpolation is not performed.

  On the other hand, if it is determined that the check has not been completed for all the frames (S704, No), the integer variable N is incremented (S705), and the current frame from the frame specified by N (ie, the second frame) is updated. The data corresponding to the occlusion area of the three-dimensional data is extracted (S702). Here, the (N + 1) th frame refers to a frame immediately before the Nth frame.

  Thereafter, steps S702 to S707 are performed for all the frames held in the time-series data holding unit 103 until the check is completed (S704, Yes). As a result, even if there is no data suitable for interpolation in the previous frame of the current 3D data, the probability of being a measurement impossible point is checked in order to check whether there is data suitable for interpolation going back to the past frame. It is possible to obtain a three-dimensional image that has been reduced and has an expanded visual field range.

  However, data suitable for interpolation may be checked only for the frame immediately before the current three-dimensional data. In this case, the time-series data holding unit 103 needs only a storage capacity to hold two frames, and thus the storage capacity size of the time-series data holding unit 103 can be reduced.

  If processing has not been completed for all output points (pixels) of the current 3D image (S507, No), the process proceeds to the next output point (S508), and steps are performed for all output points of the current frame. S505 to S507 are performed.

  When processing of all output points of the current 3D image is completed, the 3D image creation method according to this flowchart for the current frame ends, and this flowchart is executed for the next frame.

  As described above, the three-dimensional image creation system and the three-dimensional image creation method according to the present embodiment have been described. According to the present embodiment, the three-dimensional data of the stereo camera attached to the moving body is transferred to the past three-dimensional data and Interpolation is performed based on the movement vector detected by the detection means, and integration is performed. As a result, data loss due to occlusion can be reduced, and a three-dimensional image with an expanded visual field range can be obtained.

[Second Embodiment]
A three-dimensional image creation system and a three-dimensional image creation method according to the second embodiment of the present invention will be described in detail with reference to FIGS. 9 to 11 and FIG. The 3D image creation method according to the present embodiment is performed by the 3D image creation system shown in FIG. 1 as in the first embodiment.

  This embodiment is different from the first embodiment in the following points. That is, in the present embodiment, the three-dimensional data held in the time-series data holding unit 103 is limited to those of a highly reliable and stationary object, and the past three-dimensional data is interpolated when interpolating the current three-dimensional data. The point is that it is not judged whether the data is a highly reliable and stationary object.

  In the description of the present embodiment, the description overlapping with the first embodiment is omitted.

  FIG. 9 is a diagram showing a flowchart of the three-dimensional image creation method according to the second embodiment of the present invention.

  FIG. 10 shows a subroutine flowchart of step S901 in the flowchart shown in FIG.

  FIG. 11 is a diagram showing a subroutine flowchart of step S909 in the flowchart shown in FIG.

  In addition, the subroutine flowchart of step S905 of the flowchart shown in FIG. 9 is FIG. 6 like the first embodiment.

  The three-dimensional data generation unit 101 generates three-dimensional data based on the standard image data and the reference image data received from the stereo camera 100 (S900), and the time series data holding unit 103 holds the past time series held. The three-dimensional data is updated with the current three-dimensional data (S901). The time series data is updated according to the subroutine flowchart of step S501 shown in FIG.

  It is determined whether the current three-dimensional data is highly reliable (S1000). If it is determined that the current three-dimensional data is not highly reliable (S1000, No), the data in the time series data holding unit 103 is not updated (S1004). If it is determined that the current three-dimensional data is highly reliable (S1000, Yes), it is further determined whether the past three-dimensional data is more reliable than the current three-dimensional data (S1001).

  If it is determined that the past three-dimensional data is not more reliable than the current three-dimensional data (S1001, No), the data in the time-series data holding unit 103 is not updated (S1004). When it is determined that the past three-dimensional data is more reliable than the current three-dimensional data (S1001, Yes), it is further determined whether the past three-dimensional data is that of a stationary object (S1002).

  When it is determined that the current three-dimensional data is not for a stationary object (S1002, No), the data in the time series data holding unit 103 is not updated (S1004). If it is determined that the current three-dimensional data is that of a stationary object (S1002, Yes), the data in the time series data holding unit 103 is updated with the current three-dimensional data (S1003).

  It should be noted that instead of determining whether the past data is for a stationary object or determining whether the past data is highly reliable, or in addition to this, it may be determined whether the image has been taken closest. Whether or not the image is the closest image can be determined by calculating the distance from the three-dimensional data of the stereo camera 100 to the object in the image.

  Thus, in the present embodiment, the three-dimensional data held in the time-series data holding unit 103 is limited to that of a highly reliable and stationary object. As a result, the size of the memory constituting the time-series data holding unit 103 can be reduced, so that the cost and power consumption of the system can be reduced.

  After creating three-dimensional data (S900) and updating time-series three-dimensional data (S901) for all measurement points (pixels) in one frame (S902, S903), the process proceeds to step S904.

  An output data generation area is set (S904). That is, a region of a 3D image to be output and displayed on the monitor is set.

  The occlusion is determined (S905). The occlusion determination is performed based on a subroutine flowchart shown in FIG. Since the occlusion determination is the same as in the first embodiment, description thereof is omitted.

  After determining the occlusion (S905), the time series data is interpolated (S906). Time series data interpolation is performed based on a subroutine flowchart shown in FIG.

  If it is determined that it is not occlusion (S1100, No), data interpolation is not performed, and the current three-dimensional data is used as it is as the current three-dimensional data (S1105).

  If it is determined that it is occlusion (S1100, Yes), the past three-dimensional data corresponding to the occlusion area of the current three-dimensional data is searched in the time-series data holding unit 103 (S1101). When past three-dimensional data corresponding to the occlusion area of the current three-dimensional data exists in the time-series data holding unit 103 (S1102, Yes), the current three-dimensional data is interpolated with the corresponding past three-dimensional data. (S1103).

  When past 3D data corresponding to the current 3D data occlusion area does not exist in the time-series data holding unit 103 (No in S1102), the occlusion area related to the current 3D data is determined as an unmeasurable point. (S1104), no data interpolation is performed.

  If processing has not been completed for all output points (pixels) of the current three-dimensional image (S907, No), the process proceeds to the next output point (S908), and all output points of the current frame are determined. Steps S905 to S907 are performed.

  When the processing of all output points of the current 3D image is completed, the 3D image creation method according to this flowchart for the current frame ends. Then, the 3D image creation method is performed for the next frame according to this flowchart.

  As described above, the three-dimensional image creation system and the three-dimensional image creation method according to the present embodiment have been described. However, according to the present embodiment, the three-dimensional data of the stereo camera attached to the moving body is obtained as in the first embodiment. Interpolation and integration are performed based on the past three-dimensional data and the movement detected by the movement detecting means. As a result, data loss due to occlusion can be reduced, and a three-dimensional image with an expanded visual field range can be obtained.

  Furthermore, in the present embodiment, the three-dimensional data held in the time-series data holding unit is limited to those with high reliability and stationary objects. As a result, the size of the memory constituting the time-series data holding unit can be reduced, so that the system can be reduced in cost and power consumption.

[Third embodiment]
The 3D image creation system and 3D image creation method according to the third embodiment of the present invention will be described in detail.

  This embodiment is different from the first embodiment in the following points. In other words, in the present embodiment, the frame rate of the time-series three-dimensional data held in the time-series data holding unit 103 is changed based on the speed of the moving body on which the stereo camera 100 is mounted. In other respects, the present embodiment is the same as the first embodiment, and a duplicate description of the first embodiment is omitted.

  The three-dimensional image creation method according to this embodiment is performed according to the flowcharts shown in FIGS. 5 to 7 as in the first embodiment. Here, in the holding of the three-dimensional data in step S501 in FIG. 5, as described above, the frame rate of the time-series three-dimensional data held in the time-series data holding unit 103 is set to the moving body in which the stereo camera 100 is mounted. Vary based on the speed of the. Except for the contents related to step S501 in FIG. 5, the present embodiment is the same as the first embodiment, and a description thereof will be omitted.

  The frame rate of the time-series three-dimensional data held in the time-series data holding unit 103 can be set as follows, for example. That is, when the speed of the moving body is low, the frame rate of the time-series three-dimensional data held in the time-series data holding unit 103 is 4 of the frame rate of the three-dimensional data received from the three-dimensional data generation unit 101. Set to 1 / (low speed). When the speed of the moving body is medium, the frame rate of the time-series three-dimensional data held in the time-series data holding unit 103 is three times the frame rate of the three-dimensional data received from the three-dimensional data generation unit 101. 1 (medium speed). When the speed of the moving object is high, the frame rate of the time-series three-dimensional data held in the time-series data holding unit 103 is two times the frame rate of the three-dimensional data received from the three-dimensional data generation unit 101. 1 (high speed). As described above, the frame rate of the time-series three-dimensional data held in the time-series data holding unit 103 is lowered as the speed of the moving body is lowered. This is because the change in the image of the frame of the three-dimensional data is small and the change in the image of the frame is considered to be small compared to before changing the frame rate.

  The data size held in the time-series data holding unit 103 is changed by changing the frame rate of the time-series three-dimensional data held in the time-series data holding unit 103 based on the speed of the moving body on which the stereo camera 100 is mounted. Reduce. As a result, the size of the memory constituting the time-series data holding unit 103 can be reduced, so that the cost and power consumption of the system can be reduced.

  FIG. 12 is a diagram illustrating an example of a flowchart relating to holding of three-dimensional data when the frame rate of the time-series data to be held is changed. That is, the flowchart shown in FIG. 12 corresponds to a specific example of step S501 in FIG.

  According to the flowchart of FIG. 12, when the frame rate is low, the frame rate of the time-series data to be held is ¼ of the frame rate of the three-dimensional data created by the three-dimensional data generation unit 101, and is three when the frame rate is medium. One half, and one half at high speed.

  Specifically, when the frame rate is low (S1202, low speed), after assigning an initial value 0 to the integer variable n (S1200), when n = 0, n is incremented to n = 1 (S1206), The three-dimensional data created by the three-dimensional data generation unit 101 is not retained (S1207, S1208, S1210). Even when n = 1 to 2 (S1203, No), n is incremented (S1206), and the 3D data created by the 3D data generation unit 101 is not held (S1207, S1208, S1210). When n = 3 (S1203, Yes), n = 0 is forcibly set (S1205), and the three-dimensional data created by the three-dimensional data generation unit 101 is held (S1207 to S1209).

  In other words, when the frame rate is low, the three-dimensional data is retained only when the value of n immediately before the determination of whether or not to retain the three-dimensional data is 0 among 0, 1, 2, and 3 (Yes in S1208). (S1209). Therefore, the frame rate of the three-dimensional data to be held can be set to a quarter of the frame rate of the three-dimensional data created by the three-dimensional data generation unit 101.

  When the frame rate is medium speed (S1202, medium speed), after assigning an initial value 0 to n (S1200), when n = 0, n is incremented to n = 1 (S1206), and the three-dimensional data generation unit 101 Does not hold the created three-dimensional data (S1207, S1208, S1210). Even if n = 1 (S1204, No), n is incremented to n = 2 (S1206), and the 3D data created by the 3D data generation unit 101 is not held (S1207, S1208, S1210). When n = 2 (S1204, Yes), n = 0 is forcibly set (S1205), and the three-dimensional data created by the three-dimensional data generation unit 101 is held (S1207 to S1209).

  That is, when the frame rate is medium, the three-dimensional data is retained only when the value of n immediately before the determination of whether or not to retain the three-dimensional data is 0 among 0, 1, and 2 (Yes in S1208). (S1209). Therefore, the frame rate of the three-dimensional data to be held can be set to one third of the frame rate of the three-dimensional data created by the three-dimensional data generation unit 101.

  When the frame rate is high (S1202, high speed), after assigning an initial value 0 to n (S1200), n is incremented to n = 1 (S1206), and the 3D data created by the 3D data generation unit 101 is displayed. Not held (S1207, S1208, S1210). When n = 1 (S1215, Yes), n = 0 is forcibly set (S1205), and the three-dimensional data created by the three-dimensional data generation unit 101 is held (S1207 to S1209).

  That is, when the moving body is at high speed, the three-dimensional data is held only when the value of n immediately before the determination of whether or not to hold the three-dimensional data is 0 of 0 and 1 (Yes in S1208) (S1209). . Therefore, the frame rate of the three-dimensional data to be held is one half of the frame rate of the three-dimensional data created by the three-dimensional data generation unit 101.

  The holding of the three-dimensional data according to the frame rate continues until all measurement points of the frame are completed (S1211, S1212). When the processing of the frame is completed, unless there is a system stop request (S1213), the subsequent The processing of the frame to be performed is continued (S1214).

  Thus, according to this embodiment, in addition to the effect which 1st embodiment has, there exist the following effects. That is, according to the present embodiment, the time-series data is changed by changing the frame rate of the time-series three-dimensional data held in the time-series data holding unit 103 based on the speed of the moving body on which the stereo camera 100 is mounted. The data size held in the holding unit 103 is reduced. As a result, the size of the memory constituting the time-series data holding unit 103 can be reduced, so that the cost and power consumption of the system can be reduced.

[Fourth embodiment]
A 3D image creation system and a 3D image creation method according to a fourth embodiment of the present invention will be described in detail.

  This embodiment is different from the first embodiment in the following points. That is, in this embodiment, there are a plurality of stereo cameras 100, and the three-dimensional data of the wide field of view obtained by integrating the three-dimensional data of each stereo camera 100, the current three-dimensional data is replaced with past three-dimensional data and movement detection. Based on the movement detected by the means, interpolation is performed and integration is performed. As a result, missing data due to occlusion is reduced, and an around view three-dimensional image with an expanded field of view is obtained.

  Furthermore, the present embodiment integrates a three-dimensional image of a moving object (hereinafter referred to as a “moving object model”) into the three-dimensional data obtained by interpolating the occlusion region, and the moving object viewed from an arbitrary position outside the moving object. Be able to grasp the position.

  In other respects, the present embodiment is the same as the first embodiment, and a duplicate description of the first embodiment is omitted.

  FIG. 13 is a diagram illustrating a relative positional relationship between the moving body and the stereo camera of the three-dimensional image creation system according to the present embodiment.

  As illustrated in FIG. 13, the three-dimensional image creation system according to the present embodiment includes a plurality of stereo cameras 100 mounted on a moving body 1300. The stereo camera 100 is mounted on the moving body 1300 so that a three-dimensional image with a wider field of view (ideally a 360-degree field of view) can be obtained with each camera as a whole.

  As shown in FIG. 13A, when the four stereo cameras 100 are mounted so as to be shifted from each other by 90 degrees, a three-dimensional image with a wider field of view can be obtained with a minimum number of cameras. However, this embodiment does not limit the number of stereo cameras 100. For example, as shown in FIG. 13B, eight stereo cameras 100 may be used. By increasing the number of stereo cameras 100 mounted at appropriate positions on the moving body 1300, a three-dimensional image with a wider field of view can be obtained. Hereinafter, in the description of the present embodiment, a case where four stereo cameras 100 are used will be described.

  FIG. 14 is a block diagram showing a three-dimensional image creation system according to this embodiment. As in the first embodiment, the 3D image creation system 10 according to the present embodiment includes stereo cameras 100a to 100d, a 3D data generation unit 101, a movement amount calculation unit 102, a time-series data holding unit 103, and an output unit 105. Have. Furthermore, the present embodiment includes an area setting unit (area setting unit) 1400, a moving body model holding unit (moving body model image holding unit) 1401, and a display unit (display unit) 1402.

  The three-dimensional data generation unit 101 includes a three-dimensional calculation unit (individual three-dimensional data generation unit) 1403a to 1403d, a coordinate conversion unit (three-dimensional coordinate conversion unit) 1404a to 1404d, and a three-dimensional data integration unit (three-dimensional data integration unit). 1405.

  Three-dimensional calculation units 1403a to 1403d receive standard image data and reference image data from stereo cameras 100a to 100d, respectively, and calculate three-dimensional data based on these image data. The three-dimensional data is calculated from the stereo cameras 100a to 100d based on the principle of triangulation from the parallax, the base line length of the stereo cameras 100a to 100d, and the focal length from the standard image data and the reference image data. Obtained by calculating the distance to the object. That is, each of the three-dimensional calculation units 1403a to 1403d represents the images taken by the stereo cameras 100a to 100d in the coordinate system of the stereo cameras 100a to 100d (hereinafter referred to as “stereo camera coordinate system”). Calculate as 3D data.

  The coordinate conversion units 1404a to 1404d use the three-dimensional data of the stereo camera coordinate system calculated by the respective three-dimensional calculation units 1403a to 1403d as the tertiary of the coordinate system of the moving body 1300 (hereinafter referred to as “moving body coordinate system”). Convert to original data. The conversion is performed by applying conversion matrices Mat1 to Mat4 representing the positions and orientations of the stereo cameras 100a to 100d in the moving body coordinate system to the three-dimensional data of the stereo camera coordinate system. The transformation matrices Mat1 to Mat4 may be stored in advance in the memory in the three-dimensional data generation unit 101, and the coordinate transformation units 1404a to 1404d may refer to them.

  The three-dimensional data integration unit 1405 integrates the four three-dimensional data converted by the coordinate conversion units 1404a to 1404d, and generates a three-dimensional scene in the measurement space.

  The time-series data holding unit 103 holds the three-dimensional data and reliability received from the three-dimensional data generation unit 101, and the object movement vector and the moving object movement vector received from the movement amount calculation unit. The retained data can be read as data at a desired time.

  The time-series data integration unit 104 receives the three-dimensional data and the reliability from the three-dimensional data generation unit 101, and the past three-dimensional data and the reliability from the time-series data holding unit 103. When the current three-dimensional data includes occlusion area data, the data integration unit 104 interpolates and integrates the occlusion area data with past three-dimensional data that satisfies a predetermined condition. Thereby, a three-dimensional image in which the visual field range is enlarged is generated.

  The region setting unit 1400 receives the region setting signal input by the user and sets the region of the three-dimensional data generated by the time series data integration unit 104. That is, a region of 3D data that the user wants to visually recognize is set.

  The movement amount calculation unit 102 calculates a movement vector of the moving body 1300. For example, the movement vector may be calculated from the measurement result of the acceleration sensor. Further, the movement amount calculation unit 102 calculates the movement vector of the object in the image.

  The moving body model holding unit 1401 is a storage device that holds model data (moving body model) related to the appearance of the moving body 1400 for displaying the moving body 1400 superimposed on three-dimensional data.

  The output unit 105 superimposes the moving body model held by the moving body model holding unit 1401 on the 3D data integrated by the time series data integration unit 104 and outputs the superimposed data to the display unit 1402.

  The output unit 105 may cause the display device to display the area in which the time-series data integration unit 104 interpolates the three-dimensional data in one of blinking display, shaded display, and gray display. Thereby, the user can visually recognize the data interpolated by the three-dimensional image creation system 10.

  The display unit 1402 displays the 3D image created by the time series data integration unit 104. The three-dimensional image can include a moving body model.

  FIG. 15 is a diagram illustrating a flowchart of the three-dimensional image creation method according to the present embodiment.

  Hereinafter, the three-dimensional image creation method according to the present embodiment will be described in detail with reference to the flowchart shown in FIG. The 3D image creation method according to the present embodiment is performed by the 3D image creation system shown in FIG.

  Based on the images acquired by the stereo cameras 100a to 100d (S1500), the three-dimensional calculation units 1403a to 1403d create three-dimensional data expressed in the stereo camera coordinate system (S1501).

  The three-dimensional data of the stereo cameras 100a to 100d created by the three-dimensional calculation units 1403a to 1403d are converted into three-dimensional data of the moving object coordinate system by the coordinate conversion units 1404a to 1404d, and then the three-dimensional data integration unit 1405. Integrated in.

  The time-series data holding unit 103 holds the three-dimensional data integrated by the three-dimensional data integration unit 1405 (S1503).

  The region setting unit 1400 receives the region setting signal input by the user, and sets the region of the three-dimensional data generated by the time series data integration unit 104 (S1505). The time series data integration unit 104 interpolates the data of the occlusion area with the past three-dimensional data satisfying a predetermined condition when the data of the occlusion area is included in the current three-dimensional data. Integration is performed (S1505). At this time, three-dimensional data of the region set by the region setting unit 1400 is created. That is, the time-series data integration unit 104 performs coordinate conversion of the three-dimensional data in order to match the three-dimensional data of the region set by the region setting unit 1400 before or after the interpolation of the three-dimensional data.

  The output unit 105 superimposes the moving body model held by the moving body model holding unit 1401 on the three-dimensional data created by the time series data integration unit 104 and outputs the superimposed data to the display unit 1402 (S1506).

  The display unit 1402 displays an image of the three-dimensional data received from the output unit 105 (S1507).

  Steps S1500 to S1507 are repeated until a control signal for processing end is received (S1508).

  The 3D image creation system and the 3D image creation method according to the fourth embodiment of the present invention have been described above, but this embodiment has the following effects.

  A wide field of view 3D data that has a plurality of stereo cameras and integrates the 3D data of each stereo camera. Based on the past 3D data and the movement detected by the movement detection means. Interpolate and integrate. As a result, it is possible to reduce the data loss due to occlusion and to obtain an around-view three-dimensional image with an expanded visual field range.

  Furthermore, this embodiment can grasp | ascertain the position of the moving body seen from the arbitrary positions outside a moving body by integrating a moving body model into the three-dimensional data which interpolated the occlusion area | region.

  As mentioned above, although this invention has been demonstrated based on embodiment, the scope of the present invention is not limited to embodiment.

  For example, in the above-described embodiment, a stereo camera is used as the compound eye image system, but the compound eye image system may be configured by two single cameras.

10 3D image creation system,
100, 100a, 100b, 100c, 100d stereo camera,
100a reference camera,
100b reference camera,
101 3D data generation unit (3D data generation means, stationary object determination means),
102 movement amount calculation unit (movement detection means, stationary object determination means),
103 time series data holding unit (time series data holding means),
104 Time series data integration unit (time series data integration means),
105 output section (output means),
200a, 200b mobile body,
210, 210a, 210b occlusion area,
220 obstacles,
400 objects,
1300 mobile,
1400 area setting unit (area setting means),
1401 moving body model holding unit (moving body model image holding means),
1402 Display unit (display means),
1403a, 1403b, 1403c, 1403d three-dimensional calculation unit (individual three-dimensional data generation means),
1404a, 1404b, 1404c, 1404d coordinate converter (three-dimensional coordinate converter),
1405 Three-dimensional data integration unit (three-dimensional data integration means).

Claims (38)

One or more stereo cameras attached to the moving body;
Three-dimensional data generation means for generating three-dimensional data in time series from images taken by the stereo camera;
Movement detecting means for detecting movement of the moving body;
Time-series data integration means for interpolating and integrating the three-dimensional data generated by the three-dimensional data generation means based on the past three-dimensional data and the movement detected by the movement detection means;
A three-dimensional image creation system characterized by comprising:   2. The time-series data integration unit integrates data corresponding to an occlusion area of the three-dimensional data generated by the three-dimensional data generation unit by interpolating with the past three-dimensional data. 3D image creation system described in 1.   The three-dimensional image creation system according to claim 2, wherein the occlusion area is an area where the reliability of the correspondence between the images captured by the two cameras constituting the stereo camera is low.   The three-dimensional image creation system according to claim 1, wherein the movement detected by the movement detection unit is a movement vector of the moving body.   The movement detection means detects a movement vector of the moving body based on one or more measured values of an acceleration sensor, a gyro sensor, and a correlation calculation between frames of the image. The three-dimensional image creation system according to claim 4.   The three-dimensional image creation system according to claim 5, wherein the correlation calculation is performed by any one of a phase-only correlation method, a sum of absolute differences, a sum of squares of differences, and a normalized cross correlation. The number of the stereo cameras is four,
The three-dimensional data generation means includes individual three-dimensional data generation means for individually generating three-dimensional data in time series from images taken by each stereo camera,
3D for converting the 3D data of each stereo camera generated by the individual 3D data generation means into the 3D coordinate system of the mobile body based on the known positional relationship between each stereo camera and the mobile body Coordinate transformation means;
3D data integration means for integrating the 3D data of each stereo camera converted by the 3D coordinate conversion means and creating 3D data;
The three-dimensional image creation system according to claim 1, wherein: Further comprising time-series data holding means for holding the three-dimensional data generated by the three-dimensional data generation means and the movement of the moving body detected by the movement detection means in time series,
The time-series data integration unit is configured based on the three-dimensional data held by the time-series data holding unit, the past three-dimensional data held by the time-series data holding unit, and the movement of the moving body. Interpolate and integrate
The three-dimensional image creation system according to claim 1, wherein the time-series data holding unit changes a frame rate of the time-series data to be held based on the speed of the moving body. Output means for outputting the three-dimensional data integrated by the time-series data integration means for display on a display means;
Display means for displaying the three-dimensional data output by the output means;
The three-dimensional image creation system according to claim 1, further comprising: A moving body model image holding means for holding a moving body model image that is an image of the moving body;
The three-dimensional image according to claim 9, wherein the output unit outputs the moving body model image held by the moving body model image holding unit for display on the display device together with the three-dimensional data. Creation system.   The three-dimensional image creation system according to claim 10, wherein the moving body model image is a three-dimensional image.   The three-dimensional image creation system according to claim 1, further comprising a region setting unit that sets a region of the three-dimensional data created by the time-series data integration unit. Whether the image included in the image captured by the stereo camera is an image of a stationary object, based on the image captured by the stereo camera and the movement vector of the moving object, before and after the movement of the moving object It further has a stationary object judging means for judging by calculating a change in the position vector of the object,
The three-dimensional image creation system according to any one of claims 1 to 12, wherein the three-dimensional data interpolated by the time-series data integration unit is three-dimensional data of an image of a stationary object.   The three-dimensional image creation system according to any one of claims 1 to 13, wherein the three-dimensional data interpolated by the time-series data integration unit has high reliability.   The three-dimensional image creation system according to any one of claims 1 to 14, wherein the three-dimensional data interpolated by the time-series data integration unit is an image photographed closest.   The three-dimensional image creation system according to claim 8, wherein the three-dimensional data held by the data holding unit is three-dimensional data of an image of a stationary object.   The three-dimensional image creation system according to any one of claims 8 to 16, wherein the three-dimensional data held by the data holding unit has high reliability.   The three-dimensional image creation system according to claim 8, wherein the three-dimensional data held by the data holding unit is an image taken closest.   The output means displays the area interpolated by the time-series data integration means on the display device in any one of blinking display, shaded display, and gray display. The three-dimensional image creation system according to any one of 18. A three-dimensional data generation stage for generating three-dimensional data in time series from images taken by one or more stereo cameras attached to a moving body;
A movement detecting step for detecting movement of the moving body;
A time-series data integration step of interpolating and integrating the three-dimensional data generated in the three-dimensional data generation step based on the past three-dimensional data and the movement detected in the movement detection step;
A three-dimensional image creation method characterized by comprising:   21. The time-series data integration step includes integrating the data corresponding to the occlusion region of the three-dimensional data generated in the three-dimensional data generation step by interpolating with the past three-dimensional data. The three-dimensional image creation method as described in 2.   The three-dimensional image creation method according to claim 21, wherein the occlusion region is a region having a low reliability of a correspondence relationship between images captured by two cameras constituting the stereo camera.   23. The three-dimensional image creation method according to claim 20, wherein the movement detected in the movement detection step is a movement vector of the moving body.   In the movement detection step, a movement vector of the moving body is detected based on one or more measurement values of an acceleration sensor, a gyro sensor, and a correlation calculation between the frames of the image. The three-dimensional image creation method according to claim 23.   25. The three-dimensional image creation method according to claim 24, wherein the correlation calculation is performed by any one of a phase-only correlation method, a sum of absolute differences, a sum of squares of differences, and a normalized cross correlation. The number of the stereo cameras is four,
In the three-dimensional data generation step, an individual three-dimensional data generation step that individually generates three-dimensional data in time series from images taken by each stereo camera;
A tertiary that converts the three-dimensional data of each stereo camera generated in the individual three-dimensional data generation step into a three-dimensional coordinate system of the moving body based on a known positional relationship between each stereo camera and the moving body. An original coordinate transformation stage;
Integrating the three-dimensional data of each stereo camera converted in the three-dimensional coordinate conversion step, and creating a three-dimensional data, a three-dimensional data integration step;
The three-dimensional image creation method according to any one of claims 20 to 25, wherein: A time-series data holding step for holding the three-dimensional data generated in the three-dimensional data generation step and the movement of the moving body detected in the movement detection step in time series;
The time series data integration step is based on the three-dimensional data held in the time series data holding step, the past three-dimensional data held in the time series data holding step, and the movement of the moving body. Interpolate and integrate
27. The three-dimensional image creation method according to claim 20, wherein the time-series data holding step changes a frame rate of the time-series data to be held based on a speed of the moving body. An output step for outputting the three-dimensional data integrated in the time series data integration step for display on a display means;
A display step for displaying the three-dimensional data output in the output step;
The three-dimensional image creation method according to claim 20, further comprising: A moving body model image holding step for holding a moving body model image that is an image of the moving body;
29. The three-dimensional image according to claim 28, wherein the outputting step outputs the moving body model image held in the moving body model image holding step for display on a display device together with the three-dimensional data. How to make.   30. The three-dimensional image creation method according to claim 29, wherein the moving body model image is a three-dimensional image.   31. The three-dimensional image creation method according to claim 20, further comprising a region setting step of setting a region of the three-dimensional data created in the time series data integration step. Whether the image included in the image captured by the stereo camera is an image of a stationary object, based on the image captured by the stereo camera and the movement vector of the moving object, before and after the movement of the moving object A stationary object determination step of determining by calculating a change in the position vector of the object;
The three-dimensional image creation method according to any one of claims 20 to 31, wherein the three-dimensional data to be interpolated in the time series data integration step is three-dimensional data of an image of a stationary object.   The three-dimensional image creation method according to any one of claims 20 to 32, wherein the three-dimensional data to be interpolated in the time series data integration step has high reliability.   The three-dimensional image creation method according to any one of claims 20 to 33, wherein the three-dimensional data to be interpolated in the time series data integration step is an image photographed closest.   35. The three-dimensional image creation method according to claim 27, wherein the three-dimensional data held in the data holding step is three-dimensional data of a still object image.   36. The three-dimensional image creation method according to any one of claims 27 to 35, wherein the three-dimensional data held in the data holding step has high reliability.   37. The three-dimensional image creation method according to any one of claims 27 to 36, wherein the three-dimensional data held in the data holding step is an image taken closest.   29. The output stage displays the area interpolated with the three-dimensional data in the time series data integration stage on the display device in one of blinking display, shaded display, and gray display. 37. The three-dimensional image creation method according to any one of 37.