JP2004265396A - Image forming system and image forming method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image forming system and an image forming method which easily generates a realistic walk-through animation based on actually photographed images. <P>SOLUTION: The system includes a vehicle 10, video cameras 30 for photographing surrounding images to output their image data, a laser range scanner 20 outputting depth information, and a computer 40. The computer 40 identifies the positions of surrounding objects at predetermined intervals based on the depth information, identifies a running path T of the vehicle 10 relative to the identified positions of surrounding objects, and carries out a process of calculating position information presenting the position of the vehicle 10 on the running path T and advance direction information presenting an advance direction. The computer 40 further carries out a process of generating a panorama image Pn presenting images of all surrounding directions viewed from the position on the running path T at predetermined intervals based on the image data, and a process of generating a walk-through animation based on the depth information, the position information, the advance direction information and the panorama image. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

  The present invention relates to a video generation system and a video generation method, and more particularly, to a video generation system and a video generation method for generating a walk-through animation realizing realism.

In recent years, an actual city environment is virtually generated in a computer, and the virtual three-dimensional city space is used for a navigation system, a game, a movie, and the like. The user can virtually move in the virtual three-dimensional city space, and can experience as if moving in a real city by the walk-through animation.
This walk-through animation is used in various fields, such as road guidance, tourist information, museum information, and real estate sales. For example, as a tourist guide, a bus tour, a drive, and the like can be experienced through a walk-through animation in a “digital city” that is a virtual 3D city space.
Further, a two-dimensional map is converted into a three-dimensional map, which is used for a navigation system for a car, a pedestrian, and the like.

As a guide for museums, a service for conducting a virtual tour of the museum on a homepage is provided. With this service, when a desired room in the museum is selected on the homepage, a 360-degree panoramic image of the room viewed from the center of the room is displayed, and the user can freely look around the artworks arranged in each room. Can be.
In addition, as real estate sales, a service is provided that allows three-dimensional viewing of a model room or a video near a property on a homepage. With this service, you can freely move the images and tour the property as if you were in the model room. In addition, you can see the route from the nearest station to the property on the screen. With this service, users can view the environment around the property and get an image of the surroundings without having to go to the site.

There are roughly two methods for generating the virtual space and the walk-through animation as described above.
The first is a method called model-based rendering, in which a “three-dimensional shape” of an object is input (modeled). As a specific example, it is used to create a three-dimensional expression in walk-through animation, VRML, CAD, a game, or the like.

  When the two-dimensional map data is converted to three-dimensional data, it is necessary to actually obtain the height information of the feature in order to specify the three-dimensional shape. As this technique, a technique has been proposed in which an image of a feature is acquired from two directions, and the height of the feature is calculated from the image and position information (for example, see Patent Document 1). With this technology, although it is possible to efficiently acquire the height information of a feature, it takes an extremely long time to calculate the height information individually for a feature in a vast area, and a realistic three-dimensional space is formed. There was a problem that it took time and effort.

In addition, the model-based rendering requires a very specialized technique and has a problem that it is difficult to obtain optical properties such as a surface shape and a reflection parameter of an actual object. In other words, this method has a drawback that the appearance is poor compared to the actual space because a simple color and reflection model are used as the optical characteristics.
In order to improve this, a method of pasting an actual photograph as a texture on a side surface of a building in a virtual space has been proposed (for example, see Patent Document 2). However, this method has a disadvantage that it takes much time and cost to acquire the texture. Also, no matter how sophisticated the technology, it is not possible to completely reproduce the reality of the real world, so it is not possible to actually know the environment and atmosphere of the place.

The second is an image-based rendering technique. In this method, a large amount of images of the real world, such as photos and videos, are accumulated, and these real images are processed, processed, and reconstructed in various ways to create videos from viewpoints other than the shooting point. It is.
That is, in this method, the appearance of an object or a scene itself is recorded and accumulated as an input image without generating a model, and a new image is generated by re-synthesizing these. Since this method does not need to consider the shape and reflection characteristics of the target object, it can be applied to an object that is difficult to model, and can obtain a reality that is approximately the same as that of the actual space (for example, And Patent Document 3).

In order to generate a wide-area space using such image-based rendering, a large amount of real-world images of urban space, the position and direction in which the images were taken, and depth information from the shooting point of the object reflected in the actual images are obtained. Required.
For example, there is an information acquisition method using stereo matching in order to acquire depth information from a shooting point of an object. In this method, first, an actual photographed image is actually photographed and acquired in an urban space by a vehicle-mounted photographing system. In order to further obtain depth information, corresponding point matching using epipolar constraints is performed from two images with known position and orientation information obtained by GPS etc., and the depth value of the corresponding point is derived using the principle of stereo matching Take the way to. Then, a virtual space is generated based on a large amount of photographed images and depth information.

  Alternatively, there is a method of acquiring depth information by EPI analysis. In this method, a photographed image is acquired using a vehicle-mounted imaging system. For estimating the depth information, which is the distance between the camera and the building, a method of examining the inclination on an EPI (Epipolar Plane Image) image and estimating the depth information of the object is used. Based on a large amount of real images and depth information, the concept of a ray space is used to realistically construct a wide-area environment called an urban space.

JP-A-2000-74669 (pages 4 to 8) JP-A-2003-6680 (pages 3 to 19) JP-A-10-214069 (pages 4 to 10)

  In the existing technology targeting a wide area, the position and direction of the image captured in this way are acquired by GPS or the like. However, GPS positioning has a problem that there are many areas where accurate positions cannot be received, such as the valleys of buildings.

In obtaining depth information, depth information is estimated mainly by processing an obtained image. In order to obtain depth information by such image processing, it is necessary to: 1. It is difficult to obtain accurate three-dimensional information due to the influence of vibration, noise, changes in the light source environment, and the like. 2. The shooting path must be straight; There are many problems such as the condition that the speed of the camera is constant.
In fact, when performing a process of estimating a position or a depth from image data or the like acquired on a road, it has been quite difficult to obtain accurate position information and depth information due to various environmental factors.

Further, in the conventional walk-through animation, for example, when a car or the like makes a right or left turn at an intersection, the displayed image is switched from the image of the currently running road to the image of the road after a sudden turn. For this reason, there was a problem that it was not known whether the vehicle really turned or whether the vehicle turned right or left.
Furthermore, at intersections, there are many features having valuable information such as stores and signboards, and there is a problem that if the video is suddenly switched, such information may not be sufficiently grasped.

  In order to solve this problem, when a continuous image of turning right and left at an intersection (for example, a crossroad) is provided by a walk-through animation, the image to be actually acquired is a right / left turn × 4 directions per intersection. A total of twelve patterns of straight traveling × (two straight lines in the vertical direction + two straight lines in the horizontal direction) are required, which is very inefficient.

An object of the present invention is to provide a video generation system and a video that can easily generate a realistic walk-through animation by using a method called image-based rendering that expresses a wide area by video processing based on a live-action video. It is to provide a generation method.
Another object of the present invention is to provide a video generation system and a video generation method capable of generating a real right / left turn video at an intersection or a video from an arbitrary viewpoint without any trouble as in the related art. It is in.

  According to the image generation system of the present invention, the object is to provide a moving body, a camera which is provided on the moving body, captures a video in a circumferential direction of the moving body and outputs image data, and a camera which is mounted on the moving body. A horizontal laser that is provided and outputs depth information indicating a distance to an object around the moving object; and a data processing device that processes the image data and the depth information. The data processing device includes the depth information. The position of the surrounding object with respect to the moving object is specified at predetermined time intervals based on the specified moving object, and the traveling trajectory of the moving object moved with respect to the specified position of the surrounding object is identified on the traveling trajectory. A process of calculating position information indicating the position of the moving object and traveling direction information representing the traveling direction; and generating a panoramic image displaying the entire circumferential direction viewed from the traveling locus at predetermined time intervals based on the image data. A process of the depth information, the position information, on the basis of the travel direction information and the panorama image is solved by performing the processing of generating a walk-through animation.

As described above, according to the video generation system of the present invention, it is possible to specify the position information on the travel locus of the moving body from the depth information measured by the horizontal laser without using a positioning device such as a GPS. . According to the horizontal laser, accurate distance measurement is possible even at the position of a valley such as a building.
Further, by using a horizontal laser, the depth distance from the measurement point to an object such as a building can be directly measured, and the positional accuracy is improved.
Then, in the present invention, it is possible to generate a realistic walk-through animation by using the actually shot video and the depth information.

  Further, in the processing for generating the walk-through animation, the data processing device may use one of the traveling trajectories based on the depth information, the position information, the traveling direction information, and the panorama image in the traveling trajectories that intersect each other. Performing an intersection right / left turn image generation process that generates an image that changes the traveling direction to the travel locus, and in the intersection right / left turn image generation process, the viewpoint of the walk-through animation in one travel locus intersects the travel locus. , The part of the panoramic image in the one traveling locus whose gaze is shifted in the circumferential direction from the traveling direction is set as the display image of the walk-through animation.

  As described above, in the present invention, it is possible to generate a natural walk-through animation when turning from one running locus to the other running locus only by acquiring a straight ahead image on the intersecting running locus. Therefore, the number of images to be acquired can be reduced, so that no labor is required and the time and cost for generating images can be greatly reduced.

Further, the data processing device is characterized in that, in the process of generating the walk-through animation, an arbitrary viewpoint video generation process of generating a video at a viewpoint outside the travel trajectory is performed.
More specifically, in the process of generating the walk-through animation, the data processing device performs an arbitrary viewpoint video generation process of generating an image of a predetermined viewing angle range from a viewpoint outside the travel trajectory. In the viewpoint video generation processing, each intersection at which the straight line drawn from the viewpoint in each angle direction within the viewing angle range intersects with the traveling locus is specified, and the panorama image at each intersection includes the viewpoint. Cut out the image portion, calculate the distance from the intersection to the surrounding objects in the angular directions and the distance from the viewpoint to the surrounding objects in the angular directions based on the depth information, and calculate these. A distance ratio is calculated, the image portion is enlarged or reduced by the ratio to generate a partial image outside the travel path, and the partial image outside the travel path is connected to travel. Atogai can be configured to generate an image.

  As described above, in the present invention, it is possible to generate an image from a viewpoint other than the traveling locus by enlarging, reducing, and connecting the video from the traveling locus. This makes it possible to generate a natural-looking and realistic video.

  An image generation method according to the present invention includes a step of obtaining image data in a circumferential direction of a shooting position with an omnidirectional camera, and depth information between the horizontal laser and a surrounding object by a horizontal laser measuring a distance to a surrounding object. Obtaining the position of the surrounding object with respect to the moving body at predetermined time intervals based on the depth information, and a traveling locus of the moving body with respect to the specified position of the surrounding object. Calculating position information representing the position of the moving object on the travel locus and traveling direction information representing the traveling direction, and the entire periphery viewed from the travel locus at predetermined time intervals based on the image data. Generating a panoramic image indicating a direction, and generating a walk-through animation based on the depth information, the position information, the traveling direction information, and the panoramic image , Characterized by comprising a.

  In the step of generating the walk-through animation, the viewpoint of the walk-through animation in one traveling locus is determined based on the depth information, the position information, the traveling direction information, and the panoramic image in the traveling locus that intersects with each other. As the vehicle approaches the intersection of the travel trajectories, a process is performed in which a portion of the panorama image in the one travel trajectory, in which the line of sight is shifted in the circumferential direction from the traveling direction, is used as the display image of the walk-through animation. .

Further, in the step of generating the walk-through animation, a process of generating a video image at a viewpoint outside the travel locus is performed.
More specifically, in the step of generating the walk-through animation, each intersection point at which the straight line drawn from the viewpoint outside the running locus in each angle direction within the viewing angle range intersects with the running locus is specified. Of the panoramic image at each intersection, an image portion including the viewpoint is cut out, and the distance from each intersection to a surrounding object in each angular direction and the distance from the viewpoint to a surrounding object in each angular direction. Is calculated based on the depth information, and the ratio of these distances is calculated.The image portion is enlarged or reduced by the ratio to generate a partial image outside the travel locus, and the partial images outside the travel locus are connected. And generating a video image outside the traveling locus.

  In the present specification, the animation is a moving image, and more specifically, means that a moving image composed of a live-action image performs a display similar to a case where a predetermined place is moved. .

According to the present invention, there is provided an image generation system and an image generation method that can easily generate a realistic walk-through animation by using a real image obtained by an onboard camera system and depth information obtained by a laser range scanner. Can be provided.
Further, it is possible to reproduce a real right / left turn image at an intersection or an image from an arbitrary viewpoint without taking time and effort as in the related art.

  An embodiment of the present invention will be described below with reference to the drawings. In the following description, a video generation system and a video generation method according to the present invention are described. In this description example, the present invention provides a video generation method and a program for causing the video generation system to perform predetermined processing. And a storage medium.

FIGS. 1 to 37 relate to an embodiment of the present invention. FIG. 1 is an explanatory diagram of an image generating system, FIG. 2 is an explanatory diagram showing a configuration of the image generating system, and FIG. 3 is a video camera and a laser range scanner. FIG. 4 is an explanatory diagram for calculating a video overlap distance by a video camera.
5 to 9 relate to the processing of the depth information, FIG. 5 is a flowchart of the traveling trajectory data acquisition processing, FIG. 6 is an explanatory diagram showing an example of the range image, and FIG. 7 is an explanation for explaining the data processing of the range image. FIG. 8 and FIG. 8 are explanatory diagrams showing an example of the traveling locus image, and FIG. 9 is an explanatory diagram showing an example of the traveling locus data.
10 to 15 relate to the panoramic image generation processing, FIG. 10 is a flowchart of the panorama image generation processing, FIG. 11 is an explanatory diagram showing an example of images from three adjacent video cameras, and FIG. FIG. 13 is an explanatory diagram illustrating an example of a video camera image from which an overlapping portion has been deleted, FIG. 14 is a histogram of an overlapping portion at an end of an adjacent video camera image, and FIG. FIG. 8 is an explanatory diagram illustrating a process of generating a panoramic image from images obtained by eight video cameras.

16 to 18 relate to the right / left turn image generation processing, FIG. 16 is an explanatory diagram of a right / left turn image generation processing at an intersection, FIG. 17 is a flowchart of a right / left turn image generation processing at an intersection, and FIG. It is explanatory drawing of the process which produces | generates a left turn image in an intersection.
19 to 25 relate to an arbitrary viewpoint video generation process, FIG. 19 is an explanatory diagram of a process for generating an arbitrary viewpoint video, FIG. 20 is a flowchart of a process for generating an arbitrary viewpoint video, and FIG. FIG. 22 is an explanatory diagram of a generating process, FIG. 22 is an explanatory diagram of a process of generating an arbitrary viewpoint video, FIG. 23 is an explanatory diagram of a panoramic image and pixel lines, and FIG. FIG. 25 is an explanatory diagram showing the arrangement of the generated pixel lines.
26 to 37 relate to a specific example, FIG. 26 is an explanatory diagram showing a traveling locus image used for generating a left-turn image, FIG. 27 is an explanatory diagram showing an example of a left-turn image, and FIG. FIG. 29 is an explanatory diagram showing a traveling locus image used for generating a right-turn image, FIG. 30 is an explanatory diagram showing an example of a right-turn image, and FIG. 31 is an explanatory diagram showing movement parameters for each frame. FIG. 32 is an explanatory diagram showing a traveling locus image used for generating an arbitrary viewpoint video, FIG. 33 is an explanatory diagram showing traveling locus data used for generating an arbitrary viewpoint video, and FIGS. It is explanatory drawing which shows the image from a viewpoint.

As schematically shown in FIG. 1, an image generation system S according to the present invention includes an automobile 10 as a moving body, a laser range scanner 20 as a horizontal laser mounted on the automobile 10, a video camera 30, And a computer 40 as an apparatus. The image generation system S can be moved by the automobile 10.
The video camera 30 and the laser range scanner 20 are electrically connected to a computer 40, respectively, and as shown in FIG. 2, image data from the video camera 30 and depth from the laser range scanner 20 to the side of a surrounding building or the like. The information is configured to be output to the computer 40.

As shown in FIG. 1, a system carrier 11 is fixed to a roof of an automobile 10, and an angle base 12 having a length of 30 cm, a width of 30 cm, and a height of 60 cm is fixed on the system carrier 11. A pedestal 31 is provided on the angle base 12. The laser range scanner 20 and the video camera 30 are mounted on the pedestal 31. The angle base 12 is provided in order to prevent the system carrier 11 and the vehicle body from appearing in an image captured by the video camera 30 by raising the position of the video camera 30.
Further, a vibration isolator (not shown) is mounted between the angle base 12 and the pedestal 31 in order to reduce micro vibrations and pitching. In this example, an insulator MN-5 manufactured by GELTEC is used as a vibration isolator. This anti-vibration material is made of α-gel, and its natural frequency is set low, so that it is possible to perform anti-vibration in a low frequency range, and anti-vibration in all directions, up, down, left, and right is expected.
The video camera 30 arranged in this way had a height from the ground of about 2.3 m.

  The laser range scanner 20 emits a laser in the horizontal direction and measures depth information (distance information) between the image generation system S and an object such as a building. As the laser range scanner 20, for example, LD-A manufactured by IBEO lasertechnik can be used. By using the laser range scanner 20, it is possible to directly measure the distance between an object such as a building and a shooting position and acquire depth information. Therefore, according to the laser range scanner 20, it is possible to acquire depth information that is less affected by external factors and the like and is more accurate than the depth information estimated by image processing from a captured image.

The laser header is of a single-row type, and is configured to make 20 rotations per second (one rotation is 300 degrees). The maximum measurement distance is 70 m, and 480 range points can be measured by one rotation of the laser header. That is, an object located 70 m away can be measured at point intervals of about 25 cm. The measurement error is 3 cm.
By this laser range scanner 20, depth information, that is, distance information between a sensor position (the main body of the laser range scanner 20) and an object (building) around the image generation system S, and laser irradiation direction information at this time Can be obtained continuously. Then, it becomes possible to acquire the side shape of the building from the depth information.
The depth information obtained by the laser range scanner 20 is output to the computer 40 as data of one cycle every time the laser header makes one rotation. The computer 40 receives this depth information, displays it on a monitor, and saves it in the storage unit.

The computer 40 includes a CPU, an input unit, an output unit, a storage unit, a display unit, and the like. If necessary, the computer 40 includes a communication control unit.
The CPU displays, stores, and processes data according to a predetermined program. The storage unit includes an HDD, a ROM, and a RAM. The ROM stores various basic programs for hardware control of the computer system. The RAM is configured to function as a work area for computer control. The HDD stores information such as an operating program and each program.
The computer 40 acquires data from the laser range scanner 20 and the video camera 30 and performs various data processing as described later. The depth information from the laser range scanner 20 is assigned a scan line number for each time and stored in the storage unit. The image data from the video camera 30 is stored with a frame number assigned for each time. The scan line number and the frame number are associated with a predetermined correspondence relationship, whereby the depth information and the image data are synchronized in time.
In this example, the computer 40 is mounted in the automobile 10. However, the computer 40 is placed in a place different from the car 10 and the data from the laser range scanner 20 or the video camera 30 is output to the computer 40 wirelessly or by wire, or is transmitted to the computer 40 via a storage medium. It may be configured to provide.

The video camera 30 constitutes an omnidirectional camera system, acquires images over 360 degrees around the vehicle 10, and outputs the acquired digital data to the computer 40.
In this example, a digital video camera having a wide conversion lens is used as the video camera 30. Further, in this example, in order to acquire an image in all directions of 360 degrees, an omnidirectional camera system is constructed by arranging eight identical video cameras 30 radially every 45 degrees. In this example, the video camera 30 capable of acquiring a moving image is used as an imaging unit (camera). However, the present invention is not limited to this, and a camera capable of acquiring a still image may be used as the imaging unit. Good.
Here, the eight video cameras 30 are referred to as cameras 1, 2,..., 8 in a clockwise direction from a camera facing directly in front of the automobile 10.
The angle of view based on the measured values of the video camera 30 of this example is 34 degrees in height and 46 degrees in width. The angle of view when the wide conversion lens is attached is 46 degrees in height and 60 in width.

As shown in FIG. 3, the eight video cameras 30 are radially arranged on an octagonal plate-shaped pedestal 31 so as to be equiangularly spaced from one another. The pedestal 31 is formed in a regular octagon obtained by cutting off four corners of a square plate having a side of 55 cm. That is, the video cameras 30 are arranged such that the centers of the lenses are at equal intervals of 45 ° on the circumference from the center O of the circle. At this time, the video camera 30 is disposed such that the radius from the center O of the circle to the position of the lens is the shortest distance. The distance from the center O to the position of the lens of the video camera 30 is about 30 cm.
The laser range scanner 20 is provided on the pedestal 31 such that the center thereof coincides with the center O.
In the present example, eight video cameras 30 are configured, but the number is not limited to this, and the number is not limited as long as an image covering 360 ° around can be obtained.

As shown in FIG. 4, the angle of view in the horizontal direction (lateral direction) of the video camera 30 is θ, the distance from the center O to the lens position of the video camera is r, and the images of two video cameras 30 adjacent from the center 0. Assuming that the distance to the position at which X overlaps is X, these relationships are as follows.
x / r = sin (180 ° −θ / 2) / sin ((θ / 2) −22.5 °)

The angle of view of the video camera 30 of this example (when a conversion lens is attached) is 60 °, and the distance from the center O of the pedestal 31 to the lens of the video camera 30 is 30 cm. The distance r to is calculated to be about 115 cm.
That is, the images of the two video cameras 30 adjacent to each other at a distance of about 115 cm from the center O of the pedestal 31 overlap. Therefore, in this example, an object within a circle having a radius of 115 cm from the center O of the pedestal 31 is not captured in the image, but if the object is outside a circle having a radius of 115 cm from the center O, an object in all directions of 360 ° can be recorded in the video. .
It is preferable that the eight video cameras 30 perform shooting in a manual mode while setting shooting conditions constant. In this example, high-density continuous frame still image recording of 30 frames per second was performed.
In addition, the brightness settings of all the video cameras 30 are fixed to be the same.

(Processing of depth information)
Next, processing of depth information acquired by the laser range scanner 20 will be described. FIG. 5 shows the procedure of the traveling locus data acquisition process.
The computer 40 continuously receives, from the laser range scanner 20, depth information for each cycle (distance information to a side surface of a surrounding building or the like in a relative direction with respect to the laser range scanner 20). Then, the computer 40 stores the depth information as the data of each time with the scan line number Ln attached for each cycle (step S1).
The computer 40 performs a process of generating a range image as shown in FIG. 6 for each piece of depth information to which each scan line number Ln is assigned (step S2). When depth information is plotted on a plane, a range image is obtained. In the range image, the shape of the side surface W of the building is represented by a set of points. A point in the center represents the center position of the laser range scanner 20. Thereby, the position of the surrounding building or the like with respect to the laser range scanner 20 is specified every predetermined time.

FIG. 7A shows an example in which range images corresponding to a plurality of scan line numbers Ln are superimposed. The computer 40 performs a process of extracting a line segment from the range image for each scan line number Ln (step S3). By this processing, the image shown in FIG.
Then, as shown in FIG. 4C, a process of superimposing a common portion of the image from which the line segment has been extracted is performed (step S4). This is because, by recognizing the common part of the line segment, moving the image from which the line segment is extracted in the vertical and horizontal directions and rotating it, the image from which the line segment of the immediately preceding scan line number Ln is extracted Is performed by superimposing.
At this time, since the image generation system S is moving, when the range images are superimposed, the center position of the laser range scanner 20 appears at a position shifted with time. Thereby, the moving direction vector of the video generation system S is calculated.

In this way, by superimposing the range images of the respective scan line numbers Ln in the same procedure, a traveling locus image as shown in FIG. 8 can be obtained. In the figure, T represents a travel locus that the image generation system S has moved. In this manner, the movement trajectory of the moving body 10 with respect to the specified position of the surrounding object can be specified.
The computer 40 generates the X and Y coordinates (coordinate information) of the image generation system S and the rotation angle of the image at the time corresponding to each scan line number Ln as travel locus data by performing a process of superimposing the range images. It is stored (step S5). In this manner, the position information and the traveling direction information (the front direction of the vehicle 10) on the traveling locus of the laser range scanner 20 with respect to the surrounding buildings and the like are obtained. FIG. 9 shows an example of travel locus data.
Further, from the depth information of each scan line number Ln, the distance from the side of the building on the traveling locus T at each time point, the road width, and the like can be obtained.
Note that the coordinate information is relative with respect to the position of the video generation system S at the start of the measurement.

(Panorama image generation processing)
Next, a process of generating a panoramic image Pn displaying all directions of 360 ° from image data acquired by the video camera 30 will be described with reference to FIG. In this process, the computer 40 connects eight videos acquired by the eight video cameras 30 in the horizontal direction to generate a panoramic image. The panoramic image is obtained by continuously connecting images around 360 ° viewed from the viewpoint.

In this example, the image captured by the video camera 30 is an image of 640 × 480 pixels as shown in FIG. In the drawing, cameras 1, 8 and 2 are images obtained by the video camera 30 for photographing the front of the automobile 10, 45 ° left and 45 ° right, respectively. Although not shown, in the case of this example, eight images of the cameras 1 to 8 are obtained corresponding to each frame number.
As described above, the imaging ranges of the adjacent video cameras 30 partially overlap with each other, and since each video camera 30 has a difference in the brightness of shooting, the video captured by each video camera 30 is simply displayed. Simply connecting them results in an unnatural image and cannot produce a beautiful panoramic image. For this reason, in the process of generating a panoramic image, a process of deleting an overlapping portion is performed.

First, the computer 40 performs a process of reading eight images as shown in FIG. 11 from the storage unit for each frame number (step S11).
Then, as shown in FIG. 12, a non-overlapping image portion of the images obtained by the respective video cameras 30 is subjected to mask processing (step S12). The area to be masked is set to have a width of a predetermined number of pixels from the center of each image.
Then, as shown in FIG. 13, in addition to the masked portion (mask portion), a process of extracting image regions to be used by deleting image portions at both ends while leaving several pixels on both left and right sides of the mask portion. This is performed (step S13). In this example, a process of deleting an area of 125 pixels each from both ends of the image by the image area extraction processing is performed.

Next, a process of adjusting the brightness of the image of each video camera 30 is performed (step S14).
Therefore, as shown in FIG. 14, histograms of both ends of the extracted image overlapping with the adjacent image are extracted. As an example, histograms of overlapping edges of the camera 8 and the camera 1 are shown in FIGS.
Then, a process of averaging the histograms extracted from each image is performed. FIG. 14C shows the averaged histogram. Then, a process of resampling the averaged histogram into the original image is performed, and the brightness of the extracted image is adjusted.
When the brightness is adjusted, a process of connecting the eight images in the horizontal direction is performed (step S15). Thereby, a panoramic image Pn in all directions of 360 ° as shown in FIG. 15 is generated. Note that the panoramic image Pn of this example has an image in a viewing range of 180 degrees to the left and right, with the traveling direction (center direction of the camera 1) being 0 degree.
Then, the joined panoramic image Pn is stored with a frame number attached (step S16).
The brightness of the panoramic image Pn obtained in this way is adjusted at the joint between the adjacent image and the image, so that the user does not feel uncomfortable.
The computer 40 performs the processing of steps S11 to S15 for all frame numbers, generates a panoramic image Pn, and stores it in the storage unit.

A walk-through animation is generated from the traveling locus data, the panoramic image Pn, and the like generated as described above.
The walk-through animation is generated by cutting and synthesizing a viewpoint of the walk-through animation, that is, a panoramic image in the line of sight from the viewpoint of the user moving with time, as described in the following video generation processing.
That is, the coordinates of the viewpoint and the viewing angle of the user are input to the computer 40, the panorama image Pn of the frame number corresponding to the viewpoint coordinates and the viewing angle at each time is selected, and the images are synthesized.
For example, when there is a viewpoint on the traveling locus, the scan line number Ln and the frame number are specified by the coordinates of the viewpoint, and the panoramic image Pn of the frame number is selected. Then, an image in the line-of-sight direction from the viewpoint, that is, an image portion in a range determined by the viewing angle from the viewpoint is cut out from the panoramic image Pn, and a display image is generated. The display image generated in this way can be directly displayed on the display unit of the computer 40. In addition, the generated display image can be stored in the storage unit.
Processing when the viewpoint is outside the traveling locus will be described later.

(Intersection right / left turn video generation processing)
Next, generation processing of an intersection right / left turn image will be described.
In this processing, as shown in FIG. 16, when the video generation system S goes straight ahead at the intersection, the video (panorama image) obtained by the video camera 30 and the video obtained by going straight in the horizontal direction (panorama image) From the image), the video when turning around the intersection is synthesized.
At this time, the locus when moving in the vertical direction is the running locus T1, the locus when moving in the horizontal direction is the running locus T2, and the running locus T1 and T2 intersect within the intersection as shown in FIG. .

Hereinafter, a description will be given of an example of a process of generating an image when a left turn is made at an intersection where two roads are orthogonal to each other.
FIG. 17 shows a procedure for generating an intersection right / left-turn image.
It is assumed that the automobile moves on the traveling locus T1, turns left at an intersection, and further moves on the traveling locus T2.
The computer 40 reads the panorama image Pn and the traveling locus data on the traveling locus T1 and T2 (step S21). At this time, the travel trajectory data based on the travel trajectories T1 and T2 may be data of the same data in different time zones, or may be separate data but having the same origin position and direction.

Next, a process of determining a frame number at the time of intersection is performed (step S22).
As shown in FIG. 18D, the computer 40 superimposes the vertical traveling locus T1 and the horizontal traveling locus T2 obtained by the laser range scanner 20, and obtains an intersection P4 at the intersection. The intersection P4 is obtained by referring to the coordinate information on the traveling trajectories T1 and T2 of the traveling trajectory data.
Further, the computer 40 calculates the scan line number corresponding to the intersection P4 from the coordinates of the intersection P4 for the vertical travel and the horizontal travel, respectively. Since the scan line numbers and the frame numbers are associated with each other, the frame numbers of the panoramic image Pn in the vertical direction and in the horizontal direction are obtained based on the obtained scan line numbers.

Next, a process of synthesizing the panoramic image Pn is performed (Step S23).
As shown in FIG. 18A, when the car (viewpoint of the walk-through animation) is traveling straight ahead at the point P1 (on the trajectory T1) before the intersection, the line of sight is in the direction of the arrow, that is, in the straight ahead direction on the front. For this reason, the display image used for the walk-through animation at this time is a panorama image Pn from the traveling trajectory T1 in which the image portion of the camera 1 has been cut out.

  When the vehicle approaches the intersection as shown in FIG. 18B, an image portion of the panorama image Pn from the traveling locus T1 in which the viewpoint has been moved to the left by a predetermined amount is cut off so as to appear to be turning left. Used. Specifically, the panoramic image Pn is moved from left to right with respect to the display screen. In other words, the display screen is moved to the left. Then, an image synthesized from the images of the cameras 1 and 8 is gradually cut out from the image at the center of the camera 1 and displayed.

  As the car enters the intersection as shown in FIG. 18C, the viewpoint is further moved to the left. In other words, of the panoramic image Pn from the traveling trajectory T1, an image portion obtained by further moving the viewpoint to the left is cut out and used. That is, the panorama image Pn is further moved from left to right with respect to the display screen. In other words, the display screen is further moved to the left, and an image synthesized from the images of the cameras 8 and 7 is cut out and displayed. Thus, the image displayed as approaching the intersection P4 approaches the image centered on the camera 7.

  As shown in FIG. 18D, at the intersection P4, the image of the camera 7 (90 ° left) when traveling in the vertical direction (traveling trajectory T1) and the image of the camera 1 (front) when traveling in the horizontal direction (traveling trajectory T2). The images match. Therefore, when the intersection P4 is reached, the panoramic image Pn (image center is the camera 7) obtained by running in the vertical direction is replaced with a panoramic image Pn (image center is the camera 1) obtained by running in the horizontal direction. At this time, the frame numbers are switched. This makes it possible to switch between the image obtained during the vertical running and the image obtained during the horizontal running without giving the user a feeling of strangeness.

Finally, the generated images are joined together, and a video (walk-through animation) of turning left at the intersection is synthesized. This makes it possible to create right-turn and left-turn moving images when entering from four directions at an intersection by simply acquiring a straight-ahead image without actually turning at the intersection.
According to this method, when turning right or left, the screen does not suddenly switch from the screen in the straight traveling direction to the screen in the right or left turning direction, so that it is preferable that the user does not feel uncomfortable.
As described above, according to the video generation system S of the present example, it is possible to generate a walk-through animation including a natural right / left-turn video that does not cause a user to feel strange.

  Further, an image in which the viewpoint is moved by the set movement parameter can be generated by combining the arbitrary viewpoint video generation processing described below. The viewpoint movement and traveling direction images are synthesized by measuring the position of the moving object and the road width from the depth information and moving, enlarging, and reducing the panorama image based on the position information, as in the case of the intersection video. can do.

(Arbitrary viewpoint video generation processing)
Next, the arbitrary viewpoint video generation processing will be described. The computer 40 generates an arbitrary viewpoint video (a video outside the travel locus) that can be viewed from an arbitrary viewpoint other than the locus on which the automobile 10 travels, based on the generated panoramic image, the depth information from the laser range scanner 20, and the travel locus data. Perform processing.
FIG. 19 shows a conceptual diagram of arbitrary viewpoint video generation. As shown in FIG. 19, it is assumed that the image generation system S has moved along the traveling trajectory T from the origin L0 and the depth information by the laser range scanner 20 and the image data by the video camera 30 have been acquired. Then, the side surfaces W of the building are measured on both sides of the traveling locus T from the depth information. Here, let P be a new viewpoint other than on the traveling locus T.

The outline of the arbitrary viewpoint video generation processing is as follows. First, a range on the traveling trajectory T having a vector in the same direction as the line-of-sight vectors V1, V2,. A range of scan line numbers Ln having data is calculated.
Next, for the intersections L1, L2,... At each target scan line number Ln, the distance between the side surface W of the building and the viewpoint P, and the distance between the side surface W of the building and a point on the traveling locus T are determined.
Then, a process of enlarging / reducing the panorama image Pn on the traveling locus T at each frame number corresponding to each scan line number is performed according to the ratio of the two distances obtained.
Finally, the generated image is reconstructed, and a new image of the viewpoint P is generated.
FIG. 20 shows a procedure for generating an arbitrary viewpoint video.

  In the arbitrary viewpoint video generation processing, the coordinates (X, Y) of the viewpoint P and the viewing angles θ1 and θ2 that determine the viewing area of the target object are given to the computer 40 as data input processing, and the stored travel locus data and The panoramic image Pn is read (Step S31).

Next, a process of calculating an intersection between the extended line from the visual field region of the target object to the viewpoint P and the traveling locus T is performed (step S32).
At the viewpoint P (X, Y), when viewing the range from the viewing angles θ1 to θ2 with the traveling direction being 0 degrees, the range from the viewing angles θ1 to θ2 is the viewing area.
Here, as shown in FIG. 21, the intersection points L1 and L2 are the intersection points of a straight line drawn from the viewpoint P in a direction inclined by the viewing angles θ1 and θ2, respectively, and the traveling locus T. The coordinates of the origin L0, the intersection L1, and the intersection L2 are (0, 0), (X11, Y11), and (X12, Y12).
As shown in FIG. 21, an intersection point drawn from the viewpoint P (X, Y) toward the traveling locus T is defined as L3. The coordinates of the intersection L3 are (X2, Y). The computer 40 calculates X2 from the viewpoint P and the travel trajectory data, and further obtains a distance Lx (= X−X2) between the viewpoint P and the intersection L3.
The distance Ld1 between the intersection L3 and the intersection L1 is calculated by Ld1 = Lx / tan θ1, and the distance Ld2 between the intersection L3 and the intersection L2 is calculated by Ld2 = Lx / tan θ2.
Further, the Y coordinate (Y11) of the intersection L1 is calculated by Y11 = Y-Ld1, and the Y coordinate (Y12) of the intersection L2 is calculated by Y12 = Y-Ld2. Thus, the coordinate range of the intersection on the traveling locus T is calculated.
In this way, the intersections on the traveling locus T corresponding to the angles from the viewing angles θ1 to θ2 are obtained.

  Next, processing for specifying the range of the scan line number Ln of the data to be used is performed (step S33). Here, the process of comparing the Y coordinate on the traveling locus T corresponding to each angle with the Y coordinate corresponding to the scan line number Ln of the traveling locus data is performed, and the scan line number having the closest Y coordinate for each angle is performed. Ln is determined.

Next, in order to obtain an image from the viewpoint P, a process of obtaining an enlargement / reduction ratio for enlarging / reducing an image from a point on the travel locus T corresponding to each target scan line number Ln is performed (step S34). .
The enlargement / reduction ratio in each line-of-sight direction is determined by the distance Dp in each line-of-sight direction from the viewpoint P to the object (in this case, the side surface W of the building) and the distance Dl in each line-of-sight direction from a point on the traveling trajectory T to the object. Given by the ratio Z of

As shown in FIG. 22, the distance DIn from the coordinate point (Xln, Yln) corresponding to the target scan line number Ln to the target is obtained by actually acquiring the measured value by the laser range scanner 20 and storing it as depth information. I have. In the figure, the positions of the side surfaces W of the object in the θ1 direction and the θ2 direction corresponding to the intersections L1 and L2 are denoted by W1 and W2, respectively.
By reading depth information corresponding to the target angle θln to the target for the scan line number Ln in the range to be used, the distance Dln to the target can be obtained.
However, since the travel locus data is generated by rotating and connecting the respective range images for each scan line number Ln, it is necessary to consider the image rotation angle θrn in addition to the angles from the viewing angles θ1 to θ2. (See FIG. 22). The rotation angle θrn is a rotation angle corresponding to the scan line number Ln, and corresponds to a traveling direction at the time of measurement.
Thus, when the rotation angle θrn is considered with reference to the travel trajectory for the target scan line number Ln obtained for each angle from the viewing angles θ1 to θ2, the target angle θln is θln = θn−θrn.

  Therefore, of the depth information at each scan line number Ln, the distance information corresponding to the angle θln corresponds to the distance Dln in the line-of-sight direction to the target. The computer 40 reads out the data corresponding to the angle θln and stores it as the distance Dln.

Further, a process of calculating a distance Dpn in the line of sight from the viewpoint P to the target is performed.
The distance Dpn corresponds to a value obtained by subtracting the distance Dpln between the viewpoint P (X, Y) and the coordinate point (Xln, Yln) from the distance Dln calculated corresponding to the scan line number Ln. That is, Dpn = Dln-Dpln.

First, in the process of calculating the distance Dpln, the coordinate point (Xln, Yln) of each scan line number Ln is read from the travel locus data, and the coordinates of the viewpoint P (X, Y) and the coordinate point L (Xln, Yln) are read. The distance between the two points is calculated. That is, the distance Dpln is calculated by ^{Dpln = sqrt {(X-Xln} ) 2 + (Y-Yln) 2}.
Then, the distance Dpn is calculated by Dpn = Dln-Dpln.

The ratio Z of the distance Dpn between the viewpoint P (X, Y) in the line of sight to the target and the distance Dln from the coordinate point of the scan line number Ln to the target is calculated by Z = Dpn / Dln.
As a result, for each scan line number Ln, the enlargement / reduction ratio (ratio Z) of the image when the image of the new viewpoint P is generated from the original image is obtained and stored.

Next, a process of determining a region of an image to be used is performed (step S35).
First, since the scan line number Ln and the frame number of the panoramic image Pn are synchronized, the computer 40 specifies the frame number of the panoramic image Pn to be used by determining the scan line number Ln. Further, the target angle θln is calculated corresponding to the scan line number Ln.
As shown in FIG. 23, assuming that the width of the panoramic image Pn is w and the angle of the center of the panoramic image Pn is 0, the target pixel line In is
In1 = (π-θln) × w / 2π ·· (pixel line start)
In2 = (π−θln) × w / 2π + w / 2π .. (end of pixel line)
Is represented by That is, the scan line number Ln is an image in which an image composed of pixel widths from In1 to In2 of the panoramic image Pn is used.

Next, as image reconstruction processing, image enlargement / reduction processing is performed (step S36).
An image is reconstructed using the distance ratio Z between the distance Dpn and the distance Dln and the pixel line In.
As shown in FIG. 24, by multiplying each pixel line In by the distance ratio Z, the image is scaled up and the pixel line Ipn viewed from the viewpoint P is generated for each angle. The pixel line Ipn corresponds to a partial image outside the traveling locus.

Further, an image arrangement process is performed as an image reconstruction process (step S37).
The size of the pixel line Ipn generated in this manner differs for each pixel line according to the ratio Z. Therefore, as shown in FIG. 25, when the image is enlarged (when the ratio Z is 1 or more), the vertical length of the original image is used as a reference, and the portion protruding from the vertical length of the original image is deleted. Is done. In FIG. 25, some pixel lines Ipn are discretely displayed for easy understanding.
On the other hand, when the image is reduced (when the ratio Z is less than 1), a pixel line having the smallest vertical length is searched from all the pixel lines, and the vertical length of this pixel line is used as a reference. The part that goes out of the way is deleted.
Then, the center pixel of each line is calculated based on the vertical length, and the image of each line is arranged so that the center pixel is aligned with the vertical center of the image. As a result, an image (video outside the travel locus) viewed from the viewpoint P is generated.
As described above, according to the video generation system S of the present example, it is possible to generate a walk-through animation from an arbitrary viewpoint other than the traveling locus T.

As described above, according to the image generation system S of the present example, accurate travel trajectory data and position information are calculated based on the depth information obtained from the laser range scanner 20, and the image obtained from the video camera 30 is calculated. It is possible to generate a panoramic image Pn covering 360 degrees around the image based on the data.
Then, based on these data, it is possible to generate a realistic walk-through animation by right / left turn video generation processing or arbitrary viewpoint video generation processing.

Next, a specific example of the traveling image generated by the system of the present example will be described. First, a specific example of a left-turn image at an intersection is shown.
The intersection targeted for the image is an intersection where a one-way one-way (two-way round-trip) road extending from east to west and north to south intersects at about 90 °.
First, with respect to each road extending in the east, west, and north and south directions, the image generation system S acquires images (straight-moving images in a total of four directions) of going straight through the intersection from both directions, and also acquires depth information.

  Based on the data of these images and the like, an image was synthesized in which the user goes south on a north-south road, turns left at an intersection, and heads east on an east-west road. The images used at this time are a straight-ahead image that goes from north to south on a north-south road, and a straight-ahead image that goes from west to east on an east-west road.

FIG. 26 shows a traveling locus image acquired by the laser range scanner 20. From the traveling trajectory data, an intersection position when the vehicle goes straight from north to south and a straight line from west to east, and a frame number of a video at the time of intersection are obtained.
Here, the frame number of the video at the intersection position is the 129th frame from the start frame in the straight ahead video from north to south, and the 98th frame from the start frame in the straight forward video from west to east.
In addition, a panoramic image is generated for each frame based on the straight-moving image from north to south and the straight-moving image from west to east.

Furthermore, a left-turn image was synthesized from these data. FIG. 27 shows the generated left-turn image displayed every four frames.
Of these, the image up to the frame 76 is the straight image going straight from north to south. The frames from frame 77 to frame 128, which is one frame before the intersection, are synthesized by shifting the viewpoint of the straight-ahead image from north to south. Further, a frame 98 in the figure is a frame that is switched from a straight moving image from north to south to a straight moving image from west to east. Thereafter, a frame image of a straight-line video from west to east is used.

  In addition, the viewpoint is gradually moved to the left so as to pass through the locus inside (left side) of the locus from which the video was actually acquired near the intersection, thereby generating a left-turn image. FIG. 28 shows movement parameter values indicating the amount of movement of the viewpoint.

Next, a specific example of a right-turn image at an intersection is shown.
The target intersection is the same as the intersection in the left turn image. At this intersection, using the straight-moving video from north to south and the straight-moving video from east to west, a video was made to go south on the north-south road, turn right at the intersection, and go west on the east-west road.

FIG. 29 shows a running locus image acquired by the laser range scanner 20. Based on the traveling locus image, the frame number of the video when the vehicle crosses the intersection position when traveling straight from north to south and when traveling straight from east to west is calculated.
Here, the frame number at the intersection position is the 139th frame from the start frame in the straight forward video from north to south, and the 119th frame from the start frame in the straight forward video from west to east.
FIG. 30 shows a right-turn image generated from the acquired data. In FIG. 30, the generated right-turn image is displayed every four frames.
Until the intersection, a straight-ahead image from north to south is used, and after the intersection, a straight-ahead image from west to east is used. The image up to the frame 84 is the straight image from north to south. The frames from frame 85 to frame 138, which is one frame before the intersection, are synthesized by gradually shifting the viewpoint of the straight-ahead image from north to south to the right. Then, at the time of intersection, the frame is switched to a frame 119 of a straight-ahead image from west to east. Thereafter, a frame image of a straight-line video from west to east is used.

  In addition, the viewpoint was gradually moved to the right side so as to pass through a traveling locus inside (right side) of the traveling locus where the video was actually acquired near the intersection, and a right-turn image was generated. FIG. 31 shows the values of the movement parameters at that time.

As described above, by calculating the intersection frame number based on the panorama image Pn and the travel locus data and synthesizing the panorama image Pn, it is possible to automatically synthesize the right / left turn video at the intersection.
Conventionally, in order to acquire a realistic right / left turn image and straight-ahead image, it was necessary to actually drive 12 times (8 right / left turns and 4 straight runs) per one intersection. However, according to the present invention, it is possible to acquire a right / left turn image and a straight image in any direction only by a total of four straight runs per intersection. Therefore, according to the present invention, a right / left turn image and a straight-ahead image can be acquired with one-third of the labor, so that no labor is required and the cost can be reduced.

Further, it is possible to obtain a more realistic right / left-turn image at the intersection based on the movement parameter.
As for the movement parameters, the right and left turns at the beginning of the turn are almost the same. However, in the case of left-hand traffic, the turning parameter is set to be larger when turning right after the middle stage than when turning left. Therefore, the movement parameter is set to be larger when turning right than when turning left.
By using the movement parameters used when synthesizing the right / left-turn image, it is also possible to adapt when synthesizing the right / left-turn image at another intersection. In particular, in the case of a left turn, the turning route at the intersection is almost the same irrespective of the difference in the road width and the number of lanes, so that the movement parameters can be standardized. In addition, in the case of a right turn, or depending on the intersection angle of a road at an intersection or the number of lanes, the moving parameters are appropriately set, so that a right / left turn image corresponding to the intersection can be automatically generated. it can.

Next, a specific example of an arbitrary viewpoint video will be described.
First, on a straight road, a straight-moving image was obtained by the image generation system S, and depth information was obtained.
FIG. 32 shows a traveling locus image. This is generated using data of 61 scans with scan line numbers 720 to 780 among the acquired data. In the figure, the running locus is represented by T.
FIG. 33 shows the coordinates of a coordinate point on the traveling locus T and the rotation angle θrn at each scan line number. Further, a panoramic image Pn was generated based on the obtained video.

An image viewed from a new viewpoint is generated using such traveling locus data and the panoramic image Pn.
Here, as the coordinates of the new viewpoint P, P11 (−3.−3) in which the X coordinate is moved left and right around the viewpoint P15 (0, 25) on the traveling trajectory T in the range from −3.5 to +3.5. 5,25), P12 (-3,25), P13 (-2,25), P14 (-1,25), P16 (1,25), P17 (2,25), P18 (3,25), Nine points of P19 (3.5, 25) were set.
Further, as shown in FIG. 32, in each viewpoint, the viewing angle θ1 is fixed at 75 ° and the viewing angle θ2 is fixed at 30 ° in order to see a difference between images from each viewpoint, and the angle range of 45 ° is set. Image generated.
The viewpoints P11, P12, P13, and P14 are points moved by 3.5, 3, 2, 1 to the left in the X coordinate direction from the traveling locus T (viewpoint P15). The viewpoints P16, P17, P18, and P19 are points moved by 1, 2, 3, 3.5 to the right in the X coordinate direction with respect to the traveling locus T (viewpoint P15).

FIG. 34 to FIG. 37 show images from each viewpoint generated under such settings. These figures are shown diagrammatically for ease of viewing.
FIG. 34A shows an image viewed from the viewpoint P15 (0, 25). This image is a panorama image at the viewpoint P15 on the traveling trajectory T with the viewing angles θ1 = 75 ° and θ2 = 30 °, and images in these angle ranges are cut out.
Also, images viewed from the viewpoints P16 and P17 are shown in FIGS. Images viewed from viewpoints P18 and P19 are shown in FIGS. 35 (A) and (B), respectively. Images viewed from viewpoints P14 and P13 are shown in FIGS. 36 (A) and (B), respectively. Images viewed from viewpoints P12 and P11 are shown in FIGS. 37 (A) and (B), respectively.

Comparing the image from the viewpoint P15 on the traveling trajectory T with the image from each viewpoint, the moving range from the viewpoint P16 to the viewpoint P19 to the right side shows that the visible range of the object becomes narrower, and the wall of the building on the right side Appears to be approaching.
In addition, the movement from the viewpoint P16 to the viewpoint P19 does not linearly move in the direction of the viewing angle, but moves in a direction at 90 ° to the traveling direction. Therefore, the image from each viewpoint is not enlarged toward the center of the image as the viewpoint moves to the right, but the image is enlarged and the field of view is narrowed, and the field of view is shifted rightward toward the building. It moves in parallel to.

  On the other hand, when moving from the viewpoint P15 on the traveling locus T to the left side, as it moves to the viewpoint P11, the visual field range is widened, and it looks as if it is moving away from the wall of the building. That is, from the viewpoint P13 to the viewpoint P19, the tree in front of the building is not included in the visual field range and is not reflected in the image. However, at the viewpoint P11 and the viewpoint P12, the field of view is widened, and a tree is included in the field of view, which is reflected in the image.

Using the panorama image Pn generated based on the image data photographed by the video camera 30 and the traveling locus data generated based on the depth information acquired by the laser range scanner 20, the traveling locus is An image viewed from any viewpoint other than T can be synthesized.
In the image generation system S of the present invention, accurate position information and a moving object trajectory can be obtained by using the laser range scanner 20. That is, in the image generation system S, the configuration can be simplified because other position measuring devices such as a GPS and a gyro are not used.
Then, based on the obtained position information, it is possible to generate a panoramic image Pn from each viewpoint, a movement of the viewpoint, a right / left turn image at an intersection, a moving direction change image, and the like.

  Then, by setting the coordinates of the traveling route other than the traveling trajectory T and the viewing angle and generating an image, a natural walk-through animation that can be freely moved and has a high degree of reality can be generated.

FIG. 3 is an explanatory diagram of a video generation system. FIG. 1 is an explanatory diagram illustrating a configuration of a video generation system. FIG. 3 is an explanatory diagram illustrating an arrangement of a video camera and a laser range scanner. FIG. 4 is an explanatory diagram for calculating a video overlap distance by a video camera. It is a flowchart of a traveling locus data acquisition process. FIG. 4 is an explanatory diagram illustrating an example of a range image. FIG. 9 is an explanatory diagram illustrating data processing of a range image. It is explanatory drawing which shows an example of a traveling locus image. It is explanatory drawing which shows an example of travel locus data. It is a flowchart of a panorama image generation process. FIG. 3 is an explanatory diagram illustrating an example of images from three adjacent video cameras. FIG. 9 is an explanatory diagram illustrating a process of deleting an overlapping portion of a video camera image. FIG. 3 is an explanatory diagram illustrating an example of a video camera image from which an overlapping portion has been deleted. It is a histogram of the overlap part of the edge part of an adjacent video camera image. FIG. 9 is an explanatory diagram illustrating a process of generating a panoramic image from images obtained by eight video cameras. It is explanatory drawing of the process which produces | generates the right-left turn image in an intersection. It is a flowchart of the process which produces | generates the right-left turn image in an intersection. It is explanatory drawing of the process which produces | generates a left turn image in an intersection. FIG. 9 is an explanatory diagram of a process of generating an arbitrary viewpoint video. 5 is a flowchart of a process for generating an arbitrary viewpoint video. FIG. 9 is an explanatory diagram of a process of generating an arbitrary viewpoint video. FIG. 9 is an explanatory diagram of a process of generating an arbitrary viewpoint video. It is explanatory drawing of a panorama image and a pixel line. FIG. 9 is an explanatory diagram illustrating a pixel line generation process viewed from a viewpoint P. FIG. 9 is an explanatory diagram illustrating an arrangement of generated pixel lines. It is explanatory drawing which shows the driving locus | trajectory image used for left-turn image production | generation. It is explanatory drawing which shows an example of a left turn image. FIG. 9 is an explanatory diagram showing movement parameters for each frame. It is explanatory drawing which shows the driving locus | trajectory image used for right-turn image production | generation. It is explanatory drawing which shows an example of a right turn image. FIG. 9 is an explanatory diagram showing movement parameters for each frame. It is explanatory drawing which shows the driving | running locus image used for arbitrary viewpoint image | video production | generation. It is explanatory drawing which shows the driving | running locus data used for arbitrary viewpoint image | video production | generation. It is explanatory drawing which shows the position of an arbitrary viewpoint, and the image from an arbitrary viewpoint. It is explanatory drawing which shows the position of an arbitrary viewpoint, and the image from an arbitrary viewpoint. It is explanatory drawing which shows the position of an arbitrary viewpoint, and the image from an arbitrary viewpoint. It is explanatory drawing which shows the position of an arbitrary viewpoint, and the image from an arbitrary viewpoint.

Explanation of reference numerals

DESCRIPTION OF SYMBOLS 10 Car 11 System carrier 12 Angle stand 20 Laser range scanner 30 Video camera 31 Base 40 Computer S Image generation system

Claims (8)

A distance between a moving body, a camera disposed on the moving body, which captures images in a circumferential direction of the moving body and outputs image data, and a camera disposed on the moving body and representing objects around the moving body. A horizontal laser that outputs depth information, and a data processing device that processes the image data and the depth information,
The data processing device includes:
The position of a surrounding object with respect to the moving body is specified at predetermined time intervals based on the depth information, and the running trajectory of the moving body with respect to the specified position of the surrounding object is specified. A process of calculating position information indicating a position of the moving body and traveling direction information representing a traveling direction on the moving body;
A process of generating a panoramic image that displays the entire circumferential direction as viewed from the traveling locus every predetermined time based on the image data;
A video generation system that performs a process of generating a walk-through animation based on the depth information, the position information, the traveling direction information, and the panoramic image. In the processing for generating the walk-through animation, the data processing device may convert one traveling locus into another traveling locus based on the depth information, the position information, the traveling direction information, and the panoramic image in the traveling locus intersecting each other. Perform an intersection right / left turn image generation process that generates an image that changes the traveling direction to the trajectory,
In the intersection right / left turn image generation processing, as the viewpoint of the walk-through animation on one traveling locus approaches the intersection of the traveling locus, a line of sight from a traveling direction to a circumferential direction in a panoramic image on the one traveling locus is changed. The video generation system according to claim 1, wherein the shifted portion is used as a display image of the walk-through animation.   The video generation system according to claim 1, wherein the data processing device performs an arbitrary viewpoint video generation process of generating a video at a viewpoint outside the travel locus in the process of generating the walk-through animation. In the process of generating the walk-through animation, the data processing device performs an arbitrary viewpoint video generation process of generating a video within a predetermined viewing angle range from a viewpoint outside the travel locus,
In the arbitrary viewpoint video generation processing, specify each intersection where a straight line drawn from the viewpoint in each angle direction within the viewing angle range and the traveling locus intersects,
Of the panoramic image at each intersection, cut out an image portion including the viewpoint,
A distance from each intersection to a surrounding object in each angular direction and a distance from the viewpoint to a surrounding object in each angular direction are calculated based on the depth information, and a ratio of these distances is calculated. And
The image portion is enlarged or reduced by the ratio to generate a partial image outside the traveling locus,
The video generation system according to claim 1, wherein the video image outside the travel locus is generated by connecting the partial images outside the travel locus. A step of acquiring image data in all directions of the shooting position with an omnidirectional camera,
A step of obtaining depth information between the horizontal laser and the surrounding object with a horizontal laser that measures the distance to the surrounding object,
The position of a surrounding object with respect to the moving body is specified at predetermined time intervals based on the depth information, and the running trajectory of the moving body with respect to the specified position of the surrounding object is specified. Calculating position information representing the position of the moving body and traveling direction information representing the traveling direction,
A step of generating a panoramic image that displays the entire circumferential direction as viewed from above the traveling locus for each predetermined time based on the image data;
Generating a walk-through animation based on the depth information, the position information, the traveling direction information, and the panoramic image.   In the step of generating the walk-through animation, the viewpoint of the walk-through animation in one traveling locus is determined based on the depth information, the position information, the traveling direction information, and the panoramic image in the traveling locus that crosses each other. The method according to claim 1, wherein a part of the panorama image in the one traveling locus whose gaze is shifted in a circumferential direction from a traveling direction is displayed as the display image of the walk-through animation as approaching a crossing point of the locus. 6. The video generation method according to 5.   The video generation method according to claim 5, wherein in the step of generating the walk-through animation, a process of generating a video at a viewpoint outside the travel locus is performed. In the step of generating the walk-through animation,
Identify each intersection where a straight line drawn in each angle direction within the viewing angle range from the viewpoint outside the travel locus and the travel locus intersects,
Of the panoramic image at each intersection, cut out an image portion including the viewpoint,
A distance from each intersection to a surrounding object in each angular direction and a distance from the viewpoint to a surrounding object in each angular direction are calculated based on the depth information, and a ratio of these distances is calculated. And
The image portion is enlarged or reduced by the ratio to generate a partial image outside the traveling locus,
The image generation method according to claim 5, wherein the image outside the travel path is generated by connecting the partial images outside the travel path.

Images