JP2010072922A - Image generation device, method, and program - Google Patents

Abstract

An image generation apparatus is provided that generates an image viewed from a specified viewpoint, from which unnecessary subjects are removed, based on ray space data of a local region.
An image generation apparatus that generates an image based on ray space data of each local region obtained by dividing a space, and is a first local region included in the image to be generated based on a specified viewpoint and angle of view. The local area is determined, and thereby the drawing means generates an output image, and creates correspondence information indicating the correspondence between the pixel of the output image and the local area of the light space data used for generating the pixel. . Thereafter, when a pixel of the output image is designated by the removal position input means, a second local area corresponding to the designated pixel of the output image is determined, and the drawing means determines the second local area from the first local area. Redraw the output image excluding the region.
[Selection] Figure 1

Description

  The present invention relates to a free viewpoint image generation technique, and more particularly, to an image generation technique for generating a free viewpoint image by dividing a space into a plurality of local areas and constructing a ray space for each local area.

  Various proposals have been made on a free viewpoint image generation technique for reproducing an image at an arbitrary position from moving images taken by a plurality of cameras. (For example, see Patent Document 1, Non-Patent Document 1, and Non-Patent Document 2,).

  The above-described method in the prior art is called image-based rendering. A light space is constructed from images taken by a plurality of cameras, and an image viewed from an arbitrary position is interpolated based on the light space. It is generated by processing. Here, constructing a ray space means that rays passing through a certain surface are represented using appropriate coordinate axes, and rays represented as coordinates in this coordinate space are associated with pixel values of an image generated by the rays. To tell. Note that it is not realistic to prepare pixel value data in advance for all coordinates in the coordinate space. Normally, the pixel value is set only for an appropriate number of coordinates determined according to the application or the like. Prepare the data.

  Here, Patent Document 1 describes a technique for dividing a region where a subject exists into a plurality of local regions and constructing a light space for each local region to generate a free viewpoint image. According to Patent Literature 1, first, light space data of each local region is created based on images taken by a plurality of cameras. Here, the ray space data of the local region is data for specifying a ray passing through the boundary surface of the local region and a pixel value corresponding to the ray. For example, in the example shown in FIG. 2, the space is divided into local areas sa1 to sa10, and the image generation apparatus holds the light space data of each local area represented by a dotted line. In FIG. 2, reference numerals 10 to 13 denote subjects.

  When the viewpoint position is designated, the image generation apparatus described in Patent Document 1 first generates an image based on the ray space data of the local region located at the backmost position when viewed from this position, and then one before The free viewpoint image is generated by repeatedly overwriting the image based on the ray space data of the local region in FIG. For example, if the designated viewpoint is the position of reference numeral 20 and the range of the image to be generated is between the lines 21 and 22, the image generating apparatus according to Patent Document 1 firstly local regions sa1 and sa2 in the range of the image. And sa3 based on the ray space data, and then overwriting the previously generated image with the ray space data of the local regions sa5 and sa6, and finally overwriting with the ray space data of the local region sa9. Thus, an image is generated.

JP 2008-15756 A Takeshi Naemura, et al. "Ray-Based Creation of Photo-Realistic Virtual World", VSMM97, pp. 59-62, September 1997 Michael Drose, et al. “Ray-Space Interpolation based on Filtering in Disparity Domain”, Proc. of 3D Image Conference 2004, pp. 29-30, 2004

  Here, in the image viewed from the viewpoint 20, a part of the subject 11 is not visible by the subject 10. When the entire subject 11 is to be displayed on the image, the viewpoint position needs to be set between the subject 10 and the subject 11. However, the obtained image is very close to the subject 11, and when the angle of view is fixed and cannot be changed, the entire subject 11 is 1 depending on the distance between the subject 11 and the subject 10 and the size of the subject 11. Cannot be displayed on one image. In addition, when the angle of view can be set, it is possible to display the entire subject 11 in a single image by widening the image, that is, by increasing the angles of the lines 21 and 22. This will limit the angle of view.

  On the other hand, there is a case where it is desired to generate and display an image of an arbitrary angle of view viewed from an arbitrary viewpoint position and an obstructed subject removed. Therefore, the present invention provides an image generation apparatus, method, and program for generating an image viewed from a specified viewpoint, from which unnecessary subjects are removed, based on the ray space data of the local region. Objective.

According to the image generation apparatus of the present invention,
An image generation device that divides a space into a plurality of local regions and generates an image based on ray space data of each local region, and includes local regions included in an output image to be generated based on a specified viewpoint and angle of view. A local area determination means for determining a certain first local area, a specified viewpoint and angle of view, and an output image based on the first local area, and a pixel of the output image and generation of the pixel A drawing unit that creates correspondence information indicating a correspondence relationship with the local region of the used light space data, a removal position input unit for designating a pixel of the output image, and an excluded local region determination unit. When the pixel of the output image is specified by the removal position input means, the excluded local area determination means determines a second local area that is a local area corresponding to the specified pixel of the output image based on the correspondence information. Drawing Means, characterized in that the first local region by excluding the second local region redrawing output image.

According to another embodiment of the image generation apparatus of the present invention,
The drawing means overwrites the image drawn based on the local area farthest from the specified viewpoint among the first local areas based on the ray space data of the previous local area, It is also preferable to generate an output image by sequentially repeating a local region that is closest to the designated viewpoint.

According to another embodiment of the image generation apparatus of the present invention,
The model data indicating the correspondence between the subject existing in the space and the ray space data of each local region is held, and the excluded local region determination unit corresponds to the pixel of the output image specified by the removal position input unit. It is also preferable to determine the subject corresponding to the pixel of the designated output image based on the ray space data of the local region to be used, and to set all local regions where the subject exists as the second local region.

According to the program of the present invention,
A computer is caused to function as the image generation apparatus.

According to the image generation method of the present invention,
An image generation method in an image generation apparatus that divides a space into a plurality of local regions and generates an image based on ray space data of each local region, and is included in an output image to be generated based on a specified viewpoint and angle of view A first local region that is a local region to be generated, a step of generating an output image based on the specified viewpoint and angle of view, and the first local region, a pixel of the generated output image, A step of creating correspondence information indicating a correspondence relationship with the local region of the light space data used for pixel generation, a step of receiving designation of a pixel of the output image, and the designated output image based on the correspondence information Determining a second local region that is a local region corresponding to the pixel of the first pixel, and redrawing the output image by excluding the second local region from the first local region. And wherein the Rukoto.

  It is possible to generate and display an image of an arbitrary angle of view viewed from an arbitrary viewpoint position and an obstructing subject removed.

  The best mode for carrying out the present invention will be described in detail below with reference to the drawings. FIG. 1 is a simplified functional block diagram of an image generating apparatus according to the present invention. According to FIG. 1, the image generation apparatus includes a viewpoint / view angle input unit 1, a local region selection unit 2, a drawing processing unit 3, an image display unit 4, a removal position input unit 5, and a removal local region determination unit. 6 and a database unit 7.

  The database unit 7 is a database that stores ray space data and model data of each local region. As described in Patent Document 1, the ray space data includes rays that pass through the boundary surface of the local region, and the rays. Is data for specifying a pixel value corresponding to. Model data is generated based on multiple images taken to construct a ray space, and the area where the subject exists is identified from the continuity of the positions of the pixels of each shot image in the real space. The data shows the correspondence between the area where the subject exists and the light space data of each local area. The model data can be calculated from, for example, the camera parameters of each camera and the parallax of the corresponding pixel of the photographed image when photographing a subject with a plurality of cameras, as described in Patent Document 1. The operation of the image generation apparatus according to the present invention will be described below.

  The viewpoint / view angle input unit 1 is a user interface for allowing the user to input the viewpoint and the view angle of the image to be displayed, and outputs the viewpoint and the view angle designated by the user to the local region selection unit 2. The viewpoint is a value specified by the position and direction. The local area selection unit 2 selects a local area necessary for drawing an image based on the viewpoint and the angle of view specified by the user. For example, in the state shown in FIG. 2, when the angle of view corresponding to the viewpoint 20 and the range between the straight lines 21 and 22 is designated, the local region selection unit 2 selects the local regions sa1, sa2, sa3, sa5, sa6 and Sa9 is determined as a necessary local area, and the local area, the viewpoint and the angle of view designated by the user are output to the drawing processing unit 3.

  The drawing processing unit 3 generates an image based on the viewpoint and angle of view specified by the user and the light space data of the local region selected by the local region selecting unit 2 among the data stored in the database unit 7. More specifically, from the ray space data of the local regions sa1, sa2, and sa3 that are the deepest local regions when viewed from the viewpoint, a light ray directed to the viewpoint 20 is selected, and an image is generated based on the selected light ray. Next, from the ray space data of the local regions sa5 and sa6 that are one step ahead, the light beam that is directed to the viewpoint 20 is selected and overwritten with the previously generated image, and finally, the light ray of the local region sa9 that is one step ahead. Similarly, an image is generated by overwriting with spatial data.

  In the present invention, overwriting means that when a pixel value based on ray space data of a certain local region is different from a pixel value of an already generated image, the pixel value is changed by the ray space data of this certain local region. Say that. FIG. 3 is a diagram for explaining the meaning of overwriting used in the present invention. Note that FIG. 3 is obtained by deleting the display of the local region of FIG. 2 in order to make the drawing easier to see. For example, the light ray 30 from the subject 13 toward the viewpoint 20 has the same value in all the light space data of the local areas sa3, sa6, and sa9. In this case, the drawing processing unit 3 does not perform overwriting, and this light ray 30 The pixel by is recognized as being generated by the light space data of the local region sa3.

  On the other hand, the light beam traveling from the subject 11 to the viewpoint 20 is blocked by the subject 10, and instead, the light beam from the subject 10 reaches the viewpoint 20. The drawing processing unit 3 uses the value of the light beam emitted from the subject 11 as the pixel value of the pixel based on the light beam 31 when the local region sa5 is processed, but when processing the local region sa9, Recognizing that the pixel value of the ray 31 in the ray space data of sa5 and the pixel value of the ray 31 in the ray space data of the local region sa9 are different, and accordingly, the pixel corresponding to the ray 31 by the ray space data of the local region sa9. Overwrite the pixel value of. Thereby, the drawing processing unit 3 recognizes that the pixel by the light ray 31 is generated by the light space data of the local region sa9.

  As described above, the drawing processing unit 3 generates correspondence information indicating which local region is generated by the ray space data of the pixel value of each pixel of the generated image. The correspondence information is output to the excluded local region determination unit 6. The generated image is output to the image display unit 4 for display on a display, for example.

  It is assumed that the user wants to display an image from which the subject 10 in FIG. 2 is removed by viewing the image displayed on the display by the image display unit 4. In this case, the user designates the subject 10 to be removed using the removal position input unit 5. More specifically, the removal position input unit 5 is, for example, a pointing device, and the user moves the cursor displayed on the display to a position where the subject 10 is displayed to remove the subject 10. This is input to the removal position input unit 5, and the removal position input unit 5 outputs information specifying the pixel where the cursor is located at this time to the excluded local region determination unit 6.

  The excluded local region determination unit 6 determines which local region the pixel output from the removal position input unit 5 belongs to based on the correspondence information. Subsequently, if the determined local region is the local region located farthest from the viewpoint, the drawing processing unit 3 notifies the drawing processing unit 3 as an error because it cannot be removed, and the drawing processing unit 3 reports the error to the image display unit. 4 is displayed on the display.

  On the other hand, when the determined local area is not the local area located farthest from the viewpoint, but corresponds to some subject, the excluded local area determination unit 6 sends the determined local area to the drawing processing unit 3. Notice. When the drawing processing unit 3 receives the notification of the local region from the excluded local region determination unit 6, the drawing processing unit 3 excludes the received local region from the local region selected by the local region selection unit 2 and redraws the image. Do. That is, for example, when the user designates the pixel of the subject 10, the excluded local region determination unit 6 notifies the drawing processing unit 3 of the local region sa9, and the drawing processing unit 3 thereby performs the first drawing. The local region sa9 is excluded from the local regions sa1, sa2, sa3, sa5, sa6, and sa9 used in the above, and an image from the viewpoint 20 is generated again. Thereby, an image from which the subject 10 is removed is generated.

  In addition, the rendering processing unit 3 outputs the newly generated image to the image display unit 4 and displays the correspondence information based on the newly generated image, and the newly generated image and the updated correspondence relationship. Information is output to the excluded local region determination unit 6. As a result, when a plurality of subjects overlap each other, it is possible to remove the obstructing subjects in order.

  In the configuration described above, only the light space data of the local area corresponding to the position specified by the cursor is excluded. For example, in the case of a subject straddling a plurality of local areas, such as the subject 11, the designation is made by the cursor. Depending on the position, an image from which only a part of the subject has been removed is generated, and a plurality of removal designations must be made in order to delete one subject.

  When it is desired to remove a subject straddling a plurality of local areas with a single operation, the removed local area determination unit 6 is used to determine a local area from which model data is to be removed. In other words, based on the light beam corresponding to the designated pixel and the model data, it is determined in which local region the subject including the designated pixel exists, and all the local regions in which the subject exists are rendered. Output to part 3.

  The image generation apparatus according to the present invention can be realized by a program that causes a computer to function as the image generation apparatus. These computer programs can be stored in a computer-readable storage medium or distributed via a network. Furthermore, the present invention can be realized by a combination of hardware and software.

1 is a block diagram of an image generation apparatus according to the present invention. It is a figure explaining the image generation by this invention. It is a figure explaining overwriting of an image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Viewpoint / view angle input part 2 Local area | region selection part 3 Drawing process part 4 Image display part 5 Removal position input part 6 Removal local area determination part 7 Database part 10, 11, 12, 13 Subject 20 Viewpoint 21, 22 Line 30, 31 rays

Claims (5)

An image generation device that divides a space into a plurality of local regions and generates an image based on ray space data of each local region,
Local area selection means for selecting a first local area that is a local area included in the output image to be generated based on the specified viewpoint and angle of view;
Correspondence that generates an output image based on the specified viewpoint and angle of view, and the first local region, and indicates the correspondence between the pixel of the output image and the local region of the ray space data used for generating the pixel Drawing means for creating relationship information;
Removal position input means for designating pixels of the output image;
An excluded local region determination means;
With
When the pixel of the output image is specified by the removal position input means,
The excluded local region determination means determines a second local region that is a local region corresponding to the pixel of the output image specified based on the correspondence information,
The drawing means redraws the output image by excluding the second local area from the first local area,
Image generation device. The drawing means overwrites the image drawn based on the local area farthest from the specified viewpoint among the first local areas based on the ray space data of the previous local area, Generate an output image by iterating sequentially from the specified viewpoint to the local area that is at the forefront.
The image generation apparatus according to claim 1. Holding model data indicating the correspondence between the subject existing in the space and the light space data of each local region;
The excluded local region determining unit determines a subject corresponding to the designated pixel of the output image based on the light space data of the local region corresponding to the pixel of the output image designated by the removal position input unit, and the subject exists. All local regions to be the second local region,
The image generation apparatus according to claim 1.   The program which makes a computer function as an image generation apparatus of any one of Claim 1 to 3. An image generation method in an image generation apparatus that divides a space into a plurality of local regions and generates an image based on ray space data of each local region,
Selecting a first local region, which is a local region included in the output image to be generated, based on the specified viewpoint and angle of view;
Generating an output image based on the specified viewpoint and angle of view and the first local region;
Creating correspondence information indicating the correspondence between the pixel of the generated output image and the local region of the ray space data used to generate the pixel;
Receiving a pixel designation of the output image;
Determining a second local region that is a local region corresponding to a pixel of the designated output image based on correspondence information;
Redrawing the output image by excluding the second local region from the first local region;
An image generation method comprising:

Images