JP2016009245A - Intermediate viewpoint image generator - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an intermediate viewpoint generation device capable of generating an intermediate viewpoint image even in a scene where it is difficult to associate characteristic points between images.SOLUTION: An intermediate viewpoint image generation device includes: an image input unit to which a plurality of images to be processing objects are input; a characteristic point association unit for associating respective characteristic points of inputted two images with each other; a camera parameter calculation unit for calculating camera parameters of a camera having photographed the two images from the associated characteristic points; an area division unit for performing the division of the area for the inputted two images; a pixel association unit for performing the association of the pixels from the associated characteristic points included in the divided area; and an intermediate viewpoint image generation unit for generating an intermediate viewpoint image at a designated intermediate viewpoint position using the associated pixels.

Description

  The present invention relates to an intermediate viewpoint image generation device that generates an intermediate viewpoint image using a plurality of images having different viewpoints.

  Image Based Rendering is known as a method for generating an image from a viewpoint at an intermediate position of the viewpoints of the images (hereinafter referred to as an intermediate viewpoint image) based on images obtained by shooting a scene from a plurality of viewpoints. Yes. This technique has a feature that even if the three-dimensional structure of the target scene is unknown, it is possible to generate an arbitrary viewpoint image with high quality and high realism. By using this method, it is possible to generate an image from an arbitrary viewpoint that is not actually photographed (see Non-Patent Document 1, for example).

Tomonari Hatanaka, Akira Kubota, Yoshinori Hatori, "Generation of Intermediate Viewpoint Images from Stereo Images with Different Focus", IPSJ SIG Notes, 2008 (124), 23-28, 2008.

  However, in the intermediate viewpoint image generation method described in Patent Document 1, since the viewpoint interpolation method is used, it is difficult to associate the images for viewpoint interpolation. There is a problem that the feature points are limited, such as those that are easy to deal with.

  The present invention has been made in view of such circumstances, and provides an intermediate viewpoint image generation apparatus capable of generating an intermediate viewpoint image even in a scene where it is difficult to associate feature points between images. For the purpose.

  The present invention includes an image input unit that inputs a plurality of images to be processed, a feature point association unit that associates feature points of each of the two input images, and the associated feature points. A camera parameter calculation unit that calculates camera parameters of the camera that captured the two images, a region dividing unit that performs region division on the two input images, and a correspondence included in the divided regions A pixel association unit that associates pixels from the associated feature points, and an intermediate viewpoint image generation unit that generates an intermediate viewpoint image at a designated intermediate viewpoint position using the associated pixels. It is characterized by providing.

  In the present invention, the pixel association unit associates the divided areas from the associated feature points, and associates the pixels based on an aspect ratio of a rectangle including the associated areas. It is characterized by performing attachment.

  According to the present invention, it is possible to generate an intermediate viewpoint image even in a scene where it is difficult to associate feature points between images, and it is possible to widen the range of scenes in which the intermediate viewpoint image can be used. Is obtained.

It is a block diagram which shows the structure of one Embodiment of this invention. 3 is a flowchart showing a processing operation for generating an intermediate viewpoint image by the intermediate viewpoint image generating apparatus shown in FIG. 1. It is a figure which shows an example of the input image a and the image b. It is a figure which shows the image a and the image b after performing area division. It is explanatory drawing which shows the processing operation of step S5 shown in FIG. It is explanatory drawing which shows the processing operation of step S6 shown in FIG. It is a figure which shows an example of the intermediate | middle viewpoint image produced | generated.

  Hereinafter, an intermediate viewpoint image generating device according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes an image input unit for inputting at least two images taken from different viewpoints. Reference numeral 2 denotes an image processing unit that characterizes images by performing image processing on an input image. Reference numeral 3 denotes an image generation unit that generates and outputs an intermediate viewpoint image based on the output of the image processing unit 2. The intermediate viewpoint image generation device shown in FIG. 1 may be configured by a computer device.

  Next, a processing operation in which the intermediate viewpoint image generation apparatus shown in FIG. 1 generates an intermediate viewpoint image will be described with reference to FIG. FIG. 2 is a flowchart showing a processing operation in which the intermediate viewpoint image generation apparatus shown in FIG. 1 generates an intermediate viewpoint image.

  First, the image input unit 1 inputs at least two processing target images from the outside (step S1). The processing target image input here needs to have different viewpoints, and it is necessary that the subject for which an intermediate viewpoint image is to be generated be captured. The image input unit 1 holds the input processing target image therein.

  Here, a description will be given on the assumption that there are two input processing target images. It is assumed that the image input unit 1 receives and holds two images a and b (image a and image b are actually color images) shown in FIG. FIG. 3 is a diagram illustrating an example of the input image a and image b.

  Next, the image processing unit 2 reads out the processing target image held in the image input unit 1, extracts feature points from each of the two read images, and extracts the feature points based on the feature amounts of the feature points. Point correspondence is performed (step S2).

  For the correspondence between the feature points, a known method such as the method described in Reference 1 can be used. Since the method used in step S2 is a known method, detailed description thereof is omitted here.

  Reference 1 "David G. Lowe," Distinctive image features from scale-invariant keypoints, "International Journal of Computer Vision, 60, 2, pp. 91-110, 2004"

  Next, the image processing unit 2 calculates camera parameters of a camera that has captured two images from each of the associated feature points (step S3). In step S3, the following camera parameters are calculated. The camera parameters can be calculated using a known calculation method such as the method described in Reference 2. Since the calculation method used here can use a known calculation method, detailed description thereof is omitted here.

  Reference 2 “Noah Snavely, Steven M. Seitz, Richard Szeliski. Photo Tourism: Exploring image collections in 3D. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2006), 2006.”

The camera parameters are as follows. Camera parameters include external parameters (rotation, translation) and internal parameters (focal length, principal point).

  Here, rotation R is a rotation matrix for the camera in world coordinates, and translation t is a translation sequence for the camera.

The focal length f _x is the focal length of the x-axis direction in the xy coordinate mutually orthogonal to the optical axis with the origin of the camera optical axis, the focal length f _y is the focal length of the y-axis direction.

The principal point (c _x , c _y ) is the coordinates of the reference point for calculating the above-mentioned focal length (if a plurality of lenses mounted on the camera are regarded as one lens, its virtual lens The above-mentioned focal length is from the coordinates of the position where the light is present to the focal position.

The camera parameters are as _follows : I _a : internal parameter of image a, R _a : external parameter (rotation) of image a, and T _a : external parameter of image a. (Translation), I _b : internal parameter of image b, R _b : external parameter (rotation) of image b, T _b : external parameter (translation) of image b.

  Next, the image processing unit 2 performs region division processing on the two read processing target images (image a and image b) (step S4). This area division is divided into areas as shown in FIG. FIG. 4 is a diagram illustrating the image a and the image b after the area division. Each of these areas is an area where pixels having similar characteristics gather. Similar features include, for example, similar colors and similar brightness.

  This region can be divided by a known method such as the method described in Reference 3. Since a known method can be used as the method used here, detailed description thereof is omitted here.

  Reference 3 `` Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Susstrunk, SLIC Superpixels Compared to State-of-the-art Superpixel Methods, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, num. 11, p. 2274-2282, 2012. ''

  Next, the image processing unit 2 associates the divided areas with each other using the associated feature points. Then, the image processing unit 2 associates each pixel with a rectangular aspect ratio including the associated region (step S5).

  Here, the processing operation of step S5 will be described in detail with reference to FIG. FIG. 5 is an explanatory diagram showing the processing operation of step S5 shown in FIG. Assume that the feature point 401 in the image a and the feature point 402 in the image b are associated in step S2. The areas divided in step S4 including the feature point 401 and the feature point 402 (here, the area 403 in the image a and the area 404 in the image b) are associated with each other.

  Then, the rectangles including the regions 403 and 404 (here, the rectangle 405 in the image a and the rectangle 406 in the image b) are calculated. At this time, each pixel included in the rectangle 405 and the rectangle 406 (here, a plurality of pixels 407 in the image a and a plurality of pixels 408 in the image b) is extracted, and each pixel is associated according to the aspect ratio. .

  Returning to FIG. 2, next, the image generation unit 3 generates an intermediate viewpoint image from the pixels associated in the image processing unit 2 (step S <b> 6).

Here, the processing operation of step S6 will be described in detail with reference to FIG. FIG. 6 is an explanatory diagram showing the processing operation of step S6 shown in FIG. Image generating unit 3, the viewpoint V _alpha (inner from the image a interpolation ratio alpha, the image b viewpoint position of 1-alpha) internal parameters I _v viewpoint V _alpha in _an external parameter (rotation) of the viewpoint V _alpha R _v the external parameters (translation) _{T v} viewpoint V _alpha, is calculated by the following (see FIG. 6).

I _v = (1−α) × I _a + α × I _b
q _v = (1−α) × q _a + α × q _b
P _v = (1−α) × P _a + α × P _b
Here, q is a quaternion obtained from a rotation matrix, and P is a coordinate of a three-dimensional point represented by P = −R ^T T.

For all pixels of the image a, the point _P a on the image _a, P ab: _{P a} and the point _P a in the world coordinate system of the point _{P ab} on the corresponding image _b, and _{P ab} respectively calculated, the pixel The point P _v and the color I (P _v ) in the world coordinate system are calculated as follows.

_{P v = (1-α)} × P a + α × P ab
I (P _v ) = (1−α) × I (P _a ) + α × I (P _ab )

Similarly, for all the pixels of the image b, a point _P b on the image b, _{P b} and the point _P b in the world coordinate system of the point _{P ba} on the corresponding image _a, the _{P ba} respectively calculated, the pixel The point P _v and the color I (P _v ) in the world coordinate system are calculated as follows.

P _v = (1−α) × P _ba + α × P _b
I (P _v ) = (1−α) × I (P _ba ) + α × I (P _b )

An intermediate viewpoint image is generated by assigning the color I (P _v ) to the coordinates of P _v calculated as described above. For example, when two images (image a and image b) shown in FIG. 3 are input, an intermediate viewpoint image as shown in FIG. 7 is generated when α = 0.5. . FIG. 7 is a diagram illustrating an example of the generated intermediate viewpoint image.

It should be noted that the transformation between the point (u, v) in the image coordinate system and the point (X, Y, Z) in the world coordinate system follows the following equation.

  As described above, according to the configuration described above, an intermediate viewpoint image viewed from the viewpoint at the intermediate position can be generated from two images having different viewpoints. In particular, by using the region including the feature point or the rectangular region including the region, by associating the pixel of the processing target image used for calculating the color of the pixel constituting the intermediate viewpoint image, It is possible to associate even scenes that are difficult to associate.

  The intermediate viewpoint image generation device in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  Even in a scene where it is difficult to associate feature points between images, it can be applied to applications where it is essential to generate an intermediate viewpoint image.

  DESCRIPTION OF SYMBOLS 1 ... Image input part, 2 ... Image processing part, 3 ... Image generation part

Claims (2)

An image input unit for inputting a plurality of images to be processed;
A feature point associating unit for associating feature points of each of the two input images;
A camera parameter calculation unit that calculates camera parameters of a camera that has captured the two images from the associated feature points;
A region dividing unit that performs region division on the two input images;
A pixel association unit for associating pixels from the associated feature points included in the divided region;
An intermediate viewpoint image generation device, comprising: an intermediate viewpoint image generation unit that generates an intermediate viewpoint image at a specified intermediate viewpoint position using the associated pixels. The pixel association unit includes:
2. The pixel is correlated on the basis of the aspect ratio of a rectangle that includes the associated area by associating the divided areas from the associated feature points. The intermediate viewpoint image generation device described in 1.