JP2017017581A - Virtual viewpoint image generation device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a virtual viewpoint image generation device and a program capable of reducing distortion of an interpolation image while suppressing an increase in computation amount.SOLUTION: A virtual viewpoint image generation device includes: a horizontal direction processing part which generates a horizontal direction interpolation image from a first input image, a second input image, and a position of a virtual viewpoint and outputs a horizontal direction parallax vector showing relations between blocks or pixels matching or similar between the first input image and the second input image; a vertical direction processing part which determines a search range in a horizontal direction from at least the horizontal direction parallax vector and generates a vertical direction interpolation image from the vertical image in the virtual viewpoint, the first input image, and/or the second input image; and an interpolation image integration part generating a virtual viewpoint image by integrating the horizontal direction interpolation image with the vertical direction interpolation image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a virtual viewpoint image generation apparatus and program, and is particularly suitable for generating an image that would be obtained when shooting at a virtual viewpoint position from a plurality of images shot at two or more viewpoint positions. Is.

  In recent years, viewers have a stereoscopic effect due to binocular parallax and motion parallax, such as a display device that changes the display viewpoint following the movement of the viewer's eyes and a display device that projects images of two or more viewpoints simultaneously. Display devices that provide images are attracting attention.

  In such a display device, the small distance between the viewpoints to be photographed often greatly affects the subjective quality, and it is required to reduce the distance between the viewpoints as much as possible. In the case of a still image, for example, an image captured at a position of two or more viewpoints can be obtained by repeating processing such as moving the camera and shooting again after shooting with the camera. Further, at least when such a method is used, the distance between the viewpoints can be reduced by reducing the amount of movement of the camera described above.

  However, when shooting a moving image instead of a still image, it is necessary to reduce the size of the camera module in order to reduce the distance between the viewpoints. There are trade-offs in the downsizing of the camera module, such as a reduction in resolution capable of photographing, a reduction in lens performance, and an increase in design and manufacturing costs.

  For this reason, from an image captured from a limited viewpoint (hereinafter also referred to as an actual captured image), for example, captured at a viewpoint (virtual viewpoint) between a plurality of viewpoints related to the actual captured image such that a camera module cannot be installed. There is a need for a technique for estimating and generating a generated image (hereinafter also referred to as a virtual viewpoint image).

  Patent Document 1 proposes a technique for generating an image photographed from a virtual viewpoint from three images photographed from two viewpoints. The technique described in Patent Literature 1 searches for a corresponding block or pixel between images and obtains an interpolation vector therefrom, thereby generating an interpolated image as a virtual viewpoint image.

  Non-Patent Document 1 proposes a technique for complementing between frames and improving the frame rate. It can be assumed that the technique described in Non-Patent Document 1 is applied to viewpoint interpolation.

  Detailed processing contents differ between the technology described in Patent Document 1 and the technology described in Non-Patent Document 1. However, the search for the corresponding block or pixel, the calculation of the vector corresponding to the viewpoint to be interpolated (virtual viewpoint), and the generation of an interpolated image (virtual viewpoint image) using the vector are described in Patent Document 1. The description technique and the description technique of Non-Patent Document 1 are common.

  FIG. 8 is a block diagram in which the configuration of the common description technique described above for generating a virtual viewpoint image from two images taken from two viewpoints is shown. As shown in FIG. 8, the apparatus (hereinafter referred to as a conventional virtual viewpoint image generation apparatus) has a configuration in which a first input image PIC1, a second input image PIC2, and a virtual viewpoint position VPV are input and an interpolation image IPPIC0 is input. Is output.

  The first input image PIC1 and the second input image PIC2 input to the conventional virtual viewpoint image generation device 10 are actual captured images having the same vertical direction but different horizontal directions. A virtual viewpoint position VPV is designated on a line segment connecting the viewpoint of the first input image PIC1 and the viewpoint of the second input image PIC2. Therefore, as shown in FIG. 9, the conventional virtual viewpoint image generation apparatus 10 interpolates the first input image PIC1 and the second input image PIC2 in the horizontal direction according to the virtual viewpoint position VPV, and performs the interpolation image. (Virtual viewpoint image) The virtual viewpoint image is composed of a horizontal processing unit 110 that obtains IPPIC0.

  The horizontal direction processing unit 110 searches for a block (or pixel) that matches or resembles between the first input image PIC1 and the second input image PIC2, and outputs a horizontal direction disparity vector. Unit 111, an interpolation vector calculation unit 112 for obtaining an interpolation vector from the horizontal direction parallax vector, the virtual viewpoint position, and the viewpoint position of the actual captured image, and the horizontal direction interpolation image from the interpolation vector and the second input image PIC2. (Virtual viewpoint image) It has a horizontal direction interpolation image generation unit 113 for generating IPPIC0 (detailed functions of the respective units 111, 112, 113 are described in the section “Embodiment” described later).

JP 2014-72801 A

J. et al. Ascenso, C, Brites, and F.M. Pereira, “Improving frame interpolation with spatial motion smoothing for pixel domain distributed video coding,” 5th EURASIP Conference on, 2005.

  However, when a plurality of input images is associated with each block (or pixel), for example, due to the presence of occlusion or a difference in lighting, the correspondence with each block (or pixel) is performed. Errors can occur. An error in correspondence between blocks or pixels leads to distortion of the generated interpolation image. In addition, for an area where occlusion occurs, even if the occlusion area can be detected correctly, it is difficult to accurately determine how to set pixel values belonging to the same area from two viewpoints. Thus, the estimation error appears as distortion of the interpolated image.

  Therefore, there is a need for a virtual viewpoint image generation apparatus and program that can reduce the distortion of an interpolated image while suppressing an increase in the amount of calculation.

  In order to solve such a problem, the virtual viewpoint image generation device according to the first aspect of the present invention (1) generates a horizontal interpolation image from the first input image, the second input image, and the position of the virtual viewpoint. A horizontal processing unit that searches for a block or pixel that matches or is similar between the first input image and the second input image and outputs at least a horizontal disparity vector that represents the relationship; (2) At least a horizontal search range is determined from the horizontal direction parallax vector, and is taken from a viewpoint shifted in the vertical direction from a straight line passing through both viewpoints related to the first input image and the second input image. At least one vertical direction processing unit that generates a vertical direction interpolation image from one or both of the first input image and the second input image, and (3) horizontal direction interpolation. Image and vertical complement Integrating the image, characterized in that it comprises an interpolation image integration unit for generating a final virtual viewpoint image.

  According to a second aspect of the present invention, there is provided a virtual viewpoint image generation program for generating a horizontally interpolated image from a first input image, a second input image, and a position of a virtual viewpoint. A horizontal direction processing unit that searches for a block or pixel that matches or resembles between one input image and a second input image, and outputs at least a horizontal direction disparity vector representing the relationship; (2) at least, A horizontal search range is determined from the horizontal disparity vector, and a vertical image captured from a viewpoint shifted in the vertical direction from straight lines passing through both viewpoints of the first input image and the second input image, and At least one vertical direction processing unit that generates a vertical direction interpolation image from one or both input images of the first input image and the second input image; and (3) a horizontal direction interpolation image and a vertical direction interpolation image. Integrated, characterized in that to function as the interpolation image integration unit for generating a final virtual viewpoint image.

  According to the present invention, it is possible to reduce distortion of an interpolated image while suppressing an increase in calculation amount.

It is a block diagram which shows the structure of the virtual viewpoint image generation apparatus of embodiment. It is explanatory drawing of the positional relationship of the viewpoint of the input image to the virtual viewpoint image generation apparatus of embodiment, and a virtual viewpoint. It is explanatory drawing explaining the interpolation method of the vertical direction in the virtual viewpoint image generation apparatus of embodiment. It is a flowchart which shows the flow of operation | movement in the virtual viewpoint image generation apparatus of embodiment. It is explanatory drawing which shows the relationship between the horizontal image in the virtual viewpoint image generation apparatus of embodiment, a vertical image, and a virtual viewpoint image. It is explanatory drawing explaining the positional relationship of the camera which image | photographs the horizontal image in the virtual viewpoint image generation apparatus of embodiment, and the virtual viewpoint to interpolate. It is explanatory drawing of the positional relationship of the viewpoint of the input image to the virtual viewpoint image generation apparatus of deformation | transformation embodiment, and a virtual viewpoint. It is a block diagram which shows the structure of the conventional virtual viewpoint image generation apparatus. It is explanatory drawing of the relationship between the viewpoint of the input image to the conventional virtual viewpoint image generation apparatus, and a virtual viewpoint.

(A) Main Embodiments Hereinafter, embodiments of a virtual viewpoint image generation device and a program according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of Embodiment FIG. 1 is a block diagram illustrating a configuration of a virtual viewpoint image generation device according to an embodiment. In FIG. 1, the same and corresponding parts as those in FIG.

  The virtual viewpoint image generation device 100 may be constructed by connecting various hardware components, and the function of the program is applied by applying an execution configuration of a program such as a CPU, ROM, or RAM. It may be constructed so as to be realized. In any case, the functional detailed configuration of the virtual viewpoint image generation device 100 can be represented in FIG. 1 even when the construction method is applied.

  The virtual viewpoint image generation apparatus 100 includes a horizontal direction processing unit 110, a first vertical direction processing unit 120, a second vertical direction processing unit 130, and an average processing unit 101.

  The horizontal direction processing unit 110 includes a horizontal direction matching block search unit 111, an interpolation vector calculation unit 112, and a horizontal direction interpolation image generation unit 113.

  The first vertical direction processing unit 120 includes a first search range determination unit 121, a first vertical direction matching block search unit 122, a first shift vector calculation unit 123, and a first vertical direction interpolation image generation unit 124. Have. Similarly, the second vertical direction processing unit 130 includes a second search range determination unit 131, a second vertical direction matching block search unit 132, a second shift vector calculation unit 133, and a second vertical direction interpolation image generation. Part 134.

  In the virtual viewpoint image generation device 100, the viewpoint position is predetermined (for example, predetermined by the arrangement of the camera module), the first input image (hereinafter referred to as the first horizontal image) PIC1, the first Two input images (hereinafter referred to as second horizontal images) PIC2 and a third input image (hereinafter referred to as vertical images) PIC3 are input. Each of the input images PIC1 to PIC3 has an x-coordinate value in which the upper left is the origin, the horizontal direction passing through the origin and the right side is positive, and the y coordinate in the vertical direction passing through the origin and the lower side is positive. It is an image for specifying the position by the value (see FIG. 3). The viewpoint relating to the first horizontal image PIC1 and the viewpoint relating to the second horizontal image PIC2 are the same in the vertical direction but different in the horizontal direction.

  Here, as shown in FIG. 2, the virtual viewpoint position VPV is designated on a line segment connecting the viewpoints of the first horizontal image PIC1 and the second horizontal image PIC2. As shown in FIG. 2, the viewpoint (position) of the vertical image PIC3 is the same as the virtual viewpoint position VPV in the horizontal direction but different in the vertical direction.

  In the vertical image, as shown in FIG. 2, a straight line passing through the viewpoint of the vertical image and the virtual viewpoint is orthogonal to a straight line passing through both viewpoints related to the first horizontal image PIC1 and the second horizontal image PIC2. This is an image in which the viewpoint of the vertical image is set at a position.

  The horizontal direction processing unit 110 receives the first horizontal image PIC1 and the second horizontal image PIC2, and the virtual viewpoint position VPV. The horizontal direction processing unit 110 generates an interpolation image (hereinafter referred to as a horizontal direction interpolation image) HPIC when it is assumed that the image is taken from the virtual viewpoint position VPV from the first horizontal image PIC1 and the second horizontal image PIC2. To the average processing unit 141. Each part (111-113) in the horizontal direction processing part 110 acts as follows.

  The horizontal direction matching block search unit 111 searches for a block (or pixel) that matches or is similar (hereinafter referred to as “corresponding”) between the first horizontal image PIC1 and the second horizontal image PIC2, and performs horizontal processing. A directional parallax vector is obtained and output. The horizontal disparity vector is, for example, the amount of horizontal displacement (size) between corresponding blocks (or pixels), the position of the block (or pixel) in the second horizontal image PIC2 as the starting point, and the first horizontal disparity vector. This is a vector whose end point is the position of the corresponding block (or pixel) in the image PIC1.

  Further, the horizontal direction matching block search unit 111 is a horizontal direction matching degree indicating the degree of coincidence between the first horizontal image PIC1 and the second horizontal image PIC2 or the coincidence or similarity of similar blocks (or pixels). Are output to the vertical processing units 120 and 130. Here, as will be described later, various methods can be applied to the search range determination method by the vertical processing units 120 and 130. Therefore, when the vertical direction processing units 120 and 130 use the method of determining the search range without using the horizontal matching degree, the horizontal direction matching block search unit 111 performs the horizontal direction matching to the vertical direction processing units 120 and 130. The degree of output can be unnecessary.

The interpolation vector calculation unit 112 obtains an interpolation vector for each block (or pixel) by a proportional process for the horizontal direction parallax vector for each block (or pixel). When the viewpoint of the first horizontal image PIC1 and the distance between the viewpoint of the second horizontal image PIC2 and _{L 12,} the distance between the virtual viewpoint (virtual viewpoint position) and the viewpoint of the second horizontal image PIC2 and _{L V2,} The interpolation vector calculation unit 112 obtains an interpolation vector by multiplying the horizontal parallax vector by L ₁₂ / L _V2 .

  The horizontal direction interpolated image generation unit 113 moves the image content of each block (or pixel) in the second horizontal image PIC2 as the reference image by an interpolation vector for the block (or pixel), thereby moving the horizontal direction interpolated image. A block (or pixel) of HPIC is generated, and a horizontal direction interpolation image HPIC is generated by executing such a movement operation for all the blocks (or pixels).

  In this embodiment, a case where the reference image is the second horizontal image PIC2 is illustrated. However, the reference image may be the first horizontal image PIC1, and the reference image may be both the first horizontal image PIC1 and the second horizontal image PIC2.

  The first horizontal image processing unit 120 receives the first horizontal image PIC1 and the vertical image PIC3. The first vertical direction processing unit 120 is an interpolation image (hereinafter referred to as a first vertical direction interpolation image) VPIC1 when it is assumed that the first horizontal image PIC1 and the vertical image PIC3 are taken from the virtual viewpoint position VPV. Is generated and given to the average processing unit 141.

  Hereinafter, the operation of each unit (121 to 124) in the first vertical processing unit 120 will be described with reference to FIG. FIG. 3 is an explanatory diagram of a method for generating the first vertically interpolated image VPIC1 from the first horizontal image PIC1 and the vertical image PIC3. “Triangle” in FIG. 3 represents the image portion of the subject.

  The first search range determination unit 121 determines a horizontal search range from the horizontal disparity vector and the horizontal matching degree, and outputs the search range to the vertical matching block search unit 122. Detailed processing of the method of determining the horizontal search range by the search range determination unit 121 will be described in the section of operation.

  As will be described later, when the search range determination method by the search range determination unit 121 does not use the horizontal matching degree, the horizontal matching degree may be unnecessary. Further, only one of the vertical processing units 120 and 130 acquires the horizontal matching degree from the horizontal processing unit 110, and any one of the vertical processing units 120 and 130 searches using the horizontal matching degree. The range may be determined.

  The first vertical direction matching block search unit 122 limits the horizontal search range to the search range obtained by the first search range determination unit 121, and between the first horizontal image PIC1 and the vertical image PIC3. , Search for a corresponding block (or pixel), and output a first vertical disparity vector. The first vertical disparity vector is obtained by, for example, a block matching algorithm. The first vertical disparity vector corresponds to the shift amount (size) between corresponding blocks (or pixels) in the first horizontal image PIC1 with the position of the block (or pixel) in the vertical image PIC3 as the starting point. It is a vector whose end point is the position of a block (or pixel). A block BR1 in FIG. 3 is a block in the vertical image PIC3, and a block in the first horizontal image PIC1 corresponding to the block BR1 is a block BR2. Therefore, the first vertical disparity vector VTR1 for the block BR1 is a vector that connects the center of the block BR1 to the center of the block BR2.

  The first shift vector calculation unit 123 obtains the first shift vector by extracting the vertical component vector of the first vertical parallax vector for each block (or pixel). As shown in FIG. 3, when the component of the first vertical disparity vector VTR1 is (x, y) = (α, β), the component of the first shift vector VTR2 is (x, y) = (0, β). That is, the first shift vector calculation unit 123 outputs a vector in which the horizontal component is “0” in the first vertical parallax vector VTR1 as the first shift vector VTR2.

  The first vertically interpolated image generation unit 124 moves the image content of each block (or pixel) in the vertical image PIC3 by the first shift vector VTR2 for the block (or pixel) and moves the first vertical direction. A block (or pixel) in the interpolated image VPIC1 is formed, and such an operation is performed for all the blocks (or pixels), thereby generating the first vertical direction interpolated image VPIC1. A block BR3 obtained by moving the image content of the block BR1 shown in FIG. 3 by the first shift vector VTR2 is a block (interpolation block) corresponding to the block BR1 in the first vertical direction interpolation image VPIC1.

  The first vertical direction processing unit 120 generates the first vertical direction interpolation image VPIC1 from the first horizontal image PIC1 and the vertical image PIC3, but the second vertical direction processing unit 130 The second vertically interpolated image VPIC2 is generated from the horizontal image PIC2 and the vertical image PIC3. One input image to be processed is different between the first vertical processing unit 120 and the second vertical processing unit 130, but each unit (131 to 134) in the second vertical processing unit 130 is the first Since the operation is the same as the corresponding units (121 to 124) in the vertical direction processing unit 120, the description of the function is omitted.

The average processing unit 141 is provided as an interpolation image integration unit, integrates the horizontal direction interpolation image HPIC, the first vertical direction interpolation image VPIC1, and the second vertical direction interpolation image VPIC2, and finally outputs them. An interpolation image (virtual viewpoint image) IPPIC is obtained. The average processing unit 141 obtains an output interpolated image (virtual viewpoint image) IPPIC by applying the average process as the integration process. The averaging process in the first embodiment is a simple averaging process (the second embodiment described later is a weighted average to which a dynamic weighting coefficient is applied). If the average processing unit 141 applies a simple average process, the average processing unit 141 uses P as the pixel value of the horizontally interpolated image HPIC at a certain coordinate in the coordinate system (see FIG. 3), and the first vertical unit. If the pixel value of the direction interpolation image VPIC1 is Q1, and the pixel value of the second vertical direction interpolation image VPIC2 is Q2, the pixel value R of the output interpolation image IPPIC is calculated according to equation (1).
R = (P + Q1 + Q2) / 3 (1)

(A-2) Operation of Embodiment Next, the processing operation of the virtual viewpoint image generation processing in the virtual viewpoint image generation device 100 according to the embodiment having the above-described units will be described in detail.

  FIG. 4 is a flowchart showing a flow of operations in the virtual viewpoint image generation device 100 according to the embodiment.

(A-2-1) Flow of Overall Operation First, the horizontal direction disparity vector is calculated from the first horizontal image PIC1 and the second horizontal image PIC2 in the horizontal direction matching block search unit 111 in the horizontal direction processing unit 110. It is obtained (step S101). Next, the interpolation vector calculation unit 112 obtains an interpolation vector from the horizontal parallax vector and the virtual viewpoint position (step S102). Further, the horizontal direction interpolation image generation unit 113 obtains a horizontal direction interpolation image HPIC from the interpolation vector and the second horizontal image PIC2 (step S103).

  Here, when the vertical direction processing units 120 and 130 refer to only the horizontal direction parallax vector to generate the vertical direction interpolation image, the horizontal direction matching block search unit 111 does not output the horizontal direction matching degree.

  In addition, the search range determination unit 121 of the first vertical direction processing unit 120 obtains a search range from the horizontal direction parallax vector and the horizontal direction matching degree (step S104). Since a plurality of determination methods can be applied to this search range determination method, the plurality of search range determination methods will be described later.

  In the first vertical direction matching block search unit 122 in the first vertical direction processing unit 120, a first vertical direction parallax vector is obtained from the first horizontal image PIC1 and the vertical image PIC3 (step S105). Next, in the first shift vector calculation unit 123, a first shift vector is obtained from the first vertical parallax vector (step S106). Further, the first vertical direction interpolation image generation unit 124 generates the first vertical direction interpolation image VPIC1 from the first shift vector and the vertical image PIC3 (step S107).

  In the second vertical direction processing unit 130, the same operation as that of the first vertical direction processing unit 120 is executed, and the second vertical direction interpolation image VPIC2 is generated (step S108). That is, in the second search range determination unit 131, the search range is obtained from the horizontal direction parallax vector and the horizontal matching degree. The second vertical direction matching block search unit 132 obtains the second vertical direction parallax vector from the second horizontal image PIC2 and the vertical image PIC3, and then the second shift vector calculation unit 133 determines the second vertical direction parallax vector. The second shift vector is obtained from the two vertical direction parallax vectors, and the second vertical direction interpolation image generation unit 134 further calculates the second vertical direction interpolation image from the second shift vector and the vertical image PIC3. VPIC2 is generated.

  Finally, in the average processing unit 141, an output interpolation image IPPIC is obtained by averaging from the horizontal direction interpolation image HPIC, the first vertical direction interpolation image VPIC1, and the second vertical direction interpolation image VPIC2 (step S1). S109).

(A-2-2) Search Range Determination Method Next, in step S104 of FIG. 4, the search range when the search range is determined from the horizontal direction parallax vector and the horizontal direction matching degree in the search range determination unit 121. A determination method will be described.

  In the following, description will be given by taking as an example the determination of the vertical direction parallax vector from the first horizontal image PIC1 and the vertical image PIC3 unless otherwise specified.

  In the following, as shown in FIG. 5, the horizontal direction when another input image (for example, the second horizontal image PIC2) is referenced from the input image of interest (for example, the first horizontal image PIC1). The disparity vector is “v”, the virtual viewpoint position parameter indicating the position of the virtual viewpoint to be interpolated is “c”, and the code parameter is “s”.

The virtual viewpoint position parameter c is a real number having a range (0.0, 1.0). Further, as shown in FIG. 6, another input image (for example, the second horizontal image PIC2) viewed from the position of the camera that captures the target input image (for example, the first horizontal image PIC1) is captured. If the distance to the camera position is “p” and the distance from the camera position where the target input image (for example, the first horizontal image PIC1) is taken to the virtual viewpoint position to be interpolated is “q”, (2 ) Expression.
c = q / p (2)

  When the virtual viewpoint position to be interpolated is located exactly between the position of the camera that captures the first horizontal image PIC1 and the position of the camera that captures the second horizontal image PIC2, the virtual viewpoint position parameter c is c. = 0.5.

  Also, the sign parameter s is greater than the virtual viewpoint position that the camera that captures the input image wants to interpolate when the position of the object of the input image is relatively left when viewed from the position of the object of the vertical image PIC3. When it is relatively on the right side, s = −1, and in other cases, s = 1.

  Here, there are roughly two methods for determining the search range. One is a method that does not use the horizontal matching degree, and the other is a method that uses the horizontal matching degree. Hereinafter, these two types of search range determination methods will be described in order.

  First, a method not using the horizontal matching degree will be described. There are two methods for obtaining the search range without using the horizontal matching degree.

A first search range determination method that does not use the horizontal matching degree is a method of using equation (3) when the search range is “α”.
α∈ [scv−r, scv + r] (3)

  That is, the search range α is a range from a value obtained by subtracting the parameter r to the result scv obtained by multiplying the sign parameter s, the virtual viewpoint position parameter c, and the horizontal parallax vector v to a value obtained by adding the parameter r to the result scv. To do. Here, the parameter r is a predetermined parameter of 0 or more. When r = 0, the search range α is α = scv, the horizontal search is one point, and the horizontal search is virtually not performed. Conversely, when the value of the parameter r is increased, the search range is expanded.

  The smaller the value of the parameter r, the smaller the calculation amount. However, for example, when the quality of the horizontal direction parallax vector v is poor due to occlusion or the like, the first vertical direction interpolated image VPIC1, the first vertical direction processing unit 120, the second vertical direction processing unit 130, This adversely affects the quality of the second vertically interpolated image VPIC2 in the form of deterioration.

  On the contrary, if the value of the parameter r is increased, even if the quality of the horizontal parallax vector v deteriorates due to, for example, occlusion, the first vertical direction processing unit 120 and the second vertical direction processing unit 130 Reduce the possibility of adverse effects on However, there is a demerit that the amount of calculation increases.

A second search range determination method that does not use the horizontal matching degree is a method in which the search range α is set to equation (4).
α∈ [scv−r, scv + r] ∪ [scv ₁ −r, scv ₁ + r]
∪ [scv ₂ −r, scv ₂ + r]
∪. . .
∪ [scv _k −r, scv _k + r]
∪. . .
∪ [scv _n −r, scv _n + r] (4)

Here, in Equation (4), the horizontal direction parallax vector v _k is a horizontal direction parallax vector of a block (or pixel) adjacent to the block (or pixel) of interest.

  In equation (4), the horizontal disparity vectors of n neighboring blocks (or pixels) are included in the calculation formula, but the horizontal disparity vectors of how many neighboring blocks (or pixels) are included in the calculation formula. Is a predetermined parameter.

  In the first search range determination method that does not use the horizontal matching degree, if the quality of the horizontal disparity vector is poor when the parameter r is reduced, the vertical processing unit also deteriorates the quality of the interpolated image. The problem of “having an adverse effect in the form of” is often caused by differences in occlusion and lighting.

  On the other hand, the second method of determining the search range that does not use the degree of horizontal matching is that, in a natural image, (a) problems that occur due to differences in occlusion, lighting, etc. (Pixel) may not occur, and (b) the problem that the first search range is determined using the fact that the horizontal disparity vector is spatially smooth This is a method of suppressing the occurrence.

  In this way, when the second search range determination method is employed, the parameter r can be made smaller than when the first search range determination method is employed, and therefore the first vertical direction interpolation image VPIC1, It can be expected that the calculation amount is suppressed while maintaining the image quality of the second vertically interpolated image VPIC2.

  Next, a method for determining the search range using the horizontal matching degree will be described. When the horizontal matching degree is used, the search range is determined by a combination of the above-described two types of search range determination methods.

  Thereafter, the horizontal matching degree M is a real number of 0 or more. An example will be described in which the block (or pixel) matching degree between the first horizontal image PIC1 and the second horizontal image PIC2 is higher as the horizontal matching degree M is smaller.

When the value of the horizontal matching degree M is smaller than a predetermined threshold T, that is, when the matching degree is high, the search range α is calculated using the horizontal disparity vector v of the block (or pixel) of interest. Is obtained by equation (5).
α∈ [scv−r, scv + r] (5)

  When the value of the horizontal matching degree M is equal to or greater than a predetermined threshold T, that is, when the matching degree is low, the horizontal disparity vector of a block (or pixel) adjacent to the block (or pixel) of interest is calculated. The search range α is obtained by equation (6).

α∈ [scv ₁ −r, scv ₁ + r] ∪ [scv ₂ −r, scv ₂ + r]
∪. . .
∪ [scv _k −r, scv _k + r]
∪. . .
∪ [scv _n −r, scv _n + r] (6)
When the value of the degree of matching M in the horizontal direction is high, it can be predicted that this is a region where no occlusion or a difference in lighting is occurring. Therefore, the search range is determined using the horizontal disparity vector of the block (or pixel) of interest.

  When the value of the horizontal matching degree M is low, it can be predicted that the block (or pixel) of interest is a region where occlusion, illumination difference, or the like occurs. Therefore, the search range is determined using the horizontal disparity vector of the block (or pixel) in the vicinity of the block (or pixel) of interest.

  By using the degree of matching M in the horizontal direction, even if the value of the parameter r or the number n of adjacent blocks (or pixels) is reduced, problems that occur due to differences in occlusion or lighting, etc. can be suppressed. Therefore, it can be expected that the calculation amount is suppressed while maintaining the quality.

(A-3) Effect of Embodiment As described above, according to the embodiment, the existence of occlusion is obtained by generating an interpolated image using viewpoint information (vertical image) captured in the vertical direction from the virtual viewpoint. Further, the distortion of the interpolated image that occurs due to the difference in the way of lighting or lighting is reduced.

  This is because, even in an area that cannot be associated between the first horizontal image PIC1 and the second horizontal image PIC2, there is a high possibility that they can be associated in the vertical direction from the virtual viewpoint.

  Even in a region that cannot be associated in the vertical direction, there is a possibility that association can be made between the first horizontal image PIC1 and the second horizontal image PIC2. By taking advantage of both by averaging the horizontal walking interpolation image and the vertical direction interpolation image, distortion of the interpolation image that occurs due to the presence of occlusion, the difference in the way of lighting, and the like is reduced.

  Furthermore, according to the embodiment, the calculation amount of the vertical direction processing unit is reduced by using the calculation result of the horizontal direction processing unit in terms of the calculation amount, and an increase in the calculation amount can be suppressed.

(B) Other Embodiments Although various modified embodiments have been mentioned in the above-described embodiments, the present invention can also be applied to the following modified embodiments.

  (B-1) In the above-described embodiment, the case where the horizontal direction matching block search unit 111 outputs the horizontal direction parallax vector and the horizontal direction matching degree to the vertical direction processing unit 120 is exemplified. However, the horizontal direction matching block search unit 111 may output the horizontal direction parallax vector and the horizontal matching degree to the vertical direction processing unit 130 as well as the vertical direction processing unit 120.

  (B-2) In the embodiment described above, the case where the horizontal processing unit 110 generates the horizontal interpolation image HPIC from the second horizontal image PIC2 has been described, but the method of generating the horizontal interpolation image HPIC is described here. It is not limited. In short, any generation method may be used as long as the horizontal direction interpolation image HPIC can be obtained from the first horizontal image PIC1, the second horizontal image PIC2, and the virtual viewpoint position VPV. For example, a horizontally interpolated image HPIC may be generated from the first horizontal image PIC1, and the first horizontal image PIC1 and the second horizontal image PIC2 may be generated as described in Non-Patent Document 1. You may make it produce | generate the horizontal direction interpolation image HPIC from both.

  (B-3) In the above embodiment, the first vertical direction interpolated image VPIC1 is generated from the first horizontal image PIC1 and the vertical image PIC3, and the second horizontal image PIC2 and the vertical image PIC3 are The vertical interpolation image VPIC2 is generated and reflected in the final output interpolation image IPPIC. However, in order to reduce the amount of calculation, the first vertical interpolation image VPIC1 and the second vertical interpolation image VPIC2 are shown. Only one of them may be generated and reflected in the final output interpolation image IPPIC.

  (B-4) In the above-described embodiment, an example in which a vertical image from a viewpoint above a line segment connecting both viewpoints of the first horizontal image PIC1 and the second horizontal image PIC2 is applied is shown. However, a vertical image from a viewpoint below the line segment may be applied, and a vertical image from a viewpoint above the line segment and a viewpoint below the line segment may be applied. Both of the vertical images may be applied.

  As is clear from the latter case, the number of vertically interpolated images reflected in the final output interpolated image IPPIC is not limited to 1 or 2, and the number of vertically interpolated images is N (N is a natural number). ).

  (B-5) In the above embodiment, the case where an output interpolation image is generated by a simple average of a plurality of interpolation images (horizontal direction interpolation image and vertical direction interpolation image) has been exemplified. However, the average processing unit may generate an output interpolation image by weighted average of a plurality of interpolation images to which a dynamic weighting factor is applied. The average processing unit may generate an output interpolation image by weighted average of a plurality of interpolation images to which a fixed weighting factor is applied. For example, a fixed weighting factor may be determined according to the distance between the viewpoint of the image that has generated the interpolation image (the second horizontal image in the case of a horizontal direction interpolation image) and the virtual viewpoint.

  (B-6) In the operation description of the above embodiment, the operation of the horizontal processing unit 110, the operation of the first vertical processing unit 120, and the operation of the second vertical processing unit 130 are executed in this order. Showed the case. However, the order of operations is not limited to this. Two or more operations of the horizontal processing unit 110, the first vertical processing unit 120, and the second vertical processing unit 130 may be executed by parallel processing. good. Further, for example, after obtaining the horizontal direction parallax vector in the horizontal direction processing unit 110, the first vertical direction processing unit 120 obtains the first vertical direction parallax vector, and then the second vertical direction processing unit 130 obtains the second Of the horizontal direction processing unit 110 so as to execute the calculation of the interpolation vector, the calculation of the first shift vector, and the calculation of the second shift vector in this order. The operation of one vertical processing unit 120 and the operation of the second vertical processing unit 130 may be subdivided and cyclically executed in a time division manner.

  (B-7) The above embodiment shows a case where there is one virtual viewpoint. However, even if there are two or more virtual viewpoints and the virtual viewpoint image generation apparatus constitutes a multi-viewpoint image generation system. good. For example, two virtual viewpoint images as stereo images may be generated. In this case, the first vertical image from the viewpoint above the line segment connecting both viewpoints of the first horizontal image PIC1 and the second horizontal image PIC2 and the second from the viewpoint below the line segment described above. Both of the vertical images may be applied. FIG. 7 shows that a line segment connecting both viewpoints of the first horizontal image PIC1 and the second horizontal image PIC2 is divided into three equal parts, and the first and second positions on the line segment are 1/3 and 2/3. In this example, two virtual viewpoints are defined, the viewpoint of the first vertical image is located above the first virtual viewpoint, and the viewpoint of the second vertical image is located below the second virtual viewpoint. In this way, by setting the viewpoints of two vertical images up and down, interpolation at 1/3 intervals can be performed with high accuracy.

  (B-8) In the description of the above-described embodiment, the horizontal matching degree is output for each block. However, the matching degree may be output for each pixel.

  (B-9) In the above embodiment, the virtual viewpoint is on a line segment connecting both viewpoints of the first horizontal image PIC1 and the second horizontal image PIC2, and the horizontal direction interpolation image is obtained by interpolation (interpolation). The case of generating was shown. However, the technique of the present invention is also applicable when a virtual viewpoint is located on an extension of a line segment connecting both viewpoints of the first horizontal image PIC1 and the second horizontal image PIC2, and a horizontal image is generated by extrapolation. The idea can be applied. The term “interpolation” is a term representing interpolation, but “interpolation” in the claims means both interpolation and extrapolation.

100: Virtual viewpoint image generation device,
110, ... Horizontal direction processing unit, 111 ... Horizontal direction matching block search unit, 112 ... Interpolation vector calculation unit, 113 ... Horizontal direction interpolation image generation unit,
120, 130 ... vertical direction processing unit, 121, 131 ... search range determination unit, 122, 132 ... vertical direction matching block search unit, 123, 133 ... shift vector calculation unit, 124, 134 ... vertical direction interpolation image generation unit,
141: Average processing unit.

Claims (11)

A horizontal direction interpolation image is generated from the first input image and the second input image and the position of the virtual viewpoint, and the first input image and the second input image are identical or A horizontal processing unit that searches for similar blocks or pixels and outputs at least a horizontal disparity vector representing the relationship;
At least a horizontal search range is determined from the horizontal parallax vector, and is taken from a viewpoint shifted in the vertical direction from a straight line passing through both viewpoints related to the first input image and the second input image. At least one vertical direction processing unit that generates a vertical direction interpolation image from a vertical image and either or both of the first input image and the second input image;
A virtual viewpoint image generation apparatus comprising: an interpolation image integration unit that integrates the horizontal direction interpolation image and the vertical direction interpolation image to generate a final virtual viewpoint image. The vertical processing unit is
At least a search range determination unit that determines a search range in the horizontal direction from the horizontal disparity vector,
The horizontal search range is limited to the search range, and a search is made for a block or pixel that matches or is similar between the first input image or the second input image and the vertical image. A vertical matching block search unit that outputs a vertical disparity vector representing the relationship;
A shift vector calculation unit characterized in that a vector having a horizontal component of 0 among the vertical direction parallax vectors is output as a shift vector;
The virtual viewpoint image generation apparatus according to claim 1, further comprising: a vertical direction interpolation image generation unit that generates a vertical direction interpolation image at the virtual position from the shift vector and the vertical image. The position of the virtual viewpoint is set on a straight line passing through both viewpoints related to the first input image and the second input image,
The vertical image has a straight line passing through the viewpoint of the vertical image and the virtual viewpoint at a position orthogonal to a straight line passing through both viewpoints of the first input image and the second input image. The virtual viewpoint image generation apparatus according to claim 1, wherein the viewpoint is an image set. The vertical processing unit includes a first vertical processing unit and a second vertical processing unit,
The first input image is input to the first vertical processing unit, the second input image is input to the second vertical processing unit,
The interpolated image integration unit includes the horizontal direction interpolated image, the first vertical direction interpolated image obtained by the first vertical direction processing unit, and the second vertical direction obtained by the second vertical direction processing unit. The virtual viewpoint image generation apparatus according to claim 1, wherein the final virtual viewpoint image is generated by integrating the interpolation image. When the second input image is referred to from the first input image of interest, the horizontal direction parallax vector is v, the parameter indicating the position of the viewpoint to be interpolated is c, the predetermined parameter of zero or more is r, the first When the sign parameter representing the relative positional relationship when the position of the object of the input image is viewed from the position of the object of the vertical image at the virtual viewpoint is s,
The virtual viewpoint image generation device according to claim 2, wherein the search range determination unit calculates the search range α according to the following equation (A).
α∈ [scv−r, scv + r] (A) The horizontal disparity vector when the second input image is referred to from the first input image of interest is v, the horizontal disparity vector of the block or pixel adjacent to the block or pixel of interest is v _k , the neighborhood N is the number of blocks or pixels, c is a parameter representing the virtual viewpoint position to be interpolated, r is a predetermined parameter of 0 or more, and the object position of the first input image is the object position of the vertical image at the virtual viewpoint. If the sign parameter representing the relative positional relationship when viewed from the position is s,
The virtual viewpoint image generation apparatus according to claim 2, wherein the search range determination unit calculates the search range α according to the following equation (B).
α∈ [scv−r, scv + r] ∪ [scv ₁ −r, scv ₁ + r]
∪ [scv ₂ −r, scv ₂ + r]
∪. . .
∪ [scv _k −r, scv _k + r]
∪. . .
∪ [scv _n −r, scv _n + r] (B) The horizontal direction processing unit searches for a block or pixel that matches or is similar between the horizontal direction parallax vector, the first input image, and the second input image, and the horizontal direction related to the searched element Output the degree of direction matching,
The virtual viewpoint image generation according to any one of claims 2 to 4, wherein the search range determination unit determines a horizontal search range from the horizontal direction parallax vector and the horizontal direction matching degree. apparatus. The horizontal disparity vector when the second input image is referred to from the first input image of interest is v, the horizontal disparity vector of the block or pixel adjacent to the block or pixel of interest is v _k , the neighborhood N is the number of blocks or pixels, c is a parameter representing the virtual viewpoint position to be interpolated, r is a predetermined parameter of 0 or more, and the object position of the first input image is the object position of the vertical image at the virtual viewpoint. If the sign parameter representing the relative positional relationship when viewed from the position is s,
The search range determination unit
When the degree of horizontal matching is greater than or equal to a threshold, the search range α is calculated according to equation (C). When the degree of horizontal matching is less than the threshold, the search range α is calculated according to equation (D). The virtual viewpoint image generation apparatus according to claim 2, wherein the virtual viewpoint image generation apparatus is a virtual viewpoint image generation apparatus.
α∈ [scv−r, scv + r] (C)
α∈ [scv ₁ −r, scv ₁ + r] ∪ [scv ₂ −r, scv ₂ + r]
∪. . .
∪ [scv _k −r, scv _k + r]
∪. . .
∪ [scv _n −r, scv _n + r] (D) The horizontal processing unit is
A horizontal direction matching block search unit that searches for a block or pixel that matches or is similar between the first input image and the second input image, and outputs a horizontal direction disparity vector representing the relationship;
An interpolation vector calculation unit for obtaining an interpolation vector from the horizontal parallax vector and the virtual viewpoint position;
A horizontal direction interpolation image generation unit configured to generate a horizontal direction interpolation image from the interpolation vector and the input image of either or both of the first input image and the second input image. The virtual viewpoint image generation device according to claim 1. As the virtual viewpoint, a first virtual viewpoint and a second virtual viewpoint are set,
Generating a virtual viewpoint image for the first virtual viewpoint from the first input image, the second input image, and the first vertical image;
Generating a virtual viewpoint image for the second virtual viewpoint from the first input image, the second input image and the second vertical image;
The first vertical image is provided with a viewpoint above a straight line passing through both viewpoints related to the first input image and the second input image,
The second vertical image has a viewpoint that is provided below a straight line passing through both viewpoints of the first input image and the second input image. The virtual viewpoint image generation device according to any one of claims 9 to 9. Computer
A horizontal direction interpolation image is generated from the first input image and the second input image and the position of the virtual viewpoint, and the first input image and the second input image are identical or A horizontal processing unit that searches for similar blocks or pixels and outputs at least a horizontal disparity vector representing the relationship;
At least a horizontal search range is determined from the horizontal parallax vector, and is taken from a viewpoint shifted in the vertical direction from a straight line passing through both viewpoints related to the first input image and the second input image. At least one vertical direction processing unit that generates a vertical direction interpolation image from a vertical image and either or both of the first input image and the second input image;
A virtual viewpoint image generation program, wherein the horizontal direction interpolation image and the vertical direction interpolation image are integrated to function as an interpolation image integration unit that generates a final virtual viewpoint image.

Images