JP2013030898A - Image transmission method, image transmission apparatus, image sending apparatus, image receiving apparatus, image sending program, and image receiving program - Google Patents

Abstract

The total number of pixels of transmission data is reduced while maintaining the quality of a composite image.
A step of downsampling a multi-viewpoint image based on a reduction ratio and outputting a low resolution image, a step of encoding a low resolution image, a step of encoding a high resolution depth map, and a low resolution image code A step of decoding data, a step of decoding high-resolution depth map code data, and a corresponding point with decimal pixel accuracy at a different viewpoint of each pixel using depth information of each pixel obtained from the high-resolution multi-view depth map The step of setting the correspondence between viewpoints by obtaining the step, the step of obtaining the correspondence between viewpoints by projecting each three-dimensional projection point to another viewpoint, and the decimal pixel value of another viewpoint based on the correspondence between viewpoints Generating a high-resolution image from a low-resolution image by performing upsampling, and arbitrary viewing from the high-resolution image and the high-resolution depth map. And a step of synthesizing an image.
[Selection] Figure 1

Description

  The present invention relates to an image transmission method, an image transmission apparatus, an image transmission apparatus, an image reception apparatus, an image transmission program, and an image reception program.

  As one of the next-generation image media, free viewpoint images that allow viewers to freely operate the viewpoint are attracting attention. A free viewpoint image captures the target scene from various positions and angles by using a number of imaging devices, obtains the light ray information of the scene, and restores the light ray information at an arbitrary viewpoint based on this information. The image seen from is generated.

  In order to generate such an image, a large number of imaging devices must be densely installed in order to acquire all the light ray information in the scene by imaging. In practice, it is necessary to synthesize unacquired light ray information from light ray information obtained from a small number of sparsely arranged imaging devices using some interpolation method.

  As one of the interpolation synthesis methods, there is Depth Image based Rendering (DIBR) in which a virtual viewpoint image is synthesized using multi-viewpoint images and scene depth information estimated therefrom. The depth information is the distance from the camera to the subject in each pixel of the multi-viewpoint image. When considering transmission of a free viewpoint image, it is effective to estimate the depth information on the transmission side and describe and transmit it as a multi-viewpoint gray scale image (depth map). This method reduces the amount of computation on the receiving side and enables more accurate estimation by estimating depth information using a multi-viewpoint image before encoding distortion is superimposed.

  Since image data composed of such a multi-view image and a multi-view depth map has a huge amount of information, a more efficient coding method is essential, and various methods are being studied. However, another thing that must be achieved along with the reduction in code amount is that the total number of pixels of image data needs to be reduced to several times that of a normal single-viewpoint image due to the upper limit of decoder throughput and memory capacity. It has been reported. Therefore, generally, in order to reduce the total number of pixels, a method of down-sampling the multi-view depth map and reducing the number of pixels is used (for example, see Non-Patent Document 1).

Yea S, Vetro A, “VIEW SYNTHESIS PREDICTION FOR RATEOVERHEAD REDUCTION IN FTV”, 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video (2008).

  However, in the method of downsampling the multi-view depth map, the 3D information is lost due to the down-sampling of the depth map, and the accuracy of inter-viewpoint pixel correspondence is lost. There is a problem of lowering. Conversely, if the multi-viewpoint image is downsampled while maintaining the resolution of the depth map, the above problem does not occur because the three-dimensional information is accurately maintained, but the composite image is deteriorated because the quality of the pre-combination image itself is degraded. It can be easily inferred that the quality of the product also decreases.

  The present invention has been made in view of such circumstances, and a free viewpoint image transmission method, an image transmission apparatus, and an image transmission capable of reducing the total number of pixels of transmission data while maintaining the quality of a composite image. An object is to provide an apparatus, an image receiving apparatus, an image transmission program, and an image reception program.

  The present invention is an image transmission method of a free-viewpoint image comprising a high-resolution multi-viewpoint image obtained by photographing the same scene with a plurality of cameras and a high-resolution multi-view depth map in which depth information for each pixel is described as a grayscale image. A reduction rate setting step for setting a reduction rate when downsampling the high-resolution multi-viewpoint image; and downsampling for downsampling the high-resolution multi-viewpoint image and outputting a low-resolution multi-viewpoint image based on the reduction rate An encoding step of encoding the low-resolution multi-view image obtained in the down-sampling step and outputting low-resolution multi-view image code data; and encoding the high-resolution multi-view depth map to generate a high-resolution multi-view Depth map encoding step for outputting depth map code data, and the image encoding step An image decoding step for decoding the obtained low-resolution multi-view image code data, a depth map decoding step for decoding the high-resolution multi-view depth map code data obtained by the depth map encoding step, and the depth map decoding Inter-viewpoint correspondence that sets inter-viewpoint correspondences by finding corresponding points with different pixel precision at different viewpoints of each pixel using depth information of each pixel obtained from the high-resolution multiview depth map obtained in the step An up-sampling step that generates a high-resolution multi-view image from the low-resolution multi-view image obtained in the image decoding step by performing up-sampling while referring to the decimal pixel value of another viewpoint based on the correspondence relationship between the viewpoints; Sampling step, high-resolution multi-viewpoint image obtained in the up-sampling step and the high-resolution And having a free viewpoint image synthesizing step of synthesizing the image of the arbitrary viewpoint from the viewpoint depth map.

  The present invention includes a first filter selection step for selecting a filter from a predetermined filter group used for downsampling, and a second filter for selecting a filter from a predetermined filter group used for upsampling. A selection step, wherein the down-sampling step down-samples the high-resolution multi-viewpoint image using the filter selected in the first filter selection step, and the up-sampling step includes the second sampling step. The low-resolution multi-viewpoint image is up-sampled using the filter selected in the filter selection step.

  The present invention includes an additional information encoding step of encoding information for identifying the filter selected in the first filter selection step and outputting the information as additional information code data, and identifying the filter by decoding the additional information code data An additional information decoding step for outputting information, wherein the second filter selection step selects a filter to be used for upsampling based on information for identifying the filter.

  The present invention further includes a first filter setting step for setting a filter with the highest image restoration efficiency, and a second filter setting step for setting a filter with the highest image restoration efficiency. The sampling step downsamples the high-resolution multi-viewpoint image using the filter set in the first filter setting step, and the upsampling step uses the filter set in the second filter setting step. Then, the low-resolution multi-viewpoint image is upsampled.

  The present invention includes an additional information encoding step for encoding the filter identification information set in the first filter setting step and outputting additional information code data, and decoding the additional information code data to obtain the filter identification information. And an additional information decoding step for outputting, wherein the second filter setting step sets a filter used for the upsampling based on identification information of the filter.

  The present invention is an image transmission apparatus for a free-viewpoint image comprising a high-resolution multi-viewpoint image obtained by photographing the same scene with a plurality of cameras and a high-resolution multi-view depth map in which depth information for each pixel is described as a grayscale image. Reduction ratio setting means for setting a reduction ratio when down-sampling the high-resolution multi-viewpoint image; down-sampling for down-sampling the high-resolution multi-viewpoint image based on the reduction ratio and outputting a low-resolution multi-viewpoint image Means, image encoding means for encoding the low-resolution multi-view image obtained by the down-sampling means and outputting low-resolution multi-view image code data, and encoding the high-resolution multi-view depth map for high-resolution multi-view Depth map encoding means for outputting depth map code data, and low resolution multi-value obtained by the image encoding means. Image decoding means for decoding point image code data, depth map decoding means for decoding high-resolution multi-view depth map code data obtained by the depth map coding means, and high resolution obtained by the depth map decoding means An inter-viewpoint correspondence setting means for setting the inter-viewpoint correspondence by obtaining corresponding points with decimal pixel accuracy at different viewpoints of each pixel using depth information of each pixel obtained from a multi-view depth map; Up-sampling means for generating a high-resolution multi-view image from a low-resolution multi-view image obtained by the image decoding means by performing up-sampling while referring to a decimal pixel value of another viewpoint based on the inter-correspondence relationship, and the up-sampling Free viewpoint image that synthesizes an arbitrary viewpoint image from the high resolution multi-view image obtained by the means and the high-resolution multi-view depth map Characterized in that it comprises a combining means.

  The present invention provides a first filter selecting means for selecting a filter from a predetermined filter group used for downsampling, and a second filter for selecting a filter from a predetermined filter group used for upsampling. Selecting means, wherein the downsampling means downsamples the high-resolution multi-viewpoint image using the filter selected by the first filter selecting means, and the upsampling means includes the second filter. The low-resolution multi-viewpoint image is upsampled using the filter selected by the selection means.

  The present invention further includes a first filter setting unit that sets a filter that maximizes the image restoration efficiency, and a second filter setting unit that sets a filter that maximizes the image restoration efficiency. The means downsamples the high-resolution multi-viewpoint image using the filter set by the first filter setting means, and the upsampling means uses the filter set by the second filter setting means. The low-resolution multi-viewpoint image is upsampled.

  The present invention is an image transmission device that transmits a high-resolution multi-viewpoint image obtained by photographing a same scene with a plurality of cameras and a high-resolution multi-view depth map in which depth information for each pixel is described as a grayscale image. A reduction ratio setting means for setting a reduction ratio when downsampling the high-resolution multi-viewpoint image; and downsampling the high-resolution multi-viewpoint image based on the reduction ratio and outputting a low-resolution multi-viewpoint image Down-sampling means, image encoding means for encoding the low-resolution multi-view image obtained by the down-sampling means and outputting low-resolution multi-view image code data, and encoding the high-resolution multi-view depth map and high resolution And depth map encoding means for outputting multi-view depth map code data.

  The present invention provides a high-resolution multi-viewpoint image obtained by photographing the same scene with a plurality of cameras and a free-viewpoint image composed of a high-resolution multi-view depth map in which depth information for each pixel is described as a grayscale image. An image receiving apparatus for receiving transmitted free-viewpoint image code data, obtained by the image decoding means for decoding low-resolution multi-viewpoint image code data obtained by the image coding means, and the depth map coding means. Depth map decoding means for decoding the obtained high resolution multi-view depth map code data, and another viewpoint for each pixel using depth information of each pixel obtained from the high-resolution multi-view depth map obtained by the depth map decoding means Inter-viewpoint correspondence setting means for setting a correspondence relationship between viewpoints by obtaining corresponding points with decimal pixel accuracy in the above, and correspondence between the viewpoints An upsampling unit that generates a high-resolution multi-viewpoint image from a low-resolution multi-viewpoint image obtained by the image decoding unit by performing upsampling while referring to a fractional pixel value of another viewpoint based on the relationship, and the upsampling unit And a free-viewpoint image synthesizing unit that synthesizes an arbitrary-viewpoint image from the obtained high-resolution multi-viewpoint image and the high-resolution multi-viewpoint depth map.

  The present invention relates to an image transmission apparatus for transmitting a free viewpoint image composed of a high resolution multi-viewpoint image obtained by photographing the same scene with a plurality of cameras and a high-resolution multiview depth map in which depth information for each pixel is described as a grayscale image. An image transmission program for causing the computer to perform image transmission processing, a reduction ratio setting step for setting a reduction ratio when down-sampling the high-resolution multi-viewpoint image, and the high-resolution multi-viewpoint based on the reduction ratio A downsampling step of downsampling the image and outputting a low-resolution multi-viewpoint image; an image encoding step of encoding the low-resolution multi-viewpoint image obtained in the downsampling step and outputting low-resolution multi-viewpoint image code data; The high-resolution multi-view depth map is encoded and the high-resolution multi-view depth map code is decoded. Characterized in that to perform the depth map coding step of outputting the data.

  The present invention provides a high-resolution multi-viewpoint image obtained by photographing the same scene with a plurality of cameras and a free-viewpoint image composed of a high-resolution multi-view depth map in which depth information for each pixel is described as a grayscale image. An image reception program for causing a computer on an image receiving apparatus that receives transmitted free viewpoint image code data to perform image reception processing, which decodes low-resolution multi-view image code data obtained by the image encoding step Obtained from an image decoding step, a depth map decoding step for decoding high-resolution multi-view depth map code data obtained by the depth map encoding step, and a high-resolution multi-view depth map obtained by the depth map decoding step Corresponding point with decimal pixel precision from different viewpoint of each pixel using depth information of each pixel A low-resolution obtained in the image decoding step by performing the up-sampling while referring to the fractional pixel value of another viewpoint based on the inter-viewpoint correspondence, and setting the inter-viewpoint correspondence by setting An up-sampling step for generating a high-resolution multi-view image from a multi-view image, and a free-view image synthesis for synthesizing an arbitrary viewpoint image from the high-resolution multi-view image obtained in the up-sampling step and the high-resolution multi-view depth map Steps are performed.

  According to the present invention, a high-resolution multi-viewpoint image is downsampled and transmitted as a low-resolution multi-viewpoint image together with a high-resolution depth map, and pixel information between viewpoints is obtained using three-dimensional information obtained from the high-resolution depth map after reception. By referring to each other, a low-resolution multi-viewpoint image can be up-sampled and used for free-viewpoint image synthesis. As a result, the total number of pixels of transmission data can be reduced while maintaining the quality of the synthesized image. The effect is obtained.

It is a block diagram which shows the structure of the 1st Embodiment of this invention. 3 is a flowchart showing a processing operation of the image transmission apparatus 200 shown in FIG. It is a flowchart which shows the processing operation of the image receiver 300 shown in FIG. It is a block diagram which shows the structure of the 2nd Embodiment of this invention. 5 is a flowchart showing a processing operation of the image transmission device 200 shown in FIG. 5 is a flowchart showing a processing operation of the image receiving apparatus 300 shown in FIG. It is a block diagram which shows the structure of the 3rd Embodiment of this invention. It is a flowchart which shows the processing operation of the image transmission apparatus 200 shown in FIG. It is a flowchart which shows the processing operation of the image receiver 300 shown in FIG.

<First Embodiment>
Hereinafter, an image transmission apparatus according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. The image transmission device 100 includes an image transmission device 200 and an image reception device 300. The image transmission apparatus 200 includes a multi-view image input unit 201, a multi-view depth map input unit 202, a reduction rate setting unit 203, a downsampling unit 204, a depth map encoding unit 205, an image encoding unit 206, and a multiplexing unit 207. Prepare.

  The multi-view image input unit 201 inputs a high-resolution multi-view image obtained by capturing a scene for generating a free viewpoint image from multiple viewpoints. The multi-view depth map input unit 202 inputs a high-resolution multi-view depth map. The reduction rate setting unit 203 determines the reduction rate of the multi-viewpoint image. The downsampling unit 204 generates a low-resolution multi-viewpoint image by reducing the input high-resolution multi-viewpoint image at a set reduction rate. The depth map encoding unit 205 encodes the input high-resolution multi-view depth map to generate high-resolution multi-view depth map code data. The image encoding unit 206 encodes the low-resolution multi-view image and generates low-resolution multi-view image code data. The multiplexing unit 207 multiplexes the low resolution multi-view image code data and the high resolution multi-view depth map code data to generate free viewpoint image code data.

  Also, the image receiving apparatus 300 includes a code data input unit 301, a demultiplexing unit 302, an image decoding unit 303, a depth map decoding unit 304, a 3D projection point setting unit 305, an inter-viewpoint correspondence setting unit 306, and an upsampling unit. 307, a virtual viewpoint setting unit 308, and a free viewpoint image composition unit 309.

  The code data input unit 301 inputs free viewpoint image code data. The demultiplexer 302 demultiplexes the input code data into low-resolution multi-view image code data and high-resolution multi-view depth map code data. The image decoding unit 303 decodes the low-resolution multi-view image code data to obtain a low-resolution multi-view image. The depth map decoding unit 304 decodes the high-resolution multi-view depth map code data to obtain a high-resolution multi-view depth map. The three-dimensional projection point setting unit 305 calculates the projection point on each pixel in the high-resolution multi-viewpoint image. The inter-viewpoint corresponding point setting unit 306 obtains the inter-viewpoint pixel correspondence of the high-resolution multi-viewpoint image with the decimal pixel accuracy using the three-dimensional projection point of each pixel of each viewpoint. The up-sampling unit 307 up-samples the low-resolution multi-viewpoint image using the obtained inter-viewpoint pixel correspondence to obtain a high-resolution multi-viewpoint image. The virtual viewpoint setting unit 308 sets a virtual viewpoint for generating a free viewpoint image. The free viewpoint image combining unit 309 combines an image obtained by observing a scene from the set virtual viewpoint as a free viewpoint image using the high resolution multi-view image and the high resolution depth map.

  Next, the processing operation of the image transmission apparatus 200 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the processing operation of the image transmission apparatus 200 shown in FIG. FIG. 2 shows a multi-viewpoint in which the image transmission apparatus 200 shown in FIG. 1 records a multi-viewpoint image obtained by photographing the same scene from a plurality of viewpoints and depth information of the scene at each viewpoint, which are necessary for free viewpoint image synthesis. The processing operation at the time of transmitting the depth map is shown.

  First, the multi-view image input unit 201 inputs a high-resolution multi-view image, and the multi-view depth map input unit 202 inputs a high-resolution multi-view depth map (step S1). Next, the reduction ratio setting unit 203 sets the reduction ratio of the multi-viewpoint image (step S2). This reduction ratio may be a reduction ratio of the same aspect ratio, or an independent reduction ratio may be set for each of the vertical and horizontal directions. In addition, a fixed value may be used as the reduction ratio value itself, or an appropriate value may be calculated from the resolution of the multi-viewpoint image, the number of viewpoints, and the like. If the value cannot be obtained on the image receiving apparatus 300 side, the value may be encoded and transmitted on the transmitting side.

  Next, the downsampling unit 204 reduces the high-resolution multi-viewpoint image with the set reduction rate, and generates a low-resolution multi-viewpoint image (step S3). Subsequently, the depth map encoding unit 205 encodes the high-resolution multi-view depth map to generate high-resolution multi-view depth map code data, and the image encoding unit 206 encodes the low-resolution multi-view image to generate the low-resolution multi-view image code data. (Step S4). Any method may be used for encoding.

  Next, the multiplexing unit 207 multiplexes the low-resolution multi-view image code data and the high-resolution multi-view depth map code data as free-view image code data (step S5), and outputs the multiplexed code data (step S5). S6). This output method may be transmitted over a network or may be recorded on some recording medium.

  Next, the processing operation of the image receiving apparatus 300 shown in FIG. 1 will be described with reference to FIG. FIG. 3 is a flowchart showing the processing operation of the image receiving apparatus 300 shown in FIG. FIG. 3 shows that the image receiving device 300 receives the low-resolution multi-view image and the high-resolution multi-view depth map transmitted by the image transmitting device 200, and increases the resolution of the multi-view image using the high-resolution multi-view depth map. It shows the processing operation when a free viewpoint image is synthesized using them after sampling.

  First, the code data input unit 301 inputs free viewpoint image code data (step S11). The input at this time may be transmission by a network or input from some recording medium. The input free viewpoint image code data is transferred to the demultiplexing unit 302, and the demultiplexing unit 302 demultiplexes the low-resolution multi-view image code data and the high-resolution multi-view depth map code data (steps). S12). The image decoding unit 303 decodes the low-resolution multi-view image code data and outputs a low-resolution multi-view image, and the depth map decoding unit 304 decodes the high-resolution multi-view depth map code data and outputs a high-resolution multi-view depth map (Step S13). The decoding method at this time is the same as that used for encoding on the transmitting apparatus side.

  Next, the three-dimensional projection point setting unit 305 calculates the projection point on each pixel in the high-resolution multi-viewpoint image based on the depth information obtained from the high-resolution multi-viewpoint depth map and the camera parameters of each viewpoint. (Step S14). The camera parameters for each viewpoint may be obtained from the same value as that on the transmission side, and may be estimated from the image, or may be encoded and transmitted on the transmission side.

  Next, the inter-viewpoint corresponding point setting unit 306 obtains the inter-viewpoint pixel correspondence of the high-resolution multi-viewpoint image with decimal pixel accuracy (step S15). In this case, the position of the decimal pixel accuracy on the projection plane when the three-dimensional projection point of each pixel is photographed by another viewpoint camera is the position of the corresponding point at that viewpoint.

Next, the upsampling unit 307 upsamples the low-resolution multi-viewpoint image using the inter-viewpoint pixel correspondence obtained in step S15 and outputs the high-resolution multi-viewpoint image (step S16). The up-sampling process is represented by the following function U _v.

Here, HR _v is a high-resolution image at the viewpoint v, LR _k is a low-resolution image at the viewpoint k, and f _{k, l} are on the low-resolution image at the viewpoint l for each pixel of the high-resolution image at the viewpoint k obtained in step S15. Represents a function indicating the corresponding position of the decimal pixel accuracy.

Specifically, the upsampling process is performed by referring to the pixel value at the decimal pixel position of the other viewpoint obtained based on the inter-viewpoint pixel correspondence obtained in step S15. The upsampling process u _{v at} this time is expressed by the following equation.

Here, the number of viewpoints of the multi-view image to n, the corresponding point in the low-resolution image LR _l viewpoint l for pixel p _k in the high-resolution image of the view k and C _{l (p k).} Also, the pixel value LR ₁ [Cl (p _v )] at the decimal pixel position is obtained from the values of surrounding integer pixels using an arbitrary interpolation method. For example, a method for obtaining by linear interpolation from four neighboring pixels can be applied. Alternatively, referring to the value of the high-resolution depth map at the viewpoint, and when the depth value differs greatly between neighboring pixels such as the object boundary, the depth value is within a certain range based on the depth value of the nearest pixel. A method of performing interpolation using a neighboring pixel group is also conceivable. Note that the depth value for the pixel of the upsample target viewpoint may be used instead of the depth value of the nearest pixel. However, when the depth values indicating the same three-dimensional position are different between the viewpoints, it is necessary to convert the depth values for the pixels of the upsampling target viewpoint into the depth values at the viewpoint on which interpolation processing is performed.

This time as a function u _v may be an arbitrary function. For example, a weighted average represented by the following formula may be used.

At this time, if all the weights w _k (p _v ) are set to 1, the processing is based on simple averaging. Further, as the weight w _k (p _v ), a weight that increases as the distance between the up-sampling target viewpoint v and the viewpoint k may be used, or the nearest integer pixel position of C _k (p _v ) A weight that becomes a larger value as the distance from the nearer may be used, or a weight obtained by combining these two weights may be used.

Further, the weight may be obtained so as to minimize the error in the low resolution image when the high resolution image is obtained.

Note that argmin is a function that gives the minimum value of a given function and returns a parameter given at the bottom. D _v represents the downsampling process in step S3, and p _v ′ represents an integer pixel position on the low-resolution image corresponding to p _v . Here, the square of the difference is used as the error, but other error measures such as the absolute value of the difference may be used. In the above formula, only the error at the viewpoint v is considered, but the total value, maximum value, and variance of errors in the low resolution images of all viewpoints when a high resolution image is obtained may be considered. Furthermore, not only the error but also the evaluation value of the image quality in the high resolution image may be added to the error scale. A total variation norm or the like can be used as the evaluation value of the image quality.

  Further, super-resolution may be used as a method for performing upsampling by minimizing errors in low-resolution images of all viewpoints while taking into consideration the image quality of high-resolution images. The details of the super-resolution technique are described in “Literature: Farsiu S, Robinson MD, Elad M, Milanfar,“ Fast and robust multiframe super resolution ”, IEEE TRANSACTIONS ON IMAGE PROCESSING, Vol.13, No10 (2010).” Has been.

  Next, the virtual viewpoint setting unit 308 sets a virtual viewpoint for generating a free viewpoint image (step S17). This may be an external input such as a setting by the user or a fixed value. The free viewpoint image composition unit 309 then composes an image obtained by observing the scene from the set virtual viewpoint as a free viewpoint image using the high resolution multi-viewpoint image and the high resolution depth map (step S18). For detailed methods, refer to “Literature: Mori Y, Fukushima N, Fujii T, Tanimoto M,“ View generation with 3D warping using depth information for FTV ”, Image Communication, Vol. 24, No. 1-2 (2009).” Have been described. At this time, it is possible to reduce the amount of calculation using the information of the three-dimensional projection point of each viewpoint obtained in step S14. Finally, the free viewpoint image synthesis unit 309 outputs the synthesized image (step S19).

<Second Embodiment>
Next, an image transmission apparatus according to a second embodiment of the present invention will be described. FIG. 4 is a block diagram showing a configuration of an image transmission apparatus according to the second embodiment of the present invention. In FIG. 4, the same parts as those of the image transmission apparatus shown in FIG. The image transmission apparatus shown in FIG. 4 is different from the image transmission apparatus shown in FIG. 1 in that a filter selection unit 208 for selecting a filter used for downsampling is provided in place of the reduction ratio setting unit 203. A filter selection unit 310 that selects a filter used for upsampling is provided before the sampling unit 307.

  Next, the processing operation of the image transmission apparatus 200 shown in FIG. 4 will be described with reference to FIG. FIG. 5 is a flowchart showing the processing operation of the image transmission apparatus 200 shown in FIG. In FIG. 5, the same parts as those in the processing operation shown in FIG. The processing operation shown in FIG. 5 is different from the processing operation shown in FIG. 2 in that processing for selecting a filter to be used in the downsampling unit 204 is performed.

  First, the multi-view image input unit 201 inputs a high-resolution multi-view image, and the multi-view depth map input unit 202 inputs a high-resolution multi-view depth map (step S1). Next, the filter selection unit 208 selects a filter used for downsampling of the multi-viewpoint image from filters generally used for downsampling (step S2a). An appropriate one may be selected depending on the image content and the number of viewpoints so that the restoration efficiency on the receiving side is as high as possible, or may be selected by an external input such as a user input. As the selection method, for example, a method of performing downsampling and upsampling using each of the filters designed in advance and selecting the one having the highest restoration rate compared with the original image can be considered. Further, information indicating which filter is selected may be encoded and transmitted as additional information. Thereafter, the down-sampling unit 204 reduces the high-resolution multi-view image using the selected down-sampling filter, and generates a low-resolution multi-view image (step S3).

  Next, the depth map encoding unit 205 encodes the high-resolution multi-view depth map into high-resolution multi-view depth map code data, and the image encoding unit 206 encodes the low-resolution multi-view image to generate the low-resolution multi-view image code data. (Step S4). Subsequently, the multiplexing unit 207 multiplexes the low-resolution multi-view image code data and the high-resolution multi-view depth map code data as free-view image code data (step S5), and outputs the multiplexed code data (step S5). S6).

  Next, the processing operation of the image receiving apparatus 300 shown in FIG. 4 will be described with reference to FIG. FIG. 6 is a flowchart showing the processing operation of the image receiving apparatus 300 shown in FIG. In FIG. 6, the same reference numerals are given to the same parts as the processing operation shown in FIG. The processing operation shown in FIG. 6 is different from the processing operation shown in FIG. 3 in that processing for selecting a filter to be used in the upsampling unit 307 is performed.

  First, the code data input unit 301 inputs free viewpoint image code data (step S11). The input free viewpoint image code data is transferred to the demultiplexing unit 302, and the demultiplexing unit 302 demultiplexes the low-resolution multi-view image code data and the high-resolution multi-view depth map code data (steps). S12). The image decoding unit 303 decodes the low-resolution multi-view image code data and outputs a low-resolution multi-view image, and the depth map decoding unit 304 decodes the high-resolution multi-view depth map code data and outputs a high-resolution multi-view depth map (Step S13).

  Next, the three-dimensional projection point setting unit 305 calculates the projection point on each pixel in the high-resolution multi-viewpoint image based on the depth information obtained from the high-resolution multi-viewpoint depth map and the camera parameters of each viewpoint. (Step S14). Subsequently, the inter-viewpoint corresponding point setting unit 306 obtains the inter-viewpoint pixel correspondence of the high-resolution multi-viewpoint image with decimal pixel accuracy (step S15).

  Next, the filter selection unit 310 selects a filter used for upsampling of the multi-viewpoint image (step S15a). This needs to be paired with the downsampling filter used on the transmission side, but the selection method itself may be selected according to the image content and the number of viewpoints, or input from the outside such as user input You may choose by. Further, information on which filter is used on the transmission side may be encoded and transmitted as additional information.

  Next, the up-sampling unit 307 up-samples the low-resolution multi-viewpoint image by referring to the pixel value of the other viewpoint based on the inter-viewpoint pixel correspondence relationship to obtain a high-resolution other-viewpoint image (step S16). The upsampling filter uses the selected filter.

  Next, the virtual viewpoint setting unit 308 sets a virtual viewpoint for generating a free viewpoint image (step S17). The free viewpoint image composition unit 309 then composes an image obtained by observing the scene from the set virtual viewpoint as a free viewpoint image using the high resolution multi-viewpoint image and the high resolution depth map (step S18). Finally, the free viewpoint image synthesis unit 309 outputs the synthesized image (step S19).

<Third Embodiment>
Next, an image transmission apparatus according to a third embodiment of the present invention will be described. FIG. 7 is a block diagram showing a configuration of an image transmission apparatus according to the third embodiment of the present invention. 7, the same parts as those of the image transmission apparatus shown in FIG. 4 are denoted by the same reference numerals, and the description thereof is omitted. The image transmission apparatus shown in FIG. 7 is different from the image transmission apparatus shown in FIG. 4 in that, in the image transmission apparatus 200, instead of the filter selection unit 208, a filter setting unit 209 that sets a filter used for downsampling, An additional information encoding unit 210 for encoding is provided. Further, in the image receiving device 300, an additional information decoding unit 311 and a filter setting unit 312 are provided instead of the filter selection unit 310.

  Next, the processing operation of the image transmission device 200 shown in FIG. 7 will be described with reference to FIG. FIG. 8 is a flowchart showing the processing operation of the image transmission apparatus 200 shown in FIG. In FIG. 8, the same parts as those in the processing operation shown in FIG. The processing operation shown in FIG. 8 is different from the processing operation shown in FIG. 5 in that instead of selecting a filter to be used in the downsampling unit 204, an arbitrary filter is set, encoded, and transmitted.

  First, the multi-view image input unit 201 inputs a high-resolution multi-view image, and the multi-view depth map input unit 202 inputs a high-resolution multi-view depth map (step S1). Next, the filter setting unit 209 sets a filter used for downsampling on the transmission side and upsampling on the reception side of the multi-viewpoint image (step S2b). This may be set so that the restoration efficiency on the receiving side is the highest, depending on the number of viewpoints and the characteristics of each viewpoint image / depth map. For example, a multi-viewpoint image captured by a horizontal N × vertical 1 camera array is used with a downsampling filter that reduces to 1 / N × vertical 1 so that the sampling efficiency for all viewpoints is increased. A method is conceivable. Alternatively, based on the three-dimensional information obtained from the depth map, a method of determining the downsampling rate for a partial region in the image according to the number of viewpoints that can observe the region can be considered. For example, an area that can be observed only from one viewpoint is not downsampled, and an area that can be observed from all viewpoints is downsampled to 1 / N. The upsampling filter may be set based on the same interpolation method as the downsampling filter, or may be set based on another standard. For example, a method of determining an upsampling filter using an average square error minimization method for an image downsampled by a certain downsampling filter is also conceivable. In addition, the same filter may be set for all viewpoints, or may be set for each viewpoint. Thereafter, the down-sampling unit 204 reduces the high-resolution multi-view image using the set down-sampling filter, and generates a low-resolution multi-view image (step S3).

  Next, the additional information encoding unit 210 encodes the set upsampling filter to generate filter code data (step S3a). Next, the depth map encoding unit 205 encodes the high-resolution multi-view depth map into high-resolution multi-view depth map code data, and the image encoding unit 206 encodes the low-resolution multi-view image to generate the low-resolution multi-view image code data. (Step S4). Then, the multiplexing unit 207 multiplexes the low resolution multi-view image code data, the high resolution multi-view depth map code data, and the filter code data to generate free viewpoint image code data (step S5), and the multiplexed code data is generated. Output (step S6).

  Next, the processing operation of the image receiving apparatus 300 shown in FIG. 7 will be described with reference to FIG. FIG. 9 is a flowchart showing the processing operation of the image receiving apparatus 300 shown in FIG. 9, parts that are the same as the processing operations shown in FIG. 6 are given the same reference numerals. The processing operation shown in FIG. 9 is different from the processing operation shown in FIG. 6 in that instead of selecting a filter to be used in the upsampling unit 307, a process of decoding the filter code data received together with the image data and using it for upsampling. It is a point to do.

  First, the code data input unit 301 inputs free viewpoint image code data (step S11). The input free viewpoint image code data is transferred to the demultiplexing unit 302. The demultiplexing unit 302 demultiplexes the low-resolution multi-view image code data, the high-resolution multi-view depth map code data, and the filter code data. (Step S12). The image decoding unit 303 decodes the low-resolution multi-view image code data and outputs a low-resolution multi-view image, and the depth map decoding unit 304 decodes the high-resolution multi-view depth map code data and outputs a high-resolution multi-view depth map (Step S13).

  Next, the three-dimensional projection point setting unit 305 calculates the projection point on each pixel in the high-resolution multi-viewpoint image based on the depth information obtained from the high-resolution multi-viewpoint depth map and the camera parameters of each viewpoint. (Step S14). Subsequently, the inter-viewpoint corresponding point setting unit 306 obtains the inter-viewpoint pixel correspondence of the high-resolution multi-viewpoint image with decimal pixel accuracy (step S15).

  Next, the additional information decoding unit 311 decodes the filter code data to generate an upsampling filter (step S15b). The filter setting unit 312 sets the decoded upsampling filter. Subsequently, the up-sampling unit 307 up-samples the low-resolution multi-viewpoint image by referring to the pixel values of the other viewpoints based on the inter-viewpoint pixel correspondence relationship to obtain a high-resolution other-viewpoint image (step S16).

  Next, the virtual viewpoint setting unit 308 sets a virtual viewpoint for generating a free viewpoint image (step S17). The free viewpoint image composition unit 309 then composes an image obtained by observing the scene from the set virtual viewpoint as a free viewpoint image using the high resolution multi-viewpoint image and the high resolution depth map (step S18). Finally, the free viewpoint image synthesis unit 309 outputs the synthesized image (step S19).

  In the above description, an example in which the output is a free viewpoint image has been described. However, a multi-viewpoint image that has been upsampled without performing free-viewpoint image synthesis may be used as an output to be used as a multi-viewpoint image transmission apparatus. Good. Moreover, you may make it utilize as a free viewpoint video transmission apparatus by applying with respect to each flame | frame of a moving image.

  In the above description, an example has been described in which an upsampling filter is encoded and transmitted, and upsampling is performed using the upsampling filter, but instead a downsampling filter is encoded and transmitted and used on the receiving side. An upsampling process like Formula (4) may be performed, and a super-resolution process may be performed. When the upsampling filter is used internally in the super-resolution processing, it may be obtained from the filter coefficient of the downsampling filter, or both the downsampling filter and the upsampling filter may be encoded and transmitted.

  Further, although the example in which the up-sampling of the multi-viewpoint image is performed independently by the image receiving apparatus 300 has been described, the image transmitting apparatus 200 may specify the parameters used for the up-sampling, encode them, and transmit them. Since the image transmission apparatus 200 can refer to the high-resolution multi-viewpoint image before downsampling, the image transmission device 200 compares the image quality while performing a trial upsampling on the downsampled multi-viewpoint image, and an appropriate upsampling parameter is set. Can be set.

In the above description, an example has been described in which a three-dimensional projection point is calculated from a high-resolution depth map and a corresponding point between viewpoints is obtained using this. However, as another method, a homography matrix is obtained in advance. You may do it. When the point v _ω existing on the plane is v ₁ and v ₂ in the coordinate systems of the first camera and the second camera, the normal vector of the plane is n, and the distance from the plane to the second camera And d, respectively, with respect to the coordinates m ₁ and m ₂ on the image, using the respective projection transformation matrices A ₁ and A ₂ ,
sm ₁ = Hm ₂ (5)
H = A ₁ ⁻¹ (R + tn ^T / d) A ₂ (6)
Is established. This transformation can be applied to all points on the same plane, and this 3 × 3 matrix H is called a homography matrix.

  When the multi-view depth map is a grayscale image with 8 bits = 256 gradations, the depth level is also 256 gradations, so 256 homography transformation matrices from one viewpoint to another viewpoint are calculated for each depth level. By doing so, it is possible to expect a significant reduction in the amount of calculation especially when dealing with high-resolution images.

  As described above, in order to reduce the total number of pixels of free viewpoint image data consisting of multi-view images and multi-view depth maps that maintain the quality of the composite image, the total number of pixels can be reduced by down-sampling the multi-view images. The image quality is restored by performing upsampling using a multi-view depth map in the image receiving apparatus. By having a multi-view depth map with high resolution and high accuracy, the inter-viewpoint pixel correspondences of multi-viewpoint images can be obtained with decimal pixel accuracy, and accurate upsampling can be performed while referring to pixels of other viewpoints using the correspondences. Can be done.

  1, 4, and 7, the program for realizing the functions of the image transmission device 200 and the image reception device 300 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is recorded on the computer. The free viewpoint image transmission process may be performed by causing the system to read and execute. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  Although the embodiments of the present invention have been described with reference to the drawings, the above embodiments are merely examples of the present invention, and it is clear that the present invention is not limited to the above embodiments. It is. Accordingly, additions, omissions, substitutions, and other modifications of components may be made without departing from the spirit and scope of the present invention.

  The present invention can be applied to applications in which it is essential to perform image transmission by reducing the total number of pixels of transmission data while maintaining the quality of the composite image.

  201 ... multi-viewpoint image input unit, 202 ... multi-viewpoint depth map input unit, 203 ... reduction rate setting unit, 204 ... downsampling unit, 205 ... depth map encoding unit, 206 ..Image encoding unit, 207... Multiplexing unit, 208... Filter selection unit, 209... Filter setting unit, 210 .. Additional information encoding unit, 301. 302 ... Demultiplexing unit 303 ... Image decoding unit 304 ... Depth map decoding unit 305 ... Three-dimensional projection point setting unit 306 ... Inter-viewpoint correspondence setting unit 307 ..Upsampling unit, 308... Virtual viewpoint setting unit, 309... Free viewpoint image synthesis unit, 310... Filter selection unit, 311.

Claims (12)

An image transmission method of a free-viewpoint image composed of a high-resolution multi-viewpoint image in which the same scene is captured by a plurality of cameras and a high-resolution multi-viewpoint depth map describing depth information for each pixel as a grayscale image,
A reduction ratio setting step for setting a reduction ratio when downsampling the high-resolution multi-viewpoint image;
A downsampling step of downsampling the high-resolution multi-viewpoint image based on the reduction ratio and outputting a low-resolution multi-viewpoint image;
An image encoding step of encoding the low resolution multi-view image obtained in the down-sampling step and outputting low-resolution multi-view image code data;
A depth map encoding step for encoding the high-resolution multi-view depth map and outputting high-resolution multi-view depth map code data;
An image decoding step of decoding the low-resolution multi-view image code data obtained by the image encoding step;
Depth map decoding step for decoding high-resolution multi-view depth map code data obtained by the depth map encoding step;
Setting the correspondence between viewpoints by obtaining corresponding points with different pixel accuracy at different viewpoints of each pixel using the depth information of each pixel obtained from the high-resolution multi-view depth map obtained at the depth map decoding step A correspondence setting step between viewpoints,
An upsampling step of generating a high-resolution multi-viewpoint image from the low-resolution multi-viewpoint image obtained by performing the upsampling while referring to the decimal pixel value of another viewpoint based on the correspondence relationship between the viewpoints;
An image transmission method comprising: a high-resolution multi-view image obtained in the up-sampling step; and a free-viewpoint image synthesis step of synthesizing an arbitrary viewpoint image from the high-resolution multi-view depth map. A first filter selection step of selecting a filter from a predetermined filter group used for downsampling;
A second filter selection step of selecting a filter from a predetermined group of filters used for upsampling,
The down-sampling step down-samples the high-resolution multi-viewpoint image using the filter selected in the first filter selection step;
The image transmission method according to claim 1, wherein the up-sampling step up-samples the low-resolution multi-viewpoint image using the filter selected in the second filter selection step. An additional information encoding step of encoding information for identifying the filter selected in the first filter selection step and outputting the information as additional information code data;
An additional information decoding step of decoding the additional information code data and outputting information for identifying the filter;
The image transmission method according to claim 2, wherein the second filter selection step selects a filter used for upsampling based on information for identifying the filter. A first filter setting step for setting a filter with the highest image restoration efficiency;
And a second filter setting step for setting a filter with the highest image restoration efficiency,
The downsampling step downsamples the high-resolution multi-viewpoint image using the filter set in the first filter setting step,
The image transmission method according to claim 1, wherein the up-sampling step up-samples the low-resolution multi-viewpoint image using the filter set in the second filter setting step. An additional information encoding step of encoding identification information of the filter set in the first filter setting step and outputting additional information code data;
An additional information decoding step of decoding the additional information code data and outputting filter identification information;
5. The image transmission method according to claim 4, wherein the second filter setting step sets a filter used for the upsampling based on identification information of the filter. An image transmission device for a free-viewpoint image composed of a high-resolution multi-viewpoint image in which the same scene is captured by a plurality of cameras and a high-resolution multi-view depth map describing depth information for each pixel as a grayscale image,
Reduction ratio setting means for setting a reduction ratio when down-sampling the high-resolution multi-viewpoint image;
Down-sampling means for down-sampling the high-resolution multi-view image based on the reduction ratio and outputting a low-resolution multi-view image;
Image encoding means for encoding the low-resolution multi-view image obtained by the down-sampling means and outputting low-resolution multi-view image code data;
Depth map encoding means for encoding the high resolution multi-view depth map and outputting high-resolution multi-view depth map code data;
Image decoding means for decoding low-resolution multi-view image code data obtained by the image encoding means;
Depth map decoding means for decoding high resolution multi-view depth map code data obtained by the depth map encoding means;
Setting the correspondence between viewpoints by determining the corresponding points with the decimal pixel accuracy in different viewpoints of each pixel using the depth information of each pixel obtained from the high-resolution multi-view depth map obtained by the depth map decoding means A correspondence setting means between viewpoints,
Up-sampling means for generating a high-resolution multi-view image from the low-resolution multi-view image obtained by the image decoding means by performing up-sampling while referring to the decimal pixel value of another viewpoint based on the correspondence relationship between the viewpoints;
An image transmission apparatus comprising: a high-resolution multi-view image obtained by the up-sampling unit; and a free-viewpoint image synthesizing unit that synthesizes an arbitrary viewpoint image from the high-resolution multi-view depth map. First filter selection means for selecting a filter from a predetermined filter group used for downsampling;
A second filter selection means for selecting a filter from a predetermined filter group used for upsampling;
The down-sampling means down-samples the high-resolution multi-viewpoint image using the filter selected by the first filter selection means;
The image transmission apparatus according to claim 6, wherein the up-sampling unit up-samples the low-resolution multi-viewpoint image using a filter selected by the second filter selection unit. First filter setting means for setting a filter having the highest image restoration efficiency;
A second filter setting means for setting a filter with the highest image restoration efficiency,
The down-sampling means down-samples the high-resolution multi-viewpoint image using the filter set by the first filter setting means,
The image transmission apparatus according to claim 6, wherein the upsampling unit upsamples the low-resolution multi-viewpoint image using a filter set by the second filter setting unit. An image transmission device that transmits a high-resolution multi-viewpoint image obtained by photographing a single scene with a plurality of cameras and a high-resolution multi-view depth map in which depth information for each pixel is described as a grayscale image,
Reduction ratio setting means for setting a reduction ratio when down-sampling the high-resolution multi-viewpoint image;
Down-sampling means for down-sampling the high-resolution multi-view image based on the reduction ratio and outputting a low-resolution multi-view image;
Image encoding means for encoding the low-resolution multi-view image obtained by the down-sampling means and outputting low-resolution multi-view image code data;
An image transmission apparatus comprising: depth map encoding means for encoding the high resolution multi-view depth map and outputting high-resolution multi-view depth map code data. Free image sent from an image sending device that sends a high-resolution multi-viewpoint image of the same scene captured by multiple cameras and a high-resolution multi-view depth map that describes the depth information for each pixel as a grayscale image An image receiving device that receives viewpoint image code data,
Image decoding means for decoding low-resolution multi-view image code data obtained by the image encoding means;
Depth map decoding means for decoding high resolution multi-view depth map code data obtained by the depth map encoding means;
Setting the correspondence between viewpoints by determining the corresponding points with the decimal pixel accuracy in different viewpoints of each pixel using the depth information of each pixel obtained from the high-resolution multi-view depth map obtained by the depth map decoding means A correspondence setting means between viewpoints,
Up-sampling means for generating a high-resolution multi-view image from the low-resolution multi-view image obtained by the image decoding means by performing up-sampling while referring to the decimal pixel value of another viewpoint based on the correspondence relationship between the viewpoints;
An image receiving apparatus comprising: a high-resolution multi-view image obtained by the up-sampling unit; and a free-view point image synthesizing unit that synthesizes an arbitrary viewpoint image from the high-resolution multi-view depth map. An image to a computer on an image transmission device that transmits a high-resolution multi-viewpoint image obtained by shooting the same scene with a plurality of cameras and a high-resolution multi-view depth map in which depth information for each pixel is described as a grayscale image An image transmission program for performing transmission processing,
A reduction ratio setting step for setting a reduction ratio when downsampling the high-resolution multi-viewpoint image;
A downsampling step of downsampling the high-resolution multi-viewpoint image based on the reduction ratio and outputting a low-resolution multi-viewpoint image;
An image encoding step of encoding the low resolution multi-view image obtained in the down-sampling step and outputting low-resolution multi-view image code data;
A depth map encoding step of encoding the high resolution multi-view depth map and outputting high-resolution multi-view depth map code data. Free image sent from an image sending device that sends a high-resolution multi-viewpoint image of the same scene captured by multiple cameras and a high-resolution multi-view depth map that describes the depth information for each pixel as a grayscale image An image receiving program for causing a computer on an image receiving apparatus that receives viewpoint image code data to perform image receiving processing,
An image decoding step of decoding the low-resolution multi-view image code data obtained by the image encoding step;
Depth map decoding step for decoding high-resolution multi-view depth map code data obtained by the depth map encoding step;
Setting the correspondence between viewpoints by obtaining corresponding points with different pixel accuracy at different viewpoints of each pixel using the depth information of each pixel obtained from the high-resolution multi-view depth map obtained at the depth map decoding step A correspondence setting step between viewpoints,
An upsampling step of generating a high-resolution multi-viewpoint image from the low-resolution multi-viewpoint image obtained by performing the upsampling while referring to the decimal pixel value of another viewpoint based on the correspondence relationship between the viewpoints;
An image receiving program characterized by causing a high-resolution multi-view image obtained in the up-sampling step and a free-viewpoint image synthesis step to synthesize an arbitrary viewpoint image from the high-resolution multi-view depth map.

Images