JP2017182668A - Data processing apparatus, imaging apparatus, and data processing method - Google Patents

Abstract

[Problem] When correcting second image data (such as distance image data) based on first image data (such as photographed image data), the pixel values of the second image data are To satisfactorily correct second image data even when the pixel values of the image data are different. A data processing device (13) that corrects second image data using first image data and second image data, wherein the first image data is sent to a distance image data correction section (133). a smoothing processing unit 1333 that performs smoothing to generate smoothed data; and a weight that uses the smoothed data to determine weighting coefficients for surrounding pixels to be used when correcting the correction target pixel of the second image data. It includes a determining unit 1334 and a correcting unit 1335 that corrects the correction target pixel using the pixel information of the surrounding pixels and the weighting coefficient. [Selection diagram] Figure 1

Description

  The present invention relates to a data processing device, an imaging device, and a data processing method.

  There has been proposed a method for acquiring or calculating a distance image and a motion image representing a distribution of distance information and motion information corresponding to a captured image. However, the acquired distance information and motion information often include an error due to the relationship between the acquisition principle and the subject. For example, when distance information or motion information is calculated by template matching, a large error occurs in a region that is a distance or motion boundary, such as the vicinity of the subject boundary in the captured image. This error occurs when different distances or movements are included in one template, and the acquired information is incorrect information indicating an intermediate state among a plurality of different states. The extent and amount of error spread depends on the template size. In addition to template matching, information varies due to various factors. For this reason, a method has been proposed in which the information of the corresponding pixel is corrected using a captured image.

  In Patent Document 1, clustering is performed using a plurality of pixel values and distance values within a calculation range of a correction target pixel. The distance information of each pixel is corrected by calculating the distance value of the correction target pixel based on the clustering result.

  In Patent Document 2, the entire captured image is divided into small regions called SuperPixels (a set of a plurality of pixels assumed to be similar objects), and each small region has the same distance (in this document, the foreground, background, etc. 3)), the distance image is corrected.

  In Non-Patent Document 1, a distance image is corrected by a weighted cross bilateral filter using distance information, luminance information of a captured image, and reliability information derived therefrom.

Japanese Patent No. 5150698 US Patent Application Publication No. 2015/003725

Matsuo Shinya, et al., “Improvement of depth estimation accuracy by weighted cross bilateral filter”, Journal of the Institute of Image Information and Television Engineers, Vol. 66, no. 11, pp. J434-J443 (2012)

  As described above, when a distance image or a motion image is corrected using a captured image, among pixels around the correction target pixel, pixels whose luminance and color in the captured image are similar to the correction target pixel are used. Pixel information is calculated. However, the distribution of distance and motion information is not necessarily the same as the luminance and color distribution of the captured image. That is, there are cases where pixels having different luminance and color are included in regions having the same distance. For this reason, there may be a case where the pixel and the number of pixels of the captured image to be referred to differ depending on the luminance and color difference around the correction pixel in the region. As a result, even for subjects with the same distance and movement, the correction results differ depending on the brightness and color distribution of the photographed image, and there is a problem that a difference in distance or movement that does not exist originally occurs. .

  Such a problem does not occur only between a distance image or a motion image and a captured image. When correcting second image data (for example, distance image or motion image) corresponding to first image data (for example, photographed image), it is used for correction in consideration of the similarity of pixel values of the first image data. The same problem occurs when selecting the pixel of the second image data to be performed.

  An object of the present invention is to provide a data processing device that can satisfactorily correct the second image data even when the pixel values of the first image data are different in a region where the pixel values of the second image data are the same. Is to provide.

The first aspect of the present invention is:
A data processing device that corrects second image data using first image data and second image data,
Smoothing means for smoothing the first image data to generate smoothed data;
Weight determining means for determining a weighting factor for peripheral pixels used when correcting the correction target pixel of the second image data using the smoothed data;
Correction means for correcting the correction target pixel using pixel information of the peripheral pixels and the weighting factor;
Is a data processing apparatus.

The second aspect of the present invention is:
A data processing device that corrects second image data using first image data and second image data,
Smoothing means for smoothing the first image data to generate smoothed data;
Pixel selecting means for selecting a pixel to be used when correcting the correction target pixel of the second image data using the smoothed data;
Correction means for correcting the correction target pixel using pixel information of the second image data corresponding to the pixel selected by the pixel selection means;
Is a data processing apparatus.

The third aspect of the present invention is:
A data processing method for correcting second image data using first image data and second image data,
Computer
A smoothing step of smoothing the first image data to generate smoothed data;
Using the smoothed data, a weight determining step of determining a weighting factor for peripheral pixels used when correcting the correction target pixel of the second image data;
A correction step of correcting the correction target pixel using pixel information of the peripheral pixels and the weighting factor;
Is a data processing method for executing.

The fourth aspect of the present invention is:
A data processing method for correcting second image data using first image data and second image data,
Computer
A smoothing step of smoothing the first image data to generate smoothed data;
A pixel selection step of selecting a pixel to be used when correcting the correction target pixel of the second image data using the smoothed data;
A correction step of correcting the correction target pixel using pixel information of the second image data corresponding to the pixel selected in the pixel selection step;
Is a data processing method for executing.

  According to the present invention, even when the pixel values of the first image data are different in the region where the pixel values of the second image data are the same, the second image data can be corrected well.

FIG. 3 is a schematic diagram illustrating an example of an imaging apparatus including the data processing apparatus according to the first embodiment. 5 is a flowchart of a data processing method according to the first embodiment. FIG. 6 is a schematic diagram illustrating an example of an imaging device including a data processing device according to a second embodiment. FIG. 3 is an explanatory diagram of an outline of a data correction method according to the first embodiment. FIG. 3 is a schematic explanatory diagram of a bilateral filter.

  The present invention will be described in detail with reference to embodiments and drawings. However, the present invention is not limited to the configuration of each embodiment. Moreover, you may combine each embodiment suitably.

(Embodiment 1)
In the present embodiment, information to be corrected is distance information. The distance information is a value corresponding to the distance between a certain reference point and the subject. Typically, the distance (absolute distance) between the imaging device and the subject corresponds to the distance information. In general, the distance information may be another value obtained from the distance between the imaging device and the subject. Another example of the distance information is a distance between the focus position of one image and the subject, and a distance between the intermediate position of the focus position of two images and the subject. The distance information may be either a distance on the image plane side or a distance on the object side. The distance information may be represented by a distance in real space, or may be represented by an amount that can be converted into a real space distance, such as a defocus amount or a parallax amount. Thus, the distance information is a value that changes depending on the distance (absolute distance) between the imaging device and the subject. Therefore, the distance information can also be referred to as a distance dependent value.

  In the present embodiment, distance image data (second image data) is generated from captured image data (first image data), and the distance image data is corrected using the captured image data and the distance image data.

[Constitution]
FIG. 1A schematically shows a configuration when the data processing apparatus according to the present invention is applied to an imaging apparatus. The imaging apparatus 1 includes an imaging optical system 10, an imaging element 11, a control unit 12, a data processing device 13, a storage unit 14, an input unit 15, and a display unit 16.

  The imaging optical system 10 is an optical system that includes one or a plurality of lenses and forms incident light on the image plane of the imaging element 11. The image sensor 11 is an image sensor having an image sensor such as a CCD or a CMOS. An image sensor having a color filter, a monochrome image sensor, or a three-plate image sensor may be used.

  The data processing device 13 includes a signal processing unit 130, a memory 131, a distance image data generation unit 132, and a distance image data correction unit 133. The data processing device 13 can be configured using a logic circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). A part or a front function of the data processing device 13 may be configured by a computer (processor) and a program.

  The signal processing unit 130 performs various signal processing such as AD conversion, noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, and color correction of the analog signal output from the image sensor 11. Digital image data output from the signal processing unit 130 is accumulated in the memory 131 and used for display on the display unit 16, recording (saving) in the storage unit 14, calculation of distance information, generation of distance image data, and the like. . The distance image data represents a distribution of distance information. The distance image data is also referred to as a distance map.

  The distance image data generation unit 132 acquires captured image data from the signal output from the signal processing unit 130, and generates distance image data of the subject from the captured image data. There is no particular limitation on the method for acquiring the distance information of the subject, and a depth-from-defocus method (hereinafter referred to as a DFD method) that uses photographed image data with different blurs taken under different photographing conditions, or photographed image data with parallax is used. Stereo method can be mentioned. Acquisition of captured image data with parallax can be obtained by using a plurality of imaging devices or an imaging device having pixels that can divide the pupil of the optical system. Other examples of distance information acquisition methods include Time of Flight (TOF method) and Depth From Focus. The distance image data generated by the distance image data generation unit 132 is stored in the storage unit 14 or temporarily stored in the memory 131 and used for subsequent processing. Further, the distance image data may be corrected according to the acquisition method.

  The distance image data correction unit 133 has a function of correcting the distance information of each pixel of the distance image data using the acquired captured image data and the distance image data generated by the distance image data generation unit 132. As shown in FIG. 1B, the captured image data correction unit 133 includes, as its functional units, a captured image data acquisition unit 1331, a distance image data acquisition unit 1332, a smoothing processing unit 1333, a correction use pixel selection unit 1334, A correction unit 1335 is provided. The captured image data acquisition unit 1331 reads captured image data from the storage unit 14 or the memory 131. The distance image data acquisition unit 1332 reads the distance image data from the storage unit 14 or the memory 131. The smoothing processing unit 1333 smoothes the captured image data and generates smoothed data. The correction use pixel selection unit 1334 selects a pixel to be used when correcting the correction target pixel of the distance image data based on the smoothed data. The correction unit 1335 corrects the distance information of the correction target pixel using the distance image data corresponding to the selected pixel. Details of the correction processing by the distance image data correction unit 133 will be described in detail later.

  The storage unit 14 is a non-volatile storage medium that stores captured image data, distance image data, other intermediate data, parameter data used in the imaging apparatus 1, and the like. As the storage unit 14, any storage medium that can read and write at high speed and has a large capacity may be used. For example, a flash memory or the like is preferable as the storage unit 14.

  The input unit 15 is an interface that is operated by a user to input information and change settings on the imaging apparatus 1. For example, dials, buttons, switches, touch panels and the like can be used.

  It is a display means comprised with the display part 16, a liquid crystal display, an organic EL display, etc. The display unit 16 displays a composition confirmation at the time of photographing, browsing of photographed / recorded images, various setting screens, message information, and the like.

  The control unit 12 controls each unit of the imaging device 1. Functions of the control unit 12 include, for example, automatic focusing by autofocus (AF), change of focus position, change of F value (aperture), image capture, control of shutter and flash (both not shown), and storage. The unit 14, the input unit 15, and the display unit 16 are controlled.

[processing]
Next, the flow of the entire process from the start to the end of photographing according to the present embodiment will be described using the flowchart of FIG. First, in step S20, the photographer zooms in on the scene he wants to shoot, determines the composition, and simultaneously sets shooting conditions such as shutter speed and F number, and performs focus adjustment. These photographing conditions may be determined manually by the photographer or automatically by the imaging device 1.

  Next, when a photographing button (not shown) is pressed in step S21, the imaging device 1 acquires photographed image data and distance image data. The distance image data is generated by the distance image data generation unit 132 by the method described above. The captured image data needs to be captured in accordance with the distance image data generation method. For example, when distance image data is generated by DFD, it is necessary to obtain a plurality of photographed image data with different blurs by changing the photographing conditions.

  In step S22, the distance image data correction unit 133 corrects the distance image data. Details of this step will be described with reference to the flowchart of FIG.

  In step S220, the smoothing processing unit 1333 performs smoothing processing on the captured image data as preprocessing. Smoothed photographed image data (smoothed data) is stored in the memory 131 and used in the following steps.

The smoothing process in step S220 is preferably a process that smoothes minute changes in luminance and color in the captured image data, but preserves changes in luminance and color at the edge of the object. Such a smoothing process is called edge-preserving smoothing. In the edge preserving type smoothing process, pixel values are weighted and averaged using the weighting according to the distance between the pixels and the weighting according to the difference between the pixel values (pixel information) on the peripheral pixels of the target pixel. The

An example of the edge preserving type smoothing process is a bilateral filter. The bilateral filter is expressed by Equation 1. The bilateral filter obtains the filtering result _Ip by referring to the pixels in the peripheral region S at the position p.

Here, the pixel value at the position p of the captured image F is represented by F _p , and the pixel value obtained by filtering with reference to the pixels in the peripheral region S at the position _p is represented by I _p . Also, W _p in equation (1) is used for normalization and is defined as in equation (2). G _σ is a Gaussian function (normal distribution function) with a variance of σ ² and an average of 0. In Expression (2), σ _s and σ _r are parameters that determine the filtering amount of the captured image F. As for the normalized weight of Equation (1), G _σs is a Gaussian defined in the spatial direction, and the weight is controlled by the degree of separation (distance between pixels) from the pixel position of interest. G _σr is a Gaussian defined in the intensity direction, and the weight to the pixel q is controlled by a difference from the intensity Fp (difference in pixel information). Filtering in consideration of the weights in the spatial direction and the intensity direction is executed by these two parameters. The peripheral region S is a region centered on the position p, and is, for example, a square or circular region. The size of the peripheral region S may be determined as appropriate.

The concept of this bilateral filter process will be described with reference to FIG. FIG. 5 shows the intensity distribution (pixel value distribution) 50 in one dimension for the sake of simplicity. The intensity distribution 50 is a distribution indicating pixel values for each luminance or color position. Now, the position p where the filtering process is performed is defined as a position 51. The intensity distribution 50 at the position 51 has a pixel value 52. Here, it is a parameter σ _s in the spatial direction that weights with Gaussian so that the pixel value near the position 51 is used for filtering. A graph 53 shows the weighting coefficient for each position. The weighting coefficient 53 is defined in the spatial direction according to the Gaussian distribution in a wide range if σ _s is large and in a narrow range if σ _s is small. Similarly, the intensity direction parameter σ _r is weighted by Gaussian so that a pixel value close to the intensity 52 to be filtered in FIG. 5 is used for filtering. The graph 54 shows the weighting coefficient for each pixel value. The weighting factor 54 is defined in the intensity direction according to the Gaussian distribution in a wide range if _σr is large and in a narrow range if σr is small. In this way, by weighting in the spatial direction and the intensity direction, it is possible to perform a smoothing process that preserves edges.

Other examples of edge-preserving smoothing include nonlinear diffusion and guided filters (Guided
Filter), Non-local Means filter. These filters are also available in step S220.

  In the loop L1, the processes in steps S221 and S222 are repeated for each correction target pixel of the distance image data. Typically, all the pixels of the distance image data are correction target pixels, and the processing of the loop L1 is performed on all the pixels of the distance image data.

  Next, in step S221, the correction use pixel selection unit 1334 selects a reference pixel for the correction target pixel of the distance image data. The reference pixel is a pixel that is referred to when the correction value of the correction target pixel is obtained in step S222. The reference pixel can also be referred to as a pixel used for correction (correction use pixel).

  In the reference pixel selection process in step S221, the captured image data smoothed in step S220 (hereinafter referred to as smoothed data) is used. The reference pixel can be determined as a pixel in which the similarity of the pixel information of the smoothed data between the correction target pixel and the correction target pixel is equal to or higher than a predetermined threshold among the correction target pixels. The degree of similarity can be determined by comparing information on the intensity (luminance or color) of the smoothed data of the correction target pixel and its neighboring pixels, or information that can be converted therefrom. Here, the neighboring pixels are all pixels existing in a predetermined range with reference to the correction target pixel. The shape and size of the neighboring pixels may be arbitrarily determined based on the error amount of the distance image data. In addition, the neighboring pixels may or may not include the correction target pixel itself. In determining whether the pixel is a reference pixel, the similarity S (q) calculated for each neighboring pixel (reference pixel candidate pixel) q is used.

ここで、Ｉは平滑化データでの輝度値を表し、ｐは補正対象画素、ｑは近傍画素の位置を表す。Ｓは補正対象画素と近傍画素の輝度値の類似度を表し、値が小さいほど類似度は高い。 For example, if the smoothed data is a luminance image having a monochrome pixel value, the similarity S (q) is calculated as in Expression (3).

Here, I represents the luminance value in the smoothed data, p represents the correction target pixel, and q represents the position of the neighboring pixel. S represents the similarity between the luminance values of the correction target pixel and neighboring pixels, and the smaller the value, the higher the similarity.

ここで、Ｒ、Ｇ、Ｂはそれぞれ赤、緑、青のカラーチャンネルを表す。 Further, if the smoothed data is color image data, the similarity S (q) between the correction target pixel and the neighboring pixels is calculated by the Euclidean distance relating to the color as in Expression (4).

Here, R, G, and B represent red, green, and blue color channels, respectively.

  The method of calculating the similarity is not limited to the one described above, and there is no particular limitation such as the Manhattan distance. Further, the similarity may be calculated by converting the color image into another color space such as CIELab color space or YUV color space.

As described above, the distance image data correction unit 133 selects a reference pixel according to the similarity. Specifically, a pixel having a similarity S value equal to or lower than a threshold U (similar to the threshold U) is selected as a reference pixel. When the selected pixel is 1 and the pixel that was desired to be selected is 0 and expressed as W, it can be determined for each neighboring pixel as shown in Equation (5).

  Supposing that W is a weighting factor of each pixel in the correction process, step S221 of the present embodiment uses only pixels whose smoothing data similarity is equal to or greater than a threshold among the neighboring pixels of the correction target pixel. This can be regarded as a process of determining a weighting factor for neighboring pixels.

  The reference pixel determination method is not necessarily the above method. In addition to the method using the smoothed data, the reference pixel can be determined using reliability information indicating the reliability (probability) of the distance information. The reliability may be obtained as the likelihood of the distance information during the distance calculation process in the distance image data generation unit 132. In that case, reliability data can also be acquired in step S21. Further, the reliability of the distance information can be acquired by analyzing the captured image data. Depending on the distance calculation method, for example, a dark place (blackout) or a bright place (overexposed), a place where there is no texture in the subject, a place where the subject is moving, etc. should be calculated as low reliability. it can. Furthermore, the reliability can be acquired from the distance image data. In the distance image data, a pixel having a significantly different distance value or a boundary portion of the distance with respect to a neighboring pixel group is highly likely to be an error, and can be obtained with low reliability. A pixel to be used for correction can be selected using the reliability obtained in this way. It is preferable not to select unreliable distance information as reference pixels in the distance image data. Therefore, similarly to the threshold value processing of the similarity of the luminance image described above, the threshold value is processed with a certain threshold value, and only reliable distance information is selected as the reference pixel. Also, the reference pixel can be determined by combining the reliability and the similarity. Any method may be used as the method of the selection method using the reliability. The pixel selection using the smoothed data and the captured image data has been individually described above. However, the pixel selection rule may be determined by comprehensively determining each pixel selection.

Next, in step S222, the distance image data correction unit 133 corrects the distance information data of the correction target pixel using the selected pixel obtained in step S221. Now, assuming that the corrected distance information data is D ′ and the uncorrected distance information data is D, the corrected distance information data D ′ is calculated by Expression (6).

That is, in the present embodiment, the corrected distance information data D ′ is calculated as the average value of the distance information data of the reference pixel (correction use pixel) selected in step S221. W
Is regarded as a weighting factor, the corrected distance information data D ′ can be said to be a weighted average value of distance image data of pixels near the correction target pixel.

  If the peripheral pixel range Q is too wide, the assumption that distance information having similar brightness (or color) has a close value breaks down, increasing the possibility of generating a new error, and increasing the amount of calculation. is there. On the other hand, if the peripheral pixel range Q is too narrow, there are few peripheral pixels that can be referred to, and appropriate correction cannot be performed. For this reason, it is desirable that an appropriate peripheral pixel range Q is determined in advance according to the distance calculation method for the amount of error (width) generated at the object boundary.

  Here, the outline of the distance map correction described in steps S220 to S222 will be described in detail with reference to FIGS. 4 (A) to 4 (E). 4A to 4E, distance and intensity (brightness or color) are collectively described for simplicity of description, and the horizontal axis represents position and the vertical axis represents distance or intensity (brightness or color). It is. 4A shows the one-dimensional distribution of the distance image data before correction, and FIG. 4B shows the one-dimensional distribution of the captured image data at the same position as that in FIG. 4A.

  It is assumed that the distance 41 in FIG. A range 43 in FIG. 4B indicates a pixel range Q within the threshold value U of the intensity difference in Expression (5) and the surrounding pixel range Q in Expression (6). The pixel position of the pixel 44 that falls within this range 43 is specified. The distance 45 of the pixel position determined here is referred to when the distance 41 is corrected, and the average value of the distance 41 and the distance 45 (the distance 46 shown in FIG. 4C) is the corrected distance.

  As described above, when the captured image data varies as shown in FIG. 4B due to the distribution of the subject and the influence of shot noise, the number of reference pixels used for correcting the distance image data decreases. As a result, the correction effect is reduced as shown in FIG.

  On the other hand, in the present embodiment, smoothed photographed image data as shown in FIG. 4D is obtained by smoothing the photographed image data shown in FIG. In the captured image data after smoothing, variation in pixel values (luminance or color) is reduced, so that the number of pixels included in the range 43 is large. Therefore, a large number of pixels are selected as reference pixels used for correction. FIG. 4E shows a correction result by the correction processing of this embodiment. As shown in FIG. 4E, according to the present embodiment, a good correction result can be obtained.

  Further, by using the smoothed data, the threshold value U for selecting the reference pixel can be reduced. As a result, the possibility of selecting pixels having different pixel values is reduced, and the effect of improving the correction accuracy can be obtained.

Further, although it is influenced by the intensity distribution of the subject of the photographed image, the correction effect is improved when the weighting range in the spatial direction when generating the smoothed data is larger than the region Q that is referred to when correcting the distance data. It is often high. Specifically, when the region Q is a square or a circle, σ _s (standard deviation of the Gaussian function G _σs , that is, the square root of the variance σ _s ² ) in the formula (1) is the length of one side of the square of the region Q. It is better to make it larger than half the radius or circular radius. As described above, the distance image data is corrected by controlling the smoothing data generation and distance map correction parameters related to the distance map correction.

  Next, in step S23, processing according to the application using the captured image data and the corrected distance image data is performed. As an example of processing, there is a blurring process for adding blur according to distance information to captured image data.

In step S24, the result of the application process is displayed on the display unit 16. Further, the processing result is recorded in the storage unit 14 together with the photographed image data, the distance image data, and the corrected distance image data.

  As described above, according to the present embodiment, the smoothed image is used by smoothing the captured image data as preprocessing and correcting the distance image data based on the smoothed captured image. It becomes possible to realize distance correction with higher accuracy than in the case where there is not. As a result, there is an effect that various processes using the distance image can be accurately realized.

  In the above description, only the smoothing process is applied to the captured image data. However, other than the smoothing process may be applied. For example, it is possible to enhance the contour of the distance image data by performing edge emphasis processing after smoothing processing and emphasizing the difference in signal strength at the edge portion of the subject. By emphasizing the contour, it is possible to avoid using the distance information of different objects when correcting the distance.

In the above description, as shown in Expression (5), only the pixels having the similarity of the captured image data equal to or lower than the threshold U among the peripheral pixels of the correction target pixel are used for correction. That is, the weight W is binarized to 0 or 1. However, the weight W does not have to be a binary value and can be a continuous value. For example, the weight W can be a value corresponding to the similarity. In the present embodiment, the similarity S has a smaller value as it is similar. For example, W = (S _max −S) / S _max (where S _max is the maximum value of the similarity S). . Of course, the weight W may be determined by another calculation formula as long as the value of the similarity S is smaller (that is, the similarity is higher). From this point of view, the correction use pixel selection unit 1334 can be regarded as a weight determination unit that determines weighting factors for peripheral pixels used when correcting the correction target pixel.

  In the above description, the average of the distance information of the pixels used for correction is the corrected distance information. However, the correction distance information may be obtained by any method other than the above. For example, distance information after correction of a correction target pixel may be obtained by interpolation processing of a defective pixel based on a pixel used for correction.

(Embodiment 2)
In the first embodiment, the distance information is corrected using captured image data and distance image data. In contrast, in the present embodiment, it is shown that correction processing can be performed using a combination other than a combination of a captured image and a distance image. FIG. 3 shows the data processing apparatus of this embodiment. The flowchart is the same as the flowchart of the first embodiment. The image processing method of the present embodiment will be described focusing on differences from the first embodiment.

  The data processing device 31 includes a first image data input 311, a second image data input unit 312, and a second image data correction unit 314.

  In this specification, “image data” means two-dimensional array data in which numerical data is logically two-dimensionally arrayed. In this specification, “image data” is also referred to as “map”. The “pixel” of the image data means a position in the two-dimensional array of the image data. Numerical data constituting image data is not limited to data representing specific information, and includes, for example, data representing luminance information, distance information, motion information, and reliability information. Further, the format of numerical data is not limited to a specific format, and may be a scalar, a vector, a matrix, or the like. The image data is referred to as luminance image data, distance image data, motion image data, reliability image data, or the like depending on information to be handled.

Reference image data is input to the first image data input unit 311. Examples of the first image data include not only luminance image data but also already corrected distance image data, infrared light image data, polarization image data, and the like. When the first image data is luminance image data, the data processing device 31 of the present embodiment can be an imaging device as shown in FIG. When the first image data is distance image data or the like, the first image data input unit 311 may include a data generation unit. If the first image data is distance image data, F and I in equation (1) correspond to distance values.

  Second image data is input to the second image data input unit 312. The second image data is image data at substantially the same viewpoint as the first image data, and is preferably image data representing different information. The first image data and the second image data may be data representing information of the same content calculated by different calculation methods. The first image data and the second image data do not necessarily have the same angle of view, and the second image data only needs to be included in the first image data. That is, the angle of view of the first image data may be equal to the angle of view of the second image data, or may be larger than that.

  In addition, as an example of the second image data, motion data (Optical Flow) can also be cited. The motion data is data representing the relative motion of the subject or the subject and the camera. For example, the speed in the horizontal direction (x direction) and the vertical direction (y direction) is held as data in each pixel as a format. Such motion data is calculated from two images acquired at a certain time interval by template matching of the two images, and the vertical and horizontal speeds are calculated from the amount of movement and the shooting time interval. Can be calculated. As for the speed, when different speeds are mixed in a template of a certain size, different speeds are mixed, and an error having an intermediate speed value occurs. The effect is obtained if the data has such an object boundary error.

  The second image data may be an infrared light image or a polarized image. Infrared images and polarized images may have a lower resolution than RGB images due to the effects of chromatic aberration of lenses and special sensor structures. In this case also, it can be considered that an error has occurred in the object boundary.

  Further, when the second image data has a small size (small data amount) with respect to the first image data, an enlargement (data interpolation) process may be performed. In that case, an error occurs near the edge of the data (near the boundary of the object). Image data after such enlargement processing can be used as second image data. For example, an image sensor for acquiring the above-described infrared image or polarization image may have an infrared color filter or polarization filter only in a specific pixel. In this case, since the acquired infrared image or polarization image has a smaller size than the RGB image, enlargement processing may be performed.

  The second image data may be input from another device, or may be calculated from other information including the first image data. If it is motion data, D in Expression 6 corresponds to a velocity value in the horizontal direction or the vertical direction.

  As described above, if there is an error in the second image data and the first image data is data serving as a reference for correction, the correction processing of the second image data is performed in the same manner as in FIG. It can be carried out. In other words, the second image data correction unit 314 has a configuration similar to that in FIG. 1B and may perform processing similar to the processing shown in FIG. Depending on the data, there may be a plurality of correction target data, but the basic processing remains the same. For example, in the case of motion data, horizontal motion data correction processing and vertical data correction processing are independently performed using the same method.

  According to the present embodiment, it is possible to perform high-precision correction using various data having an error, not limited to the distance image data.

(Embodiment 3)
The above-described data processing method of the present invention can be preferably applied to, for example, an imaging apparatus such as a digital camera or a camcorder, or an image processing apparatus or a computer that processes data obtained by the imaging apparatus. The technology of the present invention can also be applied to various electronic devices (including mobile phones, smartphones, slate terminals, and personal computers) incorporating such an imaging device or image processing device. In the above-described embodiment, the configuration in which the image processing device is incorporated in the main body of the imaging device is shown, but the function of the image processing device may be configured in any manner. For example, an image processing device may be incorporated in a computer having an imaging device, and the computer may acquire an image captured by the imaging device, and the data processing method may be executed based on the acquired image. Further, the image processing apparatus may be incorporated in a computer that can access the network by wire or wireless, and the computer may acquire an image via the network and execute the image processing method based on the image. The obtained distance information can be used for various types of image processing such as image segmentation, generation of stereoscopic images and depth images, and emulation of blur effects.

(Other examples)
The present invention can be realized by a general-purpose processor such as a microprocessor or CPU (Central Processing Unit) and a computer having a program stored in a memory, and the general-purpose processor executing the program. The present invention can be realized by a dedicated processor such as an ASIC, FPGA, or DSP. Both the dedicated processor and the general-purpose processor that executes the program can be said to be a processor configured to provide a specific function or a processor configured to function as a specific function unit. Some functions of the present invention may be provided by a general-purpose processor (and program), and other functions may be realized by a dedicated processor. One function of the present invention may be realized by both a general-purpose processor (and program) and a dedicated processor.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

13 Image Processing Device 1333 Smoothing Processing Unit 1334 Correction Use Pixel Selection Unit (Weight Determination Unit)
1335 Correction unit

Claims (13)

A data processing device that corrects second image data using first image data and second image data,
Smoothing means for smoothing the first image data to generate smoothed data;
Weight determining means for determining a weighting factor for peripheral pixels used when correcting the correction target pixel of the second image data using the smoothed data;
Correction means for correcting the correction target pixel using pixel information of the peripheral pixels and the weighting factor;
A data processing apparatus. The smoothing means performs edge-preserving smoothing processing.
The data processing apparatus according to claim 1. The smoothing process performed by the smoothing means is a weighted average process using weighting according to the inter-pixel distance and weighting according to the difference in pixel information.
The data processing apparatus according to claim 2. The smoothing means performs weighting according to a Gaussian function according to a distance between pixels,
The area of the peripheral pixel used when correcting the correction target pixel of the second image data is a square or circular area centered on the correction target pixel,
When the variance of the Gaussian function is σ _s ² , σ _s is larger than half the length of one side of the square or the radius of the circle,
The data processing apparatus according to claim 3. The weight determining means determines a weighting factor for the neighboring pixel based on a similarity of pixel information of the smoothed data corresponding to the correction target pixel and the neighboring pixel;
The data processing apparatus according to any one of claims 1 to 4. The weight determining means determines the weighting factor so as to use only neighboring pixels whose similarity is higher than a threshold;
The data processing apparatus according to claim 5. The weight determining means determines the weighting factor also using a reliability representing the reliability of the second image data;
The data processing apparatus according to any one of claims 1 to 6. A data processing device that corrects second image data using first image data and second image data,
Smoothing means for smoothing the first image data to generate smoothed data;
Pixel selecting means for selecting a pixel to be used when correcting the correction target pixel of the second image data using the smoothed data;
Correction means for correcting the correction target pixel using pixel information of the second image data corresponding to the pixel selected by the pixel selection means;
A data processing apparatus. The first image data is luminance image data or color image data,
The second image data is distance image data or motion image data generated using the first image.
The data processing apparatus according to any one of claims 1 to 8. An image sensor;
A data processing device according to any one of claims 1 to 9,
An imaging device comprising:
The first image data is captured image data acquired from the imaging element.
Imaging device. A data processing method for correcting second image data using first image data and second image data,
Computer
A smoothing step of smoothing the first image data to generate smoothed data;
Using the smoothed data, a weight determining step of determining a weighting factor for peripheral pixels used when correcting the correction target pixel of the second image data;
A correction step of correcting the correction target pixel using pixel information of the peripheral pixels and the weighting factor;
Data processing method to execute. A data processing method for correcting second image data using first image data and second image data,
Computer
A smoothing step of smoothing the first image data to generate smoothed data;
A pixel selection step of selecting a pixel to be used when correcting the correction target pixel of the second image data using the smoothed data;
A correction step of correcting the correction target pixel using pixel information of the second image data corresponding to the pixel selected in the pixel selection step;
Data processing method to execute.   The program which makes a computer perform each process of the method of Claim 11 or 12.

Images