JP2017073038A - Image processor, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of appropriately limiting a band and suppressing blurring of an image while reducing moire. An image processing device includes a local reduction ratio output means (102) that outputs a local reduction ratio in the vicinity of a pixel of interest of an output image with respect to an input image, and a corresponding point of an input image corresponding to the pixel of interest of the output image. A correction distance output means (103) that outputs a plurality of correction distances by correcting a plurality of distances between a plurality of pixels located in the vicinity of corresponding points of an input image by using a local reduction ratio, and a plurality of correction distance output means (103). The filter coefficient output means (104) that outputs a plurality of filter coefficients based on the correction distance of, and the data of a plurality of pixels located near the corresponding points of the input image are normalized by the sum of the plurality of filter coefficients. It has a filter processing means (107) that outputs data of a pixel of interest in an output image by performing the above-mentioned filter processing. [Selection diagram] Fig. 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  When the output image is created by enlarging or reducing the input image, the gradation value of each pixel of the output image is based on information about which pixel position of the input image corresponds to each pixel position of the output image. Need to ask. Since the corresponding position in the input image of each pixel of the output image does not necessarily match the pixel position of the input image itself, the gradation value between the pixel positions of the input image is set when the image is enlarged or reduced. It is necessary to obtain by interpolation. Bilinear interpolation, bicubic interpolation, and the like are known as gradation value interpolation calculation methods.

  When the input image is reduced, it is known that image degradation called moire occurs. This is aliasing distortion explained by the sampling theorem, and occurs when the spatial frequency of the image is not appropriately limited when the image is reduced. In order to prevent moiré caused by reduction, it is necessary to perform band limitation by applying a low-pass filter to the input image in advance. However, if an excessive low-pass filter process is performed, the contrast of the image is lowered and the image looks blurred. Therefore, the low-pass filter process needs to appropriately suppress the high-frequency component of the image without excess or deficiency. In general, the frequency characteristics of the low-pass filter and the presence or absence of the low-pass filter processing itself are appropriately selected in consideration of the balance between moire, image blur, and calculation load.

  Patent Document 1 discloses an image processing apparatus having an image thinning unit that performs a process of thinning out pixel data included in image data with respect to image data represented by digitizing the image. The band limiting unit limits the frequency band of the image data from which pixel data is thinned out by the image thinning unit. The interpolating unit interpolates the image data subjected to the frequency band limitation by the band limiting unit.

  Patent Document 2 discloses an image conversion apparatus that performs geometric conversion on input image data and outputs output image data. The coordinate inverse transform unit inversely transforms the coordinate value in the coordinate system of the output image data to the coordinate value in the coordinate system of the input image data based on the geometric transformation. The coordinate inverse transform unit randomly changes the coordinate values belonging to the same segment among the inversely transformed coordinate values when the region between the pixels of the coordinate system of the input image data is divided into a plurality of segments. .

JP 2005-208983 A JP 2007-35025 A

  When performing a transformation with a uniform reduction ratio for the entire image, it is basically possible to limit the entire image without excess or deficiency by applying a single low-pass filter to the entire input image. It can be said that there is. However, there is a problem in performing non-uniform image deformation (referred to as “geometric deformation” in this specification) such that the reduction ratio or enlargement ratio differs locally. When performing geometric deformation, if a single low-pass filter is applied uniformly to the entire image, the effect of the low-pass filter may be insufficient depending on the location, or moiré may occur. There arises a problem that the effect is excessive and the image is blurred more than necessary.

  Examples of performing such geometric deformation include display distortion correction in an image observation apparatus such as a head mounted display (HMD) and an electronic viewfinder (EVF) of a digital camera, and keystone correction of a projector.

  First, the former will be described. An image observation device such as an HMD or EVF is a device that enlarges an image of an image display device such as a small display with an optical system called a display optical system and presents the image to an observer. If distortion remains in the display optical system, the observer will observe the distorted image. Therefore, if the remaining distortion is large, the display distortion correction function may correct the distortion of the display optical system. is there. The display distortion correction is a method of distorting the image displayed on the image display device in reverse so as to cancel out the distortion of the display optical system in order to provide the observer with an image without distortion. When an image distorted by the display distortion correction function is displayed on an image display device and enlarged by a display optical system, an observer can observe an image without distortion in which the distortion of the display optical system is offset. . Basically, in this display distortion correction, since it is necessary to reduce and deform the input image non-uniformly, there may be a problem that moire occurs in the place of an image with a high reduction ratio unless appropriate band limitation is performed. . Depending on the display optical system, a complicated distortion that makes it difficult to find a specific regularity may occur. In this case, the display distortion correction requires a complicated geometrical conversion that matches the distortion.

  Next, the latter will be described. As a type of projector, a projection type projector (hereinafter referred to as a projector) that projects an image of an image display device such as a small display onto a front screen through a projection optical system and presents an enlarged image to an observer is known. Yes. If the projector cannot be directly opposed to the screen due to restrictions such as the installation location, the image projected on the screen may be geometrically distorted. This distortion is generally called keystone distortion, and some projectors have a keystone correction function for correcting the keystone distortion. As one method of correction, the image itself displayed on the image display device is distorted in reverse so as to cancel the keystone distortion, and this is projected onto the screen, so that an enlarged image without distortion can be obtained as a projected video image. There is a method to display. Basically, in this type of keystone correction, the input image needs to be reduced and deformed non-uniformly in the same manner as in the image observation apparatus. There may be a problem that moiré occurs. When the positional relationship between the projector and the screen changes, the geometric deformation pattern to be created by the keystone correction changes each time.

  When performing geometric transformations including the above two examples, performing appropriate band limiting processing according to the reduction ratio prepares a huge number of low-pass filter patterns according to various reduction ratios in advance. This means it must be done and is not realistic. In particular, in the case of geometrical deformation in which the deformation pattern is complicated and specific regularity is not seen as in the example of display distortion correction, it is difficult to reduce the types of filters due to regularity. Further, when the geometric deformation pattern itself is not fixed and can be changed as in the example of keystone correction, it is necessary to prepare a low pass filter corresponding to all the possible deformation patterns. There is known a simple and effective method for minimizing image blur while reducing moire when performing geometric deformation of an uneven image with a locally changing reduction ratio. Not.

  An object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of appropriately performing band limitation and suppressing blurring while reducing moire by a simple method.

  An image processing apparatus according to the present invention is an image processing apparatus that generates an output image having a locally different reduction ratio with respect to an input image, and outputs a local reduction ratio in the vicinity of a target pixel of the output image with respect to the input image. A plurality of distances between a corresponding reduction point of the output image and a corresponding point of the input image corresponding to the target pixel of the output image and a plurality of pixels located in the vicinity of the corresponding point of the input image. A correction distance output unit that outputs a plurality of correction distances, a filter coefficient output unit that outputs a plurality of filter coefficients based on the plurality of correction distances, and a corresponding point of the input image. Filter processing means for outputting data of a pixel of interest of the output image by performing filter processing normalized with the sum of the plurality of filter coefficients on data of a plurality of pixels located in the vicinity Characterized in that it has a.

  According to the present invention, it is possible to appropriately limit the band by a simple method and suppress blurring of an image while reducing moire.

It is a block diagram which shows the structural example of the image processing apparatus by 1st Embodiment. It is a figure which shows the geometric transformation by 1st Embodiment. It is a figure which shows the attention pixel and neighboring pixel by 1st Embodiment. It is a figure which shows the image information by 1st Embodiment. It is a figure which shows the comb function by 1st Embodiment. It is a figure which shows the sampling state of the image by 1st Embodiment. It is a figure which shows the frequency characteristic of the interpolation function by 1st Embodiment. It is a figure which shows the image information by 1st Embodiment. It is a figure which shows the comb function by 1st Embodiment. It is a figure explaining the image reduction by 1st Embodiment. It is a figure which shows the frequency characteristic of the interpolation function by 1st Embodiment. It is a figure which shows the frequency characteristic of the filter by 1st Embodiment. It is a figure which shows the input image by 2nd Embodiment. It is a figure which shows the output image by 2nd Embodiment. It is a figure which shows an example of the frequency characteristic of the filter by 2nd Embodiment. It is a figure which shows the input image by 3rd Embodiment. It is a figure which shows an example of the output image by 3rd Embodiment. It is a figure which shows an example of the frequency characteristic of the filter by 3rd Embodiment.

  Hereinafter, an image processing apparatus and an image processing method according to an embodiment of the present invention will be described with reference to the drawings. First, terms and numerical values used in this specification will be described. In this specification, an image before geometric deformation is referred to as an “input image”. An image obtained as a result of geometric deformation of the input image is referred to as an “output image”. In geometric deformation, the tone value of a specific “target pixel” position that constitutes the output image is calculated by interpolation based on the information on the corresponding position where the target pixel is located in the input image. This is performed by repeating the process of obtaining the output image gradation value of the pixel. The gradation value is information on the brightness of the place, and may be expressed by a value from 0 to 255, for example, but the expression of the gradation value is not necessarily limited to this.

  In this specification, the image reduction ratio is defined as 0.5 when the information expressed by 100 pixels in the input image is expressed by 50 pixels in the output image. In a two-dimensional geometric deformation, the reduction ratio varies depending on the location and direction in the image. For example, the horizontal reduction ratio of a certain place is 0.5 and the vertical reduction ratio is 0.7. In another place, the horizontal reduction ratio is 0.6, and the vertical reduction ratio is 0.9. As described above, in the geometrical deformation, the reduction ratio basically varies from place to place, and therefore the reduction ratio at each place is referred to as a “local reduction ratio”. It should be noted that depending on the pattern of geometric deformation, there may be a situation in which the local reduction ratio is the same in an area. A local reduction ratio of 1.0 means that the size of the image information does not change at that location, and if the local reduction rate is greater than 1.0, it means that the size of the image information is enlarged at that location. To do. The embodiment of the present invention is essentially effective in a situation where the local reduction ratio is less than 1.0. Therefore, when the local reduction ratio is greater than 1.0, the interpolation calculation may be performed with the local reduction ratio as 1.0.

  Next, the interpolation function will be described. The image data has image gradation value data at discrete pixel positions corresponding to the sampling positions. A discrete pixel position is generally expressed by an integer value. However, when calculating a gradation value of a target pixel existing at a non-integer position, the gradation value of the target pixel is calculated using an interpolation function. Is calculated. Examples of the interpolation function include a bilinear interpolation function and a bicubic interpolation function. The interpolation function is a function for calculating a filter coefficient for performing a filter operation, which is a product-sum operation with a neighboring pixel, from the distance between the target pixel and the neighboring pixel. The “distance” in this specification is, for example, one-dimensionally explained. The distance between the pixel at the position 2.3 and the pixel at the position 2.0 is 0.3, and the pixel at the position 3.0 is also the same. And the distance between the pixel at position 4.5 is 1.5. Any interpolation function expressed as a distance function may be used, and the interpolation function is not limited to the above-described bilinear interpolation function or bicubic interpolation function.

  Further, in the present specification, processing for blocking or reducing high frequency components of an image is referred to as band limitation. In general, an image may be composed of a plurality of color data such as R (red), G (green), and B (blue) as color information. When processing is required for a plurality of color data such as RGB, it is clear that the method of the embodiment of the present invention should be applied to each color data. The processing will not be described.

(First embodiment)
FIG. 1 is a block diagram showing a configuration example of an image processing apparatus 100 according to the first embodiment of the present invention. The image processing apparatus 100 includes a deformation corresponding point calculation unit 101, a local reduction rate calculation unit 102, a correction distance calculation unit 103, a filter coefficient calculation unit 104, an input image data holding unit 106, and a filter processing unit 107. The filter processing unit 107 includes a normalization unit 105 and a filter calculation unit 111. The image processing apparatus 100 receives the pixel-of-interest position information 110, performs interpolation calculation for geometric transformation, and outputs an output image gradation value 108. The target pixel position information 110 indicates the position of the target pixel of the output image. The deformation corresponding point calculation unit 101 receives the target pixel position information 110 and calculates corresponding position information (deformation corresponding point) indicating which position in the input image the target pixel position information 110 corresponds to. Then, the deformation corresponding point calculation unit 101 outputs the corresponding position information to the local reduction rate calculation unit 102 and the input image data holding unit 106. Note that the deformation corresponding point calculation unit 101 may be realized by a unit that stores in advance all corresponding positions with respect to the target pixel position information 110, or when the corresponding position is represented by some relational expression, the deformation corresponding point calculation unit 101 sequentially calculates. It may be realized by means for calculating the corresponding position. The local reduction rate calculation unit 102 calculates a local reduction rate of the output image with respect to the input image in the vicinity of the target pixel position information 110 and outputs the local reduction rate to the correction distance calculation unit 103. The local reduction rate may be obtained by sequential calculation, or information on the local reduction rate for the target pixel position information 110 may be stored in advance and used. The correction distance calculation unit 103 calculates a correction distance based on the distance between the target pixel and the neighboring point and the local reduction rate output by the local reduction rate calculation unit 102, and calculates the correction distance as a filter coefficient. Output to the means 104. The filter coefficient calculation unit 104 calculates a filter coefficient from the interpolation function using the correction distance, and outputs the filter coefficient to the filter processing unit 107. The input image data holding unit 106 holds the input image data, and outputs the calculation target image data 109 near the deformation corresponding point to the filter processing unit 107 based on the deformation corresponding point output by the deformation corresponding point calculating unit 101. . The filter calculation unit 111 performs a filter calculation on the calculation target image data 109 output from the input image data holding unit 106 using the filter coefficient output from the filter coefficient calculation unit 104. The normalizing means 105 normalizes the filter calculation action of the filter calculation means 111 by the sum of the filter coefficients. The filter processing unit 107 includes a normalization unit 105 and a filter calculation unit 111, and calculates the output image gradation value 108. If the above-described series of calculations are performed for a specific target pixel constituting the output image and the gradation value of the specific target pixel is calculated, then the same processing is repeated for the remaining target pixels. An output image subjected to geometric deformation can be obtained. The input image data holding unit 106 holds an input image, and the image processing apparatus 100 can generate an output image having a locally different reduction rate with respect to the input image.

  Next, the effect of this embodiment will be described based on specific numerical values. FIG. 2 is a diagram illustrating a relationship between an input image and an output image according to the present embodiment. The 81 black circles 201 indicate the data position of the input image, where i is the horizontal position and j is the vertical position. In FIG. 2, i and j have values of 1 to 9, respectively. The twelve white circles 202 are data positions of the output image, and the horizontal position is i 'and the vertical position is j'. In FIG. 2, i ′ is a value from 1 to 3, and j ′ is a value from 1 to 4. Note that i, j, i ', and j' may be defined as negative values. In the present embodiment, since i, j, i ′, and j ′ are defined as taking positive values, there is no data in the area of i, j, i ′, and j ′ that is smaller than 1. It will be. An operation may be necessary for such an area where no data exists. In this case, for example, processing such as handling a gradation value as 0 may be performed. i, j, i ′, and j ′ represent pixel positions of the image, respectively. For simplification, FIG. 2 shows an example of a relatively narrow area, but the image is actually a larger area. It is often composed of (larger values of i, j, i ′, j ′). For example, in the case of an image format called SXGA, 1 ≦ i ≦ 1280 and 1 ≦ j ≦ 1024. Note that the total number of pixels of the input image and the total number of pixels of the output image do not always match. Pixel position data exists at each of a plurality of data positions 201 of the input image. In FIG. 2, the data position 202 of the output image does not coincide with the data position 201 of the input image, and the interval between the data positions 202 of the output image is wider than the interval of the data positions 201 of the input image. This is a state in which geometric transformation accompanied by reduction is performed. Further, the fact that the intervals of the data positions 202 of the output image are not uniform indicates that the local reduction ratio differs depending on the location.

  It is assumed that f (i ′, j ′) = (i, j) represents the position of the input image (i, j) corresponding to the output image (i ′, j ′). The left side indicates the horizontal and vertical positions of the output image, and the right side indicates the horizontal and vertical positions of the input image corresponding thereto. In FIG. 2, the relationship is as follows.

f (1,1) = (4.8, 3.8)
f (1,2) = (4.5,4.7)
f (1,3) = (4.5, 6.3)
f (1,4) = (4.8, 7.7)
f (2,1) = (6.4, 2.7)
f (2,2) = (6.6, 4.3)
f (2,3) = (6.5, 6.0)
f (2,4) = (6.6, 7.8)
f (3,1) = (8.2, 2.4)
f (3,2) = (8.5, 4.4)
f (3,3) = (8.5, 6.0)
f (3,4) = (8.3, 7.4)

  As in this embodiment, in geometric deformation, the value of f (i ′, j ′) is often a non-integer position having a value after the decimal point. The deformation corresponding point calculation unit 101 in FIG. 1 has a function of calculating f (i ′, j ′). If (i ′, j ′) is input as the target pixel position information 110, the corresponding point f ( i ′, j ′) is output. For example, if the target pixel position information 110 is (i ′, j ′) = (2, 2), the deformation corresponding point calculation unit 101 sends f (2, 2) = ( The deformation corresponding points of 6.6, 4.3) are output.

Although the local reduction ratio at a certain position of the output image with respect to the input image is not necessarily uniquely determined when the reduction ratio continuously changes, in the present embodiment, adjacent f (i ′, j ′). ) Is a reduction ratio. For example, the horizontal reduction ratio α _x of (i ′, j ′) = (2, 2) is the i component 4.5 of f (1, 2) = (4.5, 4.7) and f ( (3,2) = (8.5, 4.4), i component 8.5 difference 8.5-4.5 = 4.0 is divided by 3, and the reciprocal is taken to be 0.75. The vertical reduction ratio α _y of f (2,2) is j component 2.7 of f (2,1) = (6.4, 2.7) and f (2,3) = (6.5. 6.0) j-component 6.0 difference 6.0-2.7 = 3.3 is divided by 3, and its reciprocal is taken as 0.91. However, the method of obtaining the reduction ratio is not necessarily limited to this, and a numerical value indicating the state of reduction or enlargement can be used as appropriate. In consideration of the calculation load, the reduction rate may be represented by an average value over a certain range, and the same reduction rate may be used for a specific range. In places where there is no adjacent image data, it may be substituted with a reduction ratio of neighboring pixels.

The local reduction rate calculation unit 102 in FIG. 1 calculates the above-described local reduction rate according to the deformation corresponding point from the deformation corresponding point calculation unit 101, and outputs the local reduction rate to the correction distance calculation unit 103. Further, the local reduction rate calculation unit 102 may output the local reduction rate to the correction distance calculation unit 103 based on the local reduction rate stored in advance. In the present embodiment, if the target pixel position information 110 is (i ′, j ′) = (2, 2), the local reduction ratio output from the local reduction ratio calculation unit 102 to the correction distance calculation unit 103 is The horizontal direction reduction ratio α _x = 0.75 and the vertical direction reduction ratio α _y = 0.91. As described above, the local reduction rate calculation unit 102 is a local reduction rate output unit, and outputs a local reduction rate in the vicinity of the target pixel of the output image with respect to the input image.

  Here, a process of interpolating the gradation value of f (i ′, j ′) from the gradation value of the input image in order to obtain an output image will be described. In this embodiment, a bicubic interpolation function will be described as an example of an interpolation function. In general, the bicubic interpolation function interpolates the gradation value of the target pixel based on the input image data of the corresponding 16 neighboring points. However, in the present embodiment, the calculation target is not necessarily 16 points in the vicinity, and therefore, description will be made with respect to 36 points as the vicinity pixels.

  FIG. 3 shows only a specific region of the input image and output image of FIG. 2 extracted, and a white circle 203 is a target pixel of the output image, and (i ′, j ′) = (2, 2) in FIG. Represents. Hereinafter, numerical examples in the present embodiment will be described by taking the process of calculating the gradation value of the white circle 203 in FIG. 3 as an example unless otherwise noted. Further, gradation values n (k, m) of a plurality of pixels (neighboring points) located in the vicinity of the corresponding point of the input image corresponding to the target pixel 203 of the output image are defined as shown in FIG. The gradation value n (k, m) is pixel data of the input image. k is a meaning in the vicinity of the kth data position in the horizontal direction with respect to the target pixel 203, and m is a meaning in the vicinity of the mth data position in the vertical direction with respect to the target pixel 203. In FIG. 3, k and m are values from −2 to +3, respectively, which represent 36 points near the target pixel 203, but the neighboring region is not limited to this, and is necessary for calculation. It is appropriately selected according to the region.

  The calculation target pixel data 109 output from the input image data holding unit 106 to the filter processing unit 107 in FIG. 1 is n (k, m). The input image data holding unit 106 may hold all the pixel data of the input image, or may hold only the calculation target pixel data 109 that is a part of the input image data. The distance between the target pixel 203 of the output image and each neighboring point (k, m) is d, and the horizontal distances dx−3, dx−2, dx−1, dx + 1, dx + 2, dx + 3 and the vertical distance dy−. 3, dy-2, dy-1, dy + 1, dy + 2, dy + 3 are defined as shown in FIG. In that case, since f (2, 2) = (6.6, 4.3) in FIG. 3, the distances d are as follows.

dx-3 = 2.6
dx-2 = 1.6
dx-1 = 0.6
dx + 1 = 0.4
dx + 2 = 1.4
dx + 3 = 2.4
dy-3 = 2.3
dy-2 = 1.3
dy-1 = 0.3
dy + 1 = 0.7
dy + 2 = 1.7
dy + 3 = 2.7

  In the bicubic interpolation function, a filter coefficient h (t) to be multiplied by each input pixel is represented by the following equation (1), where t is a distance from a neighboring input pixel used for the interpolation calculation.

  Here, s is a parameter for adjusting the frequency characteristic of the filter, and generally a value in the vicinity of -1.0 is used. In the present embodiment, description will be made assuming that s = −1.0. If the value of each d is put into the expression (1), the filter coefficient h (d) is as follows.

h (dx-3) = 0.000
h (dx-2) =-0.096
h (dx-1) = 0.396
h (dx + 1) = 0.744
h (dx + 2) = − 0.144
h (dx + 3) = 0.000
h (dy-3) = 0.000
h (dy-2) =-0.147
h (dy-1) = 0.847
h (dy + 1) = 0.363
h (dy + 2) = − 0.063
h (dy + 3) = 0.000

  Since dx−3, dx + 3, dy−3, and dy + 3 are all greater than 2, their filter coefficients h (d) are 0.000. As described above, when the distance (normal distance) d between the target pixel 203 and the neighboring pixel is simply used, the calculation area is substantially limited to 16 neighboring points.

  In the process of obtaining the gradation value I (i ′, j ′) of the target pixel 203 by the product sum of the filter coefficient h (d) and the neighboring pixel n (k, m), the filter coefficient h (d) is 0. When the calculation of the neighboring pixel n (k, m) is omitted, the following expression (2) is obtained.

  Expression (2) is a general arithmetic expression of the bicubic interpolation function. If the same calculation is repeated for the desired (i ′, j ′), geometric deformation can be performed. However, when the output image is reduced as shown in the example of FIG. Image degradation called moire occurs. This phenomenon will be described using a sinc function that is a function of distance x in the spatial domain as an interpolation function, as shown in the following equation (3).

  The sinc function is a rect function in the frequency domain. The bicubic interpolation function taken up in this embodiment is a cubic polynomial approximation of a sinc function, and the frequency characteristic is not a perfect rect function, but has a shape that approximately simulates the rect function. The explanation using the sinc function is to simplify the explanation of the concept.

  FIG. 4 shows an example of the spectrum of an image formed on the image sensor surface by an original signal constituting image data, for example, a camera lens. FIG. 5 shows a comb function that performs sampling at the sampling period fs. FIG. 6 shows a state where the original signal of FIG. 4 is sampled by the comb function of FIG. In FIG. 6, the band of the original signal is appropriately limited to fs / 2 or less with respect to the sampling period fs (if the sampling interval is P, fs = 1 / P). For this reason, even if the convolution of the original signal and the comb function is performed, there is no overlapping region in the spectrum of the folded original signal. FIG. 7 is a diagram showing the frequency characteristic of the sinc function as an interpolation function. When the frequency characteristic is fs / 2 or more, the input / output intensity ratio (the ratio of the frequency component between the filtered output signal and the input signal) is zero. It is a rect function. Therefore, if the rect function of FIG. 7 is multiplied by the sampling state of FIG. 6, only the spectrum of the original image can be extracted as shown in FIG. 8, and the original signal can be restored in a complete state. This is generally called the sampling theorem. Next, from the state in which the original signal is sampled as shown in FIG. 6, consider increasing the sampling pitch, that is, reducing the image. When the sampling pitch is P, the comb function representing sampling is expressed as the following equation (4), where the spatial position is x.

  This Fourier transform is expressed as the following equation (5), where the frequency is ν.

  That is, this Fourier transform is a comb function with a period of 1 / P = fs. Reducing an image at a reduction ratio α (here α <1) means that the sampling pitch is set to P / α in equation (4). Therefore, from equation (5), the comb function in the frequency domain It can be seen that the interval of is narrowed to α times. FIG. 9 shows a spectrum of a comb function that performs sampling at intervals of α times the sampling pitch P. As described above, the interval of the comb function is narrowed by α times. A process of reducing, that is, sampling again at a narrower interval from the state of FIG. 6 in which image information has already been sampled will be described. This processing is to restore the state in which the image information is sampled as shown in FIG. 6 using the rect function shown in FIG. 7 to the state shown in FIG. 8, and then perform convolution with the comb function shown in FIG. it is conceivable that. As a result, the state shown in FIG. 10 is obtained, and it can be seen that there is an overlap between the spectra of the original signals. This is aliasing caused by the reduction, and a component having a high frequency appears on the side of the low frequency component by folding, and an image component having a low frequency that should not originally exist is observed. When aliasing occurs in an image, it is generally expressed as moire, and aliasing is also referred to as moire in this specification. In order to avoid the occurrence of moiré due to image reduction, in the above reduction process, not the rect function of FIG. 7 in which the input / output intensity ratio becomes 0 (cutoff) as an interpolation function at fs / 2 or higher, but αfs / 2 or higher. The rect function of FIG. 11 that becomes 0 at the time may be used as the interpolation function. If interpolation is considered in the spatial domain, the sinc function to be subjected to the convolution operation is not sinc (x) but sinc (αx). This is equivalent to calculating the coefficient by multiplying the distance d between the target pixel 203 to be interpolated and the neighboring pixel n (k, m) by α when calculating the filter coefficient by the interpolation function. Even in an interpolation function such as a bilinear interpolation method or a bicubic interpolation method instead of an ideal sinc function, if a filter coefficient is calculated using a correction distance obtained by multiplying the distance by α, band limitation may be performed according to the reduction ratio. it can. If this method is used, the band limiting effect can be obtained much more easily than when the low pass filter processing is individually performed in accordance with the reduction ratio. Note that the degree of band limitation effect using the correction distance depends on the frequency characteristics of the interpolation function to be used.

Here, returning to the description using the bicubic interpolation function as an example again, how the correction distance is used in the present embodiment will be specifically described. The correction distance calculation means 103 in FIG. 1 calculates the correction distance d = d × α. Hereinafter, description will be made based on the example of FIG. The correction distance calculation means 103 includes correction distances Dx-3, Dx-2, Dx-1, Dx + 1, which are obtained by multiplying the distance d by the horizontal direction reduction ratio α _x = 0.75 and the vertical direction reduction ratio α _y = 0.91. Dx + 2, Dx + 3, Dy-3, Dy-2, Dy-1, Dy + 1, Dy + 2, Dy + 3 are calculated. That is, the correction distances Dx-3, Dx-2, Dx-1, Dx + 1, Dx + 2, Dx + 3, Dy-3, Dy-2, Dy-1, Dy + 1, Dy + 2, and Dy + 3 are as follows.

Dx-3 = dx-3 × α _x = 1.95
Dx-2 = dx-2 × α _x = 1.20
Dx-1 = dx-1 * [alpha] _x = 0.45
Dx + 1 = dx + 1 × α _x = 0.30
Dx + 2 = dx + 2 × α _x = 1.05
Dx + 3 = dx + 3 × α _x = 1.80
Dy-3 = dy-3 × α _y = 2.09
Dy-2 = dy-2 × α _y = 1.18
Dy-1 = dy-1 * [alpha] _y = 0.27
Dy + 1 = dy + 1 × α _y = 0.64
Dy + 2 = dy + 2 × α _y = 1.55
Dy + 3 = dy + 3 × α _y = 2.46

The correction distance calculation unit 103 calculates a plurality of distances d between a corresponding point of the input image corresponding to the target pixel 203 of the output image and a plurality of pixels located in the vicinity of the corresponding point of the input image as a local reduction rate α _x. And a plurality of correction distances D are output by performing correction using α _y . Specifically, the correction distance calculation unit 103 is a correction distance output unit, and outputs a plurality of correction distances D by multiplying the plurality of distances d by the local reduction rates α _x and α _y .

  The filter coefficient calculation unit 104 calculates a filter coefficient h (D) obtained by substituting the correction distance D into t in Expression (1) as follows.

h (Dx-3) = − 0.002
h (Dx-2) =-0.128
h (Dx-1) = 0.686
h (Dx + 1) = 0.847
h (Dx + 2) = − 0.045
h (Dx + 3) = − 0.032
h (Dy-3) = 0.000
h (Dy-2) =-0.122
h (Dy-1) = 0.871
h (Dy + 1) = 0.447
h (Dy + 2) = − 0.112
h (Dy + 3) = 0.000

  The filter coefficient calculation means 104 is a filter coefficient output means, and outputs a plurality of filter coefficients h (D) based on a plurality of correction distances D using an interpolation function. By using the correction distance D in the calculation of Expression (1), it can be seen that the neighborhood region where t <2 in Expression (1) increases in the horizontal direction. When the correction distance D is not used, the neighboring pixels to be calculated by the bicubic interpolation method are substantially 4 × 4 = 16 pixels according to the equation (1). On the other hand, when the correction distance D is used as in the present embodiment, the neighboring pixel region to be calculated changes depending on the reduction ratio, and in the present embodiment, it is substantially 6 × 4 = 24 pixels. Basically, the larger the area used for the calculation, the easier it is to obtain a favorable image quality. Therefore, in consideration of the minimum reduction ratio that can be generated by geometric transformation, the region to be interpolated, that is, the calculation target pixel data 109 is set (that is, the values of k and m are appropriately selected), and substantially In particular, it can be said that it is preferable to change the calculation target region in accordance with the reduction ratio. However, it is not always necessary to change the substantial calculation target area according to the reduction ratio, and only the calculation area used for a general interpolation function may be set as the calculation target. For example, a bicubic interpolation function is a 4 × 4 region near the target pixel, and a bilinear interpolation function is a 2 × 2 region near the target pixel. The calculation target area may be determined as appropriate from the balance between the calculation load and the band limiting effect. Further, the image quality of the output image may be adjusted by multiplying the reduction ratio calculated according to a specific definition by a constant value or adding a constant.

  Next, the filter processing means 107 will be described. The sum total of the filter coefficients h (d) calculated by the interpolation function is normally 1.0, which means that the luminance does not change by the filter. When the filter coefficient h (D) is changed according to the reduction ratio as in this embodiment, the sum of the filter coefficients h (D) may not be 1.0. For this reason, it is necessary to correct the influence caused by the sum of the filter coefficients h (D) (sum in each of the vertical and horizontal directions) not being 1.0. In the present specification, the process for correcting this influence is referred to as normalization. In the present embodiment, a method of performing a filter operation after normalizing the filter coefficient h (D) will be described. In the example of this embodiment, the total sum of the horizontal filter coefficients is 1.326, and the total sum of the vertical filter coefficients is 1.084. Therefore, the normalizing means 105 calculates the normalized filter coefficient H (D) obtained by dividing the filter coefficient by the sum of the filter coefficients h (D) in the respective directions as follows.

H (D _x-3 ) = h (D _x-3 ) /1.326=−0.002
H (D _x-2 ) = h (D _x-2 ) /1.326=−0.097
H (D _x-1 ) = h (D _x-1 ) /1.326=0.517
H (D _{x + 1} ) = h (D _{x + 1} ) /1.326=0.539
H (D _{x + 2} ) = h (D _{x + 2} ) /1.326=−0.034
H (D _{x + 3} ) = h (D _{x + 3} ) /1.326=−0.024
H (D _y-3 ) = h (D _y-3 ) /1.084=0.000
H (D _y−2 ) = h (D _y−2 ) /1.084=−0.113
H (D _y-1 ) = h (D _y-1 ) /1.084=0.804
H (D _{y + 1} ) = h (D _{y + 1} ) /1.084=0.612
H (D _{y + 2} ) = h (D _{y + 2} ) /1.084=−0.103
H (D _{y + 3} ) = h (D _{y + 3} ) /1.084=0.000

  In this embodiment, the filter calculation unit 111 performs a filter calculation of the normalized filter coefficient H (D) and the neighboring point n (k, m) and outputs an output image gradation value 108. Note that the normalization itself is not limited to the method of performing the filtering process after calculating the normalized filter coefficient H (D). For example, after calculating the gradation value by the product-sum operation of the unnormalized filter coefficient h (D) and the calculation target pixel data 109, the calculated gradation value is summed with the filter coefficients in the horizontal and vertical directions. An equivalent result can also be obtained by processing such as dividing by the product of. Which method should be adopted may be selected as appropriate. In the present embodiment, it is possible that the sum of the normalized filter coefficient in a certain direction does not become exactly 1, for example, by the process of truncating the digits when calculating the filter coefficient, but up to how many significant digits are used. This may be determined as appropriate depending on the balance between the calculation load and the image quality.

  For example, in the interpolation calculation of (i ′, j ′) = (2, 2), the calculation target area in the horizontal direction is expanded by the reduction, and the gradation value I (2, 2) is normalized by the interpolation. Using the coefficient H (D), it is expressed as the following equation (6).

  The filter calculation unit 111 performs the filter calculation of the normalized filter coefficient H (D) and the neighboring point n (k, m) as shown in the equation (6), so that the target pixel position (i ′, j ′) = (2 , 2) is calculated, and the gradation value I (2, 2) is output as the output image gradation value 108.

  As described above, the filter coefficient calculation unit 104 outputs a plurality of filter coefficients h (D). The filter processing unit 107 filters the data of a plurality of pixels located in the vicinity of the corresponding point (203) of the input image using the plurality of filter coefficients h (D), so that the target pixel of the output image Output the data. At that time, the filter processing means 107 performs a filter process normalized by the sum of a plurality of filter coefficients h (D). Specifically, the normalizing unit 105 outputs a plurality of normalized filter coefficients H (D) obtained by dividing the plurality of filter coefficients h (D) by the sum of the plurality of filter coefficients h (D). The filter calculation unit 111 calculates the data of the pixel of interest of the output image by the product-sum operation of the plurality of normalized filter coefficients H (D) and the data of the plurality of pixels located in the vicinity of the corresponding point (203) of the input image. Output.

  FIG. 12 is a diagram illustrating the horizontal frequency characteristics of the filter used in the interpolation calculation of (i ′, j ′) = (2, 2) in the present embodiment. A solid line plot 1201 of the normal distance d is a horizontal frequency characteristic of the filter coefficient h (d) obtained by a normal bicubic interpolation function that does not correct the distance by the reduction ratio. A dotted line plot 1202 of the correction distance D is a horizontal frequency characteristic of the normalized filter coefficient H (D). In FIG. 12, the normalized angular frequency is a frequency when the sampling frequency is 2π rad. From the plot of FIG. 12, the filter characteristic represented by the dotted line plot 1202 of the correction distance D is more low-pass filter characteristic than the filter characteristic represented by the normal distance d solid line plot 1201 that does not consider the reduction ratio. You can see that Furthermore, if the same processing is repeated for the required i ′ and j ′ (in this embodiment, i ′ = 3, j ′ = 4), an output image having undergone geometric deformation is output. Can be obtained.

  According to the present embodiment, when performing geometric deformation of an image with a non-uniform reduction ratio in the image, a simple technique is used to appropriately limit the band according to the local reduction ratio and reduce moire. However, image blur can be minimized.

  The image subjected to the above-described geometric deformation is displayed by a head mounted display (HMD), an electronic viewfinder (EVF) of a digital camera, or a projector. By performing geometric deformation, distortion due to the display optical system of the HMD or EVF, or distortion due to the installation location of the projector can be reduced.

  The present embodiment can be realized by hardware such as an electronic circuit board that implements the functions described in the present embodiment. The present embodiment can also be realized by creating a program code of software that realizes the functions described in the present embodiment and executing the program code stored in a storage medium storing the program code by a computer. Can do. In that case, hardware such as an electronic circuit board or the program code itself constitutes the present embodiment.

(Second Embodiment)
In the second embodiment of the present invention, an example in which the first embodiment is applied to display distortion correction of an HMD will be described. The HMD is a device that mounts a small display (liquid crystal, organic EL, or the like) and presents an enlarged virtual image to an observer by a display optical system. In this embodiment, the display resolution of a small display will be described as a horizontal direction 1280 and a vertical direction 960. In addition, the HMD of the present embodiment employs an eccentric optical system as a display optical system in order to reduce the size of the entire HMD, and rotationally asymmetric distortion remains.

  FIG. 13 is a diagram illustrating an example of the input image 301. The number of pixels of the input image 301 is 1280 in the horizontal direction and 960 in the vertical direction. When the grid-like input image 301 shown in FIG. 13 is directly displayed on a small display, the observer observes a distorted grid due to distortion of the display optical system.

  FIG. 14 is a conceptual diagram showing an example of an output image 302 that should be created for display distortion correction. The output image 302 is an output image obtained by performing geometric deformation on the input image 301 in order to cancel distortion of the display optical system. An area 303 indicates the entire image data of 1280 × 960 size displayed on the small display. It can be seen that the output image 302 includes a non-uniformly reduced place with respect to the input image 301, and the vertical grid interval is remarkably reduced particularly in the vicinity of the upper left and right corners of the output image 302. When performing such geometric deformation, if the image processing apparatus of the first embodiment is applied, distortion correction with less blurring of the image can be realized while reducing moire. In the display distortion correction, for example, a program code of software that realizes the function described as the first embodiment is created, and geometric conversion is performed based on an instruction of the program code stored in the storage medium on which the program code is recorded. It is realizable using the computer which performs. Display distortion correction can be performed by displaying the output image on a small display, but the present invention is not necessarily limited thereto.

  In the present embodiment, an effect obtained when the first embodiment is applied to display distortion correction using a bicubic interpolation function with s = −0.5 in Expression (1) will be described. The local reduction ratios in the image vertical direction at the positions of the intersections 304, 305, and 306 of the lattice shown in FIG. 14 are 0.635, 0.763, and 0.996, respectively.

  FIG. 15 is a diagram illustrating frequency characteristics in the image vertical direction of the filter when the luminance values at the respective positions of the intersections 304, 305, and 306 are calculated by the image processing apparatus according to the first embodiment. From the plot of FIG. 15, it can be seen that the smaller the reduction rate value, the higher the band limiting effect. As described above, when the first embodiment is applied to the display distortion correction of the image observation apparatus, the observer can appropriately limit the band in accordance with the local reduction ratio, reduce the moire, and blur the image. It is also possible to observe an image with minimal distortion and low distortion.

  The example of the head mounted display has been described above, but the same applies to the electronic viewfinder. In that case, the local reduction ratio calculation means 102 outputs a local reduction ratio for correcting distortion of the display optical system of the head mounted display or the electronic viewfinder.

(Third embodiment)
In the third embodiment of the present invention, an example in which the first embodiment is applied to keystone correction of a projector will be described. The projector is a device that mounts a small display (liquid crystal, organic EL, etc.) and projects an enlarged image thereof onto a screen by a projection optical system. In the present embodiment, the display resolution of a small display will be described as a horizontal direction 1920 and a vertical direction 1080.

  FIG. 16 is a diagram illustrating an example of the input image 401. The number of pixels of the input image 401 is 1920 in the horizontal direction and 1080 in the vertical direction. When the grid-like input image 401 shown in FIG. 16 is directly displayed on a small display, keystone distortion occurs depending on the positional relationship between the projector and the screen. In this case, the observer observes the distorted grid.

  FIG. 17 is a conceptual diagram illustrating an example of an output image 402 that should be created in order to perform keystone distortion correction when the positional relationship between the projector and the screen is not directly opposed. The output image 402 is an output image obtained by performing geometric deformation on the input image 401 in order to cancel the keystone distortion. An area 403 indicates the entire image data of 1920 × 1080 size displayed on the small display. It can be seen that the output image 402 includes a locally reduced place with respect to the input image 401, and the lattice spacing is significantly reduced particularly near the right end of the output image 402. If the image processing apparatus of the first embodiment is applied when performing such geometric deformation, keystone correction with less blurring of the image can be realized while reducing moire. In the keystone distortion correction, for example, an electronic circuit board that realizes the function described as the first embodiment is created, the electronic circuit board is incorporated into the electronic circuit board of the projector, and an output image is displayed on a small display, thereby displaying the keystone. Correction can be performed. However, it is not necessarily limited to this.

  In the present embodiment, an effect when the first embodiment is applied to the keystone correction using the bicubic interpolation function with s = −0.5 in Expression (1) will be described. The local reduction ratios in the image horizontal direction at the intersections 404, 405, and 406 of the lattice shown in FIG. 17 are 0.460, 0.643, and 0.800, respectively.

  FIG. 18 is a diagram illustrating the frequency characteristics of the filter in the horizontal direction of the image when the luminance values at the respective positions of the intersections 404, 405, and 406 are calculated by the image processing apparatus according to the first embodiment. From the plot of FIG. 18, it can be seen that the smaller the reduction rate value, the higher the band limiting effect. As described above, when the first embodiment is applied to the keystone correction of the projector, the observer can appropriately limit the band according to the local reduction ratio, has less moire, and minimizes image blurring. An image with less distortion and less distortion can be observed.

  In the present embodiment, the local reduction ratio calculation unit 102 outputs a local reduction ratio for correcting distortion of an image projected on the screen by the projector, which occurs according to the position of the projector with respect to the screen.

  According to the first to third embodiments, the image processing apparatus can be industrially used for an apparatus such as a head mounted display, an electronic viewfinder, or a projector.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Deformation corresponding point calculation means 102 Local reduction rate calculation means 103 Correction distance calculation means 104 Filter coefficient calculation means 105 Normalization means 106 Input image data holding means 107 Filter processing means 111 Filter Calculation means

Claims (8)

An image processing apparatus for generating an output image having a locally different reduction ratio with respect to an input image,
A local reduction rate output means for outputting a local reduction rate in the vicinity of the target pixel of the output image with respect to the input image;
By correcting a plurality of distances between a corresponding point of the input image corresponding to the target pixel of the output image and a plurality of pixels located in the vicinity of the corresponding point of the input image using the local reduction ratio. Correction distance output means for outputting a plurality of correction distances;
Filter coefficient output means for outputting a plurality of filter coefficients based on the plurality of correction distances;
A filter that outputs data of a pixel of interest of the output image by performing filter processing normalized with the sum of the plurality of filter coefficients on data of a plurality of pixels located in the vicinity of the corresponding point of the input image And an image processing apparatus. The filter processing means includes
Normalizing means for outputting a plurality of normalized filter coefficients obtained by dividing the plurality of filter coefficients by the sum of the plurality of filter coefficients, respectively;
Filter operation means for outputting data of a pixel of interest of the output image by product-sum operation of the plurality of normalized filter coefficients and data of a plurality of pixels located in the vicinity of the corresponding point of the input image. The image processing apparatus according to claim 1, wherein:   The image processing apparatus according to claim 1, wherein the correction distance output unit outputs the plurality of correction distances by multiplying the plurality of distances by the local reduction ratio.   The image processing apparatus according to claim 1, wherein the filter coefficient output unit outputs the plurality of filter coefficients using an interpolation function.   The said local reduction rate output means outputs the said local reduction rate for correcting the distortion of the display optical system of a head mounted display or an electronic viewfinder, The any one of Claims 1-4 characterized by the above-mentioned. Image processing apparatus.   2. The local reduction rate output unit outputs the local reduction rate for correcting distortion of an image projected on the screen by the projector, which is generated according to a position of the projector with respect to the screen. The image processing apparatus according to any one of? An image processing method for generating an output image having a locally different reduction ratio with respect to an input image,
A local reduction rate output step of outputting a local reduction rate in the vicinity of the target pixel of the output image with respect to the input image by a local reduction rate output means;
The correction distance output means calculates a plurality of distances between a corresponding point of the input image corresponding to the target pixel of the output image and a plurality of pixels located in the vicinity of the corresponding point of the input image, and the local reduction rate. A correction distance output step for outputting a plurality of correction distances by using and correcting,
A filter coefficient output step of outputting a plurality of filter coefficients based on the plurality of correction distances by a filter coefficient output means;
By performing filtering processing normalized by the sum of the plurality of filter coefficients on the data of a plurality of pixels located in the vicinity of the corresponding point of the input image by the filter processing unit, the target pixel of the output image And a filter processing step for outputting data.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-6.

Images