JP2005063323A - Image modification processing method and apparatus, program, and data recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image modification processing method and apparatus, program, and data recording capable of performing image modification processing considering that the shape of blur is different for each pixel and further enhancing the image modification effect. To provide a medium.
Convolution operation matrix storage means for a value of a matrix part including at least a non-zero element among elements Q _{m, n} (x, y) of a convolution operation matrix Q _{m, n} for each coordinate (m, n). 32 and 33, and when the subject is imaged by the bifocal lens 21, the value of the output signal Z (m + x, n + y) of the image element 24 and the convolution calculation matrix storage unit 32, The value of each element A (m, n) of the subject matrix A is calculated by performing convolution operation processing using Q _{m, n} (x, y) stored in 33, and a focused image is obtained. I made it.
[Selection] Figure 1

Description

  The present invention relates to an image modification processing method and apparatus, program, and data recording medium for modifying an in-focus image formed by a lens into an in-focus image, and configures, for example, a bifocal lens. Image modification processing for obtaining an in-focus image from an image in which an in-focus image formed by one lens unit overlaps an in-focus image formed by the other lens unit, or single focus It can be used for image modification processing that removes lens blur.

  Conventionally, a bifocal lens having two lens portions having different focal lengths has been used as a bifocal contact lens. When a human wears a contact lens composed of such a bifocal lens, a human can see an in-focus image and a non-focused image (so-called out-of-focus image) formed by the two lens portions. It is thought that the user selects unconsciously and sees only the image in focus.

  By the way, if such a bifocal lens is provided in an information terminal device such as a mobile phone or a portable information terminal, for example, a normal distance that is at a standard distance from the lower limit of focal depth (for example, 0.3 m) to infinity. A subject (for example, a person or a landscape) is imaged by a long focal length lens unit having a long focal length, while a close subject (for example, a two-dimensional barcode, an iris, a character, or the like) disposed at a shorter distance. Are picked up by a short-focus lens unit having a short focal length, whereby high-resolution images can be obtained. And the information terminal device provided with such a bifocal lens has already been developed by the inventor of the present application (see Patent Document 1).

  However, in the information terminal device including the bifocal lens, for example, an optical shutter such as a liquid crystal shutter is provided between the bifocal lens and the image sensor, so that the long focal lens portion and the short focal lens portion can be switched. In such a case, a high-contrast image can be obtained. However, when such a lens unit is not switched, the focused image formed by the two lens units is not in focus. Since the image overlaps, there is a problem that the image becomes unclear.

  At this time, when a human wears a bifocal contact lens using a bifocal lens as described above, the human unconsciously selects an in-focus image and an out-of-focus image. Although it is considered that only matching images are viewed, a process similar to the image selection process in the human brain is applied to a portable information terminal device such as a normal cellular phone or a portable information terminal. If it can be executed in a short time by a central processing unit (CPU) having the performance of the installed level, the convenience and performance of the information terminal device can be improved, and it can be realized at low cost, which is convenient.

  In addition, if such processing can be performed, not only when a bifocal lens is provided in a portable information terminal device such as a mobile phone or a portable information terminal, but generally when imaging with a bifocal lens is performed. For example, when a camera having a bifocal lens is connected to a personal computer, or when a bifocal lens is used as a surveillance camera, the image quality can be improved, which is advantageous.

  Therefore, the inventor of the present application can improve the quality of the image picked up by the bifocal lens in a short time, and without using a focusing mechanism, a normal subject at a standard distance and the like. There has already been proposed an image modification processing apparatus that can obtain a clear image for any of the close subjects at a short distance (see Japanese Patent Application No. 2003-31734).

JP 2002-123825 A (paragraphs [0007], [0008], [0027], [0042], FIG. 3, summary)

  By the way, in the image modification processing apparatus proposed in Japanese Patent Application No. 2003-31734 described above, the image modification process is performed on the assumption that blur of the same shape occurs in all pixels. That is, in a state where light emitted from an arbitrary point on the subject spreads on the image sensor by the action of the bifocal lens (blurred shape), light emitted from another point on the subject is imaged by the action of the bifocal lens. It is assumed that it is the same as the state of spreading on the element (blurred shape).

  However, in the actual imaging with the bifocal lens, blur of the same shape does not occur in all pixels. Therefore, in order to further enhance the image modification effect, it is necessary to perform processing in consideration of the occurrence of blurring with different shapes in each pixel.

  Also, not only when imaging with a bifocal lens, but also when imaging with a single focus lens, all pixels are not completely focused, and there are pixels that are out of focus, and each pixel The shape of the blur is usually different. Therefore, it would be convenient if such defocusing due to a single focus lens could be removed.

  An object of the present invention is to perform an image modification process considering that the shape of blur is different for each pixel, and to further enhance the image modification effect, an image modification processing method, an apparatus thereof, a program, And providing a data recording medium.

The present invention relates to an image modification processing method for modifying an in-focus image formed by a lens into an in-focus image, and the size of an image sensor is M pixels × N pixels, and light emitted from a subject. A matrix of M rows and N columns indicating the brightness of the image is A, and a matrix of M rows and N columns indicating the output signal of the image obtained by imaging the subject with the lens is Z, and one point (m, When the subject coordinate system and the image coordinate system are set so that the imaging position of the light emitted from n) is one point (m, n) in the image coordinate system, the coordinates (m, n) for performing the convolution calculation process ) Is a value of a matrix part including at least a non-zero element among the values of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} of (2M-1) rows (2N-1) columns. The whole or a part of each coordinate (m, n) is calculated in advance based on the following formula (A-1) and is folded. Is stored in the viewing operation matrix storage means, when captured by the lens of a subject, the playback operation means, each element stored in the convolution matrix storage unit _{Q m, n (x, y} ) at least one of The value of each element A (m, n) of the matrix A of the subject based on the following equation (A-2) using the value of the part and the value of each element Z (m + x, n + y) of the matrix Z of the output signal of the image Is calculated.

Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （A-1）

A (m, n) = Σ x Σ y Q m, n (x, y) Z (m + x, n + y)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （A-2）

Here, x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ (N−1), m and n are natural numbers, and 1 ≦ m ≦ M, 1 ≦ n ≦ N, and W _{m, n} (x, y) is a value of each element of the matrix W _{m, n} of (2M−1) rows (2N−1) columns. W _{m, n} is a point spread function matrix indicating a state in which light emitted from one point (m, n) of the subject spreads on the image sensor due to the action of the lens, and W _{m, n} (0, 0) Is the value of the output signal of the pixel located in the center part of the spread, W _{m, n} (−x, −y) is the value of the output signal of the pixel located in the surrounding blur part, and power is W _m, a real number which is a power number n (0,0), a 1 ≦ power ≦ 2, the sigma _x, the sum of x = (1-M) ~ (M-1), Σ y is , Y = (1-N) to (N-1) Is the sum.

  Further, the “lens” may be a bifocal lens or a single focal lens.

  Furthermore, specifically, for example, a complementary metal oxide semiconductor (CMOS), a charge coupled device (CCD), or the like can be employed as the “imaging device”.

The power value may be determined according to the value of W _{m, n} (0, 0). At this time, if the value of W _{m, n} (0, 0) is in the vicinity of 0.5 (including 0.5; the same applies hereinafter), the power value needs to be a value other than 2. More specifically, when the value of W _{m, n} (0, 0) is in the vicinity of 0.5, the power value is set to 1 or more and less than 2, more preferably 1. On the other hand, when the value of W _{m, n} (0, 0) is not near 0.5, it is set to 1 or more and 2 or less. The same applies to the following inventions.

Further, “all or part of each coordinate (m, n)” means that the convolution calculation matrix Q _{m, n} stored in the convolution calculation matrix storage means is convolution for all coordinates (m, n). The calculation matrix Q _{m, n} may be used, or only the convolution calculation matrix Q _{m, n} for some coordinates (m, n) may be used. However, when only the convolution calculation matrix Q _{m, n} for some coordinates (m, n) is stored, it is based on the convolution calculation matrix Q _{m, n} for some of the stored coordinates (m, n). It is assumed that the convolution matrix Q _{m, n} for all coordinates (m, n) can be obtained by calculation.

Further, “value of matrix part including at least non-zero element among values of each element Q _{m, n} (x, y) of convolution arithmetic matrix Q _{m, n} of (2M−1) rows (2N−1) columns” Means that when the convolution calculation matrix Q _{m, n} for the coordinates (m, n) is stored in the convolution calculation matrix storage means, the values of all (2M−1) × (2N−1) elements are set. It is not necessary to memorize, and it is only necessary to memorize the values of the matrix part including the non-zero elements. Incidentally, as described above, in formula (A-2), Σ _x is the sum of x = (1-M) ~ (M-1), Σ y is, y = (1-N) ~ (N -1) have been described as the sum of which is the convolution matrix storage unit (2M-1) lines (2N-1) row convolution matrix Q _m, each element of the _n Q _{m, n} This means processing when all the values of (x, y) are stored. When only a partial value (a value of a matrix portion including a non-zero element) is stored, the sum of the stored values is stored. And it is sufficient. The same applies to the following inventions.

In such an image modification processing method of the present invention, each element Q _{m, n} of the convolution calculation matrix Q _{m, n} stored in the convolution calculation matrix storage means by the reproduction calculation means when the subject is imaged by the lens _{. Using} the value of at least a part of _n (x, y) and the value of each element Z (m + x, n + y) of the matrix Z of the output signal of the image obtained by imaging, the equation (A-2) ) To calculate the value of each element A (m, n) of the subject matrix A.

At this time, the calculation processing by the reproduction calculation means is performed by calculating each element Q _{m, n} (x, x, _n ) of the convolution calculation matrix Q _{m, n} calculated in advance based on the equation (A-1) and stored in the convolution calculation matrix storage means. Since the calculation is performed using the value of y), the subject matrix A can be obtained with a very small amount of calculation compared to the case where the arithmetic processing is performed using the large inverse transformation matrix T _g ⁻¹ of M × N rows and M × N columns. It is possible to calculate the value of each element A (m, n).

That is, the number of elements in the subject matrix A is M × N, and the number of elements in the matrix Z of the output signal of the image is also M × N. Therefore, when it is considered that light emitted from each coordinate corresponding to each element of the matrix A of the subject affects the pixels corresponding to each element of the matrix Z of the output signal of the image, this correspondence relationship (subject matrix The relationship of deriving the value of each element of the matrix Z of the output signal of the image from the value of each element of A can be shown using a giant transformation matrix T _g of M × N rows and M × N columns. _If the inverse matrix T _g ⁻¹ of the giant transformation matrix T _g of × N rows and M × N columns can be obtained, this giant inverse transformation matrix T _g ⁻¹ is used to reverse the matrix Z of the output signal of the image. The value of each element of the subject matrix A can be derived from the value of each element. However, the arithmetic processing using the M × N rows and M × N giant inverse transformation matrix T _g ⁻¹ has a large amount of calculation, and thus, for example, a portable information terminal device such as a normal cellular phone or a portable information terminal. In a central processing unit (CPU) having a level of performance mounted on the CPU, processing in a short time is difficult, so it is not realistic.

  On the other hand, in the present invention, the calculation processing by the reproduction calculation means has a very small calculation amount, so that the performance condition required for the CPU is relaxed. Therefore, it is possible to execute the processing in a short time even with the capability of the CPU mounted on a portable information terminal device such as a cellular phone or a portable information terminal. Therefore, if the present invention is applied to a portable information terminal device, a portable information terminal device having an image modification function can be realized, and the usability and performance of the information terminal device can be improved. Become.

In addition, since the convolution calculation matrix storage means stores the convolution calculation matrix Q _{m, n} for each coordinate (m, n), it is considered that the blur shape differs for each pixel by the reproduction calculation means. Image modification processing can be performed. For this reason, the image modification effect can be further enhanced as compared with the case of performing the image modification process assuming that the same shape blur occurs in all the pixels, and the above object is achieved by these.

In the image-modifying method described above, the convolution matrix Q _m for each coordinate (m, _n), out of _n, convolution matrix for coordinates aligned on a straight line extending from the optical axis position in one direction Q _{m , n} are selected as sampling matrices, and the convolution calculation matrix storage means stores only the values of the elements Q _{m, n} (x, y) of the sampling matrix Q _{m, n} and convolves other coordinates. The value of each element Q _{m, n} (x, y) of the operation matrix Q _{m, n} is obtained by using the axial symmetry of the lens by the convolution operation matrix rotation calculation means to calculate each element Q _{m of the} sampling matrix Q _{m, n. , n} (x, y) is preferably calculated by rotating the arrangement of the values about the optical axis position.

In this way, when only the sampling matrix Q _{m, n} is stored in the convolution calculation matrix storage means and the other convolution calculation matrix Q _{m, n} is calculated by rotating the sampling matrix Q _{m, n} , It is possible to reduce the amount of data stored in the convolution calculation matrix storage means.

Further, as described above, when only the sampling matrix Q _{m, n} is stored in the convolution calculation matrix storage means, and the other convolution calculation matrix Q _{m, n} is calculated by rotating the sampling matrix Q _{m, n} When calculating the value of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} for other coordinates by the convolution matrix rotation calculation means, each sampling matrix Q _{m, n} The first partition region corresponding to the element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} to be calculated when the arrangement of the values of the element Q _{m, n} (x, y) is rotated The second divided area corresponding to the element Q _{m, n} (x, y) of the rotated sampling matrix Q _{m, n} that overlaps the predetermined position is obtained, and the element Q _{m, n} ( x, the value of y), the calculation target convolution matrix Q _{m, n} elements Q _{m of, n} (x, To be adopted as a value of), or obtains a first divided region of a predetermined position overlaps the second divided area, the elements corresponding to the first divided region obtained Q _m, as the value of _n (x, y) It is desirable to adopt the value of the element Q _{m, n} (x, y) corresponding to the second partitioned area whose predetermined position overlaps the first partitioned area.

Here, the “predetermined position of the first partitioned area” and the “predetermined position of the second partitioned area” are respectively the first from the viewpoint of bringing the value of each element Q _{m, n} (x, y) to be obtained closer to the exact value. It is preferable to set the center position of the partition area and the center position of the second partition area, but is not limited to this, for example, the corners and sides of the first partition area and the second partition area, etc. Also good.

Thus seek second compartment area or first divided areas predetermined position overlaps the second divided region, overlaps the predetermined position of the first divided area, the sampling matrix Q _m, other than the _n convolution matrix Q _{m, When} the value of each element Q _{m, n} (x, y) of _n is calculated and determined, it corresponds to each second partitioned area based on the area ratio of each second partitioned area overlapping the first partitioned area. When the value of the element Q _{m, n} (x, y) corresponding to the first partition region is strictly calculated by weighted averaging the value of each element Q _{m, n} (x, y) (see FIG. 10) In comparison, the processing contents are simplified, and the processing time can be shortened.

In the image modification processing method described above, the lens is a bifocal lens, and the matrix Z is a focused image formed by one lens portion constituting the bifocal lens and the other lens. is a matrix showing the output signal of the blurred image and is overlapped image of the focus formed by the section, point-spread function matrix W _{m, n} are, W _{m, n} (0 by the action of mainly one lens portion , 0), and the value of W _{m, n} (−x, −y) may be determined mainly by the action of the other lens unit.

  Here, the “bifocal lens” is for imaging a normal subject (for example, a person or a landscape) at a standard distance (a distance from the lower limit of focal depth (for example, 0.3 m) to infinity). A long focal length lens unit having a long focal length, and a short focal length for imaging a close subject (for example, a two-dimensional barcode, an iris, or a character) that is closer than a standard distance subject The short focus lens unit is an imaging lens formed integrally on the same surface. In addition, the long focus lens portion and the short focus lens portion may be formed separately by separate members and then integrated, or the long focus lens portion and the short focus lens portion are formed by using one member. May be. The same surface is most preferably a surface orthogonal to the optical axis of the imaging lens.

  In addition, “one lens unit” is a lens unit that forms an in-focus image. When a normal subject at a standard distance is imaged, a long-focus lens unit corresponds to the normal lens unit. The short-focus lens unit corresponds to imaging a close subject that is closer than the subject. On the other hand, the “other lens unit” is a lens unit that forms a blurred image, and when capturing a normal subject at a standard distance, the short-focus lens unit corresponds, When imaging a close subject that is closer than a normal subject, a long-focus lens unit is applicable.

  Furthermore, the front shape (the shape seen from the direction along the optical axis of the lens) of the long-focus lens part constituting the “bifocal lens” is any one of a circle, an ellipse, and a polygon. The front shape of the lens is annular, and the short focus lens portion is arranged outside the long focus lens portion and is concentric with the long focus lens portion (arranged so that the optical axes of the lens portions coincide with each other). It is desirable from the viewpoints of simplification of the structure, ease of manufacture, easy acquisition of high-quality images, improvement of decoding accuracy, and the like. When a concentric structure having an annular front surface of the focus lens portion is obtained, a very favorable result can be obtained. Note that the arrangement relationship between the long focus lens portion and the short focus lens portion may be reversed (therefore, the front shape is reversed), and the short focus lens portion may be disposed inside and the long focus lens portion disposed outside.

The value of W _{m, n} (0,0) is determined mainly by the action of one lens part, and the value of W _{m, n} (−x, −y) is determined mainly by the action of the other lens part. The meaning is that a slight blur is formed by the action of one lens part that forms an in-focus image, which may affect the value of W _{m, n} (−x, −y), and Depending on the front shape (in other words, the shape of the blur) of the other lens unit that forms a defocused image, the defocus due to the action of the other lens unit affects the value of W _{m, n} (0, 0). It means that there are things.

  Thus, when the present invention is applied to a bifocal lens, an in-focus image formed by one lens portion constituting the bifocal lens and a defocused image formed by the other lens portion. It is possible to perform image modification processing that obtains an in-focus image from images that overlap with each other in a short time, and without using a focusing mechanism, a normal subject at a standard distance and more than this A clear image can be obtained for any of the close subjects at a short distance.

  Examples of the image modification processing apparatus for realizing the image modification processing method of the present invention described above include the following image modification processing apparatus of the present invention.

That is, the present invention is an image modification processing device that modifies a blurred image formed by a lens into a focused image, wherein the size of the image sensor is M pixels × N pixels, A matrix of M rows and N columns indicating the brightness of emitted light is A, and a matrix of M rows and N columns indicating an output signal of an image obtained by imaging the subject is Z, and one point ( When the subject coordinate system and the image coordinate system are set so that the imaging position of the light emitted from m, n) is one point (m, n) in the image coordinate system, all of the coordinates (m, n) or For some of the coordinates (m, n) for performing the convolution operation processing calculated based on the following equation (B-1), a convolution operation matrix Q _m, of (2M−1) rows (2N−1) columns _. each element Q _m of _{n, n (x,} y) values of the matrix portion containing at least non-zero elements of the value of A storage for convolution matrix storage unit, the convolution matrix elements Q _m stored in the storage _{unit, n} (x, y) each element Z of the matrix Z of at least some of the values and the image of the output signal of the Reproduction calculation means for calculating the value of each element A (m, n) of the matrix A of the subject based on the following formula (B-2) using the value of (m + x, n + y). Is.

Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （B-1）

A (m, n) = Σ x Σ y Q m, n (x, y) Z (m + x, n + y)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （B-2）

Here, x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ (N−1), m and n are natural numbers, and 1 ≦ m ≦ M, 1 ≦ n ≦ N, and W _{m, n} (x, y) is a value of each element of the matrix W _{m, n} of (2M−1) rows (2N−1) columns. W _{m, n} is a point spread function matrix indicating a state in which light emitted from one point (m, n) of the subject spreads on the image sensor due to the action of the lens, and W _{m, n} (0, 0) Is the value of the output signal of the pixel located in the center part of the spread, W _{m, n} (−x, −y) is the value of the output signal of the pixel located in the surrounding blur part, and power is W _m, a real number which is a power number n (0,0), a 1 ≦ power ≦ 2, the sigma _x, the sum of x = (1-M) ~ (M-1), Σ y is , Y = (1-N) to (N-1) Is the sum.

  In such an image modification processing apparatus of the present invention, the actions and effects obtained by the above-described image modification processing method of the present invention can be obtained as they are, thereby achieving the object.

In the image modification processing apparatus described above, the convolution calculation matrix storage means stores coordinates arranged on a straight line extending in one direction from the optical axis position in the convolution calculation matrix Q _{m, n} for each coordinate (m, n). convolution matrix Q _{m on, n} is selected in the sampling matrix Q _m, each element Q _m of _{n, n (x,} y) only the value of are stored as a sampling matrix, convolution matrix for other coordinates Q _m, each element Q _m of _{n, n (x,} y) the value of, by using the axial symmetry of the lens, the sampling matrix Q _m, each element Q _m of _{n, n (x,} y) values It is desirable to have a configuration provided with a convolution calculation matrix rotation calculating means for calculating the arrangement by rotating the arrangement around the optical axis position.

Furthermore, in the case of the configuration including the convolution calculation matrix rotation calculation unit as described above, the convolution calculation matrix rotation calculation unit calculates the value of each element Q _{m, n} (x, y) of the sampling matrix Q _{m, n} . When the arrangement is rotated, the rotated sampling matrix Q _m, which overlaps with a predetermined position of the first partition region corresponding to the element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} to be calculated _{. A} second partitioned area corresponding to the element Q _{m, n} (x, y) of _n is obtained, and the value of the element Q _{m, n} (x, y) corresponding to the obtained second partitioned area is convolved as a calculation target. The first partitioned area obtained by obtaining the first partitioned area that is adopted as the value of the element Q _{m, n} (x, y) of the operation matrix Q _{m, n} or the predetermined position of the second partitioned area overlaps As a value of an element Q _{m, n} (x, y) corresponding to the second section whose predetermined position overlaps the first section area It is desirable that the value of the element Q _{m, n} (x, y) corresponding to the region is adopted.

In the image modification processing apparatus described above, the lens is a bifocal lens, and the matrix Z is a focused image formed by one lens portion constituting the bifocal lens and the other lens. is a matrix showing the output signal of the blurred image and is overlapped image of the focus formed by the section, point-spread function matrix W _{m, n} are, W _{m, n} (0 by the action of mainly one lens portion , 0) is determined, and the value of W _{m, n} (−x, −y) may be determined mainly by the action of the other lens unit.

Further, the present invention is a program for causing a computer to function as an image modification processing device that modifies a blurred image formed by a lens into a focused image. M pixels × N pixels, M rows and N columns matrix indicating the brightness of light emitted from the subject is A, and M rows and N columns matrix indicating the output signal of the image obtained by imaging the subject with the lens are Z When the subject coordinate system and the image coordinate system are set so that the imaging position of light emitted from one point (m, n) in the subject coordinate system is one point (m, n) in the image coordinate system, For all or part of the coordinates (m, n), (2M-1) rows (2N-1) for the coordinates (m, n) for performing the convolution calculation processing calculated based on the following equation (C-1) ) convolution matrix Q _{m columns,} each element of the _{n Q m, n (x,} y) of At least a convolution operation matrix storage means for storing the values of the matrix portion containing a non-zero elements, each element Q _m stored in the convolution matrix storage _{unit, n} (x, y) of at least a portion of one of the The value of each element A (m, n) of the matrix A of the subject is calculated based on the following equation (C-2) using the value and the value of each element Z (m + x, n + y) of the matrix Z of the output signal of the image. An image modification processing apparatus characterized by comprising a reproduction calculation means for causing a computer to function.

Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （C-1）

A (m, n) = Σ x Σ y Q m, n (x, y) Z (m + x, n + y)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （C-2）

Here, x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ (N−1), m and n are natural numbers, and 1 ≦ m ≦ M, 1 ≦ n ≦ N, and W _{m, n} (x, y) is a value of each element of the matrix W _{m, n} of (2M−1) rows (2N−1) columns. W _{m, n} is a point spread function matrix indicating a state in which light emitted from one point (m, n) of the subject spreads on the image sensor due to the action of the lens, and W _{m, n} (0, 0) Is the value of the output signal of the pixel located in the center part of the spread, W _{m, n} (−x, −y) is the value of the output signal of the pixel located in the surrounding blur part, and power is W _m, a real number which is a power number n (0,0), a 1 ≦ power ≦ 2, the sigma _x, the sum of x = (1-M) ~ (M-1), Σ y is , Y = (1-N) to (N-1) Is the sum.

  The above-mentioned program or a part thereof is, for example, a magneto-optical disk (MO), a read-only memory (CD-ROM) using a compact disk (CD), a CD recordable (CD-R), a CD rewritable (CD -RW), read-only memory (DVD-ROM) using digital versatile disk (DVD), random access memory (DVD-RAM) using DVD, flexible disk (FD), magnetic tape, hard disk, It can be recorded on storage media such as read-only memory (ROM), electrically erasable and rewritable read-only memory (EEPROM), flash memory, and random access memory (RAM) for storage and distribution. And, for example, a local area network (LA ), A metropolitan area network (MAN), a wide area network (WAN), a wired network such as the Internet, an intranet, or an extranet, or a wireless communication network, or a combination thereof. It is also possible to carry it on a carrier wave. Furthermore, the above program may be a part of another program, or may be recorded on a recording medium together with a separate program.

Furthermore, the present invention is a computer-readable data recording medium that records data used in an image modification process for modifying a blurred image formed by a lens into a focused image. The size of the element is M pixels × N pixels, the matrix of M rows and N columns indicating the brightness of light emitted from the subject is A, and M rows N indicating the output signal of the image obtained by imaging the subject with the lens. The column matrix is Z, and the subject coordinate system and the image coordinate system are set so that the imaging position of light emitted from one point (m, n) in the subject coordinate system is one point (m, n) in the image coordinate system. When set, in order to perform the convolution calculation processing calculated based on the following equation (D-1) for all or a part of each coordinate (m, n) as a matrix used for calculating the matrix A from the matrix Z (2M-1) for the coordinates (m, n) of In which values were recorded matrix portion that includes at least non-zero elements among the values of the row (2N-1) convolution matrix Q _{m columns,} each element Q _m of _{n, n (x, y)} .

Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （D-1）

Here, x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ (N−1), m and n are natural numbers, and 1 ≦ m ≦ M, 1 ≦ n ≦ N, and W _{m, n} (x, y) is a value of each element of the matrix W _{m, n} of (2M−1) rows (2N−1) columns. W _{m, n} is a point spread function matrix indicating a state in which light emitted from one point (m, n) of the subject spreads on the image sensor due to the action of the lens, and W _{m, n} (0, 0) Is the value of the output signal of the pixel located in the center part of the spread, W _{m, n} (−x, −y) is the value of the output signal of the pixel located in the surrounding blur part, and power is A real number that is a power of W _{m, n} (0, 0), and 1 ≦ power ≦ 2.

  The data recording medium of the present invention includes, for example, a magneto-optical disk (MO), a read-only memory (CD-ROM) using a compact disk (CD), a CD recordable (CD-R), and a CD rewritable. (CD-RW), read-only memory (DVD-ROM) using digital versatile disk (DVD), random access memory (DVD-RAM) using DVD, flexible disk (FD), magnetic tape, A hard disk, read-only memory (ROM), electrically erasable and rewritable read-only memory (EEPROM), flash memory, random access memory (RAM), or a combination of these can be employed.

As described above, according to the present invention, the image processing is performed using the convolution calculation matrix Q _{m, n} stored in the convolution calculation matrix storage means by the reproduction calculation means, so that M × N rows M × The value of each element A (m, n) of the matrix A of the subject can be calculated with a very small amount of calculation compared with the case where the arithmetic processing is performed using the N-column giant inverse transformation matrix T _g ^−1. In addition, since the convolution operation matrix storage means stores the convolution operation matrix Q _{m, n} for each coordinate (m, n), the reproduction operation means considers that the blur shape differs for each pixel. The modification process can be performed, and the image modification effect can be further enhanced as compared with the case where the image modification process is performed on the assumption that blur of the same shape occurs in all pixels.

  An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows an overall configuration of an imaging system 10 including an image modification processing device 30 of the present embodiment. FIG. 2 shows a detailed configuration of the bifocal lens 21 that forms an image to be processed by the image modification processing device 30. The imaging system 10 is, for example, an imaging system provided in a portable information terminal device such as a mobile phone or a portable information terminal, or an imaging system configured by a personal computer and a camera connected thereto.

  In FIG. 1, an imaging system 10 includes an imaging mechanism 20 that images a subject, an image modification processing device 30 that improves the quality of an image captured by the imaging mechanism 20, and a quality that is improved by the image modification processing device 30. Display means 40 for displaying the improved image.

  The imaging mechanism 20 includes a bifocal lens 21 that captures an image of a subject and an imaging element 24 that captures an image formed by the bifocal lens 21.

  In FIG. 2, the bifocal lens 21 is composed of, for example, a glass long-focus lens portion 22 and, for example, a glass short-focus lens portion 23. The long-focus lens portion 22 and the short-focus lens portion 23 are They are arranged on the same surface and integrated. The long-focus lens unit 22 has, for example, a circular front shape and is arranged at the center, while the short-focus lens unit 23 has, for example, an annular (donut-shaped) front shape, It is arranged on the outside in contact with the outer edge of the long focal length lens portion 22. The long-focus lens portion 22 and the short-focus lens portion 23 are integrated so that their optical axes coincide with each other, that is, concentrically arranged, whereby the bifocal lens 21 is concentric bifocal. It is a lens. The surface on which these lens portions 22 and 23 are arranged is orthogonal to the optical axis of the bifocal lens 21.

  Here, the long focus lens unit 22 is long and suitable for imaging a normal subject (for example, a person or a landscape) at a standard distance from the lower limit of focus depth (for example, 0.3 m) to infinity. A lens unit having a focal length. On the other hand, the short focus lens unit 23 is a lens unit having a short focal length suitable for imaging a close subject (for example, a two-dimensional barcode, an iris, or a character) arranged at a distance closer to a normal subject. is there. Note that the arrangement relationship between the long focal lens unit and the short focal lens unit may be reversed, and the long focal lens unit may be disposed outside and the short focal lens unit disposed inside.

  As the imaging device 24, for example, a complementary metal oxide semiconductor (CMOS), a charge coupled device (CCD), or the like can be employed. The size of the image sensor 24 is assumed to be M pixels in the vertical direction × N pixels in the horizontal direction.

  The image modification processor 30 includes an output signal storage unit 31, a convolution operation matrix storage unit 32 for standard image reproduction, a convolution operation matrix storage unit 33 for close-up image reproduction, a rotation information storage unit 34, and a convolution operation. A matrix rotation calculating unit 35, a reproduction calculating unit 36, and a switching operation unit 37 are provided.

  The output signal storage means 31 extracts and stores the output signal of the image sensor 24.

The convolution calculation matrix storage means 32 for standard image reproduction picks up a normal subject (for example, a person or a landscape) at a standard distance with the bifocal lens 21, and accordingly, in this embodiment, the inner length is long. When a focused image is formed by the focus lens unit 22 and an out-of-focus image is formed by the outer annular short focus lens unit 23, convolution used for reproduction calculation processing for obtaining a focused image The operation matrix Q _{m, n} (see FIG. 6) is stored.

The convolution calculation matrix storage means 33 for reproducing the close-up image captures a close subject (for example, a two-dimensional bar code, an iris, or a character) arranged at a distance closer to that of a normal subject. When the outer annular short-focus lens unit 23 forms an in-focus image and the inner long-focus lens unit 22 forms an in-focus image, a reproduction calculation process for obtaining an in-focus image The convolution operation matrix Q _{m, n} used in the above is stored. In this case, the shape of the blurred portion in the point spread function matrix W _{m, n} is not a ring shape as shown in FIG. (See a blurred part 83 in FIG. 13 described later), and the convolution matrix Q _{m, n} also has a shape corresponding to the blurred part as shown in FIG. 6 described later. It does not have a ring shape, but has a substantially solid circular shape or a substantially solid elliptical shape (a state in which the inside is filled with non-zero elements).

Then, the convolution operation matrix storage means 32 and 33 for standard image reproduction and close-up image reproduction convolution operation matrix Q _{m, n} for each coordinate (m, n) calculated in advance based on equation (18) described later. At least a part of the values of the respective elements Q _{m, n} (x, y) of (see FIG. 4) are stored as shown in the table according to the arrangement order of x and y. Since it is at least a part, the whole may be stored, but in order to reduce the calculation capacity and memory capacity, only the matrix part including the non-zero elements (matrix part H in FIG. 4) is stored. Is preferred. Therefore, here, the description will be made assuming that only the matrix portion H including non-zero elements is stored.

Also, standard image reproduction and close-up image reproduction convolution matrix storage means 32 and 33, the convolution operation matrix Q _m for all the coordinates _{(m, n),} n (i.e., M × N number of Q _{m , n} ) is not stored _{, and} only the convolution matrix Q _{m, n for} some coordinates (coordinates aligned on a straight line) is stored as a sampling matrix in consideration of the axial symmetry of the bifocal lens 21. In the present embodiment, as an example, it is assumed that only the convolution matrix Q _{m, n} for each coordinate lined up on a line 60 in FIG. Accordingly, in the present embodiment _, the distribution of the convolution matrix Q _{m, n} stored as a sampling matrix is vertically symmetric, so that only half (for example, the upper half) of the matrix portion H including the non-zero elements described above. Remember.

The rotation information storage means 34 rotates the arrangement of the values of the elements Q _{m, n} (x, y) of the convolution arithmetic matrix Q _{m, n} as the sampling matrix stored in the convolution arithmetic matrix storage means 32,33. Rotation information used for calculating and determining the value of each element Q _{m, n} (x, y) of the convolution calculation matrix Q _{m, n} other than the sampling matrix is stored. The rotation information is obtained convolution matrix Q _{m, n} (sampling matrix other than the convolution matrix Q _{m, n)} of all (However, considering axial symmetry, may. 1/4 of the total coordinate) prepared for And the distance r from the center coordinates (M / 2, N / 2) and the angle θ with respect to the coordinates (m, n) of the convolution operation matrix Q _{m, n} to be obtained (see FIG. 12). The angle θ refers to the coordinates (m, n) and center of the convolution matrix Q _{m, n to be} obtained with respect to a straight line (line 60 in FIG. 8 described later) in which the coordinates (m, n) of the sampling matrix Q _{m, n} are arranged. This is an angle formed by a straight line connecting the coordinates (M / 2, N / 2).

The convolution calculation matrix rotation calculation means 35 is a value of each element Q _{m, n} (x, y) of the convolution calculation matrix Q _{m, n} as a sampling matrix stored in the convolution calculation matrix storage means 32 and 33, and rotation information. Using the rotation information stored in the storage means 34, processing for calculating and determining the value of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} other than the sampling matrix is performed.

The reproduction calculation unit 36 performs calculation processing for reproducing a subject, and each element Q _{m, n} (x of the convolution calculation matrix Q _{m, n} as a sampling matrix stored in the convolution calculation matrix storage units 32 and 33 is used. , Y) (each value of the matrix portion H) or each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} other than the sampling matrix calculated and determined by the convolution matrix rotation calculation means 35 And the value of each element Z (m, n) of the matrix Z indicating the output signal of the image stored in the output signal storage means 31, and based on the equation (19) to be described later, The process of calculating the value of each element A (m, n) is performed. It should be noted that a part other than the matrix part including the non-zero element (matrix part H in FIG. 4) among the elements Q _{m, n} (x, y) of the convolution arithmetic matrix Q _{m, n} , that is, the convolution arithmetic matrix storage means. The portions not stored in 32 and 33 are zero elements, and thus are not calculated.

  The switching operation unit 37 uses the data of the convolution calculation matrix storage units 32 and 33 in the calculation processing by the convolution calculation matrix rotation calculation unit 35 and the reproduction calculation unit 36, that is, picks up an image of either a normal subject or a close subject. For example, a push button type, a toggle type switch, a slide type switch or the like is used.

  As the output signal storage means 31, the convolution calculation matrix storage means 32 and 33, and the rotation information storage means 34, for example, a hard disk, ROM, EEPROM, flash memory, RAM, MO, CD-ROM, CD-R, CD- RW, DVD-ROM, DVD-RAM, FD, magnetic tape, or a combination thereof can be employed.

  The convolution calculation matrix rotation calculation means 35 and the reproduction calculation means 36 are used for various information terminal devices constituting the imaging system 10 (for example, portable information terminal devices such as a mobile phone and a portable information terminal, or a personal information terminal connected with a camera). This is realized by a central processing unit (CPU) provided inside a computer, a monitoring camera device, etc.) and one or a plurality of programs that define the operation procedure of this CPU.

  As the display means 40, for example, a liquid crystal display, a CRT display, a projector and a screen, or a combination thereof can be employed.

Hereinafter, the calculation method and the basis of each element Q _{m, n} (x, y) of the convolution calculation matrix Q _{m, n} stored in the convolution calculation matrix storage means 32 and 33, and the reproduction calculation means 36 are performed. The basis of the arithmetic processing will be described.

  As a precondition, here, when a normal subject (for example, a person or a landscape) at a standard distance is imaged by the bifocal lens 21, an image focused by the inner long focal lens unit 22 is obtained. A case will be described where an out-of-focus image is formed by the outer annular short focus lens portion 23 formed. However, the application of the present invention is not limited to such a case.

  First, basic matters regarding the subject and the image will be described. FIG. 3 is an explanatory diagram of a relationship between a subject and an image obtained by imaging the subject with the bifocal lens 21. In addition, since the image formed by the lens is generally an inverted image, in FIG. 3, by reversing the direction of the coordinate axis between the object and the image, the position of the corresponding object and the position of the image are the same. The coordinates are set. Here, as shown in FIG. 3, a subject coordinate system (s, t) composed of an s-axis (vertical axis) and a t-axis (horizontal axis), and an h-axis (vertical axis) and a k-axis (horizontal axis). ) And an image coordinate system (h, k) is set.

  The number of pixels in the vertical direction of the image sensor 24 is M, and the number of pixels in the horizontal direction is N. At this time, if each point (s, t) of the subject (the portion of coordinates (s, t) in the subject coordinate system) has a brightness of A (s, t), the entire light emitted by the subject is Represented by a matrix A of M × N (M rows and N columns) having elements of A (s, t) in s rows and t columns (s and t are natural numbers, 1 ≦ s ≦ M, 1 ≦ t ≦ N) Yes (see FIG. 4). This is called a subject matrix A.

  On the other hand, if the magnitude of the output signal of each point (h, k) on the image sensor 24 (pixel located at the coordinates (h, k) in the image coordinate system) is Z (h, k), respectively, The entire image obtained by imaging with the bifocal lens 21 has elements of Z (h, k) in h rows and k columns (h and k are natural numbers, 1 ≦ h ≦ M, 1 ≦ k ≦ N). It can be expressed by a matrix Z having M × N (M rows and N columns) (see FIG. 4). This is called an image matrix Z.

<Expression of Characteristics of Bifocal Lens 21 by Single Bright Spot Image>
As shown in FIG. 3, when a subject having a bright spot only at one point (coordinates (m, n)) is imaged by the bifocal lens 21, the coordinates (m, n) are spread on the image sensor 24. As a result, an in-focus image formed by the inner long-focus lens portion 22 and an annular blurred image formed by the outer annular short-focus lens portion 23 are formed. Here, m and n are natural numbers, 1 ≦ m ≦ M, 1 ≦ n ≦ N, where M and N are the number of pixels in the vertical and horizontal directions of the image sensor 24. Also, the coordinates (m, n) on the subject means a corresponding point on the subject projected on the coordinates (m, n) on the image sensor 24. That is, light emitted from one point (m, n) in the subject coordinate system (s, t) is imaged at a point (m, n) indicated by the same numerical value in the image coordinate system (h, k). It has become.

  Here, in the image coordinate system, as shown in FIG. 3, the x-axis (vertical axis) and the y-axis are respectively parallel to the h-axis (vertical axis) and the k-axis (horizontal axis) around the coordinates (m, n). Taking the (horizontal axis), a coordinate system composed of these x-axis and y-axis is set. In this coordinate system (x, y), if a single bright spot (coordinate (m, n)) on the subject moves, the corresponding coordinate (m, n) on the image sensor 24 also moves. This is a relative coordinate system set for each of m, n).

When the magnitude of the output signal of the pixel located at each point (x, y) in the relative coordinate system (x, y) is W _{m, n} (x, y), one point (m, n) of the subject In a state in which the light emitted from the image sensor 24 spreads on the image sensor 24 by the action of the bifocal lens 21 (PSF: Point Spread Function), the value of W _{m, n} (x, y) is used as the element (2M−1) rows 2N-1) can be expressed by a matrix W _{m, n} of columns (see FIG. 4). This is called a point spread function (PSF) matrix with respect to coordinates (m, n).

Note that the values of W _{m, n} (x, y) are all x, y (x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ ( The total value of N-1)) is the total amount of light emitted from one point (m, n) of the subject, and each element W _{m, n} (x, y) has a total value of 1. Normalize the value.

The PSF matrix W _{m, n} is a (2M−1) row (2N−1) column matrix, that is, a matrix having approximately four times the number of elements of the subject matrix A or the image matrix Z. Light emitted from one point (for example, a point of coordinates (1, 1)) is diagonally centered on one point (1, 1) of the corresponding end on the image sensor 24. It is for showing the state which expands as a whole including 1 point (M, N) of the edge part on the opposite side. However, in reality, the blurred portion does not spread widely from one end portion in the diagonal direction on the image sensor 24 to the other end portion, and as shown in FIG. 4, (2M-1) rows ( Among the elements W _{m, n} (x, y) of the 2N−1) -column PSF matrix W _{m, n} , a part including non-zero elements is a partial matrix including W _{m, n} (0,0) Only part E is present.

<Convolution for Deriving Image Matrix Z from Subject Matrix A>
FIG. 4 is an explanatory diagram of the conversion relationship between the subject matrix A and the image matrix Z. In general, the calculation for obtaining the image matrix Z from the subject matrix A is not a simple matrix calculation but a convolution calculation using the PSF matrix W _{m, n} expressed by the following equation (1).

Z (m, n) = Σ x Σ y {W m + x, n + y (-x, -y) * A (m + x, n + y)}
= W _{m, n} (0,0) * A (m, n)
+ Σx _{, y} {W _{m + x, n + y} (−x, −y) * A (m + x, n + y)}
(1)

Here, Σ _x is the sum of x = (1−M) to (M−1), and Σ _y is the sum of y = (1−N) to (N−1). Also, Σx _{, y} is the sum for all x and y except for the point (x, y) = (0,0). However, Σ _x and Σ _y and Σ _{x, y} are the sum of only the matrix portion E (see FIG. 4) including the non-zero elements of the PSF matrix W _{m + x, n + y} (−x, −y). Should be considered. In the above formula (1), the conversion coefficient between the light amount and the image output signal is set to 1 by appropriately taking the light amount unit and the image output signal unit.

  The lower right side of the above formula (1) is obtained by dividing the upper right side into points of (x, y) = (0, 0) and other points.

Incidentally, Z (m, n) is the value of the output signal of the pixel located at the point (m, n) on the image sensor 24, and this point (m, n) is a corresponding point on the subject. When considering not only the focused light from (m, n) but also the periphery of the point, that is, the subject coordinate system (s, t), the distance corresponding to x pixels in the s direction, and y in the t direction Focused light from points (m + x, n + y) that are separated by a distance corresponding to the pixel is also collected. Therefore, the value of Z (m, n) is the output signal W _{m, n} (0,0) * A () of light of brightness A (m, n) emitted from one point (m, n) on the subject. In addition to the value of m, n), the output signal W _{m + x, n + y} (−x) by the light of brightness A (m + x, n + y) emitted from the surrounding point (m + x, n + y) is added to this value. , −y) * A (m + x, n + y). And the lower right side of the formula (1) means that.

Therefore, conversely to the above equation (1), in order to derive the subject matrix A from the image matrix Z _, a convolution for performing a convolution operation process corresponding to the PSF matrix W _{m, n} in the above equation (1). It suffices if the operation matrix Q _{m, n} can be obtained. However, such a convolution matrix Q _{m, n} cannot be easily obtained from the PSF matrix W _{m, n} as in the case of obtaining an inverse matrix.

<Examination of huge matrix for converting between subject matrix A and image matrix Z>
On the other hand, the number of elements of the subject matrix A is M × N, and the number of elements of the image matrix Z is M × N. Thus, M rows and N by arranging the elements of the subject matrix A column in tandem to a M × huge vector (column vector) of N number of elements A _g, tandem each element of the image matrix Z of M rows and N columns And a giant vector (vertical vector) Z _{g of} M × N elements, the transformation relationship from these giant vectors A _g to Z _g is the giant transformation matrix T of M × N rows and M × N columns. _{Using g} , it can be expressed as the following equation (2).

_{Z g = T g A g ·} · ························· (2)

Therefore, if the inverse matrix T _g ⁻¹ of the giant transformation matrix T _g in the above equation (2) can be obtained, the transformation relationship from the giant vector Z _g to A _g can be obtained using the giant inverse transformation matrix T _g ^−1. The following equation (3) can be expressed.

A _g = T _g ^-1 Z _g (3)

Therefore, if the huge inverse transformation matrix T _g ⁻¹ is obtained in advance and the calculation of the above equation (3) is performed at the time of imaging of the subject, the value of each element of the giant vector _Ag , that is, each element of the subject matrix A Can be obtained and the subject can be reproduced.

However, the arithmetic processing using the M × N rows and M × N giant inverse transformation matrix T _g ⁻¹ has a large amount of calculation, and thus, for example, a portable information terminal device such as a normal cellular phone or a portable information terminal. In a central processing unit (CPU) having a level of performance mounted on the CPU, processing in a short time is difficult, so it is not realistic. For example, in a VGA (Video Graphics array) which is a normal image, M = 640 pixels and N = 480 pixels, and the number of pixels M × N is about 300,000. Therefore, the huge inverse transformation matrix T _g ⁻¹ is Both the number of rows and the number of columns are about 300,000 pixels, and the size (number of elements) is the square of the number of pixels (M × N) ² ≈90 billion.

Therefore, in the following, a convolution calculation matrix Q _{m, n} for deriving the subject matrix A from the image matrix Z is obtained so that the subject can be reproduced with a smaller amount of calculation than the calculation according to the above equation (3). To do.

<Convolution operation for deriving the subject matrix A from the image matrix Z>
First, after shifting the second term on the lower right side of the formula (1) described above to the left side, the left side and the right side are interchanged to obtain the following formula (4).

W _{m, n} (0,0) * A (m, n)
= Z (m, n) −Σx _{, y} {W _{m + x, n + y} (−x, −y) * A (m + x, n + y)}
(4)

  As a simple approximation method of the above equation (4), the following approximation can be performed. That is, in the first step, it is assumed that the subject and the captured image are not significantly changed, and A (m + x, n + y) is approximated by Z (m + x, n + y). As a result, the above equation (4) becomes the following equation (5).

W _{m, n} (0,0) * A (m, n)
= Z (m, n)-[Sigma] _{x, y} {Wm _{+ x, n + y} (-x, -y) * Z (m + x, n + y)}
(5)

Subsequently, in the second step, it is assumed that the blurring state at each point is similar between neighboring points, and W _{m + x, n + y} (−x, −y) = W _{m, n} ( -X, -y). As a result, the above equation (5) becomes the following equation (6).

W _{m, n} (0,0) * A (m, n)
= Z (m, n)-[Sigma] _{x, y} {Wm _{, n} (-x, -y) * Z (m + x, n + y)}
(6)

Therefore, when the left side and the right side of the above equation (6) are divided by W _{m, n} (0, 0), the following equation (7) is obtained.

A (m, n)
= Z (m, n) / W _{m, n} (0,0)
−Σ _{x, y} {W _{m, n} (−x, −y) * Z (m + x, n + y)} / W _{m, n} (0,0)
(7)

Here, Q _{m, n} (x, y) is defined as shown in the following equation (8), and W _{m, n} (x, y) in the above equation (7) is defined as Q _{m, n} (x, y). ), The following equation (9) is obtained. However, in the following equation (8), the power multiplier of W _{m, n} (0, 0) is set to power = 1.

Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （8）

A (m, n) = Q _{m, n} (0,0) * Z (m, n)
+ Σ _{x, y} {Q _{m, n} (x, y) * Z (m + x, n + y)}
= Σ _x Σ _y {Q _{m, n} (x, y) * Z (m + x, n + y)}
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ (9)

Here, Σ _x is the sum of x = (1−M) to (M−1), and Σ _y is the sum of y = (1−N) to (N−1). Also, Σx _{, y} is the sum for all x and y except for the point (x, y) = (0,0).

The above equation (9) is an equation for performing a convolution operation corresponding to the above-described equation (1), and Q _{m, n} (x, y) in the above equation (9) is the above-described equation (1). ) Corresponding to W _{m + x, n + y} (−x, −y). Accordingly, the matrix Q _{m, n} is a convolution operation matrix corresponding to the PSF matrix W _{m, n} , and the reproduction operation for obtaining the subject matrix A from the image matrix Z is performed using this convolution operation matrix Q _{m, n.} Can do.

  Next, the following approximation can be performed as a more advanced approximation method of Equation (4) described above. That is, the above-described expression (4) is an expression obtained by modifying the above-described expression (1) for obtaining the value of the output signal Z (m, n) of the pixel located at the point (m, n) on the image sensor 24. In the same manner, the value of the output signal Z (m + x, n + y) of a pixel located at a point (m + x, n + y) separated by x in the h direction and y in the k direction on the image sensor 24 is obtained. Consider an expression that is a modified expression.

  In the above-described equation (4), m is replaced with (m + x), n is replaced with (n + y), and a relative coordinate system is used as a coordinate system for expressing blur caused by light emitted from a point (m + x, n + y) of a new subject. When (u, v) is set and x is replaced with u and y is replaced with v, the following equation (10) is obtained.

W _{m + x, n + y} (0,0) * A (m + x, n + y)
= Z (m + x, n + y)
−Σ _{u, v} {W _{m + x + u, n + y + v} (−u, −v) * A (m + x + u, n + y + v)}
(10)

Here, u and v are integers, u = (1-M) to (M−1), v = (1−N) to (N−1), and Σ _{u, v} is (u, v) = the sum for all u, v except for the point (0,0).

After dividing the left side and the right side of the above equation (10) by W _{m + x, n + y} (0,0) and substituting into A (m + x, n + y) in the above equation (4), the following equation (11 )become that way.

W _{m, n} (0,0) * A (m, n)
= Z (m, n)
−Σ _{x, y} {W _{m + x, n + y} (−x, −y)
* Z (m + x, n + y) / W _{m + x, n + y} (0,0)}
+ Σ _{x, y} Σ _{u, v} [W _{m + x, n + y} (−x, −y) * {W _{m + x + u, n + y + v} (−u, −v)
* A (m + x + u, n + y + v)} / W _{m + x, n + y} (0,0)]
(11)

Then, when the left side and the right side of the above equation (11) are divided by W _{m, n} (0, 0), the following equation (12) is obtained.

A (m, n)
= Z (m, n) / W _{m, n} (0,0)
−Σ _{x, y} {W _{m + x, n + y} (−x, −y)
* Z (m + x, n + y) / W _{m + x, n + y} (0,0)} / W _{m, n} (0,0)
+ Σ _{x, y} Σ _{u, v} [W _{m + x, n + y} (−x, −y) * {W _{m + x + u, n + y + v} (−u, −v)
* A (m + x + u, n + y + v)} / W _{m + x, n + y} (0,0)] / W _{m, n} (0,0)
(12)

The above equation (12) is approximated. In the first step, the product of W _{m + x, n + y} (−x, −y) and W _{m + x + u, n + y + v} (−u, −v) is added to the third term on the right side. Therefore, if this third term on the right side is ignored, the following equation (13) is obtained.

A (m, n)
= Z (m, n) / W _{m, n} (0,0)
−Σ _{x, y} {W _{m + x, n + y} (−x, −y)
* Z (m + x, n + y) / W _{m + x, n + y} (0,0)} / W _{m, n} (0,0)
(13)

Subsequently, in the second step, it is assumed that the blurring state at each point is similar between neighboring points, and W _{m + x, n + y} (−x, −y) is changed to W _{m, n} ( Approximate by -x, -y). As a result, the above equation (13) becomes the following equation (14).

A (m, n)
= Z (m, n) / W _{m, n} (0,0)
−Σ _{x, y} {W _{m, n} (−x, −y)
* Z (m + x, n + y) / W _{m + x, n + y} (0,0)} / W _{m, n} (0,0)
(14)

Further, in the third step, W _{m + x, n + y} (0,0) is also approximated by W _{m, n} (0,0). As a result, the above equation (14) becomes the following equation (15).

A (m, n)
= Z (m, n) / W _{m, n} (0,0)
−Σ _{x, y} {W _{m, n} (−x, −y) * Z (m + x, n + y)} / W _{m, n} (0,0) ²
(15)

Here, Q _{m, n} (x, y) is defined as shown in the following equation (16), and W _{m, n} (x, y) in the above equation (15) is defined as Q _{m, n} (x, y). ), The following equation (17) is obtained. However, in the following equation (16), the power of W _{m, n} (0, 0) is set to power = 2.

Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ (16)

A (m, n) = Q _{m, n} (0,0) * Z (m, n)
+ Σ _{x, y} {Q _{m, n} (x, y) * Z (m + x, n + y)}
= Σ _x Σ _y {Q _{m, n} (x, y) * Z (m + x, n + y)}
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ (17)

Here, Σ _x is the sum of x = (1−M) to (M−1), and Σ _y is the sum of y = (1−N) to (N−1). Also, Σx _{, y} is the sum for all x and y except for the point (x, y) = (0,0).

And said Formula (17) is the same formula as Formula (9) mentioned above. Accordingly, the matrix Q _{m, n} in the above equation (17) is a convolution operation matrix corresponding to the PSF matrix W _{m, n} in the above equation (1), and the reproduction operation for obtaining the subject matrix A from the image matrix Z is as follows: This convolution operation matrix Q _{m, n} can be used.

From the above, if the convolution calculation matrix Q _{m, n} is defined as in the following equation (18) in consideration of the above-described equations (8) and (16), the matrix Q _m is based on the following equation (19). _{, n} can be used to perform a convolution operation to reproduce the subject.

Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
.... (18)

A (m, n) = Σ x Σ y {Q m, n (x, y) * Z (m + x, n + y)}
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ (19)

Here, x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ (N−1), m and n are natural numbers, and 1 ≦ m ≦ M, 1 ≦ n ≦ N, and W _{m, n} (x, y) is a value of each element of the PSF matrix W _{m, n} of (2M−1) rows (2N−1) columns, and power is, W _m, a real number which is a power number n (0,0), a 1 ≦ power ≦ 2, the sigma _x, the sum of x = (1-M) ~ (M-1), Σ _y is the sum of y = (1-N) to (N-1).

The power value may be determined according to the value of W _{m, n} (0, 0). At this time, if the value of W _{m, n} (0, 0) is in the vicinity of 0.5 (including 0.5; the same applies hereinafter), the power value needs to be a value other than 2. More specifically, when the value of W _{m, n} (0, 0) is in the vicinity of 0.5, the power value is set to 1 or more and less than 2, more preferably 1. On the other hand, when the value of W _{m, n} (0, 0) is not near 0.5, it is set to 1 or more and 2 or less.

<Distribution of each element W _{m, n} (x, y) of PSF matrix W _{m, n} >
FIG. 5 shows an example of a distribution situation of each element W _{m, n} (x, y) of the PSF matrix W _{m, n} . At the center W _{m, n} (0,0), it passes from one point (m, n) of a normal subject at a standard distance and passes through an inner long focal lens portion 22 constituting the bifocal lens 21. The focused light forms an in-focus image. Around W _{m, n} (0,0), an annular (substantially annular or substantially elliptical) defocused image is formed by the light passing through the outer annular short focus lens portion 23. . The blank part is a zero element.

In FIG. 5, the value of W _{m, n} (0, 0) at the center is c, but this value of c is the area of the inner long focal lens portion 22 relative to the entire area of the bifocal lens 21. The ratio value. However, strictly speaking, since the division of light may not be proportional to the area, if the division ratio of the amount of light does not coincide with the area ratio of the lens, the value of c above is the inner long focus lens portion. The amount of light that actually passes through the sensor 22 and reaches the sensor (imaging device 24) or the signal amount of the sensor is combined with the entire bifocal lens 21 (the inner long-focus lens portion 22 and the outer short-focus lens portion 23). And the amount of light actually passing through the sensor or the signal amount of the sensor. In FIG. 5, for simplification of description, each element (for example, W _{m, n} (1,0) or W _{m, n} (0,1) adjacent to the center W _{m, n} (0,0)) is shown. ) Etc.) is zero, but in reality, the image formed by the inner long focal point lens unit 22 is not completely focused, and some blurring occurs. Is normal (the shape of the blur is a substantially solid circular shape or a substantially solid elliptical shape), the value of c is several elements around W _{m, n} (0,0). (See the blurred portion 80 in FIG. 13 described later).

In FIG. 5, the values of the elements of the annular blur portion are indicated by e1 to e19, but this shows an outline of the shape of the blur. The width is not a width of 1 to 2 rows or 1 to 2 columns, but a width corresponding to a larger number of rows or columns. For example, in the case of the PSF matrix W _{m, n} about the center position of the image sensor 24 (the center coordinates of the image coordinate system (h, k)), for example, seven rows around W _{m, n} (0,0) or Zero elements are arranged in an annular shape with a width of 7 columns, and non-zero elements are arranged in an annular shape with a width of 8 rows or 8 columns, for example. Accordingly, in this case, the matrix portion E (see FIG. 4) including the non-zero elements in W _{m, n} (x, y) is, for example, a size that fits in 31 pixels × 31 pixels. . In this case, the ratio between the inner diameter and the outer diameter of the ring at the blurred portion of W _{m, n} (x, y) is the same as the inner diameter and the outer diameter of the outer short-focus lens portion 23 constituting the bifocal lens 21. Is equal to the ratio. Note that the size, width, and shape of the ring in the blurred portion of W _{m, n} (x, y) vary depending on each coordinate (m, n) (see FIG. 7).

Furthermore, since the blur formed by the outer short focus lens unit 23 extends over a wide range, the values of the elements of the annular blur part (values e1 to e19 in FIG. 5) in the PSF matrix W _{m, n} are The value is sufficiently smaller than the value at the center focused portion (the value of W _{m, n} (0,0) = c in FIG. 5). Specifically, the value of c is, for example, 0.8, and the values of e1 to e19 are, for example, 0.0013. However, it is not limited to these numerical values.

<Convolution matrix Q _m, each element Q _m of _{n, n} (x, y) of the distribution>
FIG. 6 shows an example of the distribution situation of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} defined by the above-described equation (18). The distribution of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} also corresponds to the distribution of each element W _{m, n} (x, y) of the PSF matrix W _{m, n} (see FIG. 5). In addition, there is a distribution in which there is a non-zero element at the center Q _{m, n} (0, 0), and the non-zero elements are arranged in a ring (substantially annular or substantially elliptical ring) around it. The blank part is a zero element.

However, the distribution of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} of FIG. 6 is the same as that of each element W _{m, n} (x, y) of the PSF matrix W _{m, n} of FIG. For the distribution, x and y are inverted to become -x and -y. This is inverted according to the definition given by the equation (18) described above.

<Relationship between PSF matrix W _{m, n} for each coordinate (m, n)>
FIG. 7 shows the distribution status of each element W _{m, n} (x, y) of the PSF matrix W _{m, n} for each coordinate (m, n). In the case of the PSF matrix W _{m, n} about the center position of the image sensor 24 (center coordinates (M / 2, N / 2) of the image coordinate system (h, k)), the blur shape is an annular shape. . On the other hand, at coordinates other than the center of the image coordinate system (h, k), the blur shape is substantially elliptical, and as the distance from the center of the image coordinate system (h, k) increases, Increases the size and width of the ring.

Further, since the bifocal lens 21 has axial symmetry, the axial symmetry is also maintained for the distribution (blurred shape) of the PSF matrix W _{m, n} for each coordinate (m, n). Therefore, considering each straight line extending radially from the center position of the image sensor 24 (the center coordinates (M / 2, N / 2) of the image coordinate system (h, k)), the radiation direction can be viewed on any straight line. The distribution (blurred shape) changes in the same way from the center to the outside, and the distribution (blurred shape) of the PSF matrix W _{m, n} is determined by the distance from the center position of the image sensor 24. The distribution of the PSF matrix W _{m, n} is also symmetric with respect to the radiation line. That is, the distribution (blurred shape) of the PSF matrix W _{m, n} with respect to coordinates having the same distance from the center position of the image sensor 24 is all the same, and is simply rotated. For example, the arrangement of each element W _{m, n} (x, y) of the PSF matrix W _{m, n} for each coordinate (m, n) arranged on the line 50 in the image coordinate system (h, k) is represented by the image coordinate system. If the center coordinates (M / 2, N / 2) of (h, k) are rotated as the center, each coordinate (line on other lines 51, 52, 53, 54 in the image coordinate system (h, k)) ( The arrangement of each element W _{m, n} (x, y) of the PSF matrix W _{m, n} for _{m, n} ) can be calculated.

<Relationship between convolution matrix Q _{m, n} for each coordinate (m, n)>
FIG. 8 shows the distribution status of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} for each coordinate (m, n). Since the distribution of the convolution calculation matrix Q _{m, n} is also defined by the above-described equation (18), the axial symmetry is maintained as with the distribution of the PSF matrix W _{m, n} . Accordingly, the distributions of the convolution matrixes Q _{m, n} for the coordinates having the same distance from the center coordinates (M / 2, N / 2) are all the same, and are simply rotated. For example, the arrangement of the elements Q _{m, n} (x, y) of the convolution matrix Q _{m, n} for the coordinates (m, n) arranged on the line 60 indicated by the one-dot chain line in FIG. If it is rotated around (M / 2, N / 2), a convolution matrix for each coordinate (m, n) arranged on the other lines 61, 62, 63, 64 indicated by dotted lines in FIG. Q _m, each element of the _n Q _m, can be calculated arrangement of _n (x, y). It should be noted that the individual distribution of the convolution matrix Q _{m, n} for each coordinate (m, n) in FIG. 8 is inverted for x and y with respect to the distribution of the PSF matrix W _{m, n} for the same coordinate in FIG. This is the same as the relationship between FIG. 5 and FIG. 6 described above.

Therefore, the convolution calculation matrix storage means 32 and 33 do not need to store the convolution calculation matrix Q _{m, n} for all coordinates (m, n) (1 ≦ m ≦ M, 1 ≦ n ≦ N). , The convolution matrix Q _{m, n} for the coordinates arranged on one of the straight lines radially extending from the central coordinates (M / 2, N / 2) is stored as a sampling matrix, and the convolution for the other coordinates The operation matrix Q _{m, n} may be obtained by rotating the sampling matrix. In the present embodiment, in order to ensure vertical symmetry of the arrangement (blurred shape) of each Q _{m, n} (x, y) to be stored, a horizontal line 60 indicated by a one-dot chain line in FIG. convolution matrix Q _m for each coordinate arranged _above shall be stored _n (convolution a distribution of vertically symmetrical as shown in FIG. 6 described above operation matrix Q _{m, n).}

However, since the diagonal lines 61 to 64 have coordinates farthest from the central coordinates (M / 2, N / 2), that is, a convolution matrix for each coordinate on the diagonal lines 61 to 64. If Q _{m, n} is sampled, the convolution operation matrix Q _{m, n} with respect to all coordinates (m, n) (1 ≦ m ≦ M, 1 ≦ n ≦ N) can be represented. When storing the convolution calculation matrix Q _{m, n} for each coordinate lined up on the line 60, as shown in FIG. 8, the same length as the diagonal lines 61 to 64 (half the maximum value of the angle of view). The line 60 is virtually extended so as to have a corresponding length).

Further, the convolution calculation matrix Q _{m, n} stored in the convolution calculation matrix storage means 32 and 33 as the sampling matrix is preferably calculated based on the PSF matrix W _{m, n} obtained by experiment, but the bifocal lens 21 is configured. The PSF matrix W _{m, n} obtained by calculation using the inner and outer diameters of the lens units 22 and 23, the focal lengths of the lens units 22 and 23, the distance between the bifocal lens 21 and the image sensor 24, and the like. You may calculate based on.

In this way _{, if} only the convolution matrix Q _{m, n} for each coordinate (m, n) arranged on the horizontal line 60 is stored, the amount of information to be stored can be reduced. For example, in the case of a VGA (Video Graphics array), M = 640 pixels and N = 480 pixels, and M × N = 307,200 data points (Q _{m, n} ( x, rather than storing distribution) of y), if configured to store the length) content which corresponds to half the maximum value of the length (angle of view half in the diagonal direction, (M ² + N ² ) Since ^1/2/2 = 400 data is stored, the amount of information to be stored can be reduced to about 1/770. Furthermore, as shown in FIG. 6, the individual distributions of Q _{m, n} (x, y) of the convolution matrix Q _{m, n} for each coordinate lined up on the horizontal line 60 are all relative to the line 60. Since it is vertically symmetrical, only the upper half (or the lower half) needs to be stored. For this reason, the amount of information to be stored is further halved and can be reduced to about 1/540. Further, when storing the upper half of the distribution shown in FIG. 6 (the same applies to the lower half) as described above, only the portion corresponding to the matrix portion H including the non-zero elements shown in FIG. 4 is stored. What should be done is as described above.

If there is a margin in the memory capacity, the convolution calculation matrix storage unit 32 stores the convolution calculation matrix Q _{m, n} for all coordinates (m, n) (1 ≦ m ≦ M, 1 ≦ n ≦ N). , 33 may be stored.

<Rotation of convolution matrix Q _{m, n} >
FIG. 9 shows, as an example _{, the} elements Q _{m, n} (x, y) of the convolution matrix Q _{m, n} for the coordinates arranged on the horizontal line 60 (see FIG. 8) shown in FIG. An arrangement rotated 45 degrees is shown. FIG. 10 shows a state in which the area 70 for nine pixels surrounded by a thick frame in the arrangement of FIG. 9 is enlarged. In the present specification, a region corresponding to an element Q _{m, n} (x, y) of a convolution calculation matrix Q _{m, n} (matrix other than a sampling matrix) to be calculated (partitioned by dotted lines in the horizontal and vertical directions in FIG. 9). region) is referred to as first divided region, a sampling matrix after rotation Q _{m, n} elements Q _{m of, n} (x, y) corresponding region (region defined by the oblique solid line in FIG. 9) Will be referred to as a second section area for explanation.

In FIG. 10, the values of f15 to f19 are the values of the elements Q _{m, n} (x, y) corresponding to the respective second partition regions in which these values are described. The value of the element Q _{m, n} (x, y) corresponding to the central first divided area (hatched area) among the nine first divided areas included in the area 70 of FIG. 10 is as follows: Can be calculated.

First, in order to calculate correctly, all the 2nd division areas which overlap with the hatched 1st division area are grasped | ascertained. In the case of FIG. 10, four second partitioned areas corresponding to four elements having respective values of f16, f17, f18, and f19 overlap. Here, the area of the overlapping portion of the hatched first partition region and the second partition region of f16 is α1, and similarly, the hatched first partition region and the second partition regions of f17, f18, and f19 are Assuming that the areas of the overlapping portions are α2, α3, and α4, respectively, the value of the element Q _{m, n} (x, y) corresponding to the hatched first partition region is expressed as the area ratio by the following equation (20). Calculated based on weighted average.

Q _{m, n} (x, y) = {f16 × α1 + f17 × α2 + f18 × α3
+ F19 × α4} / (α1 + α2 + α3 + α4) (20)

However, if the weighted average is calculated based on the area ratio according to the equation (20), the amount of calculation increases. Therefore, a second partitioned area that overlaps a predetermined position of the hatched first partitioned area (in this embodiment, the center position 71 is taken as an example) is obtained, and an element Q _{m, n} corresponding to the obtained second partitioned area is obtained. The value of (x, y) is employed as the value of element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} to be calculated. That is, the value of each element Q _{m, n} (x, y) of the sampling matrix Q _{m, n} is adopted as it is. In the case of FIG. 10, since the second partitioned area f16 overlaps the center position 71 of the hatched first partitioned area, the element Q _{m, n} (x , Y) is f16.

In the same manner as in the case of the hatched first divided area, FIG. 11 also shows the second divided areas that overlap the central positions of the other eight first divided areas included in the area 70 of FIG. The result of calculating and determining the value of each element Q _{m, n} (x, y) is shown. In FIG. 11, among the nine first divided areas included in the area 70, for the upper right and lower left first divided areas, the second divided area that overlaps the center position of these areas is a zero element. The value of the element Q _{m, n} (x, y) corresponding to the upper right and lower left first divided areas is set to zero.

  For example, when a high-performance CPU is used or when the processing speed is not limited, the weighted average may be calculated based on the area ratio according to the above-described equation (20).

As described above, the convolution calculation matrix rotation calculation means 35 calculates the rotation of the sampling matrix Q _{m, n} and the convolution calculation matrix storage means 32 and 33 store the sampling matrix Q _{m, n} . Since only half of the values considering the symmetry of the matrix portion H (see FIG. 4) including zero elements (upper half in the case of the distribution of FIG. 6) are stored, the corresponding calculation is performed. Only the value of the matrix part H including the non-zero elements in the target convolution matrix Q _{m, n} is shown.

When the convolution calculation matrix rotation calculation unit 35 rotates the sampling matrix Q _{m, n} to calculate the convolution calculation matrix Q _{m, n} other than the sampling matrix, each convolution calculation matrix Q _{m, n} to be calculated is calculated. , Rotation information is stored in advance in the rotation information storage means 34.

FIG. 12 is an explanatory diagram of the rotation information of the convolution calculation matrix Q _{m, n} . As shown in FIG. 12, when calculating the convolution calculation matrix Q _{m, n} , as rotation information _, the coordinates (m, n) and center coordinates (M / 2) of the convolution calculation matrix Q _{m, n} to be calculated are used. , N / 2) and the angle θ between the line 60 connecting the coordinates (m, n) of the convolution matrix Q _{m, n} and the central coordinates (M / 2, N / 2) to the line 60. That's fine.

r = {(m−M / 2) ² + (n−N / 2) ² } ^1/2 (21)

  θ = arctan {(m−M / 2) / (n−N / 2)} (22)

The rotation information about each convolution matrix Q _{m, n} to be calculated is the distance r and the angle θ calculated by the above formulas (21) and (22). The rotation information storage means 34 may store the distance r and angle θ for each convolution matrix Q _{m, n} to be calculated, but the bifocal lens 21 has axial symmetry. For this reason, since the axial symmetry is also maintained for the distribution of each convolution matrix Q _{m, n} , it is actually only necessary to store the coordinates of only one quadrant of the four quadrants. Therefore, the amount of information to be stored is a quarter.

  Further, when the distance r is stored, if a fraction is obtained, it is converted into an integer by rounding off or the like and stored. The distance r is stored to determine which sampling matrix on the line 60 is to be rotated. In addition, it may be stored in a state where there are fractions, and when performing the arithmetic processing by the convolution arithmetic matrix rotation calculating means 35, it may be converted into an integer by rounding off or the like to determine which sampling matrix is rotated. Further, since it is only necessary to determine which sampling matrix is to be rotated, identification information such as a number for specifying the sampling matrix may be stored instead of the distance r.

Further, when the rotation information storage means 34 stores identification information such as numbers (identification information other than the distance r) for specifying the sampling matrix as described above, a convolution operation for all coordinates on the line 60 is performed. It is not necessary to store the matrix Q _{m, n} in the convolution operation matrix storage means 32, 33 as a sampling matrix, and for each coordinate in the range _where the distribution of each convolution operation matrix Q _{m, n} is substantially the same, Only one of the convolution operation matrices Q _{m, n} may be stored. In other words, only the convolution matrix Q _{m, n} for specific coordinates determined at intervals on the line 60 may be stored.

Similarly to the case described above, when only the convolution calculation matrix Q _{m, n} for specific coordinates determined at intervals on the line 60 is stored in the convolution calculation matrix storage means 32 and 33 as a sampling matrix. The distance stored in the rotation information storage means 34 may be determined as follows. That is, regardless of whether or not the fraction r is obtained in the above-described calculation of the distance r by the equation (21), the distance r calculated by the equation (21) is not stored as it is, but each sampling matrix Q _{m, n} May be selected from the plurality of specific distances corresponding to the distance r calculated by the equation (21), and the selected specific distance may be stored in the rotation information storage unit 34. . For example, _assuming that a plurality of sampling matrices Q _{m, n} having specific distances 3, 6, 9, 12,... Are prepared, the distance r calculated by the equation (21) is 6.5 or 7. Since 6 is close, the specific distance 6 is stored, and in the case of 8 or 8.5, 9 is close and the specific distance 9 is stored. When the distance r calculated by the equation (21) is exactly the same as any one of the specific distances, the specific distance is stored.

  In the above description, in the subject coordinate system (s, t) and the image coordinate system (h, k), each coordinate value is a positive value corresponding to the row number and column number of the subject matrix A and the image matrix Z. The coordinates (m, n) indicating one point in the subject coordinate system (s, t) and the image coordinate system (h, k) are 1 ≦ m ≦ M, 1 ≦ n ≦ It was supposed to move in the range of N. Accordingly, the coordinates of the subject and the center position of the image sensor 24 have been described as (M / 2, N / 2). However, from the viewpoint of simplifying the rotation calculation and ease of grasping the symmetric position, the subject coordinate system (s, t) and the image coordinate system (h, k) are set so that each coordinate value takes a positive or negative value. For example, the coordinates of the subject and the center position of the image sensor 24 are (0, 0), and the coordinates (m, n) are −M / 2 ≦ m ≦ M / 2, −N / 2 ≦ n. You may make it move in the range of ≦ N / 2. Accordingly, the setting of each coordinate system in the claims of the present application is also a setting for convenience of explanation, and the present invention is not limited to the setting of the expression format described in the claims, and is substantially the same. Any setting may be used as long as it is possible to perform a simple process.

  In the present embodiment, the image modification processing apparatus 30 improves the quality of an image obtained by imaging the subject using the bifocal lens 21 as follows.

First, before imaging the subject, each element W _{m, n} (x of the PSF matrix W _{m, n} for each coordinate (m, n) on the line 50 (see FIG. 7) is obtained based on the equation (18). , Y), the value of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} for each coordinate (m, n) on the line 60 (see FIG. 8) is calculated in advance. Then, it is stored in the convolution operation matrix storage means 32 and 33 of the image modification processor 30 as a sampling matrix. Note that it is only necessary to store each value of the matrix portion H (see FIG. 4) including non-zero elements, and since the distribution is symmetrical in the vertical direction as shown in FIG. 6 on the line 60, only each value of the upper half is stored. It's okay.

Further, rotation information (distance r and angle θ) of the convolution calculation matrix Q _{m, n} other than the sampling matrix is calculated in advance by Expression (21) and Expression (22), and rotation information storage means of the image modification processing device 30 is obtained. 34 is stored.

  Next, it is determined whether a normal subject or a close subject is to be imaged, and a switching selection operation of the switching operation unit 37 is performed, and then the subject is imaged by the imaging mechanism 20. At this time, light emitted from the subject passes through the lens portions 22 and 23 of the bifocal lens 21 and reaches the image sensor 24. Then, the output signal of the image sensor 24 that has received light from the subject is extracted, taken into the image modification processing device 30, and stored in the output signal storage unit 31.

Subsequently, the value of each element Q _{m, n} (x, y) of the sampling matrix Q _{m, n} stored in either of the convolution calculation matrix storage means 32 or 33 by the convolution calculation matrix rotation calculation means 35 (FIG. 6). 4), and the rotation information stored in the rotation information storage means 34, and the rotation information stored in the rotation information storage means 34. The value of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} other than the sampling matrix is calculated.

Further, the value of each element Q _{m, n} (x, y) stored in one of the convolution calculation matrix storage means 32 and 33 and each element calculated by the convolution calculation matrix rotation calculation means 35 by the reproduction calculation means 36. The value of Q _{m, n} (x, y) (each value of the matrix portion H in FIG. 4) and each element Z (m + x, n + y) of the image matrix Z indicating the output signal of the image stored in the output signal storage means 31 ) And the value of each element A (m, n) of the subject matrix A is calculated based on the equation (19).

  Thereafter, using the value of each element A (m, n) of the obtained subject matrix A, the subject that is the imaging target is displayed on the screen of the display means 40. If necessary, the subject may be printed by an output unit such as a printer (not shown).

According to this embodiment, there are the following effects. That is, since the image modification processing device 30 includes the reproduction calculation means 36, when the subject is imaged by the bifocal lens 21, the convolution calculation matrix Q _m, stored in the convolution calculation matrix storage means 32, 33 is stored _. each element Q _m of _n, and at least some of the values of _n (x, y), the value of each element Z (m + x, n + y) of the image matrix Z indicating the output signal of the image obtained by imaging Can be used to calculate the value of each element A (m, n) of the subject matrix A based on the equation (19).

At this time, the calculation processing by the reproduction calculation means 36 is performed by calculating each element Q _{m, n} (x of the convolution calculation matrix Q _{m, n} calculated in advance based on the equation (19) and stored in the convolution calculation matrix storage means 32 and 33. , Y) using the value of M, N rows, M × N columns and a large inverse transformation matrix T _g ⁻¹ as shown in equation (3). The value of each element A (m, n) of the subject matrix A can be calculated with a small amount of calculation.

  Accordingly, since the performance condition required for the CPU is relaxed, the reproduction of the subject can be performed in a short time even with the capability of the CPU installed in a portable information terminal device such as a cellular phone or a portable information terminal. Can be executed in the process. Therefore, if the image modification processing device 30 is installed in a portable information terminal device, a portable information terminal device having an image modification function can be realized, and the usability and performance of the information terminal device can be improved. Can do.

Further, the convolution calculation matrix storage means 32 and 33 store the convolution calculation matrix Q _{m, n} for each coordinate (m, n) on the line 60 (see FIG. 8). In addition, it is possible to perform image modification processing considering that the shape of blur is different for each pixel. For this reason, the image modification effect can be further enhanced as compared with the case of performing the image modification process assuming that the same shape blur occurs in all the pixels.

Furthermore, since the image modification processing device 30 includes the rotation information storage unit 34 and the convolution calculation matrix rotation calculation unit 35, the optical axis position (the subject and the image sensor 24) is utilized by utilizing the axial symmetry of the bifocal lens 21. The convolution calculation matrix Q _{m, n} for each coordinate (m, n) on the line 60 (see FIG. 8) stored in the convolution calculation matrix storage means 32 and 33 as a sampling matrix is rotated around the center position). Thus, the convolution calculation matrix Q _{m, n} for other coordinates can be calculated. For this reason, the data amount memorize | stored in the convolution calculation matrix memory | storage means 32 and 33 can be reduced.

Then, as shown in FIG. 10, the convolution calculation matrix rotation calculating unit 35 obtains a second partitioned area that overlaps a predetermined position (the center position in the present embodiment) of the first partitioned area, and the value of the second partitioned area. Is used to calculate and determine the value of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} other than the sampling matrix, so that the weighted average based on the area ratio is calculated using Equation (20). As compared with the case where the value of the element Q _{m, n} (x, y) corresponding to the first partition region is strictly calculated, the processing content can be simplified and the processing time can be shortened.

  In addition, since the image modification processing apparatus 30 includes the switching operation unit 37, the data of the convolution calculation matrix storage units 32 and 33 are switched and selected according to the distance of the subject to be imaged, and the subject reproduction calculation is performed. Can do. For this reason, whether the subject is a normal subject at a standard distance or a close subject closer than this, the focus formed by one lens unit constituting the bifocal lens 21 Image modification processing that finds an in-focus image from an image in which the combined image and the out-of-focus image formed by the other lens unit overlap can be realized in a short time. A clear image can be obtained for both the subject and the close subject.

  Note that the present invention is not limited to the above-described embodiment, and modifications and the like within a scope where the object of the present invention can be achieved are included in the present invention.

That is, in the above-described embodiment, the convolution calculation matrix rotation calculating unit 35 obtains the second partitioned area that overlaps the predetermined position (the center position in the present embodiment) of the first partitioned area, as shown in FIG. The value of the element Q _{m, n} (x, y) corresponding to the first partition area is calculated and determined based on the value of the second partition area. A first partitioned area in which a predetermined position (for example, the center position) of the partitioned area overlaps is obtained, and the value of the element Q _{m, n} (x, y) corresponding to the obtained first partitioned area is determined as the first partitioned area. It is good also as a structure which employ | adopts the value of the 2nd division area where the predetermined position has overlapped. For example, in the case of FIG. 10, since the first partitioned area where the center positions of the second partitioned areas of f16 overlap is the hatched central first partitioned area, the value of the central first partitioned area F16 is adopted.

Moreover, in the said embodiment, although this invention was applied to the bifocal lens 21, this invention may be applied to a single focus lens. FIG. 13 shows a comparison result of the blur shapes (distribution of the PSF matrix W _{m, n} indicating the spread of light from a single bright spot) between the bifocal lens 21 and the single focus lens 90.

In FIG. 13, the distribution of the PSF matrix W _{m, n} when a normal subject at a standard distance is imaged by the bifocal lens 21 is considered when the long-focus lens unit 22 is considered to be an ideal lens. Formed around W _{m, n} (0,0) by the actual long-focus lens unit 22 with W _{m, n} (0,0) being the center of the image formed by the long-focus lens unit 22. A substantially solid circular shape or a substantially solid elliptical blurred portion 80 and a ring-shaped (substantially annular or substantially elliptical) blurred portion 81 formed by the short focus lens portion 23. .

In addition, the distribution of the PSF matrix W _{m, n} in the case where an object close to the standard distance is imaged by the bifocal lens 21 is considered when the short focus lens unit 23 is considered to be an ideal lens. this short focus lens unit 23 W _m of the image in focus is formed _by, centered on _n (0,0), W _m the actual short-focus lens unit _23, the periphery of the _n (0,0) A substantially solid circular shape or a substantially solid elliptical blur portion 82 formed, and a substantially solid circular shape or a substantially solid elliptical blur portion 83 formed by the long focus lens portion 22 are provided. Yes. These blurred portions 82 and 83 overlap each other.

Furthermore, the distribution of the PSF matrix W _{m, n} when a normal subject at a standard distance or a close subject at a shorter distance is imaged by the single focus lens 90 indicates that the single focus lens 90 is an ideal lens. and is a W _m of the image in focus by the single focus lens 90 is formed when _considering, centered _n (0,0) is, W _m the actual focal length lens _{90, n} (0,0 ) Are formed in a substantially solid circular shape or a substantially solid elliptical shape with a slight blur portion 91. The formation of the blurred portion 91 by the single focus lens 90 is the case where the long focus lens portion 22 forms the blurred portion 80 when a normal subject is imaged by the bifocal lens 21 and the case where a close subject is imaged by the bifocal lens 21. In principle, it has the same meaning as the formation of the blurred portion 82 by the short focus lens portion 23.

In the present invention, a PSF matrix W _{m, n} (x, y) for defining a convolution calculation matrix Q _{m, n} (x, y) used in subject reproduction calculation processing by the reproduction calculation means 36 is as follows. Can be determined. First, when an image by an ordinary subject bifocal lens 21, for the blurred portions 80 may be handled equivalent to blur portions 81, or, W _m and the sum of the values of each element of the blurred portion 80 _{, n} (0,0) may be replaced, and the value of each element of the blurred portion 80 excluding W _{m, n} (0,0) may be zero. In the latter case, the distribution is the same as that in the case where an image in which W _{m, n} (0, 0) is completely in focus is formed by the ideal long focus lens unit 22.

Next, when a close subject is imaged by the bifocal lens 21, the blurred portion 82 may be handled in the same manner as the blurred portion 83, or the value of each element of the blurred portion 82 (actually, a long Although the output signal value of the image of the blurred portion 83 by the focus lens unit 22 overlaps, the magnitude of this value is small, so it is considered that only the output signal value of the image of the blurred portion 82 by the short focus lens unit 23 is present. good.) summing the replacement value of W _{m, n} (0,0) with and, W _{m, n} the values of the elements of the blurred portion 82 except the (0,0) may be zero. In the latter case, the distribution is the same as that in the case where an image in which W _{m, n} (0, 0) is completely focused is formed by the ideal short focus lens unit 23.

Further, when a normal subject or a close subject is imaged by the single focus lens 90, the values of W _{m, n} (0, 0) are replaced by summing the values of the elements of the blur portion 91 to obtain the blur portion. This method is not adopted and the value of W _{m, n} (0, 0) is left as it is.

  As described above, the image modification processing method and apparatus, the program, and the data recording medium of the present invention include, for example, a personal digital assistant (PDA) and a personal handy phone system (PHS). ), Remote control device for operation of household appliances such as television and video, personal computer with camera, and information terminal device with image input function by bifocal lens and single focus lens such as surveillance camera device can do.

1 is an overall configuration diagram of an imaging system including an image modification processing apparatus according to an embodiment of the present invention. FIG. 2 is a detailed configuration diagram of a bifocal lens that forms an image to be processed by the image modification processing apparatus of the embodiment. Explanatory drawing of the relationship between a to-be-photographed object and the image obtained by imaging this to-be-photographed object with a bifocal lens. Explanatory drawing of the conversion relationship between the subject matrix A and the image matrix Z. The illustration figure of the distribution condition of each element Wm _{, n} (x, y) of PSF matrix Wm _{, n} . The illustration figure of the distribution condition of each element Qm _{, n} (x, y) of the convolution operation matrix Qm _{, n} . Illustration of distribution of PSF matrix W _m, the elements of _{n W m, n (x,} y) for each coordinate (m, n). Illustration of distribution of the coordinate (m, n) convolution matrix for Q _m, each element of the _{n Q m, n (x,} y). FIG. 7 is an exemplary diagram illustrating an arrangement in which each element Q _{m, n} (x, y) of a convolution arithmetic matrix Q _{m, n} for each coordinate lined up on a horizontal line shown in FIG. 6 is rotated 45 degrees. The enlarged view of the area | region for nine pixels enclosed with the thick frame among the arrangement | positioning of FIG. The figure which shows the result of having calculated | required and calculated the value of each 1st division area by the value of the 2nd division area which calculated | required the 2nd division area which overlaps the center position of each 1st division area of FIG. Explanatory drawing of the rotation information of the convolution calculation matrix Qm _{, n} . The figure which shows the comparison result of the blur shape (distribution of PSF matrix Wm _{, n} which shows the spread of the light from a single luminescent point) of a bifocal lens and a single focus lens.

Explanation of symbols

DESCRIPTION OF SYMBOLS 21 Bifocal lens 22 Long-focus lens part applicable to one or other lens part 23 Short-focus lens part applicable to one or other lens part 24 Image sensor 30 Image modification processing apparatus 32, 33 Convolution calculation matrix memory | storage means 35 Convolution calculation matrix rotation calculation means 36 Playback calculation means A Subject matrix Z Image matrix E, H Matrix portion including non-zero elements _{Wm, n} point (m, n) point spread function matrix Q _{m, n} coordinate ( convolution matrix for m, n)

Claims (10)

An image modification processing method for modifying a blurred image formed by a lens into a focused image,
The size of the image sensor is M pixels × N pixels, the matrix of M rows and N columns indicating the brightness of light emitted from the subject is A, and an output signal of an image obtained by imaging the subject with the lens is shown. The matrix of M rows and N columns is Z, and the object coordinate system and the object coordinate system are set so that the imaging position of light emitted from one point (m, n) in the subject coordinate system is one point (m, n) in the image coordinate system. When setting the image coordinate system,
The value of each element Q _{m, n} (x, y) of the (2M-1) row (2N-1) column convolution matrix Q _{m, n} for the coordinates (m, n) for performing the convolution operation processing Among them, the values of the matrix part including at least non-zero elements are calculated in advance for all or part of each coordinate (m, n) based on the following equation (A-1) and stored in the convolution matrix storage means. ,
When the subject is imaged by the lens, the reproduction calculation means outputs at least a part of values of each element Q _{m, n} (x, y) stored in the convolution calculation matrix storage means and the output of the image The value of each element A (m, n) of the subject matrix A is calculated based on the following equation (A-2) using the value of each element Z (m + x, n + y) of the signal matrix Z. An image modification processing method.
Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （A-1）
A (m, n) = Σ x Σ y Q m, n (x, y) Z (m + x, n + y)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （A-2）
here,
x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ (N−1),
m and n are natural numbers, 1 ≦ m ≦ M, 1 ≦ n ≦ N,
W _{m, n} (x, y) is the value of each element of the matrix W _{m, n} of (2M-1) rows (2N-1) columns, and this matrix W _{m, n} is one point of the subject. (M _{, n} ) is a point spread function matrix indicating a state in which light emitted from (m, n) spreads on the image pickup device by the action of the lens, and W _{m, n} (0, 0) is located at the center of the spread W _{m, n} (−x, −y) is the value of the output signal of the pixel located in the surrounding blurred portion,
power is a real number that is a power of W _{m, n} (0, 0), and 1 ≦ power ≦ 2.
The sigma _x, the sum of x = (1-M) ~ (M-1), Σ y is the sum of y = (1-N) ~ (N-1). The image modification processing method according to claim 1,
Of the convolution matrix Q _{m, n} for each coordinate (m, n), select the convolution matrix Q _{m, n} for coordinates aligned on a straight line extending in one direction from the optical axis position as the sampling matrix,
In the convolution calculation matrix storage means, only the value of each element Q _{m, n} (x, y) of the sampling matrix Q _{m, n} is stored,
The value of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} for other coordinates is calculated by the convolution matrix rotation calculation means using the axial symmetry of the lens, and the sampling matrix Q _m, each element Q _m of _{n, n (x,} y) image modification processing method and calculating by rotating the arrangement of values of about said optical axis position. The image modification processing method according to claim 2,
When calculating the value of each element Q _{m, n} (x, y) of the convolution matrix Q _{m, n} for the other coordinates by the convolution matrix rotation calculation means,
When the arrangement of values of each element Q _{m, n} (x, y) of the sampling matrix Q _{m, n} is rotated,
A calculation target convolution matrix Q _m, element Q _m of _{n, n (x,} y) the sampling matrix Q after rotation overlapping the predetermined position of the first divided area corresponding to the _{m, n} elements Q _{m of, n} A second partition region corresponding to (x, y) is obtained, and the value of the element Q _{m, n} (x, y) corresponding to the obtained second partition region is calculated from the convolution operation matrix Q _{m, n} to be calculated. Adopted as the value of element Q _{m, n} (x, y),
Alternatively, the first partition area where the predetermined positions of the second partition area overlap is obtained, and the value of the element Q _{m, n} (x, y) corresponding to the calculated first partition area is determined as the first partition area. A value of an element Q _{m, n} (x, y) corresponding to the second partitioned area where the predetermined position overlaps the area is adopted. In the image modification processing method according to any one of claims 1 to 3,
The lens is a bifocal lens;
The matrix Z indicates an output signal of an image in which an in-focus image formed by one lens unit constituting the bifocal lens and an in-focus image formed by the other lens unit overlap each other. Matrix
In the point spread function matrix W _{m, n} , the value of W _{m, n} (0,0) is determined mainly by the action of the one lens part, and W _{m, n} ( An image modification processing method characterized in that a value of -x, -y) is determined. An image modification processing apparatus for modifying a blurred image formed by a lens into a focused image,
The size of the image sensor is M pixels × N pixels, the matrix of M rows and N columns indicating the brightness of light emitted from the subject is A, and an output signal of an image obtained by imaging the subject with the lens is shown. The matrix of M rows and N columns is Z, and the object coordinate system and the object coordinate system are set so that the imaging position of light emitted from one point (m, n) in the subject coordinate system is one point (m, n) in the image coordinate system. When setting the image coordinate system,
For all or part of each coordinate (m, n), (2M−1) rows (2N−) for the coordinates (m, n) for performing the convolution calculation processing calculated based on the following equation (B-1) 1) Convolution operation matrix storage means for storing values of matrix portions including at least non-zero elements among the values of each element Q _{m, n} (x, y) of the column convolution operation matrix Q _{m, n} ;
The value of at least a part of each element Q _{m, n} (x, y) stored in the convolution calculation matrix storage means and the value of each element Z (m + x, n + y) of the matrix Z of the output signal of the image, An image modification processing apparatus comprising: a reproduction calculation means for calculating a value of each element A (m, n) of the matrix A of the subject based on the following formula (B-2) using:
Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （B-1）
A (m, n) = Σ x Σ y Q m, n (x, y) Z (m + x, n + y)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （B-2）
here,
x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ (N−1),
m and n are natural numbers, 1 ≦ m ≦ M, 1 ≦ n ≦ N,
W _{m, n} (x, y) is the value of each element of the matrix W _{m, n} of (2M-1) rows (2N-1) columns, and this matrix W _{m, n} is one point of the subject. (M _{, n} ) is a point spread function matrix indicating a state in which light emitted from (m, n) spreads on the image pickup device by the action of the lens, and W _{m, n} (0, 0) is located at the center of the spread W _{m, n} (−x, −y) is the value of the output signal of the pixel located in the surrounding blurred portion,
power is a real number that is a power of W _{m, n} (0, 0), and 1 ≦ power ≦ 2.
The sigma _x, the sum of x = (1-M) ~ (M-1), Σ y is the sum of y = (1-N) ~ (N-1). The image modification processing apparatus according to claim 5,
The said convolution matrix storage means, the coordinates (m, n) convolution matrix Q _m in the _convolution operation matrix Q _m of coordinates aligned on a straight line extending from the optical axis position in one direction of _{n, n} is Only the value of each element Q _{m, n} (x, y) of this sampling matrix Q _{m, n} selected as the sampling matrix is stored,
The values of the elements Q _{m, n} (x, y) of the convolution arithmetic matrix Q _{m, n} for other coordinates are used as the elements Q _{m of the} sampling matrix Q _{m, n} using the axial symmetry of the lens. _{, n} (x, y) is provided with convolution operation matrix rotation calculating means for calculating by rotating the arrangement of values about the optical axis position. The image modification processing apparatus according to claim 6,
The convolution operation matrix rotation calculation means is:
When the arrangement of values of each element Q _{m, n} (x, y) of the sampling matrix Q _{m, n} is rotated,
A calculation target convolution matrix Q _m, element Q _m of _{n, n (x,} y) the sampling matrix Q after rotation overlapping the predetermined position of the first divided area corresponding to the _{m, n} elements Q _{m of, n} A second partition region corresponding to (x, y) is obtained, and the value of the element Q _{m, n} (x, y) corresponding to the obtained second partition region is calculated from the convolution operation matrix Q _{m, n} to be calculated. Adopted as the value of element Q _{m, n} (x, y),
Alternatively, the first partition area where the predetermined positions of the second partition area overlap is obtained, and the value of the element Q _{m, n} (x, y) corresponding to the calculated first partition area is determined as the first partition area. An image modification processing apparatus characterized by adopting a value of an element Q _{m, n} (x, y) corresponding to the second partitioned area where the predetermined position overlaps the area. In the image modification processing apparatus according to any one of claims 5 to 7,
The lens is a bifocal lens;
The matrix Z indicates an output signal of an image in which an in-focus image formed by one lens unit constituting the bifocal lens and an in-focus image formed by the other lens unit overlap each other. Matrix
In the point spread function matrix W _{m, n} , the value of W _{m, n} (0,0) is determined mainly by the action of the one lens part, and W _{m, n} ( An image modification processing apparatus characterized in that a value of -x, -y) is determined. A program for causing a computer to function as an image modification processing device that modifies a blurred image formed by a lens into a focused image,
The size of the image sensor is M pixels × N pixels, the matrix of M rows and N columns indicating the brightness of light emitted from the subject is A, and an output signal of an image obtained by imaging the subject with the lens is shown. The matrix of M rows and N columns is Z, and the object coordinate system and the object coordinate system are set so that the imaging position of light emitted from one point (m, n) in the subject coordinate system is one point (m, n) in the image coordinate system. When setting the image coordinate system,
For all or part of each coordinate (m, n), (2M−1) rows (2N−) for the coordinates (m, n) for performing the convolution calculation processing calculated based on the following equation (C-1) 1) Convolution operation matrix storage means for storing values of matrix portions including at least non-zero elements among the values of each element Q _{m, n} (x, y) of the column convolution operation matrix Q _{m, n} ;
The value of at least a part of each element Q _{m, n} (x, y) stored in the convolution calculation matrix storage means and the value of each element Z (m + x, n + y) of the matrix Z of the output signal of the image, An image modification processing apparatus comprising: a calculation unit that calculates a value of each element A (m, n) of the subject matrix A based on the following formula (C-2) using: A program that allows a computer to function.
Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （C-1）
A (m, n) = Σ x Σ y Q m, n (x, y) Z (m + x, n + y)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （C-2）
here,
x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ (N−1),
m and n are natural numbers, 1 ≦ m ≦ M, 1 ≦ n ≦ N,
W _{m, n} (x, y) is the value of each element of the matrix W _{m, n} of (2M-1) rows (2N-1) columns, and this matrix W _{m, n} is one point of the subject. (M _{, n} ) is a point spread function matrix indicating a state in which light emitted from (m, n) spreads on the image pickup device by the action of the lens, and W _{m, n} (0, 0) is located at the center of the spread W _{m, n} (−x, −y) is the value of the output signal of the pixel located in the surrounding blurred portion,
power is a real number that is a power of W _{m, n} (0, 0), and 1 ≦ power ≦ 2.
The sigma _x, the sum of x = (1-M) ~ (M-1), Σ y is the sum of y = (1-N) ~ (N-1). A computer-readable data recording medium on which data used in an image modification process for modifying a blurred image formed by a lens into a focused image is recorded.
The size of the image sensor is M pixels × N pixels, the matrix of M rows and N columns indicating the brightness of light emitted from the subject is A, and an output signal of an image obtained by imaging the subject with the lens is shown. The matrix of M rows and N columns is Z, and the object coordinate system and the object coordinate system are set so that the imaging position of light emitted from one point (m, n) in the subject coordinate system is one point (m, n) in the image coordinate system. When setting the image coordinate system,
As a matrix used to calculate the matrix A from the matrix Z, the coordinates (m for performing the convolution calculation processing calculated based on the following equation (D-1) for all or part of the coordinates (m, n) , N) of the matrix part including at least non-zero elements among the values of each element Q _{m, n} (x, y) of the (2M−1) row (2N−1) column convolution matrix Q _{m, n} A computer-readable data recording medium on which values are recorded.
Q _{m, n} (x, y) = 1 / W _{m, n} (0,0)
(When x = 0, y = 0)
= −W _{m, n} (−x, −y) / W _{m, n} (0,0) ^power
(When other than x = 0, y = 0)
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ （D-1）
here,
x and y are integers, (1-M) ≦ x ≦ (M−1), (1-N) ≦ y ≦ (N−1),
m and n are natural numbers, 1 ≦ m ≦ M, 1 ≦ n ≦ N,
W _{m, n} (x, y) is the value of each element of the matrix W _{m, n} of (2M-1) rows (2N-1) columns, and this matrix W _{m, n} is one point of the subject. (M _{, n} ) is a point spread function matrix indicating a state in which light emitted from (m, n) spreads on the image pickup device by the action of the lens, and W _{m, n} (0, 0) is located at the center of the spread W _{m, n} (−x, −y) is the value of the output signal of the pixel located in the surrounding blurred portion,
power is a real number that is a power of W _{m, n} (0, 0), and 1 ≦ power ≦ 2.

Images