JP2020098963A - Imaging device - Google Patents

Abstract

Provided is an imaging device having a simple configuration, a thin shape, and a wide field of view at the time of close-up photography without distortion. An image pickup device includes an image sensor that converts an optical image captured by a plurality of pixels arranged in an array on an image pickup surface into an image signal and outputs the image signal, and a light receiving surface of the image sensor. A modulation unit that modulates the intensity of light using the image processing unit, an image processing unit that performs predetermined image processing on the output image output from the image sensor, and distortion correction on the image output from the image processing unit. An image distortion correction unit and an image distortion correction setting value input unit that transfers setting values for image distortion correction to the image distortion correction unit are provided. [Selection diagram] Fig. 30

Description

The present invention relates to an imaging device.

An imaging device mounted on a device such as an IoT (Internet of things) device or a wearable device is required to be thin and low in cost in order to be incorporated in a limited space. Therefore, for example, as in Patent Document 1, an imaging method capable of obtaining an object image without using a lens has been proposed. Further, for example, Patent Document 2 proposes an “imaging device that enables a finger authentication device with a small thickness that can be installed in a portable information device” using a microlens array.

US Patent Application Publication No. 2015/0219808 JP, 2011-203792, A

The above-mentioned Patent Document 1 is an imaging method in which a special lattice pattern is arranged in front of the image sensor, and an inverse problem is solved by signal processing using a diagonal pattern of the lattice pattern received by the image sensor to obtain an image of the object. .. This imaging method enables the imaging device to be thin, but has a problem that the calculation for obtaining an image becomes complicated.

In addition, Japanese Patent Laid-Open No. 2004-242242 discloses that "a microlens array in the front stage and a diaphragm array are opposed to each other, a diaphragm of the diaphragm array is provided in the vicinity of the focal point of light from a subject in the microlens array in the front stage, and a microlens in the rear stage is further provided. This is an imaging method in which the optical system used for the finger vein authentication device can be made small and a small and thin finger vein authentication device can be realized by "focusing light at the same time by the array". This imaging method enables a thin imaging device capable of close-up photography, but is thin because it requires a distance for condensing by a lens and a space for arranging two lens arrays. Has become the limit.

It is an object of the present invention to provide an image pickup apparatus having a simple structure, a thin shape, and a wide field of view at the time of close-up photography without distortion.

The present application includes a plurality of means for solving at least a part of the above problems, and examples thereof are as follows.

One embodiment of the present invention is an imaging device, which includes an image sensor that converts an optical image captured by a plurality of pixels arranged in an array on the imaging surface into an image signal and outputs the image signal, and a light-receiving surface of the image sensor. A modulation unit that modulates the intensity of light using a lattice pattern, an image processing unit that performs predetermined image processing on the output image output from the image sensor, and an output from the image processing unit. An image distortion correction unit that performs distortion correction on an image, and an image distortion correction set value input unit that transfers a set value for image distortion correction to the image distortion correction unit are provided.

According to the present invention, it is possible to provide an image pickup apparatus having a simple structure, a thin shape, and a wide field of view at the time of close-up photography without distortion.

Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

It is a figure which shows the structural example of the imaging device which concerns on 1st Embodiment. It is a figure which shows the structural example of a modulator. It is a figure which shows the other structural example of a modulator. It is a figure which shows a mode that the external object is imaged using an imaging device. It is a flow chart which shows an example of image processing of an image processing part. It is a figure explaining that the projection image from the grating substrate front surface to the back surface due to obliquely incident parallel light causes in-plane displacement. It is a figure explaining generation|occurrence|production of a moire fringe and frequency spectrum when the axis of the grating|lattice on both surfaces of a grating|lattice board|substrate is aligned, (a) shows the case of an incident angle: 0, (b) shows the case of an incident angle: +(theta). (C) shows the case where the incident angle is −θ. It is a figure which shows an example of the grating|lattice board which has arrange|positioned the axis|shaft of the surface grating|lattice and the back surface grating|lattice offset. It is a figure explaining generation|occurrence|production of a moire fringe and frequency spectrum at the time of arranging a grating|lattice on both surfaces of a grating|lattice substrate, (a) shows the case of an incident angle: 0, (b) shows the case of an incident angle: +(theta). (C) shows the case where the incident angle is −θ. It is a figure which shows an example of a lattice pattern. It is a figure which shows the other example of a lattice pattern. It is a figure which shows the further another example of a lattice pattern. It is a figure explaining the angle which the light from each point which comprises an object makes with respect to an image sensor. It is a figure explaining that a front side lattice pattern is projected when an object is in infinite distance. It is a figure which shows the example of the moire fringe produced|generated when an object exists in infinite distance. It is a figure explaining that a front side lattice pattern is expanded when an object is in a finite distance. It is a figure which shows the example of the moire fringe produced|generated when an object exists in a finite distance. It is a figure which shows the example of the moire fringe which corrected the back side lattice pattern, when an object exists in a finite distance. It is a figure which shows the structural example of the imaging device which implement|achieves a back side lattice pattern by image processing. It is a figure which shows the structural example of a modulator. It is a flow chart which shows an example of image processing of an image processing part. It is a figure which shows the structural example of the imaging device which implement|achieves a time division fringe scan. It is a figure which shows the example of the lattice pattern in a time division fringe scan. It is a figure which shows the structural example of the modulator which implement|achieves a time division fringe scan. 9 is a flowchart illustrating an example of image processing of an image processing unit that realizes time-division fringe scanning. It is a figure which shows the structural example of the imaging device which implement|achieves space division fringe scanning. It is a figure which shows the example of the lattice pattern in space division fringe scan. It is a figure explaining a mode that a light ray from a subject shifts and is projected on an image sensor by the thickness and refractive index of an optical part, and a subject position. It is a figure which shows an example of distortion generation of the to-be-photographed object. It is a figure which shows the structural example of the imaging device which implement|achieves photographing a to-be-photographed object in the state which image distortion was corrected by image distortion correction. It is a figure explaining angle of view _thetamax2 . It is a figure explaining the range and visual field of the lattice pattern which the light from a certain point which comprises an object irradiates, when an object is in a short distance. It is a figure which shows the structural example of the imaging device which implement|achieves a wide visual field by the combination of a grid pattern of a compound eye, several image sensors, and image distortion correction. It is a figure which shows an example of the lattice pattern of a compound eye. It is a figure explaining a visual field by a lattice pattern of a compound eye and a plurality of image sensors. 7 is a flowchart illustrating an example of image distortion correction processing. It is a figure explaining an example of a synthesizing method. It is a figure explaining a synthetic image. It is a figure explaining the example of arrangement of an image sensor. It is a figure explaining the other example of arrangement|positioning of an image sensor. It is a figure explaining an example of a gaze direction. It is a figure explaining the other example of a gaze direction. It is a figure which shows the structural example which memorize|stores an image distortion correction setting value in an image distortion correction part. It is a figure which shows the structural example which memorize|stores the image distortion correction setting value with the setting value learned by the image distortion correction setting value learning part. It is a figure which shows the structure which outputs the information of a sensor image and a required setting value outside from an imaging device, receives information outside, and performs image processing, image distortion correction, and image composition.

In the following embodiments, when there is a need for convenience, the description will be divided into a plurality of sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one is not There are some or all of the modifications, details, supplementary explanations, and the like.

In addition, in the following embodiments, unless otherwise specified, such as the number of elements (including the number, numerical value, amount, range, etc.), unless otherwise explicitly stated and in principle limited to a specific number, etc. However, the number is not limited to the specific number, and may be a specific number or more or less.

Further, in the following embodiments, it is needless to say that the constituent elements (including element steps and the like) are not necessarily essential unless otherwise specified or in principle considered to be essential. Yes.

Similarly, in the following embodiments, when referring to the shapes and positional relationships of constituent elements and the like, the shapes and the like of the constituent elements are substantially the same, unless otherwise specified or when it is considered that the principle is not clear. And the like, etc. are included. This also applies to the above numerical values and ranges.

In addition, in all the drawings for explaining the embodiments, the same members are denoted by the same reference symbols in principle and their repeated description is omitted. Hereinafter, each embodiment of the present invention will be described with reference to the drawings.

[First embodiment]
<Principle of shooting objects at infinity>
FIG. 1 is a diagram showing a configuration example of an image pickup apparatus according to the first embodiment. The imaging device 101 acquires an image of an object in the external world without using a lens for forming an image, which is included in a normal camera, and includes a modulator 102, an image sensor 103, and an image processing unit 106. Has been done.

FIG. 2 is a diagram showing a configuration example of the modulator. The modulator 102 is in close contact with and fixed to the light receiving surface of the image sensor 103 (FIG. 1 ), and the grating substrate 102 a and the first grating pattern 104 and the second grating formed on the upper and lower surfaces of the grating substrate 102 a. And a pattern 105. The lattice substrate 102a is made of a transparent material such as glass or plastic. Hereinafter, the image sensor 103 side of the lattice substrate 102a is referred to as a back surface, and the facing surface, that is, the imaging target side is referred to as a front surface.

These lattice patterns 104 and 105 are formed of concentric lattice patterns in which the intervals of the lattice patterns, that is, the pitches are reduced in inverse proportion to the radius from the center toward the outside. The lattice patterns 104 and 105 are formed by depositing a metal such as aluminum or chromium by a sputtering method used in a semiconductor process, for example. The metallization pattern and the non-metallization pattern create a shade.

The formation of the grid patterns 104 and 105 is not limited to vapor deposition, and may be formed by printing with an ink jet printer or the like with different shades. Furthermore, although visible light has been described as an example here, in the case of imaging with far infrared rays, for example, the grating substrate 102a is made of a material that is transparent to far infrared rays, such as germanium, silicon, or chalcogenide. That is, the grating substrate 102a can be made of a material transparent to the wavelength to be imaged, and the grating patterns 104 and 105 can be made of a material such as a metal that blocks the wavelength to be imaged. ..

Although the method of forming the grating patterns 104 and 105 on the grating substrate 102a in order to realize the modulator 102 has been described here, the present invention is not limited to this. FIG. 3 is a diagram showing another configuration example of the modulator 102. The modulator 102 can also be realized by forming the lattice patterns 104 and 105 in a thin film and holding these by a supporting member 102b provided in place of the lattice substrate 102a.

The image sensor 103 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Pixels 103a, which are light receiving elements, are regularly arranged in a grid pattern on the surface of the image sensor 103. The intensity of light transmitted through the lattice patterns 104 and 105 is modulated by the lattice patterns, and the light is received by the image sensor 103. The image sensor 103 converts the optical image received by the pixel 103a into an image signal which is an electric signal. The image signal (also referred to as “image data”) output from the image sensor 103 is subjected to image processing by the image processing unit 106 and output to the image display unit 107 or the like. The image processing unit 106 may store the processed image data in a storage device (not shown) included in the imaging device 101, or may output the processed image data to an external host computer or a recording medium.

The imaging device 101 can include, for example, a processor, a memory, a communication device, a processing circuit, and the like. Further, the image pickup apparatus 101 may include an input/output interface such as a USB (Universal Serial Bus) and an HDMI (High-Definition Multimedia Interface) that is connected to an external device. The image processing unit 106 may be realized by, for example, a dedicated image processing circuit or a processor that executes a program.

FIG. 4 is a diagram showing a state in which an object in the external world is photographed using the image pickup apparatus 101 of FIG. FIG. 4 illustrates an example in which the subject 401 is photographed by the imaging device 101 and is displayed on the image display unit 107. When the subject 401 is photographed, the surface of the modulator 102, specifically, the surface of the lattice substrate 102a on which the first lattice pattern 104 is formed faces the subject 401. Be seen.

Next, an outline of image processing by the image processing unit 106 will be described. FIG. 5 is a flowchart showing an example of image processing of the image processing unit 106.

First, the image processing unit 106 performs a two-dimensional Fourier transform operation such as a fast Fourier transform (FFT) on the Moire fringe image output from the image sensor 103 for each RGB (Red Green Blue) component of color. The frequency spectrum is obtained by the developing process according to (501). Subsequently, the image processing unit 106 cuts out data in a necessary frequency region from the frequency spectrum obtained by the processing of step 501 (502), and then calculates the intensity of the frequency spectrum (503) to acquire an image. Then, the image processing unit 106 performs noise removal processing on the obtained image (504), and subsequently performs contrast enhancement processing (505). After that, the image processing unit 106 adjusts the color balance of the image (506) and outputs it as a captured image. With the above, the image processing by the image processing unit 106 is completed.

Next, the shooting principle of the image pickup apparatus 101 will be described.

First, the concentric lattice patterns 104 and 105 in which the pitch becomes finer in inverse proportion to the radius from the center are defined as follows. In a laser interferometer, it is assumed that a spherical wave close to a plane wave interferes with a plane wave used as reference light. When the radius from the reference coordinates that is the center of the concentric circles is r and the phase of the spherical wave there is Φ(r), the spherical wave uses a coefficient β that determines the magnitude of bending of the wavefront,

と表せる。

Can be expressed as

Despite the spherical wave, it is represented by the square of the radius r because it is a spherical wave close to a plane wave and can be approximated only by the lowest order of expansion. When a plane wave is caused to interfere with light having this phase distribution,

のような干渉縞の強度分布が得られる。これは、

An intensity distribution of interference fringes such as this is,

を満たす半径位置で明るい線を持つ同心円の縞となる。縞のピッチをｐとすると、

Concentric stripes with bright lines at radial positions satisfying. If the pitch of stripes is p,

が得られ、ピッチは、半径に対して反比例して狭くなっていくことがわかる。このような縞を持つプレートは、フレネルゾーンプレートやガボールゾーンプレートと呼ばれる。本実施形態では、式（２）で定義される強度分布に比例した透過率分布をもった格子パターンを、図１に示した格子パターン１０４，１０５として用いる。

It can be seen that the pitch becomes narrower in inverse proportion to the radius. Plates having such stripes are called Fresnel zone plates and Gabor zone plates. In this embodiment, the lattice patterns having the transmittance distribution proportional to the intensity distribution defined by the equation (2) are used as the lattice patterns 104 and 105 shown in FIG.

FIG. 6 is a diagram for explaining that the projection image from the front surface to the back surface of the grating substrate due to the obliquely incident parallel light causes an in-plane shift. It is assumed that collimated light is incident on the modulator 102 having the thickness t in which the grating patterns 104 and 105 are formed on both surfaces at an angle θ ₀ . In terms of geometrical optics, the light multiplied by the transmittance of the grating on the front surface is shifted by δ=t·tan θ and enters the rear surface, and the center of the two concentric circular gratings is assumed. If they are aligned, the transmittance of the grating on the back surface is shifted by δ and the transmittance is multiplied. At this time,

のような強度分布が得られる。この展開式の第４項が、２つの格子のずれの方向にまっすぐな等間隔の縞模様を、２つの格子の重なり合った領域一面に作ることがわかる。このような縞と縞の重ね合わせによって相対的に低い空間周波数で生じる縞は、モアレ縞と呼ばれる。このようにまっすぐな等間隔の縞は、検出画像の２次元フーリエ変換によって得られる空間周波数分布に鋭いピークを生じる。その周波数の値からδの値、すなわち光線の入射角θを求めることが可能となる。このような全面で一様に等間隔で得られるモアレ縞は、同心円状の格子配置の対称性から、ずれの方向によらず同じピッチで生じることは明らかである。このような縞が得られるのは、格子パターンをフレネルゾーンプレートまたはガボールゾーンプレートで形成したことによるものであるが、全面で一様に等間隔なモアレ縞が得られるのであればどのような格子パターンを使用してもよい。

An intensity distribution such as It can be seen that the fourth term of this expansion formula forms straight, equally-spaced striped patterns in the direction of displacement of the two grids over the area where the two grids overlap. The fringes generated at a relatively low spatial frequency due to the superposition of such fringes are called moire fringes. Such straight, evenly spaced fringes produce sharp peaks in the spatial frequency distribution obtained by the two-dimensional Fourier transform of the detected image. The value of δ, that is, the incident angle θ of the light ray can be obtained from the value of the frequency. Due to the symmetry of the concentric lattice arrangement, it is clear that the moire fringes obtained at equal intervals on the entire surface occur at the same pitch regardless of the direction of deviation. Such stripes are obtained because the lattice pattern is formed by a Fresnel zone plate or a Gabor zone plate. However, as long as evenly spaced moire fringes can be obtained over the entire surface, any lattice Patterns may be used.

Here, only the component having a sharp peak from equation (5) is

のように取り出すと、そのフーリエスペクトルは、

, The Fourier spectrum is

のようになる。ここで、Ｆはフーリエ変換の演算を表し、ｕおよびｖは、ｘ方向およびｙ方向の空間周波数座標、括弧を伴うδはデルタ関数である。この結果から、検出画像の空間周波数スペクトルにおいて、モアレ縞の空間周波数のピークがｕ＝±δβ／πの位置に生じることがわかる。その様子を図７に示す。

become that way. Here, F represents a Fourier transform operation, u and v are spatial frequency coordinates in the x and y directions, and δ with parentheses is a delta function. From this result, it can be seen that in the spatial frequency spectrum of the detected image, the peak of the spatial frequency of moire fringes occurs at the position of u=±δβ/π. This is shown in FIG.

FIG. 7 is a diagram for explaining the generation of moire fringes and the frequency spectrum when the axes of the gratings on both sides of the grating substrate are aligned. In FIG. 7, from left to right, a schematic view of the arrangement of the light beam and the modulator 102, moire fringes, and a spatial frequency spectrum are shown. 7A shows the case of vertical incidence, FIG. 7B shows the case of incident light rays from the left side at an angle θ, and FIG. 7C shows the case of incident light rays from the right side at an angle θ. Shown respectively.

The first grid pattern 104 formed on the front surface side of the modulator 102 and the second grid pattern 105 formed on the back surface side have the same axis. In FIG. 7A, since the shadows of the first lattice pattern 104 and the second lattice pattern 105 coincide with each other, moire fringes do not occur.

In FIGS. 7B and 7C, the same moiré occurs because the first grating pattern 104 and the second grating pattern 105 have the same deviation, and the peak positions of the spatial frequency spectrum also match. From the spatial frequency spectrum, it becomes impossible to determine whether the incident angle of the light ray is in the case of FIG. 7B or in the case of FIG. 7C. An example of a method for avoiding this is shown in FIG.

FIG. 8 is a diagram showing an example of a grating substrate in which the axes of the front surface grid and the back surface grid are offset from each other. As shown in FIG. 8, the two grating patterns 104 and 105 are preliminarily displaced relative to the optical axis so that the shadows of the two grating patterns are shifted and overlap even with respect to a light beam that is vertically incident on the modulator 102. It is necessary to leave. When the relative shift of the shadows of the two gratings with respect to the vertically incident plane wave on the axis is δ ₀ , the shift δ caused by the plane wave with the incident angle θ is

のように表せる。このとき、入射角θの光線のモアレ縞の空間周波数スペクトルのピークは周波数のプラス側では、

Can be expressed as At this time, the peak of the spatial frequency spectrum of the moire fringes of the ray with the incident angle θ is on the plus side of the frequency,

の位置となる。画像センサの大きさをＳ、画像センサのｘ方向およびｙ方向の画素数を共にＮとすると、２次元フーリエ変換による離散画像の空間周波数スペクトルは、−Ｎ／（２Ｓ）から＋Ｎ／（２Ｓ）の範囲で得られる。このことから、プラス側の入射角とマイナス側の入射角を均等に受光することを考えれば、垂直入射平面波（θ＝０）によるモアレ縞のスペクトルピーク位置は、原点（ＤＣ：直流成分）位置と、例えば＋側端の周波数位置との中央位置、すなわち、

Position. When the size of the image sensor is S and the number of pixels in the x direction and the y direction of the image sensor are both N, the spatial frequency spectrum of the discrete image by the two-dimensional Fourier transform is -N/(2S) to +N/(2S). It is obtained in the range of. From this fact, considering that the positive side incident angle and the negative side incident angle are received evenly, the spectral peak position of the moire fringes due to the vertically incident plane wave (θ=0) is the origin (DC: DC component) position. And, for example, the center position between the + side end frequency position, that is,

の空間周波数位置とするのが妥当である。したがって、２つの格子の相対的な中心位置ずれは、

The spatial frequency position of is appropriate. Therefore, the relative center misalignment of the two gratings is

とするのが妥当である。

Is appropriate.

FIG. 9 is a diagram for explaining the generation of moire fringes and the frequency spectrum when the gratings on both sides of the grating substrate are arranged in a shifted manner. Similar to FIG. 7, schematic views of the arrangement of the light beam and the modulator 102, moire fringes, and spatial frequency spectrum are shown from left to right. 9A shows a case where the light ray is vertically incident, FIG. 9B shows a case where the light ray enters from the left side at an angle θ, and FIG. 9C shows a case where the light ray enters from the right side at an angle θ. The respective cases are shown.

The first lattice pattern 104 and the second lattice pattern 105 are arranged in advance with a shift of δ ₀ . Therefore, Moire fringes also occur in FIG. 9A, and a peak appears in the spatial frequency spectrum. As described above, the shift amount δ ₀ is set so that the peak position appears in the center of the spectrum range on one side from the origin. At this time, in FIG. 9B, the deviation δ is further increased, and in FIG. 9C, the deviation δ is decreased, which is different from FIG. The difference from FIG. 9C can be discriminated from the peak position of the spectrum. That is, the spectrum image of this peak is a bright spot showing a light flux at infinity, and is nothing but an image taken by the image pickup apparatus 101 in FIG.

If the maximum angle of incidence of parallel light that can be received is θ _max ,

より、撮像装置１０１にて受光できる最大画角は、

Therefore, the maximum angle of view that can be received by the imaging device 101 is

で与えられる。

Given in.

From the analogy with the image formation using a general lens, considering that parallel light with an angle of view θ _max is focused and received at the end of the image sensor, the effective focal length of the imaging device 101 without a lens is ,

に相当すると考えることができる。

Can be considered equivalent to.

Here, it can be understood from the expressions (13) and (14) that the angle of view can be changed by the thickness t of the modulator 102 and the size S of the image sensor 103. Therefore, for example, if the modulator 102 has the configuration of FIG. 3 and has a function (for example, a mechanical structure) capable of changing the length of the support member 102b, it is possible to change the angle of view at the time of shooting and perform shooting. Become.

Although the fast Fourier transform has been described as an example of the method for calculating the spatial frequency spectrum from the moire fringes, the method is not limited to this, and it can be realized by using a discrete cosine transform (DCT) or the like. Therefore, it is possible to further reduce the calculation amount.

Further, the transmittance distributions of the grid patterns 104 and 105 have been described assuming that they have sinusoidal characteristics as shown in the equation (2), but such a component is used as the fundamental frequency component of the grid pattern. It only needs to be included. For example, it is possible to binarize the transmittance of the lattice pattern as shown in FIG. 10 (a diagram showing an example of the lattice pattern), and further, as shown in FIG. 11 (a diagram showing another example of the lattice pattern). It is also possible to increase the transmittance by changing the duty of the grating region having a high ratio and the duty of the region having a low ratio to widen the width of the region having a high transmittance. As a result, the effect of suppressing the diffraction from the grating pattern can be obtained, and the deterioration of the captured image can be reduced.

The grating patterns 104 and 105 may be realized by phase modulation instead of transmittance modulation. For example, as shown in FIG. 12 (a diagram showing still another example of the lattice pattern), by forming the lattice substrate 102a with a cylindrical lens 1201, an intensity modulation pattern as shown in the diagram is generated on the image sensor 103. Therefore, it is possible to perform imaging in the same manner as the discussion so far. As a result, it is possible to reduce the light amount loss due to the shield portion of the grating pattern 104, improve the light utilization efficiency, and obtain the effect of suppressing the diffraction from the grating pattern. Although it is realized by a lens in FIG. 12, it can be realized by a phase modulation element having an equivalent effect.

In the above description, the incident light beams are incident at one incident angle at the same time. However, in order for the image pickup apparatus 101 to actually act as a camera, when the incident light beams are incident at a plurality of incident angles at the same time. Must be assumed. The light of such a plurality of incident angles already overlaps the images of the plurality of front side gratings at the time of being incident on the back side grating pattern. If these images produce moire fringes with each other, it is feared that the moiré fringes become noise that hinders detection of the moiré fringes with the second lattice pattern 105, which is a signal component. However, in reality, the overlapping of the images of the first grating pattern 104 does not generate the peak of the moire image, and the peak is generated only by the overlapping with the second grating pattern 105 on the back surface side. The reason will be described below.

First, the major difference is that the overlap of the shadows of the first grating pattern 104 on the surface side due to light rays having a plurality of incident angles is a sum, not a product. Regarding the overlap between the shadow of the first grating pattern 104 and the second grating pattern 105 due to the light having one incident angle, the shadow of the first grating pattern 104 has a light intensity distribution of the second grating pattern 105. The light intensity distribution after passing through the second grating pattern 105 on the back surface side can be obtained by multiplying by the transmittance of.

On the other hand, the overlapping of the shadows due to the lights incident on the first grating pattern 104 on the surface side at different angles is not the product but the sum, because the overlapping of the lights is the overlap. In the case of sum,

のように、もとのフレネルゾーンプレートの格子の分布に、モアレ縞の分布を乗算した分布となる。したがって、その周波数スペクトルは、それぞれの周波数スペクトルの重なり積分で表される。そのため、たとえモアレのスペクトルが単独で鋭いピークをもったとしても、実際上、その位置にフレネルゾーンプレートの周波数スペクトルのゴーストが生じるだけである。つまり、スペクトルに鋭いピークは生じない。したがって、複数の入射角の光を入れても検出されるモアレ像のスペクトルは、常に表面側の第１の格子パターン１０４と裏面側の第２の格子パターン１０５との積のモアレだけであり、第２の格子パターン１０５が単一である以上、検出されるスペクトルのピークは１つの入射角に対して１つだけとなるのである。

As described above, the distribution of the original lattice of the Fresnel zone plate is multiplied by the distribution of Moire fringes. Therefore, the frequency spectrum is represented by the overlap integral of each frequency spectrum. Therefore, even if the moiré spectrum has a sharp peak alone, in reality, only a ghost of the frequency spectrum of the Fresnel zone plate occurs at that position. That is, no sharp peak appears in the spectrum. Therefore, the spectrum of the moire image detected even when light with a plurality of incident angles is input is always only the moire of the product of the first grating pattern 104 on the front surface side and the second grating pattern 105 on the back surface side. Since the second grating pattern 105 is single, the number of detected spectrum peaks is only one for one incident angle.

Here, the correspondence between the parallel light which has been described so far to be detected and the light from the actual object will be schematically described with reference to FIG.

FIG. 13 is a diagram illustrating an angle formed by light from each point forming an object with respect to the image sensor. Strictly speaking, the light from each point constituting the subject 401 is a spherical wave from a point light source, and the modulator 102 and the image sensor 103 of the image pickup apparatus 101 of FIG. 1 (hereinafter, referred to as a grating sensor integrated substrate 1301 in FIG. 13). Incident on. At this time, when the grating sensor integrated substrate 1301 is sufficiently small or far from the subject 401, the incident angles of the light illuminating the grating sensor integrated substrate 1301 from each point can be regarded as the same.

Since the spatial frequency displacement Δu of the moire with respect to the minute angular displacement Δθ obtained from the equation (9) is 1/S or less which is the minimum resolution of the spatial frequency of the image sensor, the condition that Δθ can be regarded as parallel light is as follows.

のように表せる。この条件下であれば、無限遠の物体を本実施形態の撮像装置が撮像可能である。

Can be expressed as Under this condition, the object at infinity can be imaged by the imaging device of this embodiment.

<Principle of photographing a finite object>
Here, the imaging of the object at infinity described so far will be described again. FIG. 14 is a diagram illustrating that the front side lattice pattern is projected when the object is at an infinite distance. FIG. 14 shows how the first grid pattern 104 on the front surface side is projected onto the back surface. A spherical wave from a point 1401 forming an object at infinity becomes a plane wave while propagating over a sufficiently long distance and irradiates the first grating pattern 104 on the surface side, and its projected image 1402 is projected on the lower surface. It In this case, the projected image has substantially the same shape as the first grid pattern 104. As a result, the projected image 1402 is multiplied by the transmittance distribution of the back-side grating pattern (corresponding to the second grating pattern 105 in FIG. 2) to generate FIG. 15 (when the object is at an infinite distance). It is possible to obtain linear moire fringes at even intervals as shown in the figure showing an example of moire fringes.

On the other hand, imaging of an object with a finite distance will be described. FIG. 16 is a diagram illustrating that the front side lattice pattern is enlarged when the object is at a finite distance. FIG. 16 shows how the first grid pattern 104 on the front surface side is projected onto the back surface. The spherical wave from the point 1601 forming the object irradiates the first grating pattern 104 on the surface side, and the projected image 1602 is projected on the lower surface. In this case, the projected image is enlarged almost uniformly. Note that this enlargement factor α is calculated by using the distance d from the first grid pattern 104 to the point 1601.

のように算出できる。

Can be calculated as

Therefore, if the transmittance distribution of the designed back side grating pattern is multiplied as it is for parallel light, it is shown in FIG. 17 (a diagram showing an example of moire fringes generated when an object is at a finite distance). As described above, the linear moire fringes at equal intervals are not generated. However, if the second lattice pattern 105 is enlarged in accordance with the shadow of the first lattice pattern 104 on the surface side that is uniformly enlarged, the enlarged projected image 1602 is displayed again in FIG. Is a finite distance, a linear moire fringe at even intervals can be generated, as shown in a figure showing an example of the moire fringe in which the back side lattice pattern is corrected. For this purpose, the correction can be performed by setting the coefficient β of the second lattice pattern 105 to β/α ² .

Thereby, the light from the point 1601 at a distance which is not necessarily infinity can be selectively developed. As a result, it is possible to perform shooting by focusing on an arbitrary position.

<Simplified configuration>
Next, a method of simplifying the configuration of the modulator 102 described above will be described. In the modulator 102, a first grating pattern 104 and a second grating pattern 105 having the same shape are formed on the front surface and the back surface of the grating substrate 102a, offset from each other. Then, the angle of the incident parallel light is detected from the spatial frequency spectrum of the moire fringes by image processing to develop the image. The second grid pattern 105 on the back surface side is an optical element that modulates the intensity of light that comes into close contact with the image sensor 103 and has the same grid pattern regardless of the incident light. Then, you may change the structure of the imaging device 101 as follows. The difference from FIG. 1 will be mainly described.

FIG. 19 is a diagram showing a configuration example of the imaging device 101 that realizes the back side lattice pattern by image processing. The image pickup apparatus 101 includes a modulator 1901 in which the second lattice pattern 105 is omitted, and an image processing unit 1902 including an intensity modulator 1903 that performs a process corresponding to the second lattice pattern 105.

FIG. 20 is a diagram showing a configuration example of the modulator 1901. The modulator 1901 is composed of the grating substrate 102a and the first grating pattern 104 formed on the surface of the grating substrate 102a. As compared with FIG. 2, the number of lattice patterns formed on the lattice substrate 102a can be reduced by one. As a result, the manufacturing cost of the modulator can be reduced and the light utilization efficiency can be improved.

FIG. 21 is a flowchart showing an example of image processing of the image processing unit 1902. The difference from the flowchart of FIG. 5 is the processing of step 2101 before step 501. The processing of steps 501 to 506 is the same as the corresponding processing of FIG. 5, so description thereof will be omitted here.

In the process of step 2101, the intensity modulation unit 1903 described above generates a moire fringe image corresponding to the image transmitted from the image sensor 103 through the lattice pattern 105 on the back surface side. Specifically, since the calculation corresponding to Expression (5) may be performed, the intensity modulation unit 1903 generates the back surface side grid pattern 105 and multiplies the image of the image sensor 103 by the image. Further, if the lattice pattern 105 on the back surface side is a binarized pattern as shown in FIG. 10 or FIG. 11, the intensity modulation unit 1903 simply sets the value of the image sensor 103 in the area corresponding to black to 0. Good. This makes it possible to suppress the calculation load of the multiplication operation and the scale of the multiplication circuit.

Note that in this case, the pitch of the pixels 103a (FIG. 20) included in the image sensor 103 is fine enough to sufficiently reproduce the pitch of the first grid pattern 104, or the pitch of the first grid pattern 104 is the pixel 103a. It needs to be coarse enough to be reproduced at the pitch. When the grid pattern is formed on both sides of the grid substrate 102a, the pitch of the grid pattern does not necessarily have to be resolved by the pixels 103a of the image sensor 103, and only the moire image needs to be resolved. However, when a lattice pattern is reproduced by image processing, the lattice pattern and the image sensor 103 must have the same resolution.

In addition, the processing corresponding to the second lattice pattern 105 has been realized by the intensity modulator 1903. However, the second grating pattern 105 is not limited to this, and since the second grating pattern 105 is an optical element that closely modulates the intensity of light incident on the sensor, the sensitivity of the sensor can be effectively increased. It can also be realized by taking into account the setting.

<Noise cancellation>
In the above description, the discussion was focused on Equation (6) in which only the component having a sharp peak is extracted from Equation (5), but in reality, terms other than the fourth term in Equation (5) are noise. Becomes Therefore, noise cancellation based on fringe scanning is effective.

First, in the interference fringe intensity distribution of Expression (2), when the initial phase of the first grating pattern 104 is Φ _F and the initial phase of the second grating pattern 105 is Φ _B , Expression (5) yields

のように表せる。ここで、三角関数の直交性を利用し、

Can be expressed as Here, using the orthogonality of trigonometric functions,

式（１９）のように式（１８）をΦ_Ｆ、Φ_Ｂに関して積分すると、ノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。前述の議論から、これをフーリエ変換すれば、空間周波数分布にノイズのない鋭いピークを生じることになる。

When equation (18) is integrated with respect to Φ _F and Φ _B as in equation (19), the noise term is canceled and a term that is a constant multiple of the single frequency remains. From the above discussion, Fourier transform of this will yield a sharp peak with no noise in the spatial frequency distribution.

Here, the formula (19) is shown in the form of integration, but actually, the same effect can be obtained by calculating the total sum of the combinations of Φ _F and Φ _B. Φ _F and Φ _B may be set so as to equally divide an angle between 0 and 2π, and {0, π/2, π, 3π/2} are equally divided into four, {0, π/3. , 2π/3} may be equally divided.

Furthermore, equation (19) can be simplified. In Expression (19), Φ _F and Φ _B are calculated so that they can be changed independently, but Φ _F =Φ _B, that is, even if the same phase is applied to the initial phases of the grating patterns 104 and 105, the noise term can be canceled. .. If Φ _F =Φ _B =Φ in equation (19),

となり、ノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。また、Φは０〜２πの間の角度を等分するように設定すればよく、｛０、π／２、π、３π／２｝のように４等分すればよい。

Therefore, the noise term is canceled and the term that is a constant multiple of the single frequency remains. Further, Φ may be set so as to equally divide an angle between 0 and 2π, and may be equally divided into four such as {0, π/2, π, 3π/2}.

Further, the noise term can be canceled by using the orthogonal phases of {0, π/2} without even division, and the simplification can be further achieved. First, if the image processing unit 1902 implements the second lattice pattern 105 as in the configuration of FIG. 19, a negative value can be handled in the lattice pattern 105, and therefore the equation (18) is

となる（Φ_Ｆ＝Φ_Ｂ＝Φ）。格子パターン１０５は既知であるため、この式（２１）から格子パターン１０５を減算し、Φ＝｛０、π／２｝の場合について加算すれば、

(Φ _F =Φ _B =Φ). Since the lattice pattern 105 is known, if the lattice pattern 105 is subtracted from this equation (21) and added for the case of Φ={0, π/2},

のようにノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。

Thus, the noise term is canceled and a term that is a constant multiple of the single frequency remains.

Further, as described above, the first grid pattern 104 and the second grid pattern 105 separate the two developed images generated in the spatial frequency space by shifting in advance by δ ₀ . However, this method has a problem that the number of pixels of the developed image is halved. Therefore, a method for avoiding overlapping of developed images without shifting by δ ₀ will be described. In the fringe scan of equation (19), instead of cos,

のようにｅｘｐを用い複素平面上で演算する。これによりノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。式（２３）中のｅｘｐ（２ｉβδｘ）をフーリエ変換すれば、

The calculation is performed on the complex plane using exp as follows. This cancels the noise term, leaving a term that is a constant multiple of the single frequency. Fourier transform of exp(2iβδx) in equation (23) gives

となり、式（７）のように２つのピークを生じず、単一の現像画像を得られることが判る。このように、格子パターン１０４，１０５をずらす必要もなくなり、画素数を有効に使用可能となる。

Therefore, it can be seen that a single developed image can be obtained without generating two peaks as in Expression (7). In this way, it is not necessary to shift the grid patterns 104 and 105, and the number of pixels can be effectively used.

A configuration for performing the above-described fringe scan-based noise canceling method will be described with reference to FIGS. In the fringe scan, it is necessary to use at least a plurality of patterns having different initial phases as the grid pattern 104. To realize this, there are a method of switching patterns by time division (FIGS. 22 to 25) and a method of switching patterns by space division (FIGS. 26 to 27).

FIG. 22 is a diagram showing a configuration example of the image pickup apparatus 101 that realizes the time division fringe scan. This imaging device 101 has a modulator 2201 instead of the modulator 102 of FIG. The image pickup apparatus 101 also has a modulator control unit 2202 and an image processing unit 2203.

FIG. 23 is a diagram showing an example of a lattice pattern in the time division fringe scan. The modulator 2201 is composed of, for example, a liquid crystal display element capable of electrically switching and displaying a plurality of initial phases shown in FIG. 23 (that is, capable of changing a lattice pattern). In each pattern of FIG. 23, the initial phase Φ _F or the phase difference Φ is {0, π/2, π, 3π/2}, respectively. An example of the electrode arrangement in the liquid crystal display element of the modulator 2201 that realizes this is shown in FIG.

FIG. 24 is a diagram showing a configuration example of the modulator 2201 that realizes time division fringe scanning. The modulator 2201 is provided with concentric electrodes so as to divide one period of the lattice pattern into four, and every four electrodes are connected from the inside, and four electrodes are drawn out from the outer peripheral portion as drive terminals. ing. The modulator control unit 2202 temporally switches the voltage states applied to these four electrodes between two states of "0" and "1", so that the initial phase Φ _F or Φ of the lattice pattern is changed as shown in FIG. It is possible to switch to {0, π/2, π, 3π/2}. It is to be noted that in FIG. 24, the shaded electrode corresponds to the electrode to which “1” is applied shields the light, and the shaded electrode corresponds to the electrode to which “0” is applied transmits the light.

FIG. 25 is a flowchart showing an example of the image processing of the image processing unit 2203 that realizes the time division fringe scanning. The difference from the flowchart of FIG. 21 is the processing of steps 2501 to 2504 before step 501. The processing of steps 501 to 506 is the same as the corresponding processing of FIG. 21, so description thereof will be omitted here.

First, the image processing unit 2203 resets the addition result at the beginning of the fringe scan calculation (2501). Next, in the case of corresponding to Expression (20), the image processing unit 2203 sets the same initial phase as the grid pattern 104 used for imaging (2502) and generates the grid pattern 105 having the initial phase. , The output image of the image sensor 103 is multiplied (2101). The image processing unit 2203 adds the multiplication result for each initial phase pattern (2503). The image processing unit 2203 repeats the above steps 2502 to 2503 for all the patterns of the initial phase (NO in 2504). If the processing has been completed for all the phases (YES in 2504), the image processing unit 2203 advances the processing to step 501. In addition, although the above-described flowchart has been described by using the equation (20) as an example, it can be similarly applied to the equations (19), (22), and (23).

FIG. 26 is a diagram showing a configuration example of the image pickup apparatus 101 that realizes the space division fringe scan. This imaging device 101 has a modulator 2601 instead of the modulator 102 of FIG. The image pickup apparatus 101 also includes an image division unit 2602 and an image processing unit 2203.

FIG. 27 is a diagram showing an example of a lattice pattern in the space division fringe scan. The modulator 2601 has a plurality of initial phase patterns arranged two-dimensionally such that the initial phase ΦF or Φ of FIG. 27 is {0, π/2, π, 3π/2}, respectively. The image division unit 2602 divides the output image of the image sensor 103 into area images corresponding to the pattern arrangement of the modulator 2601, and sequentially transmits the area images to the image processing unit 2203. In the example of FIG. 26, the image sensor output is divided into 2×2 areas. Since the fringe scan based on the equation (20) requires four phases, the modulator 2601 has a 2×2 pattern arrangement. Since the fringe scan based on the equation (22) can be realized in two phases, the modulator 2601 can be realized in a 1×2 pattern arrangement, and the image sensor output is also divided into 1×2 regions accordingly. The subsequent processing of the image processing unit 2203 is the same as the processing of FIG. 22 which is a time-division fringe scan, and thus the description thereof is omitted.

By using this space division fringe scan, it is not necessary to electrically switch the grid pattern unlike the time division fringe scan modulator 2201, and the modulator can be manufactured at low cost. However, when the space division fringe scan is used, the image is divided and the resolution is effectively reduced. Therefore, when it is necessary to increase the resolution, the time division fringe scan is suitable.

<Distortion correction principle>
Further, let us consider a case where the first grating pattern 104 has a thickness and the refractive index n thereof is not 1. FIG. 28 is a diagram for explaining how a light beam from a subject is projected on an image sensor with a shift depending on the thickness and refractive index of the optical section and the subject position. In FIG. 28, the lattice pattern 104 is sandwiched between the optical units 2801 and 282 having a thickness l ₁ and l ₂ . Further, the principal axis of light from a certain point that constitutes an object at a short distance reaches the image sensor 103 through the center of the lattice pattern.

If 2803 points constituting the object is directly above the center of the grating pattern 104, the beam thickness _{l 1} of the optical portion 2801, the thickness _{l 2} of the optical portion 2802, and reaches the same position of the image sensor regardless of the refractive index. On the other hand, the beam position of the point from 2804 points constituting an object in a position shifted from the center of the grating pattern 104, the thickness _{l 1} of the optical portion 2801, the thickness _{l 2} of the optical portion 2802, an image by the refractive index The position to reach the sensor is different. Assuming that the positions 2804 of the points forming the object are x and y, the deviation amount δr of the arrival position is

となる。このずれが含まれた状態で被写体の撮影を行うと表示画像は歪むことになる。仮に画像センサと格子パターンの中心が一致していた場合、δｒの値はｘ，ｙが大きいほど、つまり中心から遠い箇所ほど大きく歪んだ画像として表示される。

Becomes If the subject is photographed in a state where this shift is included, the display image will be distorted. If the center of the image sensor and the center of the grid pattern are coincident with each other, the larger the x and y values of δr, that is, the farther from the center, the more distorted the image is displayed.

FIG. 29 is a diagram showing an example of occurrence of distortion of a subject to be photographed. When the lattice pattern as shown in FIG. 29 is used as a subject, if the refractive index is 1 or the thicknesses of the optical units 2801 and 2802 are both 0, the image is taken without distortion and as shown in the upper right of FIG. The same image is captured and displayed. On the other hand, the refractive index is greater than 1, and when the thickness l ₁ of the optical portion _2801, the thickness l ₂ of the optical portion 2802 is larger than 0, a distorted image as shown in the lower right Figure 29 is displayed.

Therefore, a method of correcting the distortion by using δr calculated from Expression (25) will be considered. FIG. 30 is a diagram illustrating a configuration example of an image pickup apparatus that realizes shooting of a subject in a state where image distortion is corrected by image distortion correction. Specifically, the image pickup apparatus 101 includes an image distortion correction unit 3201. Although the basic photographing principle is the same as that shown in FIG. 1, the image distortion correction unit 3201 performs correction to remove the image distortion before displaying the image. In this correction, the image distortion correction unit 3201 uses δr calculated from Expression (25). The image distortion correction unit 3201 uses the image distortion correction setting value input unit 3202 to input the parameters necessary for the calculation. Here, the set values that the image distortion correction set value input unit 3202 accepts are the thickness l ₁ and the refractive index of the optical unit 2801, the thickness l ₂ and the refractive index of the optical unit 2802, and the distance d from the surface of the optical unit 2801 to the subject. Is. By this correction process, the distortion of the image displayed on the image display unit 107 is removed, and the subject can be displayed correctly.

Here, the value input to the image distortion correction set value input unit 3202 may be a design value of the image pickup apparatus 101, or a necessary set value may be measured for each apparatus. For example the distance d becomes possible to back calculate the thickness l ₁ and thickness l ₂ of the optical unit 2802 of the optical portion 2801 if capturing a known point source as the subject. This makes it possible to eliminate individual differences when manufacturing the device.

Further, in the present embodiment, as an example, the respective optical units 2801 and 2802 are arranged in contact with the front and rear of the lattice pattern 104, but the present invention is not limited to this, and the respective optical units 2801 and 2802 are arranged in the lattice pattern. They may be separated from each other without contacting 104, and the number thereof may be any number. As a result, even if the image sensor 103 is provided with a cover glass, the influence thereof can be removed, and more accurate distortion correction can be performed.

[Second embodiment]
<Problems with close-up photography>
The above embodiment has been described on the assumption that the subject is distant from the imaging device by a certain distance or more. In the second embodiment, problems in the case of taking a close-up image of a subject will be described. First, the angle of view is redefined. In the description so far, the peak position u (equation (9)) of the spatial frequency spectrum of the moire fringes of the light ray having the incident angle θ is larger than the upper limit frequency N/(2S) determined by the pixel pitch of the image sensor 103. In some cases, the angle of view θ _max was defined (equation (13)). On the other hand, when β is small or the pixel pitch (S/N) of the image sensor 103 is small and the peak position u is smaller than the upper limit frequency, the definition of the angle of view is different.
FIG. 31 is a diagram for explaining the angle of view θ _max2 . Θ _max2 , which is the inclination of a straight line that passes through the reference coordinates of the concentric circular grid pattern that constitutes the first grid pattern 104 and the end of the image sensor 103, is the maximum field angle that can be developed. Therefore, the angle of view θ _max2 is calculated by using the size S of the image sensor 103 and the thickness t of the modulator in imaging at a distance d from the first lattice pattern.

のように表せる。また、視野の直径Ａは、θ_ｍａｘ２を用いて、

Can be expressed as Further, the diameter A of the visual field is θ _max2 ,

となる。

Becomes

Here, at the time of close-up photography, there is a problem that the field of view is narrowed due to the scattering angle from the object to be imaged and the CRA (Chief Ray Angle) characteristic of the image sensor.

First, the limitation of the visual field due to the scattering angle from the object to be imaged will be described. FIG. 32 is a diagram for explaining a range and a field of view of a lattice pattern irradiated by light from a certain point that constitutes the object when the object is at a short distance. FIG. 32 shows the spread angle (scattering angle) θ _s of light from a certain point on the object. When the scattered light from the points 3401 to 3403 forming the object irradiates the first grating pattern 104, the diameter B of the irradiation region in the first grating pattern 104 is calculated by using the scattering angle θ _s .

となる。このように、接写時には第１の格子パターン１０４のうち照射領域しか使用することができない。なお、現実には散乱光強度は散乱角が大きくなるに従い徐々に減衰するものであるが、図３２では簡単化のため、照射領域のみ散乱光が到達しているとしている。

Becomes As described above, only the irradiation area of the first grid pattern 104 can be used during close-up photography. In reality, the scattered light intensity gradually attenuates as the scattering angle increases, but in FIG. 32, the scattered light reaches only the irradiation region for simplification.

As described above, imaging can be performed when the incident light passes through the reference coordinates of the concentric lattice pattern forming the first lattice pattern 104. Therefore, the points 3401 and 3402 through which the scattered light passes the reference coordinates can be captured, but the points 3403 and the like that do not pass the reference coordinates cannot be captured. When θ _s ≦θ _max2 as shown in FIG. 32, the angle of view is determined by the scattering angle θ _s , and the diameter A _s of the visual field is

となる。

Becomes

Next, the limitation of the visual field due to the CRA characteristic of the image sensor will be described. Depending on the layout of the wiring or the like on the front surface or the back surface of the light receiving element or the design of the microlens array normally arranged on the front surface of the light receiving element, the amount of light received by the image sensor has an angle dependency called CRA characteristic. Therefore, if it is assumed that the incident light can be received up to the incident angle θ _i based on the CRA characteristics, the angle of view is determined by θ _i and the field of view A _i is determined in imaging using an image sensor that _satisfies θ _i ≦θ _max2. Is

となる。ただし、θ_ｓ≦θ_ｉの場合の視野は式（２９）となる。

Becomes However, the field of view in the case of θ _s ≦θ _i is given by equation (29).

Note that the incident angle θ _i is preferably determined by an angle that is the half-width at half maximum of the CRA characteristic, but is not limited to this, and if the angle is designed to be smaller than the half-width at half maximum, peripheral light amount attenuation is less likely to occur, and the half-width at half maximum is less. The larger the angle, the larger the field of view.

<Field of view expansion method for close-up photography>
Therefore, a method of expanding the field of view during close-up will be described.

FIG. 33 is a diagram showing a configuration example of an imaging device that realizes a wide field of view by combining a compound eye lattice pattern and a plurality of image sensors. Unlike the configuration shown in FIG. 19, the image pickup apparatus 101 includes a modulator 301 in which a compound eye lattice pattern 3503 is formed, a compound eye image sensor 3004, an image combining unit 3506, and an image processing unit 106. The compound-eye lattice pattern 3503 is configured by arranging a plurality of basic patterns (basic lattice patterns) 3502, each of which is a concentric lattice pattern in which the pitch of stripes becomes narrower from the center toward the edge. The compound eye image sensor 3004 is configured by arranging a plurality of image sensors 3505.

FIG. 34 is a diagram showing an example of a compound eye lattice pattern. In this example, the compound eye grid pattern 3503 is realized by arranging the basic patterns 3502 in 3×3 without overlapping each other. In this case, the image sensors 3505 are arranged in 3×3 so as to correspond to the basic patterns 3502.

FIG. 35 is a diagram for explaining the compound eye grid pattern shown in FIG. 33 and the field of view by a plurality of image sensors. Of the plurality of basic patterns 3502 and the plurality of image sensors 3505, the n-th field of view by the combination of the n-th basic pattern 3502 and the n-th image sensor 3505 is A _n , the angle of view is θ _n , and the n-th and n+1-th fields. It is assumed that the center-to-center distance of the basic pattern 3502 is q _n , and the number of the basic patterns 3502 and the image sensors 3505 in the one-dimensional direction is N. The visual fields A _n of the respective basic patterns are different from each other. At this time, the synthetic visual field A _N is

となる。つまり、基本パターン３５０２と画像センサ３５０５の一対一の組み合わせ（基本ユニットともいう）を複数用いることで、それぞれの組み合わせで撮像できる視野を合成し拡大することができる。

Becomes That is, by using a plurality of one-to-one combinations (also referred to as basic units) of the basic pattern 3502 and the image sensor 3505, it is possible to combine and expand the fields of view that can be imaged by each combination.

However, when the distance d from the compound-eye lattice pattern 3503 to the imaging target is short, the visual field of each basic pattern 3502 becomes small, and a gap is formed in the synthetic visual field A _N. To obtain a field of view continuous gap does not occur, _{A n} is as long or _{q n,} may be set to d and _{q n} which satisfies _{A n = 2d · tanθ n ≧} q n.

Subsequently, image processing of image pickup using the above-described compound eye image sensor 3004 and compound eye lattice pattern 3503 will be described with reference to FIGS.

FIG. 36 is a flowchart showing an example of the image distortion correction processing. The difference from the flowchart of FIG. 21 is the processing of step 3801 and step 3802 added between step 504 and step 505. The image processing unit 106 performs the processes of steps 2101 to 504 on each of the sensor images output by the plurality of image sensors 3505.

The image distortion correction unit 3201 performs image distortion correction on the output image of each image sensor 3505 according to the input value from the image distortion correction setting value input unit 3202 in step 3801. Subsequently, in step 3802, the image combining unit 3506 rearranges the output images in step 3801 according to the arrangement of the image sensors 3505 and combines these. Here, since the image distortion correction is performed in step 3801, it is possible to correctly combine the images in step 3802.

The setting values used for image distortion correction in step 3801 are the thickness l ₁ and the refractive index of the optical unit 2801, the thickness l ₂ and the refractive index of the optical unit 2802, and the optical unit 2801 which are input via the image distortion correction setting value input unit 3202. The distance d from the surface to the subject.

Here, the value input to the image distortion correction set value input unit 3202 may be a design value of the image pickup apparatus 101, or a necessary set value may be measured for each individual apparatus. For example, if a point light source with a known distance d is photographed as an object directly above the basic lattice pattern 3502, the thicknesses l _{1 and} l ₂ of the optical units 2801 and 2802 can be obtained by back calculation. This makes it possible to eliminate individual differences when manufacturing the device.

FIG. 37 is a diagram illustrating an example of a synthesizing method. In addition, FIG. 38 is a diagram illustrating a composite image. 37 and 38 are diagrams illustrating an example of the combining method in step 3802. In FIG. 37, A _N is the combined visual field A _N in FIG. 35, and A _n and A _n+1 are the visual field A _n and visual field A _n+1 by the nth and n+1th basic patterns 3502 in FIG. Also, the output images of step 3801 in FIG. 36 are represented as output images 3901 and 3902.

In FIG. 38, a composite image of the output image 3901 and the output image 3902 is shown. In Step 3802, one composite image is obtained by shifting and adding at least one of the output image 3901 and the output image 3902 by the number of pixels corresponding to the center-to-center distance q _n of the nth and n+1th basic patterns 3502. To be

Next, the luminance correction after the combination will be described with reference to FIG. An overlapping area 4001 indicates an area where the visual fields A _n and A _n+1 of the output image 3901 and the output image 3902 overlap. The brightness of the overlapping area 4001 after the combination is the sum of the output image 3901 and the output image 3902. Therefore, in order to eliminate the difference in brightness between the overlapping area 4001 and other areas where the fields of view do not overlap and to obtain a smooth brightness distribution, it is necessary to correct the brightness of the overlapping area 4001. When the luminance of the area corresponding to the overlapping area 4001 of the output image 3901 and the luminance of the area corresponding to the overlapping area 4001 of the output image 3902 match, the luminance of the overlapping area 4001 after combining is halved. By doing so, it can be easily corrected.

However, since the actual luminance has a distribution that is attenuated from the center to the edge due to the influence of CRA or the like, the luminance of the area corresponding to the overlapping area 4001 of the output image 3901 and the area corresponding to the overlapping area 4001 of the output image 3902. In many cases, the brightness does not match, and the brightness difference cannot be corrected even if the brightness after synthesis is halved. Therefore, of either the output image 3901 or the output image 3902, either one of the regions corresponding to the overlapping region 4001 in which the visual field A _n and the visual field A _n+1 overlap is adopted, or both images are combined. Images are cut out from the edge by the number of pixels corresponding to q _n /2 and the overlapping portions are eliminated, and the images are added together. By any of these methods, the brightness difference can be reduced without performing brightness correction.

It should be noted that in the present embodiment, a plurality of image sensors 3505 are used, but one image sensor 3505 may be used when the image sensor 3505 is sufficiently larger than the compound eye lattice pattern 3503. In that case, if the image output from the image sensor 3505 is divided according to the number of the basic patterns 3502, the same processing as when using a plurality of image sensors 3505 is possible.

Further, although the basic pattern 3502 and the image sensor 3505 have been described focusing on the one-dimensional array in FIG. 35 and the like, this configuration is also applicable to a two-dimensional array by expanding in the orthogonal direction as well. is there.

Further, the number of basic patterns 3502 is not limited to that shown in the figure. Since the field of view is determined by the number of basic patterns 3502, it may be appropriately changed according to the size of the imaging target.

Further, if the distance from the basic pattern 3502 to the imaging target is increased (increased), the visual field becomes wider. By doing so, it is possible to reduce the number of basic patterns 3502 required to obtain the same field of view.

FIG. 39 is a diagram illustrating an arrangement example of image sensors. FIG. 39 shows an arrangement in which the directions of the image sensors 3505 (the directions of the wirings) are aligned (in the same direction). In a general image sensor, wiring for transmitting a signal extends from the side of the light receiving surface. When arranging the plurality of image sensors 3505, it is necessary to space the image sensors 3505 so that interference between wirings and overlapping of wirings on the light receiving surface 4101 do not occur, and the light receiving surfaces are brought close to each other within a certain distance. You may not be able to.

FIG. 40 is a diagram illustrating another arrangement example of the image sensor. FIG. 40 shows an arrangement in which the orientation of each image sensor 3505 (wiring orientation) faces the outside of the image sensor 3505. Although the position of the wiring differs depending on the type of the image sensor, by arranging at least some of the image sensors 3505 so as to be different from the directions of other image sensors 3505, the distance between the light receiving surfaces due to the wiring is restricted (a fixed distance). It is possible to avoid the restriction that cannot be approached within.

According to the above method and configuration, by using a plurality of combinations of the image sensor 3505 and the basic pattern 3502, it is possible to combine and expand the fields of view that can be imaged by each. According to the present embodiment, it is possible to provide an image pickup apparatus having a simple configuration, a thin shape, a distortion corrected, and a field of view enlarged at the time of close-up photography.

[Third embodiment]
<Gaze direction design>
The gaze direction in imaging using the compound-eye lattice pattern 3503 will be described using the third embodiment.

FIG. 41 is a diagram illustrating an example of a gaze direction. In particular, FIG. 41 is a diagram for explaining an example of the gaze direction of imaging by the compound-eye lattice pattern 3503. A _n, when the angle of view by the equation (26) is determined, which indicates the field of view can be obtained by the n-th basic pattern 4301. Further, here, the reference coordinates of the concentric lattice pattern forming the basic pattern 4301 and the center coordinates of the light receiving surface of the image sensor 3505 do not match. The center coordinates of the light receiving surface of the image sensor 3505 are the intersections of the diagonal lines that connect the four corners of the light receiving surface.

As described in the first embodiment, the angle of view determined by the equation (26) is the inclination of a straight line passing through the edge of the light receiving surface of the image sensor 3505 and the reference coordinates of the concentric circular grid pattern forming the basic pattern 4301. Specified by Thus, gaze direction is the vector connecting the two points and the center of the reference coordinates and the field A _n of concentric grating pattern constituting the basic pattern 4301. That is, it can be said that the gaze direction can be changed by designing the positions of the reference coordinates of the concentric lattice pattern forming the basic pattern 4301 and the center coordinates of the light receiving surface of the image sensor 3505.

FIG. 42 is a diagram illustrating another example of the gaze direction. This example shows a configuration example in which a plurality of provided basic patterns mutually change the gaze direction. The difference between the configuration of FIG. 41 and the configuration of FIG. 42 is the position of the reference coordinate of each concentric lattice pattern forming the basic pattern 4401 and the basic pattern 4402. In FIG. 42, these reference coordinates are arranged so that the center-to-center distance l of the concentric lattice patterns of the basic pattern 4401 and the basic pattern 4402 approaches. In this way, by changing the center coordinates of the concentric circular grid pattern for each basic pattern, it is possible to change the gaze direction of imaging by each basic pattern.

Depending on the type of the image sensor 3505, the center coordinates of the light receiving surfaces of the adjacent image sensors 3505 may not be close to each other due to the arrangement of the wirings protruding from the light receiving surface. Even in such a case, by changing the position of the reference coordinates of the basic pattern 3502, it is possible to match the visual field A _N with the imaging target and obtain a continuous visual field A _N.

As described above, the central coordinates of the concentric lattice pattern forming the basic pattern 4301 and the central coordinates of the image sensor 3505 do not match (the plane center of the lattice pattern and the center of the image sensor are in any direction on the plane). Offset), it is necessary to reflect the shift amount (offset amount) of the center coordinates in the x and y values used in the equation (25) for calculating the image distortion amount. In the case where the grid pattern as shown in the first embodiment includes only one basic pattern, one shift amount δc of the center coordinates to be set may be set in the image distortion correction set value input unit 3202. When a plurality of basic patterns are included in the lattice pattern as described above, it is possible to correctly correct the distortion of all basic patterns by setting the shift amount δc of the center coordinates for each basic pattern.

In this case, the values of δc are used to correct the values of x and y, and the corrected values are specified as x′ and y′. Then, the values of x and y in the equation (25) are calculated as x′ and y′ to specify δ _r in x and y.

According to the method and configuration described above, the gaze direction can be changed by setting the reference coordinates of the concentric lattice pattern that forms the basic pattern 3502, and the imaging target can be changed even when the arrangement method of the image sensor 3505 is restricted. It is possible to turn the field of view to.

[Fourth Embodiment]
In the first to third embodiments described above, the method in which the image distortion correction setting value input unit 3202 existing outside the image pickup apparatus 101 inputs the setting value used in the image distortion correction unit 3201 has been described. In the present embodiment, a configuration in which a setting value used in the image distortion correction unit 3201 is included inside the imaging device 101 will be described.

FIG. 43 is a diagram illustrating a configuration example in which the image distortion correction setting value is stored in the image distortion correction unit. In this configuration example, it is not necessary to give a set value for distortion correction from the outside, and the operability can be simplified. In this case, the set value stored in the image distortion correction set value storage unit 4501 is unique to the image pickup apparatus 101, and it is necessary to inspect and store the set value at the time of manufacturing according to the application. What is required as set values is the thickness l ₁ and the refractive index of the optical portion 2801, the thickness l ₂ and the refractive index of the optical portion 2802, and the distance d from the surface of the optical portion 2801 to the subject, as described above. Further, when the gaze direction is provided as in the third embodiment, the shift amount δc between the lattice pattern and the center position coordinates of the image sensor is stored in advance in the image distortion correction set value storage unit 4501.

In addition, in the fourth embodiment, when the object distance other than the initially set object distance is used, the center position coordinates of the grating pattern and the image sensor, or the thickness and the refractive index of the grating pattern are changed due to an external factor, and then again. The setting value stored in the image distortion correction setting value storage unit 4501 needs to be updated (given).

[Fifth Embodiment]
In the first to fourth embodiments, the distance from the lattice pattern to the subject, the thickness and the refractive index of the optical part other than the refractive index 1 existing between the image sensor and the subject, and the two-dimensional position information of the subject are used. The distortion is corrected based on the calculated distortion amount. A configuration for learning the setting value stored in the image distortion correction setting value storage unit 4501 from the image obtained by the image processing unit 106 will be described. FIG. 44 is a configuration diagram when the image distortion correction setting value learning unit 4601 is included in the image pickup apparatus 101. The image output from the image processing unit 106 is sent to the image distortion correction setting value learning unit 4601. At this time, it is desirable that the subject to be photographed has a known pattern, and if it is a known clear and uniform pattern such as a lattice pattern, the learning is still easy.

Note that the output image from the image processing unit 106 may be sent to the image distortion correction setting value learning unit 4601 and used for re-learning each time shooting is performed, or the image distortion correction setting value may be learned according to specific conditions. Only in this case, the output image may be sent to the image distortion correction setting value learning unit 4601. In this case, the specific condition may be, for example, the case where the subject is registered in advance, the case where the operation mode of the image processing unit 106 is switched to the mode for learning the distortion correction setting value, or the like. Further, this learning includes learning by various methods and may be rule-based learning processing or machine learning or learning processing based on a learned model in a neural network.

As described above, according to the imaging apparatus 101 according to the fifth embodiment, it is possible to widen the range of setting values used by the image distortion correction unit 3201 because distortion correction is performed based on the result of actually photographing a subject. For example, when a lattice-shaped pattern is photographed, distortion correction may be performed by specifying the deviation of the positional information of the intersections of each lattice (the amount of deviation from the ideal pattern, the direction of deviation, or the difference in the distance between the intersections). This makes it possible to perform distortion correction that reflects the state of the imaging device 101.

[Sixth Embodiment]
The first to fifth embodiments perform image processing, distortion correction, and image composition processing in the image pickup apparatus 101. In the sixth embodiment, a server device, a personal computer, a smartphone, a tablet, etc. are provided with the setting values necessary for the image acquired by the image sensor 3505 and the image processing, the setting necessary for the image distortion correction, and the image combining. The configuration will be described in which the output is output to the external device and subsequent development processing is performed on the external device side.

FIG. 45 is a diagram showing a configuration in which a sensor image and information about necessary setting values are output from the image pickup apparatus to the outside, and the information is received outside to perform image processing, image distortion correction, and image combination. That is, the image pickup apparatus 101 includes an image processing set value storage unit 4701, an image distortion correction set value storage unit 4501, an image combination set value storage unit 4702, and an information transmission unit 4703. Then, the information transmission unit 4703 outputs the output of the image sensor 3505 to the external device via the network, the communication line, or the wireless communication path, the information stored in the image processing setting value storage unit 4701, and the image distortion correction setting value. The information stored in the storage unit 4501 and the information stored in the image synthesis setting value storage unit 4702 are all transmitted in a form including them.

Then, the transmitted information is received via the information receiving unit 4704 and input to the image processing device 4705. The image processing device 4705 includes an information receiving unit 4704, the image processing unit 106, an image distortion correction unit 3201, and an image synthesizing unit 3506, and performs image processing and image distortion correction processing as in the above-described embodiment. , Performs image composition processing, and sends it to the image display unit 107.

The image display unit 107 configures a screen or the like using the combined image and displays it on the display device.

With such a configuration, it is not necessary to perform image processing, image distortion correction, and image composition in the image pickup apparatus 101, and the amount of calculation of the image pickup apparatus 101 can be significantly reduced. Therefore, it becomes possible to reduce the hardware requirement level of the image pickup apparatus 101, and the unit price of the image pickup apparatus 101 can be suppressed. In this way, if it becomes possible to cooperate with another device via a network, for example, a server device in a remote place, convenience is improved.

The present invention has been described above using a plurality of embodiments. Of course, the present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described.

Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add/delete/replace other configurations with respect to a part of the configurations of the respective embodiments.

Further, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them with, for example, an integrated circuit. Further, the above-described respective configurations, functions and the like may be realized by software by the processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that realize each function is stored in a memory, a hard disk, a recording device such as SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, and a DVD (Digital Versatile Disc). Can be placed.

In addition, the control lines and information lines shown are those that are considered necessary for explanation, and not all the control lines and information lines in the product are necessarily shown. In practice, it may be considered that almost all configurations are connected to each other.

The present invention is not limited to the image capturing apparatus and the image capturing method, and may be implemented in various modes such as an image capturing system, a developing method, a computer-readable program, an image processing circuit, a finger vein authentication device, a finger vein imaging device, and a finger vein authentication method. Can be provided.

Reference numeral 101... Imaging device, 102... Modulator, 102a... Lattice substrate, 102b... Support member, 103... Image sensor, 103a... Pixel, 104... First lattice pattern, 105... Second lattice pattern, 106... Image processing unit , 107... Image display section, 401... Subject, 3201,... Image distortion correction section, 3202... Image distortion correction set value input section, 3506... Image combining section, 4501... Image distortion correction set value storage section, 4601... Image distortion correction setting Value learning unit, 4701... Image processing set value storage unit, 4702... Image combination set value storage unit, 4703... Information transmitting unit, 4704... Information receiving unit, 4705... Image processing apparatus

Claims (13)

An image sensor that converts an optical image captured by a plurality of pixels arranged in an array on the imaging surface into an image signal and outputs the image signal.
A modulator provided on the light-receiving surface of the image sensor for modulating the intensity of light using a lattice pattern,
An image processing unit that performs predetermined image processing on the output image output from the image sensor,
An image distortion correction unit that performs distortion correction on the image output from the image processing unit,
An image distortion correction setting value input unit that transfers setting values for image distortion correction to the image distortion correction unit,
An imaging device comprising: The imaging device according to claim 1, wherein
The lattice pattern is a multiple lattice pattern having a plurality of basic lattice patterns,
Each of the plurality of basic lattice patterns has different fields of view,
An imaging device characterized by the above. The imaging device according to claim 1, wherein
A plurality of the image sensors,
The lattice pattern is a multiple lattice pattern having a plurality of basic lattice patterns,
Each of the plurality of basic lattice patterns has different fields of view,
The basic lattice pattern constitutes a basic unit that is associated with the image sensor in a one-to-one correspondence,
Each of the basic units has a different field of view,
An imaging device characterized by the above. The imaging device according to claim 1, wherein
The lattice pattern is a multiple lattice pattern having a plurality of basic lattice patterns,
The basic lattice pattern is provided on a plane that has a predetermined thickness and is substantially parallel to the light receiving surface of the image sensor, and the center position of the two-dimensional direction on the plane and the center position of the light receiving surface of the image sensor. And are offset in either direction on the plane,
The image distortion correction set value input unit uses the amount of the offset or the center position of the basic lattice pattern on the plane as the set value.
An imaging device characterized by the above. The imaging device according to claim 1, wherein
A plurality of the image sensors,
The lattice pattern is a multiple lattice pattern having a plurality of basic lattice patterns,
Each of the plurality of basic lattice patterns has different fields of view,
The basic lattice pattern constitutes a basic unit that is associated with the image sensor in a one-to-one correspondence,
Each of the basic units has a different field of view,
The basic lattice pattern is provided on a plane that has a predetermined thickness and is substantially parallel to the light receiving surface of the image sensor, and the center position of the two-dimensional direction on the plane and the center position of the light receiving surface of the image sensor. And are arranged offset in any direction on the plane,
The image distortion correction unit uses the amount of the offset or the center position of the basic lattice pattern on the plane as the set value.
An imaging device characterized by the above. The imaging device according to claim 1, wherein
The image distortion correction setting value input unit is provided in at least one of the image distortion correction unit and the image processing unit,
An imaging device characterized by the above. The imaging device according to claim 3 or 6, wherein
A distortion correction setting value learning unit that learns the setting value of the image distortion correction unit by a predetermined method using a result of capturing a predetermined known pattern,
An imaging device characterized by the above. An image sensor that converts an optical image captured by a plurality of pixels arranged in an array on the imaging surface into an image signal and outputs the image signal.
A modulator provided on the light-receiving surface of the image sensor for modulating the intensity of light using a lattice pattern,
An image processing setting value storage unit that stores setting values for processing the image signal output from the image sensor,
An image distortion correction setting value storage unit that stores a setting value for correcting distortion of the image signal output from the image sensor,
An image combination set value storage unit that stores a set value for combining the image signals output from the image sensor,
The image signal output from the image sensor, the set value stored in the image processing set value storage unit, the set value stored in the image distortion correction set value storage unit, and the image composition set value storage unit. The setting value stored in, an information transmitting unit for transmitting to a predetermined external device,
An imaging device comprising: The imaging device according to claim 8, wherein
The lattice pattern is a multiple lattice pattern having a plurality of basic lattice patterns,
Each of the plurality of basic lattice patterns has different fields of view,
An imaging device characterized by the above. The imaging device according to claim 8, wherein
A plurality of the image sensors,
The lattice pattern is a multiple lattice pattern having a plurality of basic lattice patterns,
Each of the plurality of basic lattice patterns has different fields of view,
The basic lattice pattern constitutes a basic unit that is associated with the image sensor in a one-to-one correspondence,
Each of the basic units has a different field of view,
An imaging device characterized by the above. The imaging device according to claim 8, wherein
The lattice pattern is a multiple lattice pattern having a plurality of basic lattice patterns,
The basic lattice pattern is provided on a plane having a predetermined thickness and substantially parallel to the light receiving surface of the image sensor, and the center position of the two-dimensional direction on the plane and the center position of the light receiving surface of the image sensor. And are arranged offset in any direction on the plane,
The image distortion correction set value storage unit stores the amount of the offset or the center position of the basic lattice pattern on the plane as the set value.
An imaging device characterized by the above. The imaging device according to claim 10, wherein
A plurality of the image sensors,
The lattice pattern is a multiple lattice pattern having a plurality of basic lattice patterns,
Each of the plurality of basic lattice patterns has different fields of view,
The basic lattice pattern constitutes a basic unit that is associated with the image sensor in a one-to-one correspondence,
Each of the basic units has a different field of view,
The basic lattice pattern is provided on a plane that has a predetermined thickness and is substantially parallel to the light receiving surface of the image sensor, and the center position of the two-dimensional direction on the plane and the center position of the light receiving surface of the image sensor. And are arranged offset in any direction on the plane,
The image distortion correction set value storage unit stores the amount of the offset or the center position of the basic lattice pattern on the plane as the set value.
An imaging device characterized by the above. The imaging device according to claim 8, wherein
A plurality of the image sensors,
The lattice pattern is a plurality of lattice patterns having a plurality of basic lattice patterns,
Each of the plurality of basic lattice patterns has a different field of view,
The basic lattice pattern constitutes a basic unit that is associated with the image sensor in a one-to-one correspondence,
Each of the basic units has a different field of view,
A distortion correction setting value learning unit that learns the setting value of the image distortion correction unit by a predetermined method using the result of capturing a predetermined known pattern,
An imaging device characterized by the above.

Images