WO2015001992A1 - Compound-eye imaging device - Google Patents

Abstract

　The present invention provides a compound-eye imaging device capable of quickly and suitably performing image processing even when the position of any individual eye image in a sensor array is displaced due to a lens array and a holder being deformed together with a change in an environmental condition. An image processing circuit (80) executes image processing on the basis of information pertaining to a difference between at least a first position (T1) and a second position (T2) and information indicating the degree of deformation of a lens array (10) together with a change in an environment condition. For example, a positional change of any individual eye image in an sensor array (60) caused together with a change in an environment condition, etc., that is, a positional change of an image obtained corresponding to an optical image in the sensor array (60), is corrected. The displacement of any individual eye image in the sensor array (60) can thereby be quickly and accurately ascertained and excellent image processing can thereby be performed against the change in the environment condition, even if the centers of deformation of the lens array (10) and the holder (50), as viewed from the direction of an optical axis (OA) of a lens unit (10a), do not match.

Description

Compound eye imaging device

The present invention relates to a compound eye imaging device including a lens array having a plurality of lens portions arranged two-dimensionally.

Demand for thinning of imaging optical systems in recent years has been greatly increased. In order to respond to this, we have improved the manufacturing accuracy in response to the shortening of the overall length by optical design and the accompanying increase in error sensitivity, but in order to meet the demand for further thinning, one conventional optical system and image sensor The method of obtaining an image by the above is insufficient.

Therefore, the compound eye optical system used in a compound eye imaging device that divides the detection area of the image sensor, arranges the optical system so as to correspond to each, and processes the obtained image to output the final image In order to meet the demand for thinning, an optical system called “A” is attracting attention. However, in an imaging apparatus including a compound eye optical system, the arrangement pitch of the lens unit of the imaging apparatus changes due to a change in environmental conditions such as the environmental temperature. It is possible to plan.

As an example of a compound-eye imaging device that performs image processing, an imaging device in which a light shielding wall having a rectangular hole corresponding to each lens is disposed between a lens array and an imaging element, and the light shielding wall is placed at an ambient temperature. A binary image reflecting the position of the light-shielding wall is acquired from the luminance information incident on the image sensor that can capture the shadow of the light-shielding wall between individual eye images, taking the shape and material according to the shape change of the lens array due to the change. On the basis of this, there is a description that calculates and corrects the amount of distortion of the light shielding wall accompanying the change in ambient temperature (see Patent Document 1).

However, according to the study by the present inventors, it has been found that it may not be possible to sufficiently cope with changes in environmental conditions such as the environmental temperature simply by focusing on the deformation of the lens array or the light shielding wall. This is because the lens array is normally held by a holder so as to face the image sensor, and the position change of each eye image on the image sensor is apparent due to deformation of the holder due to environmental conditions. Furthermore, it is thought that it becomes complicated and it becomes impossible to respond accurately to changes in environmental conditions. More specifically, since it is practically difficult to match the center of deformation of the holder and the center of deformation of the lens array due to changes in environmental conditions, the shift of the lens portion relative to the image sensor that occurs with changes in environmental conditions is complicated. It is thought to become. In Patent Document 1, the position of the shadow of the light shielding wall is used to indirectly grasp the deformation of the lens array due to the temperature change. However, the position of the shadow of the light shielding wall depends on the state of the light source and the use environment of the imaging device. The detection result is likely to be unstable due to the fact that the detection result is unstable. In addition, it is difficult to bring the light shielding wall and the image sensor into close contact with each other, and the shadow of the light shielding wall projected onto the image sensor is likely to be blurred. Therefore, there is a possibility that the positions of the light shielding walls, particularly the positions of the four corners, cannot be obtained accurately. Further, in Patent Document 1, since the deformation of the lens array is indirectly estimated from the deformation of the light shielding wall, it is impossible to accurately grasp the deformation of the lens array and the deformation of the housing holding the lens array. Therefore, in the imaging apparatus of Patent Document 1, super-resolution processing for obtaining one high-resolution image from images of the same field formed by the individual lens units, and calculation of the parallax of the individual lens units with high accuracy. It is difficult to realize.

JP 2011-147079 A

The present invention provides a compound-eye imaging device capable of performing image processing quickly and satisfactorily even when the position of each single-eye image on the sensor array is shifted due to deformation of the lens array and the holder as the environmental conditions change. The purpose is to do.

In order to solve the above-described problems, a compound eye imaging apparatus according to the present invention includes a compound eye optical system including a lens array having a plurality of lens units arranged two-dimensionally in a direction perpendicular to the optical axis, and a compound eye optical system. Forms a plurality of optical images corresponding to the plurality of lens units, outputs a sensor signal corresponding to the plurality of optical images, a holder for holding the compound eye optical system so as to face the sensor array, and a sensor An image processing circuit for processing an image signal output from the array, and a first position that is a center position of deformation accompanying a change in the environmental condition of the holder when viewed from the optical axis direction in the reference environmental condition The lens array is fixed to the holder in a state where the second position, which is the center position of deformation accompanying the change in the environmental conditions of the lens array when viewed from the optical axis direction, is separated from the holder. At least, performs image processing based on the first position information about the differences between the second position, and information indicating the degree of deformation due to changes in environmental conditions of the lens array. Here, the first position refers to the holder that is the center of deformation when viewed from the optical axis direction of the lens unit when the holder that fixes the lens array expands or contracts as the environmental conditions change. Is the position. When the holder is fixed to the sensor array or the substrate on which the sensor array is provided, the first position can be a place where the relative position with respect to the sensor array does not substantially change. The second position refers to the lens array that becomes the center of deformation when viewed from the optical axis direction of the lens unit when the lens array fixed to the holder expands or contracts as the environmental conditions change. It is the upper position. The second position may be a place where the relative position does not substantially change with respect to the fixed place of the holder. As information regarding the difference between the first position and the second position, for example, a vector indicating movement from the first position to the second position can be used. As the information indicating the degree of deformation accompanying the change in the environmental conditions of the lens array, for example, a vector indicating the deformation for each position in the lens array derived from the linear expansion coefficient of the lens array can be used.

In the compound eye imaging device, the image processing circuit executes image processing based on at least information regarding a difference between the first position and the second position and information indicating a degree of deformation accompanying a change in environmental conditions of the lens array. To do. For example, a change in the position of each single-eye image on the sensor array caused by a change in environmental conditions such as a change in environmental temperature, that is, a change in the position of an image obtained corresponding to the optical image on the sensor array is corrected. Thereby, even when the deformation center of the lens array and the holder when viewed from the optical axis direction of the lens portion does not match, it is possible to quickly and accurately grasp the positional deviation of each eye image on the sensor array, Good image processing can be performed against changes in environmental conditions.

In a specific aspect or embodiment of the present invention, the lens array is bonded to the holder with an adhesive.

In another aspect of the present invention, the adherend portion of the lens array is provided at a point-symmetrical position with respect to the center of the lens portion arrangement region where a plurality of lens portions are provided in the lens array.

In yet another aspect of the present invention, the adherend portion of the lens array is continuous so as to surround the plurality of lens portions.

In still another aspect of the present invention, the holder has a ceiling portion and leg portions that support the ceiling portion. *

In yet another aspect of the present invention, the lens array is fixed to the ceiling of the holder. *

In yet another aspect of the present invention, the object side surface of the lens array is fixed to the image side surface of the ceiling of the holder.

In yet another aspect of the present invention, the holder has a deformation center that is substantially unchanged in relative position with respect to the sensor array when the shape changes in a direction perpendicular to the optical axis in accordance with a change in environmental conditions. Are arranged as follows.

In yet another aspect of the present invention, a substrate having a sensor array is provided, and the holder is fixed to the substrate.

In yet another aspect of the present invention, a substrate provided with a sensor array is provided, and the legs of the holder are fixed to the substrate.

In yet another aspect of the present invention, the lens array is formed of a resin.

In yet another aspect of the present invention, the holder is made of resin. A holder having a shape suitable for a lens array, a sensor array, or the like can be easily obtained.

In still another aspect of the present invention, the linear expansion coefficient α1 (1 / ° C.) of the holder, the distance L (mm) between the first position and the second position, and the pixel pitch P (mm) of the sensor array are 30 L · The relationship of α1 <5P is satisfied. Here, the above relational expression assumes a temperature change of 30 ° C.

In still another aspect of the present invention, the linear expansion coefficient α1 of the holder, the distance L between the first position and the second position, and the pixel pitch P of the sensor array satisfy the relationship of 30L · α1 <2P.

In still another aspect of the present invention, the linear expansion coefficient α1 of the holder and the linear expansion coefficient α2 of the lens array satisfy a relationship of 1.2> α2 / α1> 0.8. In this case, since the linear expansion coefficient of the lens array and the linear expansion coefficient of the holder are close to each other, the deformation amounts of both when the environmental conditions change are close, so the relative positions of the plurality of optical images formed on the sensor array The shift is relatively small.

In still another aspect of the present invention, the linear expansion coefficient α2 of the lens array and the linear expansion coefficient α3 of the sensor array satisfy the relationship of α2 / α3> 5.

In still another aspect of the present invention, the image processing circuit uses a combined vector of a first vector indicating movement from the first position to the second position and a second vector indicating deformation for each position in the lens array. Image processing.

In still another aspect of the present invention, the image processing circuit is configured to adjust the environmental condition based on information regarding a difference between the first position and the second position and information indicating a degree of deformation accompanying a change in the environmental condition of the lens array. The corresponding region of each single-eye image on the sensor array is estimated, and pattern matching of the image signal corresponding to the estimated presence region is executed, and the correction amount of the position change of each single-eye image due to the change in environmental conditions is calculated. Ask.

In yet another aspect of the present invention, the reference environmental condition is to set the environmental temperature to room temperature.

In still another aspect of the present invention, the compound-eye optical system further includes a second lens array that is disposed so as to overlap the optical axis direction of the first lens array, with the lens array as the first lens array.

1A and 1B are a plan view and a cross-sectional view illustrating a compound eye imaging device according to an embodiment. It is a figure explaining a lens array laminated body and a holder among compound eye imaging devices. FIGS. 3A to 3C are schematic views showing how the lens units shift with respect to the sensor array when the environmental conditions change. It is a figure explaining the measurement result which shows the position shift of the image on the sensor array which arises with the change of environmental conditions. FIG. 5A is a block diagram illustrating a basic configuration of the compound eye imaging apparatus according to the embodiment, and FIG. 5B is a block diagram functionally illustrating an image processing circuit provided in the digital processing circuit. It is a conceptual diagram explaining many conversion area | regions or input images arranged on the photoelectric conversion surface of a sensor array. It is a flowchart showing the flow of operation | movement in an image processing circuit. It is a figure for demonstrating the shift | offset | difference of the pixel position by the temperature change of the input image from each synthetic lens. It is a figure for demonstrating the 1st specific example of the positional offset rough estimation process as a 1st positional offset estimation process process in FIG.7 S12. It is a figure for demonstrating the 2nd specific example of a position shift outline estimation process. It is a figure for demonstrating the 3rd specific example of a position shift outline estimation process. It is a figure for demonstrating the modification of a position shift outline estimation process. 13A to 13C are diagrams for explaining a specific example of the second positional deviation estimation processing step in step S13 of FIG. It is a figure showing the flow of the super-resolution process in step S14 of FIG. It is a figure explaining the deterioration information used by FIG.14 S33. It is a figure showing the specific example of deterioration information.

[Image Detection Unit of Imaging Device]
1A and 1B are a plan view and a cross-sectional view illustrating an image detection unit for performing image detection in a compound eye imaging apparatus according to an embodiment of the present invention.
As shown in FIGS. 1A and 1B and the like, an imaging device 100 (compound eye imaging device) includes a lens array stack 1 and a sensor array 61 corresponding to each synthetic lens 1a constituting the lens array stack 1. 60, a digital processing circuit 103 including an image processing circuit 80 that performs image processing conforming to a visual field division method or a super-resolution method on an image signal detected by the sensor array 60, and a substrate on which the sensor array 60 is mounted SB. Among these, the lens array laminate 1, the sensor array 60, and the substrate SB function as an image detection unit. Here, the field division method refers to a method of obtaining one image by joining images of different fields of view formed by individual lenses and connecting the images of the fields of view by image processing. The super-resolution method refers to a method of obtaining one high-resolution image by image processing from images of the same field of view formed by individual lenses. Note that in this specification, image processing refers to correction or conversion performed on image data by an image processing circuit including a computing device such as a computer in order to obtain a reconstructed image from each individual eye image. Refers to the processing.

The illustrated lens array laminate 1 is a laminate in which a plurality (specifically, two) of lens arrays 10 and 20 are stacked, and is used as a compound eye optical system. In the following description, the lens array laminate 1 may be referred to as a lens array. These first and second lens arrays 10 and 20 are square plate-like members extending in parallel to the XY plane, and are stacked in the Z-axis direction perpendicular to the XY plane and joined to each other.

The lens array laminate 1 is stored in a rectangular frame-shaped holder 50 in a state of facing the sensor array 60. In the lens array laminate 1, the first lens array 10 on the object side is an integrally molded product made of a light-transmitting thermoplastic resin that is an optical material, in other words, an integrated product made of a thermoplastic resin. The first lens array 10 has a rectangular outline when viewed from the central axis AX direction or the Z-axis direction. Here, the central axis AX indicates an axis at the center of an array region (lens portion array region LR) of a plurality of lens units 10a described later. The first lens array 10 has a plurality of lens portions 10a each of which is an optical element, a support portion 10b that supports the plurality of lens portions 10a from the periphery, and a rectangular frame shape that extends in a strip shape outside the support portion 10b. And an edge portion 10r. The plurality of lens portions 10a constituting the first lens array 10 are two-dimensionally arranged on square lattice points (16 × 4 × 4 in the illustrated example) arranged in parallel to the XY plane. Each lens unit 10a has a first optical surface 11a that is convex on the first main surface 10p on the object side, and a second optical surface 11b that is convex on the second main surface 10q on the image side. The first and second optical surfaces 11a and 11b are, for example, aspherical surfaces. The support portion 10b is a flat plate-like portion and includes a plurality of peripheral portions 10c so as to surround each lens portion 10a. A square frame-shaped edge portion 10r on the periphery or outside in the lateral direction of the support portion 10b is a portion for joining the first lens array 10 to the second lens array 20. In the first lens array 10, it is assumed that there are an area where the lens part 10a is arranged on a lattice point and a frame-like area where the lens part 10a is not provided on the support part 10b. You can also.

The second lens array 20 on the image side is an integrally molded product made of a thermoplastic resin and has a rectangular outline when viewed from the direction of the central axis AX. The second lens array 20 has a plurality of lens portions 20a each of which is an optical element, a support portion 20b that supports the plurality of lens portions 20a from the periphery, and a rectangular frame shape that extends in a band shape outside the support portion 20b. And an edge portion 20r. The plurality of lens portions 20a are two-dimensionally arranged on square lattice points (16 × 4 × 4 in the illustrated example) arranged in parallel to the XY plane. Each lens unit 20a has a first optical surface 21a that is concave on the first main surface 20p on the object side, and a second optical surface 21b that is convex on the second main surface 20q on the image side. The first and second optical surfaces 21a and 21b are, for example, aspherical surfaces. The support portion 20b is a flat plate-like portion and includes a plurality of peripheral portions 20c so as to surround each lens portion 20a. A square frame-shaped edge portion 20r on the periphery or outside in the lateral direction of the support portion 20b is a portion for joining the second lens array 20 to the first lens array 10. In the second lens array 20, it is assumed that there are an area where the lens part 20a is arranged on a lattice point and a frame-like area where the lens part 20a is not provided on the support part 20b. You can also.

The above lens arrays 10 and 20 are positioned with respect to each other by their own structure, their own weight, electrostatic force, and the like by sequentially stacking them by machine or manual work. That is, self-alignment is performed. However, a positioning portion may be provided in the lens array, and positioning may be performed by bringing the lens array into contact with each other. Further, when the lens arrays 10 and 20 are stacked, the lens arrays 10 and 20 are bonded or bonded together by supplying, for example, a photocurable resin between the edge portions 10r and 20r and then curing the photocurable resin. By such joining or adhesion, a lens array laminate 1 including a large number of synthetic lenses 1a arranged two-dimensionally in a matrix is obtained. The optical axis OA of each synthetic lens 1a is parallel to the central axis AX of the entire area of the plurality of lens portions 10a, 20a.

Note that a thin light-shielding first aperture plate 41 is disposed between the first lens array 10 and the second lens array 20 so as to be sandwiched and fixed between the pair of support portions 10b and 20b. Although a detailed description is omitted in the first diaphragm plate 41, a large number of openings are formed corresponding to the respective lens portions 10a. On the image side of the second lens array 20, a thin light-blocking second diaphragm plate 42 fixed to the support portion 20b by bonding or the like is disposed. Although a detailed description is omitted, the second aperture plate 42 has a large number of openings corresponding to the lens portions 20a and the like.

Since the lens arrays 10 and 20 or the lens array laminate 1 described above are made of resin, they are lightweight, thin and inexpensive. On the other hand, the lens arrays 10 and 20 have a larger linear expansion coefficient than the sensor array 60, and the positional deviation of each eye image on the sensor array 60 tends to increase. For this reason, the method of assembling the lens arrays 10 and 20 or the lens array laminated body 1 to the holder 50 is important. When such an assembling method is appropriate, the positional deviation of each eye image on the sensor array 60 is corrected. Effect is more prominent. In addition, when using the lens array laminated body 1, since each eye is comprised with the optical system which consists of several groups of lens parts 10a and 20a, it becomes advantageous to image quality improvement.

The holder 50 is an integrally molded product made of a thermoplastic resin, and has a ceiling portion 50a having a rectangular outline when viewed from the central axis AX direction of the lens array laminate 1, and a ceiling portion arranged circumferentially along this outline. A wall-like leg portion 50b extending in the direction of the central axis AX from 50a. The holder 50 is fixed on the substrate SB by adhering the entire bottom surface of the leg portion 50b in a state where the lens array laminate 1, the sensor array 60, and the like are disposed so as to be surrounded by the ceiling portion 50a and the leg portion 50b. When the holder 50 is fixed on the substrate SB in this way, a circuit associated with the sensor array 60 can be disposed on the substrate SB in the vicinity of the sensor array 60. The performance can be improved. In particular, by fixing the leg portion 50b of the holder 50 to the substrate SB, it becomes easy to grasp the deformation of the holder 50 due to a change in environmental conditions, and image processing becomes easy.
By using the holder 50 having the ceiling 50a and the leg 50b as described above, the lens array laminate is provided at a position facing the sensor array 60 while ensuring the accommodation or arrangement space of the sensor array 60 in the holder 50. 1 can be arranged from the main surface side of the first lens array 10 constituting this. Further, the main surface of the first lens array 10 is bonded and fixed to the image side of the ceiling 50a of the holder 50, whereby the lens array laminate 1 in the direction perpendicular to the optical axis OA of the lens portions 10a and 20a. Alternatively, it is relatively difficult to prevent deformation including movement of the lens arrays 10 and 20, and it becomes easy to grasp the deformation of the lens arrays 10 and 20 and the like caused by changes in environmental conditions. An entrance diaphragm plate 45 fixed by adhesion to the holder 50 or the like is disposed on the object side of the holder 50 that houses the lens array stack 1. Although detailed description is omitted in the ceiling portion 50a and the entrance diaphragm plate 45 of the holder 50, a large number of openings are formed corresponding to the lens portions 10a and the like. Since the holder 50 has its legs 50b bonded and fixed to the substrate SB, the ceiling 50a is mainly deformed due to a change in environmental temperature. For this reason, the diagonal center of the ceiling part 50a substantially coincides with the deformation center. The ceiling 50a of the holder 50 is also restrained by the lens array laminate 1 (lens arrays 10 and 20) to which the holder 50 is bonded, but the size of the holder 50 is increased or a highly flexible adhesive is used. For example, the influence of the constraint on the lens arrays 10 and 20 can be reduced. In addition, by holding the lens arrays 10 and 20 with the holder 50, a space in which the compound eye optical system faces the sensor array 60 can be secured even if another member is provided around the sensor array 60.

Since the holder 50 is made of resin, it is inexpensive, and a holder having a shape suitable for the lens arrays 10, 20 and the sensor array 60 can be easily obtained. On the other hand, the linear expansion coefficient is larger than that of the sensor array 60, and the position shift of each individual eye image on the sensor array 60 tends to increase. For this reason, the effect which correct | amends the position shift of each single-eye image on the sensor array 60 expresses more notably.

The sensor array 60 is mounted on the substrate SB. The sensor area 61 provided in the sensor array 60 is a two-dimensional array of a plurality of pixels that perform photoelectric conversion, and is arranged in a localized area corresponding to each synthetic lens 1a on the sensor array 60. Alternatively, the sensor array 60 may be disposed throughout. In addition to the sensor array 60, an AD conversion unit 101b, which will be described later, and the like are mounted on the substrate SB.

Hereinafter, the lens array laminate 1, the holder 50, and the like will be described in detail with reference to FIGS. 1B, 2 and the like. As shown in FIGS. 1B and 2, the lens array laminate 1 is fixed to the image side surface of the ceiling 50 a of the holder 50 with an adhesive. As a result, the lens array laminate 1 including the lens arrays 10 and 20 is housed in the holder 50 and is stably fixed to an appropriate position in the holder 50 with high accuracy. As a result, the sensor array 60 is also accurate. It can be fixed stably and well. In the case of the present embodiment, the adherend portion QB of the lens array stack 1 is relative to the central axis AX of the region (lens portion arrangement region LR) in the lens array stack 1 where the plurality of lens portions 10a and 20a are provided. It is arranged at a point-symmetrical position. Specifically, the adherend portion QB of the first lens array 10 is disposed between the object-side main surface 10p of the first lens array 10 and the image side surface of the ceiling 50a of the holder 50 in the lens array stack 1. However, the adhesive is disposed so as to surround the plurality of lens portions 10a so as to be substantially square in a plan view (in FIGS. 1A, 1B, and 2, the adherend portion QB is on the object side). Since it is not directly observed, it is shown with a broken line). Here, on the central axis AX of the lens array stacked body 1, the support reference point OS that is the center of the region surrounded by the adherend portion QB is disposed. By arranging the adherend portion QB in this way, the lens array stack 1 is expanded or contracted isotropically and relatively undistorted from the central axis AX or the support reference point OS due to changes in environmental conditions. The central axis AX or the support reference point OS substantially coincides with a second position T2 described later. In particular, by providing the adherend portion QB of the lens array 10 so as to surround the whole of the plurality of lens portions 10a, it becomes easy to realize isotropic deformation. The lens array stack 1 is associated with a first position T1 that is a center position of deformation accompanying a change in the environment of the holder 50 when viewed from the optical axis OA direction and a change in the environment of the lens array stack 1 under reference environmental conditions. It is fixed to the holder 50 so that the second position T2, which is the center position of deformation, is separated by a distance L.

Here, the reference environmental condition is an environmental condition in which the holder 50 and the lens array stacked body 1 are shaped to serve as a reference for defining the degree of deformation. For example, if the factor causing the deformation of the holder 50 and the lens array stack 1 is the environmental temperature, the temperature at which the holder 50 and the lens array stack 1 become the reference shape (for example, room temperature of 20 ° C.) is the reference environmental condition. . In this case, the lens array stack 1, the holder 50, and the like can be assembled at room temperature, and the imaging device 100 can be easily manufactured.
Further, the first position T1 is a member that supports the holder 50 when the holder 50 that fixes the lens array stack 1 is deformed mainly in the planar direction in accordance with changes in environmental conditions (in this embodiment, It is a position on the holder 50 where the relative position when viewed from the direction of the optical axis OA with respect to the substrate SB) does not substantially change. In the case of the present embodiment, the first position T1 is the center of the portion fixed to the substrate SB in the holder 50, that is, the center of the rectangular frame (intersection of diagonal lines) in the holder 50 in plan view. If the deformation of the substrate SB and the sensor array 60 due to environmental changes is sufficiently smaller than the deformation of the holder 50, the first position T1 can also be viewed as being position-invariant with respect to the sensor array 60. Thus, the presence of the first position T1 that does not change position with respect to the sensor array 60 in the holder 50 makes it easy to grasp the deformation of the holder 50 due to a change in environmental conditions, and image processing becomes easy.
Further, in the present embodiment, the second position T2 is relative to the holder 50 when the lens array stack 1 fixed to the holder 50 is deformed mainly in the planar direction with changes in environmental conditions. This is the position on the lens array laminate 1 where the relative position does not change substantially. In the case of this embodiment, the second position T2 is the center (intersection of diagonal lines) of the frame-shaped adherend QB in the lens array stack 1. That is, the center axis AX of the lens unit arrangement region LR and the deformation center substantially coincide.

Note that the first position T1 and the second position T2 are separated from each other first because it is difficult to accurately match them. Second, because of the following factors, the shape of the holder 50 cannot be the same as the aspect ratio of the lens portion arrangement region LR of the lens array stack 1. Specifically, (1) the sensor array 60 requires a functional unit such as a reading unit in addition to the photoelectric conversion unit, and (2) the wire bonding of the sensor array 60 may not be arranged on four sides. (3) When mounting the sensor array 60 on the substrate SB, it may be necessary to arrange other electrical components such as resistors and capacitors in the vicinity of the sensor array 60, and (4) the compound eye imaging device. Design requirements (5) When the lens arrays 10 and 20 are made of plastic, a margin portion is provided in the vicinity of the gate portion for injection molding to increase the injection stability. For example, the shape is not similar to the lens portion arrangement region. For these reasons, the lens part array region LR of the lens array stack 1 is fixed at a position offset with respect to the holder 50, and the center position of the lens part array region LR of the lens array stack 1 and the center of the holder 50 are aligned. The position of each optical image formed on the sensor array 60 is isotropic starting from the center of the arrangement of the lens portions 10a and 20a as the environmental conditions such as the environmental temperature change. It will behave differently from change.

As already described, the lens array laminate 1 and the holder 50 are each formed of resin. It is preferable that the linear expansion coefficient α2 of the lens array laminate 1 (lens arrays 10 and 20) and the linear expansion coefficient α1 of the holder 50 satisfy the relationship of 1.2> α2 / α1> 0.8. The linear expansion coefficients α1 and α2 of the holder 50 and the lens array stacked body 1 are larger than the linear expansion coefficient of the sensor array 60 (for example, 5 times or more), and the positional deviation of each eye image on the sensor array 60 tends to increase. There is. In addition, since the linear expansion coefficient of the lens array laminate 1 and the linear expansion coefficient of the holder 50 are relatively close, the deformation center position when the environmental conditions change is close to the first position T1 of the holder 50, so the sensor array 60 The relative positional deviation of the plurality of optical images formed on the image becomes relatively small. Thereby, it is possible to prevent an increase in the number of pixels corresponding to non-overlapping portions occurring before and after the positional deviation, and the calculation for correction is simplified.

The linear expansion coefficient α1 (1 / ° C.) of the holder 50, the distance L (mm) between the first position T1 and the second position T2, and the pixel pitch P (mm) of the sensor array 60 are 30L · α1 <5P. Preferably, the relationship of 30L · α1 <2P is satisfied. By doing in this way, the effective area | region required for the image acquisition decided by the relationship of the relative position of the lens parts 10a and 20a of the lens arrays 10 and 20, the holder 50, and the sensor array 60 can be ensured reliably. . In other words, it is not necessary to increase the number of pixels of the sensor array 60 more than necessary. Here, the above relational expression assumes a temperature change of 30 ° C.

The linear expansion coefficient α2 of the lens array laminate 1 (lens arrays 10 and 20) and the linear expansion coefficient α3 of the sensor array 60 satisfy the relationship of α2 / α3> 5. By doing in this way, compared with the deformation amount of the lens arrays 10 and 20 accompanying the change of environmental temperature, the deformation amount of the sensor array 60 accompanying the change of environmental temperature can be made small enough, and the load on image processing It becomes easy to reduce.

Hereinafter, the state of deformation and displacement of the lens array laminate 1 when the environmental conditions change will be described. Hereinafter, for the sake of simplicity, the shape change state of the first lens array 10 will be described, but the same applies to the shape change state of the lens array stack 1. That is, the deformation and misalignment that occur in the first lens array 10 also occur in the lens array stack 1. At this time, if there is a slight difference in expansion coefficient between the first and second lens arrays 10 and 20, a deformation or a positional deviation that averages the expansion coefficients of the first and second lens arrays 10 and 20 occurs.

When it is assumed that the holder 50 is not deformed, only the first lens array 10 is deformed in the environmental condition change, and the second position T2 that is the deformation center is the support center (support reference point OS) or the center. The first lens array 10 coincides with the axis AX and isotropically deformed around the support center (support reference point OS) or the center axis AX. Therefore, as shown in FIG. 3B, the sensor array 60 is displaced radially isotropically around the second position T2. The same applies to the case where the deformation center of the holder 50, the deformation center of the lens array stack 1, and the center position of the sensor array 60 are matched.

On the other hand, as in the present embodiment, when the first position T1 that is the deformation center of the holder 50 and the second position T2 that is the deformation center of the first lens array 10 are separated by a distance L (see FIG. 2), The one lens array 10 isotropically deformed around the first position T1 and moves relative to the sensor array 60 by the deformation of the holder 50.

3A, a vector M1 indicating the movement of the second position T2 due to the deformation of the holder 50 is represented by a one-dot chain line, and a vector M2 indicating the deformation of the first lens array 10 is represented by a two-dot chain line. The direction in which the vector M1 and the vector M2 are combined becomes a deformation vector M3 for the substrate SB of the first lens array 10 and the sensor array 60 when the environmental conditions change. From the sensor array 60 side, it is observed as if each lens unit 10a moved radially about a certain point (coordinate SP0 in FIG. 3A). When the degree of change in the environmental condition is 1/2 of FIG. 3A, the deformation vector M3 of the first lens array 10 has a half size as shown in FIG. 3C. In FIGS. 3A to 3C, the vectors M1, M2, and M3 are exaggerated for easy understanding.

Therefore, as a simple model, by synthesizing the vector M2 indicating the degree of deformation of the lens array 10 itself and the vector M1 indicating the movement of the lens array 10, the positional shift caused by these effects can be grasped almost accurately. be able to. For example, by storing the vectors M1 and M2 in the reference environmental condition and correcting the vectors M1 and M2 in accordance with the change in the environmental condition, it is possible to grasp the positional deviation by combining the corrected vectors M1 and M2. it can. Actually, in order to perform the correction more accurately, it is preferable to perform the correction by predicting the deviation based on the two vectors and then narrowing down the region and analyzing the image. By doing so, correction can be performed accurately and quickly. The sensor area 61 has an area with the number of pixels that can sufficiently cover the displacement of the single-eye image due to a change in environmental conditions.

FIG. 4 is a diagram showing actual measurement results for explaining deformation and displacement due to a change in environmental conditions of the lens array laminate 1. In actual measurement, an 11 mm square plastic lens array 10 in which 4 × 4 lens portions are arranged in a matrix with a pitch of 2.5 mm, and the size of the ceiling portion 50a are horizontal (X direction) 13.9 mm, vertical (Y Direction) A 13.7 mm plastic holder 50 was used. The lens array 10 was made of cyclic polyolefin APEL (linear expansion coefficient 70 × 10 ⁻⁶ / ° C.) made of Mitsui Chemicals, and the holder 50 was made of polycarbonate (linear expansion coefficient 60 × 10 ⁻⁶ / ° C.). . A sensor array 60 having a pixel pitch P of 1.7 μm and a linear expansion coefficient of 4 × 10 ⁻⁶ / ° C. was used. The lens array 10 was adhered to the holder 50 using a thermosetting adhesive so that the distance L between the first position T1 and the second position T2 was 1.5 mm. At this time, an adhesive was applied in a quadrangular shape so as to surround the plurality of lens portions 10 a symmetrically with respect to the center of the arrangement of the plurality of lens portions 10 a and adhered to the holder 50. The leg part 50b of the holder 50 was bonded and fixed to the substrate on which the sensor array 60 was mounted. Then, the environmental temperature was changed from 26 ° C. to 58 ° C., and the obtained image was analyzed to measure the change in the center position of each individual eye image. FIG. 4 shows 16 image data corresponding to each individual eye, with only the vicinity of the center pixel extracted and arranged. As the environmental temperature rises, a positional deviation occurs, and apparently, a radial positional deviation occurs centering on a position (coordinate SP0) that is shifted from the central axis AX of the lens portion arrangement region LR. It was observed.

[Circuit part of imaging device]
FIG. 5A is a block diagram illustrating a basic circuit configuration of the imaging apparatus according to the present embodiment. As illustrated, the imaging apparatus 100 includes an image detection unit 101, a CPU (Central Processing Unit) 102, a digital processing circuit 103, an image display unit 104, a card interface (I / F) 107, and a storage unit 108. And an operation key 109.

Note that the imaging apparatus 100 can be configured as a system in which each unit is embodied as an independent apparatus, but is generally embodied as a digital camera, a personal computer, a portable communication terminal, or the like. That is, a user such as a digital camera can view a high-resolution image on the image display unit 104 by capturing an image of a subject using the imaging device 100, and store the high-resolution image in the storage unit 108. Can do. Specifically, in the imaging apparatus 100, the image detection unit 101 detects light from the subject and generates an image signal (input image) corresponding to the subject, and the image processing circuit 80 of the digital processing circuit 103 uses the image signal. On the other hand, by performing image processing as will be described later, a high-resolution output image having a higher frequency component than the input image (hereinafter also referred to as “high-resolution image”) is generated. Then, the digital processing circuit 103 outputs the high resolution image to the image display unit 104, the storage unit 108, and the like.

The image detection unit 101 is a part that generates an image of an object (an input image obtained by imaging the object) as data by capturing an image of the object, and is also called a camera module. As described above, the image detection unit 101 includes a main body part 101a including an optical system as shown in FIGS. 1A and 1B, and an A / D (Analog-to-Digital) conversion unit 101b connected to the main body part 101a. Including. The A / D conversion unit 101b converts a video signal (analog electrical signal) indicating a subject detected by the main body 101a into a digital signal and outputs the digital signal. The digital signal output from the A / D conversion unit 101b, that is, the image detection unit 101 includes information about an input image obtained by imaging the subject. Note that the image detection unit 101 may further include a control processing circuit for controlling each part of the imaging device 100.

The CPU 102 executes processes based on programs and setting values stored in advance in a ROM or RAM, so that the image detection unit 101, the digital processing circuit 103, the image display unit 104, the card interface (I / F) 107, etc. The operation is comprehensively controlled, and the entire imaging apparatus 100 is caused to perform a predetermined operation such as an operation according to a user request such as imaging or a state monitoring. The first position T1, the second position T2, the linear expansion coefficient α1 of the holder 50, the linear expansion coefficient α2 of the lens array stack 1 (lens arrays 10 and 20), the linear expansion coefficient α3 of the sensor array 60, the vector W1, Correction coefficients and the like of W2 and vectors W1 and W2 due to changes in environmental temperature are stored in ROM or RAM.

The digital processing circuit 103 operates under the control of the CPU 102 and executes various digital processes including image processing according to the present embodiment. The digital processing circuit 103 typically includes a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an LSI (Large Scale Integration), an FPGA (Field-Programmable Gate Array), and the like. The digital processing circuit 103 includes an image processing circuit 80 that performs intended image processing including super-resolution processing on the input image obtained by the image detection unit 101 to generate an output image.

The image display unit 104 displays an input image provided by the image detection unit 101, an output image generated by the digital processing circuit 103, various setting information related to the imaging apparatus 100, a control GUI (Graphical User Interface) screen, and the like. .

A card interface (I / F) 107 is an interface for writing an output image or image data generated by the image processing circuit 80 of the digital processing circuit 103 to the storage unit 108 or reading image data or the like from the storage unit 108. is there. The storage unit 108 is a storage device that stores image data generated by the image processing circuit 80 and other various types of information (setting values such as control parameters and operation modes of the imaging apparatus 100). The storage unit 108 includes a flash memory, an optical disk, a magnetic disk, and the like, and stores data in a nonvolatile manner.

FIG. 5B is a block diagram functionally illustrating the image processing circuit 80 provided in the digital processing circuit 103. The image processing circuit 80 generates a high-resolution image by performing the image processing method according to the present embodiment on the input image acquired by the image detection unit 101. More specifically, the image processing circuit 80 includes a position shift estimation unit 81 for performing a position shift estimation process described later, a super-resolution processing unit 82, and an input / output unit 83. The misregistration estimation unit 81 further includes a first estimation unit 81a for performing the first misregistration estimation process and a second estimation unit 81b for performing the second misregistration estimation process. The first estimation unit 81a includes a setting unit 81d for setting a search area to be described later. The super-resolution processing unit 82 includes a calculation unit 82a for calculating parameters used for the super-resolution processing based on the estimated positional deviation and the like.

The super-resolution processing unit 82 performs super-resolution processing described later on the input image. The super-resolution processing is processing for generating frequency information that exceeds the Nyquist frequency of the input image. At that time, the misregistration estimation unit 81 performs a first misregistration estimation processing step and a second misregistration estimation processing step, which will be described later, to estimate the misregistration from the reference image for each input image.

[Operation overview]
In the imaging apparatus 100, a high resolution image is obtained by performing super-resolution processing on a plurality of input images respectively obtained from different viewpoints obtained by photographing with the image detection unit 101 functioning as a camera array.

The imaging apparatus 100 has an environmental condition (temperature change or the like) for a large number of input images IG detected in a large number of conversion areas AR arranged in a matrix on the photoelectric conversion surface PC of the sensor array 60 shown in FIG. Super-resolution processing is performed in consideration of the displacement of the pixel position due to. At this time, the image capturing apparatus 100 uniformly expands to the periphery or uniformly to the center due to the temperature variation of the lens array stacked body 1 or the like with respect to the input image IG corresponding to the image projected on the sensor array 60. In consideration of not only the phenomenon of shrinkage but also the spatially uniform shift phenomenon having temperature dependence due to the holder 50, the lens array laminate 1, etc., the displacement of the pixel position is evaluated in advance with high accuracy. Perform super-resolution processing.

[Operation flow]
(Overall operation)
FIG. 7 is a flowchart showing an operation flow in the imaging apparatus 100 according to the present embodiment.
First, in the imaging apparatus 100, the 16 input images IG illustrated in FIG. 6 are acquired by executing the process of acquiring the input images IG by the 16 single-lens cameras constituting the image detection unit 101. (Step S11). Specifically, digital image signals corresponding to input images IG respectively obtained by photoelectric conversion in the 16 conversion areas AR of the sensor array 60 of FIG. 6 are input to the image processing circuit 80 of the digital processing circuit 103. Here, for example, a low resolution image of about 1000 × 750 pixels is input. Note that the image processing circuit 80 can perform pre-processing on the obtained digital image signal, thereby facilitating detection and correction of a displacement described later. This preprocessing includes smoothing processing such as a Gaussian filter, and an image for each color (RGB) can also be obtained by demosaic processing.

When the input image is acquired, the image processing mode is selected (step S10). Mode 1 is selected when high image quality is required, and mode 2 is selected when high speed is required. The mode selection may be switched in advance to match the image processing mode designated by the operation of the operator's operation key, or automatically switched according to the shooting mode such as still image shooting and moving image shooting. it can.

When mode 1 is selected, the process proceeds to step S12, and when mode 2 is selected, the process proceeds to step S16. In step S12, the first estimation unit 81a of the image processing circuit 80 executes a positional deviation approximate estimation process as a first positional deviation estimation processing step. In this processing step, the shift amount in pixel units (integer pixels) is estimated. Here, the positional deviation of the pixel for each input image IG due to the change in the position of each synthetic lens 1a constituting the lens array stack 1 due to a temperature change is estimated.

Next, on the basis of the shift amount in units of pixels, the second estimation unit 81b of the image processing circuit 80 executes a second positional shift estimation process step (step S13). Here, the shift amount in sub-pixel units (decimal pixels) is estimated.

Then, in the super-resolution processing unit 82 of the image processing circuit 80, super-resolution processing is executed in consideration of the positional deviation amount obtained in steps S12 and S13 (step S14), and a high-resolution image of about 4000 × 3000 pixels. Is generated as an output image OG.

(Position estimation process)
FIGS. 8 and 9 are diagrams for explaining a specific example of the misalignment rough estimation process as the first misalignment estimation process in step S12. Hereinafter, description will be made assuming that the environmental condition is temperature. FIG. 8 is a diagram for explaining the displacement of the pixel position due to the temperature change of the input image IG from each synthetic lens 1a. As an example, the input image IG before and after the temperature rises by 30 ° C. is schematically represented. It is a figure. FIG. 9 is a diagram for explaining the shift (including enlargement) of the input image IG focusing on one conversion area AR.

In the figure before the temperature rise in FIG. 8 (left figure), the 16 solid line rectangles represent the input images IG (A to P) on the sensor array 60, respectively. In the figure after the temperature rise in FIG. 8 (right figure), the 16 dotted line rectangles represent the input images A to P before the temperature rise, and the 16 solid line rectangles represent the input images A to P after the temperature rise. Yes. Reference signs A to P of the input image also identify the synthetic lens 1 a constituting the lens array stack 1. That is, the 16 synthetic lenses 1a constituting the lens array stack 1 may be represented by reference signs A to P.

As shown in FIG. 8, when the temperature rises, each synthetic lens 1a and the holder 50 expand, so that the shooting range of each input image changes as a whole. In the present embodiment, the imaging apparatus 100 uses the input image IG (K) from one synthetic lens 1a out of the four lens units arranged on the inner side as a reference for the positional deviation approximate estimation process in step S12. Used as an image. In this processing, the processing is executed on the assumption that all the input images IG (A to P) expand and contract as similar shapes and maintain a relatively congruent relationship during the above expansion and contraction.

For example, it is assumed that the expansion of the interval on the input image IG is 1/1000 times the temperature increase of 30 ° C. In this case, if the distance between the input image IG (K) as the reference image and the input image IG (A) farthest from the input image IG (A) is 4000 pixels in the oblique direction (X direction and Y direction), the input image IG (A) changes by four pixels in an oblique direction (X direction and Y direction) due to a temperature change of 30 ° C. That is, the relative position of the input image IG (A) with respect to the input image IG (K) changes within a range of 9 × 9 pixels in an environment where the standard temperature is ± 30 ° C.

In the above step S12, the remaining input image IG (A to J, L to P) on the basis of the pixel (reference pixel) CP represented by a small circle in FIG. 8 on the input image IG (K) as the reference image. For the corresponding reference pixel CP ′ above, the displacement due to expansion and contraction due to temperature is estimated in pixels. Here, the reference pixel CP on the input image IG (K) is determined based on the first position T1 and the second position T2. That is, using the first position T1, the second position T2, the first vector W1 from the first position T1 to the second position T2, and the linear expansion coefficient α2 of the lens array stack 1 (lens arrays 10 and 20), A reference pixel CP is determined by calculating a pixel that minimizes movement due to temperature change. When the reference pixel CP is small in movement due to temperature change, there is an effect that the margin width of the search area described later can be narrowed.

以上の表において、値ａ，ｂは、入力画像ＩＧのＸ方向のピッチをＰｘとし、Ｙ方向のピッチをＰｙとすると、伸縮率βを用いて、Ｐｘ×β×３０，Ｐｙ×β×３０に相当するものとなっている。 As shown in FIG. 9, among the input images IG (A to J, L to P) excluding the input image IG (K) that is the reference image, for example, in the input image IG (A), the position shift due to the temperature change occurs. The pixel CP ′ corresponding to the absence is specified as a coordinate position set in advance (dotted circle). When estimating a positional shift due to a temperature change of ± 30 ° C., a range A1 that covers the shift amount assumed in the input image IG (A) with a temperature change of ± 30 ° C. around the pixel CP. This is the minimum area for searching for a pixel corresponding to the reference pixel CP. Here, the range A1 that covers the shift amount assumed for the input image IG (A) is the pixel CP when the distance from the reference pixel CP to the pixel CP ′ is D and the expansion / contraction rate per unit temperature is β. It is obtained by expanding a certain margin width around a square defined by a diagonal line having a length of D × β × 60 (60 is a unit in degrees Celsius) centering on '. As shown in FIG. 8, the search area indicated by the range A1 is a set for each input image IG (A to J, L to P), and the input image IG (K) that is the reference image. It is set wider in the radial direction and longer in the radial direction. Specifically, the range (search area) A1 is set to an area obtained by adding a margin width to the values shown in Table 1 below.
[Table 1]

In the above table, the values a and b are Px × β × 30, Py × β × 30 using the expansion / contraction ratio β, where Px is the pitch in the X direction of the input image IG and Py is the pitch in the Y direction. It is equivalent to.

As for the search area (also referred to as the estimated presence area), the range A1 is an example, and is not limited to the range A1. In the input image IG (A to J, L to P), considering that the reference pixel CP ′ shifts along the diagonal line defining the range A1, the diagonal line is set rather than setting the entire range A1 as a search area. The search efficiency is better when the long and narrow area A2 along the corresponding line segment TL is set as the search area. That is, as shown in FIG. 9, a pixel CP1 that corresponds to the pixel CP ′ when the temperature decreases by 30 ° C. and a pixel CP2 that corresponds to the pixel CP ′ when the temperature increases by 30 ° C. are specified. Thus, a range A2 that connects the pixels CP1 and CP2 and is expanded by a certain margin width is set as a search area. FIG. 10 shows the search area range A2 for all input images IG (A to J, L to P) to be searched. In this case, the range A2 is inclined with respect to the axes X and Y of the input image IG.

For other input images IG (B to J, L to P), the range A1 that covers the shift amount assumed for the input images IG (B to J, L to P), more preferably ± A long and narrow area A2 widened by a certain margin width along the line segment TL connecting the upper limit and the lower limit of 30 ° C. is set as a search area.

The search area is based on the first position T1, the second position T2, the distance L between the first position T1 and the second position T2, and the linear expansion coefficient α2 of the lens array stacked body 1 (lens arrays 10 and 20). You may decide. In this case, since the above-described procedure is omitted, the processing becomes faster.

In the above description, it is assumed that the temperature of the image detection unit 101 cannot be specified. However, when the imaging apparatus 100 includes a temperature sensor that detects the temperature around the image detection unit 101, as illustrated in FIG. A range A3 close to a restricted square can also be used as a search area. The illustrated range A3 corresponds to a case where + 30 ° C. is raised from the standard temperature. That is, the range A3 moves within the range A2 corresponding to the temperature around the image detection unit 101.

The ranges A1, A2, and A3 as described above are based on the existence area of a single-eye image estimated by a relatively simple method, and pattern matching is performed using the ranges A1, A2, and A3 as search areas. Thus, the load on image processing can be made relatively small.

Note that the ranges A2, A3, etc. corresponding to the search area are margins that are assumed only from the temperature change in consideration of the deviation from the design value such as the actual distance between the synthetic lenses 1a in addition to the temperature change. It is possible to make the range wider than the width.

Within the search area, that is, the range A1, A2, A3 (preferably the range A2, A3), the template matching process is performed using the image including the reference pixel CP ′ of the input image IG (A), and the input image IG ( The pixel TP having the highest degree of coincidence with the reference pixel CP of K) is specified as a pixel at a position corresponding to the reference pixel CP. An example of the template matching process here is NCC (Normalized Cross Correlation). Other examples include SAD (Sum of Absolute Difference) and SSD (Sum of Squared Difference). For the other input images IG (B to J, L to P), the template matching process is performed in the ranges A1, A2, and A3 (preferably the ranges A2 and A3) that are similar search areas. In each input image IG (A to J, L to P), a pixel at a position corresponding to the reference pixel CP of the input image IG (K) is specified.

FIG. 12 corresponds to FIG. 9 and is a diagram for explaining a modification example such as setting a search area. In this case, although the reference pixel CP is on the input image IG (K), it is set at a position away from the point SP where there is no position shift due to temperature. As a result, the reference pixel CP is also shifted in a relatively wide range. However, when the imaging apparatus 100 includes a temperature sensor that detects the temperature around the image detection unit 101, the range A2 ′ depends on the ambient temperature of the image detection unit 101. A pixel CPV at a specific location in the center may be selected and used as a reference pixel CP for template matching processing.

(Position displacement estimation process 1)
Hereinafter, the positional deviation estimation processing 1 as the second positional deviation estimation processing step in step S13 of FIG. 7 will be described. In this positional deviation estimation process 1, positional deviation estimation is performed in units of sub-pixels in which the rough estimation in step S12 is improved more accurately.

FIGS. 13A to 13C are diagrams for explaining the positional deviation estimation processing 1 in step S13. As shown in FIG. 13A, a positional deviation in units of subpixels is estimated for one input image IG (A) after temperature change. In the specified range A4 of the range of, for example, 3 × 3 pixels centered on the pixel TP corresponding to the reference pixel CP ′ in the input image IG (A), identified by the first positional deviation estimation processing step in step S12, Quadratic curved surface fitting is performed based on the matching degree (for example, NCC value) for each pixel in template matching (see FIG. 13B). In the misregistration estimation process 1 in step S13, as an example, the paper “Robust Super-Resolution Processing Based on Pixel Selection Considering Misregistration Amount” (Journal of the Institute of Electronics, Information and Communication Engineers, Vol. J92-D, No. 5, pp. 650-660, 2009, May 2009) can be employed. That is, as shown in FIGS. 13B and 13C, the coordinates of the pixel having the highest matching degree among the matching degrees for each pixel are specified as the shift amount. Since pattern matching is performed after setting the search area in this way, processing can be performed at a higher speed than when pattern matching is performed on all images.

The above has described the processing for performing accurate positional deviation estimation for the input image IG (A) of the reference image. However, for the other input images IG (B to J, L to P), the first step of the above step S12. By performing the same processing as in the case of the input image IG (A) with reference to the pixel TP specified in the positional deviation estimation processing step, it is possible to perform precise positional deviation estimation.

In step S13, other methods such as fitting to a quadratic curve in each of the X coordinate and the Y coordinate may be employed instead of the quadratic curved surface fitting described above.

(Position displacement estimation process 2)
When mode 2 is selected in step S10, in step S16, a positional deviation estimation process 2 that is a simple positional deviation estimation process is executed. In this step, the displacement is simply estimated based on the vector W1 based on the difference between the first position T1 and the second position T2 and the vector W2 based on the isotropic deformation of the lens array 10. is there. Specifically, a temperature sensor is provided in the imaging device, and after correcting both vectors W1 and W2 by the measured environmental temperature, a shift amount for each pixel is calculated from the combined vector of both, and the calculation result is Presumed to indicate misalignment. Alternatively, the temperature change is estimated from a plurality of distant pixels of the image data, both vectors W1 and W2 are corrected based on the change, and the corrected vectors are combined to estimate the positional deviation. According to this method, since the accurate misregistration detection by the search area setting and pattern matching as described above is not performed, misregistration estimation can be performed at high speed. That is, it is possible to grasp the positional deviation of the single-eye image accompanying the change in the environmental conditions by relatively simple calculation, and it is possible to perform good image processing quickly. In calculating the vector W1, the first position T1 and the second position T2 may be calculated from the holder shape and the lens array shape at the reference temperature, or the temperature after the lens array 10 is bonded to the holder 50. The first position T1 and the second position T2 may be measured by moving up and down. Although the vector W2 differs depending on the position, the vector W2 can be easily calculated as a function of the position with respect to the second position T2 by regarding the isotropic deformation centered on the second position T2.

(Super-resolution processing)
FIG. 14 is a diagram showing the flow of super-resolution processing in step S14. In FIG. 14, as a specific example, the case of performing the processing described in the paper “Fast and Robust Multiframe Super Resolution” (IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 10, OCTOBER 2004 page.1327-1344) is shown. The flow of resolution processing is shown.

As shown in FIG. 14, in step S31, one of the input images is subjected to an interpolation process such as a bilinear method to convert the resolution of the input image to a high resolution that is a resolution after the super-resolution process. Thus, an output candidate image as an initial image is generated.

In step S32, the amount of BTV (Bilateral Total Variation) for robust convergence to noise is calculated.

In step S33, the generated output candidate images are compared with the 16 input images, and a residual is calculated. That is, in step S33, the generated output candidate image is converted into an input image size based on each input image and its deterioration information (information indicating the relationship between the image after super-resolution and the input image). The resolution is reduced (see step S41 in FIG. 15), and the difference from the 16 input images is calculated and recorded (see step S42 in FIG. 15). Then, the difference is returned to the size after the super-resolution processing (see step S43 in FIG. 15), and is set as a residual.

In step S34, the residual output calculated from the output candidate image generated in step S31 and the BTV amount are reduced to generate the next output candidate image.

The processing in steps S31 to S34 is repeated until the output candidate image converges, and the converged output candidate image is output as the output image after the super-resolution processing.

The repetition may be a predetermined number of times such as the number of times of convergence (for example, 200 times) approximately enough, or a convergence determination may be made for each series of processing, and may be repeated according to the result. .

FIG. 15 is a diagram for explaining the deterioration information used in step S33. Deterioration information refers to information representing the relationship of each input image with respect to a high-resolution image after super-resolution processing, and is represented, for example, in a matrix format. The deterioration information includes a shift amount, a downsampling amount, a blur amount, and the like of each of the input images estimated in step S13.

Referring to FIG. 15, the degradation information is defined by a matrix indicating the conversion when each of the input image and the high-resolution image after super-resolution processing is expressed as a one-dimensional vector. The imaging apparatus 100 calculates a parameter used for the super-resolution processing based on the positional deviation estimated in the estimation processing in steps S12 and S13, and incorporates it as deterioration information.

FIG. 16 is a diagram showing a specific example of deterioration information. When the amount of pixel shift is estimated to be 0.25 pixels and the amount of downsampling is 1/4 in each of the vertical direction and the horizontal direction, the imaging apparatus 100 displays the deterioration information as illustrated in FIG. Stipulate. When one location corresponding to one pixel in the input image corresponds to 16 pixels in 16 locations in the high-resolution image after super-resolution processing, the degradation information includes 1 in 16 locations. A coefficient of / 16 is listed. Therefore, when the pixel shift amount is 0.25 pixel, each 16 pixels of the high resolution image contributes by 1/16 of the shift amount.

Note that the super-resolution processing in step S14 is not limited to the processing shown in FIG. 14, but is reconfigurable super-resolution processing that generates one image from a plurality of input images. Other processes may be employed if any. The constraint term calculated in step S32 may be replaced with another value such as a 4-neighbor Laplacian instead of the BTV amount.

According to the compound eye imaging device described above, the image processing circuit 80 at least has information on the difference between the first position T1 and the second position T2 and the environmental conditions of the lens array stack 1 (lens arrays 10 and 20). Image processing is executed based on information indicating the degree of deformation accompanying the change. For example, a change in the position of each eye image on the sensor array 60 caused by a change in environmental conditions such as a change in environmental temperature, that is, a change in the position of the image obtained corresponding to the optical image on the sensor array 60 is corrected. To do. Accordingly, even when the deformation centers of the lens arrays 10 and 20 and the holder 50 when viewed from the direction of the optical axis OA of the lens portions 10a and 20a do not coincide with each other, the positional shift of each individual eye image on the sensor array 60 is quickly performed. In addition, it can be accurately grasped, and good image processing can be performed against changes in environmental conditions.

In the above, it is assumed that the sensor array 60 and the substrate SB hardly expand and contract. However, although the sensor array 60 and the substrate SB are less than the holder 50, there is a slight expansion and contraction due to the temperature. The calculation may be performed.

In the above embodiment, the case where the image forming position is displaced along the imaging surface due to the deformation of the holder 50, the lens array stack 1, or the first lens array 10 has been described. Along with this, there is a possibility of slight displacement (defocus) in the direction of the optical axis OA. In this case, processing that compensates for local defocusing that occurs with environmental temperature is also possible. Specifically, blur degree estimation processing is performed during the second misregistration estimation processing step shown in FIGS. 13A to 13C, and the degree of blur from the reference image is estimated for each input image. That is, in the modified step S13, the degree of blur is also estimated when the shift amount in sub-pixel units (decimal pixels) is estimated. In step S14, the super-resolution processing is executed in consideration of the positional deviation amount and the degree of blur.

In the above embodiment, the temperature deformation of the holder 50, the lens array stack 1 and the like is linear. However, there is a possibility that a nonlinear component may be caused due to the fixing condition between these members and the shape of the holder 50. is there. In this case, the direction of contraction can be calculated in consideration of such a non-linear component. In this case, for example, it is possible to perform image processing assuming that the range A2 of the search area extends in a curved manner.

As mentioned above, although the compound eye imaging device concerning an embodiment was explained, the compound eye imaging device concerning the present invention is not restricted to the thing of the above illustration. For example, the arrangement of the lens portions 10a and 20a constituting the lens arrays 10 and 20 and the shape of the optical surfaces 11a and 11b are appropriately changed according to the use or specification of the lens array stack 1 or the imaging device 100. be able to. For example, the lens units 10a and 20a are not limited to being arranged at 4 × 4 lattice points, but can be arranged at lattice points of 3 × 3, 5 × 5, and the like. The lattice is not limited to a square lattice, and may be a rectangular lattice.

In each of the above embodiments, the environmental condition is described as the environmental temperature. However, the environmental condition is not limited to this, and the humidity of the surrounding environment, the concentration of a specific component in the surrounding environment, or a combination of a plurality of these elements. May be. You may make it provide a humidity sensor, a gas sensor, etc. as needed.

In the above embodiment, two lens arrays are stacked, but a single-layer lens array may be used instead of stacking them. Three or more lens arrays may be stacked.

Although the adherend portion QB to the holder 50 of the lens array laminate 1 is substantially square in plan view, it may be rectangular or circular.

In the above embodiment, the lens arrays 10 and 20 are made of resin, but may be made of glass. The resin is not limited to a thermoplastic resin, and a thermosetting resin, a photocurable resin, or the like can be used. The lens arrays 10 and 20 may be, for example, wafer level lenses in which resin lens portions are two-dimensionally arranged on a glass substrate.

In the above embodiment, the holder 50 is fixed to the substrate SB, but may be fixed to the sensor array 60.

In the above embodiment, the support reference point OS of the lens arrays 10 and 20 is made to coincide with the central axis AX. However, the support reference point OS can be positively shifted from the central axis AX.

Claims (20)

A compound eye optical system including a lens array having a plurality of lens portions arranged two-dimensionally in a direction perpendicular to the optical axis;
A plurality of optical images corresponding to the plurality of lens units are formed by the compound eye optical system, and a sensor array that outputs image signals corresponding to the plurality of optical images;
A holder for holding the compound eye optical system so as to face the sensor array;
An image processing circuit for processing an image signal output from the sensor array;
With
In a reference environmental condition, a first position that is a center position of deformation associated with a change in the environmental condition of the holder when viewed from the optical axis direction, and an environmental condition of the lens array when viewed from the optical axis direction The lens array is fixed to the holder in a state in which the second position which is the center position of the deformation accompanying the change is separated from the second position,
The image processing circuit performs image processing based on at least information regarding a difference between the first position and the second position and information indicating a degree of deformation accompanying a change in environmental conditions of the lens array. . The compound eye imaging device according to claim 1, wherein the lens array is adhered to the holder by an adhesive. The compound-eye imaging device according to claim 2, wherein the adherend portion of the lens array is provided at a point-symmetrical position with respect to a center of a lens portion arrangement region where the plurality of lens portions are provided in the lens array. The compound eye imaging apparatus according to claim 3, wherein the adherend portion of the lens array is continuous so as to surround a plurality of lens portions. The compound eye imaging device according to any one of claims 1 to 4, wherein the holder includes a ceiling portion and a leg portion that supports the ceiling portion. The compound eye imaging apparatus according to claim 5, wherein the lens array is fixed to the ceiling portion of the holder. The compound eye imaging device according to claim 6, wherein an object side surface of the lens array is fixed to an image side surface of the ceiling portion of the holder. The holder is disposed so as to have a deformation center whose relative position is substantially unchanged with respect to the sensor array when a shape change in a direction perpendicular to the optical axis is caused in accordance with a change in environmental conditions. The compound eye imaging device according to any one of claims 1 to 7. Comprising a substrate provided with the sensor array;
The compound eye imaging apparatus according to any one of claims 1 to 8, wherein the holder is fixed to the substrate. Comprising a substrate provided with the sensor array;
The compound eye imaging apparatus according to claim 5, wherein a leg portion of the holder is fixed to the substrate. The compound eye imaging device according to any one of claims 1 to 10, wherein the lens array is formed of a resin. The compound eye imaging device according to any one of claims 1 to 11, wherein the holder is formed of a resin. The linear expansion coefficient α1 of the holder, the distance L between the first position and the second position, and the pixel pitch P of the sensor array are:
30L ・ α1 <5P
The compound eye imaging device according to any one of claims 1 to 12, which satisfies the relationship: The linear expansion coefficient α1 of the holder, the distance L between the first position and the second position, and the pixel pitch P of the sensor array are:
30L ・ α1 <2P
The compound eye imaging device according to any one of claims 1 to 12, which satisfies the relationship: The linear expansion coefficient α1 of the holder and the linear expansion coefficient α2 of the lens array are:
1.2> α2 / α1> 0.8
The compound eye imaging device according to claim 1, wherein the compound eye imaging device satisfies the relationship: The linear expansion coefficient α2 of the lens array and the linear expansion coefficient α3 of the sensor array are:
α2 / α3> 5
The compound eye imaging device according to claim 1, wherein the compound eye imaging device satisfies the relationship: The image processing circuit performs image processing using a combined vector of a first vector indicating movement from the first position to the second position and a second vector indicating deformation for each position in the lens array. The compound eye imaging device according to any one of claims 1 to 16. The image processing circuit is configured to display each single-eye image corresponding to an environmental condition based on information on a difference between the first position and the second position and information indicating a degree of deformation accompanying a change in the environmental condition of the lens array. The present invention estimates an existing area on the sensor array and executes pattern matching of an image signal corresponding to the estimated existing area to obtain a correction amount of a position change of each eye image accompanying a change in environmental conditions. The compound eye imaging device according to any one of 1 to 17. The compound eye imaging device according to any one of claims 1 to 18, wherein the reference environmental condition is to set an environmental temperature to room temperature. 20. The compound eye optical system according to claim 1, wherein the compound eye optical system further includes a second lens array, the first lens array being the first lens array, and the second lens array being disposed so as to overlap the optical axis direction of the first lens array. A compound eye imaging device according to claim 1.

Images