JP2008294741A - Imaging system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging system capable of generating an arbitrary high-resolution focal image from an arbitrary focal image at a virtual image surface position of a wide range. <P>SOLUTION: The imaging system is provided with: a photographing means having an imaging optical system 1, a micro lens array 2 for separating light transmitting areas of different openings of the imaging optical system 1 and emitting the light, and a photoelectric converter 3 for converting an image formed by the micro lens array 2 into an electrical signal to generate photographed data; an arbitrary focal image configurating part 8 capable of reconfigurating an arbitrary focal image being an image at an arbitrary image surface position within a prescribed range; a photographing control means 5 for configurating a plurality of arbitrary focal images on the basis of photographed data generated by photographing of a plurality of times; and a high resolution processing part 9 for using the plurality of arbitrary focal images to generate an arbitrary focal image in high resolution. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging system, and more particularly to an imaging system capable of obtaining arbitrary-focus images with different image plane positions with high resolution.

In recent years, a light field camera has been developed as a camera that can reconstruct an image at a different image plane position designated by a user from an image shot at a certain image plane position (see Non-Patent Document 1, for example). ). This light field camera is a camera using a technique called light field photography (light field photo rendering), and can obtain an image (arbitrarily focused image) at an arbitrary focus position substantially within a predetermined range by one shooting. It has the feature of being able to.

In light field photography, by placing a microlens array between the imaging optical system and the photoelectric conversion element, light that passes through different areas of the aperture of the imaging optical system and converges on each microlens of the microlens array, It is possible to separately form an image on different pixels of the photoelectric conversion element and reconstruct an arbitrary focus image at the image plane position set by the user from the obtained imaging data. Light incident on one microlens is separated and imaged on different pixels of the photoelectric conversion element downstream of the microlens for each incident direction (for each aperture region of the imaging optical system that passes through). Also, the range of image plane positions (hereinafter referred to as virtual image plane positions) that can be set by the user becomes wider as the number of pixels of the photoelectric conversion element for one microlens increases.
Ren Ng, 5 others, "Light Field Photograph with a Hand-held Plenoptic Camera", Stanford Tech Report CTSR 2005-02, p.1-11

  However, in the above light field camera (see, for example, Non-Patent Document 1), the number of pixels (resolution) of the arbitrarily-focused image to be generated is the same as the number of microlenses in the microlens array, compared with the number of pixels of the photoelectric conversion element. As a result, the number of pixels is greatly reduced.

  In addition, if the number of microlenses is increased in order to maintain the number of pixels (resolution) of the generated arbitrary focus image, the number of pixels of the photoelectric conversion element for one microlens is reduced, so that the range of virtual image plane positions that can be set is There was a problem of becoming narrow.

  The present invention has been made in view of the above-described problems, and provides an imaging system capable of generating a high-resolution arbitrary focus image from an arbitrary focus image in a wide range of virtual image plane positions. Objective.

  The imaging system according to the present invention separates and emits an imaging optical system through which light from an object field passes and light that has passed through the imaging optical system, and that has passed through different areas of the aperture of the imaging optical system. An imaging device having a separating optical system, and a photoelectric conversion element that receives the light separated and emitted by the separating optical system and converts an image formed by the separating optical system into an electric signal to generate photographing data; An arbitrary-focus image forming unit capable of reconstructing an arbitrary-focus image that is an image at an arbitrary image plane position within a predetermined range based on the shooting data generated by the photoelectric conversion element; Imaging control means that causes the arbitrary-focus image forming means to construct a plurality of arbitrary-focus images based on the shooting data generated by the multiple-time shooting, and a plurality of arbitrary-focus images. focus It is characterized in that and a high-resolution processing means for generating a high high-resolution arbitrary focus image resolution than the image.

  In the imaging system according to the present invention, it is possible to generate a high resolution arbitrary focus image obtained by increasing the resolution of an arbitrary focus image in a wide range of virtual image plane positions.

(Embodiment 1)
Hereinafter, an imaging system according to Embodiment 1 of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of an imaging system according to Embodiment 1 of the present invention. FIG. 1 shows a light field camera that captures a still image as an example of an imaging system. Note that the configuration of the imaging system is not limited to that shown in FIG. 1, and other components can be added or unnecessary components can be omitted as necessary.

  As shown in FIG. 1, the imaging system according to the present embodiment includes an imaging optical system 1, a microlens array 2, a photoelectric conversion element 3, a drive circuit 4, an imaging control unit 5, a memory 6, and a user I / F (interface) unit. 7, an arbitrary focus image configuration unit 8, a resolution enhancement processing unit 9, an inter-image change amount detection unit 10, and an image selection unit 11. In this embodiment, it is assumed that these components are mounted on one imaging device (camera or the like), and that an image is captured by the imaging system in a handheld state.

  The imaging optical system 1 is composed of, for example, a plurality of lens groups, and light from the object field that has passed through the imaging optical system 1 forms an image near the focal plane of the imaging optical system 1. A microlens array 2 as a separation optical system is disposed in the vicinity of the focal plane of the imaging optical system 1, and light that has passed through the imaging optical system 1 enters the microlens array 2. As will be described later, the microlens array 2 separates and emits light that has passed through different areas of the aperture of the imaging optical system 1. A photoelectric conversion element 3 is arranged behind the microlens array 2, and light emitted after being separated by the microlens array 2 is incident thereon. The photoelectric conversion element 3 uses, for example, a CCD or a CMOS, and is arranged in the vicinity of the focal length of the microlens 20 constituting the microlens array 2. In the present embodiment, it is assumed that the imaging optical system 1, the microlens array 2, and the photoelectric conversion element 3 are arranged along the optical axis A.

  The light that has passed through the microlens array 2 forms an image in the photoelectric conversion element 3, and the photoelectric conversion element 3 converts this image into an electrical signal to generate, for example, analog-format photographing data. Then, the drive circuit 4 reads out the shooting data generated by the photoelectric conversion element 3 as, for example, digital shooting data. At this time, analog image data is converted into digital image data, for example, by an A / D converter (not shown) provided in the drive circuit 4.

  The imaging control unit 5 operates the driving circuit 4 in accordance with the number of times of imaging (described later) input from the user via the user I / F unit 7, and the imaging data generated by a plurality of imagings is arbitrarily focused from the driving circuit 4. A series of operations to be sent to the image construction unit 8 or the memory 6 is repeated. In addition, the imaging control unit 5 repeatedly operates the arbitrary focus image forming unit 8 according to the number of times of shooting. The arbitrary focus image construction unit 8 receives shooting data from the photoelectric conversion element 3 or the memory 6 and forms an arbitrary focus image that is an image at an arbitrary image plane position within a predetermined range based on the shooting data. The arbitrary focus image is an image at an arbitrary image plane position within a predetermined range, but is substantially the same as an image obtained by freely changing the focus within the predetermined range. In the present embodiment, the arbitrary focus image forming unit 8 can reconstruct an arbitrary focus image at an arbitrary image plane position within a predetermined range from shooting data generated by one shooting.

  The arbitrary focus image formed by the arbitrary focus image construction unit 8 is stored in the memory 6 as image data. The arbitrarily-focused image stored in the memory 6 is sent to the resolution enhancement processing unit 9 via the inter-image change amount detection unit 10 and the image selection unit 11. The high-resolution processing unit 9 includes an image displacement amount estimation unit 9a and a super-resolution processing unit 9b, and uses a plurality of arbitrary focus images formed by the arbitrary focus image configuration unit 8 to perform arbitrary focus image configuration. A high-resolution arbitrary-focus image having a higher resolution than the arbitrary-focus image formed by the unit 8 is generated.

  The inter-image variation detection unit 10 detects the difference or similarity between a plurality of arbitrary focus images as an SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), a cross-correlation coefficient, or the like. The image selection unit 11 selects an arbitrary focus image used for generation of a high resolution arbitrary focus image in the high resolution processing unit 9 based on the degree of difference or similarity detected by the inter-image change amount detection unit 10. At this time, the inter-image change amount detection unit 10 detects, for example, the degree of difference between the reference arbitrary-focus image and another arbitrary-focus image as the amount of change in the scene, and the image selection unit 11 has the difference degree equal to or less than a predetermined threshold. Are transferred to the high resolution processing unit 9. Note that the inter-image change amount detection unit 10 and the image selection unit 11 may not be provided. For example, when the imaging target of the configured imaging system is limited to a stationary object, the inter-image change amount detection unit 10 and the image selection unit 11 may not be provided.

  FIG. 2 is an enlarged view showing a part of the microlens array 2 and the photoelectric conversion element 3. Here, a method for constructing an arbitrary focus image will be described. Each of the microlenses 20 constituting the microlens array 2 corresponds to the pixel array 30 of the photoelectric conversion element 3, and the light flux passing through one microlens 20 forms an image on the corresponding pixel array 30. The number of pixels 31 of the pixel array 30 corresponding to each microlens 20 is determined according to the range of the image plane position to be set and the resolution of the arbitrary focus image configured by the arbitrary focus image configuration unit 8. In the present embodiment, the pixel array 30 is an array of 5 rows and 5 columns, and one pixel array 30 includes 25 pixels 31.

  FIG. 3 is a diagram of a state in which a light beam that has entered the imaging optical system 1 and has passed through one microlens 20 forms an image on the photoelectric conversion element 3 as viewed from the direction perpendicular to the optical axis A. As shown in FIG. 3, the light that has passed through one microlens 20 of the microlens array 2 forms an image on the corresponding pixel array 30 at the rear.

  FIG. 4 is a diagram of the aperture of the imaging optical system 1 and the pixel array 30 as viewed from the direction of the optical axis A. 4A is a view of the aperture of the imaging optical system 1 viewed from the direction of the optical axis A, and FIG. 4B is a view of one pixel array 30 viewed from the direction of the optical axis A. . As shown in FIG. 4A, when the aperture of the imaging optical system 1 is divided into the same number of lattice-shaped divided apertures as the number of pixels 31 in one pixel array 30, one pixel 31 in the pixel array 30 includes Only incident light from one split aperture is accumulated, and the correspondence between each pixel (a1-e5) and split aperture (A1-E1) shown in FIG. 4B is point-symmetric when viewed from the optical axis A direction. Become. For example, the light emitted from the divided opening E5 enters the pixel e5 of any pixel array 30.

  Accordingly, when pixel data at a specific pixel position (for example, only a1 in FIG. 4B) is extracted from all the pixel arrays 30 on the photoelectric conversion element 3 in FIG. 1 and arranged, a specific divided aperture (here, , All the light beams emitted from the divided aperture A1) form an image (divided aperture viewpoint image) formed on the photoelectric conversion element 3. In the present embodiment, the incident light on the photoelectric conversion element 3 slightly changes for each shooting due to the influence of camera shake or the like of the imaging system. Therefore, the shooting data generated by a plurality of shootings has a difference for each divided aperture viewpoint image as a data difference.

  FIG. 5 is a view of the incident state of light on the microlens array 2 and the photoelectric conversion element 3 as viewed from the direction perpendicular to the optical axis A. FIG. Note that Z in FIG. 5 indicates a virtual image plane having the same resolution as the microlens array. In FIG. 5, pixels for one column (here, a1, b1, c1, d1, e1) are shown as a, b, c, d, and e as the pixels 31 of the pixel array 30.

  A light beam L shown in FIG. 5 is emitted from the divided aperture A1 of the imaging optical system 1 of FIG. 1 and passes through each pixel on the virtual image plane Z, and is equal to or less than the microlens width Dm on the microlens array 2. With the spread of The light beam that has passed through the microlens 20 forms an image on the pixel a of any pixel array 30. Similarly, the light beams emitted from the divided openings B1, C1, D1, and E1 form images on the pixels b, c, d, and e of any one of the pixel arrays 30, respectively.

  The arbitrary focus image construction unit 8 is emitted from all the divided apertures (25 in the present embodiment) of the imaging optical system 1 and obtains the amount of light passing through each pixel on the virtual image plane set by the user. The image on the image plane can be reconstructed.

  FIG. 6 is a diagram of when the luminous flux that passes through a certain pixel on the virtual image plane is emitted from which divided aperture of the imaging optical system 1 and enters which microlens is viewed from the direction perpendicular to the optical axis. Here, the pixel position on the virtual image plane Z is coordinate (x, y), the position of the divided aperture of the imaging optical system 1 is coordinate (u, v), and the position of the micro lens 20 on the micro lens array 2 is coordinate ( x ′, y ′), the distance from the aperture of the imaging optical system 1 to the microlens array 2 is F, and the distance from the aperture of the imaging optical system 1 to the virtual image plane Z is αF. In FIG. 6, only the x, u, and x ′ directions representing the vertical direction are shown. As shown in FIG. 6, the light fluxes that have passed through the coordinates (u, v) and the coordinates (x, y) are (u + (x−u) / α, v + (yu) / α) of the microlens array 2. (X ′, y ′) = (u + (x−u) / α, v + (y−u) / α).

  For this reason, if the pixel value of the pixel 31 of the photoelectric conversion element 3 is L (x ′, y ′, u, v), the pixel value E (x, y) of each pixel on the virtual image plane Z is expressed by the following formula ( It is obtained in 1).

  In Expression (1), α is given by the user. Therefore, if (x, y) and (u, v) are given, the position (x ′, y ′) of the microlens 20 on which the light beam enters can be known. Then, the pixel value at the position (u, v) from the pixel array 30 corresponding to the microlens 20 (for example, a1 in FIG. 4B if the divided aperture is A1 in FIG. 4A). L (x ′, y ′, u, v) is extracted. This is performed for all the divided apertures, and the pixel values E (x, y) of each pixel on the virtual image plane Z are obtained by summing (integrating) the extracted pixel values. The integration in FIG. 1 can be performed by simple addition using u and v as the representative coordinates of the divided aperture of the imaging optical system 1.

  FIG. 7 shows a case where a light beam L1 emitted from one divided aperture of the imaging optical system 1 and passing through one pixel on the virtual image plane Z is incident on two microlenses 20 in the direction perpendicular to the optical axis A. It is the figure seen from. In this case, the light beam L1 is dispersed and imaged on the pixels 1a and 2a of the two pixel arrays 30 corresponding to the microlenses ML1 and ML2. Accordingly, the pixel value L1 (x ′, y ′, u, v) by the light beam L1 is obtained by interpolation of the pixel values L1a and L2a of the pixels 1a and 2a. In the present embodiment, the pixel value L1 (x ′, y ′, u, v) is expressed by the following formula (2) using the image lengths d1 and d2 of the light beam L1 on the microlenses ML1 and ML2. It is determined by linear interpolation. Note that the image lengths d1 and d2 are experimentally obtained from the shooting parameters of the camera, the position of the virtual image plane, and the like.

  FIG. 8 is a diagram illustrating a setting range of the virtual image plane Z in which an arbitrary focus image can be accurately reconstructed in the imaging system according to the present embodiment. When the light beam L emitted from a divided aperture of the imaging optical system 1 and passing through a certain pixel on the virtual image plane Z has a spread equal to or less than the microlens width Dm on the microlens array, the expression (1) The pixel value on the virtual image plane Z can be obtained with high accuracy using. Focusing on the front side and the rear side of the microlens 20 respectively, and assuming that conjugate planes Z1 and Z2 on which the light beams L1 and L2 having a diameter Dm spread on the microlens are the same width are Z1 to Z2, the accuracy is This is a setting range of the virtual image plane Z that can reconstruct an arbitrarily focused image well. When the pixel array 30 corresponding to one microlens 20 is composed of pixels 31 of n rows and n columns, the virtual image plane settable range φ1 before and after the microlens array 2 is expressed by Expression (3). F0 in Expression (3) indicates the F value of the imaging optical system. In this embodiment, n = 5.

  FIG. 9 is a diagram showing a setting range of the virtual image plane Z that can further accurately reconstruct an arbitrary focus image. In FIG. 9, attention is paid only to light rays that are completely imaged on the virtual image plane Z, and the virtual image plane setting range is reduced to φ2 shown in Expression (4).

  The user can designate the position of an arbitrary virtual image plane Z from the virtual image plane setting range. Further, the position of the virtual image plane Z may be automatically set by the arbitrary focus image construction unit 8 using camera parameters or the like.

  FIG. 10 is a flowchart showing processing until a plurality of arbitrary focus images are formed. Here, a flow until a plurality of times of photographing are performed under the control of the imaging control unit 5 and a plurality of arbitrary focus images are formed will be described.

  First, the shooting number N is input from the user via the user I / F unit 7 (S101). At this time, in addition to the shooting number N, camera parameters at the time of shooting may be input. Next, the imaging control unit 5 causes the imaging unit including the imaging optical system 1, the microlens array 2, the photoelectric conversion element 3, etc. to perform one imaging via the drive circuit 4, and the imaging output from the photoelectric conversion element 3. Data is stored in the memory 6 (S102). Then, it is determined whether or not the number of times of photographing N input in S101 has been completed (S103), and if the photographing has not been completed, the single photographing in S102 and the saving of the photographing data are repeated. At this time, for example, shooting is continuously performed for 2 seconds, and shooting data for 60 times (when the imaging system is 30 fps) is stored.

  When N times of shooting are completed, for example, the user sets the virtual image plane position via the user I / F unit 7 (S104). The user may set a virtual image plane position every time an arbitrary focus image is configured, or may cause an arbitrary focus image configuration to be performed for a plurality of preset virtual image plane positions. As virtual ellipse positions to be set in advance, for example, positions arranged at equal intervals within a virtual image plane setting range can be considered. Then, the arbitrary focus image constructing unit 8 reads one shooting data out of the N shooting data stored in the memory 6 (S105). Then, the arbitrary focus image construction unit 8 composes one arbitrary focus image using the photographing data read in S105 and the virtual image plane position set in S104 (S106). Next, it is determined whether or not reading of shooting data for N times stored in the memory 6 has been completed (S107), and if not completed, another shooting data is read from the memory 6. When reading of the shooting data for N times is completed, the user selects whether or not to reconstruct an arbitrarily focused image at a different virtual image plane position via the user I / F unit 7 or the like (S108). When reconstructing the focus image, the processes from S104 to S108 are repeated. If the arbitrary focus image is not reconstructed, the process is terminated. In the present embodiment, an arbitrary focus image is configured for all the captured image data for N times, but an arbitrary focus image may be configured for only a part of the captured data.

  In this way, an arbitrary focus image corresponding to the number of shots can be generated for each virtual image plane position. When arbitrary focus images are compared for each photographic data at a certain virtual image plane position, the shift for each divided aperture viewpoint image is synthesized and becomes the shift of the entire arbitrary focus image.

  A plurality of arbitrary focus images formed by the processing shown in FIG. 10 are temporarily stored in the memory 6 as image data, and then used in the high resolution processing unit 9 to generate a high resolution arbitrary focus image. As described above, a plurality of arbitrary focus images are transferred from the memory 6 to the high resolution processing unit 9 via the inter-image change amount detection unit 10 and the image selection unit 11. The inter-image change amount detection unit 10 detects a difference or similarity between arbitrary focus images as a scene change amount, and the image selection unit 11 increases an arbitrary focus image whose scene change amount is a predetermined threshold value or less. Transfer to the resolution processing unit 9.

  Here, the high-resolution processing of the arbitrary focus image performed by the high-resolution processing unit 9 will be described. In the present embodiment, a plurality of arbitrary focus images configured at the same virtual image plane position are used in the resolution enhancement processing. For example, a plurality of arbitrary focus images having slightly different virtual image plane positions are used. A high resolution arbitrary focus image may be generated. In the high resolution processing unit 9, after the image displacement amount estimation unit 9a estimates the image displacement amount (corresponding to the pixel position) between the arbitrary focus images, super-resolution is performed using the pixel displacement amount and the image data of the arbitrary focus image. The processing unit 9b generates a high resolution arbitrary focus image.

  Note that the resolution enhancement processing means 9 selects one reference image from a plurality of arbitrary focus images formed by the processing of FIG. 10, and performs the resolution enhancement processing on the reference image. At this time, the resolution enhancement process may be performed on the entire reference image. For example, the user may select a partial area for which the resolution enhancement process is to be performed, and the resolution enhancement process may be performed only on the partial area. You may make it perform. In this case, the image data of the partial region of the reference image is transferred to the image displacement amount estimation unit 9a so that the image displacement amount can be estimated, and the other arbitrary focal image has an area larger than the partial region of the reference image. Transfer image data. For example, the partial area of the selected reference image is set as one block, and the image data for nine blocks including the partial area and its periphery is transferred for the other arbitrary focus images.

  FIG. 11 is a flowchart showing an algorithm of an image displacement amount estimation process performed by the image displacement amount estimation unit 9a. Hereinafter, the image displacement amount estimation processing performed by the image displacement amount estimation unit 9a will be described in accordance with the algorithm flow shown in FIG. First, one arbitrary focus image serving as a reference for image displacement amount estimation is read as a reference image (S201). In this embodiment, an arbitrary focus image composed of shooting data generated in the first shooting immediately after a shutter button (not shown) of the imaging system is fully pressed is used as a reference image. It may be an arbitrary focus image. Next, the reference image is deformed with a plurality of image displacement parameters to generate an image sequence (S202). At this time, for example, an image sequence is generated by translating and rotating the reference image within a predetermined range. Then, one image is read from the arbitrary focus image formed by the processing of FIG. 10 as a reference image for estimating the image displacement amount with respect to the standard image (S203). Then, a rough pixel position is associated between the reference image and the reference image using a pixel matching method such as a region-based matching method (S204). Thereby, the image displacement amount between the standard image and the reference image is obtained at the pixel level.

  Next, a similarity value between the image sequence generated by deforming the base image in S202 and the reference image is calculated (S205). This similarity can be obtained as a difference between an image sequence such as SSD (Sum of Squared Difference) and SAD (Sum of Absolute Difference) and a reference image. Then, a discrete similarity map is created using the relationship between the image displacement parameter when the image sequence is generated in S202 and the similarity value calculated in S205 (S206). Then, the discrete similarity map created in S206 is complemented to obtain a continuous similarity curve, and extreme values of similarity values are searched for in this continuous similarity curve (S207). As a method for obtaining a continuous similarity curve by complementing a discrete similarity map, for example, there is a parabolic fitting or a spline interpolation method. An image displacement parameter when the similarity value becomes an extreme value in the continuous similarity curve is estimated as an image displacement amount between the reference image and the reference image.

  Thereafter, it is determined whether image displacement amount estimation has been performed for all arbitrary focus images used for generation of high resolution arbitrary focus images in the super-resolution processing unit 9b (S208), and image displacement amounts for all arbitrary focus images. If the estimation has not been performed, the process from S203 to S208 is repeated using another arbitrary focus image as the next reference image among the arbitrary focus images formed by the process of FIG. 10 (S209). In S208, when it is determined that the image displacement amount estimation has been performed for all the images used in the super-resolution processing unit 9b, the process ends.

  FIG. 12 is a diagram illustrating an example of a similarity curve obtained by the image displacement amount estimation unit 9a in the process of FIG. In FIG. 12, the vertical axis indicates the similarity value, and the horizontal axis indicates the image displacement parameter when the image sequence is generated in S202 of FIG. In the example of FIG. 12, the similarity between the image sequence and the reference image is calculated by SSD, and the similarity curve is obtained by complementing a discrete similarity map by parabolic fitting. The smaller the degree value, the higher the degree of similarity. In this way, a continuous similarity curve is obtained by complementing the discrete similarity map, and by searching for its extreme value (minimum value in the example of FIG. 12), the reference image and the reference image are obtained. The amount of image displacement can be obtained at the subpixel level.

  In this way, the image displacement amount estimated by the image displacement amount estimation unit 9a and the image data of the arbitrary focus image for which the image displacement amount is obtained are transferred to the super-resolution processing unit 9b, and the resolution is increased with respect to the reference image. Process.

FIG. 13 is a flowchart showing an algorithm for high resolution processing performed in the super-resolution processing unit 9b. Hereinafter, the high resolution processing using the super-resolution processing will be described according to the algorithm flow shown in FIG. First, image data of a reference image and image data of a plurality of arbitrary focus images whose image displacement amount is estimated by the image displacement amount estimation unit 9a are read (S301). Next, using the reference image as a target image for high resolution processing, complementary processing such as bilinear interpolation or bicubic interpolation is performed on the target image to create an initial image z ₀ (S302). This interpolation process can be omitted depending on circumstances.

  Then, using the image displacement amount estimated by the image displacement amount estimation unit 9a, the pixel correspondence between the target image and the arbitrary focus image where the image displacement amount is estimated is clarified, and the enlarged coordinates of the target image are used as a reference. The registration process is performed in the coordinate space to generate a registration image y (S303). Here, y is a vector representing the image data of the registration image. For details of the method for generating this registration image y, see “Tanaka / Okutomi, High-speed Algorithm for Reconfigurable Super-Resolution Processing, Computer Vision and Image Media (CVIM) Vol. 2004, No. 113, pp. 97 -104 (2004-11) ”. In the superimposition process in S303, the pixel position is associated between each pixel value of a plurality of arbitrary focus images whose image displacement amount has been estimated in the process of FIG. 11 and the enlarged coordinates of the target image. This is done by moving the value over the closest grid point of the enlarged coordinates of the target image. At this time, a plurality of pixel values may be set on the same grid point. In this case, averaging processing is performed on these pixel values.

  Next, a point spread function (PSF) in consideration of imaging characteristics such as an optical transfer function (OTF, Optical Transfer Function) and a CCD aperture (CCD aperture) is obtained (S304). This PSF is reflected in the matrix A in the following formula (5), and for example, a Gauss function can be used simply. Then, using the registration image y generated in S303 and the PSF obtained in S304, the evaluation function f (z) represented by the following equation (5) is minimized (S305), and f (z ) Is minimized (S306).

In Expression (5), y is a column vector representing the image data of the registration image generated in S603, z is a column vector representing the image data of the high-resolution arbitrary focus image obtained by increasing the resolution of the target image, A is a PSF, etc. Is an image conversion matrix that represents the characteristics of the imaging system including. Also, g (z) is a regularization term that takes into account the smoothness of the image, the correlation between the colors of the image, and the like, and λ is a weighting factor. For example, the steepest descent method can be used to minimize the evaluation function f (z) represented by the equation (5). When the steepest descent method is used, a value obtained by partially differentiating f (z) by each element of z is calculated, and a vector having these values as elements is generated. Then, as shown in the following equation (6), by adding a vector whose element is a partial differentiated value to z, the high-resolution arbitrary focus image z is updated (S307), and f (z) is minimized. Z is obtained.

In Equation (6), z _n is a column vector representing the image data of the high-resolution arbitrary focus image that has been updated n times, and α is the step of the update amount. In the process of the first S305, it is possible to use the initial images z ₀ obtained in S302 as a high-resolution arbitrary focus image z. If it is determined in step S306 that f (z) has been minimized, the process ends, and z _n at that time is recorded in the memory 6 or the like as a final high-resolution arbitrary focus image. In this way, it is possible to obtain a high-resolution arbitrary focus image having a higher resolution than the arbitrary focus image configured by the processing of FIG.

  FIG. 14 is a block diagram illustrating a configuration example of the super-resolution processing unit 9b. Here, the high resolution processing using the super resolution processing performed in the super resolution processing unit 9b will be further described. 14 includes an interpolation enlarging unit 201, an image storage unit 202, a PSF data holding unit 203, a convolution integration unit 204, a registration image generation unit 205, an image comparison unit 206, and a convolution integration unit. 207, a regularization term calculation unit 208, an update image generation unit 209, and a convergence determination unit 210.

First, a reference image is given to the interpolation enlargement unit 201 as a target image for high resolution processing, and interpolation enlargement of the target image is performed (corresponding to S302 in FIG. 13). Examples of the interpolation enlargement method used here include bilinear interpolation and bicubic interpolation. The target image interpolated and enlarged by the interpolation enlargement unit 201 is sent to the image storage unit 202 as, for example, an initial image z ₀ and stored therein. Next, the interpolation-enlarged target image is supplied to the convolution integration unit 204, and convolution integration is performed with the PSF data (corresponding to the image conversion matrix A in Expression (5)) provided by the PSF data holding unit 203. Done.

  Further, the target image and the plurality of reference images whose image displacement amount estimation unit 9a has estimated the image displacement amount are given to the registration image generation unit 205, and the image displacement amount obtained by the image displacement amount estimation unit 20a is based on the image displacement amount. In addition, a registration image y is generated by performing an overlay process in a coordinate space based on the enlarged coordinates of the target image (corresponding to S303 in FIG. 13). The registration process in the registration image generation unit 205 is performed by, for example, changing the pixel position between each pixel value of a plurality of arbitrary focus images whose image displacement amount is estimated by the image displacement amount estimation unit 9a and the enlarged coordinates of the target image. This is done by associating each pixel value on the grid point closest to the enlarged coordinates of the target image. At this time, a plurality of pixel values may be set on the same grid point. In this case, averaging processing is performed on these pixel values.

The image data (vector) convolved and integrated in the convolution integration unit 204 is sent to the image comparison unit 206, and the pixel at the same pixel position with the registration image y generated in the registration image generation unit 205. Difference image data (corresponding to (y−Az) in Expression (5)) is generated by calculating the difference between the values. The difference image data generated in the image comparison unit 206 is given to the convolution integration unit 207, and convolution integration is performed with the PSF data given by the PSF data holding unit 203. Convolution unit 207, for example, by integrating convolution and a column vector representing a transposed matrix and the difference image data of the image transformation matrix A in equation (5), ‖y-Az‖ ² and z of formula (5) Generates a vector obtained by partial differentiation with respect to each element of.

Further, the image stored in the image storage unit 202 is given to the regularization term calculation unit 208, and the regularization term g (z) in the equation (5) is obtained, and the regularization term g (z) is changed to each of z. A vector obtained by partial differentiation with respect to elements is obtained. For example, the regularization term calculation unit 208 performs color conversion processing from RGB to YC _r C _{b on} the image data stored in the image storage unit 202, and YC _r C _b components (luminance component and color difference component) thereof. Then, a vector obtained by convolution integration of a frequency high-pass filter (Laplacian filter) is obtained. Then, a square norm (the square of the length) of this vector is used as a regularization term g (z), and a vector obtained by partial differentiation of g (z) with respect to each element of z is generated. If a Laplacian filter is applied to the C _r and C _b components (color difference components), a false color component is extracted. Therefore, the false color component can be removed by minimizing the regularization term g (z). Therefore, by including the regularization term g (z) in the expression (5), the a priori information of the image that “the color difference component of the image is generally a smooth change” is used, and the high-resolution image in which the color difference is suppressed. Can be obtained stably.

  The image data (vector) generated by the convolution integration unit 207, the image data (vector) stored in the image storage unit 202, and the image data (vector) generated by the regularization term calculation unit 208 are updated image generation units. 209. In the updated image generation unit 209, these image data (vectors) are multiplied by weighting coefficients such as λ and α shown in Equations (5) and (6) and added to generate an updated high-resolution arbitrary focus image. (Corresponding to equation (6)).

  Then, the high-resolution arbitrary focus image updated in the updated image generation unit 209 is given to the convergence determination unit 210, and the convergence determination is performed. In this convergence determination, it may be determined that the update operation of the high-resolution arbitrary focus image has converged when the number of repeated computations required for the convergence is greater than a certain number, or the high-resolution updated in the past Recording the arbitrary focus image, taking the difference from the current high resolution arbitrary focus image, the update work of the high resolution arbitrary focus image has converged when it is determined that the update amount is less than a certain value You may judge.

  When the convergence determination unit 210 determines that the update operation has converged, the updated high resolution arbitrary focus image is output to the outside as the final high resolution arbitrary focus image. If it is determined that the update operation has not converged, the updated high-resolution arbitrary focus image is given to the image storage unit 202 and used for the next update operation. This high resolution arbitrary focus image is given to the convolution integration unit 204 and the regularization term calculation unit 208 for the next update operation. By repeating the above processing and updating the high-resolution arbitrary focus image by the update image generation unit 209, a good high-resolution arbitrary focus image can be obtained. Instead of generating the registration image y, the super-resolution processing unit 9b can perform high resolution processing by applying a weighting coefficient to each reference image.

  In this embodiment, a plurality of arbitrary focus images are formed by photographing a plurality of times by photographing means having the imaging optical system 1, the microlens array 2, the photoelectric conversion element 3, and the like, and there is a slight misalignment of these. Since a high-resolution arbitrary-focus image is generated by estimating the amount of image displacement between arbitrary-focus images, it is possible to generate a high-resolution arbitrary-focus image obtained by increasing the resolution of the arbitrary-focus image in a wide range of virtual image plane positions. Become.

(Embodiment 2)
FIG. 15 is a block diagram illustrating a configuration of an imaging system according to the second embodiment of the present invention. The configuration and processing of the imaging system according to this embodiment are the same as those of the imaging system according to Embodiment 1 except for the following points, and only different parts will be described.

  The imaging system according to the present embodiment includes an imaging unit moving mechanism 15 and an imaging unit movement amount recording unit 16 in addition to the components of the imaging system according to the first embodiment. In the imaging system according to the present embodiment, the high resolution processing unit 9 does not include the image displacement amount estimation unit 9a, and is configured only by the super-resolution processing unit 9b.

  The imaging unit moving mechanism 15 moves at least one of the imaging optical system 1, the microlens array 2, and the photoelectric conversion element 3 in a plane perpendicular to the optical axis A. As this imaging unit moving mechanism 15, for example, a driving mechanism used in an existing camera shake correction mechanism or the like can be used. In the present embodiment, the microlens array 2 and the photoelectric conversion element 3 are moved in the same movement amount and in the same movement direction so that the positional relationship between the microlens array 2 and the photoelectric conversion element 3 does not change.

  The imaging unit moving amount recording unit 16 records the moving amount and moving direction of the imaging optical system 1, the microlens array 2, and the photoelectric conversion element 3 that are moved by the imaging unit moving mechanism 15 each time shooting is performed. To do. As the imaging unit movement amount recording unit 16, for example, a gyro sensor of a camera shake correction mechanism can be used.

  The imaging unit movement amount recording unit 16 transfers the movement amount and movement direction for each photographing to the high resolution processing unit 9. In the present embodiment, the imaging unit movement amount recording unit 16 reduces the above movement amount to the scale of the arbitrary focus image formed by the arbitrary focus image configuration unit 8 and sends it to the high resolution processing unit 9. . The scale of the arbitrary focus image is determined by the resolution of the arbitrary focus image, that is, the number of microlenses 20. For example, in the present embodiment, the resolution of the arbitrary focus image is 1/25 of the number of pixels of the photoelectric conversion element 3, and 1/5 of the actual movement amount is sent as the shift amount between the arbitrary focus images. In general, when the pixel array 30 corresponding to one microlens 20 has n rows and n columns, 1 / n of the actual movement amount is the shift amount between the arbitrary focus images.

  Similar to the first embodiment, the inter-image change amount detection unit 10 detects a difference or similarity between arbitrary focus images as a scene change amount, and the image selection unit 11 determines that the scene change amount is equal to or less than a predetermined threshold. A certain arbitrary focus image is transferred to the high resolution processing unit 9. In the present embodiment, only an arbitrary focus image with a smaller amount of scene change than that in the first embodiment is sent to the high resolution processing unit 9. This is because the high resolution processing unit 9 does not have the image displacement amount estimation unit 9a.

  FIG. 16 is a flowchart illustrating processing performed by the inter-image change amount detection unit 10 and the image selection unit 11. Also in the first embodiment, the inter-image change amount detection unit 10 and the image selection unit 11 perform substantially the same processing as the processing shown in FIG.

  First, one arbitrary focus image formed by the arbitrary focus image construction unit 8 is read (S401). Next, it is determined whether or not the arbitrary focus image is read for the first time (S402). If it is the first time, the arbitrary focus image is used as a reference image (S408), and the next arbitrary focus image is read (S401). For the second and subsequent arbitrary focus images other than the reference image, a difference value (such as the above difference) between the reference image and the arbitrary focus image is calculated (S403), and it is determined whether the difference value is equal to or less than a threshold value. (S404). If the difference value between the arbitrary focus image and the reference image is equal to or smaller than the threshold value, the arbitrary focus image is registered as a transfer image to be transferred to the super-resolution processing unit 9b (S405). Then, it is determined whether or not reading has been completed for all arbitrary focus images (S406). If reading has not been completed, the processing from S401 to S406 is repeated. When all the arbitrary focus images have been read, the arbitrary focus image and the reference image registered as transfer images in S405 are transferred to the super-resolution processing unit 9b.

  Thereafter, the super-resolution processing unit 9b generates a high-resolution arbitrary focus image using the transferred reference image and arbitrary focus image and the movement amount and movement direction transferred from the imaging unit movement amount recording unit 16. In the present embodiment, the super-resolution processing unit 9b generates a high-resolution arbitrary focus image using the above-described movement amount and movement direction instead of the image displacement amount in the first embodiment.

  In the present embodiment, at least one of the imaging optical system 1, the microlens array 2, and the photoelectric conversion element 3 is moved by the imaging unit moving mechanism 15, and the movement amount and the moving direction are recorded by the imaging unit movement recording unit 16. Since the high-resolution arbitrary focus image is generated using the recorded movement amount and movement direction, the high-resolution arbitrary focus image can be generated in a short time without estimating the image displacement amount. Other effects are the same as those of the imaging system according to the first embodiment.

(Embodiment 3)
FIG. 17 is a block diagram illustrating a configuration of an imaging system according to Embodiment 3 of the present invention. The configuration and processing of the imaging system according to this embodiment are the same as those of the imaging system according to Embodiment 1 except for the following points, and only different parts will be described.

  The imaging system according to the present embodiment includes an imaging unit moving mechanism 15 in addition to the components of the imaging system according to the first embodiment. The imaging unit moving mechanism 15 moves at least one of the imaging optical system 1, the microlens array 2, and the photoelectric conversion element 3 in a plane perpendicular to the optical axis A as in the second embodiment.

  However, in the present embodiment, an image displacement amount estimation unit 9a is provided in the high resolution processing unit 9, and the image displacement amount estimation unit 9a is provided between the reference image and other arbitrary focus images as in the first embodiment. Estimate the amount of image displacement. In the present embodiment, a high resolution arbitrary focus image is generated by the same processing as in the first embodiment.

  In the present embodiment, since the imaging unit moving mechanism 15 moves at least one of the imaging optical system 1, the microlens array 2, and the photoelectric conversion element 3, it is possible to spontaneously generate an appropriate positional deviation. . Other effects are the same as those of the imaging system according to the first embodiment.

  Needless to say, the present invention is not limited to the above-described embodiments, and includes various modifications and improvements that can be made within the scope of the technical idea. For example, in the above embodiment, an example is shown in which all the components are mounted on one imaging device. For example, the processing in the arbitrary focus image configuration unit 8 or the high resolution processing unit 9 is performed by an image such as a personal computer. You may make it perform with a processing apparatus. In the above-described embodiment, the resolution enhancement processing unit 9 performs the resolution enhancement processing by the super-resolution processing. However, the resolution enhancement processing other than the super-resolution processing may be performed.

1 is a block diagram illustrating a configuration of an imaging system according to Embodiment 1 of the present invention. 2 is an enlarged view showing a part of a microlens array 2 and a photoelectric conversion element 3. FIG. It is the figure which looked at a mode that the light ray which injected into the imaging optical system and passed through one microlens forms an image on a photoelectric conversion element from the perpendicular direction of the optical axis. It is the figure which looked at the opening and pixel arrangement | sequence of an imaging optical system from the direction of the optical axis. It is the figure which looked at the incident state of the light to a microlens array and a photoelectric conversion element from the orthogonal | vertical direction of the optical axis. It is a figure at the time of seeing from which perpendicular | vertical direction of an optical axis which light beam which passes a certain pixel on a virtual image surface is inject | emitted from which division | segmentation aperture of an imaging optical system, and which microlens enters. It is the figure seen from the orthogonal | vertical direction of an optical axis when the light beam inject | emitted from one division | segmentation opening of an imaging optical system and passed through one pixel on a virtual image surface straddles two microlenses. It is the figure which showed the setting range of the virtual image plane which can reconfigure | reconstruct an arbitrary focus image accurately in the imaging system which concerns on Embodiment 1. FIG. Furthermore, it is the figure which showed the setting range of the virtual image plane which can reconstruct an arbitrary focus image accurately. It is a flowchart which shows the process until several sheets of arbitrary focus images are comprised. It is a flowchart which shows the algorithm of the image displacement amount estimation process performed in an image displacement amount estimation part. It is a figure which shows the example of the similarity curve which the image displacement amount estimation part calculated | required in the process of FIG. It is a flowchart which shows the algorithm of the high resolution process performed in a super-resolution process part. It is a block diagram which shows the structural example of a super-resolution process part. It is a block diagram which shows the structure of the imaging system which concerns on Embodiment 2 of this invention. It is a flowchart which shows the process performed by the variation detection part between images, and an image selection part. It is a block diagram which shows the structure of the imaging system which concerns on Embodiment 3 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging optical system 2 Micro lens array 3 Photoelectric conversion element 4 Drive circuit 5 Imaging control part 6 Memory 7 User I / F part 8 Arbitrary focus image structure part 9 High resolution processing part 9a Image displacement amount estimation part 9b Super-resolution processing Unit 10 Inter-image change amount detection unit 11 Image selection unit 15 Imaging unit moving mechanism 16 Imaging unit movement amount recording unit 20 Micro lens 30 Pixel array 31 Pixel 201 Interpolation enlargement unit 202 Image storage unit 203 PSF data holding unit 204 Convolution integration unit 205 Registration Image Generation Unit 206 Image Comparison Unit 207 Convolution Integration Unit 208 Regularization Term Calculation Unit 209 Update Image Generation Unit 210 Convergence Determination Unit

Claims (6)

An imaging optical system through which light from the object field passes, and a separation optical system in which light that has passed through the imaging optical system is incident and separates and emits light that has passed through different areas of the aperture of the imaging optical system; A photoelectric conversion element that receives the light separated and emitted by the separation optical system and converts the image formed by the separation optical system into an electrical signal to generate photographing data;
Arbitrary focus image constructing means capable of reconstructing an arbitrary focus image, which is an image at an arbitrary image plane position within a predetermined range, based on photographing data generated by the photoelectric conversion element;
An imaging control unit that causes the imaging unit to perform a plurality of times of imaging, and causes the arbitrary focus image configuration unit to configure a plurality of arbitrary focus images based on imaging data generated by the plurality of times of imaging;
High resolution processing means for generating a high resolution arbitrary focus image having a higher resolution than the arbitrary focus image using the plurality of arbitrary focus images;
An imaging system comprising:   Image displacement amount estimation means for estimating an image displacement amount between the plurality of arbitrary focus images is provided, and the resolution enhancement processing means uses the image displacement amount estimated by the image displacement amount estimation means. The imaging system according to claim 1, wherein a high-resolution arbitrary focus image is generated. An imaging unit moving unit that moves at least one of the imaging optical system, the separation optical system, and the photoelectric conversion element in a plane perpendicular to the optical axis of the imaging unit;
The imaging control unit moves at least one of the imaging optical system, the separation optical system, and the photoelectric conversion element by the imaging unit moving unit every time the imaging unit performs imaging. Item 3. The imaging system according to Item 2. An imaging unit movement amount recording unit that records a movement amount and a movement direction of the imaging optical system, the separation optical system, and the photoelectric conversion element that are moved by the imaging unit movement unit each time the imaging unit performs imaging. With
The imaging according to claim 3, wherein the high-resolution processing unit generates the high-resolution arbitrary focus image using the movement amount and the movement direction recorded in the imaging unit movement amount recording unit. system.   The imaging system according to claim 4, wherein the imaging unit moving unit moves the separation optical system and the photoelectric conversion element with the same movement amount and the same movement direction.   Inter-image change amount detecting means for detecting a difference or similarity between the plurality of arbitrary focus images, and the high-resolution arbitrary focus image based on the difference or similarity detected by the inter-image change amount detecting means. The image pickup system according to claim 1, further comprising: an image selection unit that selects the arbitrary focus image used for generating the image.

Images