JP2005303694A - Compound eye imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To attain both a wide angle and high resolution of all objects with simple image processing in a compound eye imaging device that is formed by arranging a plurality of lenses in parallel with each other and can be made smaller because lens back becomes short. <P>SOLUTION: Lenses 3R, 3G1 and 3B of a short focus and a lens 3G2 of a long focus have different angles of view but image an object so as to include the same part of the object. A zoomed up image obtained by an imaging device corresponding to the lens 3G2 of a long focus is inserted into a portion of a wide image obtained by an imaging device corresponding to the lenses 3R, 3G1 and 3B of a short focus. Then, an image, wherein the portion of the image has high resolution and the residual part has low resolution but has a wide angle of view, can be obtained from all objects by a simple insertion combining processing. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a compound-eye imaging device that is formed by arranging a plurality of lenses in parallel with each other and can reduce the size of the device.

  In recent years, small photographic devices such as digital cameras are becoming popular. In addition, camera functions are also being installed in mobile phones and personal digital assistants, and there is a strong demand for downsizing camera devices. Particularly in the latter case, there is a strong demand for thinning the equipment. In addition, there are fields where downsizing of devices is always required, such as digital cameras for endoscopes. However, in a configuration using a single imaging lens and an imaging sensor with a mosaic filter as in a normal camera, when the size of the sensor and the required image quality are determined, the number and size of the required lenses are almost the same. Therefore, there was a limit to downsizing the device.

  Therefore, as conventional techniques that can solve such a problem, for example, Patent Documents 1 and 2 are cited. Patent Document 1 is an optical device in which a plurality of small lenses are two-dimensionally arranged to reduce the size of a camera. Patent Document 2 is an image input device in which a plurality of small lenses are two-dimensionally arranged to combine images with different viewpoints. In these conventional techniques, a technique for shortening the lens back using a lens array and miniaturizing the apparatus has been proposed.

  However, in Japanese Patent Application Laid-Open No. 2004-260260, the individual lenses capture different positions of the subject, so that the magnification is limited to the use close to 1, and it is difficult to adopt for use using a reduction optical system for photographing a general subject. There is. Further, in Patent Document 2, there is a problem that the arithmetic processing for restoring a single high-resolution image from many low-resolution images is complicated and requires a lot of cost and time.

  On the other hand, two contradictory functions of high resolution and wide field of view are required for a camera device used for monitoring the inside of a building or house, outdoors, and the like. During normal monitoring, it is necessary to broadly look over a wide area to detect the presence or absence of abnormalities at an early stage, and to look closely at the targets of abnormalities such as fires and intruders. For such applications, cameras that can control the line-of-sight direction and shooting magnification, such as pan, tilt, and zoom, are commercially available. In addition, a fovea-type camera has been proposed that can photograph the center and the periphery sparsely by devising lenses and sensors. However, in any case, the mechanism and the lens become large, and it is difficult to reduce the size of the apparatus.

Therefore, for such applications, as shown in Patent Documents 3 and 4, an approach for solving both problems using a lens array has been proposed. In Patent Document 3, an image having different magnifications is obtained by using a plurality of lenses having different focal lengths, and a signal having the second highest resolution is obtained from a signal in which the focal length is appropriate for the distance to the subject and the highest resolution is obtained. By performing signal processing for subtraction, the imaging apparatus can perform focus adjustment without moving the lens. In Patent Document 4, a plurality of lenses having different focal lengths are arranged in parallel, and a photographing area in the peripheral part is made larger than that in the central part. This is a compound eye imaging system in which
Japanese Patent Publication No.59-50042 JP 2001-61109 A JP 2000-32354 A JP 2002-171447 A

  The patent document 3 uses a plurality of images with different magnifications individually, and because the sensor is separated by each function, it is difficult to suppress positional deviation or characteristic difference for each image. There's a problem. Patent Document 4 is similar to Patent Document 1, in which each lens captures a different position of a subject, and is difficult to apply to a general subject, and also requires processing for pasting adjacent images. There is a problem that much cost and time are required.

  An object of the present invention is to provide a compound eye imaging apparatus capable of achieving both wide angle and high resolution with simple image processing for any subject.

  The compound-eye imaging device of the present invention is a compound-eye imaging device in which lenses having different focal lengths are arranged in parallel with each other. A part of an image obtained by an imaging element corresponding to a short-focus lens is long-focused. And image processing means for fitting an image obtained by an imaging device corresponding to the lens.

  According to the above configuration, a plurality of lenses are arranged in parallel with each other, and an image area can be reduced by a compound eye. Therefore, in a compound-eye imaging device that can shorten the lens back and reduce the size of the device, In a configuration in which the focal lengths are different from each other, for example, a signal from an imaging element with an appropriate focus is selected as in Patent Document 3, or a different part of the subject is imaged as in Patent Document 4, for example. On the other hand, in the present invention, the short-focus lens and the long-focus lens capture an image so as to include the same part of the subject, although the angle of view is different. In other words, the imaging range of the long-focus lens is always included (partly) in the imaging range of the short-focus lens, and the long-focus lens is compared to the short-focus lens if the image is in the same range of the subject. The resolution will be higher. Then, by fitting the zoomed-up image obtained by the image sensor corresponding to the long focus lens into the part of the wide image obtained by the image sensor corresponding to the short focus lens, the part of the image However, although the resolution of the remaining portion is low, an image with a wide angle of view can be obtained.

  Therefore, it is possible to achieve both wide angle and high resolution with a simple fitting composition process for any subject.

  In the compound eye image pickup apparatus of the present invention, the lens has two types of focal lengths, the short focus side faces an image sensor that detects R, G, and B colors, and the long focus side detects G. The image pickup element faces.

  According to the above configuration, when realizing the above wide angle and high resolution, the short focus (wide angle) side is detected in full color of R, G, and B, and the long focus (telephoto) side is detected as G. To. Then, the image processing means uses G1 on the short focus side as G1 and G2 on the long focus side as G2, while maintaining the R and B images as they are, the region of the G2 image in the G1 image. Is replaced with the G2 image, and the position is corrected and fitted.

  Therefore, a high-resolution image can be effectively obtained by replacing the G image with high visibility.

  Furthermore, the compound eye imaging device of the present invention is characterized by a 2 × 2 lens arrangement in the vertical and horizontal directions.

  According to the above configuration, when the R, G1, G2, and B images are obtained, the distance between the lenses is relatively small by adopting a 2 × 2 vertical and horizontal lens arrangement for the four lenses. The parallax is small and lenses can be arranged in a compact manner, and one rectangular image sensor can be divided into four parts and used efficiently.

  In the compound eye imaging device of the present invention, the long focus lens is provided with a displacement means for displacing the long focus lens in a plane perpendicular to the optical axis.

  According to the above configuration, it is possible to displace a long-focus (zoom-up) image area such as raising a suspicious person while keeping a short-focus (wide) image unchanged by grasping the entire view of the monitoring target area. . As a result, it is possible to arbitrarily zoom up on the part that the operator wants to pay attention to, which can improve convenience.

  Furthermore, in the compound-eye imaging device of the present invention, the image processing means performs an interpolation process on the short focus side image, which is approximately the inverse of the magnification ratio of the short focus side lens and the long focus side lens. It is characterized by that.

  According to the above configuration, for example, when the focal length of the long-focus side lens and the short-focus side lens is twice, the image processing means interpolates approximately twice the short-focus side image. Processing will be performed.

  Therefore, by fitting the double zoom image into the substantially double interpolation image, the difference in resolution between the short focus (wide) image and the long focus (zoom up) image can be made inconspicuous.

  In the compound eye imaging device of the present invention, the image processing means is activated when a predetermined event occurs.

  According to the above configuration, the image processing means does not always operate, that is, normally, R, G1, B images are synthesized and a short focus (wide) full color image is obtained. On the other hand, the image processing means is triggered by a predetermined event, for example, an operation of the operator or detection outputs of various sensors for crime prevention, and a part of the G1 image is a long-focus (zoom-up) G2 image. Is replaced.

  Therefore, normally, the fitting and synthesizing process is not performed, so that when the predetermined event occurs, a high-definition image for human recognition is obtained even if the frame frequency is decreased. Can be obtained.

  As described above, the compound eye imaging device of the present invention is formed by arranging a plurality of lenses in parallel with each other, and the image area can be reduced by the compound eye, so that the lens back can be shortened and the device can be miniaturized. In the apparatus, a short-focus lens and a long-focus lens are provided, and the wide-angle obtained by an image sensor corresponding to the short-focus lens is picked up so as to include the same part of the subject although the angle of view is different. By inserting a zoomed-up image obtained by an image sensor that supports a long-focus lens into a part of the image, the resolution of the part of the image is high, and the remaining part has a low resolution but a wide angle of view. To get an image of.

  Therefore, it is possible to achieve both wide angle and high resolution with a simple fitting composition process for any subject.

[Embodiment 1]
FIG. 1 is an exploded perspective view showing a configuration of a compound eye imaging apparatus 1 according to an embodiment of the present invention, and FIG. 2 is a diagram showing a cross section of an imaging unit. The compound eye imaging apparatus 1 is generally configured to include an imaging device 2, a lens array 3, a spectral filter 4, a partition wall 5, and an image processing unit 6.

  The image pickup device 2 has a light receiving section 2a arranged two-dimensionally, and can read out the obtained image signal for each pixel, and has a CMOS structure for reading out charges in a matrix manner. However, a CCD structure that sequentially reads out may be used. In this imaging device 2, the imaging area is virtually divided according to the number of lenses described later, and the light passing through each lens reaches only one imaging area. In this imaging device 2, the imaging region is divided into a total of 4 regions of 2 × 2 in length and width, and a partition wall 5 that functions as a hood is provided so that light that has passed through the other lenses as described above does not enter. ing. The number of pixels in each imaging region in the imaging device 2 is equal to each other.

  Each imaging region in the imaging device 2 may be one-dimensionally arranged. In this case, lenses are also arranged one-dimensionally. However, by adopting a 2 × 2 lens arrangement in the vertical and horizontal directions as described above, when obtaining four images R, G1, G2, and B, which will be described later, the distance between the lenses is relatively small, and therefore the parallax is small. In addition, the lenses can be arranged in a compact manner, and one rectangular imaging device 2 can be divided into four parts and used efficiently.

  The lens array 3 includes four image forming units 2 × 2 in length and width arranged in parallel to each other, and includes wide-angle lenses 3R, 3G1 and 3B, and a telephoto lens 3G2. G1, B and G2 images are obtained. For example, the magnification of the telephoto lens 3G2 is twice that of the wide-angle lenses 3R, 3G1, and 3B, but this ratio is not limited. In the present embodiment, the region of the subject captured by the telephoto lens 3G2 is a central portion of the wide-angle lenses 3R, 3G1, and 3B and is a region that is 1/4 in area.

  The spectral filter 4 is also divided into 4 × 2 × 2 regions corresponding to the imaging device 2 and the lens array 3, and an R on the object side of the imaging optical path of the wide-angle lenses 3R, 3G1, and 3B. , G, and B are inserted spectral filters 4R, 4G1, and 4B, and a G spectral filter 4G2 is inserted on the subject side of the imaging optical path of the telephoto lens 3G2. The spectral filter 4 may be one using a dye or one using an interference principle. Further, the transmission color of the spectral filter 4 is not limited to R, G, and B as long as it can integrate and synthesize a color image. Further, the spectral filter 4G1 and the spectral filter 4G2 may be different from each other in the center wavelength, or may be separated into four or more colors, or may be of complementary colors such as C, M, and Y. In any case, the imaging area of the imaging device 2 is divided into the number of colors + 1, and a color near G with high visibility is arranged as the spectral filter 4 in the telephoto lens.

  The partition wall 5 is provided so that the light passing through the lenses 3R, 3G1, 3B, and 3G2 reaches only one photographing region as described above, is formed in a lattice shape, and the imaging element 2 and the lens It is arranged between the array 3. The inner peripheral surfaces of the four grids of the partition wall 5 are painted in black that is not smooth so as not to reflect light.

  As will be described later, the image processing unit 6 performs processing such as interpolation, synthesis and switching of the obtained image data, conversion to a television signal and digital signal, storage and transfer.

  FIG. 3 is a diagram schematically showing how the image processing unit 6 performs image processing. FIG. 3A shows an example of images obtained by the lenses 3R, 3G1, 3B, 3G2. A wide-angle image of R, G1, and B and a telephoto image of G2 are obtained in each imaging region in the imaging device 2. During normal monitoring, the image processing unit 6 selectively reads out only wide-angle R, G1, and B images and generates a full-color image. In this case, the image sensor having the CMOS structure can read an image by designating only a necessary pixel address. On the other hand, in the image pickup device having the CCD structure, image data is sequentially transferred, and only pixel data at a necessary address is read out. Therefore, when a CMOS structure image sensor is employed, wide-angle image data required can be read at a higher speed than a CCD structure image sensor using an image sensor having the same number of pixels. In this case, the update period of the image data can be shortened (the frame rate is increased), and the time can be observed more finely.

  On the other hand, a predetermined event such as an intrusion of a suspicious person occurs, and the image processing unit 6 is triggered by, for example, an operation of an operator who monitors the monitor or detection outputs of various sensors for crime prevention. Perform image processing. Two types of the image processing unit 6 are provided for normal use and for abnormal use, and during normal operation, three color components of R, G, and B are used as described above in the same manner as a normal three-plate surveillance camera. It is also possible to operate the image processing unit on the side that performs the simple processing for combining the images, and to activate the image processing unit on the side that performs the following slightly complicated processing in the event of an abnormality.

  When the abnormality occurs, the telephoto G2 image is read together with the wide-angle R, G1, and B images. Then, as shown in FIG. 3B, the image processing unit 6 interpolates wide-angle R, G1, and B image data into a total of four times the number of pixels, two times in length and width. The interpolation method may be a known method such as a bilinear method using an average value of adjacent pixels or a cubic convolution method using convolution of surrounding pixels. In this way, by performing a double interpolation operation on the data of the wide-angle R, G1, and B images and fitting the data of the double zoom image G2 as described later, a short focus (wide) image and a long focus ( (Zoom-up) The difference in resolution from the image can be made inconspicuous.

  Next, as shown in FIG. 3 (c), a 1/4 region of the above-mentioned area at the center of the interpolated G1 image is cut out, and the G2 original image obtained by the telephoto lens is pasted. A composite image G ′ shown in (d) is obtained. Here, when the size of the imaging unit is sufficiently smaller than the subject, the deviation of the viewpoint position can be ignored.

  On the other hand, when the shift between the G1 and G2 images cannot be ignored, an average filter is applied to the boundary region to make the shift inconspicuous, as shown in FIG. There is also a method of extracting and correcting the amount of positional deviation from an image. For example, a pixel that captures the same part of the subject is detected between the G1 and G2 images by a corresponding point extraction method known for binocular stereoscopic distance measurement and the amount of deviation is calculated from the address. Based on this shift amount, the position of the telephoto G2 image is corrected vertically and horizontally and pasted on the G1 image. When the amount of deviation is equal to or less than the pixel unit, the correction equal to or less than the pixel unit is performed using the above-described interpolation processing. When the image pickup device 2, the lens array 3, and the spectral filter 4 have the structure shown in FIG. 1, when G2 is used as a reference, the horizontal shift of G1 is the same as that of the B image and the vertical shift is the same as that of the R image. Since only the amount is generated, the position is corrected in the same manner. As a result, it is possible to correct the positional deviation of all the R, G1, and B images.

  By the above processing, a composite image G ′ having a high resolution and a wide angle at the center as shown in FIG. 3D can be obtained. By combining the composite image G ′ and the wide-angle R and B images, the update period of the image data is increased by the time required for image processing (the frame rate is lowered), but the wide-angle and high-resolution are obtained. A single full color image can be obtained. Here, since human visual characteristics are sensitive to green, subject identification ability can be improved when the G image has a high resolution as in the above configuration. In addition, since the surveillance camera is often installed so that the subject to be watched is at the center, the portion can be imaged with high resolution.

  FIG. 4 is a flowchart illustrating in detail the above-described image processing operation. In step S1, the normal monitoring mode is set. In step S2, only wide-angle R, G1, and B images are selectively read. In step S3, a full-color image is generated and displayed. In step S4, it is determined whether or not an abnormality has occurred. While no abnormality has occurred, the processes in steps S1 to S3 are repeated.

  If it is determined in step S4 that an abnormality has occurred, the process proceeds to a gaze mode in step S5, and all images R, G1, B, and G2 are read in step S6. In step S7, an interpolation calculation is performed on the wide-angle R, G1, and B images to four times the number of pixels. Subsequently, in step S8, the central region of the interpolated G1 image is cut out, and the G2 original image is pasted. In step S9, a full color image is generated and displayed. Thereafter, the process returns to step S4 to determine whether or not the abnormality occurrence state is continued. If the abnormality is continued, the processing of steps S6 to S9 is repeated. -S3 is repeated.

  Here, if the number of pixels of the image sensor is 4n, in the conventional mosaic filter, G is 2n, and R and B are n pixels. On the other hand, in this embodiment, there are n R, G1, G2, and B, respectively. Therefore, when the conventional camera device has a focal length equivalent to that of the wide-angle lens of the present embodiment, R and B are the same number of pixels, G1 is 1/2 the number of pixels, and G2 for the subject. Will have twice as many pixels as G. Therefore, the composite image G ′ has a lower resolution than the conventional camera in the peripheral portion with respect to green with high visibility, but has a high resolution in the central portion. In this way, in a compound eye imaging device that can reduce the size of the device by shortening the lens back, it is possible to realize a small size, wide angle, and high resolution for any subject by simple fitting and combining processing.

[Embodiment 2]
FIG. 5 is a diagram illustrating an example of an image captured by a compound eye imaging apparatus according to another embodiment of the present invention. It should be noted that in the present embodiment, the relationship between the wide-angle lenses 3R, 3G1, 3B and the telephoto lens 3G2 in the compound-eye imaging device 1 described above is interchanged. That is, the imaging device 2, the partition wall 5 and the spectral filter 4 are the same as the compound eye imaging device 1 described above, but the lenses 3R, 3G1 and 3B are telephoto lenses, although the lens arrays are the same in a 2 × 2 arrangement. Thus, the lens 3G2 becomes a wide-angle lens. Therefore, the images R, G1, G2, and B obtained in the respective imaging regions of the imaging element 2 are as shown in FIG.

  The image processing unit 6 selectively reads only the wide-angle G2 image and displays a monochrome image during normal monitoring. In this case, since the number of pixels to be read out is small, in the case of the CMOS image sensor, the cycle can be shortened (the frame rate is increased), and the time resolution at the time of gaze is improved. On the other hand, when an abnormality occurs, the telephoto R, G1, and B images are also read out and a full-color image is synthesized.

  The spectral filter 4G2 may be a filter that reproduces a human visibility curve. Further, the spectral filter 4G2 corresponding to the wide-angle lens may be omitted, and imaging may be performed at all wavelengths. In that case, there is no decrease in the amount of light due to the filter, so the sensitivity during monitoring is improved. In addition, if an imaging device is sensitive to the near-infrared region, a living body can be monitored in a dark situation such as at night. Since the sensitivity is not improved during gaze, a full color image may be obtained by emitting a flash. Furthermore, if it is exclusively for night use, a filter that transmits only infrared rays may be used.

  FIG. 6 is a flowchart for explaining in detail the above-described image processing operation. Similar to the processing shown in FIG. 4 described above, the corresponding steps are denoted by the same step numbers. If the normal monitoring mode is set in step S1, only the wide-angle G2 image is selectively read in step S12, and a monochrome image is generated and displayed in step S13. In step S4, it is determined whether or not an abnormality has occurred. While no abnormality has occurred, the processes in steps S1, S12, and S13 are repeated.

  If it is determined in step S4 that an abnormality has occurred, the process proceeds to a gaze mode in step S5, and all images R, G1, B, and G2 are read in step S16. Subsequently, in step S18, the central region of the wide-angle G2 image is cut out, and the telephoto R, G1, and B images are pasted. In step S19, a full-color image is generated and displayed. Thereafter, returning to the step S4, it is determined whether or not the abnormality occurrence state is continued. If the abnormality occurrence state is continued, the processes of the steps S16, S18, and S19 are repeated. The processes of S1, S12, and S13 are repeated. Between the steps S16 and S18, the interpolation processing as in the step S7 may be performed on the wide-angle G2 image.

  Even in this case, a compact, wide-angle, and high-resolution compound-eye imaging device can be realized. The first and second embodiments may be properly used depending on the type of monitoring target.

[Embodiment 3]
FIG. 7 is a diagram showing a cross section of the imaging unit of the compound eye imaging device 11 according to still another embodiment of the present invention. This compound eye image pickup device 11 is similar to the compound eye image pickup device 1 described above, and corresponding portions are denoted by the same reference numerals and description thereof is omitted. It should be noted that in the compound eye imaging device 11, the telephoto lens 3G2 is provided with a displacement means 12 for displacing the telephoto lens 3G2 in a plane perpendicular to the optical axis. This makes it possible to move a high resolution area from the center. In this case, the region of the G2 image is cut out from the G1 image, and the G2 image is pasted.

  With this configuration, it is possible to change the area of the long-focus (zoom-up) image, such as raising the suspicious person while keeping the short-focus (wide) image unchanged, by grasping the entire view of the monitored area. Thus, the part that the operator wants to focus on can be arbitrarily zoomed in, and convenience can be improved.

It is a disassembled perspective view which shows the structure of the compound eye imaging device which concerns on one Embodiment of this invention. It is a figure which shows the cross section of the imaging part of FIG. It is a figure which shows typically the mode of the image processing by an image process part. 4 is a flowchart for explaining in detail an image processing operation of FIG. 3. It is a figure which shows an example of the captured image by the compound-eye imaging device which concerns on other forms of implementation of this invention. 6 is a flowchart for explaining in detail the image processing operation of FIG. 5. FIG. 7 is a diagram showing a cross section of an imaging unit of a compound eye imaging device according to still another embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,11 Compound eye imaging device 2 Image pick-up element 2a Light-receiving part 3 Lens array 3R, 3G1, 3B Wide-angle lens 3G2 Telephoto lens 4; 4R, 4G1, 4G2, 4B Spectral filter 5 Bulkhead 6 Image processing part 12 Displacement means

Claims (6)

In a compound eye imaging device in which lenses having different focal lengths are arranged in parallel with each other,
A compound eye imaging apparatus comprising image processing means for fitting an image obtained by an image sensor corresponding to a long focus lens into a part of an image obtained by an image sensor corresponding to a short focus lens.   The focal length of the lens is two types, an image sensor that detects three colors of R, G, and B faces the short focus side, and an image sensor that detects G faces the long focus side. Item 2. A compound eye imaging device according to Item 1.   The compound-eye imaging apparatus according to claim 2, wherein the compound-eye imaging device has a 2 × 2 lens arrangement.   The compound eye imaging according to any one of claims 1 to 3, wherein the long focus lens is provided with a displacement means for displacing the long focus lens in a plane perpendicular to the optical axis. apparatus.   5. The image processing unit according to claim 1, wherein the image processing unit performs an interpolation process on the short focus side image at an approximately reciprocal of a magnification ratio between the short focus side lens and the long focus side lens. The compound eye imaging device according to any one of the above.   The compound-eye imaging apparatus according to claim 1, wherein the image processing unit is activated when a predetermined event occurs.

Images