JP2008180808A - Stereoscopic imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain three-dimensional information such as depth information with high accuracy in a stereoscopic imaging apparatus which forms right and left object images having parallax with light from an object. <P>SOLUTION: The stereoscopic imaging apparatus 1 is equipped with a lens optical system 3 having right and left lenses 2 whose optical axes L are parallel with each other, a solid state image sensor 5, and right and left prisms 8R and 8L forming the condensing optical paths of light 20R and light 20L from the object for every right and left lenses 2. The right and left prisms 8R and 8L have first 45-degree rectangular prism parts 81R and 81L arranged to be opposed to the right and left lenses 2 respectively, second 45-degree rectangular prism parts 82R and 82L arranged at positions where sufficient base length D is obtained outside the right and left lenses 2, and light guide parts 83R and 83L. The light 20R and the light 20L from the subject are made incident on the right and left prisms 8R and 8L having the sufficient base length D, and separately condensed by the lenses 2 having the same distortion aberration, so that they are formed as the subject image from which distortion is easily removed. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a stereo image capturing apparatus, and more specifically, a stereo image that forms separate subject images having parallax through a condensing path in which light from a subject is separated left and right and generates a stereo image based on these subject images. The present invention relates to an imaging apparatus.

  In an in-vehicle camera, in order to detect a person approaching the vehicle by taking a stereo image of the surroundings of the vehicle, light from the same imaging range is collected from a plurality of viewpoints, each in a different area of the image sensor. A vehicle object detection device configured to obtain a parallax image (subject image) by forming an image is known (see, for example, Patent Document 1).

  In addition, an imaging apparatus (for example, refer to Patent Document 2) that can capture both a stereoscopic image and a high-definition planar image with a single apparatus, or a stereo imaging apparatus that has a compact structure in the horizontal direction (for example, Patent Document 3) is known.

Furthermore, a stereoscopic image photographing optical device (see, for example, Patent Document 4) capable of photographing a stereoscopic image by forming images from right and left optical paths on a single image sensor in a time-division manner, or a twin-lens type. There is known a stereoscopic electronic still camera (see, for example, Patent Document 5) that can obtain a stereoscopic image without requiring the above optical system.
JP 2005-117136 A Japanese Patent No. 354297 JP 2005-241791 A JP-A-8-251624 Japanese Patent No. 3112485

  FIG. 7 shows an outline of the optical system portion of the vehicle object detection device described in Patent Document 1. The optical system portion includes an imaging lens 105b disposed on the front side of the imaging element 105a, a prism 105c disposed on the front side of the imaging lens 105b, and two reflecting members 105d and 105e disposed on the left and right sides of the prism 105c. And two objective lenses 105f and 105g respectively arranged on the front side of the reflecting members 105d and 105e.

  Lights 113L and 113R from the same imaging range in front of the vehicle are condensed by the left and right objective lenses 105f and 105g, reflected by the left and right reflecting members 105d and 105e, and both side surfaces of the prism 105c, and one imaging lens 105b. And are focused on the left and right areas of the image sensor 105a. The left and right images (subject images) captured by the image sensor 105a are processed as images having parallax, and an object is detected.

  However, in the vehicle object detection apparatus described above, the lights 113L and 113R are incident on the imaging lens 105b from a position shifted to the left and right as shown in FIG. Since the image is formed by the half part and the right half part, the two subject images picked up by the image pickup element 105a are affected by the distortion aberration of the image forming lens 105b and have different distortions on the left and right. Become.

  Specifically, for example, when the imaging lens 105b is a lens having barrel distortion (see FIG. 8), as shown in FIG. 9, the left subject image 101L has a left barrel distortion. The right subject image 101R becomes an image in which the barrel distortion on the right side appears strongly.

  In general, when obtaining three-dimensional information such as depth information of a subject from left and right subject images, the left and right subject images are digitally processed to remove distortion so that the left and right subject images can be compared. If a pre-process for generating an image (for example, a predetermined rectangular image) is executed, but the distortion of the left and right subject images is different as described above, the content and extent of processing for removing the distortion Is different between the left and right subject images.

  For example, in the case of the left and right subject images 101L and 101R shown in FIG. 9, the left subject image 101L is corrected more greatly on the left side of the image, and the right subject image 101R is corrected more greatly on the right side of the image.

  As described above, the content and degree of preprocessing for distortion removal differ between the left and right, so that in the left and right subject images 101L and 101R subjected to the preprocessing, the pixel position corresponding to the same subject portion between both images is minute. There is a high possibility that a deviation remains, and such a deviation lowers the accuracy of three-dimensional information such as subject depth information calculated based on the left and right subject images 101L and 101R.

  Accordingly, the present invention provides a stereo image capturing apparatus that forms separate subject images having parallax on an image sensor through a condensing path in which light from a subject is separated left and right, and generates a stereo image based on these subject images. Provides a stereo image pickup device capable of obtaining highly accurate three-dimensional information by obtaining the same baseline length in the left and right subject images with the same image distortion caused by lens distortion. For the purpose.

  In order to achieve the above object, according to the first aspect of the present invention, separate subject images having parallax are formed on an image sensor through a condensing path in which light from a subject is separated left and right, and based on these subject images. In a stereo image pickup device for generating a stereo image, a lens optical system having left and right lenses having the same distortion aberration, the optical axes of which are parallel to each other and arranged on the same plane, and each focal position by the left and right lenses A single solid-state imaging device for converting a subject image formed into electronic image information, and a lens optical system disposed on the subject side of the lens optical system. Each of the first and second right-angle prism portions disposed opposite to the left and right lenses, and the right and left lenses, respectively. A second 45 ° right angle prism portion disposed at a position where a sufficient baseline length on the right outer side can be obtained, and a light guide portion connecting the first and second 45 ° right angle prism portions, Condensing light from a subject separately on the left and right by the prism and the left and right lenses to form left and right subject images on the solid-state imaging device, and generating a stereo image based on these subject images To do.

  According to the second aspect of the present invention, stereo image imaging is performed in which separate subject images having parallax are formed on an image sensor through a condensing path in which light from a subject is separated left and right, and a stereo image is generated based on these subject images. In the apparatus, a lens optical system having left and right lenses having the same distortion aberration whose optical axes are parallel to each other, and solid-state imaging for converting a subject image formed at each focal position by the left and right lenses into electronic image information An element and a light bending member such as a prism or a mirror that is disposed on the subject side of the lens optical system and forms a condensing path of light incident on the left and right lenses from the subject for each lens. Are collected separately by the light bending member and the left and right lenses to form a plurality of subject images on the solid-state imaging device, and stereo images are formed based on these subject images. And generating an image.

  According to a third aspect of the present invention, in the second aspect of the invention, the light bending member is a prism that guides light incident from a subject on the left and right outer sides of the left and right lenses to the left and right lenses, respectively. And

  According to the first aspect of the present invention, since the light from the subject is separately collected on the left and right by the left and right lenses having the same distortion as the prism to form the subject image, the distortion of the image due to the distortion of the lens is reduced. Since the aspect is the same in the left and right subject images, and the content and extent of the distortion removal processing are the same in the left and right subject images, the same subject portion in the left and right images (stereo images) subjected to the distortion removal processing The degree of coincidence of corresponding pixel positions can be easily increased, and a sufficient baseline length can be easily obtained by the prism, so that highly accurate three-dimensional information based on a stereo image can be obtained.

  According to the second aspect of the invention, since the light from the subject is separately collected by the left and right lenses having the same distortion aberration as the light bending member to form the subject image, the image caused by the distortion aberration of the lens can be obtained. Since the distortion mode is the same in the left and right subject images, and the content and extent of the distortion removal processing are the same in the left and right subject images, the same subject in the left and right images (stereo images) subjected to the distortion removal processing The degree of coincidence of pixel positions corresponding to portions can be easily increased, and highly accurate three-dimensional information based on stereo images can be obtained.

  According to the third aspect of the present invention, since the light from the subject is separately collected by the left and right lenses having the same distortion aberration as the light bending member to form the subject image, the image caused by the distortion aberration of the lens is formed. Since the distortion mode is the same in the left and right subject images, and the content and extent of the distortion removal processing are the same in the left and right subject images, the same subject in the left and right images (stereo images) subjected to the distortion removal processing The degree of coincidence of the pixel positions corresponding to the portions can be easily increased, and a sufficient baseline length can be easily obtained by the light bending member, so that highly accurate three-dimensional information based on a stereo image can be obtained.

  Hereinafter, a stereo image capturing apparatus according to an embodiment of the present invention will be described with reference to FIGS. 1 to 5. As shown in FIGS. 1 to 3, the stereo image capturing apparatus 1 of the present embodiment includes nine lenses 2 in three rows and three columns arranged on the same plane with optical axes L parallel to each other. The lens optical system 3, the nine images 4 (subject images) formed by the respective lenses 2, one solid-state image pickup device 5 for converting it into electronic image information, and nine pieces converted by the solid-state image pickup device 5 Of the image information of the subject image 4, the electronic circuit 7 configured to perform distortion removal processing on the image information of the left and right subject images 4 to form stereo images 6 L and 6 R (see FIG. 4), and the lens optical system 3. A display device including prisms 8L and 8R that are arranged on the subject side and that separately form right and left condensing paths for light incident on the left and right lenses 2 from the subject, and a liquid crystal panel that displays stereo images 6L and 6R, etc. 9.

  The lens optical system 3 includes nine lenses 2, a lens support frame 12 in which a diaphragm aperture 11 for light incident on the lenses 2 is formed, and a light shield that partitions the light emitted from the lens 2 so as not to interfere with each other. A frame 13 and a filter 14 disposed immediately above the solid-state imaging device 5 are provided. The nine lenses 2 have the same shape and size and the same distortion.

  The solid-state imaging device 5 is formed of a semiconductor such as a CMOS (Complementary Metal Oxide Semiconductor) having a large number of unit pixels u (see FIG. 3) disposed at the focal position of the lens 2.

  The electronic circuit 7 converts analog image information (subject image) output from the solid-state imaging device 5 into digital information, and processes the digital image information from the AD converter 15 according to a predetermined procedure. The DSP 16 (Digital Signal Processor) that generates the stereo images 6L and 6R from which the distortion of the image is removed by applying the image information and the depth information of the subject by taking the image information (stereo image) generated by the DSP 16 and performing a predetermined process And a microprocessor 17 for obtaining three-dimensional information.

  In the stereo image pickup apparatus 1 of the present embodiment, the electronic circuit 7 includes the subject image 4 ml (see FIG. 3) formed on the solid-state image pickup device 5 by the lens 2 ml in the left column of the center row, and the center row. The two pieces of image information of the subject image 4mr (see FIG. 3) formed on the solid-state imaging device by the lens 2mr in the right column are captured and processed as left and right subject images, respectively.

  Next, the prisms 8L and 8R will be described. As shown in FIGS. 1 to 3, the prisms 8L and 8R of the present embodiment include a left prism 8L extending in a wing shape to the left from the position of the lens 2 in the left column on the subject side of the lens optical system 3, and A right prism 8R extending rightward from the position of the lens 2 in the right row. Since the left prism 8L and the right prism 8R have symmetrical structures, the structure of the left prism 8L will be described below, and the description of the right prism 8R will be omitted by setting the number suffix to R.

  The left prism 8L has a first 45 ° right-angle prism portion 81L disposed to face the lens 2 in the left column of the lens optical system 3, and a predetermined distance d (hereinafter, separated from the lens 2 in the left column). A second 45 ° right-angle prism portion 82L disposed at a position separated by a distance) and a light guide portion 83L connecting the first and second 45 ° right-angle prism portions 81L and 82L.

  The entire left prism 8L composed of the first and second 45 ° right-angle prism portions 81L and 82L and the light guide portion 83L may be integrally molded, or the first and second 45 ° right-angle prisms. The portions 81L and 82L and the light guide portion 83L may be formed as separate members and then integrally joined with an adhesive or the like. When integrally molded, there is an advantage that the number of reflections of light on the boundary surface is small and optical noise is small compared to the case where a plurality of members are joined.

  As shown in FIG. 1, the light 20L and 20R from the subject is incident on the second 45 ° right-angle prism portions 82L and 82R of the left and right prisms 8L and 8R outside the lens optical system 3, Are reflected by the slopes 82La and 82Ra of the 45 ° right-angle prism portions 82L and 82R, and pass through the light guide portions 83L and 83R to reach the upper side of the left and right lenses 2. The lights 20L and 20R that have reached the upper side of the left and right rows of lenses 2 are reflected by the inclined surfaces 81La and 81Ra of the first 45 ° right-angle prism portions 81L and 81R, and are incident on each lens 2 along the optical axis L of each lens 2. Incident light is formed as a subject image 4 on the solid-state imaging device 5 by each lens 2.

  Here, the distance between the light beams 20L and 20R from the subject incident on the second 45 ° right-angle prism portions 82L and 82R of the left and right prisms 8L and 8R is the baseline length D (see FIG. 1), and the separation distance d is sufficient. By setting it to a large size, the base line length D can be obtained to a sufficient size.

  In particular, the lens optical system 3 according to the present embodiment is manufactured by a semiconductor manufacturing technique. For example, the lens 2 can be manufactured as small as the distance between the optical axes L of the lenses 2 is about 1.2 mm, but the separation distance d is sufficient. By setting it to a large size, a sufficiently large baseline length D is obtained, and as will be described later, depth information can be detected with higher accuracy within a range limited by the resolution of the solid-state imaging device 5. .

  Now, there are two cases, for example, when there are no prisms 8L and 8R between the subject and the lens 2 (substantially the separation distance d = 0) and when there are prisms 8L and 8R having a predetermined separation distance d. Differences in accuracy of the depth information obtained will be described with reference to FIG. For convenience, only the left light collecting path in FIG. 5 is described.

  When the prism 8L is not provided (FIG. 5A), the light from the subject Ta directly enters the lens 2 and forms the subject image 4a on the solid-state imaging device 5. Similarly, the light from the subject Tb behind the subject Ta forms a subject image 4b on the solid-state imaging device 5, but the angle αa at which the light from the subject Ta enters the lens 2 and the light from the subject Tb are the lens. 2 is small, the subject images 4a and 4b on the solid-state image sensor 5 are close to each other, and the distance between the subject images 4a and 4b on the solid-state image sensor 5 is the distance between unit pixels u (resolution). ), The difference in depth between the subject Ta and the subject Tb is not detected.

  On the other hand, when there is a prism 8L having a predetermined separation distance d, the difference between the angle βa at which light from the subject Ta enters the prism 8L and the angle βb at which light from the subject Tb enters the prism 8L is large. Therefore, the distance between the subject images 4a and 4b on the solid-state imaging device 5 is relatively large, and the difference in depth between the subject Ta and the subject Tb is easily detected.

  For example, when the prisms 8L and 8R are not provided, a difference in depth between the subject Ta at a distance of 50 cm and the subject Tb at a distance of 60 cm from the stereo image pickup apparatus 1 is not detected in relation to the resolution of the solid-state imaging device 5. Even so, when there are prisms 8L and 8R having a predetermined separation distance d, a difference in depth between the subject Ta at a distance of 50 cm and the subject Tb at a distance of 60 cm is detected.

  As described above, practically accurate depth information can be obtained by employing the prisms 8L and 8R having a predetermined separation distance d in relation to the resolution of the solid-state imaging device 5.

  For example, when the stereo image pickup apparatus 1 is used as an automobile collision prevention sensor, it is desirable to obtain depth information about a subject within a range of 1 m from the rear of the automobile with high accuracy. The separation distance d of the prisms can be determined to be a predetermined value in relation to the resolution (interval between unit pixels u).

  Next, a procedure for obtaining a stereo image by the stereo image capturing apparatus 1 of the present embodiment will be described.

  The stereo image pickup device 1 is directed to an appropriate subject, and light 20L, 20R from the subject is bent through the left and right prisms 8L, 8R and is incident on the left and right lenses 2 of the lens optical system 3. An image is formed as a subject image 4 on the solid-state imaging device 5.

  At this time, since the distortion aberration of each lens 2 is the same, each subject image 4 has a parallax corresponding to the position of the lens 2, but all the distortions are the same image.

  The subject image 4 is selectively read out by the electronic circuit 7 to the left column subject image 4ml and the right column subject image 4mr (see FIG. 3), and the image information is subjected to distortion removal by the DSP 16. Stereo images 6L and 6R (see FIG. 4) from which distortion has been removed by processing are generated. Since the distortion of the subject images 4ml and 4mr is the same, the content of the distortion removal processing by the DSP 16 may be the same, and the generated stereo images 6L and 6R have a predetermined shape (for example, a rectangular shape) with high accuracy. The image is formed.

  The stereo images 6L and 6R may be displayed on the display device 9 by the microprocessor 17, or the microprocessor 17 may generate three-dimensional information such as depth information based on the stereo images 6L and 6R. Good.

  For example, when the microprocessor 17 obtains the depth information based on the stereo images 6L and 6R, the microprocessor 17 determines the left and right of all the pixels of the stereo images 6L and 6R based on the color density and brightness. Stereo images 6L and 6R are compared to detect the same subject portion (for example, a person's eye Pa or background picture Pb), and the position difference between the same subject portions Pa and Pb in the left and right stereo images 6L and 6R. Is converted into depth information.

  At this time, since the stereo images 6L and 6R generated by the DSP 16 are formed in a predetermined shape with high accuracy, the difference (depth information) between the positions of the same subject portions Pa and Pb in the left and right stereo images 6L and 6R is also obtained. It is detected with high accuracy.

  As described above, in the stereo image pickup apparatus 1 of the present embodiment, the light 20L and 20R from the subject is separately collected on the left and right by the left and right lenses 2ml and 2mr having the same distortion aberration as the prisms 8L and 8R. Since the image is formed, the distortion of the image due to the distortion of the lens 2 is the same in the left and right subject images 4 ml and 4 mr, and the content and extent of the distortion removal processing are the same in the left and right subject images 4 ml and 4 mr. Therefore, the degree of coincidence of pixel positions corresponding to the same subject portion in the left and right stereo images 6L and 6R subjected to the distortion removal process can be easily increased, and a sufficient baseline length can be easily obtained by the prisms 8L and 8R. Since D can be obtained, highly accurate three-dimensional information can be obtained.

  In the stereo image pickup apparatus 1 of the present embodiment, the electronic circuit 7 uses the nine subject images 4 formed on the solid-state image pickup device 5 to obtain the image information of the left and right subject images 4ml and 4mr in the center row. However, the image information captured by the electronic circuit 7 is one of the three subject images 4 in the left column and the three subject images 4 in the right column. Any of these may be sufficient.

  Further, as shown in FIG. 6, the lens optical system 3 includes two lenses 2 that are parallel to the optical axis L and arranged on the left and right sides, and the first 45 of the prisms 8L and 8R. The right-angle prism portions 81L and 81R may be arranged to face the left and right lenses 2.

  Furthermore, instead of the left and right prisms 8L and 8R, mirrors arranged at the positions of the inclined surfaces 81La and 82La and 81Ra and 82Ra of the left and right prisms 8L and 8R may be used.

The side view of the partial cross section which shows schematic structure of the stereo image imaging device which concerns on one Embodiment of this invention (a cross section is the AA line of FIG. 2). The top view of the lens optical system and prism of the stereo image pick-up device. The perspective view of the lens optical system and prism of the stereo image pick-up device. The figure which shows the example of the stereo image produced | generated in the same stereo image imaging device. (A) is explanatory drawing which shows the relationship between a to-be-photographed object and a to-be-photographed object when there is no prism, (b) is explanatory drawing which shows the relationship between a to-be-photographed object and a to-be-photographed image with a prism. The top view which shows the modification of the lens optical system and prism in the stereo image pick-up device. 1 is a schematic configuration diagram of a conventional stereo image capturing apparatus. The figure which shows the influence which the distortion aberration of the imaging lens in the conventional stereo image imaging device has on an image. FIG. 10 is a diagram illustrating an example of a subject image in a conventional stereo image capturing apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereo image pick-up device 2, 2 ml, 2 mr Lens 3 Lens optical system 4, 4 ml, 4 mr Subject image 5 Solid-state image sensor 6L, 6R Stereo image 8L, 8R Prism (light bending member)
81L, 81R First 45 ° right angle prism portions 82L, 82R Second 45 ° right angle prism portions 83L, 83R Light guide portions 20L, 20R Light from subject D Base length L Optical axis

Claims (3)

In a stereo image imaging device that forms separate subject images having parallax on a imaging element through a condensing path that separates light from the subject left and right, and generates a stereo image based on these subject images.
A lens optical system having left and right lenses having the same distortion, the optical axes of which are parallel to each other and arranged on the same plane;
A single solid-state imaging device for converting a subject image formed at each focal position by the left and right lenses into electronic image information;
A prism that is disposed on the subject side of the lens optical system and forms a condensing path of light incident on the lens from the subject for each of the left and right lenses;
The prism is
A first 45 ° right angle prism portion disposed opposite to the left and right lenses, respectively.
A second 45 ° right angle prism portion disposed at a position where a sufficient baseline length is obtained on the left and right sides of each of the left and right lenses;
A light guide portion connecting the first and second 45 ° right angle prism portions,
The light from the subject is separately collected by the prism and the left and right lenses separately to form left and right subject images on the solid-state imaging device, and a stereo image is generated based on these subject images. Stereo image pickup device. In a stereo image imaging device that forms separate subject images having parallax on a imaging element through a condensing path that separates light from the subject left and right, and generates a stereo image based on these subject images.
A lens optical system having left and right lenses having the same distortion aberration, the optical axes of which are parallel to each other;
A solid-state imaging device that converts a subject image formed at each focal position by the left and right lenses into electronic image information;
A light bending member such as a prism or a mirror, which is disposed on the subject side of the lens optical system and forms a condensing path of light incident on the left and right lenses from the subject for each lens;
Light from a subject is separately collected on the left and right by the light bending member and the left and right lenses to form a plurality of subject images on the solid-state imaging device, and a stereo image is generated based on these subject images. A featured stereo image imaging device.   3. The stereo image capturing apparatus according to claim 2, wherein the light bending member is a prism that guides light incident from a subject to the left and right lenses at the left and right outer sides of the left and right lenses, respectively.

Images