JP2015186188A - Imaging apparatus and display method therefor - Google Patents

Abstract

An imaging apparatus capable of displaying a live view image that is not blurred as a whole and a display method of the imaging apparatus are provided. A microlens array is disposed between an imaging lens and an imaging element. One microlens 6 is assigned to each of the plurality of image pickup devices 6 (image pickup device group 7) of the image pickup device 5. A live view image represented by a signal obtained from the photoelectric conversion element 6 located at the same position relative to the pupil position of the imaging lens through which the light beam from the subject passes is displayed on the display screen of the display device. Since the live view image is not a multi-viewpoint image, the entire image is not blurred. [Selection] Figure 8

Description

  The present invention relates to an imaging device and a display method thereof.

  A camera called a light field camera has been realized. In the light field camera, the image data obtained from the image sensor includes information on the light traveling direction in addition to the light intensity distribution on the light receiving surface. As a result, observation images from a plurality of viewpoints and directions can be reconstructed. In a light field camera, an image in which a desired portion is not blurred can be reconstructed by performing refocusing after imaging. For this reason, if a live view image obtained by live view imaging is displayed on the display screen as it is, the live view image becomes totally blurred as shown in FIG. 15, and the subject cannot be confirmed. For this purpose, there is a camera equipped with an optical viewfinder and a subject is confirmed using the optical viewfinder (Patent Document 1). Also, there is a live view image in which a distance map is created, divided for each distance, refocused for each divided area, and the refocused divided areas are synthesized (Patent Document 2).

JP 2013-165475 A JP 2013-145980

  However, in the thing of patent document 1, since an optical finder is required, a camera will become expensive. In addition, the camera cannot be downsized. Moreover, in the thing of patent document 2, since the load of CPU is large, it is difficult to display a live view image. A time lag may occur when the shutter is pressed while viewing an image obtained by the method described in the cited document 2.

  An object of the present invention is to enable shooting with a light field camera and display a live view image without blur while reducing costs.

  An image pickup apparatus according to the present invention includes an image pickup lens and photoelectric conversion elements arranged, is arranged between an image pickup element that captures a subject in live view imaging, the image pickup lens and the image pickup element, and is 1 for a plurality of photoelectric conversion elements of the image pickup element. Microlens to which one microlens is assigned, and causes the light to be incident on the photoelectric conversion element existing at the same position relative to the pupil position of the imaging lens through which the light from the subject passes among the plurality of photoelectric conversion elements An array and a display control unit that displays a live view image represented by a signal obtained from a photoelectric conversion element present at a relatively same position among a plurality of photoelectric conversion elements on a display screen of the display device. It is characterized by.

  The present invention also provides a display method for an imaging apparatus. That is, in this method, an image pickup device on which photoelectric conversion elements are arranged picks up a live view image of a subject, and is arranged between the image pickup lens and the image pickup device. The microlens array to which the lens is assigned causes the light beam to be incident on the photoelectric conversion element existing at the same position relative to the pupil position of the imaging lens through which the light beam from the subject passes among the plurality of photoelectric conversion elements. The display control unit displays a live view image represented by a signal obtained from a photoelectric conversion element relatively located at the same position among the plurality of photoelectric conversion elements on the display screen of the display device.

  You may further provide the switching instruction | indication part which instruct | indicates switching of the photoelectric conversion element which exists in the same position relatively among several photoelectric conversion elements. In this case, for example, the display control unit detects an instruction from the switching instruction unit and displays a live view image represented by a signal obtained from the switched photoelectric conversion element on the display screen of the display device. Let's go.

  You may further provide the signal calculating means which combines or selects the signal obtained from the adjacent photoelectric conversion element which exists in the relatively same position of a micro lens array among several photoelectric conversion elements. In this case, the display control unit will display the live view image combined by the signal calculation means or represented by the selected signal on the display screen of the display device.

  First calculation control for determining whether the spatial frequency of the subject is low, and first calculation control for causing the signal calculation unit to perform a combination when the first determination unit determines that the spatial frequency of the subject is low. Means may further be provided.

  An imaging mode switching unit and a second calculation control unit that performs selection or combination by the signal calculation unit when the imaging mode is switched by the switching unit may be further provided.

  The second determination means for determining whether or not the subject brightness is equal to or less than the threshold value and the second determination means determine that the subject brightness is equal to or less than the threshold value. The calculation control means may be further provided.

  You may further provide the imaging control means to output the signal output only from the photoelectric conversion element which exists in the relatively same position of a microlens array among several photoelectric conversion elements from an imaging element. In this case, the display control unit will display the live view image represented by the signal output from the image sensor under the control of the imaging control means on the display screen of the display device.

  You may further provide the memory which memorize | stores the signal output from the image pick-up element. In this case, the display control unit reads the signal output from the photoelectric conversion elements present at the relatively same position of the microlens array among the plurality of photoelectric conversion elements from the signals stored in the memory. The live view image represented by the received signal will be displayed on the display screen of the display device.

  According to the present invention, among the plurality of photoelectric conversion elements assigned to one microlens, a live signal represented by a signal obtained from a photoelectric conversion element existing at the same position relative to the pupil position of the imaging lens. The view image is displayed on the display screen of the display device. Since a live view image represented by a signal obtained from such a photoelectric conversion element is a collection of light rays obtained from the same direction, the parallax is reduced and the subject is a live view image that is not blurred. Live view images can be obtained with relatively low cost. Since the load on the CPU can also be suppressed, a signal output from the photoelectric conversion element can be displayed in a live view image in a short time.

It is a block diagram which shows the electric constitution of a light field camera. A microlens array is shown. An image sensor is shown. A microlens array and an image sensor are shown. The relationship between an imaging lens, a microlens array, and an imaging device is shown. The pupil region of the imaging lens is shown. The relationship between a microlens and an image sensor is shown. A microlens array and an image sensor are shown. It is an example of a live view image. It is an example of a live view image. The relationship between a microlens and an image sensor is shown. The relationship between a microlens and an image sensor is shown. It is a flowchart which shows the process sequence of a light field camera. It is a flowchart which shows the process sequence of a light field camera. It is an example of a live view image.

  FIG. 1 is a block diagram showing an electrical configuration of a light field camera (imaging device) 1 according to an embodiment of the present invention.

  The overall operation of the light field camera 1 is controlled by the control device 20.

  The light field camera 1 includes an imaging element 5 in which an imaging lens 2 and a photoelectric conversion element are arranged. A microlens array 3 is disposed between the imaging lens 2 and the imaging device 5.

  FIG. 2 is a front view of the microlens array 3.

  A plurality of microlenses 4 are arranged in the microlens array 3 in the horizontal direction and the vertical direction.

  FIG. 3 is a front view of the image sensor 5.

  In the image sensor 5, photoelectric conversion elements 6 are arranged in the horizontal direction and the vertical direction. In this embodiment, one photoelectric conversion element group (a plurality of photoelectric conversion elements) composed of a total of 25 (not necessarily 25) in the horizontal direction and the vertical direction (not necessarily 5). 6) One microlens 4 is assigned to 7.

  FIG. 4 is a front view of the image sensor 5 viewed from the subject side.

  As described above, it can be seen that one microlens 4 is assigned to each photoelectric conversion element group 7 composed of 25 photoelectric conversion elements 6.

  FIG. 5 shows a state in which the light beam that has passed through the imaging lens 2 enters the photoelectric conversion element 6 of the imaging element 5.

  The light beam from the subject is refracted by the imaging lens 2 and enters the photoelectric conversion element 6 of the imaging element 5 through the microlens 4 constituting the microlens array 3. The imaging lens 2 is divided into five pupil regions a1 to a5 in the vertical direction in FIG. As shown in FIG. 6, the imaging lens 2 is divided into 25 pupil regions in consideration of the horizontal direction and the vertical direction, but in FIG. 5, since it is viewed from the side, five pupil regions divided in the vertical direction for convenience. a1 to a5 are represented. The light beam L1 that has passed through the pupil region a1 enters the photoelectric conversion element p1 in the photoelectric conversion element group 7. Similarly, light rays L2, L3, L4 and L5 that have passed through pupil region a2, pupil region a3, pupil region a4 or pupil region a5 are incident on photoelectric conversion elements p2, p3, p4 and p5 of photoelectric conversion element group 7, respectively. To do. The same applies not only to the specific photoelectric conversion element group 7 but also to all the photoelectric conversion element groups 7 of the imaging element 5.

  FIG. 6 shows a state where the pupil region of the imaging lens 2 is divided into 25 photoelectric conversion elements 6 constituting the photoelectric conversion element group 7.

  The imaging lens 2 includes 25 pupil regions a11 to a15 (pupil region a1), a21 to a25 (pupil region a2), a31 to a35 (pupil region a3), a41 to a45 (pupil region a4), and a51 to a55 ( The pupil region is divided into 25 pupil regions a5).

  FIG. 7 shows the relationship between the microlens 4 and the 25 photoelectric conversion elements 6 constituting the photoelectric conversion element group 7.

  The 25 photoelectric conversion elements 6 constituting the photoelectric conversion element group 7 are denoted by reference numerals p11 to p15 (photoelectric conversion element p1), p21 to p25 (photoelectric conversion element p2), p31 to p35 (photoelectric conversion element p3), and p41 to p45. (Photoelectric conversion element p4) and p51 to p55 (photoelectric conversion element p5).

  Light rays that have passed through the pupil regions a11, a12, a13, a14, or a15 of the imaging lens 2 are incident on the photoelectric conversion elements p11, p12, p13, p14, or p15 by the microlens array 3. Similarly, the light beam that has passed through the pupil regions a21, a22, a23, a24, or a25 of the imaging lens 2 is incident on the photoelectric conversion elements p21, p22, p23, p24, or p25 by the microlens array 3. Light rays that have passed through the pupil regions a31, a32, a33, a34, or a35 enter the photoelectric conversion elements p31, p32, p33, p34, or p35 by the microlens array 3, and the pupil regions a41, a42, a43 of the imaging lens 2 , A44 or a45 is incident on the photoelectric conversion elements p41, p42, p43, p44 or p45 by the microlens array 3 and passes through the pupil regions a51, a52, a53, a54 or a55 of the imaging lens 2. The light beam is incident on the photoelectric conversion elements p51, p52, p53, p54 or p55 by the microlens array 3. The pupil regions a11 to a15, a21 to a25, a31 to a35, a41 to a45 and a51 to a55 of the imaging lens 2 as described above, the photoelectric conversion elements p11 to p15, p21 to p25, and p31 to p35 of the photoelectric conversion element group 7. , P41 to p45 and p51 to p55 are not limited to the specific photoelectric conversion element group 7, but are established for all the photoelectric conversion element groups 7 included in the imaging element 5. As described above, the microlens array 3 is present at the same position relative to the pupil position of the imaging lens 2 through which the light beam from the subject passes among the plurality of photoelectric conversion elements 6 constituting the photoelectric conversion element group 7. The light is incident on the photoelectric conversion element 6 that performs the above operation.

  FIG. 8 shows a correspondence relationship between the photoelectric conversion element 6 included in the imaging element 5 and the microlens 4 included in the microlens array 3, and corresponds to FIG.

  In this embodiment, when the image pickup device 5 performs live view image pickup, it is obtained from the photoelectric conversion devices 6 that are relatively located at the same position among the plurality of photoelectric conversion devices 6 constituting the photoelectric conversion device group 7. The output signal is output from the image sensor 5. For example, as indicated by hatching in FIG. 8, a signal obtained from the photoelectric conversion element 6 indicated by reference numeral p <b> 44 in FIG. Other photoelectric conversion elements 6 may be used instead of the photoelectric conversion element 6 indicated by reference sign p44, and a plurality of adjacent or non-adjacent photoelectric conversion elements 6 may be used as long as the photoelectric conversion elements 6 are relatively located at the same position. A signal obtained from the element 6 may be an output signal of the imaging element 5. Normally, a signal obtained from the photoelectric conversion element 6 indicated by a symbol p33 (lens center) in FIG. 7 is used as an output signal of the imaging element 5.

  Returning to FIG. 1, when the imaging mode is set by the switch included in the operation device 17, live view imaging is performed, and the images are relatively located at the same position by imaging control by the control device (imaging control means) 20. A signal obtained from the photoelectric conversion element 6 becomes an output signal of the imaging element 5. The image signal output from the image sensor 5 is subjected to analog signal processing such as correlated double sampling, signal amplification, and analog / digital conversion in the analog signal processing circuit 11, and is output as digital image data. The digital image data output from the analog signal processing circuit 11 is subjected to digital signal processing such as reference level adjustment in the digital signal processing circuit 12. The digital image data output from the digital signal processing circuit 12 is subjected to predetermined image processing in the image processing device 13. The digital image data output from the image processing device 13 is given to the display device 18 via the control device 20.

  In the case of a light field camera, light rays from different viewpoints (different directions) are incident on each of the plurality of photoelectric conversion elements 6 constituting the photoelectric conversion element group 7, and thus are obtained from all the photoelectric conversion elements 6. When a live view image is displayed using a signal, the entire image 32 is blurred as shown in FIG. In this embodiment, a live view image represented by a signal obtained from the photoelectric conversion element 6 on which only light rays from a specific viewpoint (specific direction) are incident is displayed by the control device (display control unit) 20 on the display device 18. Is displayed on the display screen. As shown in FIG. 9, a sharp live view image 30 is displayed.

  When the shutter release button included in the operation device 17 is pressed, the signals obtained from all the photoelectric conversion elements 6 become the output signals of the imaging element 5 unlike the case of live view imaging. This output signal is given to the memory 14 via the analog signal processing circuit 11, the digital signal processing circuit 12, and the image processing device 13, and is temporarily stored. The digital image data is read from the memory 14, and the read image data is recorded as multi-viewpoint image data on the memory card 16 by the recording control device 15. Since multi-viewpoint image data is recorded in the memory card 16, a refocus process is performed to remove blur from the desired portion of the captured image so that an image with a sharp desired portion can be reproduced. Become. Such a refocus process is performed by the image processing apparatus 13, for example.

  In the above-described embodiment, in the case of live view imaging, not all the imaging elements 6 but a plurality of photoelectric conversion elements 6 assigned to the microlens 4 (a part of the plurality of photoelectric conversion elements 6), A signal obtained from the photoelectric conversion element 6 that is relatively located at the same position is used as an output signal of the imaging element 5, and a live view image is generated from the output signal. However, when the image signals obtained from all the image sensors 6 are used as output signals of the image sensor 5 as they are and all the digital image data converted from the output signals are temporarily stored in the memory 14 and read from the memory 14. , The image data output from the photoelectric conversion element 6 existing at the relatively same position of the microlens array 3 is read, and the live view image represented by the read image data (signal) is displayed on the display screen of the display device 18. You may be made to do.

  FIG. 10 is an example of a live view image obtained as described above.

  A ghost 31 is generated in the live view image 30A. A ghost 31 may be generated by light that is repeatedly reflected by the imaging lens 2 or the like. As described above, when the ghost 31 is generated in the live view image 30 </ b> A, switching of the photoelectric conversion elements 6 existing relatively at the same position among the plurality of photoelectric conversion elements 6 configuring the photoelectric conversion element group 7 is performed. You may make it instruct | indicate by the operating device (switching instruction | indication part) 17. FIG. The live view represented by the signal obtained from the switched photoelectric conversion element 6 (the same applies to the case of reading from the memory 14 as described above) by detecting the instruction from the operation device 17 by the control device 20. The image is displayed on the display screen of the display device 18 by the control device (display control unit) 20.

  FIG. 11 shows the relationship between the microlens 4 and the plurality of photoelectric conversion elements 6 constituting the photoelectric conversion element group 7, and corresponds to FIG.

  In the above-described embodiment, a signal obtained from the photoelectric conversion element 6 indicated by reference sign p44 is used as an output signal of the image pickup element 5, and a live view image 30A is generated. In such a case, as described above, when the ghost 31 is generated, the photoelectric conversion elements 6 that are relatively located at the same position are switched. In this embodiment, as indicated by hatching, a signal obtained from the photoelectric conversion element 6 indicated by reference numeral p33 existing at the relatively same position is used as an output signal of the imaging element 5. Even in this case, as shown in FIG. 9, the live view image has no blur.

  FIG. 12 shows a modification and shows the relationship between the microlens 4 and the plurality of photoelectric conversion elements 6 constituting the photoelectric conversion element group 7.

  In the above-described embodiment, among the plurality of photoelectric conversion elements 6 assigned to the microlens 4, there is one photoelectric conversion element 6 from which a signal is read, but as shown by hatching in the upper diagram of FIG. A live view image may be generated by reading signals from the photoelectric conversion elements 6 (reference symbols p33, p34, p43, and p44). When signals are read out from two or more photoelectric conversion elements 6 that are adjacent (not necessarily adjacent) as indicated by hatching among the plurality of photoelectric conversion elements 6 as shown in the lower diagram of FIG. Thus, the photoelectric conversion element 6 may be switched so that a part overlaps before and after switching of the photoelectric conversion element 6 from which a signal is read.

  As shown in the upper diagram of FIG. 12, among the plurality of photoelectric conversion elements 6, signals obtained from adjacent photoelectric conversion elements existing at relatively the same position of the microlens array 3 are combined or selected. The control device (signal calculation means) 20 may perform this. In that case, the live view image combined or selected by the control device 20 will be displayed on the display screen of the display device 18 by the control device 20.

  Further, whether or not the spatial frequency of the subject is low is determined by the control device (first determination means) 20 (can be determined based on a signal output from the image sensor 5), and it is determined that the spatial frequency of the subject is low. Thus, a combination of signals obtained from the photoelectric conversion element 6 may be performed.

  Further, when the shooting mode is switched by the shooting mode switch (switching means) included in the operation device 17 (for example, when switching to the landscape shooting mode), a combination of signals obtained from the photoelectric conversion element 6 is combined. Or a selection may be made.

  Further, the control device (second determination means) 20 determines whether or not the subject brightness is equal to or less than the threshold value, and the combination of signals obtained from the photoelectric conversion element 6 when the subject brightness is determined to be equal to or less than the threshold value. Or the selection may be made.

  For the combination and selection of the signals described above, the operation device 17 may be provided with a cross key, a touch sensor, etc., and the photoelectric conversion element 6 for obtaining a signal may be indicated using these cross key, touch sensor, etc. The operation device 17 may be provided with a push button, and the photoelectric conversion element 6 for obtaining a signal may be switched at random by simply pressing the push button.

  FIG. 13 is a flowchart showing a processing procedure of the light field camera 1 when the imaging mode is set.

  When the imaging mode is set, the aperture (not shown) is opened and the subject is captured in live view (step 41). Subsequently, among the plurality of photoelectric conversion elements 6 constituting the photoelectric conversion element group 7 as described above, the photoelectric conversion elements 6 that are relatively located at the same position are determined (step 42). Of course, it may be determined in advance. The determined signal output from the photoelectric conversion element 6 is used as the output signal of the imaging element 5, and the output signal is subjected to gain-up processing in the analog signal processing circuit 11 (step 43). Based on the gain-up signal, a live view image is displayed on the display screen of the display device 18 as described above (step 44).

  FIG. 14 is a flowchart showing another processing procedure of the light field camera 1 when the imaging mode is set.

  The subject is imaged at the full aperture value (step 51), and the multi-viewpoint image data output from all the photoelectric conversion elements 6 of the image sensor 5 (excluding the photoelectric conversion elements 6 not used for recording) is stored in the memory as described above. 14 (step 52). Subsequently, among the image data recorded in the memory 14, the image data used for displaying the live view image is determined as described above (step 53). As described above, image data obtained from the photoelectric conversion elements 6 that are relatively located at the same position among the plurality of photoelectric conversion elements 6 assigned to the microlens 4 are read. The read image data is subjected to gain-up processing in the image processing device 13 (step 55), and a live view image is displayed on the display screen of the display device 18.

1 Light field camera (imaging device)
2 Imaging lens 3 Micro lens array 5 Imaging element
14 memory
17 Operating device (switching instruction unit, switching means)
20 Control device (display control unit, signal calculation means, first determination means, first calculation control means, second calculation control means, second determination means, third calculation control means, imaging control means)

Claims (9)

Imaging lens,
An imaging device in which photoelectric conversion elements are arranged and images a subject,
One microlens is allocated to the plurality of photoelectric conversion elements of the image pickup element, which is disposed between the image pickup lens and the image pickup element, and light rays from a subject are out of the plurality of photoelectric conversion elements. A microlens array that causes the light beam to be incident on a photoelectric conversion element that exists at the same position relative to the pupil position of the imaging lens that passes through, and a plurality of photoelectric conversion elements that exist at the same position. A display control unit for displaying a live view image represented by a signal obtained from the photoelectric conversion element on a display screen of the display device;
An imaging apparatus comprising: A switching instruction unit for instructing switching of photoelectric conversion elements that are relatively located at the same position among the plurality of photoelectric conversion elements;
The display control unit
Detecting an instruction from the switching instruction unit, and displaying a live view image represented by a signal obtained from the switched photoelectric conversion element on the display screen of the display device;
The imaging device according to claim 1. A signal calculation means for combining or selecting signals obtained from adjacent photoelectric conversion elements present at relatively the same position of the microlens array among the plurality of photoelectric conversion elements;
The display control unit displays a live view image combined or selected by the signal calculation means on the display screen of the display device;
The imaging device according to claim 1. First determination means for determining whether or not the spatial frequency of the subject is low, and a first determination means for performing a combination by the signal calculation means when the first determination means determines that the spatial frequency of the subject is low. Arithmetic control means,
The imaging apparatus according to claim 3, further comprising: Shooting mode switching means, and second calculation control means for performing selection or combination by the signal calculation means when the shooting mode is switched by the switching means,
The imaging device according to claim 3 or 4, further comprising: When the subject brightness is determined to be lower than the threshold value by the second determination means for determining whether the subject brightness is equal to or lower than the threshold value, the combination by the signal calculating means is performed. Third arithmetic control means,
The imaging device according to any one of claims 3 to 5, further comprising: An image pickup control means for outputting, from the image pickup device, a signal output from only the photoelectric conversion device existing at a relatively same position of the microlens array among the plurality of photoelectric conversion devices;
The display control unit
A live view image represented by a signal output from the image sensor under the control of the imaging control means is displayed on a display screen of a display device;
The imaging device according to any one of claims 1 to 6. A memory for storing a signal output from the image sensor;
The display control unit reads a signal output from a photoelectric conversion element present at a relatively same position of the microlens array among the plurality of photoelectric conversion elements from the signals stored in the memory. The live view image represented by the received signal is displayed on the display screen of the display device.
The imaging device according to any one of claims 1 to 7. An image sensor in which photoelectric conversion elements are arranged images the subject,
A microlens array, which is arranged between the imaging lens and the imaging element and in which one microlens is assigned to the plurality of photoelectric conversion elements of the imaging element, is formed from the subject among the plurality of photoelectric conversion elements. The light beam is incident on a photoelectric conversion element existing at the same position relative to the pupil position of the imaging lens through which the light beam passes,
The display control unit displays a live view image represented by a signal obtained from a photoelectric conversion element relatively located at the same position among the plurality of photoelectric conversion elements on the display screen of the display device;
Display method of imaging apparatus.

Images