JP2014220564A - Image reproducing device, image reproducing method, program, and imaging device - Google Patents

Abstract

To provide an image reproducing apparatus that enables reconstruction of light field data by an operation that allows an intuitive and easy-to-understand shooting of the F value of the focus and aperture during reproduction. An image reproducing apparatus that reconstructs pixel data that can be refocused and reproduces an image includes an instruction unit that acquires an instruction to reproduce the image, and a setting unit that sets information necessary to reconstruct the pixel data And an update means for acquiring information for updating information necessary for reconstruction set by the setting means, and a generation means for reconstructing pixel data and generating an image according to the information set by the setting means The setting means sets the initial information of information necessary for reconstruction when an instruction to reproduce an image is acquired, and sets the acquired information for update when information for update is acquired. Information necessary for reconstruction is set based on the information, and when the setting means sets information necessary for reconstruction, the generation means reconstructs pixel data according to the set information and generates an image. [Selection] Figure 1

Description

  The present invention relates to an image reproduction apparatus for light field data acquired by a light field camera, and more particularly to an image reproduction apparatus, an image reproduction method, and an image reproduction apparatus related to refocusing at the time of reproduction and adjustment of the depth of field and storage of the adjusted image. The present invention relates to an imaging apparatus.

  2. Description of the Related Art In recent years, an imaging apparatus (hereinafter, a light field camera) that can acquire not only light intensity distribution but also information on the incident direction of light in an imaging apparatus such as an electronic camera has been proposed. For example, according to Non-Patent Document 1, a microlens array is arranged between a photographic lens and an image sensor, and one microlens is associated with a plurality of pixels of the image sensor, thereby allowing light that has passed through the microlens. Is photoelectrically converted into a pixel signal for each incident direction by a plurality of pixels.

  By applying a technique called “Light Field Photography” to the pixel signal (light field information) acquired in this way, an image focused on an arbitrary image plane (refocus plane) is re-captured after shooting. Can be configured and played. Note that a normal captured image can also be generated by taking the sum of a plurality of pixels corresponding to one microlens.

  As a case where the user needs to reconstruct an image by refocusing, for example, when the image is confirmed after shooting, a desired subject is not in focus. With a conventional camera, if the desired subject is not in focus, it must be re-taken. For this reason, as in Patent Document 1, a technique has been devised that makes it easy for the user to focus on a desired subject during shooting.

  On the other hand, in the light field camera, a refocus image can be reconstructed using the captured pixel signal, so that a desired image can be obtained without re-taking.

JP 2000-92387 A JP 2010-093422 A

Ren. Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  However, in the reproduction or preview display of the image data of the light field information, it is necessary to save the display or the image by reflecting the intention of the photographer or the player with respect to the focus of the display image and the setting of the depth of field. is there.

  Furthermore, at least two parameters of focus and depth of field are required for display by image reconstruction, but, for example, an intuitive operation method using an operation member mounted on a camera has not been established.

  In relation to this, Patent Document 1 discloses a technique in which a zoom function using a zoom button is used not only for zooming at the time of shooting but also for zoom setting of a display image at the time of reproduction. However, Patent Document 1 does not consider the image captured by the light field camera, and does not touch on resetting the focus or changing the depth of field.

  Japanese Patent Application Laid-Open No. 2004-228867 discloses a technique for recognizing a subject at the time of shooting, displaying only the subject on a distance graph, and shooting by specifying a focus and a depth of field at a position on the displayed graph. It is disclosed. However, in Patent Document 2, it is difficult to directly and specifically know which part of the image is in focus, so it is difficult to know whether the refocus setting is as intended by the photographer.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an image reproducing apparatus and an image reproducing apparatus which can edit and save an image by setting a focus and a depth of field by an operation that is intuitive and easy to understand when reproducing light field data. An imaging device is provided.

  In order to achieve the above object, according to the present invention, an image reproducing apparatus that reconstructs refocusable pixel data and reproduces an image includes an instruction unit that acquires an instruction to reproduce the image, Pixel data is reconstructed in accordance with setting means for setting information necessary for configuration, update means for acquiring information for updating information necessary for reconstruction set by the setting means, and information set by the setting means Generating means for generating an image, and when the instruction for reproducing the image is obtained by the instruction means, the setting means sets initial information of information necessary for reconstruction, and the updating means for updating When information is acquired, information necessary for reconfiguration is set based on the acquired information for update, and when the setting means sets information necessary for reconfiguration, the set information is set. Reconstruct pixel data according to To generate an image Te.

  In an imaging apparatus capable of capturing or editing an image that can be refocused to an arbitrary focus and displaying the image on a display device, the operation state of the imaging apparatus is a shooting mode for capturing an image or a playback mode for editing a captured image. Mode specifying means for outputting discriminable mode information, focus lens moving means for outputting first focus information in the playback mode and second focus information in the shooting mode, and refocusing in the playback mode Recording instruction means for outputting an instruction to record an edited image, and in the photographing mode, the photographed image and the second focus information are recorded on a storage medium by pressing a shutter button, and the recording instruction is given in the reproduction mode. Recording means for recording the received refocused edited image on the storage medium, and the first focus information in the playback mode. Is a setting adjustment unit that outputs third focus information using one of the second focus information, and edits the captured image using the third focus information in the playback mode, and And image editing means for generating a focused edited image.

  According to the present invention, the setting of the focus and the depth of field in the reproduction of the light field data can be performed intuitively with an operation feeling when shooting. Accordingly, it is possible to provide an image reproduction device and an imaging device that facilitate the reproduction, display, editing, and storage of light field data.

1 is a configuration diagram of a main part of an imaging apparatus to which an image reproduction apparatus according to a first embodiment of the present invention is applied. 1 is a diagram schematically illustrating an appearance of an imaging apparatus according to a first embodiment of the present invention. The figure for illustrating the angle of view of a picked-up image. The figure which shows the histogram for every distance of the picked-up image of FIG. The figure which shows the flowchart of the operation | movement reproduction | regeneration mode of the image reproduction apparatus which concerns on 1st Example of this invention. The figure which shows the flowchart of operation | movement of the image reproduction apparatus which concerns on 1st Example of this invention. The block diagram of the principal part of the imaging device to which the image reproduction apparatus which concerns on 2nd Example of this invention is applied. FIG. 6 is a diagram schematically illustrating an appearance of an imaging apparatus according to a second embodiment of the present invention. The figure which shows the flowchart of the operation | movement reproduction | regeneration mode of the image reproduction apparatus which concerns on 2nd Example of this invention. The figure which shows the flowchart of operation | movement of the image reproduction apparatus which concerns on 2nd Example of this invention. FIG. 9 is a configuration diagram of a main part of an imaging apparatus to which an image reproduction apparatus according to a third embodiment of the present invention is applied. The figure which shows the example of the warning display by the image reproduction apparatus which concerns on 3rd Example of this invention. The figure which shows the histogram of the object detection frequency for every normalized distance which concerns on 3rd Example of this invention. The figure which shows the flowchart of operation | movement of the image reproduction apparatus which concerns on the 3rd Example of this invention. The figure which shows the operation | movement flowchart of the image reproduction apparatus which concerns on the 3rd Example of this invention. The figure which shows the histogram of the object detection frequency for every distance in case the normalization which concerns on 4th Example of this invention is not carried out. 7 shows a flowchart of the operation of an image reproduction apparatus according to a fourth embodiment of the present invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  With reference to FIGS. 1 to 6, the configuration of an imaging apparatus to which the image reproducing apparatus according to the first embodiment of the present invention is applied will be described.

  FIG. 1 is a configuration diagram of a main part of an imaging apparatus 100 that is a light field camera to which an image reproducing apparatus according to a first embodiment of the present invention is applied. FIG. 1 shows only components related to reproduction, display, and storage of light field data that is refocusable pixel data. For the configuration related to the generation of light field data composed of an imaging optical system that forms an optical image of a subject, an image sensor that captures an optical image, an image processing unit that generates pixel data, and their control units, etc. The description is omitted except for parts related to the invention. Note that the control of the entire imaging apparatus is performed by executing a program held in a memory (not shown) by a CPU of a control unit (not shown).

  In FIG. 1, reference numeral 101 denotes a mode instruction unit that instructs an operation mode of the imaging apparatus. Modes that can be switched by the mode instruction unit 101 are at least a shooting mode and a playback mode playback mode. Reference numeral 102 denotes a focus lens moving unit. When the focus position is changed by driving the photographing optical system, the focus lens moving unit 102 outputs focus information. Reference numeral 103 denotes an image editing unit. The image editing unit 103 reconstructs a captured image using settings at the time of shooting, and edits the refocused image. Reference numeral 104 denotes a recording unit. In the shooting mode, the recording unit 104 records the shot image and settings at the time of shooting in a storage medium (not shown), and in the playback mode, records the edited refocus image in a storage medium (not shown). Reference numeral 105 denotes a shutter button unit that gives a shooting instruction in the shooting mode. The shutter button unit 105 outputs a recording instruction for recording a photographed image in the photographing mode, and outputs a recording instruction for recording the edited refocus image in the reproduction mode. Reference numeral 106 denotes a display unit. In the reproduction mode, the display unit 106 displays the edited refocus image on a display device (not shown). Reference numeral 107 denotes a setting adjustment unit that generates focus information necessary for refocusing using focus information at the time of shooting and focus information at the time of editing, and outputs the focus information to the image editing unit 103.

  FIG. 2 schematically shows the appearance of the imaging apparatus according to the first embodiment of the present invention, and the same parts as those shown in FIG. The focus lens moving unit 102 and the shutter button unit 105 have a configuration similar to that of a conventional digital single-lens reflex camera that is not a light field camera, and a configuration in which the operational feeling in the playback mode is not impaired as compared with shooting. desirable. For example, the focus lens moving unit 102 includes a rotatable lens driving unit disposed in a ring shape around the lens tube, and the shutter button unit 105 includes a push button disposed on the camera body 100. However, in the present embodiment, the present invention is not limited to this. For example, an operation such as a focus adjustment or a shutter button press may be performed by a UI operation of a touch panel provided on the camera or a cross key. From this point, the image reproduction apparatus according to the present embodiment can be applied to an apparatus such as an information processing apparatus capable of inputting light field data. In this case, the operation of an instruction device such as a touch panel or a mouse can be performed. A function of adjusting the focus or pressing the shutter button may be performed.

  FIG. 3 is a diagram illustrating an example of an angle of view of an image captured by the imaging apparatus. In the figure, 300 is the entire angle of view. Reference numerals 301 to 304 denote characteristic subjects. In this image, the main subject is the subject 301.

  FIG. 4 shows a histogram of subject detection frequency for each distance corresponding to FIG. In the figure, the horizontal axis represents distance, and the vertical axis represents appearance frequency. In the figure, parts corresponding to the parts shown in FIG. Therefore, 301 to 304 correspond to the subjects 301 to 304 in FIG. Note that 401 to 403 relate to information that is not acquired in the image reproduction operation of the present embodiment of the present invention, and thus description thereof is omitted here.

  As shown in FIG. 4, in this embodiment, the focus distance that can be set by the focus lens moving unit 102 is 16 steps from the distance 0 to the distance 15 with the camera position as the distance 0. It is assumed that a subject 301 exists at a distance 3, a subject 302 exists at a distance 7, a subject 303 exists at a distance 9, and a subject 304 exists at a distance 12.

  In shooting in shooting mode, various settings in image processing such as aperture adjustment and white balance (not shown) are used, and an original image in a light field (hereinafter referred to as an LFRAW image) is stored in the recording unit 104 in accordance with a recording instruction from the shutter button unit 105. ).

  Light field photography is characterized in that a refocus image can be generated using light field data, but the refocusable range is limited by the imaging device, microlens array, and lens characteristics. For this reason, in the various settings, the main focus position may be set by the focus lens moving unit 102 to perform shooting. In addition, it is assumed that the setting values of each setting in the above-described shooting are stored as data associated with the LFRAW image or are included in the LFRAW image.

  With reference to FIGS. 5 and 6, the operation of the imaging apparatus according to the present embodiment in the playback mode will be described in detail. FIG. 5 shows a main flowchart of the reproduction mode in a case where at least one LFRAW image is held, and FIG. 6 shows a detailed flowchart regarding acquisition of the image initial value in FIG.

  In S500, under the control of a control unit (not shown), the light field camera 100 shifts to the playback mode according to the mode instruction unit 101, and the main playback operation is started. In step S501, the focus lens moving unit 102 is initialized. This initialization operation is performed according to the flowchart shown in FIG.

  In S600, when the initialization process in S501 is started, in S6011, the image editing unit 103 acquires the initial focus position from the LFRAW image. For example, it is assumed that the distance 3 that is the subject 301 is set as the initial focus position. Further, the focus initial position may not be from the LFRAW image but may be a position where the focus lens moving unit 102 is preset as a fixed value as the camera body.

  In step S <b> 602, the focus lens moving unit 102 sets the focus information to be output to the image editing unit 103 to the distance 3. In step S <b> 603, the focus lens moving unit 102 sets the focus position to the distance 3. Thereafter, in S604, the initialization process in S501 is terminated.

  Returning to the main flowchart, in S502, the image editing unit 103 reads out the LFRAW image from the recording unit 104, reconfigures the LFRAW image so that the distance 3 is in focus, and edits the image data for display. The display image data is image data that conforms to a data format of a display device (not shown). The edited display image data is output to the display unit 106.

  In step S503, the display unit 106 displays a display image on a display device (not illustrated). In step S504, the display unit 106 determines whether the recording instruction from the shutter button unit 105 has been received. If this is the case, S505 and S506 are executed. If not received, S507 is executed.

  In S505, the image editing unit 103 reads out the LFRAW image from the recording unit 104, reconfigures the LFRAW image according to the focus information received in S501, and edits the image data for storage. The image data for storage is, for example, data in a data format that conforms to the JPEG format, which is an image format that can be reproduced by an image reproduction device that is not light field data. The storage image data that has been edited is output to the recording unit 106, and holds the storage image data received by the recording unit 106 in S506.

  In step S <b> 507, the image editing unit 103 determines whether focus information has been received from the focus lens moving unit 102. If focus information has been received, S508 is executed, and if focus information has not been received, S509 is executed. In step S <b> 508, the image editing unit 103 updates the focus information to the focus information received from the focus lens moving unit 102. In the present embodiment, for example, when the focus position is changed by the user to the distance 7 that is the subject 302 by operating the focus lens moving unit 102, the focus information is changed to the distance 7 and the image editing unit 103 is changed. Output. In step S502, the image editing unit 103 reconstructs the LFRAW image so that the distance 7 is in focus, and edits the display image data.

  In S509, it is determined whether or not mode switching has been instructed. If not instructed, S504 is executed. Thereafter, the operations from S502 to S509 are repeatedly executed until the mode switching instruction by the mode instruction unit 101 is detected in S509. If mode switching is instructed from the mode instruction unit 101 in S509, the control unit for the entire light field camera (not shown) performs a reproduction mode end process. Next, in S510, the playback mode is terminated.

  In the present embodiment, the reproduction operation by image reconstruction when the focus information output from the focus lens moving unit 102 is changed from the distance 3 to the distance 7 has been described. However, the focus information is actually applied to the distance 4 where there is no subject. It is also possible to change and set.

  As described above, according to the present invention, the focus lens moving unit 102 and the shutter button unit 105 can be used in both the shooting mode and the playback mode. As a result, even in the playback mode, it is possible to reconstruct and play back an image adjusted to the desired focus from the LFRAW image with the same focus adjustment operation feeling as when shooting.

  Next, with reference to FIGS. 3, 4, 7, 8, 9, and 10, the configuration of the imaging apparatus to which the image reproducing apparatus according to the second embodiment of the present invention is applied will be described. In addition to the focus lens moving unit 102 and the shutter button unit 105 in the first embodiment, the present embodiment is characterized by the image reproducing operation of the imaging apparatus that also serves as the aperture adjustment unit during shooting and playback. . 7 to 10, the same parts as those in the first embodiment are denoted by the same reference numerals. 3 and FIG. 4, the description of the portion already described in the first embodiment is omitted here.

  In FIG. 4, reference numerals 401, 402, and 403 denote depths of field that can be reconstructed (edited) with an LFRAW image. Reference numeral 401 denotes a state in which the distance 2 to the distance 13 are in focus, that is, all of the subjects 301 to 304 are in focus. Reference numeral 402 denotes a state in which only the subject 301 is in focus. Reference numeral 403 denotes a state where the subjects 302 to 304 are in focus. For the sake of explanation, the depth of field indicates a certain position of the subject, but may be set including a range where there is no subject.

  FIG. 7 illustrates a configuration of a main part of the imaging apparatus 100 that is a light field camera to which the image reproducing apparatus according to the present embodiment is applied. In the figure, reference numerals 101, 102, 104, 105 and 106 are the same as those in FIG. Reference numeral 700 denotes an entire light field camera. Reference numeral 701 denotes an aperture adjustment unit that outputs aperture information set in the shooting mode and the playback mode as F-number information. A setting adjustment unit 702 calculates a focus position and a depth of field from the focus information and the F value information and outputs the calculated position as setting information. An image editing unit 703 reconstructs an LFRAW image according to setting information and edits the image into display image data or storage image data. A recording unit 704 records an LFRAW image and shooting setting information in a storage medium (not shown) in the shooting mode, and records a refocused image obtained by editing the LFRAW image in a storage medium (not shown) in the playback mode. It is.

  FIG. 8 is a diagram schematically illustrating the appearance of the imaging apparatus according to the present embodiment. In the figure, parts similar to those in FIG. As shown in FIGS. 7 and 8, the image pickup apparatus according to the present embodiment has a configuration in which an aperture adjustment unit 701 is added to the first embodiment. Similar to the focus lens moving unit 102 and the shutter button unit 105, the aperture adjusting unit 701 has the same configuration as a conventional digital single-lens reflex camera that is not a light field camera. However, similarly to the first embodiment, it is desirable that the operational feeling in the playback mode is not impaired compared with that during shooting. For example, the aperture value, so-called F value, is changed by a rotation operation by the user. However, the present embodiment is not limited to this, and an aperture value operation may be performed by, for example, a touch panel or cross key UI operation. Therefore, similarly to the first embodiment, the image reproducing apparatus of the present embodiment can be applied to an apparatus such as an information processing apparatus capable of inputting light field data.

  Next, with reference to FIGS. 9 and 10, the operation of the imaging apparatus according to the present embodiment in the playback mode will be described in detail. FIG. 9 shows a main flowchart of the playback mode when at least one LFRAW image is held. FIG. 10 shows a detailed flowchart for obtaining the image initial value in FIG.

  A reproduction mode in which the light field camera 700 shifts to the reproduction mode in accordance with the mode instruction unit 101 and the main editing operation is started under the control of a control unit (not shown) in S900. In step S901, the focus lens moving unit 102 and the aperture adjusting unit 701 are initialized.

  The initialization process of S901 is started in S1000 of the flowchart shown in FIG. In step S1001, the image editing unit 703 acquires an F value from the LFRAW image. For example, it is assumed that the depth of field 402 corresponding to the F value 402 focused on only the subject 301 is set as the initial information on the depth of field. Further, the initial value of the depth of field may be a configuration in which the aperture adjustment unit 701 has a fixed value as the camera body instead of the LFRAW image. In step S <b> 1002, the aperture adjustment unit 701 sets the F value information to the F value 402.

  In FIG. 10, S601 to S603 are the same as those in the first embodiment, and the description thereof is omitted here.

  Returning to the main flowchart shown in FIG. 9, in step S902, the setting adjustment unit 702 calculates the depth of field corresponding to the F value 402, and outputs the calculated depth of field 402 as setting information. Further, the image editing unit 703 reconstructs the LFRAW image according to the setting information so that the focus information is in the distance 3 and the range of the depth of field 402 is in focus, and edits the image data for display. The display image data is image data that conforms to a data format of a display device (not shown). The edited display image data is output to the display unit 106.

  Since S503 to S508 are the same as those in the first embodiment, description thereof is omitted.

  In step S909, the setting adjustment unit 702 confirms whether or not F value information is received from the aperture adjustment unit 701. If the F value information has been received, S910 is executed, and if it has not been received, S911 is executed.

  In step S <b> 910, the setting adjustment unit 702 calculates the depth of field from the F value information received from the aperture adjustment unit 701 and outputs the depth of field to the image editing unit 703. In the present embodiment, for example, when the user performs an operation of changing the F value to the F value 401, the aperture adjustment unit 701 outputs the F value 401 to the setting adjustment unit 702. The setting adjustment unit 702 calculates the depth of field from the F value 401 and outputs the depth of field 401 to the image editing unit 703.

  In step S <b> 902, the image editing unit 703 reconstructs the LFRAW image so that the range of the distance 7 and the depth of field 401 is in focus and edits the image data for display.

  If mode switching is not instructed in S911, S504 is executed. Thereafter, S902 to S91111 are repeatedly executed until a mode switching instruction is detected by the mode instruction unit 101 in S911. If a mode switching instruction is detected by the mode instructing unit 101 in S911, the control unit for the entire light field camera (not shown) performs a reproduction mode end process. In S912, the operation in the present playback mode is terminated.

  As described above, in this embodiment, the focus information output from the focus lens moving unit 102 is changed from the distance 3 to the distance 7, and the F value information output from the aperture value adjusting unit is changed from the F value 402 to the F value 401. The image playback operation has been described. However, the change of the focus information can be set for the distance 4 where there is no subject, for example, as in the first embodiment. The setting change of the F value information is the same as that of the focus information.

  According to the present embodiment, the reproduction mode focus lens moving unit 102 (focus adjustment unit), the shutter button unit 105, and the aperture adjustment unit 701 can be used in both the photographing mode and the reproduction mode. As a result, it is possible to reconstruct an image adjusted to the desired focus from the LFRAW image with an operational feeling during shooting in the playback mode.

  Next, a third embodiment of the present invention will be described with reference to FIGS. 3, 9, 11, 12, 13, 14, and 15. FIG. This embodiment is an example of an image reproduction operation performed in the first or second embodiment when the imaging apparatus includes a camera body and a focus lens moving unit that can be attached to and detached from the camera body. Here, for example, a light field imaging device having a detachable focus lens moving unit is, for example, an imaging device having a lens unit that is integrated with the focus lens moving unit and is detachable. 11 to 15, the same parts as those in the first and second embodiments are denoted by the same reference numerals. Since FIG. 3 is the same as that of the first and second embodiments, description thereof is omitted here. Also for FIG. 9, operations different from those of the second embodiment will be described, and other operations are the same as those of the second embodiment, and thus description thereof is omitted here.

  FIG. 11 is a configuration diagram of a main part of an imaging apparatus that is a light field camera to which the image reproducing apparatus according to the present embodiment is applied. In the figure, reference numerals 101, 104, 105, 106, 701, and 703 are the same as those in the second embodiment, and the description thereof is omitted.

  Reference numeral 1100 denotes a camera body. Reference numeral 1101 denotes a lens unit that can be attached to and detached from the camera body 1100. Reference numeral 1102 denotes a focus lens moving unit integrated with the lens unit 1101. A setting adjustment unit 1103 receives focus information from the focus lens moving unit 1102 and F-number information from the aperture adjustment unit 701, calculates a focus position and a depth of field, and outputs the setting information to the image editing unit 703. It is. The setting adjustment unit 1103 outputs a warning instruction to the display unit 1104 described later when the lens unit 1101 is a lens unit having different characteristics in the shooting mode and the reproduction mode. The setting adjustment unit 1103 holds information on the focusable range. Here, the focusable range is a value indicating a range that can be set when the focus position is changed. A display unit 1104 displays display image data on a display device (not shown). When receiving a warning instruction from the setting adjustment unit 1103, the display unit 1104 combines the warning image with the display image data and displays it on the display device.

  FIG. 12 is a diagram illustrating an example of a warning screen displayed when lenses having different focus characteristics are attached. In the figure, the portions denoted by reference numerals 105, 701, and 1100 correspond to the portions denoted by the same reference numerals in FIG. In FIG. 12, reference numeral 1200 denotes a display device (not shown), on which display image data is displayed by the display unit 106. A warning image 1201 is combined with display image data by the display unit 1104 when a warning instruction is output from the setting adjustment unit 1103.

  FIG. 13 is a histogram of subject detection frequency for each distance according to the present embodiment, where the horizontal axis represents the distance that can be set by the focus lens moving unit 1102, and the vertical axis represents the appearance frequency. The difference from the histogram of FIG. 3 is that the distance that can be set is normalized in this embodiment.

  In FIG. 13, reference numerals 301 to 304 denote objects indicated by the same reference numerals in FIG. The distance (a), the distance (b), and the distance (c) are scales obtained by normalizing the distance setting of the focus lens moving unit 1102 having different characteristics according to the actual distance of the LFRAW image. In the present embodiment, the scale according to the characteristics of the lens that captured the LFRAW image is set as the distance (b), and the distance is set in 16 steps. By normalization, it is assumed that the lens of distance (a) has eight levels and the lens of distance (c) has 32 levels. The focusable range corresponding to each distance is as follows: distance (a) is focus range (a) from distance 0 to distance 7, distance (b) is focus range (b), distance 0 to distance 15, and distance (c) is The distance is 0 to 31 in the focal range (c).

  Next, with reference to FIG. 9, FIG. 14, and FIG. 15, the operation in the reproduction mode of the imaging apparatus according to the present embodiment will be described. The operation of this embodiment is the same as the operation according to the flowchart of FIG. 9 of the second embodiment, but the process of S901 is different from that of the second embodiment.

  14 and 15 show a detailed flowchart of the image initial value acquisition process (focus position normalization process) in S901 of FIG. Playback mode

  In step S1400, the focus position initialization operation is started. In step S1401, the setting adjustment unit 1103 acquires characteristic information of the attached lens. Here, the characteristic information of the attached lens is individual information that allows the type of the lens unit 1101 currently attached to the camera body 1100 to be identified.

  In step S1402, the setting adjustment unit 1103 acquires characteristic information of the photographing lens. Here, the characteristic information of the photographing lens is individual information that can identify the type of the lens unit 1101 attached to the camera body 1100 when photographing the LFRAW image.

  In step S1403, the setting adjustment unit 1103 compares the characteristic information of the mounted lens with the characteristic information of the photographing lens. If both are the same, the process branches to S1404; otherwise, the process branches to S1410.

  In step S1410, the setting adjustment unit 1103 acquires a focus range (a) that can be realized by the focus lens moving unit 1102 of the lens unit 1101 currently mounted. In step S1411, the setting adjustment unit 1103 acquires a focus range (b) that can be realized by the focus lens moving unit 1102 of the lens unit 1101 that was attached when the LFRAW image was captured. In step S1412, the setting adjustment unit 1103 compares the focal range (a) and the focal range (b). If they are the same, the process branches to S1404, and if they are not the same, the process branches to S1413.

  In A1404, the image editing unit 703 acquires the initial focus position from the LFRAW image. For example, it is assumed that the distance 3 that is the subject 301 is set as the initial focus position.

  In S1413, focus adjustment processing is performed. Here, the focus adjustment processing means that the setting adjustment unit 1103 normalizes the focus range, acquires the initial focus position, generates focus information, and transmits the focus information to the image editing unit 701.

  The processing in S1413 will be described using the flowchart shown in FIG.

  In S1500, the focus adjustment process is started. In step S1501, the setting adjustment unit 1103 normalizes the focal range (b) to the focal range (a). For example, the focal range (a) corresponds to the distance (a), and the focal range (b) corresponds to the distance (b). The distance of each subject at the focal length (b) before normalization is 3 for the subject 301, 7 for the subject 302, 9 for the subject 303, and 12 for the subject 304. The distance of each subject after normalization to the focal length (a) is 1 for the subject 301, 3 for the subject 302, 4 for the subject 303, and 6 for the subject 304.

  In step S1502, the setting adjustment unit 1103 acquires the initial focus position from the LFRAW image. For example, it is assumed that the distance 3 of the subject 301 that is the focal position at the time of shooting is set as the initial focus position. In step S <b> 1503, the setting adjustment unit 1103 normalizes the acquired distance 3 to set the initial focus position as the distance 1. In step S1504, the focus adjustment process ends.

  Returning to the flowchart of FIG. 14, in step S <b> 1414, the setting adjustment unit 1103 outputs a warning instruction to the display unit 1104.

  In step S <b> 1405, the focus lens moving unit 1102 sets the focus information output to the setting adjustment unit 1103 to distance 1.

  Next, in S1406, the focus lens moving unit 1102 sets the focus position to the distance 1, and in S1406, the focus position initialization process ends.

  If the display unit 1104 has received a warning instruction after the focus position initialization process of the present embodiment described above, the warning image is changed to a display image as shown in 1201 in FIG. Synthesize.

  In S508, the focus adjustment is performed in S1413 to normalize the focus position. For example, even when the focal range (b) is larger than the lens unit 1101 in the reproduction mode and the number of steps of the distance that can be set in the lens unit 1101 in the shooting mode is larger than the lens unit 1101 in the reproduction mode, as shown in FIG. Reproduction is possible by performing normalization in the same manner.

  The aperture adjustment unit 701 can also be integrated with the lens adjustment unit 701 by performing normalization in the same manner. In this case, the setting adjustment unit 1103 may be included in the lens unit 1101.

  In the above description, the example in which the present embodiment is applied to the second embodiment (FIG. 9) has been described. However, the present embodiment may be applied to the first embodiment (FIG. 5).

  As described above, according to the present embodiment, in a light field camera having a lens unit in which the focus lens moving unit and the lens are integrated and detachable, a desired focus image can be generated without impairing the operational feeling during shooting. . In particular, even when the focus lens moving unit has different characteristics in the shooting mode and the playback mode, it is possible to combine the focus lens moving unit 102 and the shutter button unit 105 without impairing the operational feeling during shooting in both modes. Become.

  A fourth embodiment of the present invention will be described below with reference to FIGS. 3, 9, 11, 12, 14, 16, and 17. FIG. In this embodiment, in a light field imaging device having a detachable focus lens moving unit, even if the characteristics of the focus lens moving unit are different between the shooting mode and the playback mode, the LFRAW image is reproduced without normalizing the focus position. Is an example of 16 and 17, the same parts as those in the first and second embodiments are denoted by the same reference numerals. 3, 11, and 12 are the same as those in the third embodiment, and thus the description thereof is omitted here. 9 and 14, operations different from those of the third embodiment will be described, and the other operations are the same as those of the third embodiment, and thus description thereof will be omitted here.

  FIG. 16 shows a subject detection frequency histogram for each distance according to the present embodiment. In the figure, the horizontal axis represents the distance that can be set by the focus lens moving unit, and the vertical axis represents the appearance frequency. Reference numerals 301 to 304 correspond to subjects indicated by the same reference numerals in FIG. The distance (d), the distance (e), and the distance (f) are scales representing the distance setting of the focus lens moving unit 1102 having different characteristics.

  In this embodiment, the distance (e) is a scale according to the characteristics of the lens unit that captured the LFRAW image. The distance (e) is set in 16 steps. The distance (d) lens has 8 stages, and the distance (f) lens has 32 stages. Further, the focusable range corresponding to each distance is the distance 0 to the distance 7 at the distance (d), the distance 0 to the distance 7 at the distance (e), the distance 0 to the distance 15 from the distance (e), and the distance (f). The distance is 0 to 31 in the focal range (f).

  FIG. 17 shows a detailed flowchart of the focus adjustment operation in S1413 of FIG.

  First, with respect to the case where the lens unit in the focal range (e) is mounted in the shooting mode and the lens unit in the focal range (d) is mounted in the reproduction mode, this embodiment will be described with reference to FIGS. The image playback operation in the playback mode will be described.

  In step S1410, the setting adjustment unit 1103 acquires a focus range (d) that can be realized by the focus lens moving unit 1102 of the lens unit 1101 mounted in the playback mode. In step S1411, the setting adjustment unit 1103 acquires a focus range (e) that can be realized by the focus lens moving unit 1102 of the lens unit 1101 that was attached when the LFRAW image was captured. In step S1412, the setting adjustment unit 1103 compares the focal range (d) and the focal range (e). If they are the same, the process branches to S1404, and if they are not the same, the process branches to S1413.

  Here, the S1413 focus adjustment operation will be described with reference to FIG.

  In step S1700, focus adjustment is started. In step S <b> 1701, the setting adjustment unit 1103 compares the focal range (d) of the mounted lens unit with the focal range (e) of the lens unit used in the LFRAW image, and determines which range is narrower. . Here, since the focal range (d) is narrower, the process branches to S1702. In S1702, the setting adjustment unit 1103 sets the focusable range as the focus range (d).

  In step S <b> 1703, the setting adjustment unit 1103 acquires the distance 3 that is the initial focus position of the LFRAW image. In step S1704, the setting adjustment unit 1103 determines whether the initial focus position is included in the focal range (d) of the lens being mounted. Since it is included here, the process branches to step S1705, and the focus adjustment ends in step S1705. .

  Next, referring to FIGS. 9, 14, and 17, the lens unit in the focal range (e) in the shooting mode and the lens unit in the focal range (f) in the playback mode are mounted. The focus adjustment operation reproduction mode in which the focal position is not normalized will be described.

  In step S1410, the setting adjustment unit 1103 acquires a focus range (f) that can be realized by the focus lens moving unit 1102 related to the lens unit 1101 attached in the playback mode. In step S1411, the setting adjustment unit 1103 acquires a focus range (e) that can be realized by the focus lens moving unit 1102 related to the lens unit 1101 that was attached when the LFRAW image was captured. Next, in step S1412, the setting adjustment unit 1103 compares the focal range (f) and the focal range (e). If they are the same, the process branches to S1404, and if they are not the same, the process branches to S1413.

  Here, the S1413 focus adjustment operation will be described with reference to FIG.

  Focus adjustment is started in S1700.

  In S1701, the setting adjustment unit 1103 branches to S1710 because the focal range (e) is narrower as a comparison result of the focal range. In step S1710, the setting adjustment unit 1103 sets the focusable range as the focus range (e).

  In step S1703, the setting adjustment unit 1103 acquires the distance 3 as the initial focus position from the LFRAW image. In step S1704, the setting adjustment unit 1103 determines whether the initial focus position is included in the distance (e) that is the focal range of the currently mounted lens unit. If included, the process branches to step S1705. If it is not included, the process branches to S1720. Since it is included here, the process branches to S1705.

  Although not branched in this embodiment, in S1720, the focus lens moving unit 1102 sets the maximum value of the focal range (d) of the lens unit mounted as the initial focus position. This corresponds to, for example, the distance 7 in the focal range (d). However, the initial focus position is not limited to the maximum value of the focus range, and may be an arbitrary value.

  Thereafter, the focus adjustment is terminated in S1705.

  In S507 of FIG. 9, it is assumed that the focus information output from the focus lens moving unit is a distance exceeding the focusable range. For example, when the distance is 16 to 31. In this case, in step S508, the setting adjustment unit 1103 sets the focus information to the distance 15 that is the maximum value of the focusable range. That is, even if the setting of the distance 15 or more is set in the focus lens moving unit 1102, the display image data does not change.

  In this embodiment, the distance (d), the distance (e), and the distance (f) in FIG. 16 are associated with the position of the distance 0 as a reference. Good. In this case, the distance within a narrower range where the focusable range can be set is set.

  Further, by performing the same operation in the aperture adjustment unit 701, it is possible to integrate the lens unit 1101 with the aperture adjustment unit 701. In this case, the setting adjustment unit 1103 may be included in the lens unit 1101.

  In the above description, the example in which the present embodiment is applied to the second embodiment (FIG. 9) has been described. However, the present embodiment may be applied to the first embodiment (FIG. 5).

  As described above, according to the present embodiment, a desired focus image can be generated even in a light field camera in which the focus lens moving unit and the lens are integrated and removable. In particular, even when the focus lens moving unit has different characteristics in the shooting mode and the playback mode, the operational feeling at the time of shooting may be limited by the difference in characteristics, but in both modes, the focus lens moving unit 102 and the shutter button unit It becomes possible by combining 105.

  In the above-described embodiment, each process shown in the flowchart of the figure is realized by reading a program including a program code for realizing the function of each process from the memory and executing it by the CPU of the control unit 1218. It is.

  Note that the present invention is not limited to the above-described configuration, and all or a part of the functions shown in the flowchart of the drawing may be realized by dedicated hardware. The memory described above may be a non-volatile memory such as a magneto-optical disk device or a flash memory, a storage medium that can only be read such as a CD-ROM, or a volatile memory other than a RAM. Moreover, you may comprise from the computer-readable / writable storage medium by those combination.

  Further, a program for realizing the function of each process shown in the flowchart of the figure is recorded in a computer-readable storage medium, and the program recorded in the storage medium is read into a computer system and executed. Processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices. Specifically, the program read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  The “computer-readable storage medium” refers to a storage device such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, and a hard disk built in a computer system. Furthermore, the “computer-readable storage medium” includes a medium that holds a program for a certain period of time. For example, it includes a volatile memory (RAM) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  A program product such as a computer-readable recording medium in which the above program is recorded can also be applied as an embodiment of the present invention. The above program, recording medium, transmission medium, and program product are included in the scope of the present invention.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

Claims (17)

In an image reproducing apparatus that reconstructs refocusable pixel data and reproduces an image,
An instruction means for obtaining an instruction to reproduce the image;
Setting means for setting information necessary for reconstruction of the pixel data;
Update means for acquiring information for updating information necessary for reconfiguration set by the setting means;
In accordance with the information set by the setting means, the image forming apparatus includes a generation unit that reconstructs the pixel data and generates an image.
The setting unit sets initial information of information necessary for the reconstruction when the instruction to reproduce the image is acquired by the instruction unit, and when the update unit acquires information for the update. Sets information necessary for the reconfiguration based on the acquired information for update, and when the setting means sets information necessary for the reconfiguration, the generation unit sets the information necessary for the reconfiguration according to the set information. An image reproducing apparatus that reconstructs the pixel data to generate an image.   The information necessary for the reconstruction is any one of a focus position and a depth of field of the image generated by the generation unit, and the information for updating is from either the focus adjustment unit or the aperture adjustment unit. 2. The image reproducing apparatus according to claim 1, wherein the setting information is information on an operation to be acquired, and the setting unit calculates the depth of field based on the operation information acquired from the aperture adjustment unit. .   3. The image reproducing apparatus according to claim 2, wherein the updating unit initializes the focus adjusting unit according to the initial information of the focal position when the setting unit sets initial information of the focal position. .   4. The image reproduction device according to claim 1, wherein the setting unit acquires the initial information from the pixel data or information preset in the image reproduction device. 5.   The update means further acquires characteristic information of the photographing lens and characteristic information of the focus adjusting means related to the pixel data, and the setting means is based on the characteristic information of the photographing lens and the characteristic information of the focus adjusting means. 5. The information required for the reconstruction is set, and a warning instruction is generated according to a result of comparison between the characteristic information of the photographing lens and the characteristic information of the focus adjusting unit. The image reproducing device according to one item.   6. The image reproducing apparatus according to claim 5, wherein the setting unit sets the focal position by normalizing characteristic information of the photographing lens with characteristic information of the focus adjusting unit.   The said setting means compares the focus range of the said imaging lens with the focus range of the said focus adjustment means based on the acquired characteristic information, and sets the said focus position according to the comparison result. The image reproducing device described in 1.   8. The image reproducing apparatus according to claim 1, further comprising a recording unit that acquires a recording instruction from the recording instruction unit and stores the generated image in accordance with the recording instruction. In an imaging apparatus for generating pixel data that can be refocused by capturing an optical image of a subject formed by a photographing optical system,
Mode instruction means for instructing an operation mode of the imaging apparatus;
Drive means for driving the photographing optical system for focus adjustment when the mode instruction means designates a photographing mode;
Shooting instruction means for instructing shooting of the subject when the mode instruction means indicates a shooting mode;
The image reproduction device according to any one of claims 1 to 8, wherein when the mode instruction unit instructs a reproduction mode, the pixel reproduction data is reconstructed to generate an image.
The operation information of the focus adjustment unit and the recording instruction unit acquired in the image reproduction apparatus is information of the operation of the driving unit and the imaging instruction unit when the reproduction mode is instructed. apparatus.   A diaphragm setting unit for setting a diaphragm of the photographing optical system is further provided, and information on operation of the diaphragm adjustment unit acquired in the image reproduction apparatus includes information on operation of the diaphragm setting unit when the reproduction mode is instructed. The imaging apparatus according to claim 9, wherein the imaging apparatus is information.   The photographing optical system includes a lens unit that can be attached to and detached from the imaging device, and the photographing lens characteristic information and the focus adjustment unit characteristic information acquired by the image reproducing device each indicate the photographing mode. The imaging apparatus according to claim 9, wherein the imaging apparatus includes characteristic information of a lens unit attached to the lens and characteristic information of a lens unit attached when the reproduction mode is instructed.   The image processing apparatus further includes display means for displaying an image generated by the image reproduction apparatus, wherein the display means displays the warning image combined with the image in accordance with the warning instruction generated by the image reproduction apparatus. The imaging device according to any one of claims 9 to 11. In an image reproduction method for reconstructing refocusable pixel data and reproducing an image,
An instruction step for obtaining an instruction to reproduce the image;
A setting step for setting information necessary for reconstruction of the pixel data;
An update step for obtaining information for updating information necessary for reconfiguration set by the setting means;
In accordance with the information set in the setting step, comprising a generation step of reconstructing the pixel data and generating an image,
In the setting step, when an instruction to reproduce the image is acquired in the instruction step, initial information of information necessary for the reconstruction is set, and information for the update is acquired in the update step. When the information necessary for the reconfiguration is set based on the acquired information for update, the generation step is set when the information necessary for the reconfiguration is set in the setting step. An image reproduction method, wherein the image is generated by reconstructing the pixel data according to information. A program for controlling an image playback device that reconstructs refocusable pixel data and plays back an image,
Computer
Instruction means for obtaining an instruction to reproduce an image;
Setting means for setting information necessary for reconstruction of the pixel data;
Updating means for obtaining information for updating information necessary for reconfiguration set by the setting means;
In accordance with information set by the setting means, including program code that functions as generating means for reconstructing the pixel data and generating an image,
The program code sets the setting means, and when the instruction to reproduce the image is acquired by the instruction means, sets initial information of information necessary for the reconstruction, and the updating means When the information is acquired, the function for generating and setting the information necessary for the reconfiguration based on the acquired information for update is set, and the generation unit is configured to provide information necessary for the reconfiguration. A program code for causing the pixel data to be reconstructed according to the set information and functioning to generate an image.   A computer-readable storage medium storing the program according to claim 14.   A program that causes a computer to function as each unit of the image reproduction device according to any one of claims 1 to 8.   A storage medium storing a program according to any one of claims 1 to 8 for causing a computer to function as each unit of an image reproducing device.

Images