WO2016071973A1 - Image capture apparatus, image capture apparatus control method, image capture apparatus control program, and image file generation method - Google Patents

Abstract

In a technique for acquiring a proper image by performing an image processing of an intermediate image to which a given optical modulation has been imparted by an optical system, the present invention makes it possible to identify an image capture mode of an image capture apparatus from the data of an image captured by the image capture apparatus and/or to appropriately perform the image processing of the intermediate image. An image capture apparatus of the present invention is characterized in that the image capture apparatus comprises: an image capture optical system that includes an optical modulation element and that acquires an intermediate image optically modulated by the optical modulation element; and an image information generation unit that generates an image file including both the intermediate image and information associated with the intermediate image, wherein said information associated with the intermediate image includes: identification information for identifying an image capture mode of the image capture optical system including the optical modulation element; and image processing information related to an image processing for demodulating the optically modulated intermediate image.

Description

IMAGING DEVICE, IMAGING DEVICE CONTROL METHOD, IMAGING DEVICE CONTROL PROGRAM, AND IMAGE FILE GENERATION METHOD

The present invention relates to an imaging apparatus, an imaging apparatus control method, an imaging apparatus control program, and an image file generation method.

As background art of this technical field, there are US Pat. No. 5,748,371 (Patent Document 1), US Patent Application Publication No. 2007/0258706 (Patent Document 2), and Japanese Patent No. 2963990 (Patent Document 3). . In Patent Document 1, "A system for increasing the depth of field of an incoherent optical system and reducing lens aberrations that cause sensitivity to wavelength and defocus is a non-coherent optical mask. It is introduced to the system. " Japanese Patent Application Laid-Open No. 2004-228561 describes that “shutter opening / closing at the time of exposure at the time of acquiring one image is encoded and blurring is corrected by an arithmetic process using the information on the shutter opening / closing”. Further, in Patent Document 3, “the aperture mask 1 has two pinholes 6 and 7 as a coded aperture (pupil) shape structured so as to facilitate the analysis of the blur amount. , 8, and 9 to estimate the in-focus position, and the aperture mask 1 and the lens system 2 are telecentric in which the image magnification between the three images obtained by the CCDs 7, 8, and 9 is equal. It constitutes an optical system. "

US Pat. No. 5,748,371 US Patent Application Publication No. 2007/0258706 Japanese Patent No. 2963990

The background art described above is called computational photography. Computational photography technology is premised on image processing after shooting by an imaging device such as a camera, and the image desired by the user is an intermediate image acquired by the optical system by light information and image processing determined from the optical system of the imaging device. Correct to. In other words, the computational photography technique generates an intermediate image by applying a predetermined optical modulation to the image in the optical system (in the case of a focal depth expansion technique, for example, “blur” with respect to an image in a predetermined range as the optical modulation). The intermediate image is subjected to image processing for demodulating (or restoring) the optical modulation to obtain a desired image. This intermediate image is intermediate representation data including optical information (usually lost) inside the camera (preferably having the maximum amount of optical information), such as image processing for correcting optical characteristics. By the information processing, for example, it is possible to correct to a plurality of images with different depths of focus (focus) in the imaging range.

In order to correct such an intermediate image to an image desired by the user by image processing, a method of computational photography constituting an optical system for acquiring the intermediate image is specified and used for image processing according to the method. Parameter to be calculated. Here, for example, when an image processing device such as a personal computer (PC) is connected to the imaging device, and the image processing device acquires an image from the imaging device and observes the intermediate image from the imaging device. Even if the information is acquired, it cannot be corrected to a desired image unless information necessary for image processing (for example, a method of computational photography) is obtained.

In order to solve such a problem, it is desired that the imaging apparatus has a function for appropriately performing image processing of intermediate images in the image processing apparatus. On the other hand, depending on the method of computational photography, mounting such a function for each imaging device may complicate the functions that make up the imaging device or increase the workload for mounting. This is not preferable.

The present invention has been made in view of such a problem, and an image captured by an imaging device in a technique for obtaining a predetermined image by image processing on an intermediate image given a predetermined optical modulation by an optical system. Therefore, it is possible to identify the image pickup method (optical modulation method) of the image pickup apparatus from the data and / or to appropriately perform image processing on the intermediate image.

In order to solve the above-described problems, the present invention adopts, for example, the configurations described in the claims. The present application includes a plurality of components that solve the above-described problems. For example, an imaging optical system that includes an optical modulation element and acquires an intermediate image optically modulated by the optical modulation element And an image information generation unit that generates an image file including the intermediate image and accompanying information of the intermediate image, the accompanying information identifying an imaging method of an imaging optical system including the optical modulation element And image processing information related to image processing for demodulating the optically modulated intermediate image.

According to the present invention, it is possible to identify the imaging method (optical modulation method) of the imaging device from the data of the image captured by the imaging device, and / or to appropriately perform image processing on the intermediate image. it can. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is a block diagram which illustrates the functional composition of the imaging device concerning the embodiment of the present invention. It is a figure which illustrates the structure of the imaging optical system which concerns on embodiment of this invention. It is a block diagram which illustrates the functional composition of the controller concerning the embodiment of the present invention. It is a figure which illustrates the system information stored in the system information storage part which concerns on embodiment of this invention. It is a figure which illustrates the parameter information stored in the parameter information storage part which concerns on embodiment of this invention. It is a figure which illustrates the structure of the compression image file for the comparison with the image file which the image file generation part which concerns on embodiment of this invention produces | generates. It is a figure which illustrates the structure of the image file which the image file generation part which concerns on embodiment of this invention produces | generates. 3 is a flowchart illustrating an example of the entire operation of the imaging apparatus according to the embodiment of the present invention. 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to an embodiment of the present invention. 3 is a flowchart illustrating an example of the overall operation of the image processing apparatus according to the embodiment of the present invention. It is a figure which illustrates the structure which further contains compression image data in the image file which the image file generation part which concerns on embodiment of this invention produces | generates. It is a figure which illustrates the structure of the image file which the image file generation part which concerns on embodiment of this invention produces | generates.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a description will be given by taking as an example an imaging apparatus such as a digital video camera and a digital camera that captures a moving image or a still image.

FIG. 1 is a diagram illustrating a functional configuration of the imaging apparatus 1 according to the present embodiment. As illustrated in FIG. 1, the imaging apparatus 1 includes an imaging optical system 110, an optical system control unit 120, a controller 130, an SP (Serial / Parallel) conversion unit 101, a buffer memory 102, an optical characteristic correction processing unit 103, and an image processing unit. 104, a display unit 105, a display control unit 106, an operation unit 107, and an operation control unit 108.

FIG. 2 is a diagram illustrating the configuration of the imaging optical system 110. For example, the imaging optical system 110 illustrated in FIG. 2A includes a lens 111, a diaphragm 112, a phase plate 113, and an image sensor 114. For example, the imaging optical system 110 illustrated in FIG. 2B includes a lens 111, a diaphragm 112, a microlens array 115, and an image sensor 114. In addition, the structure to which the same code | symbol is attached | subjected in Fig.2 (a) and (b) has the same function.

The imaging optical system 110 of the imaging apparatus according to the present embodiment is configured according to a computational photography system (hereinafter referred to as “imaging system”) employed in the imaging apparatus 1. For example, when Wave Front Coding (hereinafter referred to as “WFC”) that expands the depth of field of the imaging system is adopted as an imaging method, the lens module surrounded by a broken line in FIG. In addition, a lens 111, a stop 112, and a cubic function type phase plate 113 for providing “blur” as optical modulation to an image in a predetermined range are incorporated. The phase plate 113 is incorporated in the lens module as an interchangeable lens for WFC. The optical modulation may be applied by the diaphragm 112 or a combination of the diaphragm 112 and the phase plate 113.

In addition, for example, when Light Field Camera (hereinafter referred to as “LFC”) that changes the imaging distance (focus) and depth of field is adopted as the imaging method, it is surrounded by a broken line in FIG. In the lens module, a lens 111, an aperture 112, and a microlens array 115 for providing optical modulation are incorporated. Again, the optical modulation may be applied by the diaphragm 112 or a combination of the diaphragm 112 and the phase plate 113.

The image sensor 114 photoelectrically converts the light beam that has passed through the lens 111, the diaphragm 112, and the phase plate 113 to generate a digitized electrical signal for each pixel, and outputs it to the SP conversion unit 101. The image sensor 114 is, for example, a CCD (Charge Coupled Device) or a CMOS sensor (Complementary Metal Oxide Semiconductor Sensor).

The SP conversion unit 101 converts the electrical signal for each pixel input from the image sensor 114 from a serial signal to a parallel signal, and stores it in the buffer memory 102 as intermediate image data (RAW data) acquired by the imaging optical system 110. Remember me. The buffer memory 102 temporarily stores the intermediate image data input from the SP conversion unit 101 and outputs the intermediate image data to the optical characteristic correction processing unit 103 and the controller 130 as necessary.

Intermediate image data is settable data generated by the imaging optical system 110 and capable of setting the depth of focus in the imaging range by image processing. The intermediate image data is, for example, a state in which the entire image is blurred, and is not necessarily an image that looks good for the user.

The optical characteristic correction processing unit 103 corrects the intermediate image data stored in the buffer memory 102 to image data focused on a certain viewpoint, that is, image processing for demodulating the given optical modulation. Do. Hereinafter, such image processing is referred to as “optical characteristic correction processing”. Details of the optical characteristic correction processing unit 103 will be described later.

The optical system control unit 120 includes a shutter control unit 121, a sensor position control unit 122, a lens driving unit 123, an aperture control unit 124, and a zoom control unit 125. The optical system control unit 120 controls each component of the imaging optical system 110 illustrated in FIG. 2 in accordance with an imaging operation using the operation unit 107 described later.

Specifically, for example, the shutter control unit 121 controls the shutter function of the image sensor 114 according to the imaging operation, and the sensor position control unit 122 controls the position of the image sensor 114 according to the imaging operation. The lens driving unit 123 drives the lens 111 according to the imaging operation, the aperture control unit 124 controls the aperture 112 according to the imaging operation, and the zoom control unit 125 controls the lens 111 according to the imaging operation. Control the zoom function.

Each unit of the optical system control unit 120 controls each component of the imaging optical system 110 by a control parameter value for controlling each component of the imaging optical system 110. For example, the sensor position control unit 122 controls the position of the image sensor 114 based on a control parameter value indicating the position information of the image sensor 114. The optical system control unit 120 outputs control parameter values used in the respective units of the optical system control unit 120 set in accordance with the imaging operation as described above to the controller 130 as imaging information.

The controller 130 plays a role of controlling each unit such as the optical system control unit 120, the optical characteristic correction processing unit 103, the image processing unit 104, and the operation control unit 108, which will be described later, and gives commands to each unit. Further, the controller 130 specifies an imaging method based on information related to the configuration of the imaging optical system 110 (hereinafter referred to as “optical information”). Further, the controller 130 calculates a parameter value used when the intermediate image data is subjected to the optical characteristic correction processing based on the optical information and the imaging information input from the optical system control unit 120. Details of the imaging method specifying process and the parameter value calculating process by the controller 130 will be described later.

The parameter value calculated by the controller 130 is, for example, a point spread function (PSF) representing the blurring or blurring of the image. In addition, the parameter values calculated by the controller 130 are a group of parameter values for each viewpoint used for focusing at any viewpoint when performing optical characteristic correction processing on the intermediate image data, for example.

The optical characteristic correction processing unit 103 performs optical characteristic correction processing on the intermediate image data using any one of the parameter values calculated by the controller 130 by an arithmetic method corresponding to the imaging method. Do. For example, when “WFC” is adopted as the imaging method, the optical characteristic correction processing unit 103 selects convolution calculation using a spatial filter as the calculation method, and uses the coefficient numerical data of the spatial filter that is a parameter value. To correct.

Also, for example, when “LFC” is adopted as the imaging method, the optical characteristic correction processing unit 103 selects the shift addition method as the calculation method, and corrects it using the parameter value. That is, the group of parameter values is a value for calculation used in the calculation by the calculation method. There are a plurality of LFC calculation methods such as a shift addition method and a light beam projection method. In this embodiment, the shift addition method having a lower calculation cost than the other methods is selected. Further, when there are a plurality of calculation methods, which calculation method is used may be selected by a user operation.

The image processing unit 104 performs image processing such as demosaicing, gamma correction, color processing, resolution conversion, compression processing, and the like on the image data corrected by the optical characteristic correction processing unit 103, JPEG (Joing Photographic Experts Group), etc. Are generated and output to the display control unit 106. That is, the optical characteristic correction processing unit 103 and the image processing unit 104 function as an image data generation unit that generates image data set to a predetermined depth of focus from the intermediate image data.

The display unit 105 is an output interface that visually displays an image based on the state of the imaging apparatus 1 or a compressed image input from the image processing unit 104, such as a liquid crystal display panel. The display control unit 106 outputs an image signal based on the compressed image to the display unit in order to display an image on the display unit 105. For example, the display control unit 106 decodes (decompresses) the compressed image input from the image processing unit 104 by an expansion method corresponding to the compression method (for example, JPEG), and outputs the decoded image to the display unit 105. Further, when the display unit 105 displays an image, the image processing unit 104 may not perform compression. In this case, the decoding process in the display control unit 106 is unnecessary. In this way, the compressed image generated through the processing by the optical characteristic correction processing unit 103 and the image processing unit 104 from the intermediate image data is displayed on the display unit 105, so that the user can determine whether a desired image has been generated. Can be confirmed.

The operation unit 107 is an input interface for operating the imaging apparatus 1 such as a camera shutter button. The operation control unit 108 notifies the controller 130 of operation information performed via the operation unit 107. For example, when the user presses a shutter button of a camera that is the operation unit 107, control parameter values of the respective units of the optical system control unit 120 at that timing are set as imaging information. When the display unit 105 is a display panel or the like and can accept an input operation from the user as a touch panel, the display unit 105 may also function as the operation unit 107.

As described above, in the imaging apparatus 1, the optical characteristic correction processing unit 103 generates image data set to a certain depth of focus in the imaging range from the intermediate image data. Even if the image processing apparatus (for example, PC), which is a device different from the imaging apparatus 1, acquires the image data generated in this way from the imaging apparatus 1, the image processing apparatus (for example, a PC) corrects the image data to be focused from different viewpoints. I can't. This is because the acquired image data is already focused at a certain viewpoint in the imaging apparatus 1, and if an optical characteristic correction process is further performed, artifacts (false images) or the like are generated in the image.

In addition, even if only the intermediate image data is acquired from the imaging device 1, the image processing device generates an image focused at a desired viewpoint if information necessary for performing the optical characteristic correction processing cannot be obtained. I can't. Therefore, even if the intermediate image data is acquired in the imaging device 1 and a group of parameter values for focusing at any viewpoint is calculated, the image processing apparatus can generate an image focused at a desired viewpoint. The intermediate image data cannot be fully utilized.

The controller 130 according to the present embodiment uses intermediate image data generated in the imaging apparatus 1 in an image processing apparatus such as a PC that is used by being connected to the imaging apparatus 1 and is a device different from the imaging apparatus 1. Process as you can. Such processing is one of the gist according to the present embodiment. Hereinafter, details of the processing of the controller 130 will be described.

FIG. 3 is a block diagram illustrating a functional configuration related to processing for using the intermediate image data in the image processing apparatus among the functions of the controller 130. As shown in FIG. 3, the controller 130 includes an intermediate image data acquisition unit 131, an imaging information acquisition unit 132, an optical information acquisition unit 133, a method information storage unit 134, a method identification unit 135, a parameter information storage unit 136, and a parameter value calculation. A unit 137, a demodulation information acquisition unit 138, an image file generation unit 139, and an image file storage unit 141.

The intermediate image data acquisition unit 131 acquires the intermediate image data stored in the buffer memory 102 and outputs it to the image file generation unit 139. The imaging information acquisition unit 132 acquires imaging information from the optical system control unit 120 described above and outputs it to the parameter value calculation unit 137. That is, the imaging information acquisition unit 132 functions as a control information acquisition unit that acquires control information (imaging information) for the imaging optical system 110 when generating the intermediate image data.

The optical information acquisition unit 133 acquires optical information and outputs it to the method specifying unit 135 and the parameter value calculation unit 137. The optical information includes, for example, component information indicating the type of optical component attached to the lens module according to the imaging method, feature information indicating the characteristics of the optical component, and setting information of the image sensor 114.

For example, in the case of the imaging optical system 110 shown in FIG. 2A, the component information is “phase plate”, the feature information indicates the cubic function coefficient (phase shape) of the phase plate, and the setting information of the image sensor 114 is The number of pixels and the pixel pitch are shown. It is assumed that the optical information is input in advance by the manufacturer of the imaging device 1 or the imaging optical system 110 or the user via the operation unit 107 and stored in an optical information storage unit (not shown).

FIG. 4 is a diagram exemplifying method information stored in the method information storage unit 134. As shown in FIG. 4, the method information stored in the method information storage unit 134 is a table in which component information is associated with the imaging method of the imaging optical system 110 having the optical component indicated by the component information. For example, as illustrated in FIG. 4, when the component information is “phase plate”, the imaging method of the imaging optical system 110 having the “phase plate” is WFC.

The method specifying unit 135 refers to the method information stored in the method information storage unit 134 and specifies the imaging method employed in the imaging apparatus 1. Specifically, for example, when the component information included in the optical information input from the optical information acquisition unit 133 indicates “phase plate”, the method specifying unit 135 refers to the method information as illustrated in FIG. Thus, it is specified that the imaging method is “WFC”. The method specifying unit 135 that specified the imaging method outputs identification information (hereinafter referred to as “method identifier”) for identifying the imaging method to the image file generation unit 139.

For example, the method identifier whose imaging method is “WFC” is “WF”, the method identifier whose imaging method is “LFC” is “LF”, and the method identifier whose imaging method is “Coded Expose” (encoded exposure) is “ “CE”, and the scheme identifier whose imaging scheme is “Coded Aperture” is “CD”. In addition, the format identifier may be in any format as long as the imaging scheme can be identified, such as defining a unique numerical value.

In the present embodiment, the case where the method specifying unit 135 outputs the method identifier to the image file generating unit 139 will be described as an example. However, this is merely an example, and if the image processing apparatus can identify or identify the imaging method, the method specifying unit 135 uses the specified name of the imaging method (for example, “WFC”) itself as the image file generation unit 139. May be output.

FIG. 5 is a diagram illustrating parameter information stored in the parameter information storage unit 136. As shown in FIG. 5, the parameter information stored in the parameter information storage unit 136 associates each imaging method with a list of control parameters necessary for calculating parameters used in the optical characteristic correction processing. Table.

For example, as shown in FIG. 5, when the imaging method is “WFC”, the necessary control parameters are, for example, “open F value”, “focal length”, “cubic function coefficient”, “number of pixels”, “pixels”. “Pitch”, “Sensor position” and “Kernel size”. For example, “open F value” is a control parameter value of the aperture controller 124, and “focal length” and “sensor position” are control parameter values of the sensor position controller 122. Further, for example, “cubic function coefficient” is feature information included in the optical information, and “number of pixels” and “pixel pitch” are the number of pixels and the pixel pitch of the image sensor 114 being used. Further, for example, “kernel size” is a kernel size used in the correction processing by the optical characteristic correction processing unit 103.

Note that the value of “kernel size” is stored as setting information of the imaging apparatus 1 in a setting information storage unit (not shown), for example. In addition, the value of “kernel size” may be input as setting information by the manufacturer of the imaging apparatus 1 or the imaging optical system 110 or the user via the operation unit 107.

The parameter value calculation unit 137 calculates a group of parameter values used in the optical characteristic correction process based on information input from each unit, and outputs the group to the image file generation unit 139. That is, the parameter value calculation unit 137 functions as a calculation value calculation unit that calculates a calculation value that is a group of parameter values. Specifically, first, the parameter value calculation unit 137 refers to the parameter information stored in the parameter information storage unit 136, and obtains a list of control parameters associated with the imaging method input from the method specifying unit 135. get.

Next, the parameter value calculation unit 137 acquires each control parameter value in the list of acquired control parameters. For example, the parameter value calculation unit 137 acquires the values of “open F value”, “focal length”, and “sensor position” from the imaging information input from the imaging information acquisition unit 132 and inputs from the optical information acquisition unit 133. The value of “cubic function coefficient”, which is characteristic information included in the optical information, and the “number of pixels” and “pixel pitch” included in the optical information are acquired. For example, the parameter value calculation unit 137 acquires the value of “kernel size” from a setting information storage unit (not shown).

Subsequently, the parameter value calculation unit 137 calculates a group of parameter values (for example, PSF) used in the above-described optical characteristic correction processing from the control parameter values acquired as described above, and sends them to the image file generation unit 139. Output.

The demodulation information acquisition unit 138 outputs at least one of the group of parameter values calculated by the parameter value calculation unit 137 to the optical characteristic correction processing unit 103. The image file generation unit 139 includes a method identifier input from the method specification unit 135, imaging information input from the imaging information acquisition unit 132, a group of parameter values input from the parameter value calculation unit 137, and an intermediate image data acquisition unit 131. An image file including the intermediate image data input from is generated. The storage processing unit 140 stores the image file generated by the image file generation unit 139 in the image file storage unit 141 that is a storage medium.

In this embodiment, the image file is assumed to be image information generated in accordance with a file format defined by a predetermined standard, for example, the Exif (Exchangeable Image File Format) standard. The Exif standard is a standard that defines image data, the type of accompanying information (metadata) related to the image data, the storage format of the accompanying information, and the like. That is, the image file generation unit 139 functions as an image information generation unit that generates image information. In an image file generated in accordance with the file format of the Exif standard, generally, image data that has been captured, thumbnail image data that is reference image data that is smaller in size than the image data, the date and time of image capture, the manufacturer name of the image capture device, etc. Metadata is stored in a predetermined format.

FIG. 6 shows an example of the file structure of an image file created in the file format defined in the Exif standard. In FIG. 6, “compressed image data” is compressed image data such as JPEG compressed by the image processing unit 104. Hereinafter, the image file including the compressed image data is referred to as a “compressed image file”. On the other hand, FIG. 7 is a diagram illustrating a file structure of an image file generated by the image file generation unit 139 according to the present embodiment.

As shown in FIG. 6, the compressed image file includes Exif (Exchangeable Image File Format) attached information (hereinafter referred to as “Exif information”) included in the header portion and compressed image data. Further, as shown in FIG. 6, the Exif information is composed of thumbnail image data (Thumbnail), Exif IFD (Image File Directory), and the like. In the Exif IFD, a MakerNote tag is defined, and information unique to each digital camera manufacturer is stored as the value of the MakerNote tag.

The image file generated by the image file generation unit 139 according to the present embodiment is composed of Exif information and intermediate image data included in the header portion as shown in FIG. That is, intermediate image data is stored in the image file in this embodiment instead of the compressed image data shown in FIG. Further, as shown in FIG. 7, the Exif IFD according to the present embodiment stores imaging information, a method identifier, and a parameter value (group) as a MakerNote tag value. That is, the Exif information according to the present embodiment includes identification information for identifying an imaging method (an optical modulation method) and image processing information (an imaging method is used for image processing for demodulating the optical modulation). For example, in the case of WFC, information including image processing information for removing blur and information for setting a depth of focus is included, and the image processing information demodulates optical modulation associated with system information. Image processing information related to the image processing to be performed is included. Further, as described above, this image processing information includes parameter information used for various calculations in image processing.

Further, as shown in FIG. 7, the Exif information according to the present embodiment stores thumbnail image data of compressed image data as in the case of FIG. The thumbnail image data in FIGS. 6 and 7 is generated from the compressed image data by the image processing unit 104, for example. In addition, instead of the method identifier or in addition to the method identifier, calculation method information indicating the calculation method selected according to the imaging method may be stored as the value of the MakerNote tag.

For example, when the imaging method is “WFC”, the calculation method information is “convolution calculation”, and when the imaging method is “LFC”, the calculation method information is “shift addition method” and “ray bundle projection method”. There are two. When there are a plurality of calculation method information, the calculation method information is not stored as the value of the MakerNote tag, and only the imaging method may be used. That is, the method identifier and the calculation method information are calculation specifying information that is information for specifying a calculation method used according to the imaging method for the optical characteristic correction processing.

In the present embodiment, the case where each storage unit such as the method information storage unit 134, the parameter information storage unit 136, and the image file storage unit 141 illustrated in FIG. . However, this is an example, and all or a part of each storage unit may be included in a device external to the imaging device 1, a server, and the like. In this case, each component that refers to information stored in each storage unit accesses external devices and servers via a network such as the Internet, and refers to the information stored in each storage unit. .

Next, an overall operation example of the imaging apparatus 1 according to the present embodiment will be described. FIG. 8 is a flowchart illustrating an example of the overall operation of the imaging apparatus 1 according to the present embodiment. As shown in FIG. 8, the optical information acquisition unit 133 acquires optical information (S801), the imaging information acquisition unit 132 acquires imaging information (S802), and the intermediate image data acquisition unit 131 reads from the buffer memory 102. Intermediate image data is acquired (S803). Note that the processing in S801, the processing in S802, and the processing in S803 are not limited in context, and may be executed in different orders or may be executed in parallel.

The method specifying unit 135 that acquired the optical information specifies an imaging method based on the acquired optical information (S804). Note that the processing in S802, the processing in S803, and the processing in S804 are not limited in context, and may be executed in different orders or may be executed in parallel. The parameter value calculation unit 137 calculates a parameter value for optical characteristic correction processing based on the acquired various information (S805).

The demodulation information acquisition unit 138 acquires one parameter value from the calculated parameter value group (S806). Then, the optical characteristic correction processing unit 103 performs an optical characteristic correction process on the intermediate image data using the parameter value acquired in the process of S806 by the calculation method according to the imaging method specified in the process of S804 (S807). ).

The image processing unit 104 performs various types of image processing on the image data whose optical characteristics are corrected by the optical characteristic correction processing unit 103 to generate a compressed image (S808), and the display control unit 106 generates The compressed image is decoded (expanded) and displayed on the display unit 105 (S809). When it is determined that the compressed image is a desired image (S810 / YES), the image file generation unit 139 creates an image file including the intermediate image data acquired in the process of S803 and Exif information storing the acquired various types of information. Generate (S811).

Note that, for example, the controller 130 determines whether or not the compressed image is a desired image. Specifically, for example, a user who has confirmed the image displayed on the display unit 105 performs an operation of notifying that the image is a desired image via the operation unit 107. When the controller 130 receives a notification that the image is a desired image, the controller 130 determines that the compressed image is a desired image.

If it is determined that the compressed image is a desired image (S810 / YES), the storage processing unit 140 stores the image file generated by the image file generation unit 139 in the image file storage unit 141 (S812). On the other hand, when it is determined that the compressed image is not a desired image (S810 / NO), the demodulation information acquisition unit 138 acquires one parameter value different from the parameter value already acquired from the calculated parameter value group. (S806), and the subsequent processing is repeated.

Note that one different parameter value acquired when it is determined that the compressed image is not the desired image is acquired by the demodulation information acquisition unit 138 according to a predetermined order, for example. In addition, the demodulation information acquisition unit 138 may acquire a parameter value selected by a user operation via the operation unit 107.

In the above embodiment, the case where it is determined whether or not the compressed image is a desired image has been described as an example. By such processing, a compressed image that more conforms to the user's wishes can be generated. However, such a configuration is not essential, and the image file generation unit 139 may generate an image file without determining whether or not the compressed image is a desired image. In this case, a configuration in which the display control unit 106 displays an image based on the compressed image on the display unit 105 is not essential.

In the above embodiment, the case where the image file generation unit 139 generates an image file when the compressed image is determined to be a desired image has been described as an example. However, the generation of the image file by the image file generation unit 139 is not affected by whether or not the compressed image is a desired image. Therefore, an image file may be generated regardless of the determination result of whether or not the compressed image is a desired image.

Next, a functional configuration of the image processing apparatus 2 that is connected to the imaging apparatus 1 and is a device different from the imaging apparatus 1 will be described. The image processing apparatus 2 according to the present embodiment is connected to the imaging apparatus 1 through an interface cable for communicating digital signals such as HDMI (registered trademark) (High-Definition Multimedia Interface), USB, and the like. An information processing apparatus such as a PC (Personal Computer) in which software for image editing such as retouching software and a printer driver is installed. The image processing apparatus 2 may be not only a PC but also a smartphone, a tablet terminal, or the like. In the present embodiment, the image processing apparatus 2 in which retouching software is installed will be described as an example. Under the control of retouching software installed in the image processing apparatus 2, processing such as optical characteristic correction processing and image processing is performed on the image file generated in the imaging apparatus 1.

FIG. 9 is a block diagram illustrating a functional configuration related to processing for an image file generated in the imaging device 1 among the functions of the image processing device 2. As illustrated in FIG. 9, the image processing apparatus 2 includes an image file acquisition unit 201, an optical characteristic correction processing unit 202, an image processing unit 203, a display unit 204, a display control unit 205, an operation unit 206, an operation control unit 207, and an image. An editing unit 208 and a post-processing unit 209 are included.

The image file acquisition unit 201 acquires an image file stored in the image file storage unit 141 of the imaging device 1 via the interface cable and outputs the image file to the optical characteristic correction processing unit 202. The optical characteristic correction processing unit 202 performs optical characteristic correction processing on the intermediate image data using the intermediate image data and various information included in the image file input from the image file acquisition unit 201 and outputs the intermediate image data to the image processing unit 203. To do.

Specifically, first, the optical characteristic correction processing unit 202 specifies an imaging method from the method identifier included in the image file, and acquires at least one parameter file from the group of parameter values included in the image file. Next, the optical characteristic correction processing unit 202 performs an optical characteristic correction process on the intermediate image data using the acquired parameter value by a calculation method according to the specified imaging method.

Similar to the image processing unit 104 of the imaging apparatus 1, the image processing unit 203 performs predetermined processing such as demosaic, gamma correction, color processing, resolution conversion, and JPEG on the image data corrected by the optical characteristic correction processing unit 202. Image processing such as compression processing based on the compression method is performed to generate a compressed image. Further, the image processing unit 203 outputs the generated compressed image to the display control unit 205 and the image editing unit 208. The compressed image data generated by the image processing unit 203 is stored in the storage unit 210 such as a memory or a hard disk.

The display unit 204 and the display control unit 205 have the same functions as the display unit 105 and the display control unit 106 of the imaging device 1. The operation unit 206 is a user interface for a user to input information to the image processing apparatus 2 such as a keyboard, a mouse, various hard buttons, and a touch panel. The operation control unit 207 outputs operation information performed via the operation unit 206 to the image editing unit 208. When the display unit 204 is a display panel or the like and accepts an input operation from the user as a touch panel, the display unit 204 may also function as the operation unit 206.

The display control unit 205 decodes (decompresses) the compressed image data from the image processing unit 203 and supplies the decoded image data to the display unit 204. The display unit 204 displays an image based on the supplied image data. The user can perform various edits according to the user's preference by operating the operation unit 206 while viewing the display image. The image editing unit 208 performs image editing, retouching, image synthesis, and the like on the compressed image data input from the image processing unit 203 or image data obtained by decoding the compressed image data in accordance with a user operation via the operation unit 206. Perform the editing process. The post-processing unit 209 performs various post-processings on the image edited by the image editing unit 208 and outputs it to the display control unit 205. The image data post-processed by the post-processing unit 209 is also stored in the storage unit 210.

Next, an overall operation example of the image processing apparatus 2 according to the present embodiment will be described. FIG. 10 is a flowchart showing an example of the overall operation of the image processing apparatus 2 according to the present embodiment. As shown in FIG. 10, the image file acquisition unit 201 acquires an image file generated in the imaging device 1 (S1001).

The optical characteristic correction processing unit 202 acquires a method identifier, a group of parameter values, and intermediate image data included in the image file acquired by the image file acquisition unit 201 (S1002, S1003, S1004). Note that the processing of S1002, the processing of S1003, and the processing of S1004 are not limited in context, and may be executed in different orders or may be executed in parallel.

The optical characteristic correction processing unit 202 that has acquired the group of parameter values acquires at least one parameter value as demodulation information from the acquired group of parameter values (S1005). Then, the optical characteristic correction processing unit 202 performs optical characteristic correction processing on the acquired intermediate image data using the acquired parameter value (demodulation information) by the calculation method according to the imaging method identified by the acquired method identifier. (S1006).

The image processing unit 203 performs various kinds of image processing on the image data subjected to the optical characteristic correction processing to generate a compressed image (S1007), and the display control unit 106 decodes the compressed image and displays the image on the display unit 105. Is displayed (S1008). When it is determined that the display image is a desired image (S1009 / YES), the image editing unit 208 performs an image editing process on the compressed image according to an operation from the user (S1010).

On the other hand, when it is determined that the compressed image is not a desired image (S1009 / NO), the optical characteristic correction processing unit 202 demodulates the parameter value different from the parameter value already acquired from the acquired parameter value group. (S1005) and the subsequent processing is repeated.

Note that, for example, the image editing unit 208 determines whether or not the compressed image is a desired image. Specifically, for example, a user who has confirmed the image displayed on the display unit 204 performs an operation of notifying that the image is a desired image via the operation unit 206. When the image editing unit 208 receives a notification that the image is a desired image, the image editing unit 208 determines that the compressed image is the desired image.

Further, different parameter values acquired when it is determined that the compressed image is not a desired image are acquired by the optical characteristic correction processing unit 202 in accordance with a predetermined order, for example. In addition, the optical characteristic correction processing unit 202 may acquire a parameter value selected by a user operation via the operation unit 206.

In the above embodiment, the case where it is determined whether or not the compressed image is a desired image has been described as an example. By such processing, a compressed image that more conforms to the user's wishes can be generated. However, such a configuration is not essential, and the image editing unit 208 may perform image editing processing according to a user operation without determining whether or not the compressed image is a desired image. In this case, a configuration in which the display control unit 205 displays a compressed image on the display unit 204 is not essential.

As described above, the imaging apparatus 1 according to the present embodiment includes an image file that includes intermediate image data, information for identifying an imaging method, and information on parameter values used to correct the intermediate image data in accordance with image file standards. Is generated. The image processing device 2 acquires the image file generated by the imaging device 1, and based on the intermediate image data and various information included in the image file, the calculation method according to the imaging method from which the intermediate image data was acquired, The intermediate image data is corrected using various parameter values.

That is, the imaging apparatus 1 according to the present embodiment includes an intermediate image in which the optical modulation is applied to the configuration of the imaging apparatus 1 including an optical system that applies predetermined optical modulation to an image, and the intermediate image. As an additional information, a configuration for generating an image file including information necessary for image processing for demodulating the intermediate image (removing optical modulation) is added. For this reason, according to the present embodiment, for example, it is possible to appropriately perform image processing on the intermediate image in a device different from the imaging device.

In the above embodiment, the case where the retouching software is installed in the image processing apparatus 2 has been described as an example. However, the retouching software is a service on the network and may be downloaded from the network. Further, the image processing device 2 refers to the method identifier included in the accompanying information of the intermediate image, and obtains information for acquiring an optical correction processing calculation method or the like according to the imaging method, such as the manufacturer of the imaging device. You may make it acquire via a network from a server. In other words, if at least the method identifier is included as the accompanying information of the intermediate image, even if software or an application for processing the intermediate image is not installed in the image processing apparatus 2 in advance, an appropriate optical device corresponding to the imaging method is used. Software or an application including a calculation method for correction processing can be downloaded via a network.

Further, in the above embodiment, the case where the image file generation unit 139 generates an image file including intermediate image data as illustrated in FIG. 7 has been described as an example. In addition, the image file generation unit 139 may generate an image file that further includes the compressed image data generated by the image processing unit 104 of the imaging device 1 in addition to the intermediate image data.

FIG. 11 is a diagram illustrating a configuration further including compressed image data in the image file generated by the image file generation unit 139 according to the present embodiment. As shown in FIG. 11, the image file includes compressed image data in addition to the intermediate image data. In this case, the image processing unit 104 outputs the generated compressed image data to the image file generation unit 139 (see FIG. 3). The image file generation unit 139 generates an image file including the compressed image data input from the image processing unit 104.

With such a configuration, the image processing apparatus 2 can also acquire the compressed image data generated in the imaging apparatus 1, and thus starts the image editing process without processing from the intermediate image data to generate the compressed image data. Can do. For example, in the image processing device 2, there is no request to generate a compressed image that is focused from a different viewpoint from the compressed image generated by the imaging device 1, and image editing may be performed on some compressed image. In such a case, the image processing apparatus 2 does not need to process the intermediate image data, so that image editing can be performed immediately and the processing load on the image processing apparatus 2 can be reduced.

In the above embodiment, as described with reference to FIGS. 6 and 7, the case where thumbnail image data is stored in the Exif information has been described as an example. If thumbnail image data is stored in the Exif information, the image processing apparatus 2 can display the thumbnail image data on the display unit 204 before processing the intermediate image data.

As described above, the intermediate image is in a state where the entire image is blurred, and is not necessarily an image that looks good for the user. For this reason, even if only the intermediate image data is confirmed, the user cannot grasp what kind of image it is and needs to wait for confirmation until the intermediate image data is processed and displayed. With the configuration in which the thumbnail image data is stored in the Exif information, the user can immediately grasp what kind of image it is based on the displayed thumbnail image.

However, it is not essential that the Exif information includes a thumbnail image. The thumbnail image data may be generated by the image file generation unit 139. In this case, the image file generation unit 139 acquires compressed image data from the image processing unit 104.

In the above embodiment, the case where the group of parameter values calculated by the parameter value calculation unit 137 is stored in the Exif information has been described as an example. In addition, information necessary for calculating the parameter value group may be stored in the Exif information instead of the parameter value group. That is, information necessary for calculating a group of parameter values or a group of parameter values is information regarding parameter values (calculation values).

FIG. 12 is a diagram illustrating the configuration of an image file generated by the image file generation unit 139. As shown in FIG. 12, in this case, the image file generation unit 139 generates an image file in which optical information, imaging information, a method identifier, and setting information are stored in Exif information. In this case, for example, the optical characteristic correction processing unit 202 of the image processing apparatus 2 acquires imaging information, optical information, and setting information from the image file, and calculates a group of parameter values by the same process as the parameter value calculation unit 137. .

In addition, this invention is not limited to the above-mentioned Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, it is possible to replace a part of the configuration of a certain embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of a certain embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

In addition, each of the above-described configurations, functions, processing units, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files that realize each function can be stored in a storage medium, storage, or the like. Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Image processing apparatus 110 Imaging optical system 131 Intermediate image data acquisition part 132 Imaging information acquisition part 133 Optical information acquisition part 135 Method specification part 137 Parameter value acquisition part 139 Image file generation part 140 Storage processing part 141 Image file storage part

Claims (11)

An imaging optical system that includes an optical modulation element and obtains an intermediate image optically modulated by the optical modulation element;
An image information generation unit that generates an image file including the intermediate image and accompanying information of the intermediate image;
With
The accompanying information includes identification information for identifying an imaging method of an imaging optical system including the optical modulation element, and image processing information related to image processing for demodulating the optically modulated intermediate image. An imaging apparatus characterized by the above. The imaging apparatus according to claim 1, wherein the image processing information includes parameter information associated with the identification information and used for calculation in the image processing. The imaging optical system is configured according to an imaging method,
The parameter information includes calculation specifying information which is information for specifying a calculation method used in accordance with image processing corresponding to the imaging method, and information regarding a calculation value used for calculation by the calculation method. The imaging apparatus according to claim 1. An optical information acquisition unit for acquiring optical information indicating the configuration of the imaging optical system;
A control information acquisition unit for acquiring control information for the imaging optical system when generating the intermediate image;
A calculation value calculation unit for calculating the calculation value based on the acquired optical information and the control information, and
The imaging apparatus according to claim 3, wherein the image information generation unit generates the image file having the calculation specifying information and the calculation value as the accompanying information. An optical information acquisition unit for acquiring optical information indicating the configuration of the imaging optical system;
A control information acquisition unit that acquires control information for the imaging optical system when generating the intermediate image,
The imaging apparatus according to claim 3, wherein the image information generation unit generates the image file using the acquired optical information and the control information as information related to the calculation value. An optical information acquisition unit for acquiring optical information indicating the configuration of the imaging optical system;
A method specifying unit that specifies the imaging method based on the acquired optical information, and
The imaging apparatus according to claim 3, wherein the image information generation unit generates the image file using information for identifying the specified imaging method as the calculation specifying information. An image data generation unit that generates corrected image data desired by the user from the acquired intermediate image data,
The imaging apparatus according to claim 1, wherein the image information generation unit generates an image file including the generated corrected image data, the intermediate image data, and the accompanying information. An image data generation unit that generates corrected image data desired by the user from the acquired intermediate image data, and generates reference image data having a size smaller than the image data from the generated image data;
The imaging apparatus according to claim 1, wherein the image information generation unit generates an image file including the generated reference image data, the intermediate image data, and the accompanying information. An intermediate image is obtained by an imaging optical system including an optical modulation element that optically modulates the image;
Obtaining identification information for identifying the optical modulation method and image processing information necessary for demodulating the optically modulated intermediate image by image processing;
In accordance with a predetermined standard relating to image data and accompanying information of the image data, an image file including the intermediate image data and accompanying information including the identification information and the image processing information is generated.
The generated image file is stored in a storage medium as information obtained in accordance with an imaging operation. Obtaining an intermediate image by an imaging optical system including an optical modulation element that optically modulates the image;
Performing identification information for identifying the optical modulation scheme and image processing information necessary for demodulating the optically modulated intermediate image by image processing;
Generating an image file including the intermediate image data and the accompanying information including the identification information and the image processing information according to a predetermined standard relating to the image data and the accompanying information of the image data;
Storing the generated image file in a storage medium as information obtained in accordance with an imaging operation;
A control program for an imaging apparatus, characterized in that An image optically modulated by a predetermined optical modulation method is captured by the imaging optical system, and image processing for restoring the optical modulation is performed on the image captured by the imaging optical system. In the image file generation method in the configured imaging apparatus,
A method for generating an image file, comprising generating information by adding information including an identifier indicating the type of the optical modulation method to data of the optically modulated image.

Images