JP2018148420A - Image processing device and image processing method - Google Patents

Abstract

An image processing apparatus and an image processing method capable of performing HDR processing with a higher degree of freedom are provided.
An image processing apparatus includes a composition ratio map generation unit 2164 that generates a composition ratio map in which a composition ratio in HDR processing is associated with each coordinate of first image data, and first image data and exposure. And an image processing unit 2165 for performing HDR processing for synthesizing at least one second image data different from each other for each coordinate according to the synthesis ratio map.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus and an image processing method.

  As one of the functions of recent imaging apparatuses, for example, a high dynamic range (HDR) function as proposed in Patent Document 1 is known. The HDR function is a function of performing HDR processing that obtains an image with a wider dynamic range than the original specification of the imaging apparatus by combining image data obtained by multiple imaging with different exposure values. By using such an HDR function, for example, black crushing and whiteout in a scene with a large contrast can be suppressed.

JP 2012-205244 A

  In the conventional HDR processing, processing is uniformly performed on an area having a luminance comparable to that of a blackout area or a whiteout area. For this reason, a part which is not intended by the user is also processed, and an image obtained after the HDR process may not necessarily be what the user desires. As a method for adjusting the luminance of only a part of the area in the image, there is also a method of performing shooting by attaching a half ND filter. However, even in the case of a half ND filter, it is not always possible to adjust the luminance only in the region intended by the user.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus and an image processing method capable of performing HDR processing with a higher degree of freedom.

  In order to achieve the above object, the image processing apparatus according to the first aspect of the present invention generates a composition ratio map in which a composition ratio in HDR processing is associated with each coordinate of first image data. A map generation unit; and an image processing unit that performs HDR processing for combining the first image data and at least one second image data having a different exposure for each coordinate according to the combination ratio map.

  In order to achieve the above object, the image processing method according to the second aspect of the present invention generates a composite ratio map in which a composite ratio in HDR processing is associated with each coordinate of the first image data. Performing HDR processing for synthesizing the first image data and at least one second image data having a different exposure for each coordinate according to the synthesis ratio map.

  According to the present invention, it is possible to provide an image processing apparatus and an image processing method capable of performing HDR processing with a higher degree of freedom.

1 is a block diagram illustrating a configuration of an imaging apparatus as an application example of an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the basic operation | movement of an imaging device. It is a flowchart which shows the basic operation | movement of an imaging device. It is a flowchart which shows the basic operation | movement of an imaging device. It is a flowchart shown about the production | generation process of the synthetic | combination ratio map in one Embodiment. It is the figure shown about the setting of the synthetic | combination ratio in burning mode. It is the figure shown about the 1st example of the setting of the composition ratio in gradation mode. It is the figure shown about the 2nd example of the setting of the composition ratio in gradation mode. It is a figure for demonstrating the specific example of the HDR process in one Embodiment. It is a figure for demonstrating the specific example of the HDR process in one Embodiment. It is a figure for demonstrating a mask pattern. It is a figure for demonstrating a mask pattern. It is a figure for demonstrating mask touch mode. It is a figure for demonstrating mask touch mode. 10 is a flowchart showing a composition ratio map generation process in Modification 2. 10 is a flowchart showing a composition ratio map generation process in Modification 2. It is a figure for demonstrating the modification which produces | generates a synthetic | combination ratio map automatically by learning.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus as an application example of an image processing apparatus according to an embodiment of the present invention. An imaging apparatus 1 illustrated in FIG. 1 is, for example, an interchangeable lens digital camera, and includes an interchangeable lens 100 and a main body 200. The interchangeable lens 100 is configured to be detachable from the main body 200. When the interchangeable lens 100 is attached to the main body 200, the interchangeable lens 100 and the main body 200 can communicate with each other. Thereby, the interchangeable lens 100 operates according to the control of the main body 200. Note that the imaging apparatus 1 shown in FIG. 1 is an interchangeable lens digital camera. However, the imaging apparatus 1 does not necessarily have to be a lens interchangeable type. For example, the imaging device 1 may be a lens-integrated digital camera. Further, the lens itself may have an imaging function. In this case, information may be transmitted by wireless communication between the lens and the main body. Furthermore, the technique of this embodiment does not necessarily have to be applied to an imaging apparatus. That is, the imaging function is not an essential function in the present embodiment.

  As shown in FIG. 1, the interchangeable lens 100 includes an imaging optical system 102, a drive unit 104, a position detection unit 106, a lens control unit 108, a recording unit 110, an operation unit 112, and a communication unit 114. Have. Here, each block of the interchangeable lens 100 is configured by hardware, for example. However, some blocks of the interchangeable lens 100 may be configured by software or a combination of hardware and software.

  The imaging optical system 102 includes a lens and a diaphragm, and causes a light beam from a subject (not shown) to enter the imaging element 202 of the main body 200. The lens of the imaging optical system 102 may include a focus lens. The imaging optical system 102 may include a zoom lens.

  The driving unit 104 includes a driving mechanism such as a motor for driving the lens and the diaphragm of the imaging optical system 102. The driving unit 104 drives the lens and the diaphragm of the imaging optical system 102 under the control of the lens control unit 108.

  The position detection unit 106 is an encoder, for example, and detects the position of the focus lens and the position of the zoom lens of the imaging optical system 102. Further, the position detection unit 106 detects the aperture amount of the stop of the imaging optical system. The detection result of the position detection unit 106 is acquired by the lens control unit 108.

  The lens control unit 108 is a control circuit such as a CPU and an ASIC, and controls the operation of the interchangeable lens 100. The function equivalent to the lens control unit 108 may be realized by software, or may be realized by a combination of hardware and software.

  The recording unit 110 is configured by, for example, a flash ROM. In the recording unit 110, characteristic information of the interchangeable lens 100 is recorded. This characteristic information includes information on the focal length of the imaging optical system 102, for example.

  The operation unit 112 includes operation members such as a focus ring and a zoom ring for the user to operate the interchangeable lens 100. The operation content of the operation unit 112 is detected by the lens control unit 108. The lens control unit 108 performs processing according to the operation content of the operation unit 112.

  The communication unit 114 is an interface on the interchangeable lens 100 side for communication between the interchangeable lens 100 and the main body 200. For example, the communication unit 114 includes an electrical contact and makes a wired connection between the interchangeable lens 100 and the main body 200. As described above, the communication unit 114 may perform communication between the interchangeable lens 100 and the main body 200 by wireless communication.

  The main body 200 includes an image sensor 202, a storage unit 204, a display unit 206, a touch panel 208, an operation unit 210, a recording unit 212, a face detection unit 214, a control unit 216, a communication unit 218, an external device And a communication unit 220.

  The imaging element 202 includes, for example, a CMOS image sensor or a CCD image sensor, images an object, and acquires image data related to the object. The image sensor 202 may include a phase difference detection pixel so that the distance to the object to be photographed can be detected.

  The storage unit 204 is, for example, a DRAM, and temporarily stores image data acquired by the image sensor 202. In addition, the storage unit 204 temporarily stores various processing results in the control unit 216.

  The display unit 206 is a liquid crystal display or an organic EL display, for example, and displays various images.

  The touch panel 208 is disposed so as to overlap the display screen of the display unit 206, for example. This touch panel 208 detects a user's touch operation on the display screen. Here, the touch panel 208 may be configured not only to detect the touched coordinates but also to detect the strength of the touch as a touch operation on the display screen of the user. Further, the touch panel 208 is not necessarily arranged so as to overlap the display screen of the display unit 206.

  The operation unit 210 includes operation members such as buttons, switches, and dials other than the touch panel 208. The operation unit 210 includes, for example, a release button, a moving image button, a setting button, a selection key, and a power button. The release button is an operation member for instructing still image shooting. The moving image button is an operation member for instructing the start or end of moving image shooting. The setting button is an operation member for displaying a setting screen of the imaging apparatus 1. The selection key is an operation member for selecting and determining an item on the setting screen, for example. The power button is an operation member for turning on or off the power of the imaging apparatus 1. Note that the touch panel 208 may be configured to perform operations equivalent to the release button, moving image button, setting button, selection key, and power button described above. Furthermore, the operation unit 210 may have other operation members other than those described above.

  The recording unit 212 is configured by a flash ROM, for example. An image file generated by the control unit 216 is recorded in the recording unit 212. The recording unit 212 may record various programs and setting values used for controlling the main body 200.

  The face detection unit 214 detects a human face from the image data. For example, template matching is used as the face detection method. However, the face detection method is not particularly limited in the present embodiment. The face detection unit 214 may be configured to detect a facial expression of a person. Furthermore, the face detection unit 214 may be configured to detect other than a human face.

  The control unit 216 is a control circuit such as a CPU and an ASIC, and comprehensively controls the operation of the main body 200. The control unit 216 includes an imaging control unit 2161, a display control unit 2162, a contour detection unit 2163, a composition ratio map generation unit 2164, and an image processing unit 2165. The function of each block of the control unit 216 may be realized by software, or may be realized by a combination of hardware and software. The functions of some blocks of the control unit 216 may be provided separately from the control unit 216.

  The imaging control unit 2161 controls the imaging (exposure) operation by the imaging element 202. The imaging control unit 2161 controls the exposure time of the imaging element 202, for example. Further, the imaging control unit 2161 controls the aperture amount of the diaphragm of the imaging optical system 102. As will be described in detail later, during the HDR process, the imaging control unit 2161 controls the imaging device 202 so that imaging with the imaging device 202 is performed a plurality of times with different exposure values.

  A display control unit 2162 performs control when displaying various images on the display unit 206. For example, the display control unit 2162 reads out the image file recorded in the recording unit 212 and reproduces it on the display unit 206.

  The contour detection unit 2163 detects the contour from the image data by, for example, high-pass filter (HPF) processing. The contour is used, for example, to specify a mask pattern described later.

  The composite ratio map generation unit 2164 generates a composite ratio map for HDR processing. The composition ratio map is a map in which a composition ratio of a plurality of image data with different exposures for generating an HDR image is recorded in association with the coordinates of the image data. In HDR processing, image data is synthesized according to a synthesis ratio map. Details will be described later.

  The image processing unit 2165 performs various types of image processing on the image data. This image processing includes basic image processing for using image data such as white balance correction and color correction for display and recording. Furthermore, in the present embodiment, the image processing includes HDR processing. As described above, the HDR processing in this embodiment is performed according to the synthesis ratio map.

  The communication unit 218 is an interface on the main body 200 side for communicating between the interchangeable lens 100 and the main body 200. For example, the communication unit 218 includes an electrical contact and makes a wired connection between the interchangeable lens 100 and the main body 200. As described above, the communication unit 218 may perform communication between the interchangeable lens 100 and the main body 200 by wireless communication.

  The external communication unit 220 is an interface on the main body 200 side for communicating with an external device of the imaging apparatus 1. The external communication unit 220 communicates with an external device by wireless communication, for example.

  Hereinafter, the operation of the imaging apparatus 1 of the present embodiment will be described. 2A, 2B, and 2C are flowcharts showing the basic operation of the imaging apparatus 1. FIG. 2A to 2C are controlled by the control unit 216 of the main body 200.

  In step S1, the control unit 216 determines whether or not the current operation mode of the imaging device 1 is the shooting mode. The imaging apparatus 1 has a shooting mode, a playback mode, and a communication mode as operation modes. The shooting mode is an operation mode for performing processing for recording a still image or a moving image. The reproduction mode is an operation mode for performing processing for reproducing the image file recorded in the recording unit 212 on the display unit 206. The communication mode is a mode for performing communication such as transmitting an image file recorded in the recording unit 212 to an external device or receiving an image file from the external device. The operation mode is set, for example, by the user operating the touch panel 208. If it is determined in step S1 that the current operation mode of the imaging apparatus 1 is the shooting mode, the process proceeds to step S2. If it is determined in step S1 that the current operation mode of the imaging apparatus 1 is not the shooting mode, the process proceeds to step S28.

  In step S2, the control unit 216 determines whether or not the interchangeable lens 100 has been replaced. If it is determined in step S2 that the interchangeable lens 100 has been replaced, the process proceeds to step S3. If it is determined in step S2 that the interchangeable lens 100 has not been replaced, the process proceeds to step S4.

  In step S <b> 3, the control unit 216 communicates with the lens control unit 108 of the interchangeable lens 100 via the communication unit 218 and acquires characteristic information recorded in the recording unit 110. Thereafter, the process proceeds to step S4. Thereafter, the control unit 216 refers to the acquired characteristic information as necessary.

  In step S4, the control unit 216 starts an imaging operation for displaying a through image. As an imaging operation for displaying a through image, the control unit 216 causes the imaging device 202 to perform an imaging operation at a predetermined frame rate. At this time, when a face is detected, the control unit 216 controls the imaging operation of the image sensor 202 so that the exposure value of the face becomes a predetermined reference value. When the face is not detected, the control unit 216 controls the imaging operation of the imaging element 202 so that the exposure value at the center of the screen becomes the reference value. In addition, the control unit 216 may be configured to control the imaging operation of the imaging device 202 so that the average value of the exposure values of the entire screen becomes the reference value.

  In step S <b> 5, the control unit 216 performs live view display by causing the display unit 206 to display an image based on image data sequentially obtained by the imaging operation of the image sensor 202 on the display unit 206.

  In step S6, the control unit 216 determines whether or not the current shooting mode is the HDR shooting mode. The imaging apparatus 1 has a normal shooting mode and an HDR shooting mode as shooting modes. The normal shooting mode is a shooting mode that does not involve HDR processing. The HDR shooting mode is a shooting mode that involves HDR processing. As described above, the HDR processing in the present embodiment is processing for combining a plurality of image data with different exposures according to the combination ratio map. The shooting mode is set by, for example, a user operating the touch panel 208. If it is determined in step S6 that the current shooting mode is the HDR shooting mode, the process proceeds to step S7. If it is determined in step S6 that the current shooting mode is not the HDR shooting mode, the process proceeds to step S14.

  In step S <b> 7, the control unit 216 determines whether a screen touch operation by the user is detected from the output of the touch panel 208. If it is determined in step S7 that a screen touch operation has been detected, the process proceeds to step S8. If it is determined in step S7 that no screen touch operation has been detected, the process proceeds to step S9.

  In step S <b> 8, the control unit 216 generates a synthesis ratio map according to the output of the touch panel 208. Thereafter, the process proceeds to step S9. Here, the composition ratio map generation processing will be described. FIG. 3 is a flowchart showing the composition ratio map generation processing in this embodiment.

  In step S101, the control unit 216 determines whether or not the current composition ratio map generation mode is the burn-in mode. In the present embodiment, the control unit 216 has a burning mode, a dodging mode, and a gradation mode as the synthesis ratio map generation mode. The burn-in mode is a mode in which HDR processing is performed so as to change the exposure of a portion designated by the user. The dodging mode is a mode in which HDR processing is performed so as to change exposure other than the portion designated by the user. The burn-in mode and the dodging mode are modes for realizing the same expression as the burn-in and dodging in a film photograph by image processing. On the other hand, the gradation mode is a mode in which the HDR process is performed so that the exposure is gradually changed in the direction specified by the user. The generation ratio map generation mode is set, for example, by the user operating the touch panel 208. There are other methods of setting with switches and dials, but the user's operation history is recorded and the preference is used as a reference, and priority is given to those generally used in scene analysis. The determination may be made according to the determination program. If it is determined in step S101 that the current composition ratio map generation mode is the burn-in mode, the process proceeds to step S102. If it is determined in step S101 that the current composition ratio map generation mode is not the burn-in mode, the process proceeds to step S107.

  In step S102, the control unit 216 acquires the touch time for the touched coordinates. Note that the control unit 216 may be configured to acquire the touch strength, the number of touches, and the like instead of the touch time. With such touch operations, various intuitive and sensual analog changes can be made both in terms of coordinates and in terms of image quality of the image to be synthesized, depending on touch changes and touch variations. Can do. As a result, it is possible to perform control that is completely different from the uniform HDR processing technology. In other words, the conventional HDR processing technique often performs predetermined synthesis on the entire screen, and cannot sufficiently cope with the user's particular picture creation.

  In step S103, the control unit 216 sets the composition ratio at the touched coordinates. Thereafter, the process proceeds to step S104. FIG. 4 is a diagram showing the setting of the composition ratio in the burn-in mode. FIG. 4 shows that the darker portion has a higher synthesis ratio. As shown in FIG. 4, the composition ratio is set to be proportional to the touch time. That is, the longer the touch time, the greater the synthesis ratio. Since such control is possible, control can be performed both in terms of coordinates and composition ratio. If the touch continues further, it is possible to change the exposure of the image to be synthesized. At this time, in the touched coordinates of the HDR image obtained in the subsequent HDR processing, the ratio of the image data whose exposure is adjusted other than the face portion and the central portion is increased. The synthesis ratio is assumed to have an initial value of 0 and a value from 0 to 1, for example. The set synthesis ratio is held until it is reset. Therefore, in the burn-in mode, a composite ratio map is generated in which only the composite ratio of touched coordinates has a value larger than 0, and the composite ratio of other coordinates is 0. The composite ratio map generated in this way is stored in the storage unit 204, for example.

  Further, the composition ratio may be set so as not to be proportional to the touch strength or the number of touches, but to the touch strength or the number of touches, or not to be proportional to the touch time, the touch strength, or the number of touches. For example, the amount of change in the composition ratio may be set to be small at the initial stage of touch, and the amount of change in the composition ratio may be increased as the touch time increases.

  In step S104, the control unit 216 determines whether or not the combination ratio is a limit. Assuming that the composition ratio takes a value from 0 to 1, if the touch has been made until the time when the composition ratio exceeds 1, the composition ratio is determined to be the limit. If it is determined in step S104 that the synthesis ratio is the limit, the process proceeds to step S105. If it is determined in step S104 that the composite ratio is not the limit, the process proceeds to step S106.

  In step S105, the control unit 216 warns the user that the brightness of the touched coordinate does not change even if the touch is further performed. Thereafter, the process proceeds to step S106. The warning in step S105 is performed by displaying a message on the display unit 206, for example. In addition, the warning in step S105 may be performed by voice or the like.

  In step S106, the control unit 216 determines whether or not the touch by the user has been terminated. For example, when it is detected that the user's finger is away from the touch panel 208, it is determined that the touch by the user is finished. On the other hand, when the user's finger is not separated from the touch panel 208 and only the coordinates are changed (when the user's finger slides on the touch panel 208 or the like), it is determined that the user's touch is not finished. . If it is determined in step S106 that the touch by the user has not been completed, the process returns to step S101. In this case, the composition ratio is reset at the current coordinates. If it is determined in step S106 that the touch by the user has been completed, the processing in FIG. 3 ends.

  In step S107 when it is determined in step S101 that the current composite ratio map generation mode is not the burn-in mode, the control unit 216 determines whether or not the current composite ratio map generation mode is the dodging mode. To do. If it is determined in step S107 that the current composition ratio map generation mode is the dodging mode, the process proceeds to step S108. If it is determined in step S107 that the current composition ratio map generation mode is not the dodging mode, the process proceeds to step S110.

  In step S108, the control unit 216 acquires the touch time for the touched coordinates. Note that the control unit 216 may be configured to acquire the touch strength, the number of touches, and the like instead of the touch time.

  In step S109, the control unit 216 sets a composition ratio in coordinates other than the touched coordinates. Thereafter, the process proceeds to step S104. As in the burn-in mode, in the dodging mode, the composition ratio is set to be proportional to the touch time. However, in the burn-in mode, the composition ratio of only the touched coordinates is set, whereas in the dodging mode, the composition ratio of coordinates other than the touched coordinates is set. Note that, as in the burn-in mode, in the dodging mode, the synthesis ratio is assumed to be 0 to 1 with an initial value of 0, for example. The set synthesis ratio is held until it is reset. Therefore, in the dodging mode, only the composite ratio of the touched coordinates is 0, and a composite ratio map is generated in which the composite ratio of the other coordinates is greater than zero. The composite ratio map generated in this way is stored in the storage unit 204, for example. Since such control is possible, control can be performed both in terms of coordinates and composition ratio. Furthermore, if the touch operation is continued for a long time, it is possible to change the exposure of the image to be synthesized. For example, the exposure difference from the reference image may be changed with a touch pattern including a touch time. In recent years, there is a panel that can obtain information from before touching the touch panel, and the movement of a finger in the air on the panel may be reflected. That is, the feature of the present application is that such a device can be improved and applied widely to image-related devices having a touch panel. A composite ratio map that generates a composite ratio map in which the composite ratio at the time of HDR processing or the exposure difference of the composite image at the time of HDR processing is associated with the coordinates of the operation of the touch panel and the coordinates of the first image data By providing an image processing apparatus comprising: a generation unit; and an image processing unit that performs HDR processing for combining the first image data and the second image data of the exposure difference described above for each coordinate according to the combination ratio map. Different from conventional uniform HDR processing, it is possible to acquire an image that has been created to reflect the user's preference.

  In step S110 when it is determined in step S107 that the current composition ratio map generation mode is not the dodging mode, the control unit 216 determines whether or not the current composition ratio map generation mode is the gradation mode. . If it is determined in step S110 that the current composition ratio map generation mode is the gradation mode, the process proceeds to step S111. If it is determined in step S110 that the current composition ratio map generation mode is not the gradation mode, the process returns to step S101.

  In step S111, the control unit 216 determines whether or not the touch by the user is finished. The determination in step S111 is performed similarly to the determination in step S106. If it is determined in step S111 that the touch by the user has not been completed, the process returns to step S110. If it is determined in step S111 that the touch by the user has been completed, the process proceeds to step S112.

  In step S <b> 112, the control unit 216 calculates a composition ratio having gradation in an area including the touch start coordinates and the end coordinates. The generated combination ratio map is stored in the storage unit 204, for example. Thereafter, the process of FIG. 3 ends.

  FIG. 5A is a diagram showing a first example of setting of the composition ratio in the gradation mode. Here, an arrow A in FIG. 5A indicates the movement of the user's finger on the touch panel 208. That is, in FIG. 5A, the user's finger slides in the right direction from the center of the touch panel 208. In the gradation mode, the composition ratio is set for a rectangular region R including the coordinates of the start point and end point of the touch of the user's finger. In FIG. 5A, the horizontal length of the region R is the coordinates of the start point and the end point of the touch, and the vertical length is the vertical length of the touch panel 208 (that is, the vertical direction of the screen of the display unit 206). Length). The vertical length of the region R does not need to match the vertical length of the touch panel 208.

  In the first example, the composition ratio is set to 0 at the coordinates of the starting point, gradually increasing from the coordinates of the starting point toward the coordinates of the ending point, and to 1 at the coordinates of the ending point. The number of gradations of the synthesis ratio between the start point coordinates and the end point coordinates is, for example, 256 gradations. In this case, the composition ratio changes in 256 steps from the start point coordinates to the end point coordinates.

  FIG. 5B is a diagram showing a second example of setting the composition ratio in the gradation mode. Here, an arrow B in FIG. 5B indicates the movement of the user's finger on the touch panel 208. That is, in FIG. 5B, the user's finger slides up and down from the center of the left end of the touch panel 208 to the center of the right end of the touch panel 208.

  In the second example, the composition ratio is set according to the position in the vertical direction. In other words, the composition ratio is set so that the touched coordinate approaches 0 as it goes down the touch panel, and approaches 1 as it goes up. Also in this case, a composition ratio having gradation is set with the movement of the finger.

  In addition, after the composition ratio map is generated, when the composition ratio map is generated again, the composition ratio map may be generated after resetting the previous composition ratio map. The composite ratio map may be generated so as to overlap the previous composite ratio map. Further, the method for generating the composition ratio map is not limited to the method described above. That is, the composite ratio map can be generated by various methods other than the burn mode, the dodge mode, and the gradation mode.

  Here, it returns to description of FIG. 2A-FIG. 2C. In step S9 after the synthesis ratio map generation process, the control unit 216 starts an imaging operation for the HDR process. Thereafter, the process proceeds to step S10. As the imaging operation in step S9, the control unit 216 sets the exposure value lower or higher by a predetermined number of steps than the imaging operation for displaying a through image (for example, +3 steps or −3 steps with respect to the reference value). The imaging operation of the imaging device 202 is controlled so that Note that whether the image capturing operation is performed with a low exposure value or a high exposure value is set in advance by the user, for example.

  In addition, as the imaging operation in step S9, the control unit 216, for example, has predetermined exposure values of coordinates on the image data touched by the user at the start of generation of the composite ratio map (that is, the coordinates touched first). The imaging operation of the second imaging element 202 may be controlled so as to be the reference value. Furthermore, the imaging operation for HDR processing may be performed twice or more. For example, the control unit 216 performs an imaging operation with an exposure value that is lower by a predetermined number of stages than an imaging operation for displaying a through image (for example, −3 stages with respect to a reference value) and an imaging operation with a high exposure value (for example, It may be configured to control +3 steps) with respect to the reference value. In this case, HDR processing is performed using three pieces of image data. Note that when the HDR processing is performed using three pieces of image data, the first piece is compared with the combination ratio map used for the combination of the first piece of image data and the second piece of image data. The combination ratio map used for combining the result of combining the second image data with the second image data and the third image data may be changed.

  In step S10, the control unit 216 performs HDR processing. Thereafter, the process proceeds to step S11. In step S10, when the composite ratio map is not generated, the control unit 216 performs a normal HDR process. As normal HDR processing, the control unit 216 performs, for example, an overexposure or underexposure region in image data (that is, image data in which the face or the center of the screen is properly exposed) obtained by an imaging operation for displaying a through image. The image data obtained by the imaging operation for HDR processing is synthesized. On the other hand, when the composite ratio map is generated in step S10, the control unit 216 displays the image data (that is, the face or the center of the screen) obtained by the imaging operation for the through image display immediately before the screen touch operation is performed. Are combined with the image data obtained by the imaging operation for the HDR processing according to the combination ratio map.

For example, the value of the pixel of the coordinate (x, y) of the image data obtained by the imaging operation for displaying a through image is I1 (x, y), and the image data obtained by the imaging operation for HDR processing is Assuming that the pixel value at the coordinates (x, y) is I2 (x, y) and the composition ratio at the coordinates (x, y) is a (0 ≦ a ≦ 1), the composition is performed at the coordinates (x, y). The pixel value I3 (x, y) of the image data is obtained by the following equation. HDR image data is generated by performing the following processing on all coordinates of the image data.
I3 (x, y) = (1-a) * I1 (x, y) + a * I2 (x, y)

  Hereinafter, the HDR processing of this embodiment will be specifically described. For example, in the burn-in mode, the user moves the finger F on the image I1 displayed on the through image (actually on the touch panel 208 corresponding to each coordinate of the image I1) while gradually increasing the touch time. Suppose that it slides in the direction of the arrow C shown in the upper stage. The composite ratio map along the direction of the arrow C at this time is a composite ratio map as shown in the lower part of FIG.

  On the other hand, the image data obtained by imaging for the through image is the image data I1 shown in FIG. 7, and the image data obtained by imaging for the subsequent HDR processing is the image data I2 shown in FIG. To do. Note that the image data I1 is image data in which exposure is adjusted to the face. For this reason, in the image data I1, the background is white. The image data I2 is image data whose exposure is adjusted with respect to the background. For this reason, the face is crushed black.

  When the image data I1 and the image data I2 are combined according to the combination ratio map shown in FIG. 6, HDR image data I3 shown in FIG. 7 is generated. Note that FIG. 7 schematically shows HDR image data I3 in a state where the composition is performed only in the direction along the arrow C. As shown in FIG. 7, the brightness at the left end of the HDR image data I3 is the brightness of the image data I1 because no composition is performed. On the other hand, as the HDR image data I3 moves from the left end to the right end, the proportion of the image data I2 increases and the background shown in the image data I2 starts to be displayed. Then, the brightness of the right end of the HDR image data I3 becomes the brightness of the image data I2, and a background with appropriate exposure is displayed.

  Although not shown in the figure, in the dodging mode, HDR image data in which the brightness other than the portion indicated by the arrow C in FIG. 6 gradually changes from the left end to the right end is generated. In the gradation mode, HDR image data in which the brightness of the entire image gradually changes from the left end to the right end is generated. In the gradation mode, an effect similar to that obtained when a so-called half ND filter is mounted can be expressed by image processing.

  As described above, in the present embodiment, the HDR processing composition ratio can be changed for each coordinate of the image data. Thereby, in this embodiment, HDR image data of various expressions can be generated. Thus, in this embodiment, the degree of freedom of HDR processing can be increased.

  Here, it returns to description of FIG. 2A-FIG. 2C. In step S11, the control unit 216 causes the display unit 206 to display a through image based on the image data obtained by the HDR process.

  In step S12, the control unit 216 determines whether or not there is a scene change. The scene change includes a change in brightness, a change in contrast, a change in subject, and the like. When these changes are equal to or greater than a predetermined value, it is determined that a scene change has occurred. If it is determined in step S12 that there has been a scene change, the process proceeds to step S13. If it is determined in step S12 that there is no scene change, the process proceeds to step S14.

  In step S13, the control unit 216 resets the composite ratio map. Thereafter, the process proceeds to step S14. If HDR processing is performed according to the composition ratio map before the scene change, there is a high possibility that processing different from the user's intention is performed. Therefore, when there is a scene change, the composition ratio map is reset. Note that the process of step S13 may be omitted. That is, even if there is a scene change, the composite ratio map may be held.

  In step S14, the control unit 216 determines whether or not a shooting operation has been performed by the user. The shooting operation is, for example, a release button operation or a moving image button operation by the user. If it is determined in step S14 that the user has performed a shooting operation, the process proceeds to step S15. If it is determined in step S14 that the user has not performed a shooting operation, the process proceeds to step S25.

  In step S15, the control unit 216 determines whether or not the current shooting mode is the HDR shooting mode. If it is determined in step S15 that the current shooting mode is the HDR shooting mode, the process proceeds to step S16. If it is determined in step S15 that the current shooting mode is not the HDR shooting mode, the process proceeds to step S26.

  In step S16, the control unit 216 starts an imaging operation for recording HDR image data. Thereafter, the process proceeds to step S17. As the imaging operation in step S <b> 16, the control unit 216 controls the imaging operation of the imaging element 202 so that the exposure value of the face or the center of the screen becomes a predetermined reference value. Thereafter, the control unit 216 controls the imaging operation of the image sensor 202 so that the exposure value of the face or the center of the screen is lower or higher than the predetermined reference value by a predetermined number of steps. As described above, the control unit 216 may be configured to perform an imaging operation three times or more.

  In step S17, the control unit 216 determines whether there is a composite ratio map. When it is determined in step S17 that there is a composite ratio map, the process proceeds to step S18. When it is determined in step S17 that there is no composite ratio map, the process proceeds to step S19.

  In step S18, the control unit 216 performs HDR processing according to the composite ratio map. Thereafter, the process proceeds to step S20. The HDR processing method may be the same as that in step S10.

  In step S19, the control unit 216 performs normal HDR processing. Thereafter, the process proceeds to step S20. As a normal HDR process, the control unit 216 synthesizes the second image data with, for example, a whiteout or blackout region in the first image data.

  In step S20, the control unit 216 causes the display unit 206 to display an image based on the image data obtained by the HDR process.

  In step S21, the control unit 216 determines whether or not a determination operation has been performed by the user. The determination operation is, for example, a touch operation on the determination button displayed together with the HDR image in step S20. In step S21, when the control unit 216 determines that the determination operation is not performed by the user, the process proceeds to step S22. In step S21, when the control unit 216 determines that the determination operation has been performed by the user, the process proceeds to step S24.

  In step S <b> 22, the control unit 216 determines whether a screen touch operation by the user is detected from the output of the touch panel 208. If it is determined in step S22 that a screen touch operation has been detected, the process proceeds to step S23. If it is determined in step S22 that no screen touch operation has been detected, the process proceeds to step S17.

  In step S <b> 23, the control unit 216 generates a synthesis ratio map according to the output of the touch panel 208. Thereafter, the process proceeds to step S17. Note that the composition ratio map generation process may be the same as in FIG. When the composite ratio map generation process is performed in step S23, the HDR process is performed in step S18 according to the composite ratio map generated in step S23 when the HDR image is displayed again. That is, in the present embodiment, the HDR process can be adjusted even during shooting.

  In step S24, the control unit 216 records the HDR image data in the recording unit 212 as an image file (HDR image file). Thereafter, the process proceeds to step S25. Here, in the present embodiment, the image data used for generating the HDR image data is also recorded as one HDR image file. Thereby, the HDR process can be adjusted also by the editing process at the time of reproducing the image file. The HDR image data and the image data used for generating the HDR image data may be recorded in separate files. In this case, for example, both files need to be associated by a file name.

  In step S25, the control unit 216 determines whether the power source of the main body 200 is turned off. When it is determined in step S25 that the power source of the main body 200 has been turned off, the processing in FIGS. 2A to 2C ends. If it is determined in step S25 that the main body 200 is not turned off, the process returns to step S1.

  In step S26 when it is determined in step S15 that the current shooting mode is not the HDR shooting mode, the control unit 216 starts an imaging operation for normal recording. Thereafter, the process proceeds to step S27. As the imaging operation in step S26, the control unit 216 controls the imaging operation of the image sensor 202 so that the exposure value of the face or the center of the screen becomes a predetermined reference value. Even in step S26, at the time of continuous shooting or moving image shooting, the image pickup operation for recording is repeatedly performed until the user instructs the end.

  In step S27, the control unit 216 records the image data in the recording unit 212 as an image file. Thereafter, the process proceeds to step S25.

  In step S28 when it is determined in step S1 that the current operation mode of the imaging device 1 is not the shooting mode, the control unit 216 determines whether or not the current operation mode of the imaging device 1 is the playback mode. In step S28, when it is determined that the current operation mode of the imaging apparatus 1 is the reproduction mode, the process proceeds to step S29. If it is determined in step S28 that the current operation mode of the imaging apparatus 1 is not the playback mode, the process proceeds to step S41.

  In step S29, the control unit 216 causes the display unit 206 to display a list of image files recorded in the recording unit 212. Thereafter, the process proceeds to step S30.

  In step S30, the control unit 216 determines whether an image file has been selected by the user. If it is determined in step S30 that an image file has been selected by the user, the process proceeds to step S31. If it is determined in step S30 that the user has not selected an image file, the process proceeds to step S40.

  In step S31, the control unit 216 reproduces the selected image file on the display unit 206. Note that when the selected image file is an HDR image file, the control unit 216 reproduces the HDR image data.

  In step S32, the control unit 216 determines whether or not the image file being reproduced is an HDR image file. If it is determined in step S32 that the image file being reproduced is an HDR image file, the process proceeds to step S33. If it is determined in step S32 that the image file being reproduced is not an HDR image file, the process proceeds to step S39.

  In step S <b> 33, the control unit 216 determines whether a screen touch operation by the user is detected from the output of the touch panel 208. If it is determined in step S33 that a screen touch operation has been detected, the process proceeds to step S34. If it is determined in step S33 that no screen touch operation has been detected, the process proceeds to step S39.

  In step S <b> 34, the control unit 216 generates a synthesis ratio map according to the output of the touch panel 208. Thereafter, the process proceeds to step S35. Note that the composition ratio map generation processing is performed in the same manner as shown in FIG.

  In step S35, the control unit 216 performs HDR processing according to the composite ratio map. Thereafter, the process proceeds to step S36. In step S <b> 35, the control unit 216 performs HDR processing using a plurality of image data with different exposures used for creating the HDR image data recorded in the currently reproduced image file. The HDR processing method may be the same as that in step S10.

  In step S36, the control unit 216 causes the display unit 206 to display an image based on the image data obtained by the HDR process.

  In step S37, the control unit 216 determines whether or not a determination operation has been performed by the user. In step S37, when the control unit 216 determines that the determination operation is not performed by the user, the process proceeds to step S39. In step S37, when it is determined that the determination operation has been performed by the user, the control unit 216 proceeds to step S38.

  In step S38, the control unit 216 records the HDR image data after the change in step S35 in the recording unit 212 as an HDR image file. Thereafter, the process proceeds to step S39. Also in step S38, the image data used to generate the HDR image data is recorded as an HDR image file. When the image data used for generating the HDR image data is in a file different from the HDR image data, it is not necessary to record the image data used for generating the HDR image data again.

  In step S39, the control unit 216 determines whether or not to end the reproduction of the image file. In step S39, for example, when the end of the reproduction of the image file is instructed by the operation of the operation unit 210 by the user, it is determined that the reproduction of the image file is ended. In step S39, when it is determined not to end the reproduction of the image file, the process returns to step S31. In this case, the reproduction of the image file is continued. If it is determined in step S39 that the reproduction of the image file is to end, the process returns to step S29.

  In step S40 when it is determined in step S30 that an image file has not been selected by the user, the control unit 216 determines whether or not to end the processing in the playback mode. If it is determined in step S40 that the playback mode process is not to end, the process returns to step S29. If it is determined in step S40 that the playback mode process is to be terminated, the process proceeds to step S25.

  In step S <b> 41 when it is determined in step S <b> 28 that the current operation mode of the imaging device 1 is not the playback mode, the control unit 216 performs processing in the communication mode. In the communication mode, the control unit 216 performs processing of transmitting an image file designated by the user to an external device or receiving an image file or the like from the external device. After the communication mode process ends, the process proceeds to step S25.

  As described above, in the present embodiment, a composite ratio map for HDR processing is generated according to a user operation. By performing the HDR process according to the synthesis ratio map, it is possible to generate HDR image data of various expressions. Thereby, the freedom degree of HDR processing can be raised.

  Here, in the above-described embodiment, the coordinates for setting the composition ratio are designated in accordance with the user's operation on the touch panel 208. However, the operation of the touch panel 208 is not necessarily required as long as the coordinates can be designated.

[Modification 1]
Hereinafter, modifications of the present embodiment will be described. In the above-described embodiment, an example in which the composition ratio is set for the touched coordinates has been described. On the other hand, the composition ratio may be set not only for the touched coordinates but also for the mask pattern including the touched coordinates. For example, when there is a touch on the coordinates in the region of the person O shown in FIG. 8A, the control unit 216 detects an outline including the touched coordinates and masks the detected outline as shown in FIG. 8B. Let it be pattern M. Then, the control unit 216 sets a synthesis ratio for the mask pattern M. For example, in the burn-in mode, the control unit 216 sets the composition ratio of each coordinate included in the mask pattern M according to the touch time, the touch strength, the number of touches, and the like. In the dodging mode, the control unit 216 sets the composition ratio of each coordinate excluding the mask pattern M according to the touch time, the touch strength, the number of touches, and the like. Further, in the gradation mode, the control unit 216 sets the composition ratio so that the mask pattern M has gradation. Thus, by setting the composition ratio based on the mask pattern M, it is possible to reduce the number of user operations.

[Modification 2]
In the embodiment described above, basically, a composition ratio is set for the touched coordinates. On the other hand, setting of the composition ratio may be prohibited even if the coordinates are touched. For example, in the above-described example, when a slide operation including a face as shown in FIG. 6 is performed in the burn-in mode, the face is also subject to HDR processing. In this case, a face image having an exposure value lower than the reference value is combined with the face image whose exposure value is matched with the reference value, so that the face exposure value is lower than the reference value. Modification 2 is an example in which the exposure value of only the other part is changed without changing the exposure value of the face even when the slide operation similar to that in FIG. 6 is performed. Hereinafter, a mode in which a part of such image data can be excluded from the HDR processing target is referred to as a mask touch mode. In the mask touch mode, for example, as shown in FIG. 9, a display D indicating the mask touch mode may be displayed.

  In the mask touch mode, the user operates the touch panel 208 with a plurality of fingers. As shown in FIG. 9, the user touches the coordinates to be excluded from the HDR processing target with the finger F1. Further, as shown in FIG. 9, the user performs the slide operation as described above with the other finger F2. At this time, the control unit 216 generates a composite ratio map by excluding the coordinates surrounded by the contour including the coordinates touched by the finger F1 from the HDR processing target. Then, the HDR processing is performed according to the composite ratio map generated in this manner, so that the face is excluded from the HDR processing target as shown in FIG. 10, and the exposure value of the face remains the reference value.

  FIG. 11A and FIG. 11B are flowcharts showing the composition ratio map generation processing in the second modification. Here, in FIGS. 11A and 11B, the same processes as those in FIG. 3 are denoted by the same reference numerals as those in FIG. Therefore, the description of the processes denoted by the same reference numerals is omitted as appropriate.

  In step S101, the control unit 216 determines whether or not the current composition ratio map generation mode is the burn-in mode. If it is determined in step S101 that the current composition ratio map generation mode is the burn-in mode, the process proceeds to step S102. If it is determined in step S101 that the current composition ratio map generation mode is not the burn-in mode, the process proceeds to step S107.

  In step S102, the control unit 216 acquires the touch time for the mask pattern including the touched coordinates. Note that the control unit 216 may be configured to acquire the touch strength, the number of touches, and the like instead of the touch time. Thereafter, the process proceeds to step S113.

  In step S113, the control unit 216 determines whether or not a mask touch is performed. For example, when a touch on two or more coordinates included in the same mask pattern is detected, it is determined that a mask touch is being made. If it is determined in step S113 that no mask touch has been made, the process proceeds to step S103. If it is determined in step S113 that a mask touch has been made, the process proceeds to step S114.

  In step S <b> 114, the control unit 216 excludes the mask pattern including the coordinates on which the mask is touched from the synthesis ratio setting target. Thereafter, the process proceeds to step S104.

  Steps S103-S106 are the same as steps S103-S106 in FIG. Therefore, the description is omitted.

  In step S107 when it is determined in step S101 that the current composite ratio map generation mode is not the burn-in mode, the control unit 216 determines whether or not the current composite ratio map generation mode is the dodging mode. To do. If it is determined in step S107 that the current composition ratio map generation mode is the dodging mode, the process proceeds to step S108. If it is determined in step S107 that the current composition ratio map generation mode is not the dodging mode, the process proceeds to step S110.

  In step S108, the control unit 216 acquires the touch time for the touched coordinates. Note that the control unit 216 may be configured to acquire the touch strength, the number of touches, and the like instead of the touch time. Thereafter, the process proceeds to step S114.

  In step S115, the control unit 216 determines whether or not a mask touch is performed. For example, when a touch on two or more coordinates included in the same mask pattern is detected, it is determined that a mask touch is being made. If it is determined in step S115 that the mask touch has not been made, the process proceeds to step S109. If it is determined in step S115 that a mask touch has been made, the process proceeds to step S116.

  In step S116, the control unit 216 excludes the mask pattern including the coordinates where the mask is touched from the synthesis ratio setting target. Thereafter, the process proceeds to step S104.

  In step S110 when it is determined in step S107 that the current composition ratio map generation mode is not the dodging mode, the control unit 216 determines whether or not the current composition ratio map generation mode is the gradation mode. . If it is determined in step S110 that the current composition ratio map generation mode is the gradation mode, the process proceeds to step S111. If it is determined in step S110 that the current composition ratio map generation mode is not the gradation mode, the process returns to step S101.

  In step S111, the control unit 216 determines whether or not the touch by the user is finished. If it is determined in step S111 that the touch by the user has not been completed, the process returns to step S110. If it is determined in step S111 that the touch by the user has been completed, the process proceeds to step S117.

  In step S117, the control unit 216 determines whether a mask touch has been performed. If it is determined in step S117 that the mask touch is not performed, the process proceeds to step S112. If it is determined in step S117 that the mask touch has been made, the process proceeds to step S118.

  In step S <b> 118, the control unit 216 excludes the mask pattern including the coordinates where the mask touch is made from the synthesis ratio setting targets. Thereafter, the processing of FIGS. 11A and 11B ends.

  By installing the mask touch mode as described above, HDR image data intended by the user can be generated.

[Modification 3]
In the embodiment and the modification described above, the composite ratio map is generated when the user performs a screen touch operation. On the other hand, when it is determined that the scene is similar to the scene in which the composite ratio map was previously generated by learning the relationship between the user's screen touch operation tendency and the scene, for example, by machine learning, Specifically, the control unit 216 may be configured to generate a previously generated composite ratio map. For example, the control unit 216 is equipped with an AI, and image data at the time of shooting after learning that the HDR image data I3 is generated in a scene where the image data I1 and the image data I2 in FIG. 12 are obtained. When it is determined that the scene can obtain I1, the composition ratio map MP may be generated and the HDR process may be performed without waiting for the user's screen touch operation. At this time, the user may be notified that the composite ratio map has been automatically generated. Such a configuration is expected to reduce the number of user operations.

  Although the present invention has been described based on the above embodiment, the present invention is not limited to the above-described embodiment, and various modifications and applications are naturally possible within the scope of the gist of the present invention. For example, the imaging device shown in the above-described embodiment is a lens interchangeable digital camera. However, the imaging apparatus assumed in this embodiment includes various apparatuses having an imaging function other than the digital camera, such as an endoscope, a microscope, a monitoring camera, and an in-vehicle camera. Furthermore, as described above, the HDR processing in the present embodiment can be performed during the playback mode. In other words, the technique of this embodiment does not necessarily require an imaging operation, and the technique of this embodiment can be applied to various image processing apparatuses to which a plurality of image data with different exposures are input. This is because in recent years, an increasing number of devices acquire various exposure and focus position information at the time of imaging and record them for image composition.

  Further, in the description of each operation flowchart described above, the operation is described using “first”, “next”, and the like for convenience, but this does not mean that it is essential to perform the operations in this order. Absent.

  Each process according to the above-described embodiment may be stored as a program that can be executed by the control unit 216 that is a computer. In addition, it can be stored and distributed in a storage medium of an external storage device such as a magnetic disk, an optical disk, or a semiconductor memory. And the control part 216 can perform the process mentioned above by reading the program memorize | stored in the storage medium of this external storage device, and operation | movement being controlled by this read program.

The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention when it is practiced. Further, the embodiments may be implemented in combination as appropriate, and in that case, the combined effect can be obtained. Furthermore, the invention includes various inventions, and various inventions can be extracted by combinations selected from a plurality of disclosed constituent elements. For example, even if several constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the problem can be solved and an effect can be obtained, the configuration from which the constituent requirements are deleted can be extracted as an invention.
For example, the following invention can be extracted in addition to the invention described in the initial claims of the present application.
[1] A composition that has a touch panel and associates the composition ratio at the time of HDR processing or the exposure difference of the composite image at the time of HDR processing for each coordinate of the first image data by the operation position and touch pattern of the touch panel. A composite ratio map generator for generating a ratio map;
An image processing unit for performing HDR processing for combining the first image data and the second image data of the above-described exposure difference for each coordinate according to the combination ratio map;
An image processing apparatus comprising:

DESCRIPTION OF SYMBOLS 1 Imaging device, 100 Interchangeable lens, 102 Imaging optical system, 104 Drive part, 106 Position detection part, 108 Lens control part, 110 Recording part, 112 Operation part, 114 Communication part, 200 Main body, 202 Imaging element, 204 Storage part, 206 Display unit 208 Touch panel 210 Operation unit 212 Recording unit 214 Face detection unit 216 Control unit 218 Communication unit 220 External communication unit 2161 Imaging control unit 2162 Display control unit 2163 Contour detection unit 2164 Synthesis Ratio map generation unit, 2165 image processing unit.

Claims (6)

A composition ratio map generation unit that generates a composition ratio map in which the composition ratio at the time of HDR processing is associated with each coordinate of the first image data;
An image processing unit that performs HDR processing for combining the first image data and at least one second image data having a different exposure for each coordinate according to the combination ratio map;
An image processing apparatus comprising: A display unit for displaying an image based on the first image data;
A touch panel configured to specify coordinates on an image displayed on the display unit by touch;
Further comprising
The composition ratio map generation unit calculates the composition ratio for the designated coordinates according to at least one of a touch time, a touch strength, and the number of touches for the coordinates designated by the touch panel. The image processing apparatus described.   The image processing apparatus according to claim 2, wherein the composition ratio map generation unit calculates the composition ratio for coordinates surrounded by an outline including coordinates designated by the touch panel. The touch panel is further configured to be able to specify coordinates for which HDR processing is not performed from coordinates on the image displayed on the display unit,
The image processing apparatus according to claim 2, wherein the synthesis ratio map generation unit calculates the synthesis ratio with respect to coordinates excluding coordinates for which the HDR processing is not performed.   The image processing apparatus according to claim 4, wherein the synthesis ratio map generation unit sets coordinates that are not subjected to the HDR processing for coordinates surrounded by a contour including coordinates that are not subjected to the HDR processing. Generating a composite ratio map in which the composite ratio at the time of HDR processing is associated with each coordinate of the first image data;
Performing HDR processing for combining the first image data and at least one second image data having different exposures for each coordinate according to the combination ratio map;
An image processing method comprising: