JP2012119761A - Electronic apparatus, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a dynamic range expanded image complying with user's preference.SOLUTION: An HDR image having an expanded dynamic range is created by combining a plurality of input images having different exposure conditions, and a plurality of combined result images are created by applying a plurality of gradation conversion processing (tone mapping) individually to the HDR image thus created. Meanwhile, a tone correction image having a dynamic range expanded artificially is created by performing tone correction of a single input image. Based on the movement, presence/absence of flicker or saturation of an object in an input image array, priority is set in a plurality of combined result images and tone correction signals, and then display control or recording control of the plurality of combined result images and tone correction signals is performed according to the priority.

Description

  The present invention relates to an electronic apparatus such as an imaging apparatus. The present invention also relates to an image processing method and program for performing various processes on an image.

  A method of generating an HDR (High Dynamic range) image by expanding the dynamic range of an input image obtained by photographing is known.

  In a general HDR process for realizing this method, a dynamic range is expanded by synthesizing a plurality of input images having different exposure conditions. In many cases, final HDR images are generated by performing gradation conversion for the purpose of bit length compression or the like on an image with an expanded dynamic range. There has also been proposed a method of expanding the dynamic range (apparent dynamic range) by performing appropriate gradation correction on a single image.

  In addition, the following first and second conventional methods exist as conventional methods for expanding the dynamic range.

  An imaging apparatus according to a first conventional method combines a first dynamic range expansion unit that expands a dynamic range by performing gradation correction on a single image, and a plurality of images captured under different exposure conditions. A second dynamic range expansion unit that expands the dynamic range, and a selection unit that selects which of the first and second dynamic range expansion units is used to acquire an image with an expanded dynamic range. Means for causing the image pickup means to take an image required by the dynamic range expansion means (see, for example, Patent Document 1 below).

  The image pickup apparatus according to the second conventional method includes image combining means for expanding a dynamic range by combining a plurality of images taken under different exposure conditions. In the image combining means, details of the focused state of the image The dynamic range expansion width is changed according to the degree of the degree (see, for example, Patent Document 2 below).

JP 2010-093679 A JP 2009-218895 A

  Naturally, the result image of the HDR processing using a plurality of images changes depending on the contents of the processing, and particularly changes greatly depending on the above-described gradation conversion. It is conceivable that the user designates the contents of HDR processing including gradation conversion so that a result image that suits the user's preference is generated, but such designation imposes a relatively large operational burden on the user. . It goes without saying that it would be beneficial if a result image suitable for the user's preference could be provided without imposing an excessive burden on the user. However, in the conventional methods including the first and second conventional methods, this is sufficient. Not realized.

  SUMMARY An advantage of some aspects of the invention is that it provides an electronic device, an image processing method, and a program that contribute to providing a dynamic range expanded image that suits a user's preference.

  The first electronic device according to the present invention includes an image data acquisition unit that acquires image data of an input image sequence including n input images (n is an integer of 2 or more) photographed under different exposure conditions; A synthesis processing unit that generates an image having a dynamic range wider than the dynamic range of each input image by synthesizing a plurality of input images included in the n input images, and a generated image of the synthesis processing unit, And a tone conversion unit that generates a plurality of combined result images by performing tone conversion using a plurality of different tone conversion conditions individually.

  Accordingly, the user can extract a synthesis result image (dynamic range expanded image) according to his / her preference from a plurality of synthesis result images.

  And, for example, a priority setting unit that sets priority for the plurality of combined result images based on image data of the input image sequence, and controls display or recording of the plurality of combined result images according to the priority It is preferable that an image output control unit to be provided is further provided in the first electronic device.

  As a result, it is possible to preferentially display or record a synthesis result image that is presumed to be suitable for the user's preference.

  For example, the first electronic apparatus may further include a gradation correction unit that generates a gradation correction image by performing gradation correction on one input image included in the n input images. .

  And, for example, a priority setting unit that sets priority for the plurality of combined result images and the gradation correction image based on image data of the input image sequence, and the plurality of combined result images according to the priority An image output control unit that controls display or recording of the gradation correction image may be further provided in the first electronic device.

  As a result, it is possible to preferentially display or record the synthesis result image or the gradation correction image that is presumed to match the user's preference.

  Specifically, for example, the priority order setting unit, based on the image data of the input image sequence, the movement of the object on the input image sequence, the presence or absence of flicker in the input image sequence, and the color in the input image sequence. One or more of the degrees may be evaluated, and the priority order may be set based on the evaluation result.

  Further, for example, the first electronic device is an imaging device that acquires image data of the input image sequence by photographing, and an exposure condition setting unit that sets an exposure condition of each input image forming the input image sequence; A necessity determining unit that determines whether or not it is necessary to generate the plurality of combined result images, and whether or not to generate the plurality of combined result images is determined according to the determination result of the necessity. The input image captured prior to the n input images includes the first and second determination input images, and the exposure condition setting unit has a predetermined value in the first determination input image. The first exposure condition at the time of shooting the first determination input image is set so that the number of pixels having a luminance equal to or higher than the first reference luminance is equal to or less than a predetermined first reference value, and the second In the predetermined input image A second exposure condition at the time of shooting the second determination input image is set so that the number of pixels having a luminance equal to or lower than a reference luminance is equal to or lower than a predetermined second reference value, and the necessity determination unit includes: The necessity determination may be performed based on the first and second exposure conditions.

  The second electronic device according to the present invention includes an image data acquisition unit that acquires image data of an input image sequence including n input images (n is an integer of 2 or more) photographed under different exposure conditions, A synthesis processing unit that generates an image having a dynamic range wider than the dynamic range of each input image by synthesizing a plurality of input images included in the n input images, and for the generated image of the synthesis processing unit, A tone conversion unit that generates a composite result image by performing tone conversion, wherein the tone conversion unit changes the content of the tone conversion based on image data of the input image sequence. And

  Thereby, provision of a synthesis result image (dynamic range expanded image) that suits the user's preference is expected.

  The first image processing method according to the present invention includes an image data acquisition step of acquiring image data of an input image sequence including n input images (n is an integer of 2 or more) photographed under different exposure conditions; A synthesis processing step for generating an image having a dynamic range wider than the dynamic range of each input image by synthesizing a plurality of input images included in the n input images, and for the generated image of the synthesis processing step, A gradation conversion step of generating a plurality of combined result images by performing gradation conversion using a plurality of different gradation conversion conditions individually.

  The second image processing method according to the present invention includes an image data acquisition step of acquiring image data of an input image sequence including n input images (n is an integer of 2 or more) photographed under different exposure conditions; A synthesis processing step for generating an image having a dynamic range wider than the dynamic range of each input image by synthesizing a plurality of input images included in the n input images, and for the generated image of the synthesis processing step, A gradation conversion step of generating a composite result image by performing gradation conversion, and in the gradation conversion step, the content of the gradation conversion is changed based on image data of the input image sequence It is characterized by.

  For example, a program for causing a computer to execute the image data acquisition step, the synthesis processing step, and the gradation conversion step in the first or second image processing method may be formed.

  ADVANTAGE OF THE INVENTION According to this invention, it is possible to provide the electronic device, the image processing method, and program which contribute to provision of the dynamic range expansion image suitable for a user's liking.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is an internal block diagram of the imaging part of FIG. It is a figure which shows the relationship between a two-dimensional image space and a two-dimensional image. It is a block diagram of the part which is especially concerned with dynamic range expansion. FIG. 4A is a diagram showing an input image as a short exposure image, an input image as a long exposure image, and an HDR image generated from the input image, and FIG. 4B is a diagram showing a process for increasing the brightness of the short exposure image. It is a figure showing the relationship between the incident light quantity and signal level in an input image. It is a figure showing the relationship between the incident light quantity and signal level in a HDR image. It is a figure which shows a mode that a synthetic | combination result image is obtained from an HDR image by gradation conversion. It is a figure which shows the tone curve used by the 1st gradation conversion process. It is a figure which shows a mode that a synthetic | combination result image is produced | generated from an HDR image by the 2nd gradation conversion process. It is a figure for demonstrating the simple bit length compression in a 2nd gradation conversion process. It is a brightness | luminance histogram of the HDR image and synthetic | combination result image referred in a 2nd gradation conversion process. It is a figure which shows a mode that a gradation correction image is obtained from an input image. FIG. 4A is a diagram showing a luminance histogram of an HDR image and a tone correction image, and FIG. 3C is a diagram showing a gain function in tone correction processing. FIG. 4A is a diagram illustrating a relationship between a HDR imaging necessity determination period and a main imaging period, and FIG. 4B is a diagram for explaining an input image captured during those periods. It is a figure which shows the image sequence for determination containing the image for short exposure determination, and the image for long exposure determination. It is a brightness | luminance histogram of the image for short exposure determination, and the image for long exposure determination. It is a figure which shows a mode that three types of synthetic | combination result images are obtained from two object input images. It is a figure which shows a mode that three types of synthetic result images are obtained from five object input images. 2 is an operation flowchart of the imaging apparatus in FIG. 1. It is a figure for demonstrating the example of the image display method according to a priority. It is the figure which showed the some detection part which can be provided in the imaging device of FIG. It is a figure which shows a mode that the optical flow is derived | led-out between images. It is a figure for demonstrating the setting method of the priority with respect to the 1st-3rd output HDR image.

  Hereinafter, an example of an embodiment of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 has each part referred by the codes | symbols 11-28. The imaging apparatus 1 is a digital video camera, and can capture a moving image and a still image, and can also capture a still image during moving image capturing. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25. The display unit 27 and / or the speaker 28 may be provided in an external device (not shown) of the imaging device 1.

  FIG. 2 shows an internal configuration diagram of the imaging unit 11. The imaging unit 11 drives and controls the optical system 35, the diaphragm 32, the imaging element 33 such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the optical system 35 or the diaphragm 32. And a driver 34. The optical system 35 is formed from a plurality of lenses including the zoom lens 30 and the focus lens 31. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. The driver 34 drives and controls the positions of the zoom lens 30 and the focus lens 31 and the opening of the diaphragm 32 based on a control signal from a CPU (Central Processing Unit) 23, so that the focal length (view angle) of the imaging unit 11 is controlled. ), The focal position, and the amount of light incident on the image sensor 33 (in other words, the aperture value).

  The image sensor 33 photoelectrically converts an optical image representing a subject incident through the optical system 35 and the diaphragm 32 and outputs an electric signal obtained by the photoelectric conversion to an AFE (Analog Front End) 12. More specifically, the image sensor 33 includes a plurality of light receiving pixels arranged two-dimensionally in a matrix, and each light receiving pixel stores a signal charge having a charge amount corresponding to the exposure time in photographing each image. An analog signal from each light receiving pixel having a magnitude proportional to the amount of stored signal charge is sequentially output to the AFE 12 in accordance with a drive pulse generated in the imaging device 1.

  The AFE 12 amplifies the analog signal output from the imaging unit 11 (image sensor 33), and converts the amplified analog signal into a digital signal. The AFE 12 outputs this digital signal to the video signal processing unit 13 as RAW data. The degree of signal amplification in the AFE 12 is controlled by the CPU 23.

  The video signal processing unit 13 performs necessary image processing on the image represented by the RAW data, and generates a video signal for the image after the image processing. The microphone 14 converts the ambient sound of the imaging device 1 into an analog audio signal, and the audio signal processing unit 15 converts the analog audio signal into a digital audio signal.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 using a predetermined compression method. The internal memory 17 is composed of a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores various data. The external memory 18 as a recording medium is a non-volatile memory such as a semiconductor memory or a magnetic disk, and records the video signal and the audio signal compressed by the compression processing unit 16 in a state of being associated with each other.

  The decompression processing unit 19 decompresses the compressed video signal and audio signal read from the external memory 18. The video signal expanded by the expansion processing unit 19 or the video signal from the video signal processing unit 13 is sent to the display unit 27 such as a liquid crystal display via the display processing unit 20 and displayed as an image. Further, the audio signal that has been expanded by the expansion processing unit 19 is sent to the speaker 28 via the audio output circuit 21 and output as sound.

  The TG (timing generator) 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and gives the generated timing control signal to each unit in the imaging apparatus 1. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync. The CPU 23 comprehensively controls the operation of each part in the imaging apparatus 1. The operation unit 26 includes a recording button 26a for instructing start / end of moving image shooting and recording, a shutter button 26b for instructing shooting and recording of a still image, an operation key 26c, and the like. Accept the operation. The operation content for the operation unit 26 is transmitted to the CPU 23.

  The operation mode of the imaging apparatus 1 includes a shooting mode in which an image (still image or moving image) can be shot and recorded, and an image (still image or moving image) recorded in the external memory 18 is reproduced and displayed on the display unit 27. Playback mode to be included. Transition between the modes is performed according to the operation on the operation key 26c.

  In the photographing mode, the subject is photographed one after another, and photographed images of the subject are sequentially acquired. A digital video signal representing an image is also called image data. The RAW data is also a kind of image data.

  Since the compression and expansion of image data is not related to the essence of the present invention, the following description ignores the existence of compression and expansion of image data (ie, recording compressed image data, for example). Simply expressed as recording image data). In this specification, image data of a certain image may be simply referred to as an image. Further, in this specification, when the term “display” or “display screen” is used, it refers to the display or display screen in the display unit 27.

  FIG. 3 shows a two-dimensional image space XY. The image space XY is a two-dimensional coordinate system on a spatial domain having the X axis and the Y axis as coordinate axes. The arbitrary two-dimensional image 300 can be considered as an image arranged on the image space XY. The X axis and the Y axis are axes along the horizontal direction and the vertical direction of the two-dimensional image 300, respectively. The two-dimensional image 300 is formed by arranging a plurality of pixels in a matrix in each of the horizontal direction and the vertical direction, and the position of a pixel 301 that is any pixel on the two-dimensional image 300 is (x, y ). In this specification, the position of a pixel is also simply referred to as a pixel position. x and y are coordinate values of the pixel 301 in the X-axis and Y-axis directions, respectively. In the two-dimensional coordinate system XY, when the position of a certain pixel is shifted to the right by one pixel, the coordinate value of the pixel in the X-axis direction increases by 1, and when the position of a certain pixel is shifted downward by one pixel, The coordinate value of the pixel in the Y-axis direction increases by 1. Therefore, when the position of the pixel 301 is (x, y), the positions of the pixels adjacent to the right side, the left side, the lower side, and the upper side of the pixel 301 are (x + 1, y) and (x-1, y), respectively. , (X, y + 1) and (x, y-1).

  The imaging apparatus 1 is provided with a function for expanding the dynamic range of an input image obtained by shooting. FIG. 4 shows a block diagram of a part particularly related to this function. Each part shown in FIG. 4 can be provided in the imaging apparatus 1. For example, the image processing unit 50 and the image data acquisition unit 51 can be provided in the video signal processing unit 13 of FIG. Can be realized. The image processing unit 50 includes each part referred to by reference numerals 53 to 57.

  The image data acquisition unit 51 acquires image data of the input image sequence. The image data of the input image sequence is supplied to the image processing unit 50 and the exposure control unit 52 via the image data acquisition unit 51. An image sequence represented by an input image sequence refers to a collection of a plurality of images arranged in time series. Therefore, the input image sequence is composed of a plurality of input images arranged in time series. The image sequence is also read as a moving image. The input image is a still image represented by RAW data itself, or a still image obtained by performing predetermined image processing (for example, noise removal processing or demosaicing processing) on a still image represented by RAW data. Point to. The image data acquisition unit 51 can also acquire image data of an input image sequence in the reproduction mode. In this case, the image data acquisition unit 51 acquires image data of the input image sequence by reading an arbitrary image sequence recorded in the external memory 18 from the external memory 18 as an input image sequence. However, hereinafter, it is assumed that image data of the input image sequence is acquired in the shooting mode unless otherwise specified.

  The exposure control unit 52 sets exposure conditions for the input image. At this time, based on the image data of the already captured input image sequence, it is possible to set the exposure condition of the input image to be obtained by the subsequent shooting (details will be described later). The input image exposure condition is a condition for determining the exposure state of the input image. The exposure condition of the input image includes the exposure time (that is, shutter speed) of the input image, the aperture value at the time of shooting the input image, and the ISO sensitivity at the time of shooting of the input image. The exposure time of the input image refers to the length of time that the image sensor 33 is exposed to obtain image data of the input image. The ISO sensitivity means sensitivity defined by ISO (International Organization for Standardization), and the brightness (luminance level) of the input image can be adjusted by adjusting the ISO sensitivity. Actually, the amplification factor of the signal amplification in the AFE 12 is determined according to the ISO sensitivity. The amplification degree is proportional to the ISO sensitivity. If the ISO sensitivity is doubled, the degree of amplification is also doubled, whereby the luminance of each pixel of the input image is also doubled (however, saturation is ignored).

  The composition processing unit 53 performs image composition processing for composing a plurality of input images photographed under different exposure conditions, and thereby an HDR (High Dynamic Range) image having a dynamic range wider than the dynamic range of each input image. Is generated. The bit length of the pixel signal at each pixel position of the HDR image is longer than that of the input image. Therefore, the HDR image may not be displayed on the display unit 27 while maintaining the bit length of the HDR image, or the HDR image may not be recorded in the external memory 18.

  For this reason, the gradation conversion unit 54 performs gradation conversion processing for compressing the gradation of the generated HDR image on the HDR image, and generates the HDR image after the gradation conversion processing as a synthesis result image. The conditions for gradation compression in the gradation conversion unit 54, that is, the conditions for gradation conversion processing for generating a composite result image from the HDR image are referred to as gradation conversion conditions. The gradation conversion process and the gradation conversion condition in the gradation conversion unit 54 can also be referred to as a gradation compression process and a gradation compression condition, respectively. The gradation conversion unit 54 can perform gradation conversion processing on the HDR image by individually using a plurality of different gradation conversion conditions, thereby generating a plurality of different composite result images.

  The HDR image and the synthesis result image correspond to images obtained by expanding the dynamic range of the input image. As a method for generating an image in which the dynamic range of an input image is expanded from a plurality of input images, a known method (for example, Japanese Patent Laid-Open No. 2008-109176, Japanese Patent Laid-Open No. 2008-136113, or Japanese Patent Laid-Open No. 2010-93679). Any dynamic range expansion method can be used, including the described method).

  The gradation correction unit 55 corrects the gradation of the input image by performing gradation correction processing on one input image. The input image after gradation correction is called a gradation corrected image. The tone correction unit 55 performs tone correction processing so that the apparent dynamic range of the input image is expanded (in other words, the dynamic range of the input image is pseudo-expanded).

  The tone correction image also corresponds to an image in which the dynamic range of the input image is expanded. As a method for expanding the dynamic range of an input image by gradation correction for one input image, any dynamic range expansion method including a known method (for example, a method described in Japanese Patent Laid-Open No. 2010-93679) is used. Is possible.

  The HDR shooting necessity determination unit 56 (hereinafter sometimes abbreviated as the necessity determination unit 56) determines whether HDR shooting is necessary based on the image data of the input image sequence. HDR imaging refers to imaging of a plurality of input images that are generation sources of the synthesis result image and the gradation correction image. The necessity determination unit 56 or the CPU 23 controls the imaging unit 11 so that a plurality of input images that are generation sources of the synthesis result image and the gradation correction image are captured only when it is determined that HDR imaging is necessary. can do. Only when it is determined that HDR shooting is necessary, the synthesis result image and the gradation correction image are generated. Therefore, the necessity determination unit 56 determines whether the synthesis result image and the gradation correction image need to be generated. It can also be said.

  The priority order setting unit 57 sets priorities for the plurality of synthesis result images obtained from the tone conversion unit 54 and the tone correction images obtained from the tone correction unit 55. The priority order setting unit 57 can set the priority order based on the image data of the input image sequence.

  The image output control unit 58 performs display control for displaying a plurality of synthesis result images and gradation correction images on the display unit 27 and recording control for recording in the external memory 18. The image output control unit 58 can perform display control and recording control according to the priority order set by the priority order setting unit 57.

  Hereinafter, the operation of each part shown in FIG. 4 will be described in more detail.

[Image composition processing]
An example of image composition processing by the composition processing unit 53 will be described. Reference is made to FIGS. 5 (a) and 5 (b). Here, a method of generating the HDR image 333 from the two input images 331 and 332 will be described. The input images 331 and 332 are input images obtained by photographing a common subject, and are taken at a time as close as possible to each other. For example, the input images 331 and 332 are two input images taken continuously.

The exposure control unit 52 makes the exposure condition of the input image 331 different from the exposure condition of the input image 332. Specifically, for example, the exposure time of the input image 331 is set shorter than the exposure time of the input image 332. Here, for the sake of concrete description, it is assumed that the exposure time of the input image 332 that is a long exposure image is twice the exposure time of the input image 331 that is a short exposure image, and the exposure conditions other than the exposure time are input images. Assume that 331 and 332 are the same. The pixel signal at the pixel position (x, y) of the input image 331, the pixel signal at the pixel position (x, y) of the input image 332, and the pixel signal at the pixel position (x, y) of the HDR image 333 are respectively S _S. (X, y), S _L (x, y) and S _H (x, y). The composition processing unit 53 can generate the input image 331 ′ of FIG. 5B by multiplying the pixel signal at each pixel position of the input image 331 by 2. The pixel signal at the pixel position (x, y) of the input image 331 ′ is represented by S _S ′ (x, y). S _S ′ (x, y) = 2 × S _S (x, y). A pixel signal at a certain pixel or pixel position indicates a video signal at the pixel or the pixel position, and the video signal is expressed in a RAW data format, or the video signal includes a luminance signal and a color difference signal.

  In FIG. 6, the solid line 341 represents the relationship between the incident light amount and the signal level in the input image 331, and the solid line 342 represents the relationship between the incident light amount and the signal level in the input image 332. A broken line 341 'represents the relationship between the incident light amount and the signal level in the input image 331'. The incident light amount here refers to the incident light amount to the image sensor 33 per unit time, and the signal level refers to the magnitude of the pixel signal in the input image. At each pixel position, the signal level of the input image 332 is ideally twice the signal level of the input image 331. As a result, the solid line 342 and the broken line 341 'ideally coincide completely. However, since the image sensor 33 is an analog device, there is a non-linearity between the amount of incident light and the accumulated charge amount of each light receiving pixel of the image sensor 33. In addition, non-linearity exists in the amplifier that amplifies the signal of the accumulated charge. For this reason, precisely, the signal level of the input image 332 does not double the signal level of the input image 331, and as a result, the solid line 342 and the broken line 341 'do not completely overlap.

Please refer to FIG. Combining processor 53, for the amount of incident light is light intensity IL ₁ following pixel position, the pixel signal of the input image 332 to be used as a pixel signal of the HDR image 333, and, the amount of incident light is light intensity IL ₂ or more For the pixel position, the pixel signal of the input image 331 ′ is used as the pixel signal of the HDR image 333, and for the pixel position where the incident light amount is greater than the light amount IL ₁ but less than the light amount IL _2. The pixel signal of the HDR image 333 is generated so that the mixed signal of the pixel signal of the input image 332 and the pixel signal of the input image 331 ′ is used as the pixel signal of the HDR image 333. Here, 0 <IL ₁ <IL ₂ .

More specifically, for example,
When S _L (x, y) ≦ S _TH1 is satisfied, S _H (x, y) = S _L (x, y)
When S _S ′ (x, y) ≧ S _TH2 is satisfied, S _H (x, y) is generated so that S _H (x, y) = S _S ′ (x, y). When both S _L (x, y) ≦ S _TH1 and S _S ′ (x, y) ≧ S _TH2 are not satisfied, by mixing S _L (x, y) and S _S ′ (x, y) That is,
S _H (x, y) = k _L · S _L (x, y) + k _S · S _S ′ (x, y)
To generate S _H (x, y). S _TH1 and S _TH2 are signal threshold _values corresponding to the light _amounts IL ₁ and IL ₂ , and satisfy 0 <S _TH1 <S _TH2 .

The coefficients k _S and k _L are
k _S = (S _L (x, y) −S _TH1 ) / (S _TH2 −S _TH1 ), and
k _L = 1−k _S
It is set to satisfy. However, the values of k _S and k _L can be arbitrarily determined within a range satisfying k _S > 0 and k _L > 0 and k _S + k _L = 1 (for example, k _S = k _L = 0. 5 is also possible).

The composition processing unit 53 performs the above-described processing for obtaining the pixel signal S _H (x, y) of the HDR image 333 for each pixel position. Thereby, pixel signals at all pixel positions of the HDR image 333 are obtained (that is, the HDR image 333 is completed). Since the input image 331 ′ is an image obtained by doubling each pixel signal of the input image 331, the signal level of the relatively high luminance portion of the input image 331 ′ is the signal of the image sensor 33. The saturation level S _SAT is exceeded (see FIG. 7). Accordingly, the dynamic range 352 of the HDR image 333 is wider than the dynamic range 351 of the image sensor 33.

[Tone conversion processing]
Next, the gradation conversion process by the gradation conversion unit 54 will be described. As an example, a method of performing gradation conversion processing on the above-described HDR image 333 (see FIG. 5A) will be described. An image 360 in FIG. 8 is an image obtained by performing gradation conversion processing on the HDR image 333, and corresponds to the above-described synthesis result image (see FIG. 4). Pixel position of the synthesized result image 360 (x, y) of the pixel signal in representing at _{S H '(x, y)} .

As described above, the gradation conversion unit 54 can perform gradation conversion processing on the HDR image 333 using a plurality of mutually different gradation conversion conditions. The tone conversion process described below is also called tone mapping, and the bit length BL _HDR of each pixel signal of the HDR image 333 is converted to a desired bit length BL _OUT by tone mapping. 0 <BL _OUT <BL _HDR . For example, the bit length BL _OUT is set to be the same as the bit length of an image that can be displayed on the display unit 27, or is set to be the same as the bit length of an image according to a predetermined image recording format. The bit length BL _OUT may be the same as the bit length of each pixel signal of the input image. The plurality of gradation conversion conditions include different first and second gradation conversion conditions.

A gradation conversion process using the first gradation conversion condition (hereinafter also referred to as a first gradation conversion process) will be described. In the first gradation conversion process, a common gray-level transformation function is set for the entire HDR image 333, and each pixel of the HDR image 333 is set using the common gradation transformation function. The signal is converted into each pixel signal of the synthesis result image 360. A graph representing the tone conversion function is generally called a tone curve. FIG. 9 shows a tone curve 380 used when the first gradation conversion condition is adopted. Gradation conversion function represented by the tone curve 380, the pixel signals of the HDR image 333 _S H (x, y) pixel signal with synthesized result image _{360 S H '(x, y} ) a function which defines the relationship between is there. The tone conversion process using the tone curve 380 is the most common tone conversion process, but natural tone compression can be realized.

A gradation conversion process using the second gradation conversion condition (hereinafter also referred to as a second gradation conversion process) will be described. In the second tone conversion processing, painting-like tone mapping with a high degree of freedom in tone compression is possible. Hereinafter, an example of a method will be described. In the second gradation conversion processing, first, as shown in FIG. 10, the HDR image 334 is generated by performing simple bit length compression on the HDR image 333. That is, for example, as shown in FIG. 11, the HDR image 334 is generated by truncating the lower (BL _HDR -BL _OUT ) bits of the digital signal representing each pixel signal of the HDR image 333. Then, the pixel position of the HDR image 334 (x, y) pixel signal _S HA in (x, y) digital values of a pixel signal _S H (x, y) _{1 /} digital values (BL HDR _{-BL OUT)} become. On the other hand, the bit length of each pixel signal of the HDR image 334 is BL _OUT .

FIG. 12A shows a luminance histogram HST ₃₃₄ of the HDR image 334. In the second gradation conversion process, the reference luminance L _REF is set, and the luminance histogram HST ₃₃₄ of the HDR image ₃₃₄ is changed so that the luminance level is concentrated near the reference luminance L _REF . The luminance histogram after this change is the luminance histogram HST ₃₆₀ of the synthesis result image 360 shown in FIG. Specifically, the following may be performed.

All the pixels of the HDR image 334, the dark pixel is a pixel having a lower luminance than the reference brightness _{L REF,} or, classifying the bright pixel with the reference luminance _{L REF} more brightness. Each pixel signal of each dark pixel is multiplied by a gain G _{UP of} 1 or more, while each pixel signal of each bright pixel is multiplied by a gain G _{DOWN of} 1 or less. A pixel signal is generated. That is, the pixel signal S _H ′ (x, y) at the pixel position (x, y) of the synthesis result image 360 is obtained when the pixel at the pixel position (x, y) of the HDR image 334 is a dark pixel.
S _H ′ (x, y) = G _UP × S _HA (x, y)
When the pixel at the pixel position (x, y) of the HDR image 334 is a bright pixel,
S _H ′ (x, y) = G _DOM × S _HA (x, y)

The gain G _UP may be set to a value larger than 1. Further, it is preferable to set the gain G _UP value for each dark pixel according to the brightness of the dark pixel so that (G _UP −1) increases as the difference between the brightness of the dark pixel and the reference brightness L _REF increases. . However, it is also possible to fix the value of the gain G _{UP to} a constant value larger than 1.
The gain G _DOWN may be set to a value smaller than 1 (where G _DOWN > 0). In addition, the value of the gain G _DOWN may be set for each bright pixel according to the brightness of the bright pixel so that (1-G _DOWN ) increases as the difference between the brightness of the bright pixel and the reference brightness L _REF increases. . However, it is also possible to fix the value of G _{DOWN to} a constant value smaller than 1.

It is arbitrary what luminance is set as the reference luminance _LREF . Usually, the reference luminance L _REF is set near the intermediate gradation of the synthesis result image 360. The reference luminance L _REF can be set based on the luminance having the maximum frequency in the luminance histogram HST ₃₃₄ .

By multiplying the pixel signal of each dark pixel by the gain G _UP and multiplying the pixel signal of each bright pixel by the gain G _DOWN , the luminance frequency is concentrated in the vicinity of the reference luminance L _REF . The state of concentration can be seen from the luminance histogram HST _{360 in} FIG. It can be said that the synthesis result image 360 corresponding to the luminance histogram HST ₃₆₀ is a pictorial HDR image. A picture-like image is an image having an atmosphere of a picture drawn on paper using an art material such as paint. A picture drawn on paper using an art material such as a paint generally has high saturation but little change in shading. In the second gradation conversion process, since the change in shading is reduced while maintaining the saturation, a painting-like composite result image 360 sometimes having a fantastic atmosphere is obtained.

  A known method can be used as the second gradation conversion processing method for obtaining the painting-like composite result image 360. For example, the non-patent document “Raanan Fattal, the other two,“ Gradient Domain High Dynamic Range Compression ", [online], [October 31, 2010 search], Internet <URL: http://www.cs.huji.ac.il/~danix/hdr/>", or Non-patent literature “Erik Reinhard, three others,“ Photographic Tone Reproduction for Digital Images ”, [online], [October 31, 2010 search], Internet <URL: http://www.cs.utah.edu The gradation conversion method described in “/ ~ reinhard / cdrom />” can also be used.

[Tone correction processing]
Next, the gradation correction process by the gradation correction unit 55 will be described. As an example, a method of performing gradation correction processing on an input image 331 (see FIG. 5A) as a short exposure image will be described. An image 390 in FIG. 13 is an image obtained by performing gradation correction processing on the input image 331, and corresponds to the above-described gradation correction image (see FIG. 4). A pixel signal at the pixel position (x, y) of the gradation correction image 390 is represented by S _CS (x, y).

In FIGS. 14 (a) and (b), respectively show, the luminance histogram _{HST 390} of the luminance histogram _{HST 331} and tone correction image 390 of the input image 331. A solid line 395 in FIG. 14C represents a gain function used when obtaining the gradation corrected image 390 from the input image 331. Each pixel signal of the gradation-corrected image 390 is obtained by multiplying a gain G _C according to the gain function 395 in each pixel signal of the input image 331. That is,
S _CS (x, y) = G _C × S _S (x, y).

The value of the gain G _C is set for each pixel position according to the respective pixel positions of the input image 331 brightness. G _CH and _{G CL,} respectively, an upper limit value and the lower limit of the numerical range gain _{G C} can take. Gain _{G C} to be multiplied to the pixel signal _{S S} (x, y) is set pixel position of the input image 331 (x, y) to the upper limit value _{G CH} when the brightness in is zero or near zero, the input image 331 as the brightness at the pixel position (x, y) is increased from zero, is caused to decrease toward the lower limit value G _CL from the upper limit value G _CH. This decrease may be a linear decrease or a non-linear decrease. 0 <G _CL <G _CH , for example, G _CL = 1 and G _CH = 2.

By the gradation correction using the gain G _C as described above, the gradation of the input image 331 each pixel of the short-exposure image is shifted to the high luminance side, the degree of shift is larger as against a dark pixels . As a result, the image information on the low luminance side is expressed by using a relatively large bit length while maintaining the image information on the high luminance side, and the gradation corrected image 390 whose apparent dynamic range is expanded. Can be generated.

  In the above-described method example, the gradation correction image is generated from the input image 331 as the short exposure image, but the gradation correction image can also be generated from the input image 332 as the long exposure image.

[Necessity determination of HDR shooting]
Next, a method for determining necessity of HDR shooting by the necessity determination unit 56 of FIG. 4 will be described. Reference is made to FIGS. 15 (a) and 15 (b). The necessity determination unit 56 determines whether or not HDR shooting is necessary based on the image data of the input image shot during the HDR shooting necessity determination period (hereinafter sometimes referred to as a determination period). it can. As time progresses, time t _A , t _B and t _C shall be visited in this order. The determination period is a period from time t _A to time t _B. A period from time t _B to time t _C is referred to as a main imaging period. The main shooting period is a period during which a plurality of input images that are the basis of the synthesis result image and the gradation correction image are shot. In the following, an input image shot during the main shooting period is also referred to as a target input image. (Refer FIG.15 (b)). Each of the input images 331 and 332 in FIG. 5A is a target input image.

Further, referred to the input images taken before time t _B with the preview image, further, also called determination image each input image to be photographed in the determination period (see FIG. 15 (b)). The determination image is a kind of preview image. The preview images are sequentially taken at a predetermined frame period, and the obtained preview image sequence is displayed as a moving image on the display unit 27 and used for composition confirmation by the user.

The time t _B that is the start time of the main photographing period can be determined by a user instruction. Time t _A is the start time of the judgment period may be determined by a user's instruction, the time t may be determined on the basis of the _B (e.g., time t _B before a predetermined time from the time t _A May be determined). Here, similarly to the time t _B, assume that also the time t _A is determined by the instruction of the user, half-pressed the shutter button 26b, point to full-press operation is performed, respectively, the time t _A , T _B. The shutter button 26b is a button that can be pressed in two steps. When the user lightly presses the shutter button 26b from an open state where no pressure is applied to the shutter button 26b, the shutter button 26b is in a half-pressed state. If the user further presses the shutter button 26b from that state, the shutter button 26b is fully pressed. An operation for changing the state of the shutter button 26b from the open state to the half-pressed state is a half-press operation, and an operation for changing the state of the shutter button 26b from the half-pressed state to the fully-pressed state is a full-press operation.

In determination period, the exposure control unit 52 (see FIG. 4) sets the long exposure time ET _L than the exposure time ET _S and exposure time ET _S, as shown in FIG. 16, the input image by the exposure time ET _S It performs exposure control such photographing and the photographing of the input image by the exposure time ET _L are alternately performed. In determination period, as and exposure time ET _L so that the exposure time ET _S is converged to a certain exposure time _{ET SO} converges to a specific exposure time _{ET LO,} exposure time ET _S and ET _L are sequentially changed The The exposure time ET _LO is longer than the exposure time ET _SO .

In FIG. 16, images 411 to 416 are determination images that are continuously captured in the determination period. Of determination image, the determination image that has been captured by the exposure time ET _S is also called short-exposure determination image, the determination image that has been captured by the exposure time ET _L referred to as the long-exposure determination image. Images 411, 413, and 415 are the first, second, and third short exposure determination images, respectively, and images 412, 414, and 416 are the first, second, and third images, respectively. It is an image for long exposure determination.

Figure 17 (a) and (b), respectively, represent the luminance histogram HST _L of the luminance histogram HST _S and any length exposure determination image of any short-exposure determination image. The necessity determination unit 56 is based on the image data of the short exposure determination image and the long exposure determination image, and the total number P _NUMS and long exposure determination of the pixels having a luminance _equal to or higher than a predetermined reference luminance L _TH1 in the short exposure determination image. The total number P _NUML of pixels having a luminance equal to or lower than a predetermined reference luminance L _{TH2 in the} image for use is obtained. The total number P _NUMS corresponds to the area of the hatched area in FIG. 17A, and the total number P _NUML corresponds to the area of the hatched area in FIG. The total number P _NUMS is derived for each short exposure determination image, and the total number P _NUML is derived for each long exposure determination image.

Exposure control unit 52, referring to the value of the derived _{P nums,} with searching for the exposure time ET _S to the _{P nums} below the reference value REF _S, with reference to the derived value of _{P NUML} searches the exposure time ET _L to the _{P NUML} below the reference value REF _L. More specifically, based on the _{P nums} derived for the short-exposure determination image 411, _{P nums} will be derived for the short-exposure determination image 413 or 415 is _{"REF S -ΔREF S ≦ P NUMS} ≦ REF so as to satisfy _S ", the exposure control unit 52 adjusts the exposure time ET _S for short-exposure determination image 413 or 415. Similarly, based on the P _NUML derived for the long exposure determination image 412, the P _NUML to be derived for the long exposure determination image 414 or 416 _satisfies “REF _L −ΔREF _L ≦ P _NUML ≦ REF _L ”. as satisfied, the exposure control unit 52 adjusts the exposure time ET _S for long exposure determination image 414 or 416. Each value of REF _S , REF _L , ΔREF _S and ΔREF _L can be set in advance. REF _S , REF _L , ΔREF _S and ΔREF _L all have positive integer values, REF _S > ΔREF _S , and REF _L −ΔREF _L.

By adjusting the exposure time ET _S and ET _L, the determination period, the exposure time ET _S converge to a specific exposure time _{ET SO} and the exposure time ET _L is converged to a certain exposure time _{ET LO.} For example, as a result of these adjustments, P _NUMS derived for the short exposure determination image 415 is derived so as to _satisfy “REF _S −ΔREF _S ≦ P _NUMS ≦ REF _S ” and for the long exposure determination image 416. and _{P NUML} is to meet the _{"REF L -ΔREF L ≦ P NUML} ≦ REF L", the exposure time of the determination image 415 and 416 are set. Then, the substitutes exposure time ET _S short-exposure determination image 415 to the exposure time _{ET SO,} can be substituted for the exposure time ET _L long exposure determination image 416 to the exposure time _{ET LO.}

The necessity determination unit 56 determines whether HDR shooting is necessary based on the exposure conditions of the short exposure determination image and the long exposure determination image obtained through the above adjustment. In particular,
Necessity determination inequality “ET _LO / ET _SO ≧ J”
Is satisfied, it is determined that HDR imaging is necessary. On the other hand, if the necessity determination inequality is not satisfied, it is determined that HDR imaging is unnecessary. J is a predetermined value larger than 1, for example, J = 2. As will be described later, the exposure times of the input images 331 and 332 (see FIG. 5) photographed during the main photographing period are set with reference to the exposure times ET _SO and ET _LO . Considering this, when “ET _LO / ET _SO <J” (that is, when there is not much difference between ET _SO and ET _LO ), even if processing for expanding the dynamic range is performed, The effect of the treatment is small or little. From this, the significance of the above necessity determination inequality is understood.

[About the target input image]
When it is determined that HDR shooting is necessary, the number of target input images shot during the main shooting period may be two (see FIGS. 15A and 15B). In this case, in the main shooting period, the two target input images 501 and 502 shown in FIG. 18 are continuously shot. The target input images 501 and 502 are the same as the input images 331 and 332 in FIG. The exposure control unit 52 sets the exposure times of the target input images 501 and 502 based on the exposure times ET _SO and ET _LO , respectively. For example, the exposure time ET _SO can be set as it is as the exposure time of the target input image 501 and the exposure time ET _LO can be set as it is as the exposure time of the target input image 502. Alternatively, for example, the exposure time ET _SO ′ may be obtained by increasing or decreasing the exposure time ET _SO according to a predetermined rule, and the exposure time ET _SO ′ may be set as the exposure time of the target input image 501. Similarly, the exposure time ET _LO ′ may be obtained by increasing or decreasing the exposure time ET _LO according to a predetermined rule, and the exposure time ET _LO ′ may be set as the exposure time of the target input image 502. However, the exposure time ET _LO ′ is longer than the exposure times ET _SO and ET _SO ′.

By applying the first HDR processing to the target input images 501 and 502, a synthesis result image 511 is obtained, and by applying the second HDR processing, a synthesis result image 512 is obtained and the third HDR processing is performed. As a result, a gradation corrected image 513 is obtained. The first and second HDR processes are performed by the synthesis processing unit 53 and the gradation conversion unit 54, and the third HDR process is performed by the gradation correction unit 55.
The first HDR process includes an image composition process and a first gradation conversion process. That is, after an HDR image is generated by an image combining process that combines the target input images 501 and 502, a first gradation conversion process is performed on the HDR image to obtain a combined result image 511. The synthesis result image 511 corresponds to the above-described synthesis result image 360 (see FIG. 8) when the first gradation conversion process is used.
The second HDR process includes an image composition process and a second gradation conversion process. That is, after an HDR image is generated by an image combining process that combines the target input images 501 and 502, a second gradation conversion process is performed on the HDR image to obtain a combined result image 512. The synthesis result image 512 corresponds to the above-described synthesis result image 360 (see FIG. 10) when the second gradation conversion process is used.
The third HDR process is a gradation correction process. That is, a gradation correction image 513 is obtained by performing gradation correction processing on the target input image 501 or 502. The tone correction image 513 corresponds to the tone correction image 390 of FIG.

When it is determined that HDR shooting is necessary, the number of target input images shot during the main shooting period may be three or more. For example, during the main shooting period, the five target input images 521 to 525 shown in FIG. 19 may be continuously shot. The target input images 521 to 525 are obtained by photographing a common subject. Exposure control unit 52, the exposure time of the target input image 521, 523 and 525, set on the basis of the exposure time _{ET SO.} For example, the exposure time ET _SO or ET _SO ′ can be set to the exposure time of the target input images 521, 523, and 525. The exposure conditions of the target input images 521, 523, and 525 may be the same or different from each other. The exposure control unit 52 sets the exposure time of the target input images 522 and 524 with reference to the exposure time ET _LO . For example, the exposure time ET _LO or ET _LO ′ can be set to the exposure time of the target input images 522 and 524. The exposure conditions of the target input images 522 and 524 may be the same or different from each other. Although different from the above description, the exposure time ET _LO or ET _LO ′ can be set as the exposure time of the target input image 525.

A composite result image 531 is obtained by subjecting the target input images 521 and 522 to the first HDR processing. That is, an HDR image is generated by an image combining process that combines the target input images 521 and 522, and then a first gradation conversion process is performed on the HDR image to obtain a combined result image 531.
By performing the second HDR processing on the target input images 523 and 524, a synthesis result image 532 is obtained. That is, an HDR image is generated by an image combining process that combines the target input images 523 and 524, and then a second gradation conversion process is performed on the HDR image to obtain a combined result image 532.
A gradation correction image 533 is obtained by performing a third HDR process on the target input image 525. That is, a gradation correction image 533 is obtained by performing gradation correction processing on the target input image 525.

  In the following (see FIGS. 18 and 19), the synthesis result image (511 or 531) obtained by the first HDR processing is also referred to as a first output HDR image, and the synthesis obtained by the second HDR processing. The result image (512 or 532) is also referred to as a second output HDR image, and the tone correction image (513 or 533) obtained by the third HDR process is also referred to as a third output HDR image.

[Operation flowchart]
Before describing the function of the priority setting unit 57 shown in FIG. 4, the operation procedure of the imaging apparatus 1 will be described with reference to FIG. FIG. 20 is a flowchart showing a procedure of operations involved in HDR imaging.

  When the operation in the shooting mode is started, shooting of the preview image sequence is started in step S11. The obtained preview image sequence is displayed as a moving image. On the other hand, in step S12, the CPU 23 monitors whether or not a half-press operation has been performed on the shutter button 26b. When the half-press operation is performed, a transition from step S12 to step S13 occurs. In step S13, the necessity determination unit 56 determines whether HDR shooting is necessary based on the image data of the determination image that is a kind of preview image. On the other hand, in step S14, the shutter button 26b is fully pressed. The CPU 23 monitors whether or not the above has been performed. In step S14, when it is confirmed that the full-press operation has been performed, a transition to step S15 occurs, and thereafter, the processing in steps S16 to S19 or the processing in step S20 is executed. When the necessity determination unit 56 determines that HDR shooting is necessary, the processing of steps S16 to S19 is sequentially executed, and when it is determined that HDR shooting is unnecessary, the processing of step S20 is executed. .

  In step S <b> 20, one input image is photographed, and the obtained input image is displayed and recorded in the external memory 18. On the other hand, in step S16, a plurality of target input images (for example, target input images 501 and 502 in FIG. 18 or target input images 521 to 525 in FIG. 19) are continuously photographed under a plurality of exposure conditions. The In step S17, the first to third HDR processes are executed on the plurality of target input images obtained in step S16, thereby generating the first to third output HDR images.

  In the subsequent step S18, the priority order setting unit 57 of FIG. 4 executes a process for setting the priority order for the first to third output HDR images (hereinafter referred to as a priority order setting process). Through the priority order setting process, the first, second, or third priority order is assigned to each of the first to third output HDR images. The first priority is a higher priority than the second priority, and the second priority is a higher priority than the third priority. In subsequent step S <b> 19, the image output control unit 58 of FIG. 4 displays the first to third output HDR images and records them in the external memory 18 according to the priority order. In step S19, only one or two output HDR images may be displayed, or only one or two output HDR images may be recorded in the external memory 18.

  As examples of the display and recording methods of the first to third output HDR images according to the priority order, the first to third display / recording methods will be described.

A first display / recording method will be described. In the first display / recording method, as shown in FIG. 21A, different display areas 551 to 553 are provided in the entire display area of the display screen, and unless otherwise specified by the user, FIG. ), The output HDR images to which the first to third priorities are given are displayed in the display areas 551 to 553, respectively. Here, the size of the display area 551 is larger than those of the display areas 552 and 553. The size of the display area 552 may be larger than that of the display area 553, and the sizes of the display areas 552 and 553 may be the same. The user can switch the output HDR image displayed in the display area 551 among the first to third output HDR images by a predetermined operation or a touch panel operation on the operation unit 26.
In the first display / recording method, it is preferable that only the output HDR image displayed in the display area 551 is recorded in the external memory 18 when the user instructs to record. The recording instruction is realized by a predetermined operation on the operation unit 26 or a touch panel operation.

A second display / recording method will be described. In the second display / recording method, different display areas 561 to 564 are provided in the entire display area of the display screen as shown in FIG. 21C, and display areas 562 to 564 are provided as shown in FIG. In addition, the output HDR images to which the first to third priorities are given are displayed. The display areas 562 to 564 can have the same size. On the other hand, the size of the display area 561 is larger than those of the display areas 562 to 564. In the display area 561, an output HDR image selected from the first to third output HDR images is enlarged and displayed. Unless there is a special instruction from the user, the output HDR image to which the first priority is given is displayed in the display area 561 (see FIG. 21D). The user can switch the output HDR image displayed in the display area 561 among the first to third output HDR images by a predetermined operation or a touch panel operation on the operation unit 26.
In the second display / recording method, it is preferable to record only the output HDR image displayed in the display area 561 in the external memory 18 when the user instructs to record.

A third display / recording method will be described. In the third display / recording method, only one output HDR image is displayed at one time as shown in FIG. Unless there is a special instruction from the user, only the output HDR image to which the first priority is given is displayed. The user can switch the output HDR image displayed on the display screen among the first to third output HDR images by giving a scroll instruction to the imaging apparatus 1. The scroll instruction is realized by a predetermined operation on the operation unit 26 or a touch panel operation. When a scroll instruction for one time is made in a state where the output HDR image having the first priority is displayed, the display image is changed from the output HDR image having the first priority to the output HDR image having the second priority. When the scroll instruction for one time is made while the output HDR image of the second priority is displayed, the display image is changed from the output HDR image of the second priority to the first or third priority. The output HDR image is switched to.
In the third display / recording method, it is preferable to record only the displayed output HDR image in the external memory 18 when the user instructs to record.

  In the first to third display / recording methods, only the output HDR image having the first priority may be always recorded in the external memory 18, or the first, second, and third methods may be used. The first to third output HDR images are rearranged so that the priority output HDR images are recorded in the external memory 18 first, second, and third, respectively. All the output HDR images may be recorded in the external memory 18. The above-described display / recording method examples according to the priorities are merely examples, and any method that gives some order according to the priorities is adopted for the display or recording of the first to third output HDR images. Is possible.

[Priority setting processing]
The priority order setting unit 57 can set the priority order for the first to third output HDR images based on the setting index extracted from the image data of the preview image sequence. The priority setting process will be described in detail while explaining the setting index extraction method.

To set the priority exposure control unit 52 (see FIG. 4) sets the two exposure times ET _S and ET _L, of the input image by the exposure time ET _S shooting with the input image by the exposure time ET _L Exposure control is performed so that shooting is performed alternately. The period during which such exposure control is performed is set before the main photographing period in FIG. 15A, and may be the same as or different from the above-described determination period. Here is performed to set the priority, alternating shooting the input image by the exposure time ET _S and the input image according to the exposure time ET _L is considered to be made during determination period in FIG. 15 (b). Then, the image data of the determination images 411 to 416 in FIG. 16 is used for setting the priority order.

  A motion detection unit 71 (see FIG. 22) for deriving an optical flow between any two input images can be provided in the priority order setting unit 57 or the image processing unit 50 in FIG. The motion detection unit 71 derives an optical flow between two short exposure determination images that are temporally adjacent and derives an optical flow between two long exposure determination images that are temporally adjacent. For the determination images 411 to 416, as shown in FIG. 23, an optical flow between the images 411 and 413, an optical flow between the images 412 and 414, an optical flow between the images 413 and 415, and an image 414 and The optical flow between 416 is derived.

  The optical flow between the first and second input images is a bundle of motion vectors of objects appearing on the first and second input images. The motion vector of a certain object is a vector quantity representing the direction and magnitude of the motion of the object. For each derived optical flow, the motion detection unit 71 obtains an average value or integrated value of the magnitudes of motion vectors forming the optical flow as a motion evaluation value. The priority order setting unit 57 can set the priority order after using the motion evaluation value as the first setting index for setting the priority order (see FIGS. 24A and 24B for details of the setting). Reference later).

  Further, the flicker detection unit 72 of FIG. 22 can be provided in the priority order setting unit 57 or the image processing unit 50 of FIG.

  The image sensor 33 has an electronic shutter function, and performs exposure of each light receiving pixel by a so-called rolling shutter. In the rolling shutter, the timing (time) at which each light receiving pixel on the imaging surface is exposed differs for each horizontal line. That is, on the imaging surface, the exposure timing is different between different horizontal lines. When photographing is performed using an imaging device 33 (XY address type CMOS image sensor or the like) having a rolling shutter characteristic under illumination of a non-inverter type fluorescent lamp, the exposure time of each input image is set as the light emission cycle of the fluorescent lamp. If it is made considerably shorter, the luminance unevenness in the vertical direction occurs in each input image. Such luminance unevenness that can occur in each input image is called intra-frame flicker. The non-inverter type fluorescent lamp means a fluorescent lamp that is directly lit by a commercial AC power source without using an inverter. Further, when the photographing frame rate and the frequency of the commercial AC power source that drives the non-inverter type fluorescent lamp are different, the state of uneven luminance in the vertical direction changes between frames, and luminance flicker occurs in the time direction. Such luminance flicker occurring in the time direction (flicker due to luminance change between different input images) is called inter-frame flicker.

Flicker detection unit 72, the exposure time ET _S by the input image photographing and the exposure time ET _L by the input image determination image string obtained by performing alternately shooting and the (e.g., determination image 411 in FIG. 16 416), the presence or absence of flicker can be detected. The flicker that is subject to presence / absence detection includes at least intra-frame flicker and may further include inter-frame flicker. The flicker detection unit 72 can use a known flicker detection method (for example, Japanese Patent Application Laid-Open No. 2008-109253). For example, to set the frame rate of the determination image column 60fps (frame per second), and sets the exposure time ET _L with an exposure time ET _S is set to 1/240 second to 1/60 second. In this case, if the light source of the image pickup apparatus 1 is a non-inverter type fluorescent lamp, luminance unevenness occurs in the vertical direction in the short exposure determination image, while luminance unevenness in the vertical direction does not occur in the vertical direction. It hardly occurs. Therefore, by comparing the vertical luminance distribution between the temporally adjacent short exposure determination image and the long exposure determination image, it is determined whether or not flicker exists in the determination image sequence. Can do. If the flicker detection unit 72 determines that flicker is present in the determination image sequence, it can determine that flicker is also present in the target input image sequence.

  The flicker detection unit 72 outputs a flicker presence / absence signal indicating whether or not flicker exists in the determination image sequence and the target input image sequence. When it is determined that flicker exists in the determination image sequence and the target input image sequence, a flicker presence / absence signal having a value of “1” is output, and it is not determined that flicker exists in the determination image sequence and the target input image sequence. In this case, a flicker presence / absence signal having a value of “0” is output. The priority order setting unit 57 can set the priority order using the flicker presence / absence signal as the second setting index for setting the priority order (see FIGS. 24A and 24B for details of the setting). Reference later).

  Further, the saturation detection unit 73 of FIG. 22 can be provided in the priority order setting unit 57 or the image processing unit 50 of FIG. The saturation detection unit 73 extracts one or more determination images from the determination image obtained in the determination period, and calculates a saturation evaluation value corresponding to the saturation of the extracted determination image. An arbitrary determination image can be extracted from the determination image within the determination period. In this case, if the average value of the saturation of each pixel of the extracted determination image is calculated as the saturation evaluation value Good. Further, for example, it is possible to extract an arbitrary plurality of determination images from the determination image within the determination period, and in this case, an average value of the saturation in each pixel of the extracted plurality of determination images is a saturation evaluation value Can be calculated as The priority order setting unit 57 can set the priority order after using the saturation evaluation value as the third setting index for setting the priority order.

  During the determination period, it is possible to extract the first to third setting indexes or information based on the first to third setting indexes from the image data of the preview image sequence and store them in the internal memory 17. In this case, the priority setting unit 57 can obtain the first to third setting indices from the stored information after the full pressing operation. However, the image data of the preview image sequence is temporarily recorded in the internal memory 17 or the external memory 18 as much as necessary, and the first to third setting indexes are derived from the temporarily recorded image data after the full pressing operation. You may make it do.

  The priority order setting unit 57 can set the priority order based on at least one of the first to third setting indices.

FIG. 24A shows the setting contents of the priority order when the first and third setting indexes (that is, the motion evaluation value and the saturation evaluation value) are used. The priority setting unit 57 determines that “the motion is large” when the motion evaluation value is equal to or greater than a predetermined reference motion value, and determines that “the motion is small” when the motion evaluation value is less than the reference motion value. The priority setting unit 57 determines that “the saturation is high” when the saturation evaluation value is equal to or higher than the predetermined reference saturation value, and “saturation” when the saturation evaluation value is less than the reference saturation value. It is judged that the degree is low. And
When it is judged that the saturation is high and the movement is small, according to the first rule,
When it is judged that the saturation is low and the movement is small, according to the second rule,
When it is judged that the saturation is low and the movement is large, according to the third rule,
When it is judged that the saturation is high and the movement is large, according to the fourth rule,
Set the priority.

When the first rule is applied, the first, second, and third priorities are given to the second, first, and third output HDR images, respectively.
When the second rule is applied, the first, second, and third output HDR images are given first, second, and third priorities, respectively.
When the third rule is applied, first, second, and third priorities are given to the third, first, and second output HDR images, respectively.
When the fourth rule is applied, the first, second, and third priorities are given to the third, second, and first output HDR images, respectively.

FIG. 24B shows the setting contents of the priority order when the second and third setting indexes (that is, the flicker presence / absence signal and the saturation evaluation value) are used. When the value of the flicker presence / absence signal is “1”, it is determined that there is flicker, and when the value of the flicker presence / absence signal is “0”, it is determined that there is no flicker.
When it is judged that the saturation is high and there is no flicker, according to the first rule,
If it is determined that the saturation is low and there is no flicker,
If it is determined that the saturation is low and flicker is present,
If it is determined that the saturation is high and flicker is present,
Priorities can be set.

When the saturation of the input image is high, it is presumed that the user desires to acquire a bright image with high saturation, so that the painting-like image is more effective than the first output HDR image using natural tone mapping. The second output HDR image, which should be referred to as the HDR image, is more likely to match the user's preference. Considering this, when it is determined that the saturation is high from the saturation evaluation value, the priority given to the second output HDR image is higher than that of the first output HDR image.
On the contrary, when the saturation of the input image is low, it is presumed that a painting-like HDR image is relatively unlikely to be desired. Therefore, the output HDR image using natural tone mapping is preferred by the user. It seems to be rare. Considering this, when it is determined that the saturation is low from the saturation evaluation value, the priority given to the first output HDR image is higher than that of the second output HDR image.

  In addition, when the movement of an object on the input image is large or flicker exists, it is often difficult to combine a plurality of images, and the combination may cause blurring of the outline, generation of a double image, or the like. Considering this, when it is determined that the motion is large from the motion evaluation value, or when it is determined that there is flicker from the flicker presence / absence signal, the priority given to the third output HDR image is set to the first and the second priority. 2 higher than those of the output HDR image. Conversely, when it is determined that the motion is small from the motion evaluation value or when it is determined that there is no flicker from the flicker presence / absence signal, it is considered that the dynamic range expansion by combining a plurality of images functions effectively. The priority given to the output HDR image is set lower than those of the first and second output HDR images.

In FIGS. 24A and 24B, the priority order is determined using two setting indexes, but the priority order may be determined using all of the first to third setting indexes. In this case, the first condition that the movement is small, the second condition that there is no flicker, and the third condition that the saturation is high are set. And for example,
When all of the first to third conditions are satisfied, according to the first rule,
If the third condition is satisfied but at least one of the first and second conditions is not satisfied, according to the fourth rule,
If the third condition is not met but both the first and second conditions are met, according to the second rule:
If the third condition is not satisfied and at least one of the first and second conditions is not satisfied, according to the third rule,
Priorities can be set.

It is also possible to determine the priority order using only the first setting index.
In this case, for example,
When it is judged that the movement is small, according to the first rule,
When it is determined that the movement is large, priority is set according to the fourth rule.
Or, for example,
When it is judged that the movement is small, according to the second rule,
When it is determined that the movement is large, priority is set according to the third rule.

It is also possible to determine the priority order using only the second setting index.
In this case, for example,
When it is determined that there is no flicker, according to the first rule,
When it is determined that there is flicker, priority is set according to the fourth rule.
Or, for example,
When it is determined that there is no flicker,
When it is determined that there is flicker, priority is set according to the third rule.

It is also possible to determine the priority order using only the third setting index.
In this case, for example,
When it is judged that the saturation is high, according to the first rule,
When it is determined that the saturation is low, priority is set according to the second rule.
Or, for example,
When it is judged that the saturation is high, according to the fourth rule,
When it is determined that the saturation is low, priority is set according to the third rule.

  According to the imaging device 1 described above, a plurality of output HDR images can be generated by a plurality of HDR processes. Therefore, the user can selectively display or record an output HDR image that suits his / her preference from among a plurality of output HDR images. In addition, by the display control or the recording control according to the priority order, it is possible to display or record the output HDR image presumed to meet the user's preference with priority. Needless to say, priority display or priority recording according to the user's preference contributes to improving the convenience of the user (reducing the operation burden on the user). Even if the output HDR image given a high priority does not meet the user's preference, the user can selectively select the desired output HDR image from the output HDR images given a low priority. Can be displayed or recorded. Moreover, it is possible to avoid execution of HDR imaging and HDR processing that are less necessary by determining whether HDR imaging is necessary.

  Of course, the HDR processing is executed only when it is determined that HDR shooting is necessary and the target input image is shot. Therefore, it can be said that the necessity determination of the HDR shooting by the necessity determination unit 56 is the necessity determination of the HDR processing (determination of whether or not the HDR processing is necessary), and the necessity determination of the output HDR image (output HDR) It can also be said that the determination of whether or not image generation is necessary.

[Generate only high priority images]
In the operation of FIG. 20, the priority order is set for the generated first to third output HDR images. However, the priority order setting process in step S18 is executed before the process in step S17, Of the third output HDR images, only the output HDR images set with the first priority order or only the output HDR images set with the first and second priority orders may be generated. . A method for realizing such generation is referred to as a method α for convenience.

In method α, for example,
Under the circumstances where the first rule applies,
Of the first to third output HDR images, only the second output HDR image or only the second and first output HDR images are generated,
In situations where the second rule above applies,
Of the first to third output HDR images, only the first output HDR image or only the first and second output HDR images are generated,
Under circumstances where the above third rule applies:
Of the first to third output HDR images, only the third output HDR image or only the third and first output HDR images are generated,
Under the circumstances where the fourth rule applies,
Of the first to third output HDR images, only the third output HDR image or only the third and second output HDR images are generated.
The image output control unit 58 of FIG. 4 can display the generated output HDR image on the display unit 27 and can record the output HDR image in the external memory 18.

  In the method α, only one or two HDR processes among the first to third HDR processes are selectively executed, and one or two output HDRs of the first to third output HDR images are selected. Only an image is generated. For this reason, in the method α, it can be said that the content of the HDR processing for generating the output HDR image is changed according to the priority order. Since the priority can be set from the image data of the preview image sequence, it can be said that the method α changes the content of the HDR processing for generating the output HDR image based on the image data of the preview image sequence.

Furthermore, the method α can be modified to the following method α _A. In the method α _A , only the output HDR image having a higher priority order is generated from the first and second output HDR images. Thus, when using method α _A :
Under circumstances where the above first or fourth rule applies,
Of the first and second output HDR images, only the second output HDR image is generated,
In situations where the second or third rule above applies,
Of the first and second output HDR images, only the first output HDR image is generated.
The image output control unit 58 of FIG. 4 can display the generated output HDR image on the display unit 27 and can record the output HDR image in the external memory 18.

In the method α _A , only one HDR process is selected from the first and second HDR processes, and one output HDR of the first and second output HDR images is selected by using the selected HDR process. Only an image is generated. Therefore, similarly to the method α, in the method α _A , it can be said that the content of the HDR processing for generating the output HDR image is changed according to the priority order, and the content of the HDR processing for generating the output HDR image Can be said to be changed based on the image data of the preview image sequence. On the other hand, the first and second output HDR images are always generated through an image synthesis process and a gradation conversion process. Therefore, in the method α _A , it can be said that the content of the gradation conversion processing for generating the output HDR image is changed according to the priority order, and the content of the gradation conversion processing for generating the output HDR image is the preview image sequence. It can be said that it is changed based on the image data.

According to the method α or α _A , it is possible to selectively generate an output HDR image that is presumed to meet the user's preference.

<< Deformation, etc. >>
The embodiment of the present invention can be appropriately modified in various ways within the scope of the technical idea shown in the claims. The above embodiment is merely an example of the embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the above embodiment. The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As annotations applicable to the above-described embodiment, notes 1 to 3 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
In the above-described operation example, the number of types of HDR processing that can be realized by the image processing unit 50 is three. However, the number of types is not limited as long as the number of types is two or more. From the image processing unit 50, the same number of output HDR images as the number of types of HDR processing can be obtained.

[Note 2]
The image processing unit 50, the image data acquisition unit 51, the image output control unit 58 shown in FIG. 4 and any part (display unit 27 etc.) shown in FIG. It may be provided, and each operation described above may be realized on the electronic device. The electronic device is, for example, a personal computer, a portable information terminal, or a mobile phone. The imaging device 1 is also a kind of electronic device.

[Note 3]
The imaging apparatus 1 and the electronic apparatus in FIG. 1 can be configured by hardware or a combination of hardware and software. When the imaging apparatus 1 and the electronic device are configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. In particular, all or part of the functions realized in each part of FIG. 4 are described as a program, and all or part of the functions are realized by executing the program on a program execution device (for example, a computer). You may make it do.

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 27 Display part 50 Image processing part 51 Image data acquisition part 52 Exposure control part 53 Composition processing part 54 Gradation conversion part 55 Gradation correction part 56 HDR imaging | photography necessity determination part 57 Priority order setting part 58 Image Output control unit

Claims (10)

An image data acquisition unit for acquiring image data of an input image sequence including n input images (n is an integer of 2 or more) photographed under different exposure conditions;
A synthesis processing unit that generates an image having a dynamic range wider than the dynamic range of each input image by synthesizing a plurality of input images included in the n input images;
A tone conversion unit that generates a plurality of synthesis result images by performing tone conversion on the generated image of the synthesis processing unit by individually using a plurality of different tone conversion conditions. Electronic equipment. A priority setting unit that sets a priority for the plurality of combined result images based on image data of the input image sequence;
The electronic apparatus according to claim 1, further comprising: an image output control unit that controls display or recording of the plurality of composite result images according to the priority order. The electronic apparatus according to claim 1, further comprising a gradation correction unit configured to generate a gradation correction image by performing gradation correction on one input image included in the n input images. . A priority setting unit that sets priority for the plurality of combined result images and the gradation correction image based on image data of the input image sequence;
The electronic apparatus according to claim 3, further comprising: an image output control unit that controls display or recording of the plurality of combined result images and the gradation correction image according to the priority order. The priority order setting unit is based on the image data of the input image sequence,
Movement of an object on the input image sequence;
Presence or absence of flicker in the input image sequence, and
Saturation in the input image sequence,
The electronic apparatus according to claim 2, wherein one or more of the electronic devices are evaluated, and the priority order is set based on an evaluation result. The electronic device is an imaging device that acquires image data of the input image sequence by photographing,
An exposure condition setting unit for setting an exposure condition of each input image forming the input image sequence;
A necessity determination unit that determines whether or not it is necessary to generate the plurality of combined result images,
Whether to generate the plurality of combined result images is determined according to the determination result of necessity,
The input image captured prior to the n input images includes the first and second determination input images,
The exposure condition setting unit includes the first determination input so that the number of pixels having a luminance equal to or higher than a predetermined first reference luminance in the first determination input image is equal to or less than a predetermined first reference value. A first exposure condition at the time of image shooting is set, and the number of pixels having a luminance equal to or lower than a predetermined second reference luminance in the second determination input image is equal to or lower than a predetermined second reference value. , Setting a second exposure condition at the time of shooting the second determination input image,
The electronic device according to claim 1, wherein the necessity determination unit performs the necessity determination based on the first and second exposure conditions. An image data acquisition unit for acquiring image data of an input image sequence including n input images (n is an integer of 2 or more) photographed under different exposure conditions;
A synthesis processing unit that generates an image having a dynamic range wider than the dynamic range of each input image by synthesizing a plurality of input images included in the n input images;
A gradation conversion unit that generates a composition result image by performing gradation conversion on the generated image of the composition processing unit;
The electronic device according to claim 1, wherein the gradation conversion unit changes the content of the gradation conversion based on image data of the input image sequence. An image data acquisition step of acquiring image data of an input image sequence including n input images (n is an integer of 2 or more) photographed under different exposure conditions;
A synthesis processing step of generating an image having a dynamic range wider than the dynamic range of each input image by synthesizing a plurality of input images included in the n input images;
A gradation conversion step of generating a plurality of combined result images by performing gradation conversion on the generated image of the combining processing step using a plurality of different gradation conversion conditions individually. An image processing method. An image data acquisition step of acquiring image data of an input image sequence including n input images (n is an integer of 2 or more) photographed under different exposure conditions;
A synthesis processing step of generating an image having a dynamic range wider than the dynamic range of each input image by synthesizing a plurality of input images included in the n input images;
A gradation conversion step of generating a composition result image by performing gradation conversion on the generated image of the composition processing step;
In the gradation conversion step, the content of the gradation conversion is changed based on image data of the input image sequence.   A program for causing a computer to execute the image data acquisition step, the composition processing step, and the gradation conversion step according to claim 8 or 9.