JP2018148353A - Imaging apparatus and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus and an imaging method capable of generating HDR still image data by selecting image data suitable for generation of the HDR still image data.SOLUTION: An imaging apparatus 100 includes a temporary storage unit 104a that temporarily stores a plurality of pieces of image data obtained by performing imaging a plurality of times under different exposure conditions. A first image processing unit 106a generates HDR moving image data from a first set of pieces of image data obtained by performing imaging a plurality of times under different exposure conditions. The control unit 118 selects a second set of pieces of image data different from the first set of pieces of image data based on the deviation of an object in the plurality of pieces of image data. A recording processing unit 106c records the second set of pieces of image data in a recording unit 112.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus and an imaging method.

  Some recent imaging apparatuses have an HDR image recording function for acquiring an image having a wider dynamic range than the specification of the original imaging apparatus by combining a plurality of images having different exposure conditions. Among such imaging devices having an HDR image recording function, for example, the imaging device proposed in Patent Document 1 is configured to perform imaging for HDR movie recording when there is a still image shooting instruction during HDR movie recording. Increases the number of image data for generating HDR still images by adding imaging with different exposure settings.

JP2015-15622A

  The imaging apparatus of Patent Literature 1 creates HDR still image data by combining a plurality of continuous image data immediately after the timing at which a still image capturing instruction is given. That is, in the imaging device of Patent Document 1, image data for creating HDR still image data is determined in advance. However, since there are deviations in the imaging timing of the plurality of image data, these plurality of image data are not necessarily suitable for creating HDR still image data.

  The present invention has been made in view of the above circumstances, and provides an imaging apparatus and an imaging method capable of generating HDR still image data by selecting image data suitable for generating HDR still image data. With the goal.

  In order to achieve the above object, an imaging apparatus according to a first aspect of the present invention includes an imaging unit that repeatedly performs imaging a plurality of times under different exposure conditions to generate a plurality of image data, and a plurality of the image data And a first set of image data obtained by imaging a plurality of times under the different exposure conditions from among the plurality of image data temporarily stored in the temporary storage unit. A first image processing unit that performs a first combining process to generate moving image data, and the first set of the plurality of image data based on a shift of the object in the plurality of image data. A control unit for selecting a second set of image data different from the image data of the selected image data, and generating a still image data by performing a second combining process on the selected second set of image data The second set of image data so that ; And a recording unit for recording the recording unit.

  In order to achieve the above object, the imaging method according to the second aspect of the present invention includes generating a plurality of image data by repeatedly performing imaging a plurality of times under different exposure conditions by an imaging unit, A first set obtained by temporarily storing the image data in a temporary storage unit, and from a plurality of the image data temporarily stored in the temporary storage unit, by a plurality of times of imaging under the different exposure conditions; Generating the moving image data by performing a first combining process on the image data of the plurality of image data and the first set of the plurality of the image data based on a shift of the object in the plurality of the image data. Selecting a second set of image data different from the image data of the selected image data, and generating a still image data by performing a second combining process on the selected second set of image data. The second set of images as possible ; And a recording the over data to the recording unit.

  According to the present invention, it is possible to provide an imaging apparatus and an imaging method capable of generating HDR still image data by selecting image data suitable for generating HDR still image data.

It is a block diagram which shows the structure of an example of the imaging device which concerns on one Embodiment. 6 is a flowchart illustrating main processing of the imaging apparatus according to the embodiment. 6 is a flowchart illustrating main processing of the imaging apparatus according to the embodiment. It is a timing chart which shows the operation | movement after the start of HDR moving image imaging. 5 is a flowchart illustrating HDR still image creation processing. It is a figure for demonstrating the method of determination of the propriety of HDR composition. It is a figure which shows the structure of a HDR moving image file. It is a flowchart shown about another process. It is a flowchart shown about another process.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an example of an imaging apparatus according to an embodiment of the present invention. An imaging apparatus 100 illustrated in FIG. 1 is various devices having an imaging function such as a digital camera, a smartphone, and a mobile phone with a camera function. 1 includes an imaging unit 102, a storage unit 104, an image processing unit 106, a playback unit 108, a display unit 110, a recording unit 112, a communication unit 114, an operation unit 116, And a control unit 118. Here, each block of the imaging apparatus 100 is configured by, for example, a combination of hardware and software. Each block of the imaging apparatus 100 may not be configured by a single hardware or software, but may be configured by a plurality of hardware or a plurality of software.

  The imaging unit 102 includes an imaging optical system 102a, an imaging element 102b, and a driving unit 102c. The imaging optical system 102a has a diaphragm, a lens, and the like, and makes a light beam from a subject (not shown) enter the imaging element 102b. The imaging optical system 102a includes a focus lens for adjusting the in-focus state. The imaging element 102b includes, for example, a CMOS image sensor or a CCD image sensor, images an object, and acquires image data (RAW data) related to the object. The image sensor 102b may include a phase difference detection pixel so that the distance to the object to be photographed can be detected. Further, the image sensor 102b in the present embodiment may be configured to be movable in a plane orthogonal to the optical axis with respect to the imaging optical system 102a. The drive unit 102c drives the focus lens of the imaging optical system 102a in the optical axis direction or drives the imaging element 102b according to control by the control unit 118.

  The storage unit 104 is, for example, a DRAM, and temporarily stores image data acquired by the imaging unit 102. Here, the storage unit 104 in the present embodiment is provided with a storage area as the temporary storage unit 104a. The temporary storage unit 104a is a storage area for copying and storing image data sequentially acquired by the imaging unit 102 along with HDR moving image imaging. Note that the temporary storage unit 104 a may be provided separately from the storage unit 104.

  The image processing unit 106 performs image processing on the image data acquired by the imaging unit 102 and stored in the storage unit 104. The image processing unit 106 includes a first image processing unit 106a, a second image processing unit 106b, and a recording processing unit 106c.

  The first image processing unit 106 a performs image processing related to generation of moving image data on the image data stored in the storage unit 104. Here, the moving image data in the present embodiment includes normal moving image data and HDR moving image data. The normal moving image data is moving image data in which each of a plurality of image data acquired by the imaging unit 102 is a moving image frame. The HDR video data is obtained by synthesizing each piece of composite image data configured by combining image data of different sets (first set) of exposure conditions obtained by multiple imaging under different exposure conditions by the imaging unit 102. This is video data with a wide dynamic range for video frames.

  The second image processing unit 106 b performs image processing related to generation of still image data on the image data stored in the storage unit 104. Here, the still image data in the present embodiment includes normal still image data and HDR still image data. The normal still image data is still image data generated from one image data acquired by the imaging unit 102. The HDR still image data is a still image with a wide dynamic range configured by combining image data of a set (second set) with different exposure conditions obtained by multiple imaging under different exposure conditions by the imaging unit 102. Image data. In moving images, if there is an accurate movement in an image, the gradation of details is often not concerned, and what is seen only needs to be reproduced smoothly and smoothly as it is. On the other hand, still images are required to be works that can be enjoyed everywhere. For this reason, unlike a moving image, it is desirable that the still image is an HDR image that is suitable for a still image. Assuming such a situation, the HDR still image data may be subjected to image processing so as to be image data having a different expression from the HDR moving image data. In addition, since the still image has a printing purpose, the characteristics of the HDR still image data to be synthesized may be changed according to the characteristics of the printing press.

  The recording processing unit 106c performs processing for generating a recording image file from the image data generated by the first image processing unit 106a or the second image processing unit 106b. For example, the recording processing unit 106c converts the moving image data to H.264. The video data is compressed by a predetermined video compression method such as the H.264 method, and predetermined tag information is added to the compressed video data to generate a video file. For example, the recording processing unit 106c compresses still image data by a predetermined still image compression method such as the JPEG method, and adds predetermined tag information to the compressed still image data to generate a still image file. . Here, when the still image file is generated during moving image shooting, the recording processing unit 106c performs processing for associating the still image file with the moving image file. For example, the recording processing unit 106c records information for associating a still image file and a moving image file (for example, the file name of the associated file) as tag information. Further, when generating the HDR still image data generated during HDR moving image shooting, the recording processing unit 106c records the image data used for the synthesis of the HDR still image data in the RAW format. By recording such RAW data, HDR still image data having an image representation different from that of the already recorded HDR still image data can be generated in later editing.

  The reproduction unit 108 performs processing for reproducing the image data generated by the image processing unit 106. The playback unit 108 includes a first playback processing unit 108a and a second playback processing unit 108b.

  The first reproduction processing unit 108a performs processing for reproducing moving image data on the display unit 110. For example, the first reproduction processing unit 108a inputs the moving image data generated by the first image processing unit 106a to the display unit 110 and causes the display unit 110 to display an image. When the image data is compressed, the first reproduction processing unit 108a also decompresses the compressed image data.

  The second reproduction processing unit 108b performs a process for reproducing the still image data on the display unit 110. For example, the second reproduction processing unit 108b inputs the still image data generated by the second image processing unit 106b to the display unit 110 and causes the display unit 110 to display an image. When the image data is compressed, the second reproduction processing unit 108b also decompresses the compressed image data.

  The display unit 110 is, for example, a liquid crystal display or an organic EL display, and displays various images such as an image based on the image data input from the reproduction unit 108.

  The recording unit 112 is configured by a flash ROM, for example. In the recording unit 112, the image file generated by the recording processing unit 106c of the image processing unit 106 is recorded. The recording unit 112 may store various programs used for controlling the imaging apparatus 100.

  The communication unit 114 performs processing for mediating communication between the imaging apparatus 100 and various devices. For example, the communication unit 114 performs processing for communicating an image with a television, communicating an image with a printer, and communicating an image with a cloud server.

  The operation unit 116 is an operation member for the user to operate the imaging apparatus 100. The operation unit 116 includes, for example, a release button, a moving image button, a setting button, a selection key, a power button, a touch panel, and the like. The release button is an operation member for instructing still image shooting. The moving image button is an operation member for instructing the start or end of moving image shooting. The setting button is an operation member for displaying a setting screen of the imaging apparatus 100. The selection key is an operation member for selecting and determining an item on the setting screen, for example. The power button is an operation member for turning on or off the power of the imaging apparatus 100. The touch panel is provided integrally with the display screen of the display unit 110 and detects a user's touch operation on the display screen. You may be comprised so that operation equivalent to the release button mentioned above, the moving image button, the setting button, the selection key, and the power button can be performed with a touch panel. Furthermore, the operation unit 116 may have other operation members other than those described above.

  The control unit 118 is a control circuit such as a CPU and an ASIC, and comprehensively controls the operation of the imaging apparatus 100. Functions equivalent to the control unit 118 may be realized by software, or may be realized by a combination of hardware and software. In addition, some functions of the control unit 118 may be provided separately from the control unit 118.

  Next, the operation of the imaging apparatus according to the present embodiment will be described. 2A and 2B are flowcharts illustrating main processing of the imaging apparatus according to the present embodiment. The processing in FIGS. 2A and 2B is performed mainly by the control unit 118.

  The process of FIGS. 2A and 2B is started when the power is turned on. In step S1, the control unit 118 determines whether or not the current operation mode is the shooting mode. In accordance with the operation mode set by the operation of the operation unit 116 by the user, the control unit 118 determines whether or not the current operation mode is the shooting mode. In step S1, when it is determined that the operation mode of the imaging apparatus 100 is the shooting mode, the process proceeds to step S2. In step S1, when it is determined that the operation mode of the imaging apparatus 100 is not the shooting mode, the process proceeds to step S23.

  In step S <b> 2, the control unit 118 acquires image data by performing imaging by the imaging unit 102 for through image display. In step S3, the control unit 118 controls the reproduction unit 108 based on the image data acquired in step S2 to perform a through image display.

  In step S <b> 4, the control unit 118 determines whether or not a setting instruction is given by the operation of the operation unit 116 by the user. If it is determined in step S4 that a setting instruction has been given, for example, by operating a setting button, the process proceeds to step S5. If it is determined in step S4 that no setting instruction has been given, the process proceeds to step S6.

  In step S <b> 5, the control unit 118 controls the playback unit 108 to display the setting screen on the display unit 110. After the setting screen is displayed, the control unit 118 changes various settings in accordance with a user operation on the setting screen. After the setting on the setting screen is completed, the process proceeds to step S6.

  In step S6, the control unit 118 determines whether or not an instruction to start moving image shooting has been given. For example, it is determined that the start of moving image shooting is instructed when a moving image button is operated by the user. When it is determined in step S6 that the start of moving image shooting has been instructed, the process proceeds to step S7. When it is determined in step S6 that the start of moving image shooting has not been instructed, the process proceeds to step S20.

  In step S7, the control unit 118 determines whether or not the HDR recording mode is on. The HDR recording mode is turned on / off by, for example, a user operation on the setting screen in step S5. If it is determined in step S7 that the HDR recording mode is on, the process proceeds to step S8. If it is determined in step S7 that the HDR recording mode is not turned on, the process proceeds to step S15. Here, the HDR recording mode is simply turned on / off. In this case, there is an advantage that the moving image HDR and the still image HDR can be set collectively. On the other hand, the HDR mode may be set separately for still images and moving images.

  In step S8, the control unit 118 performs HDR moving image imaging. That is, the control unit 118 performs imaging by the imaging unit 102 while appropriately changing the exposure condition (for example, exposure time) so that an HDR moving image frame can be generated. Thereafter, the process proceeds to step S9.

  FIG. 3 is a timing chart showing the operation after the start of HDR moving image imaging. Here, “imaging rate” in FIG. 3 indicates the timing of imaging and the exposure setting in each imaging. In addition, “recording rate” in FIG. 3 indicates the timing of moving image frame generation and the generated moving image frame. Further, “RAW” in FIG. 3 indicates RAW format image data stored in the temporary storage unit 104a in step S9 described later. Also, the “still image synthesis pattern” in FIG. 3 is a diagram showing an example of a HDR still image synthesis pattern created in an HDR still image creation process described later. Further, “shooting” in FIG. 3 is a timing when an instruction to record a still image, which will be described later, is given.

  In HDR video imaging, imaging with different exposure is repeatedly performed. For example, in FIG. 3, the first set of imaging including the overexposed imaging (overexposed 1 in FIG. 3) and the underexposed imaging (underexposed 1 in FIG. 3) is repeatedly performed. Here, the over-exposure imaging is an imaging performed with a higher exposure setting than the standard imaging (for example, +1 step exposure setting with respect to the standard exposure setting). Under-exposure imaging is imaging performed with an exposure setting lower than that of standard imaging (for example, -1 step exposure setting with respect to the standard exposure setting). Standard imaging is imaging performed with exposure settings during normal video imaging that is not HDR video imaging, and the exposure of main objects in the scene (for example, a central object or a person) is set to an appropriate value. It is imaging performed by exposure setting. The exposure is changed, for example, by changing the exposure time. That is, during overexposed imaging, the exposure time is set longer than the exposure time in standard imaging. In the case of underexposed imaging, the exposure time is set shorter than the exposure time in standard imaging. Here, main objects in the scene may be configured to be preset by the user. Further, the main object in the scene may be determined by learning the shooting tendency of the user's object.

  In step S <b> 9, the control unit 118 causes the storage unit 104 to store image data obtained by imaging by the imaging unit 102. In addition, the control unit 118 copies the image data obtained by imaging by the imaging unit 102 and stores the image data in the temporary storage unit 104 a of the storage unit 104.

  In step S <b> 10, the control unit 118 generates an HDR moving image frame by the first image processing unit 106 a of the image processing unit 106. As shown in FIG. 3, the generation of the HDR moving image frame is performed every time imaging of image data for two frames is completed. The HDR moving image frame is generated from the image data of the two most recent frames. For example, after completion of overexposed 1 imaging and underexposed 1 imaging, the HDR moving image frame 1 is generated using these two pieces of image data. The HDR moving image frame is performed, for example, by combining underexposed image data with an overexposed portion of overexposed image data. Note that the method of generating the HDR moving image frame is not limited to the method shown here. Here, in the present embodiment, the generation of the HDR moving image frame is performed using the copy source image data stored in the storage unit 104. That is, the copied image data is left in the temporary storage unit 104a. The copied image data is used for HDR still image recording to be described later.

  In step S <b> 11, the control unit 118 determines whether or not an instruction to end HDR video shooting has been given. For example, when the user operates the moving image button again, it is determined that the end of moving image shooting has been instructed. When it is determined in step S11 that the end of moving image shooting has not been instructed, the process proceeds to step S12. When it is determined in step S11 that the end of moving image shooting has been instructed, the process proceeds to step S14.

  In step S12, the control unit 118 determines whether to record an HDR still image. For example, when the user instructs to record a still image, it is determined that the HDR still image is recorded. The still image recording instruction is, for example, a pressing operation of a release button. If it is determined in step S12 that HDR still image recording is to be performed, the process proceeds to step S13. If it is determined in step S12 that HDR still image recording is not performed, the process returns to step S8. In this case, HDR moving image capturing is continued.

  In step S13, the control unit 118 performs processing for creating an HDR still image file. After the HDR still image file creation process, the process returns to step S8. Hereinafter, a process of creating an HDR still image file will be described. FIG. 4 is a flowchart showing HDR still image creation processing.

  In step S101, the control unit 118 selects, from the image data (RAW data) stored in the temporary storage unit 104a, two frames of image data close to the timing of the still image recording instruction. In the example of FIG. 3, a still image recording instruction is given during overexposed 2 imaging. In this case, the control unit 118 first includes overexposed 2 image data obtained by imaging at the timing when the still image recording instruction is given, and underexposed 1 image data obtained by immediately preceding imaging. 2 sets are selected (combination of synthesis pattern A in FIG. 3).

  In step S102, the control unit 118 compares the selected two pieces of image data and determines whether the two pieces of image data are image data that can be combined. If it is determined in step S102 that the two pieces of image data can be combined, the process proceeds to step S103. If it is determined in step S102 that the two pieces of image data are not image data that can be combined, the process proceeds to step S108.

  Here, the determination in step S102 is performed based on the deviation of the object in the selected two pieces of image data. The shift here includes a shift in the position of the object and a shift in the shape of the object. The deviation of the shape of the object includes a deviation of the contour between main objects (center object, person, etc.) of the image data, and a deviation of facial expression if it is a person. Note that the displacement of the position of the object is determined by, for example, a motion vector of the object between the two pieces of image data. Further, the deviation of the shape of the object is performed by extracting and comparing the contours of the image data, specifying facial expressions using a face detection technique, and comparing the facial expressions. Here, since the exposure of the two image data to be compared is different, it is desirable to take this exposure difference into consideration in the determination in step S102. For example, it is desirable to perform the determination in step S102 after performing exposure correction processing for applying gain to one of the image data so that both exposure differences are eliminated.

  In step S102, it is determined that the composition is possible if both the displacement of the position of the object and the displacement of the shape of the object are smaller than a predetermined value. For example, in the comparison of the overexposed image data B and the underexposed image data D1 shown in FIG. 5, the position of the person as the object is shifted between the overexposed image data B and the underexposed image data D1. There is no. However, in the underexposed image data D1, the facial expression of the person who is the object is crushed in black. In this case, since there is a large difference in facial expression between the overexposed image data B and the underexposed image data D1, it is determined that the overexposed image data B and the underexposed image data D1 cannot be combined. . On the other hand, in the comparison between the overexposed image data B and the underexposed image data D2 shown in FIG. 5, in the underexposed image data D1, the expression of the person who is the object is underexposure but not blackened. . In this case, it is determined that the over-exposure image data B and the under-exposure image data D2 can be combined with each other because the difference in facial expression between the over-exposure image data B and the under-exposure image data D1 is small. The In this case, HDR still image data HDR is generated by subsequent processing.

  Returning to the description of FIG. In step S103, the control unit 118 determines whether or not the luminance difference between the two pieces of image data determined to be compositeable is greater than a predetermined value (for example, 2.5 levels). The predetermined value for this determination can be set as appropriate. If it is determined in step S103 that the luminance difference between the two pieces of image data determined to be compositable is greater than a predetermined value, the process proceeds to step S104. If it is determined in step S103 that the luminance difference between the two pieces of image data determined to be compositeable is within a predetermined value, the process proceeds to step S107.

  In step S104, the control unit 118 stands by until the next HDR moving image capturing timing, and when the next HDR moving image capturing timing comes, the exposure setting is different from the most recent two frames, and the image capturing unit 102 performs image capturing. Do. In the example of FIG. 3, at the timing of imaging after overexposure 2, the control unit 118 performs imaging with a standard exposure setting different from both underexposure and overexposure, which are exposure settings in the latest two frames. (Standard 2 in FIG. 3). Further, at the timing of subsequent imaging, the control unit 118 performs imaging with an underexposure setting different from both overexposure and standard, which are exposure settings in the imaging of the two most recent frames (underexposure 2 in FIG. 3). After imaging, the control unit 118 causes the storage unit 104 to store image data obtained by imaging by the imaging unit 102. In addition, the control unit 118 copies the image data obtained by imaging by the imaging unit 102 and stores the image data in the temporary storage unit 104 a of the storage unit 104. Further, the control unit 118 generates an HDR moving image frame by the first image processing unit 106 a of the image processing unit 106. As described above, the HDR moving image frame is generated from the image data of the two most recent frames. However, in the example of FIG. 3, the image data obtained by standard imaging is directly used as an HDR moving image frame. This is to reduce the difference in image quality between HDR video frames generated by overexposed image data and underexposed image data. However, an HDR video frame may be generated from image data obtained by overexposed 2 imaging and image data obtained by standard 2 imaging. On the other hand, the image data obtained by the under-exposure 2 imaging in FIG. 3 and the image data obtained by the subsequent over-exposure 3 imaging may be used as they are for the generation of the HDR video frame. After the generation of the HDR video frame, the process proceeds to step S105.

  In step S105, the control unit 118 controls the latest three images with different exposures (two images determined to be compositable in step S102 and one image obtained in step S104. In the example of FIG. ) Is determined as to whether or not the image data can be synthesized. The determination in step S105 is performed in the same manner as the determination in step S102. If it is determined in step S105 that the three pieces of image data are image data that can be combined, the process proceeds to step S106. If it is determined in step S105 that the three pieces of image data are not image data that can be combined, the process proceeds to step S107.

  In step S <b> 106, the control unit 118 uses the second image processing unit 106 b of the image processing unit 106 using the three pieces of image data (overexposed, standard, underexposed) determined to be compositable in step S <b> 105. HDR still image data is generated. Then, the control unit 118 causes the storage unit 104 to store the generated HDR still image data. Thereafter, the process of FIG. 4 ends.

  In step S107, the control unit 118 generates HDR still image data by the second image processing unit 106b of the image processing unit 106 using the two pieces of image data determined to be compositeable in step S102. Then, the control unit 118 causes the storage unit 104 to store the generated HDR still image data. Thereafter, the process of FIG. 4 ends.

  In step S108, the control unit 118 determines whether or not a predetermined time has elapsed from the instruction to record the still image. This predetermined time is, for example, an imaging period for three frames. If it is determined in step S108 that a predetermined time has elapsed since the instruction to record the still image, the process proceeds to step S109. If it is determined in step S108 that the predetermined time has not elapsed since the still image recording instruction, the process proceeds to step S110.

  In step S109, since the predetermined time has elapsed since the instruction to record the still image, the control unit 118 collects portions that can be used for composition from the image data selected in the processing of step S101 so far, and The HDR still image data is generated by the second image processing unit 106 b 106. Then, the control unit 118 causes the storage unit 104 to store the generated HDR still image data. Thereafter, the process of FIG. 4 ends. For example, in the overexposed image data B and the underexposed image data D1 shown in FIG. 5, the facial expression of the person who is the object cannot be used for composition, but the other portions can be used for composition. In such a case, other than the face of a person is used to create HDR still image data. Similarly, parts that can be combined from other image data are collected and finally one piece of image data is generated.

  In step S110, the control unit 118 stands by until the next HDR moving image capturing timing, and when the next HDR moving image capturing timing comes, the exposure setting is different from the latest two frames and the image capturing unit 102 performs the image capturing. Do. Thereafter, the process returns to step S101.

  Note that in the second and subsequent processing of step S101, the control unit 118 selects two pieces of image data obtained by imaging at a timing close to the timing at which the still image recording instruction is given. For example, in the second process of step S101, the control unit 118 selects image data obtained by overexposed 2 imaging and image data obtained by standard 2 imaging (composite pattern B in FIG. 3). Combination). Furthermore, although not shown in FIG. 3, in the process of step S <b> 101 for the third time, the control unit 118 obtains image data obtained by overexposed 2 imaging and image data obtained by underexposed 2 imaging. select. In this embodiment, the description has been made so that another piece of image data is selected on the basis of overexposed 2 image data obtained by imaging at the timing when a still image recording instruction is given. Changes such as selecting another piece of image data based on underexposed 1 image data obtained by imaging immediately before the timing at which the image recording instruction is given may be made.

  Here, it returns to description of FIG. 2A and FIG. 2B. In step S14 when it is determined in step S11 that HDR movie shooting is to be terminated, the control unit 118 generates an HDR movie file by the recording processing unit 106c of the image processing unit 106. Thereafter, the process returns to step S1.

  FIG. 6 is a diagram showing the structure of an HDR video file. As illustrated in FIG. 6, the HDR moving image file 200 includes HDR moving image data 201, HDR still image data 202, RAW data 203, and tag information 204. Although not shown in FIG. 6, audio data acquired via a microphone mounted on the imaging apparatus 100 may also be recorded in the HDR moving image file 200.

  The HDR moving image data 201 is data including HDR moving image frames F1, F2,.

  The HDR still image data 202 is data composed of still image compressed HDR still image data A1. Note that a plurality of HDR still image data 202 may be recorded in the HDR moving image file 200.

  The RAW data 203 is data in which image data used to generate the HDR still image data 202 is recorded in the RAW format. By recording the RAW data 203, HDR still image data can be generated in a later editing operation.

  The tag information 204 includes file information such as the file name of the HDR moving image file 200. The tag information 204 includes related information including information for specifying HDR still image data associated with each HDR video frame (for example, an address where HDR still image data or RAW data is recorded). ing.

  In FIG. 6, the HDR still image data is recorded in the HDR moving image file. On the other hand, the HDR still image data may be recorded as a file different from the HDR moving image file. Similarly, the RAW data may be recorded as a file different from the HDR moving image file.

  Here, it returns to description of FIG. 2A and FIG. 2B. In step S15 when it is determined in step S7 that the HDR recording mode is not turned on, the control unit 118 performs normal moving image imaging. That is, the control unit 118 performs imaging by the imaging unit 102 with the exposure condition set as a standard exposure setting. Then, the control unit 118 instructs the image processing unit 106 to generate a moving image frame for each imaging. In response to this instruction, the first image processing unit 106a of the image processing unit 106 generates a normal moving image frame from the image data stored in the storage unit 104 along with normal moving image shooting. Thereafter, the process proceeds to step S16.

  In step S16, the control unit 118 determines whether or not an instruction to end moving image shooting has been given. For example, when the user operates the moving image button again, it is determined that the end of moving image shooting has been instructed. When it is determined in step S16 that the end of moving image shooting has not been instructed, the process proceeds to step S17. If it is determined in step S16 that the end of moving image shooting has been instructed, the process proceeds to step S19.

  In step S17, the control unit 118 determines whether or not to perform normal still image recording. Whether or not to perform normal still image recording may be determined in the same manner as in HDR still image recording. If it is determined in step S17 that normal still image recording is to be performed, the process proceeds to step S18. If it is determined in step S17 that normal still image recording is not performed, the process returns to step S15.

  In step S <b> 18, the control unit 118 generates normal still image data by the second image processing unit 106 b of the image processing unit 106 using image data stored in the storage unit 104 along with normal moving image shooting. Thereafter, the control unit 118 stores the normal still image data in the storage unit 104. Thereafter, the process returns to step S15.

  In step S19, the control unit 118 generates a normal moving image file by the recording processing unit 106c of the image processing unit 106. Thereafter, the process returns to step S1. The normal moving image file is different from the HDR moving image file shown in FIG. 6 in that the moving image data to be recorded is normal moving image data.

  In step S20 when it is determined in step S6 that the start of moving image shooting has not been instructed, the control unit 118 determines whether or not the start of still image shooting has been instructed. For example, when the release button is operated by the user, it is determined that the start of still image shooting is instructed. When it is determined in step S20 that the start of still image shooting has been instructed, the process proceeds to step S21. If it is determined in step S20 that the start of still image shooting has not been instructed, the process returns to step S1.

  In step S21, the control unit 118 performs normal still image capturing. That is, the control unit 118 performs one imaging by the imaging unit 102 in accordance with the exposure condition set in step S5. Thereafter, the process proceeds to step S22.

  In step S22, the control unit 118 generates normal still image data by the second image processing unit 106b of the image processing unit 106. After the normal still image data is generated, the control unit 118 generates a normal still image file by the recording processing unit 106 c of the image processing unit 106 and records it in the generated recording unit 112. Thereafter, the process returns to step S1.

  In step S23 when it is determined in step S1 that the operation mode is not the shooting mode, the control unit 118 performs other processes other than the shooting mode. After the other processes, the process returns to step S1.

  7A and 7B are flowcharts showing other processes. In step S201, the control unit 118 determines whether or not the current operation mode is the reproduction mode. In step S201, when it is determined that the operation mode of the imaging apparatus 100 is not the reproduction mode, the process proceeds to step S216. In step S201, when it is determined that the operation mode of the imaging apparatus 100 is the reproduction mode, the process proceeds to step S202.

  In step S202, the control unit 118 controls the reproduction unit 108 to display a list of image files recorded in the recording unit 112. Thereby, for example, a thumbnail image indicating a list of image files recorded in the recording unit 112 is displayed on the display unit 110. Thereafter, the process proceeds to step S203.

  In step S203, the control unit 118 determines whether or not a moving image file has been selected by the user from the list display. If it is determined in step S203 that the moving image file has been selected by the user, the process proceeds to step S204. If it is determined in step S203 that the moving image file has not been selected by the user, the process proceeds to step S208.

  In step S <b> 204, the control unit 118 instructs the playback unit 108 to display a moving image frame corresponding to the selected moving image file on the display unit 110. In response to this, the first reproduction processing unit 108a of the reproduction unit 108 causes the display unit 110 to display the moving image frame.

  In step S205, the control unit 118 determines whether it is time to reproduce a still image. For example, when there is still image data associated with the moving image frame being reproduced, it is determined that it is time to reproduce the still image. If it is determined in step S205 that it is time to reproduce a still image, the process proceeds to step S206. If it is determined in step S205 that it is not time to reproduce the still image, the process proceeds to step S207.

  In step S206, the control unit 118 instructs the playback unit 108 to display on the display unit 110 a still image (including an HDR still image) associated with the moving image frame being played back. In response to this, the second reproduction processing unit 108b of the reproduction unit 108 causes the display unit 110 to display a still image. As a result, the still image is reproduced on the display unit 110.

  In step S207, the control unit 118 determines whether or not to end the moving image reproduction. In step S207, it is determined that the moving image reproduction is to be ended when the reproduction of the final moving image frame has been completed or when the end of the moving image reproduction is instructed by the operation of the operation unit 116 by the user. In step S207, when it is determined not to end the moving image reproduction, the process returns to step S204. In this case, the moving image reproduction is continued. If it is determined in step S207 that the moving image reproduction is to be terminated, the process proceeds to step S215.

  In step S208 when it is determined in step S203 that the moving image file has not been selected by the user, the control unit 118 determines whether or not a slide show reproduction instruction has been issued by the operation of the operation unit 116 of the user. If it is determined in step S208 that a slide show playback instruction has been issued, the process proceeds to step S209. If it is determined in step S208 that a slide show reproduction instruction has not been issued, the process proceeds to step S212.

  In step S209, the control unit 118 instructs the playback unit 108 to sequentially start playback of the still image files recorded in the recording unit 112. In response to this, the second reproduction processing unit 108b of the reproduction unit 108 causes the display unit 110 to display a still image. As a result, the still image is reproduced on the display unit 110.

  In step S210, the control unit 118 determines whether or not to end still image reproduction. In step S <b> 210, it is determined that the still image reproduction is to be terminated when an instruction to end the slide show reproduction is given by the user operating the operation unit 116. If it is determined in step S210 that the still image reproduction is not finished, the process proceeds to step S211. If it is determined in step S210 that the still image reproduction is to end, the process proceeds to step S215. In step S210, the control unit 118 determines whether or not to end still image reproduction. In step S211, the control unit 118 determines whether there is a next still image. If it is determined in step S211 that there is a next still image, the process returns to step S209. In this case, the next still image file is played back. If it is determined in step S211 that there is no next still image, the process proceeds to step S215.

  In step S212 when it is determined in step S208 that the user has not instructed playback of the slide show, the control unit 118 determines whether or not a still image file has been selected by the user from the list display. If it is determined in step S212 that a still image file has been selected by the user, the process proceeds to step S213. If it is determined in step S212 that the still image file has not been selected by the user, the process proceeds to step S215.

  In step S213, the control unit 118 instructs the playback unit 108 to start playback of the still image file recorded in the recording unit 112. In response to this, the second reproduction processing unit 108b of the reproduction unit 108 causes the display unit 110 to display a still image. As a result, the still image is reproduced on the display unit 110.

  In step S214, the control unit 118 determines whether or not the end of still image reproduction has been instructed. If it is determined in step S214 that the end of still image reproduction has not been instructed, the process returns to step S213. When it is determined in step S214 that the end of still image reproduction has been instructed, the process proceeds to step S215.

  In step S215, the control unit 118 determines whether or not to end the playback mode process. If it is determined in step S215 that the playback mode process is not to end, the process returns to step S202. When it is determined in step S215 that the playback mode process is to be ended, the process ends in FIGS. 7A and 7B. In this case, the process returns to step S1 in FIG. 2A.

  In step S216, the control unit 118 determines whether or not the current operation mode is the communication mode. In step S216, when it is determined that the operation mode of the imaging apparatus 100 is the communication mode, the process proceeds to step S217. In step S216, when it is determined that the operation mode of the imaging apparatus 100 is not the communication mode, the processing in FIGS. 7A and 7B ends. In this case, the process returns to step S1 in FIG. 2A.

  In step S217, the control unit 118 performs communication mode processing. In the processing in the communication mode, the control unit 118 controls the communication unit 114 to transmit, for example, an image file recorded in the recording unit 112 to an external device or receive an image file recorded in the external device. Process. After the communication mode processing, the processing in FIGS. 7A and 7B ends. In this case, the process returns to step S1 in FIG. 2A.

  As described above, in the present embodiment, when recording an HDR still image from image data obtained during HDR moving image shooting, HDR is considered in consideration of the timing at which a still image recording is instructed and a shift between image data. Image data suitable for composition is selected. Thereby, the image quality of the HDR still image can be improved.

  In this embodiment, it is first determined whether HDR still image composition can be performed from two pieces of image data. When the HDR still image can be combined and the luminance difference between the image data is large, additional imaging is performed with an exposure different from that of the two selected image data. The image quality of the HDR still image can be improved by performing additional imaging with different exposures. However, the additional imaging has a slight effect on the image quality of the HDR video. In this embodiment, the image quality of the HDR video can be maintained by performing additional imaging only when necessary. In the present embodiment, additional imaging is performed when the luminance difference is large. However, the additional imaging may be configured not to be performed.

  In the present embodiment, HDR moving image data is generated from image data obtained by two imaging operations at different exposures. If the imaging rate allows, the number of times of imaging may be three or more. In this case, HDR still image data may also be generated from three or more pieces of image data from the beginning. In this case, when a user performs a pause operation during playback of a moving image captured at a high imaging rate, an HDR still image that combines the image data of the stopped frame and the frame data before and after the frame is displayed. It may be. Thus, by setting the image displayed during the pause as the HDR still image, it is possible to pause more effectively for the user than simply displaying one frame image.

  In the technique of the present embodiment, HDR moving image shooting can be replaced with HDR live view display. In this case, the technique of the present embodiment can be applied to an imaging apparatus such as an endoscope apparatus, a microscope apparatus, and a monitoring apparatus.

  Further, in the description of each operation flowchart described above, the operation is described using “first”, “next”, and the like for convenience, but this does not mean that it is essential to perform the operations in this order. Absent.

  Each process according to the above-described embodiment may be stored as a program that can be executed by the control unit 118 that is a computer. In addition, it can be stored and distributed in a recording medium of an external storage device such as a magnetic disk, an optical disk, or a semiconductor memory. The control unit 118 reads the program stored in the recording medium of the external storage device, and the operation described above can be executed by the operation being controlled by the read program.

  DESCRIPTION OF SYMBOLS 100 Imaging device, 102 Imaging part, 102a Imaging optical system, 102b Imaging element, 102c Drive part, 104 Storage part, 104a Temporary storage part, 106 Image processing part, 106a 1st image processing part, 106b 2nd image processing part 106c Recording processing unit, 108 playback unit, 108a first playback processing unit, 108b second playback processing unit, 110 display unit, 112 recording unit, 114 communication unit, 116 operation unit, 118 control unit.

Claims (8)

An imaging unit that repeatedly performs imaging at different exposure conditions to generate a plurality of image data; and
A temporary storage unit that temporarily stores a plurality of the image data;
A first combining process is performed on a first set of image data obtained by multiple imaging under the different exposure conditions from the plurality of image data temporarily stored in the temporary storage unit. A first image processing unit for generating moving image data;
A control unit that selects a second set of image data different from the first set of image data from a plurality of the image data based on a shift of an object in the plurality of image data;
The second set of image data is recorded in a recording unit so that still image data can be generated by performing a second combining process on the selected second set of image data. A recording processing unit;
An imaging apparatus comprising: An operation unit for instructing still image recording;
The imaging apparatus according to claim 1, wherein the temporary storage unit temporarily stores a plurality of the image data obtained by imaging up to a predetermined time before and after the timing when the still image recording instruction is given.   The imaging apparatus according to claim 1, wherein the control unit selects a plurality of pieces of image data with a small positional deviation of the object as the second set of image data.   The imaging device according to claim 1, wherein the control unit selects a plurality of pieces of image data having a small shape deviation of the object as the second set of image data.   The imaging apparatus according to claim 1, further comprising a second image processing unit configured to perform a second synthesis process on the selected second set of image data to generate still image data.   The second image processing unit extracts an image data portion having a small shift between the plurality of image data when the control unit does not select the second set of image data, and extracts the extracted image data. The imaging apparatus according to claim 5, wherein the still image data is generated by combining portions.   The imaging apparatus according to claim 1, wherein the recording processing unit records the second set of image data with lossless compressed data or non-compressed data. The imaging unit repeatedly generates a plurality of images under different exposure conditions to generate a plurality of image data,
Temporarily storing a plurality of the image data in a temporary storage unit;
A first combining process is performed on a first set of image data obtained by multiple imaging under the different exposure conditions from the plurality of image data temporarily stored in the temporary storage unit. Generating video data,
Selecting a second set of image data different from the first set of image data from a plurality of the image data based on deviations of objects in the plurality of image data;
The second set of image data is recorded in a recording unit so that still image data can be generated by performing a second combining process on the selected second set of image data. And
An imaging method comprising: