JP2015033020A - Imaging apparatus, imaging method, program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve image quality in the case where the combination/non-combination of captured images is switched during motion picture recording.SOLUTION: An imaging apparatus is provided which comprises: imaging means; control means for controlling the imaging means so as to continuously perform imaging at a first frame rate or a second frame rate that is higher than the first frame rate; and image processing means for processing image data continuously produced by the continuous imaging, the image processing means being configured to operate under a non-combination mode in which image data are processed without being combined with the other image data, or under a combination mode in which processing for combining a predetermined number of image data items is performed. When switching the image processing means from the non-combination mode to the combination mode during the continuous imaging, the control means switches the first frame rate to the second frame rate and when switching the image processing means from the combination mode to the non-combination mode during the continuous imaging, the control means switches the second frame rate to the first frame rate.

Description

  The present invention relates to an imaging apparatus, an imaging method, a program, and a recording medium.

  In recent years, an imaging apparatus such as a digital camera can generally capture and record a moving image with sound. In addition, an imaging apparatus that can record an image captured at a high frame rate at a low frame rate is known. When the recorded moving image is reproduced, the effect of slow reproduction can be obtained.

  In recent years, a technique for synthesizing and recording a plurality of continuously captured images has been known. Patent Document 1 discloses an image processing method for determining whether or not to perform synthesis in accordance with a change in a subject or a change in imaging conditions with respect to a subject when a plurality of images continuously captured in time series are combined. ing. When synthesizing, the imaging apparatus continuously captures images while alternately changing the exposure time between long seconds and short seconds, and synthesizes each image, thereby moving a wide dynamic range (hereinafter referred to as “HDR movie”). Can be generated. If the composition / non-composition is switched according to the state of the subject, it is also possible to dynamically switch between the HDR moving image and the normal (non-combined) moving image during recording. When switching from an HDR video to a normal video during video recording, the imaging device thins out and records the captured image in order to perform recording at the same recording frame rate as that of the HDR video.

JP 2007-036742 A

  As described above, when switching from an HDR video to a normal video during video recording according to the technique of Patent Document 1, a video that is thinned out in time series is generated, so that the image quality deteriorates (discontinuous images). There was a problem. Further, the exposure time is limited by the imaging frame rate higher than the recording frame rate, and there is a problem that the image quality is deteriorated. Such a problem occurs in the process of synthesizing a plurality of images, and may occur not only when synthesizing to generate an HDR video, but also when performing a cyclic noise reduction process.

  The present invention has been made in view of such a situation, and an object of the present invention is to provide a technique for improving the image quality when the synthesis / non-synthesis of a captured image is switched during moving image recording.

  In order to solve the above-described problem, the present invention controls the imaging unit and the imaging unit to continuously perform imaging at a first frame rate or a second frame rate higher than the first frame rate. A control means for processing and image processing means for processing image data continuously generated by the continuous imaging, wherein each image data is processed without being combined with other image data, or Image processing means that operates in a synthesis mode for performing a process of synthesizing a predetermined number of image data, and the control means is configured to change the image processing means from the non-synthesis mode to the synthesis mode during the continuous imaging. When switching to, the first frame rate is switched to the second frame rate, and the image processing means changes from the synthesis mode to the non-synthesis mode during the continuous imaging. Ri If alternative, to provide an imaging apparatus, characterized in that switching from the second frame rate to the first frame rate.

  Other features of the present invention will become more apparent from the accompanying drawings and the following description of the preferred embodiments.

  According to the present invention, it is possible to improve the image quality when the synthesized / non-synthesized captured image is switched during moving image recording.

1 is a block diagram illustrating a configuration of an imaging apparatus 100. FIG. 5 is a flowchart of a moving image shooting process according to the first embodiment. The flowchart of the switching process from a normal moving image to a HDR moving image. The flowchart of a HDR moving image process. The flowchart of the switching process from a HDR moving image to a normal moving image. The flowchart of normal moving image processing. The conceptual diagram which represented the mode of the switching process from a normal moving image to a HDR moving image in time series. The conceptual diagram which represented the mode of the switching process from a HDR moving image to a normal moving image in time series. The flowchart of the cyclic | annular NR process which concerns on 2nd Embodiment.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The technical scope of the present invention is determined by the claims, and is not limited by the following individual embodiments. In addition, not all combinations of features described in the embodiments are essential to the present invention.

[First Embodiment]
FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus 100 according to the first embodiment. The imaging apparatus 100 includes a zoom lens 110, a shift lens 111 that corrects camera shake of the imaging apparatus 100, a focus lens 112, a mechanical shutter 113 that blocks a light beam to the subsequent stage, and a diaphragm 114 that adjusts the light beam to the subsequent stage. . The imaging apparatus 100 also includes an imaging element 115 and a timing generator 116 that drives the imaging element 115 and generates timing pulses necessary for sampling.

  The imaging apparatus 100 also includes an image processing unit 117. The image processing unit 117 is an SSG circuit that generates a synchronization signal for imaging driving, a preprocessing circuit that receives digital output from imaging and performs preprocessing, an AF evaluation value calculation circuit, a luminance integration circuit, a signal processing circuit, and image synthesis A circuit, a reduction circuit, a raster-block conversion circuit, and a compression circuit.

  The SSG circuit receives an imaging drive clock from the timing generator 116, generates horizontal and vertical synchronization signals, and outputs them to the timing generator 116 and the image sensor 115. The preprocessing circuit distributes the input image to the luminance integration circuit and the signal processing circuit in units of rows. Further, the preprocessing circuit performs processing such as data correction between channels necessary for imaging data. The AF evaluation value calculation circuit performs horizontal filtering on the luminance component of the image signal input in the set plurality of evaluation areas, selects the maximum value while extracting a predetermined frequency representing the contrast, and selects the vertical direction Perform the integral operation. The luminance integration circuit mixes and generates luminance components from the RGB signals, divides the input image into a plurality of regions, and generates a luminance component for each region. The signal processing circuit performs color carrier removal, aperture correction, gamma correction processing, and the like on the output data of the image sensor 115 to generate a luminance signal. At the same time, the signal processing circuit performs color interpolation, matrix conversion, gamma processing, gain adjustment, and the like to generate color difference signals, and stores YUV format image data in the memory unit 123. Further, the signal processing circuit aggregates the luminance signals (Y) of the YUV format image data to be generated for each level, and generates luminance distribution data for each image data. The image composition circuit acquires a plurality of RGB-format captured images stored in the memory unit 123 or a plurality of YUV-format image data obtained by performing signal processing on the captured images, and performs pixel unit processing on the image data. A composite image is generated by multiplying and adding the coefficients set in (1). The reduction circuit receives the output of the signal processing circuit, performs cutout, thinning, linear interpolation processing, and the like of input pixel data, and performs pixel data reduction processing in the horizontal and vertical directions. The raster-block conversion circuit converts raster scan image data scaled by the reduction circuit into block scan image data. Such a series of image processing is realized by using the memory unit 123 as a buffer memory. The compression circuit outputs the moving image bit stream by compressing the YUV image data converted into the block scan image data according to the moving image compression method.

  The exposure control unit 118 controls the mechanical shutter 113 and the diaphragm 114. The lens driving unit 119 moves the zoom lens 110 and the focus lens 112 along the optical axis, and forms an object scene image on the image sensor 115. In addition, the lens driving unit 119 drives the shift lens 111 in accordance with outputs from an angular velocity sensor and an acceleration sensor (not shown), and optically corrects camera shake of the imaging apparatus 100. The release switch 120 is for a user to give a shooting instruction. The movie button 180 is for the user to instruct movie recording.

  The system control unit 121 includes a CPU and its interface circuit, a DMAC (Direct Memory Access Controller), a bus arbiter, and the like. A program executed by the CPU is stored in the flash memory 122. The memory unit 123 is a DRAM or the like, and temporarily stores data in the middle of each process. In addition, the program in the flash memory 122 is developed and used in the memory unit 123. Various controls of the imaging apparatus 100 are performed by the system control unit 121.

  The I / F unit 124 is an interface to the recording medium 200. The connector 125 is a connector for connecting the recording medium 200. The recording medium attachment / detachment detection switch 126 detects attachment / detachment of the recording medium 200. The microphone 127 converts the input sound into an audio signal. The A / D converter 128 converts the sound output of the microphone 127 into a digital sound signal. The audio processing unit 129 performs predetermined audio processing on the digital audio data and outputs an audio bitstream.

  The power supply control unit 130 controls a battery detection circuit and a DC-DC converter (not shown), and supplies a necessary voltage to each unit including the recording medium 200 for a necessary period. The power connector 131 is a connector for the power source 132, and the power source 132 is a power source including a primary battery, a secondary battery, an AC adapter, or the like.

  The reproduction circuit 150 converts the image data generated by the image processing unit 117 and stored in the memory unit 123 into a display image and transfers it to the monitor 151. The monitor 151 displays the image output from the reproduction circuit 150. The reproduction circuit 150 separates the YUV format image data into a luminance component signal Y and a modulated color difference component C, and performs LPF processing on the analog Y signal that has undergone D / A conversion. In addition, the reproduction circuit 150 performs BPF processing on the analog C signal that has been subjected to D / A conversion, and extracts only the frequency component of the modulated color difference component. Based on the signal component thus generated and the subcarrier frequency, the reproduction circuit 150 converts and generates a Y signal and an RGB signal and outputs the Y signal and the RGB signal to the monitor 151. In this manner, an electronic viewfinder (EVF) is realized by sequentially processing and displaying the image data from the image sensor 115.

  The recording medium 200 includes a recording unit 201 composed of a semiconductor memory, an I / F unit 202 that is an interface with the imaging device 100, a connector 203 that connects to the imaging device 100, and a medium recording prohibition switch 204.

  From here, the moving image shooting processing according to the first embodiment will be described in detail. According to the moving image shooting process of the present embodiment, the image processing unit 117 of the imaging apparatus 100 can operate in the non-compositing mode or the combining mode in accordance with an instruction from the system control unit 121. In the non-synthesizing mode, the image processing unit 117 processes each piece of image data continuously generated by the image sensor 115 without being combined with other image data. In the combination mode, the image processing unit 117 generates an HDR image by combining a predetermined number of image data generated by the image sensor 115 by imaging with different exposure times. The image processing unit 117 can dynamically switch between the synthesis mode and the non-synthesis mode during moving image shooting. For example, the system control unit 121 automatically determines the shooting scene, and the image processing unit 117 switches between the synthesis mode and the non-synthesis mode based on the determination result.

  In this embodiment, the image sensor 115 is a CMOS sensor, and the recording frame rate is 60 FPS. The image sensor 115 can operate at an imaging frame rate of 60 FPS (first frame rate) or 120 FPS (second frame rate). In a case where HDR is required, the imaging apparatus 100 captures an image at 120 FPS and generates a single HDR frame by synthesizing a predetermined number of image data. In this embodiment, one HDR frame is created by combining two pieces of image data obtained by long-second exposure and short-second exposure. On the other hand, in the case where HDR is not necessary, the imaging apparatus 100 switches the imaging frame rate to 60 FPS.

  FIG. 2 is a flowchart of the moving image shooting process according to the first embodiment. Each process in the flowchart described below including FIG. 2 is realized by the system control unit 121 developing and executing the program stored in the flash memory 122 in the memory unit 123 unless otherwise specified. FIG. 2 shows processing after the moving image recording start is instructed by the moving image button 180.

  In step S <b> 201, the system control unit 121 instructs the image processing unit 117 to perform an imaging drive for moving image recording. In response to the imaging drive instruction, the image processing unit 117 sets an imaging synchronization signal in the SSG circuit. In addition, the image processing unit 117 performs moving image timing settings for the image sensor 115 and the timing generator 116.

  In step S202, the system control unit 121 initializes a flag “HDR_FLAG” indicating whether or not an HDR video is selected to 0. This indicates that a normal moving image is selected in the initial state.

  In step S203, the image processing unit 117 divides the image input from the image sensor 115 and integrates it for each block, and the system control unit 121 performs exposure control based on the result.

  In step S <b> 204, the image processing unit 117 processes the sequentially read image data into display image data by the signal processing circuit and the reduction circuit, and stores the display image data in the memory unit 123. Then, the image corresponding to the display image data is displayed on the monitor 151 through the reproduction circuit 150.

  In S205, the image processing unit 117 determines the difference in brightness between the high-luminance block and the low-luminance block in the image based on the integration result for each block of the image data obtained by appropriate exposure (hereinafter referred to as the proper exposure image). Is detected. In step S206, the system control unit 121 determines whether or not the light / dark difference is equal to or greater than a predetermined threshold. If it is less than the threshold, the process proceeds to S207, and if it is greater than or equal to the threshold, the process proceeds to S213.

  In S207, the system control unit 121 determines whether HDR_FLAG is 1, that is, whether an HDR video is selected. When HDR_FLAG is 1, in S208, switching processing from the HDR moving image to the normal moving image is performed, and then the process proceeds to S209. Details of S208 will be described later. If HDR_FLAG is not 1 in S207, that is, if a normal moving image has already been selected, the process skips S208 and proceeds to S209. In S209, normal moving image processing is performed. Details of S209 will be described later. In S210, the system control unit 121 sets HDR_FLAG to 0.

  In step S211, the system control unit 121 determines whether or not the moving image button 180 has been pressed again. If pressed, the system control unit 121 stops moving image recording in S212, and the processing of this flowchart ends. If it is determined in S211 that the moving image button 180 has not been pressed, the process returns to S203.

  If the brightness difference is greater than or equal to the predetermined threshold value in S206, in S213, the system control unit 121 determines whether HDR_FLAG is 0, that is, whether a normal moving image is selected. If HDR_FLAG is 0, in S214, switching processing from the normal moving image to the HDR moving image is performed, and then the process proceeds to S215. Details of S214 will be described later. If HDR_FLAG is not 0 in S213, that is, if an HDR video has already been selected, the process skips S214 and proceeds to S215. In S215, HDR moving image processing is performed. Details of S215 will be described later. In S216, the system control unit 121 sets HDR_FLAG to 1, and advances the process to S211.

  FIG. 3 is a flowchart of the switching process from the normal moving image to the HDR moving image. In step S301, the system control unit 121 calculates an appropriate exposure time Exp_just based on the exposure information and brightness difference obtained in steps S203 and S205 in FIG. In addition, the system control unit 121 calculates each exposure time Exp_long and Exp_short for obtaining two pieces of image data serving as a base of the composite image. Hereinafter, the image data obtained at the exposure time Exp_long is referred to as an over image, and the image data obtained at the exposure time Exp_short is referred to as an under image.

  In step S302, the system control unit 121 substitutes an appropriate exposure time Exp_just as an initial value of the exposure time Exp. In step S303, the system control unit 121 obtains the imaging V cycle Va after switching to the HDR video. In the present embodiment, since the imaging frame rate during HDR video is 120 FPS, Va is the reciprocal thereof, that is, Va = 8.33 (ms).

  In step S304, the system control unit 121 compares the exposure time Exp with the imaging V cycle Va after switching to the HDR video. If Exp> Va, the process proceeds to S305; otherwise, the process proceeds to S308.

  In step S305, the system control unit 121 shortens the exposure time by Va / 2 (Exp = Exp−Va / 2). In step S <b> 306, the system control unit 121 performs an exposure / readout operation with the exposure time after shortening on the image sensor 115. In addition, the system control unit 121 changes (increases) the gain at the time of reading in order to compensate for the luminance decrease due to the shortening of the exposure time. The read image data is temporarily stored in the memory unit 123. The system control unit 121 reads out the image data, and stores the recording image in the memory unit 123 through the signal processing circuit, the raster-block conversion circuit, and the compression circuit in the image processing unit 117. In S307, the recording images are sequentially read and written to the recording medium 200. By repeating the processing from S304 to S307, the exposure time is shortened step by step until it is equal to or less than the reciprocal of the imaging frame rate for HDR moving images.

  In step S308, the system control unit 121 assigns the overimage exposure time Exp_long to the exposure time Exp. In S309, the system control unit 121 controls the SSG circuit in the image processing unit 117 so that the imaging vertical synchronization signal (imaging VD) is changed from 16.67 (ms) to 8.33 (ms), so that the frame Change the rate from 60 FPS to 120 FPS. In steps S310 and S311, the system control unit 121 performs image exposure, reading, and recording in the same manner as in steps S306 and S307. With the above processing, the switching process from the normal moving image to the HDR moving image is completed.

  FIG. 4 is a flowchart of the HDR moving image process. In step S401, the system control unit 121 calculates exposure times Exp_long and Exp_short for the over image and the under image, which are the basis of the composite image, based on the exposure information and brightness difference obtained in steps S203 and S205 of FIG. calculate.

  In step S <b> 402, the system control unit 121 performs exposure / reading moving images of the under image exposed with the exposure time Exp_short. In step S <b> 403, the system control unit 121 performs an exposure / read-out operation for the over image exposed with the exposure time Exp_long. The read over image and under image are temporarily stored in the memory unit 123. In step S <b> 404, the system control unit 121 reads out an over image and an under image from the memory unit 123 and performs image composition in the image processing unit 117, thereby generating image data (HDR image) with an expanded dynamic range. In step S405, the system control unit 121 writes the HDR image to the recording medium 200 in the same manner as in step S307 in FIG. With the above processing, recording of an HDR image for one frame is completed.

  FIG. 5 is a flowchart of the switching process from the HDR moving image to the normal moving image. In step S502, the system control unit 121 substitutes the exposure time Exp_long for the over image into the exposure time Exp. In S503 and S504, the system control unit 121 performs image exposure, reading, and recording in the same manner as in S306 and S307 in FIG. In step S505, the system control unit 121 substitutes an intermediate value ((Exp + Exp_just) / 2) between the current exposure time Exp and the appropriate exposure time Exp_just for the exposure time Exp. In step S506, the system control unit 121 controls the SSG circuit in the image processing unit 117 so that the imaging vertical synchronization signal (imaging VD) is changed from 8.33 (ms) to 16.67 (ms). Change the rate from 120 FPS to 60 FPS. In S507 and S508, the system control unit 121 performs exposure, reading, and recording of an image in the same manner as in S306 and S307 of FIG. With the above processing, the switching process from the HDR moving image to the normal moving image is completed.

  FIG. 6 is a flowchart of normal moving image processing. In step S601, the system control unit 121 calculates an appropriate exposure time Exp_just based on the exposure information and the light / dark difference obtained in steps S203 and S205 in FIG. 2, and substitutes the appropriate exposure time Exp_just into the exposure time Exp. In steps S602 and S603, the system control unit 121 performs image exposure, reading, and recording in the same manner as in steps S306 and S307 in FIG. With the above processing, recording of a normal image for one frame is completed.

  FIG. 7 is a conceptual diagram showing the state of switching processing from a normal moving image to an HDR moving image in time series. 7A shows an imaging vertical synchronization signal (imaging VD), and FIG. 7B shows a time at each imaging VD timing. (C) represents an exposure image, which is expressed by a parallelogram because there is a difference in exposure timing between the upper part and the lower part of the screen by a rolling shutter method unique to the CMOS sensor. (D) represents the timing at which reading is actually performed from the image sensor. In this embodiment, the exposure image is sequentially read from the next imaging VD timing, and the time required for reading is tr = 4 ms. . (E) represents a recording timing, and (f) represents a recording vertical synchronizing signal (recording VD). Each recorded image (output image) is assigned an image number corresponding to the exposure image. (G) represents the gain level applied when reading an image from the image sensor. This gain can usually be adjusted inside the image sensor 115, but it may be a mechanism that can be changed by a pre-processing circuit in the image processing unit 117.

  At time t1, normal moving image recording is being performed, the imaging frame rate is 60 FPS, and the vertical period is 16.67 ms. When a switching instruction from the normal moving image to the HDR moving image is issued from the system control unit 121 between times t1 and t2, preparation for image composition is performed. First, at times t2 to t3, the system control unit 121 sets the exposure time slightly shorter than the exposure image (1) (S305 in FIG. 3). The system control unit 121 applies the gain g1 when reading the exposure image (2) thus captured from the image sensor 115 at time t3, so that the exposure image (2) has the same imaging output level as the exposure image (1). Adjust so that From time t3 to t4, the system control unit 121 sets the exposure time to the overimage exposure time (S308 in FIG. 3). In addition, the system control unit 121 controls the SSG circuit in the image processing unit 117 to change one vertical period of imaging to 120 FPS equivalent (8.33 ms) (S309 in FIG. 3). At time t4, the system control unit 121 applies the gain g2 when reading the exposure image (3) from the image sensor 115 so that the exposure image (3) has the same imaging output level as the exposure image (1). Adjust. Between times t4 and t5, the system control unit 121 captures an image with an under image exposure time, and generates an exposure image (4) (S402 in FIG. 4). Between times t5 and t6, the system control unit 121 captures an image with an overimage exposure time, and generates an exposure image (5) (S403 in FIG. 4). Reading of the exposure image (4) and the exposure image (5) is performed with a gain g3 since image synthesis is a premise. Then, the system control unit 121 uses the image processing unit 117 to combine the exposure image (4) and the exposure image (5) to generate an HDR image. During time t6 to t8, the same processing as during time t4 to t6 is repeated.

In FIG. 7, the magnitude relationship between the gain levels at each time is as follows.
g2>g1>g3> g0 (g0 is a gain level at time t1)
However, this is because the gain control is performed so as to absorb the fluctuation of the exposure time.

  As described above, in the process of switching from the normal moving image to the HDR moving image, the frame rate is changed through the transient state in which the exposure time and the gain level are changed stepwise. As a result, it is possible to optimally control the exposure time while suppressing rapid fluctuations in image quality. Further, by changing the imaging frame rate without changing the moving image shooting drive mode, it is possible to dynamically switch from the normal moving image to the HDR moving image while keeping the recording rate.

  Note that this transient state may be skipped under some condition. Conditions include, for example, (1) when switching from a normal movie to an HDR movie is triggered by a user instruction, and (2) when the shooting mode is set to a predetermined shooting mode.

  First, a case where switching from a normal moving image to an HDR moving image is performed using a user instruction as a trigger will be described. The user can manually switch from the normal moving image to the HDR moving image by operating an operation unit (not shown) of the imaging apparatus 100. In this case, in FIG. 2, the process proceeds to S213 regardless of the determination result of S206, and the process of S214 is skipped regardless of the determination result of S213.

  Next, a case where the shooting mode is set to a predetermined shooting mode will be described. The user can set the shooting mode by operating an operation unit (not shown) of the imaging apparatus 100. Examples of shooting modes that can be set include a sports mode and a night view mode. When a predetermined shooting mode (for example, sport mode) is set, the process of S214 is skipped regardless of the determination result of S213 in FIG.

  FIG. 8 is a conceptual diagram showing the state of the switching process from the HDR moving image to the normal moving image in time series. In FIG. 8, the meanings of (a) to (g) are the same as those in FIG.

  At time t1, the HDR moving image is being recorded, the imaging frame rate is 120 FPS, and the vertical period is 8.33 ms. When the system control unit 121 gives an instruction to switch from the HDR movie to the normal movie between times t3 and t4, the composition is canceled from the next exposure image. First, at time t2 to t3, the system control unit 121 sets the exposure time to the overimage exposure time (S502 in FIG. 5). The system control unit 121 applies the gain g1 when reading the exposure image (5) thus captured from the image sensor 115 at time t5, so that the exposure image (5) becomes the exposure image (3) + (4). Adjust to the same imaging output level. At times t5 to t6, the system control unit 121 sets the exposure time to an intermediate value ((Exp + Exp_just) / 2) between the current exposure time and the appropriate exposure time (S505 in FIG. 5). In addition, the system control unit 121 controls the SSG circuit in the image processing unit 117 to change one vertical period of imaging to 60 FPS equivalent (16.67 ms). The system control unit 121 applies the gain g2 when reading the exposure image (6) thus captured from the image sensor 115 at time t6, so that the exposure image (6) has the same imaging output level as the exposure image (5). Adjust so that Between times t6 and t7, the system control unit 121 sets the exposure time to an appropriate exposure time (S601 in FIG. 6). The system control unit 121 applies the gain g3 when reading the exposure image (7) thus captured from the image sensor 115 at time t7, so that the exposure image (7) has the same imaging output level as the exposure image (6). Adjust so that

In FIG. 8, the magnitude relationship between the gain levels at each time is as follows.
g1>g2>g0> g3 (g0 is the gain level at time t1)
However, this is because the gain control is performed so as to absorb the fluctuation of the exposure time.

  As described above, in the switching process from the HDR moving image to the normal moving image, it is possible to shorten the time from the switching instruction until the recorded image is actually switched to the normal moving image. Further, since the exposure time can be increased as much as the imaging frame rate is 60 FPS in a normal moving image, the gain level can be lowered and an image with less noise can be provided.

  As described above, according to the present embodiment, the imaging apparatus 100 captures images at 60 FPS during normal moving image processing and captures images at 120 FPS during HDR moving image processing. Therefore, there is no need to thin out images during normal moving image processing, and the exposure time can be made longer than in HDR moving image processing. Therefore, the image quality when combined / non-combined captured images are switched during moving image recording. Can be improved.

[Second Embodiment]
In the first embodiment, an example in which image composition is performed to generate an HDR video has been described. However, the basic idea of the first embodiment is also applicable to image composition for other purposes. Is possible. In the second embodiment, cyclic noise reduction (cyclic NR) generally used during moving image recording is taken up as another example of the purpose. In this embodiment, the basic configuration of the imaging apparatus 100 is the same as that of the first embodiment (see FIG. 1), but the configuration of the image processing unit 117 is appropriately changed so that cyclic NR can be performed.

  The cyclic NR is a method of extracting noise by using the image signal of the previous frame from which noise has been removed and the image of the current frame, and subtracting this noise from the image signal of the current frame to remove the noise. In the cyclic NR, because of the mechanism, an afterimage is generated when there is a motion in an image. Therefore, it is necessary to take measures such as changing the cyclic coefficient according to the motion or not performing the tour itself.

  FIG. 9 is a flowchart of the cyclic NR process according to the second embodiment. All the images (denoted as frames) in this flowchart indicate a state after signal processing after exposure reading from the image sensor 115 and before compression. These frames are temporarily stored in the memory unit 123 and updated at a predetermined frame rate by moving image capturing processing.

  In step S901, the imaging apparatus 100 acquires the current frame IMG1. In S902, the imaging apparatus 100 subtracts the previous frame IMG0 from this frame to generate a difference frame IMG_S. In S903, the imaging apparatus 100 extracts a motion amount from the difference frame IMG_S. In S904, the imaging apparatus 100 compares the amount of motion with the threshold values B0 and B1 (B1> B0). If B1> motion amount> B0 in S904, the process proceeds to S906 after the imaging frame rate is set to 120 FPS in S909. If the amount of motion <B0 in S904, the process proceeds to S906 after the imaging frame rate is set to 120 FPS in S909.

  In S906, the imaging apparatus 100 next extracts a high frequency component in IMG_S and substitutes it into IMG_NOISE_H. In S907, the imaging apparatus 100 subtracts IMG_NOISE_H from IMG_1, and substitutes the result into the output frame IMG. In step S908, the imaging apparatus 100 substitutes IMG for IMG0 for the next reference, and ends the process.

  If the amount of motion> B1 in S904, the imaging frame rate is set to 60 FPS in S910. In this case, in S911, the imaging apparatus 100 substitutes the current frame IMG1 as it is for the output frame IMG, and the process proceeds to S908.

  In this embodiment, for example, when the amount of motion is small, the imaging apparatus 100 performs cyclic NR at an imaging frame rate of 60 FPS, and when the amount of motion increases to some extent, performs cyclic NR at an imaging frame rate of 120 FPS. To do. When the amount of motion further increases, the imaging device 100 does not perform the cyclic NR itself.

  With such a configuration, the noise removal effect is increased in a scene with relatively small motion, and when the motion becomes large, it is possible to record a moving image while suppressing the occurrence of an afterimage.

  As described above, according to the present embodiment, it is possible to smoothly and optimally combine a plurality of frames during moving image recording. In the case of non-synthesis, a lower imaging frame rate can be used than in the case of performing synthesis (S909 and S910 in FIG. 9).

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

Imaging means;
Control means for controlling the imaging means so as to continuously perform imaging at a first frame rate or a second frame rate higher than the first frame rate;
Image processing means for processing image data continuously generated by the continuous imaging, wherein each image data is processed without being combined with other image data, or a predetermined number of image data Image processing means that operates in a synthesis mode for performing a process of synthesizing
With
The control means switches from the first frame rate to the second frame rate when the image processing means switches from the non-synthesizing mode to the synthesizing mode during the continuous imaging. When the image processing means switches from the synthesis mode to the non-synthesis mode during imaging, the imaging apparatus switches from the second frame rate to the first frame rate. When the image processing unit operates in the synthesis mode, the control unit controls the imaging unit so that the predetermined number of image data is generated by imaging with different exposure times, and the image processing unit includes: The image pickup apparatus according to claim 1, wherein image data having an expanded dynamic range is generated by combining the predetermined number of image data. A determination means for determining whether an exposure time of the imaging means is longer than a reciprocal of the second frame rate when the image processing means is switched from the non-synthesis mode to the synthesis mode;
When it is determined that the exposure time of the imaging means is longer than the reciprocal of the second frame rate, the control means reduces the exposure time of the imaging means to be equal to or less than the reciprocal of the second frame rate, The imaging apparatus according to claim 2, wherein after the imaging with the shortened exposure time is performed, the first frame rate is switched to the second frame rate. The control means reduces the exposure time of the imaging means stepwise until it becomes less than the reciprocal of the second frame rate when reducing the exposure time of the imaging means to be less than or equal to the reciprocal of the second frame rate. The imaging apparatus according to claim 3, wherein: The said control means changes the gain applied to the image data produced | generated by the imaging with the said shortened exposure time according to the change of the exposure time by the said shortening. The Claim 3 or 4 characterized by the above-mentioned. Imaging device. Receiving means for accepting a user instruction to switch the image processing means from the non-synthesis mode to the synthesis mode;
When the image processing means switches from the non-synthesis mode to the synthesis mode in accordance with the user instruction, the control means does not perform the shortening regardless of the exposure time length of the imaging means. The imaging apparatus according to any one of claims 3 to 5, wherein the frame rate is switched to the second frame rate. It further comprises setting means for setting the shooting mode,
When the shooting mode set by the setting unit is a predetermined shooting mode, the control unit starts from the first frame rate without performing the reduction regardless of the length of the exposure time of the imaging unit. The imaging apparatus according to claim 3, wherein the imaging apparatus is switched to the second frame rate. When the image processing means switches from the non-synthesis mode to the synthesis mode, the image processing means processes the image data generated by imaging at the second frame rate in the non-synthesis mode, and then performs the synthesis. The imaging device according to any one of claims 2 to 7, wherein an operation in a mode is started. When the image processing means switches from the synthesis mode to the non-synthesis mode, the control means has processed the image data generated by imaging at the second frame rate in the non-synthesis mode. The imaging apparatus according to any one of claims 2 to 8, wherein later, the second frame rate is switched to the first frame rate. An imaging method in an imaging apparatus comprising an imaging means,
A control step in which the control unit of the imaging apparatus controls the imaging unit so as to continuously perform imaging at a first frame rate or a second frame rate higher than the first frame rate;
The image processing means of the imaging apparatus is an image processing step of processing image data continuously generated by the continuous imaging, and each image data is processed without being combined with other image data. An image processing step that operates in a mode or a synthesis mode for performing a process of synthesizing a predetermined number of image data;
With
In the control step, when the image processing step switches from the non-synthesis mode to the synthesis mode during the continuous imaging, the first frame rate is switched to the second frame rate, and the continuous When the image processing step is switched from the synthesis mode to the non-synthesis mode during imaging, the imaging method is characterized by switching from the second frame rate to the first frame rate.   The program for functioning a computer as each means except the said imaging means of the imaging device of any one of Claims 1 thru | or 9.   A computer-readable recording medium storing a program for causing a computer to function as each unit other than the imaging unit of the imaging apparatus according to claim 1.

Images