JP2015186037A - Image processing device, control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of generating a frame in which a subject is suitably in focus when a moving image having its frame composed of light-beam space information is reproduced.SOLUTION: Light-beam space information included in each frame of a moving image is acquired, a plurality of different subject distances are set, and a plurality of reconfigured images are generated based upon the subject distances. A subject distance for output is determined based upon a focusing state of a subject in the reconfigured images, and a reconfigured image is generated from the determined subject distance for output. A plurality of subject distances set for a following frame is so set as to be closer to the subject distance determined for the previous frame than the plurality of subject distances set for the previous frame.

Description

  The present invention relates to an image processing apparatus, a control method, and a program, and more particularly, to a moving image reproduction technique in which a frame includes light space information.

  In recent years, a light field camera has been proposed as a camera that can focus on an arbitrary subject distance after shooting. It is known that a light field camera can record the intensity distribution of a reflected light beam from a subject as light field information together with information on the incident direction.

  In the configuration capable of recording light space information, a plurality of microlenses are arranged in front of the image sensor, and a plurality of pixels are associated with one microlens to separate the light flux that has passed through the pupil region. There is something to be recorded. Since the light space information has the intensity distribution of the reflected light beam for each incident direction, the intensity distribution of the reflected light beam on the surface at an arbitrary subject distance can be reproduced from the information. Therefore, an image focused on an arbitrary subject distance (reconstructed image) using the light space information can be generated (reconstructed) after the light space information is recorded.

  Patent Document 1 discloses a technique for realizing an autofocus function using a reconstructed image. In this technique, a plurality of reconstructed images having different subject distances to be focused are generated from the acquired light space information, and the in-focus state with respect to the subject is evaluated for the generated reconstructed images. Then, the focus corresponding to the appropriate subject distance is controlled by specifying the reconstructed image whose in-focus state is closest to the in-focus state. By generating a plurality of reconstructed images from the light space information, an operation of driving the optical system to pick up a plurality of images becomes unnecessary, and the focus control can be speeded up.

JP 2009-258610 A

  In order to obtain a reconstructed image in which the subject is in a suitable focus state from the light space information acquired by the light field camera, a large number of reconstructed images with different subject distances to be focused are generated, and the subject in each reconstructed image is generated. It is necessary to evaluate the in-focus state. For this reason, much processing time is required in order to generate a reconstructed image and to evaluate the in-focus state.

  By the way, in the case where it is not necessary to perform processing other than the display processing of the reconstructed image after the generation, for example, when a reconstructed image is generated from the light space information recorded for the still image, the in-focus state evaluation process It may be acceptable to spend time on However, when, for example, recorded as a moving image, or when the recorded light space information is sequentially acquired and displayed at a predetermined frame rate, the time that can be allocated to the evaluation process of the in-focus state related to one frame is It is limited to the time determined by the frame rate. For this reason, in order to complete the focused state evaluation process within a limited time, a method of reducing the number of subject distances for evaluating the focused state is conceivable. However, since the subject distance interval of the reconstructed image is widened and there is a possibility that the in-focus state cannot be evaluated for an appropriate subject distance, a reconstructed image in a suitable in-focus state may not be obtained.

  The present invention has been made in view of the above-described problems of the prior art, and is an image processing device capable of generating a frame that is suitably focused on a subject in the reproduction of a moving image in which the frame is composed of light space information. An object of the present invention is to provide a control method and a program.

  In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is, using an acquisition means for sequentially acquiring image signals from a moving image including an image signal included in each frame capable of generating a reconstructed image focused on an arbitrary subject distance, and specific information for specifying a subject distance to be focused And generating means for generating a reconstructed image focused on the subject distance from the image signal, setting means for setting a plurality of specific information for specifying different subject distances for the image signal, and setting means. Determination of determining specific information for output that specifies subject distances in which the focus state of the subject satisfies a predetermined condition based on the focus state of the subject in the plurality of reconstructed images generated using the plurality of specific information Means and an output means for outputting a reconstructed image generated using the specific information for output, and the setting means outputs a duplicate for the second frame subsequent to the first frame. The subject distance specified by the specific information for the output is determined by the specific information for output determined for the first frame, rather than the subject distance specified by the plurality of specific information set for the first frame. A plurality of specific information about the second frame is set so as to be dense within a predetermined subject distance range including the distance.

  According to the present invention, it is possible to generate a frame that is suitably focused on a subject in the reproduction of a moving image in which the frame is composed of ray space information.

1 is a block diagram showing a functional configuration of a digital video camera as an example of an image processing apparatus according to the present invention. The figure which shows typically the pixel arrange | positioned at the microlens array as an example of the digital video camera which concerns on this invention, and an imaging surface The figure which shows typically the reflected light beam from the photographic subject received with an image sensor The figure which shows typically the correspondence of the position in an imaging lens and a reconstruction surface about the light beam received with an image sensor. The block diagram which shows the function structure of the refocus processing part which concerns on Embodiment 1 of this invention. The figure which shows the relationship between the refocus coefficient between the frames which concern on Embodiment 1 of this invention, and a focus evaluation value. The block diagram which shows the function structure of the refocus processing part which concerns on Embodiment 2 of this invention. The figure which shows the relationship between the refocus coefficient between the frames which concern on Embodiment 2 of this invention, and a focus evaluation value. The flowchart which shows a series of operation | movement of the refocus process which concerns on Embodiment 1 of this invention. The flowchart which shows a series of operation | movement of the refocus process based on Embodiment 2 of this invention.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, the present invention is applied to a digital video camera capable of recording light space information (LF data) composed of pixel values having information on the intensity and incident angle of a reflected light beam from a subject as an example of an image processing apparatus. An example will be described. However, the present invention is not limited to a digital video camera capable of recording LF data, but can be applied to any device that can read recorded LF data and reproduce a moving image focused on a desired subject distance. . These devices may include, for example, mobile phones, game machines, tablet terminals, personal computers, and the like.

(1 Configuration of digital video camera)
FIG. 1 is a diagram illustrating a functional configuration example of the digital video camera according to the present embodiment. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. May be. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject.

  The reflected light beam from the subject is collected by the imaging lens 101, passes through the microlens array 102, is projected onto the imaging element 103, and is subjected to photoelectric conversion. The imaging lens 101 is an imaging optical system including a single lens or a plurality of lenses, and may have a focus function and a zoom function. As shown in FIG. 2A, the microlens array 102 is disposed between the imaging lens 101 and the imaging element 103 and in the vicinity of the imaging element 103. The image sensor 103 is a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a plurality of pixels are arranged in a grid pattern.

  In the microlens array 102, for example, a plurality of microlenses are arranged in a lattice pattern. One microlens is arranged so that n × m pixels of the image sensor 103 correspond to each microlens. For example, in FIG. 2B, 5 × 5 pixels of the image sensor 103 with respect to the microlens 201. Are arranged to correspond.

  A reflected light beam from a subject imaged on one microlens is developed by the microlens and received by a pixel corresponding to each passing region of the imaging lens 101. As shown in FIG. 3, the light beam reflected from one point (302) of the subject 301 enters the imaging lens 101. The incident light beam is condensed by the imaging lens 101 and imaged on the imaging surface 303 on the surface of the microlens array 102. The imaged light flux is developed and received by the microlens on the corresponding pixels 321 to 325 for each pupil region that has passed. For example, in FIG. 3, the light beam 311 that has passed through the upper end portion of the imaging lens 101 is received by the pixel 321, and the light beam 315 that has passed through the lower end portion of the imaging lens 101 is received by the pixel 325. Since each pixel of the image sensor 103 receives the light beam developed by the microlens in this way, the reflected light beam from the subject 301 is an image signal divided for each pupil region of the image pickup lens 101, that is, LF data. Configure. The image sensor 103 outputs LF data obtained by photoelectric conversion by each pixel to the LF data input unit 104. The image sensor 103 sequentially outputs LF data of the number of frames determined by the frame rate per second. The sequentially output LF data constitutes a moving image including LF data in each frame.

  The LF data input unit 104 performs amplification processing and A / D conversion processing on the LF data input from the image sensor 103. The LF data input unit 104 performs development processing such as flaw correction and color balance correction on the LF data using, for example, preset conversion characteristics, and stores the processed LF data in the memory 105. The memory 105 is, for example, a dynamic RAM that can be randomly accessed at high speed.

  The media I / F 108 reads the LF data sequentially stored in the memory 105 by the imaging operation in accordance with an instruction from the controller 111 and writes the LF data to the recording medium 109. The media I / F 108 writes data in a file system format such as FAT, for example. Further, the media I / F 108 generates a file of LF data on the recording medium 109 and controls the file system. Further, when reproducing the LF data recorded on the recording medium 109, the media I / F 108 reads out the LF data from the recording medium 109 and stores it in the memory 105.

  The recording medium 109 is configured by a nonvolatile semiconductor memory such as a hard disk drive or a flash memory, and writes and reads LF data under the control of the media I / F 108.

  In the case of an imaging operation, the refocus processing unit 110 acquires LF data stored in the memory 105 by imaging, and refocuses a specific subject distance (specific focal plane and focal position) for each frame. A configuration image is generated (refocus processing). Although details of the refocus processing will be described later, a video (reconstructed video) focused on a specific subject distance is generated by sequentially outputting the reconstructed images generated by the refocus processing unit 110 for each frame. The refocus processing unit 110 outputs the generated reconstructed image and stores it in the memory 105. On the other hand, when playing back the LF data recorded on the recording medium 109, the refocus processing unit 110 sequentially acquires the LF data stored in the memory 105 via the media I / F 108 and performs the refocus processing. .

  The video output unit 106 sequentially acquires the reconstructed video stored in the memory 105 according to an instruction from the controller 111 and outputs the reconstructed video to the display unit 107 and a video output terminal (not shown). The display unit 107 displays the reconstructed video input from the video output unit 106. When the imaging operation is performed, the reconstructed video being imaged is displayed on the display unit, and the user can perform imaging while confirming the reconstructed video. When the LF data relating to the moving image recorded on the recording medium 109 is reproduced, the reconstructed video of the LF data is displayed on the display unit 107.

  The operation unit 112 is configured by a touch panel sensor disposed on the surface of the display unit 107, in addition to a button for user operation to start capturing a moving image and a switch for setting. When an operation by the user is notified from the operation unit 112 to the controller 111, each functional block is controlled by an instruction from the controller 111, and an imaging operation, a reproduction operation, and the like are performed. Further, the user can designate an area in the image to be focused on the reconstructed video through the operation unit 112.

  The controller 111 is a controller that performs overall operation control of the digital video camera, and is a so-called CPU. The CPU expands the program stored in the ROM (not shown) in the work area of the memory 105 and executes it, thereby controlling the operation of the digital video camera.

  Each functional block described above is connected to the data bus 113 and transmits / receives control signals to / from the controller 111. The data bus 113 controls transmission / reception of data from each functional block by time division.

(2 Processing of refocus processing unit)
The process of the refocus processing unit 110 will be described in more detail. The refocus processing unit includes functional blocks shown in FIG.

  In the refocus processing unit 110, the reconstructed image is generated by the evaluation image generating unit 601 and the reconstructed image generating unit 605. Generation of a reconstructed image in the functional block will be described. As shown in FIG. 4A, the optical path of the light beam incident on each pixel of the LF data obtained by imaging is the coordinates (u, v) of the pupil region in the imaging lens 101 and the position of the corresponding microlens. It is defined by coordinates (x ′, y ′). In the generation of a reconstructed image, a light beam having an optical path passing through the point is integrated with respect to a pixel (x, y) on a reconstruction plane (focal plane) corresponding to an arbitrary subject distance for generating the reconstructed image. Thus, the pixel value can be obtained.

FIG. 4B shows an optical path of a light beam viewed from the vertical direction in the lateral position of the digital video camera 100. If the coordinates of the pupil region on the imaging lens surface are (u, v) and the coordinates of the pixel on the reconstruction surface are (x, y), a light beam passing through the pupil region and the pixel on the reconstruction surface is incident. The coordinates (x ′, y ′) of the position of the microlens are


It is represented by F is a distance from the imaging lens to the imaging plane, and αF is a distance from the exit pupil of the imaging lens to the reconstruction surface (α is a refocus coefficient: a variable coefficient for determining the distance to the reconstruction surface). is there.

If the output of the photoelectric conversion element that receives the luminous flux is L (x ′, y ′, u, v), the pixel output E (x, y) of the coordinates (x, y) of the image formed on the reconstruction surface y) is obtained by integrating L (x ′, y ′, u, v) with respect to the pupil region of the photographing lens,


It is represented by By using (u, v) as the representative coordinates of the pupil region, the equation can be calculated by simple addition.

  The evaluation image generation unit 601 generates a plurality of reconstructed images (evaluation images) for evaluating the in-focus state from one frame of LF data acquired from the memory 105. The plurality of evaluation images are reconstructed images generated so as to be focused on different subject distances, and have different degrees of focus on the subject. The plurality of evaluation images are generated based on the plurality of refocus coefficients set by the evaluation coefficient setting unit 604. Here, the refocus coefficient set for generating the evaluation image is referred to as an evaluation coefficient. The evaluation image generation unit 601 may generate a reconstructed image limited to only a part of a subject area to be focused (a focus target area) among the captured image areas. The focus target area is, for example, a face area when the subject is a person. Further, it may be an image area set by the user through the operation unit 112. By reducing the area to be processed in this way, it is possible to reduce the amount of processing required to generate one evaluation image, and therefore it is possible to evaluate the in-focus state for a larger subject distance. The evaluation image generation unit 601 outputs the generated plurality of evaluation images to the focus evaluation unit 602.

  The focus evaluation unit 602 calculates a focus evaluation value for evaluating the in-focus state for each of the plurality of evaluation images generated by the evaluation image generation unit 601. The focus evaluation unit 602 outputs the focus evaluation value calculated for each evaluation image to the refocus coefficient determination unit 603.

  The refocus coefficient determination unit 603 determines a refocus coefficient used by the reconstructed image generation unit 605 from a plurality of focus evaluation values calculated by the focus evaluation unit 602. First, the refocus coefficient determination unit 603 selects a corresponding evaluation image by selecting the highest focus evaluation value among the acquired focus evaluation values. Then, the value of the evaluation coefficient corresponding to the selected evaluation image is determined as a refocus coefficient for generating a reconstructed image for output (frame image displayed by reproduction). The refocus coefficient determination unit 603 outputs the determined refocus coefficient to the reconstructed image generation unit 605. Also, the refocus coefficient determination unit 603 outputs the determined refocus coefficient to the evaluation coefficient setting unit 604.

  The reconstructed image generation unit 605 uses the refocus coefficient determined by the refocus coefficient determination unit 603 to generate a reconstructed image for the entire image, that is, a reconstructed image corresponding to the entire area of the subject being photographed. . The generated reconstructed image is stored in the memory 105.

  The evaluation coefficient setting unit 604 sets a new evaluation coefficient for the next frame using the refocus coefficient determined by the refocus coefficient determination unit 603. The evaluation coefficient set in the evaluation coefficient setting unit 604 is set based on the refocus coefficient determined for the immediately preceding frame. In general, since images of consecutive frames in a moving image have high correlation, it is considered that the subject distance of the subject focused on in the Nth frame does not vary greatly in the subsequent N + 1th frame. That is, it is highly likely that the subject distance focused on the subject in the (N + 1) th frame is close to the subject distance focused on the subject in the Nth frame, that is, a predetermined range including the subject distance. In other words, when there is a high possibility that there is a subject distance that focuses on the subject in the (N + 1) th frame in the vicinity of the subject distance, if the focus state is closely evaluated at a plurality of subject distances in the vicinity, the subject is more It is easy to find the subject distance to focus on. Therefore, if the focus state of the reconstructed image in the (N + 1) th frame is evaluated closely in the vicinity of the subject distance in which the focus state is closest to the focus in the Nth frame, the subject distance focused on the subject is determined. It can be suitably determined. In the processing of the present embodiment, it is assumed that the reconstructed image for in-focus state evaluation or output is specified by the refocus coefficient. However, the focal length of the imaging optical system specified by the refocus coefficient is described. Needless to say, the information is equivalent to the subject distance. That is, in order to focus on a specific subject distance, it is necessary to change the focal length of the imaging optical system to a predetermined distance, so there is a causal relationship between the two according to the characteristics of each imaging optical system. There is. Accordingly, in the generation of the reconstructed image according to the present embodiment, a specific reconstructed surface is defined by the refocus coefficient, but other than the refocus coefficient can be used for defining the reconstructed surface. That is, in the embodiment of the present invention, the information defining the reconstruction plane includes information (specific information) that can specify a subject distance to be focused in the generated reconstructed image, such as a focal length and a subject distance, in addition to a refocus coefficient. ) Any information may be used. In the following description, the Nth frame is referred to as a frame N, and the N + 1th frame is referred to as a frame N + 1.

  In the frame N, the evaluation coefficient is set by the evaluation image generation unit 601, and the in-focus state is evaluated by the focus evaluation unit 602 for the reconstructed image corresponding to each evaluation coefficient. The relationship between the distribution of evaluation coefficients and the focus evaluation value in the frame N is, for example, as shown in FIG. In FIG. 6A, the horizontal axis indicates the refocus coefficient value, and the vertical axis indicates the focus evaluation value. In the frame N, the interval between the evaluation coefficients is set wide. The evaluation coefficient with the highest focus evaluation value is α3. For this reason, among the set evaluation coefficients, α3 is a refocus coefficient in which the in-focus state is closest to the in-focus state in the frame N.

  In the frame N + 1, the evaluation image generation unit 601 sets the evaluation coefficient setting interval in the vicinity of the median value with the α3 value having the highest focus evaluation value in the frame N as the median value. In the drawing, as an example, the evaluation coefficients are set densely in the vicinity of the median value, and an example is shown in which the interval between the assessment coefficients is increased as the distance from the median value is increased. The relationship between the evaluation coefficient set in the frame N + 1 and the focus evaluation value is, for example, as shown in FIG. The value of α3 determined in the frame N is set as the value of α4 in the frame N + 1, and the in-focus state is evaluated closely in the vicinity of α4. As a result, the value of the refocus coefficient that is closest to the in-focus state is determined as the value of α5.

  As described above, even when the subject distance of the subject changes, generally the change in the subject distance in successive frames is small. Accordingly, similarly, in the frame N + 2, as shown in FIG. 6C, by setting the evaluation coefficient closely in the vicinity of the value of the evaluation coefficient α5 determined in the frame N + 1, the refocus coefficient that focuses on the subject. Can be determined.

  It should be noted that the evaluation coefficient setting interval may be changed so that the evaluation coefficient intervals become closer according to the imaging frame rate. This is because, when the frame rate is high, the movement of the subject between frames becomes smaller, so that the correlation between the images between frames is high and the change in the focus state is small. Therefore, when the frame rate is high, if the setting interval of the evaluation coefficient is narrowed, a higher focusing evaluation value can be obtained and focusing accuracy can be improved.

  When the evaluation coefficient setting unit 604 sets a plurality of evaluation coefficients by the above-described method, the evaluation coefficient setting unit 604 outputs the set evaluation coefficients to the evaluation image generation unit 601.

(3 Operation of refocus processing unit)
Next, the operation of the refocus processing unit will be described with reference to the flowchart shown in FIG. The flowchart of FIG. 9 starts when the controller 111 detects an instruction for an imaging operation by the user and starts imaging, and the captured LF data is stored in the memory 105.

  In S101, the controller 111 instructs reading of LF data for one frame from the memory 105, and the refocus processing unit 110 reads the LF data. Inside the refocus processing unit 110, the evaluation image generation unit 601 reads the supplied LF data, issues an evaluation coefficient acquisition request to the evaluation coefficient setting unit 604, and advances the processing to S102.

  In S102, the evaluation coefficient setting unit 604 determines whether the refocus coefficient is determined in the immediately preceding frame. The evaluation coefficient setting unit 604 determines that the refocus coefficient is determined when the refocus coefficient value of the immediately preceding frame determined by the refocus coefficient determination unit 603 is not a preset fixed value, and performs processing. To S103. On the other hand, if the value of the refocus coefficient of the immediately preceding frame is a preset fixed value, it is determined that the refocus coefficient has not been determined. When the first frame from which reading of LF data is started is a processing target, the value of the refocus coefficient is set to a fixed value because the refocus coefficient of the immediately preceding frame has not been determined. As will be described later in S110, when it is determined that the refocus coefficient cannot be reused, the immediately preceding refocus coefficient is reset and set to a fixed value. If the evaluation coefficient setting unit 604 determines that the refocus coefficient has not been determined, the process proceeds to S104.

  In S103, the evaluation coefficient setting unit 604 sets an evaluation coefficient based on the refocus coefficient of the immediately preceding frame determined by the refocus coefficient determination unit 603. The evaluation coefficient setting unit 604 sets seven evaluation coefficients according to, for example, a standard determined in advance through experiments or the like. The evaluation coefficient setting unit 604 sets, for example, seven evaluation coefficients as α1 to α7 in order from the smallest value, and sets the value of the refocus coefficient of the immediately preceding frame to α4 as the central evaluation coefficient. The evaluation coefficient setting unit 604 previously determines the displacement from α4 at the center for each coefficient α1 to α7 and records it as a displacement table of evaluation coefficients. The displacement recorded in the table is set so that the displacement from the adjacent evaluation coefficient increases as the evaluation coefficient moves away from α4 having the median value. The evaluation coefficient setting unit 604 determines the evaluation coefficients α1 to α3 and α5 to α7 according to the displacement read from the table to the set value of α4. When the setting of the seven evaluation coefficients is completed, the evaluation coefficient setting unit 604 outputs the evaluation coefficient to the evaluation image generation unit 601 and advances the process to S105.

  In S104, the evaluation image generation unit 601 sets the evaluation coefficients at equal intervals. For example, a value obtained by equally dividing a range that can be taken by a preset evaluation coefficient is set as each evaluation coefficient α1 to α7. When the setting of each evaluation coefficient is completed, the evaluation coefficient setting unit 604 outputs the evaluation coefficient to the evaluation image generation unit 601 and advances the process to S105.

  In S105, the evaluation image generation unit 601 generates an evaluation image corresponding to each evaluation coefficient using the obtained seven evaluation coefficients. The evaluation image generation unit 601 generates the evaluation image by calculating the pixel output E (x, y) on the reconstruction plane described above. When the focus target area is set, the evaluation image generation unit 601 acquires the set focus target area. As the focus target area, for example, a face area of a person detected by a known subject recognition technique for the immediately preceding frame or the current frame may be set. When the focus target area is set, the evaluation image generation unit 601 acquires the coordinates of the image area, and generates an evaluation image limited to the area. When the evaluation image generation unit 601 generates evaluation images for all of the acquired evaluation coefficients, the evaluation image generation unit 601 outputs the evaluation image to the focus evaluation unit 602 and advances the processing to S106.

  In S106, the focus evaluation unit 602 evaluates the focus state for each evaluation image. The evaluation of the in-focus state is performed by calculating the in-focus degree of the image by a generally known method. For example, the degree of focus of the image can be calculated by using methods such as the intensity of the edge component, the dispersion value, and the contrast ratio. Therefore, detailed description in this embodiment is omitted. The focus evaluation unit 602 calculates, for example, an average value of the intensity of the edge component in the evaluation image and sets it as the focus evaluation value of each evaluation image. The focus evaluation unit 602 outputs the calculated focus evaluation value to the refocus coefficient determination unit 603, and the process proceeds to S107.

  In S107, the refocus coefficient determination unit 603 determines a refocus coefficient that satisfies a predetermined condition based on the acquired focus evaluation value. For example, one refocus coefficient whose focus state is closest to the focus is determined. The refocus coefficient determination unit 603 identifies the highest focus evaluation value by comparing the acquired focus evaluation values. The refocus coefficient determination unit 603 determines an evaluation coefficient corresponding to the focus evaluation value as a refocus coefficient for output. The refocus coefficient determination unit 603 outputs the determined refocus coefficient to the reconstructed image generation unit 605, and the process proceeds to S108.

  In S108, the reconstructed image generation unit 605 generates a reconstructed image corresponding to the acquired refocus coefficient. The reconstructed image generation unit 605 stores the generated reconstructed image in the memory 105 when the generation of the reconstructed image is completed.

  In S109, the refocus coefficient determination unit 603 determines whether the refocus coefficient can be reused in the next frame. The refocus coefficient determination unit 603 determines that the refocus coefficient is not reusable when the focus evaluation value corresponding to the determined refocus coefficient is equal to or less than a predetermined value. This is because, when the in-focus evaluation value is equal to or less than a predetermined value, the correlation between frames is lowered due to a rapid movement of the subject or the like, and the set evaluation coefficient may not be appropriate. If the focus evaluation value is greater than or equal to the predetermined value, the refocus coefficient determination unit 603 advances the process to S110. On the other hand, when the value is larger than the predetermined value, the refocus coefficient determination unit 603 outputs the refocus coefficient to the evaluation coefficient setting unit 604 and advances the process to S111.

  In S110, the refocus coefficient determination unit 603 resets the refocus coefficient notified to the evaluation coefficient setting unit 604. The refocus coefficient determination unit 603 sets the value of the refocus coefficient notified to the evaluation coefficient setting unit 604 to 0, and then outputs the value to the evaluation coefficient setting unit 604, and the process proceeds to S111.

  In S111, the refocus processing unit 110 determines whether the processing target frame is the last frame. If the processing target frame is not the last frame, the process returns to S101 again. When the processing target is the last frame, the series of processing ends.

  The reconstructed image output from the refocus processing unit 110 is sequentially output to the memory 105 and then displayed on the display unit 107 as a reconstructed video focused on a specific subject.

  The refocus coefficient determination unit 603 obtains a curve that matches the distribution of the focus evaluation values from the relationship between the discrete focus evaluation values and the evaluation coefficient values, and the highest focus is obtained based on the curve. A refocus coefficient having an evaluation value may be obtained. It is possible to estimate a focus evaluation value peak that appears between discretely distributed evaluation coefficients, and to determine a refocus coefficient with improved focusing accuracy.

  As described above, according to the present embodiment, the generated reconstructed image is matched by setting the subject distance for evaluating the in-focus state in the vicinity of the subject distance determined for output in the immediately preceding frame. It becomes possible to improve the focusing accuracy. That is, in the reproduction of a moving image in which the frame is composed of the light space information, it is possible to generate a frame that is suitably focused on the subject.

(Embodiment 2)
Next, Embodiment 2 will be described. In the second embodiment, the refocus processing unit 110 detects the motion of the subject in the depth direction, and the evaluation coefficient is corrected according to the result of the motion detection. In the present embodiment, the movement in the depth direction will be described as a displacement of the subject distance. The digital video camera of this embodiment is the same as that of Embodiment 1 except for the internal configuration of the refocus processing unit 110 and the operation of the refocus processing unit 110. For this reason, the overlapping description will be omitted, and differences will be mainly described.

  Details of the refocus processing unit 110 according to the present embodiment will be described with reference to FIG. 7 and differences from the first embodiment.

  The LF data for one frame input to the refocus processing unit 110 is input to the detection image generation unit 701 in addition to the evaluation image generation unit 601.

  The detection image generation unit 701 generates a reconstructed image (detection image) used for motion detection in the depth direction of the subject from LF data for one frame. The detection image generation unit 701 may generate the detection image by limiting it to a partial image region, for example, the above-described focus target region. The detection image generation unit 701 outputs the generated reconstructed image to the motion detection unit 702. The detection image generation unit 701 holds the generated detection image for motion detection in the next frame.

  The motion detection unit 702 detects the movement of the subject in the depth direction between consecutive frames using the detection image acquired from the detection image generation unit 701. The motion detection unit 702 detects a motion in the depth direction by detecting a motion vector for a block having a predetermined number of pixels between frames in the detection image.

  The evaluation coefficient setting unit 604 corrects the refocus coefficient determined in the immediately previous frame based on the motion in the depth direction acquired from the motion detection unit 702 and sets the evaluation coefficient. Specifically, the evaluation coefficient setting unit 604 converts the magnitude of the motion in the depth direction into a correction value for the evaluation coefficient, and corrects the evaluation coefficient with the correction value. The evaluation coefficient setting unit 604 sets the obtained value of the corrected refocus coefficient as the central evaluation coefficient.

  FIG. 8C shows an example of correcting the refocus coefficient. FIG. 8A and FIG. 8B show that the evaluation coefficient for the frame N and the frame N + 1 is set in the vicinity of the high value of the focus evaluation value in common with the first embodiment. FIG. 8C shows the refocus coefficient corrected by using the magnitude of the motion in the depth direction and the refocus when the subject moves in the depth direction during the frame N + 1 in the frame N + 2. The coefficient for evaluation set from the coefficient is shown. The value of the refocus coefficient α5 determined in the frame N + 1 is corrected by the correction value β, and another evaluation coefficient is set with the corrected refocus coefficient value being α4. By correcting with the correction value β, the evaluation coefficient can be set in the vicinity of the refocus coefficient that increases the focus evaluation value even if the subject moves in the depth direction.

  When all the evaluation coefficients are set, the evaluation coefficient setting unit 604 outputs the evaluation coefficient generation unit 601 to the evaluation image generation unit 601. Subsequent processes executed by the evaluation image generation unit 601 are the same as those in the first embodiment.

  Next, the operation of the refocus processing unit 110 according to the second embodiment will be described.

  In step S <b> 201, the controller 111 instructs reading of LF data for one frame from the memory 105, and the detection image generation unit 701 reads LF data for one frame from the memory 105. When the reading of the LF data is completed, the detection image generation unit 701 advances the process to S202. In parallel, the evaluation image generation unit 601 reads the supplied LF data.

  In S202, the detection image generation unit 701 generates a detection image. The detection image generation unit 701 generates a reconstructed image having a deep depth of field in order to generate a detection image. The detection image generation unit 701 calculates the pixel output E (x, y) in the above-described generation of the reconstructed image only for pixels that receive the light beam that has passed through the center of the pupil region. Since this corresponds to reducing the lens aperture during imaging, the detection image generation unit 701 can generate a reconstructed image with a deep depth of field. The detection image generation unit 701 outputs the generated detection image to the motion detection unit 702, and the process proceeds to S203.

  In step S <b> 203, the motion detection unit 702 detects a motion in the depth direction with respect to the detection image generated by the detection image generation unit 701. The motion detection unit 702 acquires a detection image for the immediately preceding frame, and executes a motion vector detection process between the focus target region of the image and the detection image for the current frame. A known motion estimation method may be used for motion vector detection. The motion detection unit 702 determines that there is motion in the depth direction when the motion vector distribution in the focus target region is configured to have a radial component. The motion detection unit 702 calculates, for example, an average value of the magnitudes of the detected motion vectors in the radial direction, and uses the calculated value as information indicating the magnitude of motion. The motion detection unit 702 outputs information representing the magnitude of the motion to the evaluation image generation unit 601 and advances the process to S102.

  In S102, the evaluation coefficient setting unit 604 determines whether the refocus coefficient is determined in the immediately preceding frame, as in the first embodiment. If the immediately preceding refocus coefficient value is a fixed value, the process proceeds to S104. On the other hand, if the refocus coefficient is not a fixed value, the process proceeds to S204.

  In S204, the evaluation coefficient setting unit 604 determines whether there is a movement in the depth direction in successive frames. The evaluation coefficient setting unit 604 refers to the information indicating the magnitude of motion acquired from the motion detection unit 702 and compares the magnitude with a predetermined threshold. If the magnitude is larger than the threshold, the evaluation coefficient setting unit 604 determines that there is a movement in the depth direction, and advances the process to S205. On the other hand, if the size is equal to or smaller than the threshold value, it is determined that there is no movement in the depth direction, and the process proceeds to S103.

  In step S <b> 205, the evaluation coefficient setting unit 604 sets the corrected evaluation coefficient according to the magnitude of the motion in the depth direction acquired from the motion detection unit 702. The evaluation coefficient setting unit 604 converts the magnitude of the movement in the depth direction into the displacement of the refocus coefficient. For example, a table in which correction values determined experimentally for the size of the detected human face region and the size of the motion vector are recorded, and the evaluation coefficient setting unit 604 obtains the obtained depth direction. The correction value β is set in accordance with the movement of. The evaluation coefficient setting unit 604 corrects the refocus coefficient determined in the immediately preceding frame using the correction value β, and sets a central evaluation coefficient (α4). The evaluation coefficient setting unit 604 sets other evaluation coefficients based on the central evaluation coefficient as in the first embodiment, and advances the process to S105.

  The refocus processing unit 110 subsequent to S105 is the same as in the first embodiment, and ends the series of processing when the processing target frame is the last frame.

  In the present embodiment, the method of detecting a motion vector has been described for detecting the motion in the depth direction. However, the motion detection unit 702 detects the change in the size of the subject in the image to detect the motion in the depth direction. Motion may be detected. For example, an image obtained by reducing and enlarging the previous detection image in multiple stages is generated, matched with the detection image for the current frame, and a predetermined size in the depth direction is assigned according to the generated enlargement ratio May be. In the present embodiment, the focus target area is described as a face area when the subject is a person. However, the focus target subject is not limited to the face, and a predetermined pattern capable of pattern detection such as the whole person or a building. It may be a subject.

  As described above, in the digital camera according to the present embodiment, by setting the evaluation coefficient corrected based on the result of the motion detection in the depth direction, even if the motion in the depth direction occurs in the subject, it follows the motion. And focusing evaluation can be performed. That is, even if there is a movement of the subject, a reconstructed image in a suitable in-focus state can be displayed at a higher speed.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 100 Digital video camera, 102 Micro lens array, 103 Image sensor, 110 Refocus processing part, 601 Evaluation image generation part, 602 Focus evaluation part, 603 Refocus coefficient determination part, 604 Evaluation coefficient setting part, 605 Reconfiguration Image generator

Claims (11)

Acquisition means for sequentially acquiring the image signal from a moving image including an image signal in each frame capable of generating a reconstructed image focused on an arbitrary subject distance;
Generating means for generating a reconstructed image focused on the subject distance from the image signal using specific information for specifying a subject distance to be focused;
Setting means for setting a plurality of the specific information for specifying different subject distances for the image signal;
Based on the focus state of the subject in the plurality of reconstructed images generated using the plurality of specific information set by the setting means, an output for specifying a subject distance in which the focus state of the subject satisfies a predetermined condition Determining means for determining specific information for
Output means for outputting a reconstructed image generated using the output specific information;
Have
The setting means includes a subject distance specified by the plurality of specific information set for the first frame, the subject distance specified by the plurality of specific information for the second frame following the first frame. Rather, the plurality of identifications for the second frame so as to be dense within a predetermined subject distance range including the subject distance identified by the output identification information determined for the first frame. An image processing apparatus characterized by setting information.   The image processing according to claim 1, wherein the setting unit includes the output specific information determined for the first frame in the plurality of specific information set for the second frame. apparatus.   The setting means is configured such that the higher the frame rate of the moving image, the denser the subject distance specified by the plurality of specific information about the second frame is within the range of the predetermined subject distance. The image processing apparatus according to claim 1, wherein the plurality of specific information for the second frame is set.   For each of the plurality of reconstructed images, the determining means determines the subject distance that satisfies the predetermined condition of the subject in-focus state based on the relationship between the in-focus state of the subject and the subject distance used to generate the reconstructed image. The image processing apparatus according to claim 1, wherein the output specific information is determined.   The determining means determines, as the output specific information, specific information used to generate a reconstructed image in which the in-focus state of the subject is closest to the in-focus state among the plurality of reconstructed images. The image processing apparatus according to any one of claims 1 to 3. The specific information is information indicating a focal length of the imaging optical system focused on the subject distance,
The setting means sets the plurality of specific information for the second frame, with a focal length of the output specific information determined for the first frame as a median value. The image processing apparatus according to any one of 1 to 5. Detection means for detecting movement of the subject in the depth direction between the first frame and the second frame;
The setting means changes the subject distance specified by the output specific information determined for the first frame in accordance with the magnitude of movement of the detected subject in the depth direction, and the changed subject distance The image processing apparatus according to claim 1, wherein the plurality of pieces of specific information for the second frame are set based on the first frame.   8. The generation unit according to claim 1, wherein the generation unit generates the plurality of reconstructed images limited to a partial region of the reconstructed image that can be generated from the image signal. 9. Image processing apparatus.   The image processing apparatus according to claim 8, wherein the partial area is an area where a predetermined subject is detected in a reconstructed image that can be generated from the image signal. An acquisition step of sequentially acquiring the image signal from a moving image including an image signal that can generate a reconstructed image focused on an arbitrary subject distance in each frame;
A generating step of generating a reconstructed image focused on the subject distance from the image signal, using specific information for specifying a subject distance to be focused;
A setting step in which the setting means sets a plurality of the specific information for specifying different subject distances for the image signal;
Subject distance by which the determining means satisfies a predetermined condition based on the in-focus state of the subject in the plurality of reconstructed images generated using the plurality of specific information set in the setting step A determination step for determining specific information for output identifying
An output step of outputting a reconstructed image generated using the output specific information;
Have
In the setting step, the setting means determines that the subject distance specified by the plurality of specific information for the second frame following the first frame is based on the plurality of specific information set for the first frame. About the second frame so as to be denser in a predetermined subject distance range including the subject distance specified by the output specific information determined for the first frame than the specified subject distance. A control method for an image processing apparatus, wherein the plurality of specific information is set.   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 thru | or 9.