JP6070061B2 - Image processing apparatus, photographing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, a photographing apparatus, and a program.

There is known a stereo imaging device that acquires a stereo image composed of a right-eye image and a left-eye image using two photographing optical systems.
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP-A-8-47001

  When playing back a stereo image in time series like a video, if the amount of parallax generated between the right-eye image and the left-eye image changes rapidly in time, the viewer can follow the change in the convergence angle. Without feeling uncomfortable.

  The image processing apparatus according to the first aspect of the present invention acquires a parallax amount indicating parallax for the same target in time series, calculates a temporal variation amount of the parallax amount, and a variation amount calculation And a moving image generation unit that generates a moving image with a reduced amount of change with respect to the target when the amount of change acquired by the unit is larger than a threshold.

  The image processing method according to the second aspect of the present invention includes a change amount calculation procedure for acquiring a disparity amount indicating disparity for the same target in time series, and calculating a temporal change amount of the disparity amount, and the change A moving image generation procedure for generating a moving image in which the amount of change with respect to the object is reduced at least when the amount of change acquired by the amount calculation procedure is larger than a threshold value;

  The program according to the second aspect of the present invention acquires a parallax amount indicating parallax for the same target in time series, calculates a temporal variation amount of the parallax amount, and calculates the variation amount When the amount of change acquired by the procedure is larger than a threshold, the computer is caused to execute a moving image generation procedure for generating a moving image in which the amount of change with respect to the target is reduced.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

1 shows a digital camera and a PC according to the present embodiment. The relationship between the amount of movement of the subject, the vergence angle of the viewer, and the amount of parallax on the display device is shown. It is a figure explaining the method to reduce the temporal variation | change_quantity of the amount of parallax. It is a figure explaining the structure of a digital camera. It is a conceptual diagram which represents notably the mode that a part of imaging device was expanded. It is a figure explaining the example of a production | generation process of 2D image data and parallax image data. It is a figure explaining the concept of defocusing. It is a figure which shows the light intensity distribution which a parallax pixel outputs. It is explanatory drawing explaining the production | generation process of a parallax map. It is a figure which shows the concept of the process which transplants primary color information with reference to a parallax map. The relationship between the aperture value, the contrast indicating the sharpness of an image, and the amount of parallax in a digital camera is schematically shown. It is a flowchart which shows operation | movement of a digital camera. It is a flowchart which shows the other operation | movement of a digital camera. 7 shows another example of generating color image data for the right eye and color image data for the left eye from the RAW image data set shown in FIG. 7 shows another example of generating color image data for the right eye and color image data for the left eye from the RAW image data set shown in FIG. It is a flowchart which shows the other operation | movement of a digital camera. It is a flowchart which shows the other operation | movement of a digital camera. The example of the moving image produced | generated by step S114 of FIG. 17 is shown. The functional block of the main body of PC is shown. It is a flowchart which shows operation | movement of a main body. An example of the correction | amendment moving image produced | generated by the flowchart of FIG. 20 is shown.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 shows a digital camera 10 and a personal computer 40 (sometimes referred to as a PC) according to this embodiment. The digital camera 10 is an example of an imaging device and an example of an image processing device that generates a moving image.

  The digital camera 10 and the PC 40 transmit and receive image data such as moving images via the connection cable 32, for example. The PC 40 includes a display 42, a main body 44, and a keyboard 46 that are connected to each other. The PC 40 is also an example of an image processing apparatus, and displays a moving image generated by the main body 44 on the display 42 based on input from the keyboard 46.

  FIG. 2 shows the relationship between the amount of movement of the subject, the convergence angle of the viewer 50 who is viewing the display, and the amount of parallax on the display 42. If the triangle in the figure is a real object, the viewer 50 recognizes the position L0 in the front-rear direction of the object based on the convergence angle θ0 generated by the parallax between the right eye 51 and the left eye 52 when the object is at the position L0. To do. Furthermore, the viewer 50 determines that when the object is at the position L0 and at the position L1, the convergence angles θ0 and θ1 are different, so that the object moves forward and backward along with other information such as a change in the size of the object. Recognize that it is moving in the direction.

  Further, the viewer 50 displays the right-eye image 55 directed toward the right eye 51 and the left-eye image 56 directed toward the left eye 52 on the display 42 in accordance with the convergence angle θ0 of the viewer 50. A stereoscopic effect in the front-rear direction of the object can be obtained in a pseudo manner. Here, the difference in position on the display 42 between the right-eye image 55 and the left-eye image 56 is an example of the parallax amount D0. The parallax amount D0 may be normalized by counting the difference in position by the number of pixels and dividing by the total number of pixels in the horizontal direction of the display 42.

  Further, a right-eye image 55 and a left-eye image 56 having a parallax amount D0 corresponding to the convergence angle θ0, and a right-eye image 57 and a left-eye image 58 having a parallax amount D1 corresponding to the convergence angle θ1 are animated in time series. As a result, the viewer 50 can obtain a stereoscopic effect in which the object is moved in the front-rear direction. Here, when the parallax amount D0 and the parallax amount D1 change suddenly, the viewer 50 cannot follow the change of the convergence angle, and a sense of incongruity occurs (this is sometimes called a contradiction between convergence and adjustment). Therefore, in the present embodiment, the digital camera 10 or the PC 40 generates a moving image with reduced parallax time change.

  Any method may be used for displaying the right-eye image 55 and the left-eye image 56 on the display 42. For example, it may be displayed in a time-division manner, an interlace in which strips are arranged horizontally or vertically, or a side-by-side arrangement on one side and the other side of the screen.

  FIG. 3 is a diagram illustrating a method for reducing the temporal change amount of the parallax amount. FIG. 3 shows an example in which the moving image C1 changes from the parallax amount D0 at time T0 to the parallax amount D1 at time T1.

  In order to reduce the temporal change amount of the parallax of the moving image C1, at least two methods are conceivable. The first is to generate the moving image C2 using the parallax amount D2 smaller than the parallax amount D1 at time T1. The second is to generate the moving image C3 at a time T2 that is later than the time T1 to reach the same parallax amount D1. The above two methods may be executed at the time of shooting a moving image, or may be executed at the time of display on a display device after shooting of a moving image. Each specific example will be described later.

  The digital camera 10 according to the present embodiment, which is a form of the image processing apparatus and the imaging apparatus, is configured to be able to generate an image with a plurality of viewpoints for one scene by one shooting. Each image having a different viewpoint is called a parallax image.

  FIG. 4 is a diagram illustrating the configuration of the digital camera 10. The digital camera 10 generates a plurality of parallax images in which subject light is sequentially captured at a specific frame rate in time series, and arranges frames based on each of the plurality of parallax images in time series to thereby generate parallax. Generate a video containing information.

  The digital camera 10 includes a photographic lens 20 as a photographic optical system, and guides a subject light beam incident along the optical axis 21 to the image sensor 100. The digital camera 10 further includes an aperture 22 that shields the periphery of the subject light flux. The taking lens 20 may be an interchangeable lens that can be attached to and detached from the digital camera 10 together with the aperture 22. The digital camera 10 includes an image sensor 100, a control unit 201, an A / D conversion circuit 202, a memory 203, a drive unit 204, an image processing unit 205, a memory card IF 207, an operation unit 208, a display unit 209, an LCD drive circuit 210, and an AF. A sensor 211 is provided.

  As shown in the figure, the direction parallel to the optical axis 21 toward the image sensor 100 is defined as the Z-axis plus direction, the direction toward the front of the drawing on the plane orthogonal to the Z-axis is the X-axis plus direction, and the upward direction on the drawing is Y. The axis is defined as the plus direction. In relation to the composition in photographing, the X axis is the horizontal direction and the Y axis is the vertical direction. In the following several figures, the coordinate axes are displayed so that the orientation of each figure can be understood with reference to the coordinate axes in FIG.

  The taking lens 20 is composed of a plurality of optical lens groups, and forms an image of a subject light flux from the scene in the vicinity of its focal plane. In FIG. 4, for convenience of explanation, the photographic lens 20 is represented by a single virtual lens arranged in the vicinity of the pupil.

  The aperture 22 is disposed in the vicinity of the pupil of the photographic lens 20 and shields the peripheral region of the subject light flux incident on the image sensor 100. The aperture amount of the aperture 22 is variable and is set by control from the control unit 201. An example of the diaphragm 22 is an iris diaphragm, but is not limited thereto.

  The image sensor 100 is disposed near the focal plane of the photographic lens 20. The image sensor 100 is an image sensor such as a CCD or CMOS sensor in which a plurality of photoelectric conversion elements are two-dimensionally arranged. The image sensor 100 is controlled in timing by the drive unit 204, converts the subject image formed on the light receiving surface into an image signal, and outputs the image signal to the A / D conversion circuit 202.

  The A / D conversion circuit 202 converts the image signal output from the image sensor 100 into a digital image signal and outputs the digital image signal to the memory 203. The image processing unit 205 performs various image processing using the memory 203 as a work space, and generates image data.

  The control unit 201 controls the digital camera 10 in an integrated manner. For example, the aperture of the diaphragm 22 is adjusted according to the set diaphragm value, and the photographing lens 20 is advanced and retracted in the optical axis direction according to the AF evaluation value. Further, the position of the photographing lens 20 is detected, and the focal length and the focus lens position of the photographing lens 20 are grasped. Furthermore, a timing control signal is transmitted to the drive unit 204, and a series of sequences until the image signal output from the image sensor 100 is processed into captured image data by the image processing unit 205 is managed.

  The image processing unit 205 acquires parallax image data for the same target in time series to calculate a parallax amount, a change amount calculation unit 232 that calculates a temporal change amount of the parallax amount, and When the change amount is larger than the threshold, the moving image generation unit 233 generates a moving image in which the change amount with respect to the target is reduced. Details of each processing will be described later.

  The image processing unit 205 also has general image processing functions such as adjusting image data according to the selected image format. The generated image data is converted into a display signal by the LCD drive circuit 210 and displayed on the display unit 209. The data is recorded on the memory card 220 attached to the memory card IF 207.

  The AF sensor 211 is a phase difference sensor in which a plurality of distance measuring points are set for the subject space, and detects the defocus amount of the subject image at each distance measuring point. A series of shooting sequences is started when the operation unit 208 receives a user operation and outputs an operation signal to the control unit 201. Various operations such as AF and AE accompanying the imaging sequence are executed under the control of the control unit 201. For example, the control unit 201 analyzes the detection signal of the AF sensor 211 and executes focus control for moving a focus lens that constitutes a part of the photographing lens 20.

  FIG. 5 is a conceptual diagram conceptually showing a state in which a part of the image sensor 100 is enlarged. In the pixel area, 20 million or more pixels are arranged in a matrix. In the present embodiment, 64 pixels of adjacent 8 pixels × 8 pixels form one basic lattice 110. The basic grid 110 includes four Bayer arrays having 4 × 2 × 2 basic units in the Y-axis direction and four in the X-axis direction. As shown in the figure, in the Bayer array, a green filter (G filter) is arranged for the upper left pixel and the lower right pixel, a blue filter (B filter) is arranged for the lower left pixel, and a red filter (R filter) is arranged for the upper right pixel.

  The basic grid 110 includes parallax pixels and non-parallax pixels. The parallax pixel is a pixel that receives a partial light beam that is deviated from the optical axis among incident light beams that pass through the photographing lens 20. The parallax pixel is provided with an aperture mask having a deviated opening that is deviated from the center of the pixel so as to transmit only the partial light flux. For example, the opening mask is provided so as to overlap the color filter. In the present embodiment, the parallax Lt pixel defined so that the partial light beam reaches the left side with respect to the pixel center and the parallax specified so that the partial light beam reaches the right side with respect to the pixel center by the aperture mask. There are two types of Rt pixels. On the other hand, the non-parallax pixel is a pixel that is not provided with an aperture mask, and is a pixel that receives the entire incident light beam that passes through the photographing lens 20.

  Note that the parallax pixel is not limited to the aperture mask when receiving the partial light beam that is deviated from the optical axis, but has various configurations such as a selective reflection film in which the light receiving region and the reflective region are separated, and a deviated photodiode region. Can be adopted. In other words, the parallax pixel only needs to be configured to receive a partial light beam that is deviated from the optical axis, among incident light beams that pass through the photographing lens 20.

Pixels in the basic grid 110 are denoted by _PIJ . For example, the upper left pixel is _{P 11,} the upper right pixel is _{P 81.} As shown in the figure, the parallax pixels are arranged as follows.
P ₁₁ : Parallax Lt pixel + G filter (= G (Lt))
P ₅₁ ... Parallax Rt pixel + G filter (= G (Rt))
P ₃₂ ... Parallax Lt pixel + B filter (= B (Lt))
P ₆₃ ... Parallax Rt pixel + R filter (= R (Rt))
P ₁₅ ... Parallax Rt pixel + G filter (= G (Rt))
P ₅₅ ... Parallax Lt pixel + G filter (= G (Lt))
P ₇₆ ... Parallax Rt pixel + B filter (= B (Rt))
P ₂₇ ... Parallax Lt pixel + R filter (= R (Lt))
The other pixels are non-parallax pixels, and are any of the non-parallax pixel + R filter, the non-parallax pixel + G filter, and the non-parallax pixel + B filter.

  When viewed as a whole of the image sensor 100, the parallax pixels are classified into one of a first group having a G filter, a second group having an R filter, and a third group having a B filter. Includes at least one parallax Lt pixel and parallax Rt pixel belonging to each group. As in the example in the figure, these parallax pixels and non-parallax pixels may be arranged with randomness in the basic lattice 110. By arranging with randomness, RGB color information can be acquired as the output of the parallax pixels without causing bias in the spatial resolution for each color component, so that high-quality parallax image data can be obtained. can get.

  Next, a concept of processing for generating a RAW image data set including 2D image data and parallax image data from captured image data output from the image sensor 100 will be described. FIG. 6 is a diagram illustrating an example of processing for generating 2D image data and parallax image data.

  As can be seen from the arrangement of parallax pixels and non-parallax pixels in the basic grid 110, image data representing a specific image is not obtained even if the output of the image sensor 100 is aligned with the pixel arrangement. Only when the pixel outputs of the image sensor 100 are separated and collected for each pixel group characterized in the same manner, image data representing one image in accordance with the characteristics is formed. For example, when the left and right parallax pixels are gathered together, left and right parallax image data having parallax can be obtained. In this way, each piece of image data separated and collected for each identically characterized pixel group is referred to as plane data.

  The image processing unit 205 receives raw raw image data in which output values (pixel values) are arranged in the order of pixel arrangement of the image sensor 100, and executes plane separation processing for separating the raw image data into a plurality of plane data. The left column of the figure shows an example of processing for generating 2D-RGB plane data as 2D image data.

In generating the 2D-RGB plane data, the image processing unit 205 first removes the pixel values of the parallax pixels to form an empty grid. Then, the pixel value that becomes the empty grid is calculated by interpolation processing using the pixel values of the surrounding pixels. For example, the pixel values of the vacancy _{P 11} is the pixel value of the G filter pixels adjacent in an oblique _{direction, P -1-1, P 2-1, P} -12, averages calculates the pixel values of _{P 22} To calculate. Further, for example, the pixel value of the empty lattice P ₆₃ is calculated by averaging the pixel values of P ₄₃ , P ₆₁ , P ₈₃ , and P ₆₅ that are adjacent R filter pixel values by skipping one pixel vertically and horizontally. To do. Similarly, for example, the pixel value of the air grating _{P 76} is the pixel value of the adjacent B filter skipping one pixel vertically and _{horizontally,} and averaging operation of the pixel values of _{P 56, P 74, P 96} , P 78 calculate.

  Since the 2D-RGB plane data interpolated in this way is the same as the output of a normal imaging device having a Bayer array, various processes can be performed as 2D image data thereafter. That is, known Bayer interpolation is performed to generate color image data in which RGB data is aligned for each pixel. The image processing unit 205 performs image processing as a general 2D image according to a predetermined format such as JPEG when generating still image data and MPEG when generating moving image data.

  In the present embodiment, the image processing unit 205 further separates the 2D-RGB plane data for each color, performs the above-described interpolation processing, and generates each plane data as reference image data. That is, three types of data are generated: Gn plane data as green reference image plane data, Rn plane data as red reference image plane data, and Bn plane data as blue reference image plane data.

  The right column of the figure shows an example of generation processing of two G plane data, two R plane data, and two B plane data as parallax pixel acquisition data. The two G plane data are GLt plane data as left parallax image data and GRt plane data as right parallax image data. The two R plane data are RLt plane data and right parallax image data as left parallax image data. The two B plane data are the BLt plane data as the left parallax image data and the BRt plane data as the right parallax image data.

In generating the GLt plane data, the image processing unit 205 removes pixel values other than the pixel values of the G (Lt) pixels from all output values of the image sensor 100 to form a vacant lattice. As a result, two pixel values P ₁₁ and P ₅₅ remain in the basic grid 110. Therefore, it divided into four equal basic grid 110 vertically and horizontally, the 16 pixels of the top left is represented by an output value of the P _11, is representative of the 16 pixels in the lower right in the output value of the P _55. Then, for the upper right 16 pixels and the lower left 16 pixels, average values of neighboring representative values adjacent in the vertical and horizontal directions are averaged and interpolated. That is, the GLt plane data has one value in units of 16 pixels.

Similarly, when generating the GRt plane data, the image processing unit 205 removes pixel values other than the pixel value of the G (Rt) pixel from all the output values of the image sensor 100 to obtain an empty grid. Then, two pixel values P ₅₁ and P ₁₅ remain in the basic grid 110. Therefore, the basic grid 110 is divided into four equal parts vertically and horizontally, the upper right 16 pixels are represented by the output value of P ₅₁ , and the lower left 16 pixels are represented by the output value of P ₁₅ . The upper left 16 pixels and the lower right 16 pixels are interpolated by averaging the peripheral representative values adjacent vertically and horizontally. That is, the GRt plane data has one value in units of 16 pixels. In this way, it is possible to generate GLt plane data and GRt plane data having a resolution lower than that of 2D-RGB plane data.

In generating the RLt plane data, the image processing unit 205 removes pixel values other than the pixel value of the R (Lt) pixel from all output values of the image sensor 100 to form a vacant lattice. Then, the primitive lattice _110, the pixel values of _{P 27} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. Similarly, when generating the RRt plane data, the image processing unit 205 removes pixel values other than the pixel value of the R (Rt) pixel from all output values of the image sensor 100 to form a vacant lattice. Then, the pixel value P ₆₃ remains in the basic grid 110. This pixel value is set as a representative value for 64 pixels of the basic grid 110. In this way, RLt plane data and RRt plane data having a resolution lower than that of 2D-RGB plane data are generated. In this case, the resolution of the RLt plane data and the RRt plane data is lower than the resolution of the GLt plane data and the GRt plane data.

In generating the BLt plane data, the image processing unit 205 removes pixel values other than the pixel values of the B (Lt) pixels from all output values of the image sensor 100 to form a vacant lattice. Then, the primitive lattice _110, the pixel values of _{P 32} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. Similarly, when generating the BRt plane data, the image processing unit 205 removes pixel values other than the pixel value of the B (Rt) pixel from all the output values of the image sensor 100 to obtain an empty grid. Then, the primitive lattice _110, the pixel values of _{P 76} remains. This pixel value is set as a representative value for 64 pixels of the basic grid 110. In this way, BLt plane data and BRt plane data having a resolution lower than that of 2D-RGB plane data are generated. In this case, the resolution of the BLt plane data and the BRt plane data is lower than the resolution of the GLt plane data and the GRt plane data, and is equal to the resolution of the RLt plane data and the RRt plane data.

  In the present embodiment, the image processing unit 205 uses these plane data to generate left-viewpoint color image data and right-viewpoint color image data. The plane data may be arranged in time series to form a moving image, or images of each color in which primary color information is transplanted using a parallax map described later may be arranged in time series to form a moving image.

  FIG. 7 is a diagram for explaining the concept of defocusing. The parallax Lt pixel and the parallax Rt pixel receive the subject luminous flux that arrives from one of the two parallax virtual pupils set as the optical axis target as a partial region of the lens pupil. In the optical system of the present embodiment, since the actual subject light flux passes through the entire lens pupil, the light intensity distributions corresponding to the parallax virtual pupil are not distinguished from each other until the parallax pixel is reached. However, the parallax pixel outputs an image signal obtained by photoelectrically converting only the partial light flux that has passed through the parallax virtual pupil by the action of the aperture mask that each has. Therefore, the pixel value distribution indicated by the output of the parallax pixel may be considered to be proportional to the light intensity distribution of the partial light flux that has passed through the corresponding parallax virtual pupil.

  As shown in FIG. 7A, when an object point that is a subject exists at the focal position, the output of each parallax pixel is the corresponding image point regardless of the subject luminous flux that has passed through any parallax virtual pupil. This shows a steep pixel value distribution centering on this pixel. If the parallax Lt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Further, even when the parallax Rt pixels are arranged in the vicinity of the image point, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. That is, even if the subject luminous flux passes through any parallax virtual pupil, the output value of the pixel corresponding to the image point is the largest, and the output value of the pixels arranged in the vicinity rapidly decreases. Match each other.

  On the other hand, as shown in FIG. 7B, when the object point deviates from the focal position, the peak of the pixel value distribution indicated by the parallax Lt pixel corresponds to the image point, compared to the case where the object point exists at the focal position. Appearing at a position away from the pixel in one direction, and its output value decreases. In addition, the width of the pixel having the output value is increased. The peak of the pixel value distribution indicated by the parallax Rt pixel appears at a position away from the pixel corresponding to the image point in the opposite direction to the one direction in the parallax Lt pixel and at an equal distance, and the output value similarly decreases. Similarly, the width of the pixel having the output value is increased. That is, the same pixel value distribution that is gentler than that in the case where the object point exists at the focal position appears at an equal distance from each other. Further, as shown in FIG. 7C, when the object point further deviates from the focal position, the same pixel value distribution that is more gentle than the state of FIG. 7B appears more distantly. . That is, it can be said that the amount of blur and the amount of parallax increase as the object point deviates from the focal position. In other words, the amount of blur and the amount of parallax change in conjunction with defocus. That is, the amount of blur and the amount of parallax have a one-to-one relationship.

  FIGS. 7B and 7C show the case where the object point shifts away from the focal position. When the object point moves away from the focal position, as shown in FIG. Compared to FIGS. 7B and 7C, the relative positional relationship between the pixel value distribution indicated by the parallax Lt pixel and the pixel value distribution indicated by the parallax Rt pixel is reversed. Due to such a defocus relationship, when viewing a parallax image, the viewer visually recognizes a subject existing far behind the focal position and visually recognizes a subject present in front.

  When the change of the pixel value distribution described in FIGS. 7B and 7C is graphed, it is expressed as shown in FIG. In the figure, the horizontal axis represents the pixel position, and the center position is the pixel position corresponding to the image point. The vertical axis represents the output value (pixel value) of each pixel. As described above, this output value is substantially proportional to the light intensity.

  A distribution curve 1804 and a distribution curve 1805 represent the pixel value distribution of the parallax Lt pixel and the pixel value distribution of the parallax Rt pixel in FIG. As can be seen from the figure, these distributions have a line-symmetric shape with respect to the center position. Further, a combined distribution curve 1806 obtained by adding them shows a pixel value distribution of pixels with no parallax with respect to the situation of FIG. 7B, that is, a pixel value distribution when the entire subject luminous flux is received, and a substantially similar shape.

  A distribution curve 1807 and a distribution curve 1808 represent the pixel value distribution of the parallax Lt pixel and the pixel value distribution of the parallax Rt pixel in FIG. As can be seen from the figure, these distributions are also symmetrical with respect to the center position. Further, a combined distribution curve 1809 obtained by adding them shows a shape that is substantially similar to the pixel value distribution of the non-parallax pixels in the situation of FIG.

  FIG. 9 is an explanatory diagram illustrating a process for generating a parallax map. In the example illustrated in FIG. 9, the parallax amount calculation unit 231 performs the following parallax map generation processing based on the GLt plane data and the GRt plane data.

  The parallax amount calculation unit 231 determines a target pixel 311 having a pixel value among the pixels included in the target block, and determines a local window 312 for the target pixel 311. Then, matching processing is performed between the two images using the local window 312 as a reference, and the amount of parallax pixels in the target block including the target pixel 311 is determined. Specifically, the parallax amount calculation unit 231 sets the local window 314 on the GRt plane image indicated by the GRt plane data corresponding to the local window 312 on the Lt plane image, and changes the local window 314 to the local window 312. On the other hand, image regions having good matching with each other are searched while being relatively shifted. Then, the position of the local window 314 where matching is determined to be good is determined, and the coordinate value of the search pixel 313 that is the center coordinate is calculated. The amount of parallax pixels is determined by converting the difference between the coordinate value of the target pixel 311 and the coordinate value of the search pixel 313 in units of parallax blocks. Then, the parallax amount calculation unit 231 sequentially executes the above matching process while sequentially scanning the target pixel 311 from the upper left to the lower right on the GLt plane image, and calculates the parallax pixel amounts of the GLt plane image and the GRt plane image. To do.

  Then, the parallax amount calculation unit 231 completes the parallax map with each of the calculated parallax pixel amounts as a half value in order to use the 2D image as a reference image. The parallax map is expressed as a parallax pixel amount corresponding to each subject area of the boy 301, the girl 302, the woman 303, and the background 304, which are the subjects, for example, in the example of the figure, as the above calculation result. In the example in the figure, the boy 301 is in focus, the parallax pixel amount is 0 in the area of the boy 301, -1 in the area of the girl 302 existing in front of the boy 301, and the area of the woman 303 existing in the back. In this case, the value is +1, and the background in the far distance is +4.

  If image matching processing is performed between the GLt plane data and the GRt plane data, the amount of parallax pixels in each parallax block can be calculated, so which pixel block in one parallax image corresponds to which in the other parallax image You can see if it corresponds to a pixel block. That is, it is possible to grasp the relative positional relationship between the pixel blocks capturing the same minute area of the subject between both parallax images. Further, as described above, the parallax pixel amount with respect to the reference image is half of the parallax pixel amount between the parallax images. Therefore, it is possible to enumerate how many blocks the GLt plane image indicated by the GLt plane data is shifted from the non-parallax image over all blocks, and the enumerated number sequence forms a so-called parallax map.

  FIG. 10 is a diagram illustrating a concept of processing for transplanting primary color information with reference to a parallax map. As described above, when the parallax map is generated, the moving image generation unit 233 refers to the parallax map and generates a frame of color image data including the parallax information.

The color image data that is the reference image has pixel values corresponding to each of the RGB filters as primary color information in each pixel block such as the Rn plane data shown in FIG. When n and m are natural numbers, primary color information for the pixel block (n, m) is represented as C _nm . Then, the moving image generation unit 233 refers to the parallax map as primary color information of the pixel block (n, m) in the parallax image data, and primary color information C _n of the pixel block (n ′, m ′) such as Rn plane data. transplant _'m' . The moving image generation unit 233 repeats this operation for all blocks, and generates a frame of color image data including disparity information.

This will be specifically described with reference to the example in the figure. When acquiring the primary color information of the pixel block (1, 1) of the left parallax image data, the pixel block (1, 1) of the parallax map is referred to. At this time, since the pixel block (1, 1) of the parallax map indicates the parallax pixel amount 0, the moving image generation unit 233 searches the pixel block (1, 1) of the 2D image data as the corresponding pixel block as it is. . The moving image generation unit 233, a primary color information _{C 11} pixel block of the 2D image data (1, 1), to obtain the primary information by transplanting into pixel blocks (1,1) of the left parallax image data. Moving image generation unit 233, since the parallax pixel amount is 0, the same applies to the pixel blocks (1,1) of the right parallax image data, transplanted primary information _{C 11} pixel block of the 2D image data (1,1) To do.

Next, when acquiring the primary color information of the pixel block (1, 2) of the left parallax image data, the pixel block (1, 2) of the parallax map is referred to. At this time, since the pixel block (1, 2) of the parallax map indicates the parallax pixel amount 1, the moving image generation unit 233 is the pixel block on the right side of the corresponding pixel block (1, 2) of the color image data. Search for (1,3). The moving image generation unit 233, a primary color information _{C 13} pixel block of the 2D image data (1,3), to obtain the primary information by transplanting into the left parallax image data pixel blocks (1, 2).

Subsequently, the moving image generation unit 233 acquires primary color information of the pixel block (1, 2) of the right parallax image data. The parallax pixel amount of the pixel block (1, 2) of the referenced parallax map is 1, but when the primary color information is transplanted to the right parallax pixel data, the sign is reversed. And Therefore, the moving image generation unit 233 searches for the pixel block (1, 1) that is one left side of the corresponding pixel block (1, 2) of the 2D image data. The moving image generation unit 233, a primary color information _{C 11} pixel block (1,1) of the 2D image data, and acquires the primary information by transplanting into the right parallax image data pixel blocks (1, 2).

That is, the moving image generation unit 233 applies the primary color information C _{nm + k} of the 2D image data to the pixel block (n, m) of the left parallax image data and the pixel block ( n, m), the primary color information C _nm-k of 2D image data is transplanted, and the respective parallax image data primary color information is acquired.

  FIG. 11 schematically shows the relationship between the aperture value, the contrast indicating the sharpness of the image, and the amount of parallax in the digital camera 10. In the digital camera 10 of FIGS. 4 to 10, as in the case of a digital camera that obtains a 2D image without parallax information, as the aperture value is larger, the position before and after the focus position increases as shown in FIGS. An image having a high contrast and a deep so-called depth of field is obtained.

  Furthermore, in the digital camera 10 of the present embodiment, as shown in FIGS. 11A to 11C, the larger the aperture value, the smaller the amount of parallax before and after the focus position becomes. This is because the parallax pixels of the image sensor 100 of the digital camera 10 receive a light beam from a biased position in the exit pupil, as shown in FIG. This is based on the fact that the light flux from the biased side is blocked. That is, comparing FIGS. 11A to 11C, the amount of parallax before and after the focus position and the amount of parallax after the focus position are increased as the F value becomes 4 and 8, compared with the case where the F value of the diaphragm 22 is 1.4. The amount becomes smaller.

  FIG. 12 is a flowchart showing the operation of the digital camera 10. The flowchart of FIG. 12 is an example of generating a moving image by reducing the amount of temporal change in parallax by reducing the amount of parallax using the relationship between the aperture value and the amount of parallax in FIG. The flowchart shown in FIG. 12 starts when an instruction to generate a moving image is input to the digital camera 10 by the user.

  The control unit 201 sets the aperture value of the aperture 22 to an initial value (S100). The initial value of the aperture value may be an open value that is a minimum value, or may be determined from other shooting conditions such as the amount of light of a subject light beam.

  The control unit 201 drives the image sensor 100 via the drive unit 204 to generate a temporary image obtained by imaging subject light with the aperture value (S102). Here, the temporary image is an image used to calculate the amount of parallax, and an example thereof is a RAW image data set obtained by the operations and processes shown in FIGS. . Note that the processing may be speeded up by generating the temporary image by a method separate from the main image. Examples of separate methods include thinning out reading, omission of interpolation processing, and the like.

  The parallax amount calculation unit 231 calculates the parallax amount from the temporary image (S104). In this case, the parallax amount calculation unit 231 performs the process described in FIG. 9, for example, and calculates the parallax pixel amount by generating a parallax map.

  The change amount calculation unit 232 compares the amount of parallax calculated in step S104 with the amount of parallax calculated in step S104 with respect to the temporary image previously generated in time series, and compares the amount of parallax over time. The amount of change is calculated (S106). In this case, for example, the change amount calculation unit 232 calculates the change amount by dividing the amount of parallax pixels of the same subject in the two parallax maps with respect to the temporal images that are temporally changed by the time interval indicated by the frame rate.

  The moving image generation unit 233 determines whether or not the amount of change calculated in step S106 is greater than a threshold (S108). In this case, for example, when the amount of change calculated in step S106 is different for each subject, the moving image generation unit 233 compares the largest amount of change with a threshold value. The threshold value may be set in advance based on whether or not the viewer 50 feels uncomfortable when displaying the moving image generated with the initial aperture setting value, and industry guidelines for a display that makes the viewer feel a three-dimensional effect. Etc. may be set based on the above.

  When the amount of change is larger than the threshold value in step S108 (S108: Yes), the moving image generating unit 233 inputs to the control unit 201 that the aperture value is increased (S110). In this case, the control unit 201 may set the aperture value with reference to a table in which the aperture value is set in accordance with the degree of deviation from the threshold, or one step from the initial value regardless of the degree of deviation. Alternatively, the aperture value may be increased by several steps.

  In step S <b> 110, when the control unit 201 receives a change in aperture value from the moving image generation unit 233, the control unit 201 resets other imaging conditions such as the accumulation time and sensitivity of the photoelectric conversion element 108. As a result, unnaturalness such as a sudden change in the brightness of the image before and after the aperture value change can be alleviated.

  Subsequent to step S110 or when the amount of change is equal to or smaller than the threshold value in step S108 (S108: No), the control unit 201 drives the image sensor 100 via the drive unit 204 to capture the subject light. An image is generated (S112). The moving image generation unit 233 generates an image for one frame having parallax information based on the main image, and generates a moving image having parallax information by temporally arranging the frames with the previous frame (S114). In this case, for example, the moving image generating unit 233 generates an image for one frame including the right-eye image and the left-eye image from the main image by using a set of primary color plane data shown in FIG. Instead, the moving image generation unit 233 obtains a parallax map by the above-described processing of the parallax amount calculation unit 231, and one frame including the right-eye image and the left-eye image by the processing of FIG. 10 using the parallax map. Minute images may be generated.

  Unless an instruction for ending the moving image is input from the user (S116: No), the processes of steps S100 to S114 are repeated. When an instruction to end the moving image is input from the user (S116: Yes), the flowchart ends. As a result, the moving image generation unit 233 generates and generates time-series images in a state where the aperture value is larger than when the amount of change is smaller than the threshold when the amount of temporal change in parallax is larger than the threshold. A moving image is generated from the plurality of images. Therefore, the moving image generation unit 233 can generate a moving image with a reduced amount of temporal change in parallax, and can reduce discomfort when the viewer 50 views the moving image.

  FIG. 13 is a flowchart showing another operation of the digital camera 10. The flowchart of FIG. 13 is an example of generating a moving image by reducing the amount of temporal change in parallax by reducing the amount of parallax using the parallax maps of FIGS. 9 and 10. In the flowchart of FIG. 13, the same operations as those in the flowchart of FIG.

  In the flowchart of FIG. 13, the control unit 201 sets an imaging condition such as an aperture value and drives the image sensor 100 to generate an image of subject light (S130). Based on the image, the parallax amount calculation unit 231 performs the processing described with reference to FIG. 9 to generate a parallax map (S132). Subsequent steps S106 and S108 are the same as those in the flowchart of FIG.

  In step S108, when the amount of change is larger than the threshold (S108: Yes), the moving image generating unit 233 corrects the parallax map (S134). For example, the moving image generation unit 233 corrects the parallax map so that the parallax pixel amount of the subject corresponding to the change amount exceeding the threshold value is reduced to the threshold value or less.

  The moving image generation unit 233 uses the corrected parallax map when the parallax map is corrected in step S134, and corrects when the change amount is determined to be equal to or less than the threshold value in step S108 (S108: No). An image for one frame including the right-eye image and the left-eye image is generated by the processing of FIG.

  Thereby, the moving image generation unit 233 generates a moving image in which the change amount is reduced to the threshold value or less when the temporal change amount of the parallax is larger than the threshold value. Therefore, the moving image generation unit 233 can generate a moving image with a reduced amount of temporal change in parallax, and can reduce discomfort when the viewer 50 views the moving image.

  14 and 15 show another example of generating color image data for the right eye and color image data for the left eye from the RAW image data set shown in FIG. In particular, in the examples of FIGS. 14 and 15, color image data in which the parallax amount is arbitrarily adjusted within a range in which the main subject is not blurred is generated by introducing the stereoscopic adjustment parameter.

  FIG. 14 illustrates an example of a process for generating red right parallax image data and red left parallax image data among color image data.

The red parallax image data is generated using the pixel value of the Rn plane data described with reference to FIG. 6, and the luminance value of the RLt plane data and the RRt plane data. Specifically, for example, when calculating the pixel value RLt _mn of the target pixel position (i _m , j _n ) of the left parallax image data, first, the moving image generation unit 233 first determines the same pixel position (i _m , The pixel value Rn _mn is extracted from j _n ). Next, moving image generation unit 233, the same pixel position of RLt plane data _(i m, _{j n)} the luminance value _{RLt mn} from the same pixel position of the RRT plane data _(i m, _{j n)} the luminance value _{RRT mn} from Extract. The moving image generation unit 233, the pixel values _{Rn mn,} by multiplying the value obtained by distributing the luminance value _{RLt mn} and _{RRT mn} stereoscopic adjustment parameter C, and calculates the pixel value RLct _mn. Specifically, it is calculated by the following equation (1). However, the three-dimensional adjustment parameter C is preset in the range of 0.5 <C <1.
RLct _mn = 2Rn _mn x {C · RLt _mn + (1-C) · RRt _mn } / (RLt _mn + RRt _mn ) (1)

Similarly, when calculating the pixel value RRct _mn of the target pixel position (i _m , j _n ) of the right parallax image data, the moving image generation unit 233 uses the luminance value RLt _mn and the luminance value RRt _mn as the pixel value Rn _mn. Calculated by multiplying the value distributed by the three-dimensional adjustment parameter C. Specifically, it is calculated by the following equation (2).
RRct _mn = 2Rn _mn × {C · RRt _mn + (1-C) · RLt _mn } / (RLt _mn + RRt _mn ) (2)

Moving image generation unit 233, such processing is sequentially executed from the pixel at the left end and the upper end (1,1) to a right end and the lower end of the coordinate (i _{0, j 0).}

  When the generation process of the red right parallax image data and the left parallax image data is completed, the green right parallax image data and the left parallax image data, and the blue right parallax image data and the left parallax image data are similarly processed. Generate. Through the above processing, one frame of color image data including parallax information is completed. Further, the moving image generation unit 233 generates a moving image by arranging the frames in time series.

  FIG. 15 is a diagram for explaining a change in RGB pixel value distribution. FIG. 15A shows a G (Lt) pixel, a G (Rt) pixel, and an R (Lt) pixel when a white subject light beam from an object point located at a position deviated by a certain amount from the focal position is received. , R (Rt) pixels, B (Lt) pixels, and B (Rt) pixels.

  FIG. 15B shows the R (N) pixel, G (N) pixel, and B (N) pixel that are non-parallax pixels when a white subject light beam from the object point in FIG. 15A is received. It is the graph which arranged the output value. It can be said that this graph also represents the pixel value distribution of each color.

  When the above-described processing is performed for each corresponding pixel with C = 0.8, the light intensity distribution represented by the graph of FIG. As can be seen from FIG. 15 and the above formulas (1) and (2), the parallax amount is larger as the three-dimensional adjustment parameter C is closer to 1 in the range of 0.5 <C <1, and the parallax amount is smaller as it is closer to 0.5. Become. That is, the three-dimensional adjustment parameter C includes the parallax amount indicated by the parallax Lt pixel and the parallax Rt pixel when the parallax amount based on the parallax amount indicated by the parallax Lt pixel and the parallax amount indicated by the parallax Rt pixel is assigned to the non-parallax pixel. The weight with the amount of parallax shown is adjusted. One of the weights of the parallax amount indicated by the parallax Lt pixel and the parallax amount indicated by the parallax Rt pixel is larger as the stereoscopic adjustment parameter C is closer to 1, and the parallax amount closer to 0.5 is closer to 0.5. Assigned to a pixel.

  FIG. 16 is a flowchart showing another operation of the digital camera 10. The flowchart in FIG. 16 is an example of generating a moving image by reducing the amount of temporal change in parallax by reducing the amount of parallax using the stereoscopic adjustment parameter C in FIGS. 14 and 15. In the flowchart of FIG. 16, the same operations as those in the flowcharts of FIGS. 12 and 13 are denoted by the same reference numerals, and the description thereof is omitted.

  In the flowchart of FIG. 16, based on the image generated in step S <b> 130, the parallax amount calculation unit 231 performs the process described in FIG. 9, for example, and calculates the parallax amount (S <b> 104). Subsequent steps S106 and S108 are the same as those in the flowchart of FIG.

  In step S108, when the amount of change is larger than the threshold (S108: Yes), the moving image generating unit 233 corrects the three-dimensional adjustment parameter C (S142). In this case, the moving image generation unit 233 uses a predetermined three-dimensional adjustment parameter C that is closer to 0.5, for example, for a subject corresponding to an amount of change that exceeds a threshold. For example, the stereoscopic adjustment parameter C before correction is set to 1, and the corrected stereoscopic adjustment parameter C is set to 0.75.

  When the stereoscopic adjustment parameter C is corrected in step S134, the moving image generation unit 233 uses the corrected stereoscopic adjustment parameter C and determines that the change amount is equal to or less than the threshold value in step S108 (S108: No). ) Is used to generate an image for one frame including the image for the right eye and the image for the left eye by using the uncorrected stereo adjustment parameters by the processing of FIGS. 14 and 15.

  Thereby, the moving image generation unit 233 generates a moving image in which the change amount is reduced to the threshold value or less when the temporal change amount of the parallax is larger than the threshold value. Therefore, the moving image generation unit 233 can generate a moving image with a reduced amount of temporal change in parallax, and can reduce discomfort when the viewer 50 views the moving image.

  Here, since the temporal change amount of the parallax is reduced by the stereoscopic adjustment parameter C, it is possible to adjust the size of the parallax amount while maintaining the blur amount of the 2D color image due to the non-parallax pixels. Therefore, if these image data are reproduced by a 3D image-compatible reproducing apparatus, the viewer 50 can appreciate a 3D image in which the stereoscopic effect is appropriately adjusted as a color image. In particular, processing is easy and image data can be generated at high speed, which is suitable for moving images.

  FIG. 17 is a flowchart showing another operation of the digital camera 10. The flowchart in FIG. 17 is an example of generating a moving image by reducing the amount of temporal change in parallax by increasing the frame rate at the time of shooting and delaying the time until a certain amount of parallax is displayed. In the flowchart of FIG. 17, the same operations as those in the flowchart of FIG.

  The control unit 201 sets the frame rate to an initial value (S120). The initial value of the aperture value may be a predetermined constant value, or may be determined from other shooting conditions such as the light amount of the subject light flux.

  The control unit 201 drives the image sensor 100 via the drive unit 204 to generate a temporary image at the frame rate (S102). Further, as in FIG. 12, the amount of parallax and the amount of change are calculated, and the amount of change is compared with a threshold (S104, S106, S108).

  When the amount of change is larger than the threshold value in step S108 (S108: Yes), the moving image generation unit 233 inputs to the control unit 201 that the frame rate is increased (S122). In this case, the control unit 201 may set the frame rate by referring to a table in which the frame rate is set in accordance with the degree of deviation from the threshold, or one step for the initial value regardless of the degree of deviation. Alternatively, the frame rate may be increased by several stages.

  In step S <b> 110, when the control unit 201 receives a change in the frame rate from the moving image generation unit 233, the control unit 201 resets other imaging conditions such as the accumulation time and sensitivity of the photoelectric conversion element 108. Thereby, unnaturalness such as a sudden change in the brightness of the image before and after the change of the frame rate can be alleviated.

  Subsequent to step S110 or when the amount of change is equal to or less than the threshold value in step S108 (S108: No), the control unit 201 generates a main image at the frame rate set at that time (S112), and disparity information (S114). Thereby, the moving image generation unit 233, when the amount of temporal change in parallax is larger than the threshold, generates a moving image from a plurality of images captured and generated at a higher frame rate than when the amount of change is smaller than the threshold. Is generated.

  FIG. 18 shows an example of the moving image 330 and the corrected moving image 340 generated in step S114 of FIG. In the example of FIG. 18, it is assumed that the amount of change in parallax at time t2 exceeds the threshold value. In this case, frames 332 corresponding to a frame rate of 120 fps, which are higher than the frame rate of 60 fps at other times t1 and t3 below the threshold, are generated.

  The moving image generation unit 233 generates a corrected moving image 340 using a frame 342 obtained by converting a frame 332 at a high frame rate of 120 fps to a low frame rate of 60 fps. Thereby, the time t2 when the amount of change in parallax exceeds the threshold value is converted into a long time (t5 + t6). By reproducing the corrected corrected video 340 as usual on the PC 40 or the like, so-called slow reproduction is performed at times t5 and t6, which is longer than the actual time t2 at the time of shooting, and the temporal change in parallax is reduced. By reducing the temporal change amount of the parallax, it is possible to reduce a sense of discomfort when the viewer 50 views the moving image.

  The moving image generation unit 233 may generate a moving image 330 that reflects the frame rate at the time of shooting instead of generating the corrected moving image 340 that matches the frame rate. In this case, the moving image generation unit 233 further displays information indicating that the frame rate at time t2 is higher than the other times t1 and t3, or that the amount of change in parallax at time t2 exceeds the threshold value. It is preferable to store in the header portion 330 or the like. Thereby, a display device such as the PC 40 can reproduce the corrected moving image 340 with reference to the header information of the moving image 330 to reduce the temporal change amount of the parallax. Further, it is possible to perform reproduction corresponding to real time without reducing the amount of change in parallax according to the instruction of the viewer 50 or the like.

  FIG. 19 shows functional blocks of the main body 44 of the PC 40. The main body 44 includes a moving image acquisition unit 410, a change amount calculation unit 412, and a moving image correction unit 414. The main body 44 displays a moving image using the display 42.

  The moving image acquisition unit 410 acquires a moving image from the digital camera 10 or the like. The moving image acquisition unit 410 may acquire the moving image from a memory or a network server instead of directly communicating with the digital camera 10 to acquire the moving image.

  Based on the moving image acquired by the moving image acquisition unit 410, the change amount calculation unit 412 acquires a parallax amount indicating the parallax for the same target in time series, and calculates a temporal change in the parallax amount. When the change amount acquired by the change amount calculation unit 412 is larger than the threshold, the moving image correction unit 414 generates a corrected moving image in which at least the change amount for the target is reduced with respect to the original moving image.

  FIG. 20 is a flowchart showing the operation of the main body 44. The flowchart in FIG. 20 illustrates an example in which the moving image on which the moving image is displayed is corrected and displayed on the side where the temporal change in parallax is reduced. In the flowchart of FIG. 20, the same operations as those in FIGS. 12, 21, and 24 are denoted by the same reference numerals, and the description thereof is omitted.

  In the flowchart of FIG. 16, the moving image acquisition unit 410 acquires an original moving image including at least stereoscopic information from the digital camera 10 or the like (S150). The stereoscopic information may be a pair of parallax image data as shown in FIGS. 10 and 14, or may be information related to the parallax amount itself as shown in FIG. 9.

  The change amount calculation unit 412 calculates the amount of parallax based on the original moving image acquired by the moving image acquisition unit 410 (S104). When the information indicating the parallax amount is directly included in the original moving image acquired by the moving image acquiring unit 410, the change amount calculating unit 412 acquires the information. On the other hand, when the information indicating the parallax amount is not directly included in the original moving image acquired by the moving image acquiring unit 410, the change amount calculating unit 412 performs the disparity map illustrated in FIG. 9 based on each frame of the original moving image. For example, the amount of parallax is calculated. The change amount calculation unit 412 further compares the amount of parallax in step S104 between a plurality of time-series frames, and calculates a temporal change in the amount of parallax for the same target (S106).

  In step S108, when the amount of change is larger than the threshold (S108: Yes), the moving image correction unit 414 corrects the parallax amount (S152). In this case, the moving image correction unit 414 may reduce the amount of parallax in each frame with respect to the target, or may delay the time until the amount of change changes by the same amount of parallax.

  When reducing the amount of parallax in each frame, for example, when the original moving image uses the parallax map shown in FIG. 9, the moving image correction unit 414 reduces the amount of parallax in the parallax map. Instead of this, or in addition to this, the moving image correction unit 414 sets the stereoscopic adjustment parameter C in FIG. 14 between 1 and 0.5, for example, 0.75, for a pair of parallax image data of the original moving image. A frame with a reduced amount of parallax may be generated by generating a new pair of parallax image data.

  The moving image correcting unit 414 generates a moving image using the corrected frame in which the amount of parallax has been reduced in step S152 (S154). In this case, the moving image correction unit 414 uses the frame that does not exceed the threshold value in step S108 as it is.

  The moving image correction unit 414 displays the corrected moving image generated in step S154 in accordance with the display method of the display 42. This is the end of the flowchart of FIG. 20 may be executed by a computer program installed in the main body 44 via a medium such as the CD-ROM 416 or a network.

  FIG. 21 shows an example of the corrected moving image 360 generated by the flowchart of FIG. The moving image correcting unit 414 may generate the corrected moving image 360 corrected in step S152 so that the time until the change by the same amount of parallax is delayed.

  In this case, for example, in the original video 350, it is assumed that the change in the amount of parallax at time t12 is larger than the threshold value, and is smaller than the threshold value at times t11 and t13. In this case, the moving image correction unit 414 allocates each frame 352 at time t12 of the original moving image 350 as a frame 362 at twice the time (t15 + t16) in the corrected moving image 360. Furthermore, the moving image correction unit 414 generates the interpolation frame 364 so as to maintain the same frame rate as other times t14 and the like, for example, 60 fps. An example of the interpolation frame 364 is to copy the immediately preceding frame 362 as it is, but instead of this, the interpolation frame 364 may be generated by another method.

  By reproducing the corrected moving image 360 from the time t14 to the time t17 at the same frame rate, the time t12 when the amount of change in parallax in the original moving image 350 is enlarged is doubled at the time of reproduction, and the time (t15 + t16) is obtained. Played. Thereby, by reproducing the time t12 when the change amount of the parallax is large with a reproduction time longer than that of the original moving image 350, the change amount of the parallax per time can be reduced and displayed on the display 42.

  As described above, according to the embodiment shown in FIGS. 1 to 21, by reducing the parallax with respect to a moving image with a large change in parallax, it is possible to reduce discomfort when the viewer 50 views the moving image. Can do. Although the digital camera 10 and the PC 40 are shown in FIG. 1, the present embodiment may be applied to at least one of them.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

10 digital camera, 20 taking lens, 21 optical axis, 22 aperture, 32 connection cable, 40 personal computer, 42 display, 44 main body, 46 keyboard, 50 viewer, 51 right eye, 52 left eye, 55 right eye image, 56 left eye Image, 57 Right eye image, 58 Left eye image, 100 Image sensor, 110 Basic grid, 201 Control unit, 202 A / D conversion circuit, 203 Memory, 204 Drive unit, 205 Image processing unit, 207 Memory card IF, 208 Operation Part, 209 display part, 210 LCD drive circuit, 211 AF sensor, 220 memory card, 231 parallax amount calculation part, 232 change amount calculation part, 233 video generation part, 301 boy, 302 girl, 303 woman, 304 background, 311 target Pixel, 312 local window, 31 4 Local window, 313 search pixel, 410 moving image acquisition unit, 412 change amount calculating unit, 414 moving image correction unit, 416 CD-ROM, 330 moving image, 332 frame, 340 corrected moving image, 342 frame, 350 original moving image, 352 frame, 360 Corrected video, 362 frames, 364 interpolation frames, 1804 distribution curve, 1805 distribution curve, 1806 composite distribution curve, 1807 distribution curve, 1808 distribution curve, 1809 composite distribution curve

Claims (11)

A change amount calculation unit that obtains a disparity amount indicating disparity with respect to the same object in time series, and calculates a temporal change amount of the disparity amount; and
A moving image generating unit that generates a moving image with a reduced amount of change for at least the object when the amount of change acquired by the change amount calculating unit is greater than a threshold ;
An image for obtaining a reference image, a first parallax image having a first parallax with respect to the reference image, and a second parallax image having a second parallax in a direction opposite to the first parallax with respect to the reference image. for example Bei an acquisition unit to <br/>,
The moving image generation unit assigns a parallax amount based on the parallax amount indicated by the first parallax image and the parallax amount indicated by the second parallax image to the reference image for each corresponding pixel. To generate the video based on
The moving image generation unit weights the parallax amount indicated by the first parallax image when the change amount acquired by the change amount calculation unit is larger than a threshold value, compared to when the change amount is smaller than the threshold value. An image processing apparatus that assigns the parallax amount, which makes the weight of the parallax amount indicated by the second parallax image closer, to the reference image .   The image processing apparatus according to claim 1, wherein the moving image generation unit generates the moving image in which the amount of change is reduced to the threshold value or less. The moving image generation unit generates the moving image that is played back with a longer playback time when the change amount acquired by the change amount calculation unit is larger than a threshold value than when the change amount is smaller than the threshold value. The image processing apparatus according to claim 2 . An image processing device;
A plurality of two-dimensionally arranged photoelectric conversion elements that photoelectrically convert incident light into an electrical signal, and light from a partial region in a cross-sectional area of the incident light, respectively, to the corresponding plurality of photoelectric conversion elements An imaging unit having a light-shielding unit provided with an incident aperture, and imaging an object to generate an image;
With
The image processing apparatus includes:
A change amount calculation unit that obtains a disparity amount indicating disparity with respect to the same object in time series, and calculates a temporal change amount of the disparity amount; and
A moving image generating unit that generates a moving image in which at least the amount of change with respect to the target is reduced when the amount of change acquired by the amount of change calculating unit is greater than a threshold;
Have
The moving image generation unit generates the moving image from a plurality of images of the target imaged in time series by the imaging unit,
The moving image generation unit is generated when the change amount acquired by the change amount calculation unit is photographed by the photographing unit at a higher frame rate than when the change amount is smaller than the threshold value. An imaging device that generates the moving image from a plurality of images . The imaging device according to claim 4, wherein the moving image generation unit generates the moving image in which the amount of change is reduced to the threshold value or less. Further comprising a diaphragm with a variable aperture value that shields a peripheral region of the incident light incident on the imaging unit;
When the change amount acquired by the change amount calculation unit is larger than a threshold value, the moving image generation unit increases the aperture value in a state where the aperture value is larger than when the change amount is smaller than the threshold value. The imaging device according to claim 4, wherein the moving image is generated from a plurality of images generated by time series imaging. The imaging device according to claim 4 , wherein the moving image generation unit outputs the frame rate information in association with the moving image. An image for obtaining a reference image, a first parallax image having a first parallax with respect to the reference image, and a second parallax image having a second parallax in a direction opposite to the first parallax with respect to the reference image. An acquisition unit,
The moving image generation unit assigns a parallax amount based on the parallax amount indicated by the first parallax image and the parallax amount indicated by the second parallax image to the reference image for each corresponding pixel. To generate the video based on
The moving image generation unit weights the parallax amount indicated by the first parallax image when the change amount acquired by the change amount calculation unit is larger than a threshold value, compared to when the change amount is smaller than the threshold value. The imaging device according to claim 4, wherein the parallax amount that makes the weight of the parallax amount indicated by the second parallax image closer is assigned to the reference image. The moving image generation unit generates the moving image that is played back with a longer playback time when the change amount acquired by the change amount calculation unit is larger than a threshold value than when the change amount is smaller than the threshold value. The imaging device according to claim 4 . A change amount calculation procedure for acquiring a disparity amount indicating disparity with respect to the same object in time series, and calculating a temporal change amount of the disparity amount;
A moving image generating procedure for generating a moving image with a reduced amount of change for at least the target when the amount of change acquired by the change amount calculating procedure is greater than a threshold ;
An image for obtaining a reference image, a first parallax image having a first parallax with respect to the reference image, and a second parallax image having a second parallax in a direction opposite to the first parallax with respect to the reference image. for example Bei the acquisition procedure <br/>,
The moving image generation procedure includes a new pair of parallax images in which a parallax amount based on a parallax amount indicated by the first parallax image and a parallax amount indicated by the second parallax image is assigned to the reference image for each corresponding pixel. To generate the video based on
In the moving image generation procedure, when the change amount acquired by the change amount calculation procedure is larger than a threshold value, the weight of the parallax amount indicated by the first parallax image is larger than when the change amount is smaller than the threshold value. And an image processing method for allocating, to the reference image, the parallax amount that makes the weight of the parallax amount indicated by the second parallax image closer . A change amount calculation procedure for acquiring a disparity amount indicating disparity with respect to the same object in time series, and calculating a temporal change amount of the disparity amount;
A moving image generating procedure for generating a moving image with a reduced amount of change for at least the object when the amount of change acquired by the change amount calculating procedure is greater than a threshold;
An image for obtaining a reference image, a first parallax image having a first parallax with respect to the reference image, and a second parallax image having a second parallax in a direction opposite to the first parallax with respect to the reference image. Let the computer execute the acquisition procedure and <br/>
The moving image generation procedure includes a new pair of parallax images in which a parallax amount based on a parallax amount indicated by the first parallax image and a parallax amount indicated by the second parallax image is assigned to the reference image for each corresponding pixel. To generate the video based on
In the moving image generation procedure, when the change amount acquired by the change amount calculation procedure is larger than a threshold value, the weight of the parallax amount indicated by the first parallax image is larger than when the change amount is smaller than the threshold value. And a program for allocating the parallax amount, which makes the weight of the parallax amount indicated by the second parallax image closer, to the reference image .

Images