JP2012015872A - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device or the like capable of obtaining a high-resolution still image from a moving image by a simple process.SOLUTION: An imaging device includes an image acquisition part, a storage part, an estimation calculation part 200, and an image output part 210. The image acquisition part acquires an addition pixel value obtained by adding pixel values of a plurality of pixels included in a light-receiving unit while shifting a pixel sequentially with superposing the light-receiving unit. The storage part stores a low-resolution frame image by the addition pixel value. The estimation calculation part 200 estimates the pixel value of each pixel included in the light-receiving unit, based on a plurality of low-resolution frame images. The image output part 210 outputs a high-resolution frame image with a higher resolution than the low-resolution frame image, based on the estimated pixel value. The estimation calculation part 200 calculates a variation amount of the adjacent addition pixel values among the plurality of addition pixel values used for the estimation of an estimation target pixel, and estimates the pixel value of the estimation target pixel based on the variation amount.

Description

  The present invention relates to an imaging device and the like.

  In recent years, digital still cameras and digital video cameras have been improved in resolution and multifunction as the technology advances, and there is no functional difference between them. In addition, cameras that can shoot both still images and moving images with a single camera have become common.

JP 2009-124621 A JP 2008-243037 A

  However, in the conventional digital camera, the number of pixels that can be photographed and recorded as a moving image is much smaller, for example, about 2 million pixels, compared to, for example, that a still image having 10 million pixels or more can be photographed and recorded. Therefore, there is a problem that it is difficult to obtain a high-resolution still image during moving image shooting.

  For example, if a still image is captured by pressing the shutter during moving image capturing, the moving image capturing is stopped while the still image is captured. For this reason, a period in which no moving image is captured occurs.

  Alternatively, if a frame image in a moving image at the timing when the shutter is pressed is a still image, a high-resolution still image cannot be obtained. Specifically, in moving image shooting, 30 fps or 60 fps pixel signals are read and processed, but there is a limit to the band of images that can be read from the image sensor. For this reason, when the reading frame rate is increased, the number of pixels that can be read out per frame decreases, and the number of moving image pixels becomes smaller than that of a still image. In addition, the number of moving picture pixels is limited by the format, such as about 2 million pixels of full HD. For this reason, the number of pixels is reduced in accordance with the image size of the format and read out from the image sensor. For example, as a method of reducing the number of pixels, a method of adding and reading data from the image sensor and a method of reading out by thinning out pixels are used. As described above, since a moving image has a lower resolution than a still image, if an image taken as a moving image is used as a still image, a low-resolution image without a sense of resolution is obtained.

  To solve this problem, there is a technique for generating a high resolution image by super-resolution processing. In this method, low-resolution images are read out while sequentially shifting the position, and modeled based on the plurality of position-shifted images. Then, the estimated image is degraded to generate a low-resolution image, which is compared with the original low-resolution image, and the high-definition image is estimated by deforming the assumed high-definition image so that the difference is minimized. To do.

For example, Patent Document 1 discloses a technique for sequentially capturing low-resolution images pixel-shifted during moving image shooting in time series and generating a high-resolution image by synthesizing the plurality of low-resolution images. Has been. In this method, the super-resolution processing described above is performed on the assumed high-resolution image, and a high-resolution image with high likelihood is estimated. However, in this method, since the estimation accuracy is increased by repetitive calculation, the calculation amount is very large, the processing scale is increased, and processing time is required. For this reason, it is difficult to apply to devices having processing capacity and cost limitations, which is not practical.
Patent Document 2 discloses a technique for generating a high-definition image for the purpose of use in digital archives, remote sensing, production technology, measuring equipment, and the like. In this method, a high-resolution image is generated from the plurality of low-resolution images using a pixel-shifted low-resolution image. Specifically, the luminance value of the position obtained from the original image is set as the initial value of the high-resolution image, and the luminance value of the high-resolution image is determined by the estimation method based on the luminance value of the original image and the set initial value. Calculate. Next, for areas with large changes in luminance values, the luminance value of the high-resolution image is calculated by the average method, and the calculation result of the luminance value by the estimation method and the calculation result of the luminance value by the average method are merged to achieve high resolution. Perform image generation processing. However, in this method, if the initial value is not specified, the estimation error becomes very large. That is, in this method, in order to set the initial value, a special pattern having a uniform luminance value is inserted into a part of the original image, so that it takes time to search for the area. In addition, when such a region in the image cannot be detected, a high-definition image with high accuracy cannot be obtained.

  According to some aspects of the present invention, it is possible to provide an imaging device or the like that can obtain a high-resolution still image from a moving image by simple processing.

  In one embodiment of the present invention, a light receiving unit that is a unit for obtaining an added pixel value is set for each of a plurality of pixels of an imaging element, and a plurality of pixels included in the light receiving unit are sequentially shifted while superimposing the light receiving units. An image acquisition unit that acquires the added pixel value obtained by adding the pixel values of the pixels, a storage unit that stores a low-resolution frame image based on the added pixel value, and a plurality of low-resolution images stored in the storage unit Based on a frame image, an estimation calculation unit that estimates a pixel value of each pixel included in the light receiving unit, and on the basis of the pixel value estimated by the estimation calculation unit, a higher resolution than the low-resolution frame image An image output unit that outputs a high-resolution frame image, wherein the estimation calculation unit obtains a change amount of adjacent addition pixel values among a plurality of addition pixel values used for estimation of the estimation target pixel, and the change Based on quantity It relates to an imaging apparatus for estimating a pixel value of the serial estimation target pixel.

  According to one aspect of the present invention, an additional pixel value is acquired while sequentially shifting pixels while superimposing light reception units, and a low-resolution frame image based on the additional pixel value is acquired. Then, a pixel value is estimated based on the plurality of low resolution frame images, and a high resolution frame image is output based on the pixel values. At this time, a change amount of adjacent addition pixel values among a plurality of addition pixel values used for estimation is obtained, and a pixel value of the estimation target pixel is estimated based on the change amount. Thereby, it is possible to obtain a high-resolution still image from a moving image by a simple process.

  In the aspect of the invention, the estimation calculation unit may calculate the difference between the adjacent addition pixel values in the horizontal direction among the plurality of addition pixel values and the plurality of additions as the change amount of the adjacent addition pixel values. You may obtain | require the difference of the addition pixel value adjacent to a perpendicular direction among pixel values.

  In this way, the pixel value of the estimation target pixel can be estimated based on the difference between the addition pixel values adjacent in the horizontal direction and the difference between the addition pixel values adjacent in the vertical direction.

  In one aspect of the present invention, the estimation calculation unit includes a pixel value estimation unit that obtains a plurality of candidate pixel values for the estimation target pixel, and a candidate having a small change amount from the plurality of candidate pixel values. And a determination unit that determines a pixel value and outputs the determined candidate pixel value as a pixel value of the estimation target pixel.

  In this way, since a candidate pixel value with a small amount of change can be output as the pixel value of the estimation target pixel, the estimation error can be suppressed.

  In one aspect of the present invention, the pixel value estimation unit estimates the plurality of candidate pixel values using the addition pixel values of different combinations for each candidate pixel value, and the determination unit includes the plurality of candidates. The amount of change is obtained from the added pixel value used for estimating a pixel value, a combination having a smaller amount of change is determined among the different combinations, and a candidate pixel value corresponding to the determined combination is determined as the estimation target You may output as a pixel value of a pixel.

  In this way, by obtaining a plurality of candidate pixel values from the added pixel values of different combinations and determining a combination with a small change amount, a candidate pixel value with a small change amount can be output as the pixel value of the estimation target pixel. .

  In one aspect of the present invention, the estimation calculation unit corresponds to a pixel value estimation unit that estimates a pixel value of the estimation target pixel by an estimation method selected from a plurality of estimation methods, and each estimation method. A determination unit that obtains the amount of change and determines and selects an estimation method having a small amount of change among the plurality of estimation methods;

  In this way, the estimation error can be suppressed because the pixel value of the estimation target pixel can be estimated by an estimation method with a small amount of change.

  In the aspect of the invention, the pixel value estimation unit may perform the first estimation method in which the resolution is increased in the horizontal direction and then the resolution in the vertical direction, and the horizontal after the resolution is increased in the vertical direction. A pixel value of the estimation target pixel is estimated by a selected estimation method from among second estimation methods that achieve high resolution in the direction, and the determination unit determines the amount of change in the addition pixel value adjacent in the horizontal direction. When the first change amount and the second change amount that is the change amount of the addition pixel values adjacent in the vertical direction are obtained, and it is determined that the first change amount is smaller than the second change amount Alternatively, when the first estimation method is selected and it is determined that the second change amount is smaller than the first change amount, the second estimation method may be selected.

  In this way, the magnitude relationship between the amount of change in the horizontal direction and the amount of change in the vertical direction is determined, and the estimation method is selected based on the determination result. High resolution can be achieved in the direction.

  In the aspect of the invention, the light receiving unit is sequentially set to a first position and a second position next to the first position by the pixel shift, and the light receiving unit of the first position is set. When the light receiving unit at the second position overlaps, the pixel value estimation unit obtains a difference value between the added pixel values at the first and second positions and superimposes the light receiving unit at the first position. A first intermediate pixel value that is a light receiving value of the first light receiving region excluding the region, and a second light receiving value of the second light receiving region that is the light receiving unit of the second position excluding the overlapping region. A relational expression with an intermediate pixel value is expressed using the difference value, the first and second intermediate pixel values are estimated using the relational expression, and the estimated first intermediate pixel value is used for the estimation. The pixel value of the estimation target pixel may be obtained.

  In this way, it is possible to estimate the intermediate pixel value from the light reception value read while sequentially shifting the pixels while superimposing the light reception units, and to obtain the final estimated pixel value from the estimated intermediate pixel value. Thereby, pixel value estimation of a high resolution frame image can be simplified.

  In the aspect of the invention, the pixel value estimation unit may include a continuous intermediate pixel value including the first and second intermediate pixel values as the intermediate pixel value pattern. The relational expression between the intermediate pixel values is expressed using the added pixel values of the first and second positions, and the continuous added pixel values including the added pixel values of the first and second positions are added pixel values. In the case of a pattern, the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values is compared with the added pixel value pattern to evaluate similarity, and based on the similarity evaluation result, An intermediate pixel value included in the intermediate pixel value pattern may be determined so that the similarity is the highest.

  In this way, it is possible to estimate the intermediate pixel value based on a plurality of light reception values acquired by pixel shifting while superimposing the light reception units.

  In one aspect of the present invention, the pixel value estimation unit obtains an evaluation function representing an error between the intermediate pixel value pattern and the addition pixel value pattern expressed by a relational expression between the intermediate pixel values, and The intermediate pixel value included in the intermediate pixel value pattern may be determined so that the value of the evaluation function is minimized.

  In this way, by determining the intermediate pixel value so that the value of the evaluation function is minimized, the intermediate pixel value is determined so that the similarity between the intermediate pixel value pattern and the received light value is the highest. it can.

2 is a configuration example of an imaging device. Explanatory drawing about the acquisition method of an addition pixel value. Explanatory drawing about the acquisition method of an addition pixel value. Explanatory drawing about the acquisition method of an addition pixel value. Explanatory drawing about the acquisition method of an addition pixel value. Explanatory drawing about the recording format of an animation. Explanatory drawing about a basic estimation method. Explanatory drawing about a basic estimation method. The 1st structural example of a maximum likelihood image estimation part. Explanatory drawing about the method of calculating | requiring a some candidate pixel value. The flowchart of the process which estimates a pixel value with the 1st estimation method. The detailed flowchart of the process which determines the probability. Explanatory drawing about the difference of an adjacent addition pixel value. The 2nd structural example of a maximum likelihood image estimation part. Example characteristics of estimation error magnitude. The flowchart of the process which estimates a pixel value with the 2nd estimation method. FIG. 17A is a detailed flowchart of processing for determining the estimation order. FIG. 17B is an explanatory diagram of a processing block. Explanatory drawing about the 1st estimation method and the 2nd estimation method.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. Imaging Device FIG. 1 shows a configuration example of a digital camera (imaging device) that can obtain a high-resolution still image from a moving image by simple processing. This digital camera includes an imaging lens 102 and a body 101 (main body portion). The body 101 includes a mechanical shutter 103, an image sensor 104 (sensor), a mechanical shutter drive unit 108, a memory 110, an AE processing unit 111 (AE: Automatic Exposure), an AF processing unit 112 (AF: Auto-Focus), and a system controller 113. , Image processing unit 114, compression / decompression unit 115, system bus 116, lens control unit 117, UI control unit (UI: User Interface) 118, display control unit 119, maximum likelihood image estimation unit 120, recording medium control unit 121 (memory) Device control unit), LCD 122 (display unit in a broad sense, LCD: Liquid Crystal Display), EVF 123 (Electronic View Finder), recording medium 124 (storage device), and image acquisition unit 130.

  This digital camera is a camera system that can shoot still images and still images, and that can shoot still images during moving image shooting. These shooting modes are switched according to user designation or the like, and control of the digital camera is switched depending on the shooting mode. For example, this digital camera is an interchangeable lens digital camera or a compact camera in which a lens and a body are integrated.

  An input signal is input to the UI control unit 118 from a user interface for a user to operate the digital camera, and the UI control unit 118 controls the digital camera based on the input signal. Specifically, the UI control unit 118 includes user interfaces such as various buttons and a dial (not shown). The UI control unit 118 transmits input information from the user interface to the system controller 113 and controls the operation of the digital camera.

  The imaging lens 102 (interchangeable lens) is composed of optical elements such as a focus lens, a zoom lens, and a diaphragm. The focus lens is driven based on focus position data output from the AF processing unit 112. The zoom lens is driven to a predetermined zoom position based on a user's zoom operation via the UI control unit 118. As for the aperture, the aperture diameter (F value) is controlled based on aperture value data output from the AE processing unit 111. In the interchangeable lens digital camera, the body 101 and the imaging lens 102 are connected via a communication path. For this reason, the lens control unit 117 communicates at a predetermined timing to control the optical element, whereby the imaging lens 102 is controlled.

  The mechanical shutter 103 is used when taking a still image. The mechanical shutter 103 includes a motor and a motor driver, and is driven by a mechanical shutter driving unit 108. The mechanical shutter drive unit 108 controls the opening / closing of the mechanical shutter 103 in accordance with the timing of the pressing signal of the shutter button from the UI control unit 118. The mechanical shutter drive unit 108 controls the opening and closing of the mechanical shutter 103 according to the shutter speed data output from the AE processing unit 111. At the time of moving image shooting, the mechanical shutter drive unit 108 is set not to operate (non-operating state).

  The image acquisition unit 130 controls an imaging operation such as exposure or charge readout, and acquires a captured image. Specifically, the image acquisition unit 130 includes an image sensor 104, a CDS 105 (Correlated Double Sampling circuit), an AMP 106 (amplifier), an A / D converter 107, and an image sensor drive unit 109.

  The image sensor 104 has a photocathode on which a large number of light receiving elements are arranged in a matrix. A subject image that has passed through the imaging lens 102 is formed on the photoelectric surface on the image sensor 104, and the image sensor 104 photoelectrically converts the image. The image sensor 104 has RGB color filters arranged in a mosaic pattern.

  The image sensor driving unit 109 supplies a vertical transfer clock signal and a horizontal transfer clock signal, and the image sensor 104 reads out the electric charge accumulated for each pixel in synchronization with these signals and outputs it as an image signal. The image sensor driving unit 109 supplies the image sensor 104 with an electronic shutter drive signal for controlling the charge exposure time in each pixel. The image signal output from the image sensor 104 is subjected to noise removal by a CDS (correlated double sampling circuit) 105, gain adjustment is performed, amplified by an AMP 106, and converted into digital image data by an A / D converter 107. Converted.

  The system controller 113 controls the operation and processing of each component of the digital camera. For example, the system controller 113 performs timing control of the image sensor driving unit 109 during moving image shooting. That is, the system controller 113 starts imaging when an instruction to start imaging is given by an input from the UI control unit 118, and controls charge reading of the image sensor 104. The system controller 113 controls analog signal processing by the CDS 105 (correlated double sampling circuit), the AMP 106, and the A / D converter 107, and performs control to write digital image data into the memory 110.

  The AF processing unit 112 determines an in-focus position based on the AF area information set by the UI control unit 118 and image data captured at a predetermined capturing timing. The AF processing unit 112 calculates focus position data for driving the imaging lens 102 by processing the image data continuously written in the memory 110.

  Similar to the AF processing unit 112, the AE processing unit 111 calculates subject luminance from image data captured at a predetermined photographing timing and written in the memory 110, and based on the subject luminance, the shutter speed, the aperture value, and the like are calculated. Determine the exposure conditions.

  The display control unit 119 performs control to display an image on the LCD 122 and the EVF 123 (electronic viewfinder), and control to display the estimated high resolution image on the LCD 122. For example, the display control unit 119 displays the image data written in the memory 110 on the LCD 122 or EVF 123 as a through image, and converts the image data into a format that can be displayed on the LCD 122 or EVF 123. When the through image display mode is set by the UI control unit 118, the display control unit 119 sequentially displays images updated at predetermined intervals on the LCD 122 and the EVF 123 in the memory 110. When the shooting mode is set by the UI control unit 118, the display control unit 119 displays the image acquired by the image acquisition unit 130 on the LCD 122 or the EVF 123 as a through image.

  The image processing unit 114 performs image processing on the image data written in the memory 110. Specifically, the image processing unit 114 performs colorization processing, gamma correction, contrast correction, contour enhancement processing, and the like as image processing.

  The compression / decompression unit 115 performs image data compression processing in a predetermined compression format, and outputs the compressed image data to the recording medium control unit 121. The recording medium control unit 121 writes the compressed image data into the recording medium 124. In the playback mode, the compression / decompression unit 115 reads a compressed moving image file (image data) from the recording medium 124 and performs decompression processing on the moving image file. The decompressed image data is output to the display control unit 119 as a moving image, and the display control unit 119 displays the moving image on the LCD 122 or the EVF 123.

  The recording medium control unit 121 controls writing to the recording medium 124 and reading from the recording medium 124. For example, the recording medium 124 is a flash memory, a hard disk, an optical disk, or the like.

  The maximum likelihood image estimation unit 120 performs processing for estimating a high resolution still image from a low resolution moving image. For example, when still image shooting is performed during moving image shooting, the maximum likelihood image estimation unit 120 reconstructs a still image at a timing when the user presses the shutter from the shot moving image. In addition, when generating a still image after moving image shooting, the user views the moving image displayed on the LCD 122 and designates a timing at which a still image is desired. Then, the maximum likelihood image estimation unit 120 configures the still image at that timing from the moving image recorded on the recording medium 124.

2. Next, a method for estimating a high-resolution still image from a low-resolution moving image in the moving-image still image shooting mode (a mode for acquiring a still image from a moving image) will be described.

  First, a method for obtaining an added pixel value constituting each frame image of a low resolution moving image will be described. In the moving image still image shooting mode, when the moving image still image shooting mode is selected in advance by the UI control unit 118 and an instruction to start shooting is issued, shooting is started, charge reading of the image sensor 104 is controlled, and the CDS 105 and the AMP 106 are controlled. Then, analog signal processing is performed by the A / D converter 107 and the image data is written in the memory 110.

  2 to 5 are explanatory diagrams showing a method for reading out charges from the image sensor 104. FIG. For example, RGB color filters are arranged in the image sensor 104, and only specific color signals are input to each pixel (for example, a Bayer array). 2-5 is explanatory drawing about the addition pixel value in this case.

  Note that the frame used in the following description is, for example, the timing at which one frame image is captured by the image sensor or the timing at which one frame image is processed in image processing. Alternatively, one frame image in the image data is also referred to as a frame as appropriate.

FIG. 2 shows a method for reading G1 pixels (first G pixel). As shown in FIG. 2, four frames f _n , f _{n + 1} , f _{n + 2} , and f _{n + 3} (n is a natural number) are taken as one cycle, and reading is repeatedly performed for each frame. In FIG. 2, p represents a pixel pitch, and a dotted square represents that pixels at positions separated by 2p in the horizontal and vertical directions are added and read. The four pixels within the dotted square are light reception units, and the pixel value obtained by addition reading is the addition pixel value (light reception value).

In the following expression (1), the pixel value after readout in the frame f _n is as ₀₀ , the pixel value after readout in the frame f _{n + 1} is as ₀₁ , the pixel value after readout in the frame f _{n + 2} is as ₁₀ , and the pixel value in the frame f _{n + 3} is The pixel value after readout is shown as as ₁₁ . As shown in FIG. 2, when attention is paid to the pixels, in the frame f _n and the frame f _{n + 1} , the pixel value _G102 and the pixel value G1 ₂₂ are read out in both frames. In the frame f _{n + 1} and the frame f _{n + 2} , the pixel value G1 ₂₂ and the pixel value G1 ₂₄ are read out in both frames, and in the frame f _n and the frame f _{n + 3} , the pixel value G1 ₂₂ is read out in both frames. That is, there is an overlapping area in the area read between the frames.

FIG. 3 shows an R pixel readout method, FIG. 4 shows a G2 pixel (second G pixel) readout method, and FIG. 5 shows a B pixel readout method. The following formulas (2) to (4) show the added pixel values in the frames f _{n to} f _{n + 3} . Since the method for obtaining these added pixel values is the same as that for the G1 pixel described above, description thereof is omitted.

As shown in FIGS. 2 to 5, in the frame f _n , a frame image is constituted by the added pixel values as ₀₀ , at ₀₀ , au ₀₀ , and av ₀₀ . In the frame f _{n + 1} , a frame image is configured by the addition pixel values as ₀₁ , at ₀₁ , au ₀₁ , and av ₀₁ , and in the frame f _{n + 2} , the frame is formed by the addition value pixel values as ₁₀ , at ₁₀ , au ₁₀ , and av _10. An image is configured, and in the frame f _{n + 3} , a frame image is configured by the addition value pixel values as ₁₁ , at ₁₁ , au ₁₁ , and av ₁₁ .

  The signal read out from the image sensor 104 in this way is subjected to predetermined processing and written into the memory 110. The image written in the memory 110 is subjected to processing such as colorization by the image processing unit 114, compressed in a predetermined compression format by the compression / decompression unit 115, and written to the recording medium 124.

Next, a recording format of a moving image shot in the moving image still image shooting mode will be described with reference to FIG. As shown in A1 of FIG. 6, at the time of moving image shooting, the image generated by the image processing unit 114 is compressed by the compression / decompression unit 115 to generate image data PD _{1 to} PD _{n + 2} (frame image). The image data is recorded for each frame together with the header indicated by A2 and the shutter flag indicated by A3.

As shown in A4, when the user presses the shutter during moving image shooting, the UI control unit 118 detects that the shutter has been pressed and notifies the system controller 113 of it. For example, the shutter has been pressed in the image pickup operation period T _n in the frame f _n. Then, the system controller 113, at the timing when the image of the frame f _n the shutter is pressed is compressed, the compression decompression unit 115, and notifies that the shutter has been pressed by the frame f _n. Then, as indicated by A5, the compression / decompression unit 115 sets “1” to the shutter flag when writing the image data PD _n . In FIG. 6, a shutter flag “0” represents a state where the shutter is not pressed, and a shutter flag “1” represents a state where the shutter is pressed.

This operation is continued while the user is shooting in the moving image still image shooting mode, and is stopped when shooting is finished. The maximum likelihood image estimation unit 120 generates a still image from the recorded moving image data after the end of shooting. The generation timing may be, for example, when the user presses a generation start button (not shown), or may be generated automatically after completion of shooting. For example, the still image is generated from the images PD _{n−2 to} PD _{n + 1 of} two frames before and after the frame f _n to which the flag “1” is assigned (a total of four frames).

3. Basic Estimation Method for Pixel Value Next, a method for estimating a high-resolution still image from the addition pixel value included in the frame image will be described. First, the basic estimation method will be described, and then the first and second estimation methods using the basic estimation method will be described.

  FIG. 7 and FIG. 8 are explanatory diagrams for a basic estimation method. In the following description, when the pixels of the image sensor are arranged in an orthogonal two-axis coordinate system, the direction along one axis is called the horizontal direction, and the direction along the other axis is called the vertical direction. . For example, the horizontal direction is the horizontal scanning direction in the imaging operation. In the following description, as to the direction on the image data, the direction along one of the two orthogonal axes is referred to as a horizontal direction, and the direction along the other axis is referred to as a vertical direction.

As shown in B1 of FIG. 7, the image before addition (mixing) is an image to be obtained, and pixels are represented by v ₀₀ ′, v ₁₀ ′, v ₂₀ ′,. The pre-addition image is an image acquired when it is assumed that the image is captured at the original resolution of the image sensor. As shown in B2, an image is read from the image sensor 104 by pixel addition. In this captured image, each pixel is represented by a ₀₀ , a ₀₁ , a _02,.

Next, as shown in B3, an intermediate estimated image (estimated image 1) is estimated from the captured image. At this time, the intermediate estimated image is composed of strip-like pixels, and intermediate estimated pixel values b ₀₀ , b ₁₀ , b ₂₀ ,... Are calculated. Since this pixel value corresponds to the pixel value obtained by adding the two pre-addition pixels, the 2-pixel addition value is calculated from the 4-pixel addition value in the first estimation. For example, b ₀₀ corresponds to the addition pixel of v ₀₀ ′ and v ₀₁ ′, b ₀₁ corresponds to the addition pixel of v ₀₁ ′ and v ₀₂ ′, and b ₀₀ and b ₀₁ are the overlapping region v ₀₁ ′. Is superimposed. That is, the pixel value of the intermediate estimated image corresponds to a pixel value obtained by pixel shift while superimposing the added pixel value of two pixels in the vertical direction (or horizontal direction).

As shown in B4, final estimated pixels v ₀₀ , v ₁₀ , v ₂₀ ,... Are calculated from the intermediate estimated image, and a final estimated image (estimated image 2) is estimated. When the estimation accuracy of the final estimated value is sufficiently good, the pixel values of the pre-addition image and the final estimated image are the same value.

A method for estimating b _ij from a _ij will be described with reference to FIG. FIG. 8 shows an example of the first row in the horizontal direction of the addition pixel a _ij having an overlapping region. When the pixel values are a ₀₀ and a _{10 in the} order of shift, the following equation (5) is obtained. When b ₀₀ is an unknown and the intermediate estimated pixel values b ₁₀ and b ₂₀ are obtained as a function of b ₀₀ , the following equation (6) is obtained from the following equation (5).

As in the above equation _{(6), b} a _{b 00} a unknown _00, b _10, the combination pattern of the _{b 20} are determined, we do not know the absolute value of the unknown _{b 00.} Therefore, it is necessary to calculate or set the unknown b _00. Therefore, as shown in the following equation (7), the pattern {a ₀₀ , a ₁₀ } based on the added pixel value is compared with the pattern {b ₀₀ , b ₁₀ , b ₂₀ } based on the intermediate estimated pixel value, and the error E is minimized. An unknown number b ₀₀ is derived. For example, the error E is obtained by the error E _j shown in the following equation (7), and b ₀₀ that minimizes this E _j is obtained. Then, the obtained value is set as an intermediate estimated pixel value b ₀₀ and substituted into the above equation (6) to obtain intermediate estimated pixel values b ₁₀ and b ₂₀ .

Note that v _ij is estimated from b _ij by the same method as described above. That _{is, v 00} as unknowns, representing the estimated pixel values _{v 00, v 01, v 02} in relation of _{v 00.} Then, an error between the pattern {v ₀₀ , v ₀₁ , v ₀₂ } and the pattern {b ₀₀ , b ₀₁ } is obtained, v _{00 at} which the error is minimized is obtained, and the v ₀₀ is substituted into the relational expression to obtain v _01. , seek _{v 02.} In this way, four intermediate estimated pixel values b ₀₀ , b ₁₀ , b ₀₁ , b ₁₁ are estimated from the four added pixel values a ₀₀ , a ₁₀ , a ₀₁ , a _11, and the four intermediate pixels are estimated. Four estimated pixel values v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ are estimated from the values.

4). First Estimation Method for Pixel Value Next, a first estimation method using the above basic estimation method will be described. FIG. 9 shows a configuration example in the case where the pixel value is obtained by the first estimation method as a first configuration example of the maximum likelihood image estimation unit 120. The maximum likelihood image estimation unit includes an estimation calculation unit 200 and an image output unit 210.

Estimation calculation section 200 obtains a plurality of candidates for each estimated pixel values v _ij, selects the final estimated pixel values v _ij from the candidates. Specifically, the estimation calculation unit 200 includes a pixel value estimation unit 220 and a determination unit 230 (selection unit). The pixel value estimation unit 220 obtains a plurality of candidates for one v _ij from the combination of different added pixel values a _ij using the above basic estimation method. The determination unit 230 obtains a difference value between adjacent horizontal pixel values a _{ij in} the horizontal direction and the vertical direction, determines a magnitude relationship between the difference values, and determines a final estimated pixel value v _ij from the determination result.

The image output unit 210 outputs a high resolution image based on the estimated pixel value v _ij . As described above with reference to FIG. 2 and the like, when the estimated pixel is obtained from the added pixel value obtained by the Bayer array image sensor, a mosaic image corresponding to the Bayer array of the original image sensor is obtained. In this case, the image output unit 210 performs demosaicing processing on the estimated high-resolution mosaic image, and outputs a color high-resolution still image. Further, the image output unit 210 may perform image processing such as gradation conversion processing and filter processing.

With reference to FIG. 10, an example case of obtaining four candidate against v _11, it will be described a method of obtaining a plurality of candidates for each estimated pixel value. As shown in FIG. 10, in the first pattern, candidate pixels of addition pixels a ₀₀ , a ₁₀ , a ₀₁ , a ₁₁ to v ₁₁ are calculated. In the second pattern, candidate pixels of v ₁₁ are calculated from the addition pixels a ₁₀ , a ₂₀ , a ₁₁ , a ₂₁ . In the third pattern, and calculates the candidate pixel pixel-sum _{a 01, a 11, a 02} , a 12 from _{v 11.} In the fourth pattern, v ₁₁ candidate pixels are calculated from the added pixels a ₁₁ , a ₂₁ , a ₁₂ , and a ₂₂ . In this way, v ₁₁ can be calculated from four combinations of addition pixels having overlapping regions, and four candidate pixels can be obtained.

FIG. 11 shows a flowchart of processing for estimating a pixel value by the first estimation method. When this process is started, it sets the estimated target pixel to be processed for estimating the pixel values from the captured image (e.g., the position of the pixel of v ₁₁ in FIG. 10) (step S1). Next, a candidate pixel value is calculated from the addition pixel including the estimation target pixel. That is, candidate pixel values corresponding to each combination are calculated from the four combinations of addition pixels shown in FIG. 10 (step S2).

  Next, the probability is determined from the calculated four candidate pixel values, and an appropriate estimated pixel value is selected (step S3). Next, it is determined whether the estimation of all the pixels constituting the image has been completed (step S4). If the estimation of all the pixels has not been completed, the calculation of the estimated pixel is continued by changing the position of the target pixel (NO). If all pixels have been estimated, the estimation process ends (YES). This processing flow is performed by the maximum likelihood image estimation unit 120. When the processing is completed, the estimated image is stored in the memory 110. The stored estimated image is written to the recording medium 124 as a high-definition still image.

FIG. 12 shows a detailed flowchart of the process of determining the probability from the four candidate pixel values (step S3 in FIG. 11). When this process is started, and calculates a difference value D1~D4 from the addition pixel value _{a ij} (step S10 to S13). With reference to FIG. _13, as an example the case of estimating the _{v 11,} described difference value D1 to D4. As shown in FIG. 13, nine addition pixels are used to obtain four candidates. At this time, the difference value D1 is the sum of the vertical difference values of the added pixels in the first and second rows, and the difference value D2 is the vertical difference value of the added pixels in the second and third rows. Is the sum of The difference value D3 is the sum of the horizontal difference values of the addition pixels in the first and second columns, and the difference value D4 is the sum of the horizontal difference values of the addition pixels in the second and third columns. is there. In this way, the difference between pixel values between adjacent columns or rows is calculated to calculate the total value.

  As shown in FIG. 12, D1 and D2 are then compared (step S14). If D1 is small (YES), D3 and D4 are further compared (step S15). If D3 is small (YES), the estimated value calculated from the first pattern combination is selected (step S16). If D4 is equal to or less than D3 (NO), the estimated value calculated from the combination of the third patterns is selected (step S17). If D2 is equal to or less than D1 in step S14 (NO), D3 and D4 are compared (step S18). When D3 is small (YES), the estimated value calculated from the combination of the second patterns is selected (step S19). When D4 is equal to or less than D3 (NO), an estimated value calculated from the combination of the fourth patterns is selected (step S20). In this way, a candidate obtained from a combination pattern having a smaller difference value between the horizontal direction and the vertical direction is selected. When the selection of the estimated pixel value is completed, the process ends.

  As described above, the digital camera has a problem that it is difficult to obtain a high-resolution still image during moving image shooting. In addition, there is a problem of improving estimation accuracy when estimating a high resolution image from a moving image. In addition, the super-resolution processing (for example, Patent Documents 1 and 2 described above) that obtains a high-resolution still image from a low-resolution moving image has a problem of a large amount of calculation and a large processing load, and an area for initial value setting. There is a problem that it is necessary to search.

In this regard, according to the present embodiment, as illustrated in FIG. 2 and the like, the image acquisition unit 130 sets a light reception unit, which is a unit for acquiring the added pixel value a _ij, for each of the plurality of pixels of the image sensor 104 ( For example, the light receiving unit of a ₀₀ includes G1 ₀₀ , G1 ₂₀ , G1 ₀₂ , G1 ₂₂ ). Image acquisition unit 130, while superimposing the light receiving unit _{(a 00} and _{a 10} are _{G1 20} and superimposed _{G1 22)} while sequentially pixel shifted light receiving units (a predetermined period (4 frames) _{a 00, a 10} , A ₁₁ and a ₀₁ are sequentially shifted), and an added pixel value a _ij obtained by adding the pixel values of a plurality of pixels included in the light receiving unit is obtained.

Then, as illustrated in FIG. 6, the storage unit (the recording medium 124 or the memory 110) stores the low-resolution frame images PD _{1 to} PD _{n + 2} based on the added pixel value a _ij . As shown in FIG. 7, the estimation calculation unit 200 is included in the light receiving unit based on a plurality of low-resolution frame images (added pixel values a _{ij for} four frames shown in B2) stored in the storage unit. The pixel value of each pixel (estimated pixel value v _ij shown in B4) is estimated. The image output unit 210 outputs a high resolution frame image having a higher resolution than the low resolution frame image based on the pixel value v _ij estimated by the estimation calculation unit 200.

At this time, as illustrated in FIG. 12, the estimation calculation unit 200 includes a change amount (difference value D1) of adjacent addition pixel values among the plurality of addition pixel values a _{00 to} a ₂₂ used for estimation of the estimation target pixel v _11. to D4) the calculated estimates the pixel value v ₁₁ of the estimation target pixel based on the amount of change.

Thereby, it is possible to obtain a high-resolution still image from a moving image by a simple process. Specifically, by estimating the pixel value v _ij using the added pixel value a _ij acquired while sequentially shifting the pixels while superimposing the light receiving units, the estimation process is simplified and the circuit has a relatively small amount of calculation. A high-definition still image can be generated by processing with a small scale. Thereby, a digital camera having a function of shooting a high-definition still image at the time of moving image shooting can be developed at a lower cost.

  Further, by estimating the pixel value based on the change amount of the adjacent addition pixel value, it is possible to perform the estimation considering the change amount of the adjacent addition pixel value. Therefore, the estimation accuracy of the pixel value can be improved as compared with the estimation that does not consider the amount of change (for example, estimation using only the pattern 1 shown in FIG. 10).

  In addition, since a still image can be acquired from a moving image, there is no need to interrupt the moving image shooting for still image shooting. In addition, since a still image at an arbitrary timing in a moving image can be obtained after shooting the moving image, there is no fear of missing a photo opportunity as compared with a case where a still image is shot by pressing the shutter.

In this embodiment, as shown in FIG. 13, the estimation calculation unit 200 uses the addition pixel values adjacent in the horizontal direction among the plurality of addition pixel values a _{00 to} a ₂₂ as the change amount of the adjacent addition pixel values. Differences D3 and D4 and differences D1 and D2 between the addition pixel values adjacent in the vertical direction are obtained.

  In this way, the pixel value of the estimation target pixel can be estimated based on the difference between the addition pixel values adjacent in the horizontal direction and the difference between the addition pixel values adjacent in the vertical direction. As shown in FIG. 15, the smaller the difference between the addition pixel values, the smaller the estimation error. Therefore, the estimation accuracy can be improved by performing the estimation taking into account the difference between the addition pixel values.

  In the present embodiment, as illustrated in FIG. 9, the estimation calculation unit 200 includes a pixel value estimation unit 220 and a determination unit 230. The pixel value estimation unit 220 obtains a plurality of candidate pixel values for the estimation target pixel. Then, the determination unit 230 determines a candidate pixel value having a small change amount from the plurality of candidate pixel values, and outputs the determined candidate pixel value as a pixel value of the estimation target pixel.

Specifically, as illustrated in FIG. 10, the pixel value estimation unit 220 uses a plurality of candidate pixel values by using different combinations of added pixel values (a _ij of combinations of patterns 1 to 4) for each candidate pixel value. (V ₁₁ corresponding to the patterns 1 to 4) is estimated. As illustrated in FIG. 12, the determination unit 230 determines the addition pixel values (all a _ij included in the patterns 1 to 4 included in the patterns 1 to 4, a _{00 to} a ₂₂ illustrated in FIG. 13) used for the estimation of the plurality of candidate pixel values. A change amount is obtained (steps S10 to S13). Then, the determination unit 230 determines a combination with a small change amount among the different combinations (step S14 and the like). Determination unit 230, the candidate pixel values corresponding to the determined combination, and outputs the pixel value v ₁₁ of the estimated target pixel (step S16 etc.).

  In this way, it is possible to determine a combination with a small amount of change by obtaining a plurality of candidate pixel values from the added pixel values of different combinations and comparing the magnitude relationship of the amount of change corresponding to each combination. Moreover, since the candidate pixel value calculated | required from the combination of the addition pixel value with a small variation | change_quantity can be selected, the estimation error of an estimation object pixel can be reduced.

In the above embodiment, one candidate pixel is selected from the four candidate pixels. However, in this embodiment, the four candidate pixel values are weighted and added by the weighting coefficient corresponding to the amount of change, and the pixel of the estimation target pixel A value may be obtained. Specifically, when the candidate pixel values of patterns 1 to 4 are vk ₁ to vk ₄ and the weighting coefficients are w ₁ to w ₄ , v ₁₁ = w ₁ · vk ₁ + w ₂ · vk ₂ + w ₃ · The pixel value v ₁₁ is obtained by vk ₃ + w ₄ · vk ₄ . The weighting coefficient of the pattern determined to have a small difference value is set to 1, and the other coefficients are set to 0. For example, when the pattern 1 is selected, w ₁ = 1, w ₂ = w ₃ = w ₄ = 0.

  In the present embodiment, the light receiving unit is sequentially set to the first position and the second position next to the first position by the pixel shift. The light receiving units at the first and second positions overlap. Then, the pixel value estimation unit 220 calculates a difference value between the light reception values of the light reception units at the first and second positions. The pixel value estimation unit 220 includes a first intermediate pixel value (first intermediate estimated pixel value) that is a light reception value of the first light reception region obtained by removing the overlap region from the light reception unit of the first position, and a second A relational expression with a second intermediate pixel value (second intermediate estimated pixel value) that is a light reception value of the second light reception region obtained by removing the overlap region from the light reception unit of the position is expressed using a difference value. The pixel value estimation unit 220 estimates the first and second intermediate pixel values using the relational expression, and obtains the pixel value of each pixel of the light receiving unit using the estimated first intermediate pixel value.

For example, as shown in FIG. 7, the first position to the light receiving unit to obtain the received-light value a ₀₀ is set, in the next frame, the light-receiving unit is set to the second position for obtaining the received-light value a ₁₀ . These light receiving units are overlapped in a region including the estimated pixels v ₁₀ and v ₁₁ . A region including the estimated pixels v ₀₀ and v ₀₁ corresponds to the first light receiving region, and an intermediate pixel value b ₀₀ of the region corresponds to the first intermediate pixel value. In addition, a region including the estimated pixels v ₂₀ and v ₂₁ corresponds to the second light receiving region, and an intermediate pixel value b ₂₀ in the region corresponds to the second intermediate pixel value. Then, as described with reference to FIG. 8, the difference value δi ₀ between the received light values a ₀₀ and a ₁₀ is obtained, and the relational expression b ₂₀ = b ₀₀ + δi ₀ is obtained. An unknown number b ₀₀ is estimated, and b ₂₀ is estimated using a relational expression. Estimated pixel values v ₀₀ and v ₀₁ are obtained using b ₀₀ .

  In this way, the estimation process of the high resolution image can be simplified by once estimating the intermediate pixel value from the superposed shifted light reception value and obtaining the estimated pixel value from the superposed shifted intermediate pixel value. For example, compared with the above-mentioned Patent Document 1, the amount of calculation for super-resolution processing can be reduced, so that the circuit scale can be suppressed. Further, as compared with Patent Document 2 described above, it is not necessary to search for a special area for setting an initial value from a captured image.

  In the present embodiment, the pixel value estimation unit 220 uses the intermediate pixels included in the intermediate pixel value pattern when successive intermediate pixel values including the first and second intermediate pixel values are used as the intermediate pixel value pattern. A relational expression between the values is expressed using the added pixel values of the first and second positions. The pixel value estimation unit 220 uses an intermediate pixel value pattern represented by a relational expression between intermediate pixel values when successive addition pixel values including the addition pixel values at the first and second positions are used as the addition pixel value pattern. And the added pixel value pattern are compared to evaluate the similarity. Then, the pixel value estimation unit 220 determines an intermediate pixel value included in the intermediate pixel value pattern based on the similarity evaluation result so that the similarity becomes the highest.

For example, as described with reference to FIG. 8, successive intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } correspond to the intermediate pixel value pattern, and as shown in the above equation (6), {b ₀₀ , b ₁₀ , B ₂₀ } is expressed using received light values a ₀₀ , a ₁₀ . Then, the intermediate pixel value pattern {b ₀₀ , b ₁₀ , b ₂₀ } and the received light value pattern {a ₀₀ , a ₁₀ } are compared, and {b ₀₀ so that the similarity represented by the evaluation function E _j is the highest. , B ₁₀ , b ₂₀ } are determined. Here, the light reception value pattern {a ₀₀ , a ₁₀ } is a light reception value that is continuous in the horizontal direction or the vertical direction (in a continuous order).

  In this way, it is possible to estimate the intermediate pixel value based on a plurality of light reception values acquired by pixel shifting while superimposing the light reception units.

  In the present embodiment, the pixel value estimation unit 220 obtains an evaluation function representing an error between the intermediate pixel value pattern and the added pixel value pattern expressed by the relational expression between the intermediate pixel values, and the value of the evaluation function is The intermediate pixel value included in the intermediate pixel value pattern is determined so as to be minimized.

For example, as shown in the above formulas (6) and (7), the intermediate pixel value pattern {b ₀₀ , b ₁₀ , b ₂₀ } is represented as a function of the unknown b ₀₀ , and the intermediate pixel value pattern {b ₀₀ , b ₁₀ , B ₂₀ } and the received light value pattern {a ₀₀ , a ₁₀ } are represented by an evaluation function E _j . Then, an unknown number b ₀₀ = α (initial value) that minimizes the value of the evaluation function E _j is obtained, and the obtained values of b _{00 to} b ₂₀ are determined by substituting the obtained b ₀₀ into the above equation (6). The

  In this way, the value of the intermediate pixel value can be estimated by expressing the error by the evaluation function and obtaining the intermediate pixel value corresponding to the minimum value of the evaluation function. For example, as described above, the initial value of the intermediate pixel estimation can be set with a simple process by obtaining the unknown using the least square method. That is, it is not necessary to search for an area for setting an initial value as compared with the above-described Patent Document 2.

5. Second Estimation Method for Pixel Value Next, a second estimation method using the basic estimation method described above will be described. FIG. 14 shows a configuration example in the case where the pixel value is obtained by the second estimation method as a second configuration example of the maximum likelihood image estimation unit 120. The maximum likelihood image estimation unit includes an estimation calculation unit 200 and an image output unit 210. In FIG. 9 and the like, the same components as those described above are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

Estimation calculation unit 200 selects a method of estimating the intermediate estimated pixel values b _ij, to estimate the estimated pixel values v _ij from the intermediate estimated pixel values b _ij determined by the selected method. Specifically, the estimation calculation unit 200 includes a pixel value estimation unit 220 and a determination unit 230 (selection unit). Or determination unit 230, or obtains the intermediate estimated pixel values b _ij and higher definition the sum pixel values a _ij in the horizontal direction, obtaining the intermediate estimated pixel values b _ij and higher definition the sum pixel values a _ij vertically Determine. The pixel value estimation unit 220 obtains b _ij by increasing the definition of a _ij in the determined direction, and estimates v _ij from the b _ij .

The process which the estimation calculating part 200 performs is demonstrated in detail using FIGS. 15-18. First, the estimation error will be described. As described above, in the first estimation method, the estimation in the horizontal direction is performed when the estimation candidate image is generated, and then the estimation in the vertical direction is performed. This is because the output of the image sensor 104 is generally output for each horizontal line. In the estimation of the intermediate estimated pixel value b _ij, the unknown b ₀₀ that minimizes the error E _j shown in the above equation (7) is obtained, but the magnitude of the estimated error E _j differs depending on the combination of the added pixel values a _ij .

FIG. 15 schematically shows a characteristic example of the magnitude of the error E _j . FIG. 15 shows the magnitude of error (ratio of error to estimated value) when the horizontal axis is the absolute value of the difference between the added pixels used for estimation. As shown in FIG. 15, when the difference between the addition pixels used for estimation increases, the generated error also increases. Further, in this embodiment, since the estimation calculation is performed twice in the horizontal direction and the vertical direction, if an error occurs in the estimated value in the first estimation, the second estimated value is also affected, and an accurate estimated value is obtained. I can't get it. Therefore, in the second estimation method, the estimation error is reduced by controlling the order of the direction in which pixel estimation is performed.

FIG. 16 shows a flowchart of processing for estimating a pixel value by the second estimation method. When this process is started, a process block to be estimated is determined (step S20). This processing block is a combination of addition pixels used in one estimation process in the captured image, and is set for every four addition pixels, for example. For example, a ₀₀ , a ₁₀ , a ₀₁ , and a ₁₁ are set as the first processing block, and a ₂₀ , a ₃₀ , a ₂₁ , and a ₃₁ are set as the second processing block. Alternatively, a ₁₀ , a ₂₀ , a ₁₁ , and a ₂₁ may be set as the second processing block.

Next, the estimation order is determined from the difference value of the addition pixels belonging to the processing block (step S21). Next, the intermediate estimated pixel value b _ij is estimated in the estimation order, and the estimated pixel value v _ij is estimated from b _ij (step S23). Next, it is determined whether or not the generation of the estimated image has been completed (step S23). If all the pixels have been estimated, the process is terminated (YES), and if not, the process proceeds to step S20. Return (NO).

FIG. 17A shows a detailed flowchart of the process for determining the estimation order (step S21). Hereinafter, as illustrated in FIG. 17B, a case where a ₀₀ , a ₁₀ , a ₀₁ , and a ₁₁ are set as processing blocks will be described as an example. When this process is started, the amount of change P1 in the horizontal pixel value is calculated (step S30). Next, the amount of change P2 in the vertical pixel value is calculated (step S31).

Next, the amount of change P1 and the amount of change P2 are compared (step S32). If P1 is small, the resolution is increased in the horizontal direction (YES). That is, as shown in E1 in FIG. 18, the addition pixel value _{a 00, a 10, a 01} , a 11 from the intermediate estimated pixel value _b of the vertical two pixels in one horizontal pixel _00, b _10, b _{01, b 11} And a first intermediate estimated image (estimated image 1a) is obtained (step S33). Next, a final estimated image is estimated using the intermediate estimated pixel values b ₀₀ , b ₁₀ , b ₀₁ , and b ₁₁ (step S34). On the other hand, if P2 is greater than or equal to P1, high resolution is achieved in the vertical direction (step S32, NO). That is, as indicated by E2 in FIG. 18, the sum pixel values _{a 00, a 10, a 01} , a 11, a horizontal second intermediate estimated pixel value of the vertical one pixel pixel _{c 00, c 01, c 10} , c 11 And a second intermediate estimated image (estimated image 1b) is obtained (step S35). Next, a final estimated image is estimated using the intermediate estimated pixel values c ₀₀ , c ₀₁ , c ₁₀ and c ₁₁ (step S36). When the final estimated image is obtained, the process is terminated.

  According to the above embodiment, as shown in FIG. 14, the estimation calculation unit 200 includes a pixel value estimation unit 220 and a determination unit 230. The pixel value estimation unit 220 estimates the pixel value of the estimation target pixel using an estimation method selected from a plurality of estimation methods. The determination unit 230 obtains a change amount corresponding to each estimation method, and determines and selects an estimation method having a small change amount from among a plurality of estimation methods.

  Specifically, as illustrated in FIG. 18, the pixel value estimation unit 220 includes a first estimation method (E1) in which high resolution is achieved in the horizontal direction and then high resolution in the vertical direction, and high in the vertical direction. The pixel value of the pixel to be estimated is estimated by the estimation method selected from the second estimation method (E2) that performs high resolution in the horizontal direction after the resolution. As illustrated in FIG. 17A, the determination unit 230 includes a first change amount P1 that is a change amount of the addition pixel value adjacent in the horizontal direction and a first change amount that is a change amount of the addition pixel value adjacent in the vertical direction. 2 change amount P2 is obtained (steps S30 and S31). If the determination unit 230 determines that the first change amount P1 is smaller than the second change amount P2, the determination unit 230 selects the first estimation method (step S33). On the other hand, if the determination unit 230 determines that the second change amount P2 is smaller than the first change amount P1, the determination unit 230 selects the second estimation method (step S35).

  In this way, by comparing the magnitude relationship between the first change amount and the second change amount, an estimation method having a small change amount can be determined between the first estimation method and the second estimation method. Further, since the pixel value of the estimation target pixel can be estimated by an estimation method with a small change amount, the estimation error of the estimation target pixel can be reduced.

  Here, in the above embodiment, each unit configuring the maximum likelihood image estimation unit 120 is configured by hardware, but is not limited thereto. For example, it is configured such that the CPU processes each unit on an image acquired in advance using an imaging unit including the imaging lens 102, the mechanical shutter 103, and the imaging element 104, and is realized as software by the CPU executing a program. It is good to do. Alternatively, a part of processing performed by each unit may be configured by software.

  When the imaging unit is separated and the processing performed by each unit of the maximum likelihood image estimation unit 120 is realized as software, a known computer system such as a workstation or a personal computer can be used as the image processing apparatus. Then, a program (image processing program) for realizing processing performed by each unit of the maximum likelihood image estimation unit 120 is prepared in advance, and this image processing program is executed by the CPU of the computer system. In this case, for example, the CPU executes the processes of the flowcharts shown in FIGS. 11, 12, 16, and 17A.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, in the specification or the drawings, terms (addition, addition pixel, intermediate estimated pixel value, final estimation) described at least once together with different terms (mixed, mixed pixel, intermediate pixel value, estimated pixel, etc.) having a broader meaning or the same meaning Pixel etc.) may be replaced with the different terms anywhere in the specification or drawings. Further, the configurations and operations of the maximum likelihood image estimation unit, the imaging device, and the like are not limited to those described in the present embodiment, and various modifications can be made.

101 body, 102 imaging lens, 103 mechanical shutter, 104 imaging device,
107 A / D converter, 108 mechanical shutter drive unit, 109 image sensor drive unit,
110 memory, 111 AE processing unit, 112 AF processing unit,
113 system controller, 114 image processing unit, 115 compression / decompression unit,
116 system bus, 117 lens control unit, 118 UI control unit,
119 display control unit, 120 maximum likelihood image estimation unit, 121 recording medium control unit,
124 recording medium, 130 image acquisition unit, 200 estimation calculation unit, 210 image output unit,
220 pixel value estimation unit, 230 determination unit,
a _ij addition pixel value, b _ij , c _ij intermediate estimation pixel value, v _ij estimation pixel value,
δi ₀ difference value, E _j evaluation function, f _{n to} f _{n + 4} frames,
PD ₁ -PD _{n + 2} image data, T _n imaging operation period, D1 to D4 difference values,
P1 first change amount, P2 second change amount

Claims (9)

A light receiving unit, which is a unit for obtaining an added pixel value, is set for each of a plurality of pixels of the image sensor, and pixel values of a plurality of pixels included in the light receiving unit are added while sequentially shifting the pixels while superimposing the light receiving units. An image acquisition unit for acquiring the added pixel value,
A storage unit for storing a low-resolution frame image based on the added pixel value;
Based on a plurality of low-resolution frame images stored in the storage unit, an estimation calculation unit that estimates a pixel value of each pixel included in the light receiving unit;
An image output unit that outputs a high-resolution frame image having a higher resolution than the low-resolution frame image based on the pixel value estimated by the estimation calculation unit;
Including
The estimation calculation unit includes:
An imaging apparatus, wherein a change amount of adjacent addition pixel values among a plurality of addition pixel values used for estimation of an estimation target pixel is obtained, and a pixel value of the estimation target pixel is estimated based on the change amount. In claim 1,
The estimation calculation unit includes:
As a change amount of the adjacent addition pixel values, a difference between the addition pixel values adjacent in the horizontal direction among the plurality of addition pixel values and a difference between the addition pixel values adjacent in the vertical direction among the plurality of addition pixel values. An imaging device characterized by being obtained. In claim 1 or 2,
The estimation calculation unit includes:
A pixel value estimation unit for obtaining a plurality of candidate pixel values for the estimation target pixel;
A determination unit that determines a candidate pixel value having a small amount of change from the plurality of candidate pixel values, and outputs the determined candidate pixel value as a pixel value of the estimation target pixel;
An imaging device comprising: In claim 3,
The pixel value estimating unit
Estimating the plurality of candidate pixel values using the summed pixel values in different combinations for each candidate pixel value;
The determination unit
The change amount is obtained from the added pixel values used for the estimation of the plurality of candidate pixel values, a combination in which the change amount is reduced among the different combinations is determined, and a candidate pixel value corresponding to the determined combination is determined. An image pickup apparatus that outputs a pixel value of the estimation target pixel. In claim 1 or 2,
The estimation calculation unit includes:
A pixel value estimation unit that estimates a pixel value of the estimation target pixel by an estimation method selected from a plurality of estimation methods;
A determination unit that obtains the amount of change corresponding to each estimation method, and determines and selects an estimation method with a small amount of change among the plurality of estimation methods;
An imaging device comprising: In claim 5,
The pixel value estimating unit
From the first estimation method for high resolution in the vertical direction after high resolution in the horizontal direction and the second estimation method for high resolution in the horizontal direction after high resolution in the vertical direction, Estimating a pixel value of the estimation target pixel by the selected estimation method;
The determination unit
A first change amount that is the change amount of the addition pixel value adjacent in the horizontal direction and a second change amount that is the change amount of the addition pixel value adjacent in the vertical direction;
If it is determined that the first change amount is smaller than the second change amount, the first estimation method is selected,
An imaging apparatus characterized by selecting the second estimation method when it is determined that the second change amount is smaller than the first change amount. In any one of Claims 1 thru | or 6.
By the pixel shift, the light receiving unit is sequentially set to a first position and a second position next to the first position, and the light receiving unit of the first position and the light receiving unit of the second position. Are superimposed,
The pixel value estimating unit
Obtaining a difference value between the added pixel values of the first and second positions;
A first intermediate pixel value that is a light reception value of the first light receiving region obtained by removing the overlap region from the light reception unit of the first position, and a second value obtained by removing the overlap region from the light reception unit of the second position. A relational expression with the second intermediate pixel value that is a light reception value of the light receiving region is expressed using the difference value,
An imaging apparatus, wherein the first and second intermediate pixel values are estimated using the relational expression, and the pixel value of the estimation target pixel is obtained using the estimated first intermediate pixel value. In claim 7,
The pixel value estimating unit
When successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern, a relational expression between intermediate pixel values included in the intermediate pixel value pattern is expressed as the first and second It is expressed using the added pixel value of the position of
The intermediate pixel value pattern and the addition pixel represented by the relational expression between the intermediate pixel values when successive addition pixel values including the addition pixel values of the first and second positions are used as the addition pixel value pattern. Compare with value patterns to evaluate similarity,
An imaging device, wherein an intermediate pixel value included in the intermediate pixel value pattern is determined based on the similarity evaluation result so that the similarity becomes the highest. In claim 8,
The pixel value estimating unit
An evaluation function representing an error between the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the addition pixel value pattern is obtained, and the intermediate pixel value pattern is set so that the value of the evaluation function is minimized. An image pickup apparatus that determines an intermediate pixel value to be included.

Images