WO2012011484A1 - Image capture device - Google Patents

Abstract

The objective of the present invention is to provide an image capture device which allows simple exposure control. An image capture device of the present invention includes an imaging element (103) for capturing a subject image, a readout control unit (160) which takes the weighted sum of pixel values of a plurality of pixels of the imaging element (103) to read out thereof as a summed pixel value, a coefficient setting unit (130) for setting a weighting coefficient in the weighted summing, and an exposure control information output unit (140) for outputting exposure control information for performing exposure control for an image capture unit (100) on the basis of the weighting coefficient.

Description

Imaging device

The present invention relates to an imaging device and the like.

Some modern digital cameras and video cameras can be used by switching between still image shooting mode and movie shooting mode. For example, there is a camera that can shoot a still image with a resolution higher than that of a moving image by a user operating a button during moving image shooting.

JP 2008-147715 A JP 2007-295401 A

However, with this method, moving image shooting is interrupted when shooting a high-resolution still image. Alternatively, if the moving image shooting is performed at a resolution equivalent to a still image in order not to interrupt the moving image shooting, the frame rate of the moving image is lowered. As described above, there is a problem of achieving both high resolution still image shooting and high frame rate moving image shooting.

In order to solve this problem, the present inventor is considering generating a high-resolution still image from a low-resolution moving image by using a method of addition reading. Specifically, at the time of moving image shooting, the pixel values of a plurality of pixels are weighted and added and read out from the image sensor, and a high resolution image is restored from the pixel values obtained by the weighted addition.

However, with this method, there is a problem in that the range of pixel values read out by addition is different when the weighting coefficient is different. For example, in the automatic exposure control, different pixel values are read due to the difference in the weighting coefficient even when the exposure amount is the same, and thus different control is required depending on the weighting coefficient. This complicates the automatic exposure control.

Note that Patent Document 1 discloses a technique for mechanically shifting pixels of an optical system to perform moving image shooting and acquiring a high-definition image from the moving image. Patent Document 2 discloses a technique for performing exposure control according to a live view display gain.

According to some aspects of the present invention, it is possible to provide an imaging device or the like that enables simple exposure control.

One embodiment of the present invention sets an imaging element that captures a subject image, a reading control unit that performs weighted addition of pixel values of a plurality of pixels of the imaging element and reads the result as an added pixel value, and a weighting coefficient in the weighted addition. The present invention relates to an imaging apparatus including a coefficient setting unit and an exposure control information output unit that outputs exposure control information for performing exposure control of the imaging unit based on the weighting coefficient.

According to one aspect of the present invention, a weighting coefficient is set, weighted addition is performed using the weighting coefficient, an added pixel value is read, and exposure control information is output based on the weighting coefficient. Thereby, simple exposure control and the like are possible.

In the aspect of the invention, the coefficient setting unit sets a first weighting coefficient in the first imaging mode, sets a second weighting coefficient in the second imaging mode, and performs the exposure control. The information output unit obtains a ratio of the sum of the first weighting factors and the sum of the second weighting factors as a weighting factor ratio, and outputs the exposure control information using a photometric evaluation value based on the weighting factor ratio May be.

In this way, the exposure control information is output using the photometric evaluation value based on the weighting coefficient ratio between the first imaging mode and the second imaging mode, so that the exposure control for performing the exposure control of the imaging unit is performed. Information can be output.

In the aspect of the invention, the coefficient setting unit may set a coefficient having the same value as the first weighting coefficient for each pixel to be weighted and added in the first imaging mode. In the second mode, the second weighting coefficient different from the first weighting coefficient is set, and the exposure control information output unit obtains a photometric evaluation value from the added pixel value in the first imaging mode. Outputting the exposure control information using the obtained photometric evaluation value, and obtaining the photometric evaluation value from a pixel value obtained by multiplying the added pixel value by the weighting coefficient ratio in the second imaging mode, The exposure control information may be output using the obtained photometric evaluation value.

In this way, in the first imaging mode, a photometric evaluation value is obtained from the added pixel value that is not multiplied by the weighting factor ratio, and in the second imaging mode, the pixel obtained by multiplying the added pixel value by the weighting factor ratio. The photometric evaluation value is obtained from the value. This makes it possible to share exposure control in the first shooting mode and the second shooting mode, thereby simplifying exposure control.

Further, according to one aspect of the present invention, a display control unit that adjusts the luminance of the display image based on the weighting coefficient and performs control to display the adjusted display image may be included.

In this way, by adjusting the brightness of the display image based on the weighting coefficient, it is possible to perform display control with appropriate brightness independent of the weighting coefficient.

In the aspect of the invention, the coefficient setting unit sets a first weighting coefficient in the first imaging mode, sets a second weighting coefficient in the second imaging mode, and performs the display control. The unit may obtain a ratio of the sum of the first weighting coefficients and the sum of the second weighting coefficients as a weighting coefficient ratio, and may adjust the luminance of the display image based on the weighting coefficient ratio .

In this way, the brightness of the display image can be adjusted based on the weighting coefficient ratio between the first imaging mode and the second imaging mode.

In the aspect of the invention, the coefficient setting unit may set a coefficient having the same value as the first weighting coefficient for each pixel to be weighted and added in the first imaging mode. In the second mode, the second weighting coefficient different from the first weighting coefficient is set, and the display control unit is weighted and added by the first weighting coefficient in the first imaging mode. Control the display of the display image based on the added pixel value, and multiply the added pixel value weighted and added by the second weighting coefficient in the second imaging mode by the weighting coefficient ratio. You may perform control which displays the said display image by the said addition pixel value after.

In this way, in the first imaging mode, a display image with the added pixel value not multiplied by the weighting coefficient ratio is displayed, and in the second imaging mode, a display image with the added pixel value multiplied by the weighting coefficient ratio is displayed. Is displayed. Thereby, the display image can be adjusted to the same brightness in the first shooting mode and the second shooting mode.

In one aspect of the present invention, the light receiving unit is configured based on a storage unit that stores an image based on the added pixel value as a low-resolution frame image, and a plurality of low-resolution frame images stored in the storage unit. An estimation calculation unit that estimates a pixel value of each pixel included, and an image output that outputs a high-resolution frame image having a higher resolution than the low-resolution frame image based on the pixel value estimated by the estimation calculation unit The readout control unit sets a light receiving unit, which is a unit for obtaining the added pixel value, for each of a plurality of pixels of the imaging element, and sets pixel values of the plurality of pixels included in the light receiving unit. The weighted addition is performed, and the added pixel values are read while sequentially shifting the pixels while superimposing the light receiving units, and the estimation calculation unit is configured to obtain a plurality of added pixel values obtained by sequentially shifting the light receiving units. Based on, it may estimate the pixel value of each pixel included in the light receiving unit.

In this way, an additional pixel value is acquired while sequentially shifting pixels while superimposing light receiving units, and a low-resolution frame image based on the additional pixel value is acquired. Then, a pixel value is estimated based on the plurality of low resolution frame images, and a high resolution frame image is output based on the pixel values. Thereby, it is possible to obtain a high-resolution still image from a moving image by a simple process.

In the aspect of the invention, the light receiving unit is sequentially set to a first position and a second position next to the first position by the pixel shift, and the light receiving unit of the first position is set. And the light receiving unit of the second position overlap, the estimation calculation unit obtains a difference value between the added pixel values of the first and second positions, and overlaps the light receiving unit of the first position. A first intermediate pixel value that is a light receiving value of the first light receiving region excluding the light receiving value, and a second intermediate value that is a light receiving value of the second light receiving region excluding the overlapping region from the light receiving unit of the second position. A relational expression with a pixel value is expressed using the difference value, the first and second intermediate pixel values are estimated using the relational expression, and the light reception is performed using the estimated first intermediate pixel value. Even if the pixel value of each pixel included in the unit is obtained There.

In this way, it is possible to estimate the intermediate pixel value from the added pixel value read while sequentially shifting the pixels while superimposing the light receiving units, and obtain the final estimated pixel value from the estimated intermediate pixel value. Thereby, pixel value estimation of a high resolution frame image can be simplified.

In the aspect of the invention, the estimation calculation unit may be included in the intermediate pixel value pattern when successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern. A relational expression between intermediate pixel values is expressed using the added pixel values of the first and second positions, and successive added pixel values including the added pixel values of the first and second positions are added pixel value patterns. And comparing the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the added pixel value pattern to evaluate similarity, and based on the similarity evaluation result, The intermediate pixel value included in the intermediate pixel value pattern may be determined so that the similarity is the highest.

In this way, the intermediate pixel value can be estimated based on a plurality of added pixel values acquired by pixel shifting while superimposing the light receiving units.

In the aspect of the invention, the estimation calculation unit obtains an evaluation function representing an error between the intermediate pixel value pattern and the addition pixel value pattern expressed by a relational expression between the intermediate pixel values, and the evaluation The intermediate pixel value included in the intermediate pixel value pattern may be determined so that the function value is minimized.

In this way, by determining the value of the intermediate pixel value so that the value of the evaluation function is minimized, the value of the intermediate pixel value is set so that the similarity between the intermediate pixel value pattern and the added pixel value is the highest. Can be determined.

FIG. 1 is a comparative example of this embodiment. 2A and 2B are explanatory diagrams of a weighted addition method. FIG. 3A is an explanatory diagram of a range of added pixel values. FIG. 3B is an example of a program diagram for exposure control. FIG. 4 is an explanatory diagram of exposure control according to the present embodiment. FIG. 5 is a configuration example of the imaging apparatus according to the present embodiment. FIG. 6 is a flowchart of processing performed by the present embodiment. FIG. 7 is a flowchart of a modification. FIG. 8 is a flowchart of a second modification. FIG. 9 is a detailed explanatory diagram of the shooting mode. FIG. 10 is an explanatory diagram of addition readout control when performing pixel shift. FIG. 11 is an explanatory diagram of addition readout control when pixel shift is performed without weighting. FIG. 12 is an explanatory diagram of addition readout control when performing pixel shift with weighting. FIG. 13 is an explanatory diagram of addition readout control when pixel shift is not performed. FIG. 14A is an explanatory diagram of addition readout control when no pixel shift is performed without weighting. FIG. 14B is an explanatory diagram of addition readout control when weighted and pixel shift is not performed. FIG. 15 is an explanatory diagram of weighting coefficients. FIG. 16A is an explanatory diagram of an added pixel value and an estimated pixel value. FIG. 16B illustrates the intermediate pixel value. FIG. 17 is an explanatory diagram of an intermediate pixel value estimation method. FIG. 18 is an explanatory diagram of an intermediate pixel value estimation method. FIG. 19 is an explanatory diagram of an intermediate pixel value estimation method.

Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. First, a problem when a still image is shot during moving image shooting will be described, and a method in which the present embodiment acquires a high-resolution still image from a low-resolution moving image will be described.

FIG. 1 shows a comparative example of this embodiment. When the moving image switch 108 is turned on, the imaging apparatus starts moving image shooting, the imaging unit 100 captures an image, and the moving image signal processing circuit 116 processes the image to acquire moving image data. When the still image switch 109 is turned on during moving image capturing, the moving image capturing is temporarily stopped, the image capturing unit 100 captures an image, and the still image signal processing circuit 117 processes the image to acquire still image data.

For example, if a 12-megapixel high-pixel sensor can be driven at a high speed of 60 fps (fps: “frame” per-second), a 12-megapixel moving image can be captured, and one of them can be acquired as a still image. In this case, a high-resolution still image can be acquired without interrupting moving image shooting (image disappearance). However, there is no sensor capable of such driving at present. Also, recording a 12 megapixel moving image causes an increase in storage capacity, resulting in a decrease in recording time. Thus, there is a problem that it is difficult to achieve both high frame rate moving images and high resolution still images.

In response to this problem, the above-described Patent Document 1 discloses a shift unit that shifts the incident position of an optical image on an image sensor using a camera shake control signal from an optical camera shake control circuit and a pixel shift control signal from a pixel shift control circuit. Discloses a method of shifting pixels and obtaining a high-resolution still image from the pixel-shifted image.

However, in this method, since the pixel shift is performed by a mechanical method or an optical method, it is necessary to control the shift amount in the XY directions. For this reason, it is difficult to control and the influence on the readout time of the moving image is considered, which is not practical. Further, in order to achieve both camera shake control and pixel shift control, the control may be complicated. Also, with this method, the opportunity to obtain a high-resolution still image during moving image shooting is only when the shutter is pressed, and the shutter chance is limited.

Therefore, in this embodiment, mechanical shift is not performed, and pixel values of four pixels are added and read, and a moving image is shot while shifting the positions of the four pixels in each frame. Then, a high-resolution still image is estimated afterwards from the moving image obtained by the pixel shift, and it becomes possible to capture a high-resolution still image at a high frame rate and arbitrary timing. The estimation of the still image is performed by a method described later with reference to FIGS. 15 to 19, for example.

Also, in this embodiment, weighted addition is performed in reading out pixel values to further improve the reproducibility of high frequency components. However, when a different weighting coefficient is used depending on the mode, the range (signal level) of the pixel value obtained by addition and reading differs depending on the mode, which affects the automatic exposure control and the brightness of the live view display. There are challenges.

Next, the problem due to the weighted addition will be described in detail, and the exposure control and display control methods performed by the present embodiment will be described.

FIG. 2A and FIG. 2B schematically show a weighted addition method performed by this embodiment. As shown in FIG. 2A, in the normal moving image mode, the same weighting is applied to the four pixels, and the four-pixel added value a = 1 · v ₁ + 1 · v ₂ + 1 · v ₃ + 1 · v ₄ is acquired. In this case, the sum of the weighting coefficients (first weighting coefficients) is 1 + 1 + 1 + 1 = 4. On the other hand, in the fused moving image mode (for example, the moving image still image fused moving image 1 mode shown in FIG. 9), the same weighting is not applied to all four pixels, for example, a 4-pixel addition value a = 1 · v ₁ + 1/2 · v _2. + 1/2 · v ₃ + 1/4 · v ₄ is acquired. In this case, the sum of the weighting coefficients (second weighting coefficients) is 1 + 1/2 + 1/2 + 1/4 = 2.25.

As shown in FIG. 3A, for example, the range of pixel values v1 to v4 is set to 256 gradations. Then, as indicated by A1, the range of the 4-pixel added value a in the normal moving image mode is 256 × 4 = 1024 gradations. On the other hand, the range of the 4-pixel added value a in the fusion moving image mode is 256 × 2.25 = 576 gradations. As described above, when the weighting coefficients are different, the range of the 4-pixel addition value is different.

Fig. 3 (B) shows an example of a program diagram for exposure control. In this example, for simplicity, a photometric evaluation value is assumed to be a 4-pixel addition value, and an exposure time program diagram will be described as an example. As shown in FIG. 3B, when the 4-pixel addition value is 1024, the exposure time is controlled to be T1, and when the 4-pixel addition value is 576, the exposure time is T2. Controlled. In this way, if exposure control is performed with the same program diagram using the 4-pixel addition value of each mode as it is, even with the same exposure amount, the exposure time is controlled to be different. Further, in order to achieve the same exposure time, a program diagram is required for each mode, and the control becomes complicated.

Also, in live view display of moving images, if display is performed using the 4-pixel addition value of each mode as it is, the brightness of the display varies depending on the range of the 4-pixel addition value. That is, in the fused moving image mode, the display becomes darker by about 4 / 2.25 = 1.78 times = 5 dB than in the normal moving image mode.

Therefore, in the present embodiment, as shown in FIG. 4, the 4-pixel addition value in the fusion moving image mode is increased by 1.78 times, and exposure control is performed using the increased 4-pixel addition value. In the fusion video mode, a video with a gain-up 4-pixel addition value is displayed in live view. In this way, by adaptively increasing the gain according to the weighting coefficient, it becomes possible to simplify exposure control and display a moving image with appropriate brightness. The exposure control may be performed not only by controlling the exposure time but also by controlling the aperture value.

Note that the above-described Patent Document 2 discloses a technique that enables a photographer to display an image displayed in a live view with a desired brightness, and also allows an image in a more preferable exposure state to be captured. . However, this technique does not mention a form for capturing an image displayed in the live view and an image in a more preferable exposure state. That is, no mention is made of performing exposure control or display control according to the weighting coefficient.

2. Imaging Device FIG. 5 shows a configuration example of the imaging device of the present embodiment that performs gain control and exposure control by increasing the 4-pixel added value according to the weighting coefficient. The imaging apparatus includes an imaging unit 100, an A / D conversion unit 104, a user I / F unit 106, a control unit 113, and an imaging control unit 118.

In the following, a process that enables acquisition of a still image at the same time as recording a moving image is referred to as `` moving image still image fusion '', a mode for performing this processing is referred to as a `` fusion movie mode '', and a moving image suitable for this processing is referred to as `` This is called “fusion video”. This “fusion video” is an image that can generate a high-resolution still image. For example, it is acquired by a pixel shift described later in FIG. 10 and the like, and a still image is acquired by an estimation method described later in FIGS. Is possible.

The imaging control unit 118 includes an aperture control unit 120 and an imaging element control unit 119. The imaging control unit 118 drives and controls the imaging unit 100. The image sensor control unit 119 includes a read control unit 160 that controls the image sensor 103 and controls reading of pixel values, and an exposure control unit 161 that controls exposure time.

The imaging unit 100 (optical system) is an optical system for performing imaging, and includes an imaging lens 101, an aperture 102, an imaging element 103 such as a CMOS sensor, and a shutter (not shown). In response to a command from the system controller 105, the diaphragm controller 120 drives the diaphragm 102 and the shutter, whereby the operations of the diaphragm 102 and the shutter are performed.

The A / D conversion unit 104 converts an analog signal obtained by imaging by the imaging unit 100 into digital data.

The system controller 105 controls each part of the imaging apparatus (system). The system controller 105 includes a coefficient setting unit 130 that sets a weighting coefficient used for weighted addition.

The user I / F unit 106 (switch unit) includes a mode switch 107 for setting a shooting mode by the user, a moving image switch 108 for instructing start / stop of moving image recording, and a still image switch 109 for instructing still image recording. For example, the user I / F unit 106 includes a touch panel, operation buttons, and the like.

The external memory 110 records captured video data and still image data. The display device 111 is, for example, a liquid crystal display device, and performs live view display and display of reproduced moving images and still images. The recording medium 112 is a medium for recording image data. The display device 111 and the recording medium 112 may be incorporated in the imaging device, or may be an external device that can be attached and detached by a USB or the like.

The control unit 113 (signal processing system) includes a system controller 105, a compression / decompression circuit 121 (compression / decompression circuit, compression / decompression unit), a recording medium I / F circuit 126 (recording medium I / F unit), and an exposure control information output unit. 140 (AE processing system), a still image processing unit 141 (signal processing system), a moving image processing unit 142 (signal processing system), and a display control unit 150 (display processing system). The control unit 113 performs processing of a captured image and control of each component.

The moving image processing unit 142 processes the moving image data from the A / D conversion unit 104. The moving image processing unit 142 includes an electronic image stabilization circuit 114, a line memory 115, and a moving image signal processing circuit 116.

The electronic image stabilization circuit 114 is an image stabilization circuit that electronically corrects camera shake by image processing. The line memory 115 holds image data for one line so that the electronic image stabilization circuit 114 performs a camera shake correction process of less than one pixel. The moving image signal processing circuit 116 performs processing such as luminance signal conversion and color difference signal conversion on the image data from the electronic image stabilization circuit 114.

The still image processing unit 141 processes still image data from the A / D conversion unit 104. The still image processing unit 141 includes a still image signal processing circuit 117, a high resolution processing circuit 127 (estimation calculation unit), and a frame memory 128 (storage unit).

The high resolution processing circuit 127 performs high resolution processing for resolving a moving image and estimating a still image. The frame memory 128 holds a frame image in order to perform resolution enhancement processing (estimation processing) by the high resolution processing circuit 127. The still image signal processing circuit 117 performs image processing on a still image that has been subjected to high resolution processing and image processing on a still image that has been shot in the normal still image mode. For example, the still image signal processing circuit 117 performs processing such as luminance signal conversion and color difference signal conversion on still image data.

The exposure control information output unit 140 outputs AE control information (exposure control information) for the imaging control unit 118 to perform AE control (exposure control, AE: Auto Exposure). Exposure control information output unit 1
40 includes an AE processing circuit 122 and an AE gain setting circuit 123.

The AE processing circuit 122 obtains an AE evaluation value from the digital image data from the A / D conversion unit 104. The system controller 105 controls the imaging control unit 118 based on this AE evaluation value, the aperture control unit 120 sets the aperture 102, and the imaging element control unit 119 sets the accumulation time of the imaging element 103. . In this way, the exposure is controlled to be appropriate. The AE gain setting circuit 123 sets an AE gain for adjusting the difference in the image signal range for AE processing in the normal moving image mode and the fused moving image mode. The AE processing circuit 122 obtains an AE evaluation value from the digital image data multiplied by the AE gain.

The display control unit 150 performs control to display a display image on the display device 111, and includes a display device control circuit 124 and a display gain setting circuit 125. In the following, the operation of the display control unit 150 when used as a monitor during recording will be described as an example.

The digital image data output from the A / D conversion circuit 104 is input to the display device control circuit 124 via the moving image signal processing circuit 116 and the system controller 105. The display device control circuit 124 performs control to send a display image with an appropriate signal level to the display device 111. The display gain setting circuit 125 sets a display gain for adjusting the difference in the range of the display image signal in the normal moving image mode and the fused moving image mode. The display device control circuit 124 performs control to display digital image data multiplied by the display gain.

The compression / decompression circuit 121 compresses still image data generated by the still image signal processing circuit 117, compresses moving image data generated by the moving image signal processing circuit 116, and compresses the compressed image data. Perform decompression processing. For example, the compression / decompression circuit 121 compresses image data into a JPEG image or compresses moving image data into an MPEG image.

The recording medium I / F circuit 126 controls reading and writing with respect to the recording medium 112. The system controller 105 performs read and write access to the external memory 110.

3. AE Control and Display Control Next, AE control and display control performed by the imaging apparatus will be described in detail. First, a problem of AE processing and display processing caused by the weighting coefficient described in FIG. 3B and the like will be described with a specific example, and then a flowchart of processing performed by the present embodiment will be described. Hereinafter, a calculation example in the case where the pixel value GR11 is calculated from the 4-pixel addition value gr11 described later with reference to FIG. 10 and the AE gain is set using the GR11 will be described.

When the weighting coefficients are W1, W2, W3, and W4, they are expressed by the following equation (1). The four-pixel addition value gr11 is expressed by the following expression (2). Here, r is a real number with r ≧ 1, and GR11, GR13, GR31, and GR33 are pixel values that are added and read.
W1 = 1, W2 = 1 / r, W3 = 1 / r, W4 = 1 / r ² (1)
W * gr11 = W1 * GR11 + W2 * GR13 +
W3 * GR31 + W4 * GR33 (2)

For example, when r = 2, the following expression (3) is established from the above expression (1). At this time, it is assumed that the following expression (4) is acquired as the 4-pixel addition value.
W1 = 1, W2 = 1/2, W3 = 1/2, W4 = 1/4 (3)
W * gr11 = 576 (4)

Here, it is assumed that although the weighted addition is actually performed, the AE control is performed without considering the weighting coefficient, and the pixel value GR11 is gained up to the saturation value. It is assumed that the saturation value of the pixel value GR11 is 1024. Further, it is assumed that GR11 = GR13 = GR31 = GR33 holds assuming that light of the same brightness strikes the entire surface of the image.

In this case, since the weighting coefficient is regarded as W1 = W2 = W3 = W4 = 1, GR11 = 576/4 = 144 is obtained from the above equation (4). In order to set the pixel value GR11 to 1024 using this average value, it is necessary to multiply 1024/144 = 7.11 times. However, since the actual pixel value is GR11 = 256 as shown in the following formula (5), if it is multiplied by 7.11, it becomes 256 * 7.11 = 1989, and the pixel value is equal to or higher than the saturation value. .
GR11 = 576 / (1 + 1/2 + 1/2 + 1/4) = 256 (5)

In this respect, in this embodiment, AE control is performed in consideration of the weighting coefficient, and 4 / (1 + 1/2 + 1/2 + 1/4) = 1.78 is set as the AE gain. Therefore, the gain of the four-pixel addition value gr11 is increased to 576 * 1.78 = 1024, and GR11 = 1024/4 = 256 is obtained. If the average value is used to set the pixel value GR11 to 1024, 1024/256 = 4 times, and AE control can be performed within the saturation value.

On the other hand, when weighting addition is not performed and r = 1 in the above equation (1), W1 = W2 = W3 = W4 = 1 in the first place, so W * gr11 = 1 * 256 * 4 = 1024. . Therefore, 1 is set to the AE gain.

Thus, in this embodiment, a value different from the AE gain when weighted addition is not performed is set as the AE gain when weighted addition is performed. Even in the display control, the same problem as described above occurs with respect to the brightness of the display. Therefore, the display gain is set according to the weighting coefficient. Although the case where GR11 is obtained from gr11 has been described above, the value of each pixel can be obtained similarly from the other four-pixel addition values.

Next, the processing of this embodiment for performing AE control and display control according to such weighting coefficients will be described in detail. FIG. 6 shows a flowchart of processing performed by the present embodiment.

As shown in FIG. 6, when this process is started, a mode is selected (step S1). When the normal moving image mode is selected, the aperture is controlled (step S2), and the exposure time of the image sensor is controlled (step S3). Next, pixel readout control is set to no weighting and no pixel shift (step S4), and 4-pixel addition readout is performed (step S5). Next, AE processing is performed without multiplying the 4-pixel addition value by AE gain (step S6), an AE evaluation value is obtained (step S7), and an aperture value and exposure time are set based on the AE evaluation value (step S7). S2, S3). Further, the live view display is controlled without applying the display gain (step S8), and the live view image is displayed (step S9). Further, control is performed to record the captured moving image (step S10), and recording is performed on the recording medium (step S11).

On the other hand, when the fused video mode is selected in step S1, the aperture is controlled (step S12), and the exposure time of the image sensor is controlled (step S13). Next, pixel readout control is set to weighted (step S14), and a weighting coefficient is set (step S15). Next, four pixels are added and read (step S16), the sum of the weighting coefficients is calculated (step S17), and the AE gain and the display gain are set (step S18). For example, in the above equation (1), if r = 3 in the fused movie mode and r = 1 in the normal movie mode, the AE gain and the display gain are set to 2.25 from the following equation (6).
(1 + 1 + 1 + 1) / (1 + 1/3 + 1/3 + 1/9) = 2.25
(6)

Next, the 4-pixel addition value is multiplied by the AE gain (step S19), AE processing is performed using the image (step S20), an AE evaluation value is obtained (step S21), and the aperture value is determined by the AE evaluation value. Settings and exposure time are set (steps S12 and S13). Further, the 4-pixel addition value is multiplied by the display gain (step S22), display control of the live view image is performed (step S23), and the live view image is displayed (step S24). Further, control is performed to record the captured moving image (step S25), and recording is performed on a recording medium (step S26). This recorded video is not multiplied by gain.

Here, the AE process is, for example, a process of setting an area for performing photometric evaluation on a captured image, or a process of setting an aperture value setting characteristic and an exposure time setting characteristic (program diagram) with respect to the exposure amount. . The AE evaluation value is a photometric evaluation value obtained based on the pixel value in the area set in the captured image.

4). Modified Example FIG. 7 shows a flowchart of a modified example of processing performed by the present embodiment. This modification is an example in which processing independent of the mode is made common. As shown in FIG. 7, when this process is started, the aperture is controlled (step S50), the exposure time of the image sensor is controlled (step S51), and the mode is selected (step S52).

When the normal moving image mode is selected, pixel readout control is set (step S53), and 4-pixel addition readout is performed (step S54). Next, AE control (steps S61, S62, S50, S51), display control (steps S64, S65), and recording control (steps S66, S67) are performed.

On the other hand, when the fused video mode is selected in step S52, pixel readout control is set (step S55), and a weighting coefficient is set (step S56). Next, four pixels are added and read (step S57), the sum of the weighting coefficients is calculated (step S58), and the AE gain and the display gain are set (step S59). Next, the 4-pixel addition value is multiplied by the AE gain (step S60), and AE control is performed using the image (steps S61, S62, S50, S51). Further, the display gain is multiplied by the 4-pixel addition value (step S63), and display control of the live view image is performed (steps S64 and S65). Further, control for recording a moving image (steps S66 and S67) is performed.

5. Second Modification FIG. 8 shows a flowchart of a second modification of the process performed by this embodiment. This modification is an example in which the weighting coefficient setting process is shared. As shown in FIG. 8, when this process is started, the aperture is controlled (step S100), the exposure time of the image sensor and the like are controlled (step S101), the readout control is set to weighted addition, and the mode is set according to the mode. The presence / absence of pixel shift is set (step S102).

Next, the weighting coefficient is set to the value shown in the above formula (1) (step S103). In the normal moving image mode, r = 1 is set, and in the fused moving image mode, r> 1 is set. Next, the pixel values are weighted (step S104), and four pixels are added and read (step S105). Next, a sum of weighting coefficients is calculated (step S106), and an AE gain and a display gain are set (step S107). In the normal moving image mode, the gain is 1, and in the fused moving image mode, the gain is a value represented by the following expression (7).
4 / (1 + 1 / r + 1 / r + 1 / r ² ) (7)

Next, the 4-pixel addition value is multiplied by the AE gain (step S108), and AE control is performed (steps S109, S110, S100, and S101). Further, the 4-pixel addition value is multiplied by the display gain (step S111), and display control is performed (steps S112 and S113). Further, control for recording a moving image (steps S114 and S115) is performed.

As described above, when a different weighting coefficient is used depending on the mode, the range of the pixel value obtained by addition and reading differs depending on the mode, which affects the automatic exposure control and the brightness of the live view display. There is a problem.

In this regard, according to the present embodiment, as illustrated in FIG. 5, the imaging device weights and adds the pixel values of the imaging element 103 that captures the subject image and a plurality of pixels of the imaging element 103 to obtain an added pixel value. A readout control unit 160 that reads out, a coefficient setting unit 130 that sets a weighting coefficient in weighted addition, an exposure control information output unit 140 that outputs exposure control information for performing exposure control of the imaging unit 100 based on the weighting coefficient, including.

This enables simple exposure control and the like. Specifically, by performing exposure control of the imaging unit 100 based on the weighting coefficient, it becomes possible to share exposure control without dividing exposure control for each shooting mode.

Here, the exposure control information is, for example, a photometric evaluation value, or information indicating an aperture value and an exposure time obtained from the photometric evaluation value. Exposure control is performed by setting the aperture and exposure time based on these pieces of information.

Note that the imaging unit 100 may be configured integrally with the imaging device such as a compact camera. Alternatively, as in the interchangeable lens type camera, the imaging unit 100 may be configured such that the imaging element 103 is integrated with the imaging device (body), and the interchangeable lens including the diaphragm 102 and the optical system 101 is configured separately.

In this embodiment, as described above in step S103 of FIG. 8, the coefficient setting unit 130 sets the first weighting coefficient in the first imaging mode (for example, the normal moving image mode). The coefficient setting unit 130 sets a second weighting coefficient in the second imaging mode (for example, the fused moving image mode). Then, as shown in the above equation (7), the exposure control information output unit 140 obtains the ratio of the sum of the first weighting coefficients and the sum of the second weighting coefficients as the weighting coefficient ratio. The exposure control information output unit 140 outputs exposure control information using a photometric evaluation value based on the weighting coefficient ratio.

In this way, the exposure control information for performing the exposure control of the imaging unit 100 can be output by outputting the exposure control information using the photometric evaluation value based on the weighting coefficient ratio.

More specifically, in the present embodiment, the coefficient setting unit 130 sets the same value coefficient W1 = W2 = W3 = W4 = 1 for each pixel to be weighted and added in the first imaging mode, In the second mode, W1 = 1, W2 = W3 = 1 / r, and W4 = 1 / r2 (r> 1) are set as coefficients that are not the same for all the pixels to be weighted and added. Then, as described above in steps S108 to S110 in FIG. 8, in the first imaging mode, the exposure control information output unit 140 obtains a photometric evaluation value from the added pixel value (such as gr11 described later in FIG. 10), and In the second imaging mode, a photometric evaluation value is obtained from a pixel value obtained by multiplying the added pixel value by a weighting coefficient ratio.

In this way, in the first imaging mode, a photometric evaluation value is obtained from the added pixel value that is not multiplied by the weighting factor ratio, and in the second imaging mode, the pixel obtained by multiplying the added pixel value by the weighting factor ratio. The photometric evaluation value is obtained from the value. Thereby, the AE process and the AE evaluation value calculation process (steps S109 and S110) can be made common in the first shooting mode and the second shooting mode. Thereby, exposure control and AE circuit can be shared.

In this embodiment, as shown in FIG. 5, the imaging apparatus includes a display control unit 150 that adjusts the luminance of the display image based on the weighting coefficient and performs control to display the adjusted display image.

This makes it possible to control live view display with appropriate brightness. That is, by adjusting the luminance of the display image based on the weighting coefficient, it is possible to perform display control with appropriate brightness that does not depend on the weighting coefficient.

In the present embodiment, as shown in the above equation (7), the display control unit 150 obtains the ratio of the sum of the first weighting coefficients and the sum of the second weighting coefficients as the weighting coefficient ratio. Then, the display control unit 150 adjusts the luminance of the display image based on the weighting coefficient ratio.

In this way, the brightness of the display image can be adjusted based on the weighting coefficient ratio between the first imaging mode and the second imaging mode.

More specifically, in the present embodiment, as described above in steps S111 to S113 in FIG. 8, the display control unit 150 performs the first weighting coefficient W1 in the first imaging mode (for example, the normal moving image mode). Control is performed to display a display image based on an added pixel value (gr11 or the like described later in FIG. 10) weighted and added by = W2 = W3 = W4 = 1. In the second imaging mode (for example, the fused video mode), the display control unit 150 performs weighted addition using the second weighting coefficients W1 = 1, W2 = W3 = 1 / r, W4 = 1 / r2 (r> 1). Control is performed to multiply the added pixel value by a weighting coefficient ratio and display a display image based on the added pixel value after the multiplication.

In this way, in the first imaging mode, a display image with the added pixel value not multiplied by the weighting coefficient ratio is displayed, and in the second imaging mode, a display image with the added pixel value multiplied by the weighting coefficient ratio is displayed. Is displayed. Thereby, the live view display can be adjusted and displayed with the same brightness in the first shooting mode and the second shooting mode.

6). Normal Movie Mode The shooting mode of the present embodiment will be described in detail with reference to FIG. First, the normal moving image mode will be described. This mode is a mode in which only a moving image is captured without performing still image shooting in the middle, and addition reading is performed without weighting and without pixel shift.

The operation in the normal moving image mode will be described by taking the imaging apparatus shown in FIG. 5 as an example. In this mode, the system controller 105 controls the imaging unit 100, the aperture control unit 120, the imaging element control unit 119, and the control unit 113 in accordance with the mode setting content shown in FIG. The system controller 105 performs various settings according to instructions from the mode switch 107.

The pixel addition signal without weighting and without superposition shift is read from the image sensor 103. The addition reading is performed by a method described later with reference to FIG.

The image used for AE control and the image used for display are images that do not gain up and are the same as the images recorded on the recording medium 112.

The subject image formed on the image sensor 103 is converted into an electrical signal and sequentially read out. The read image is converted into a digital image by the A / D conversion unit 104 and then input to the electronic image stabilization circuit 114. The image subjected to the image stabilization process is processed by the moving image signal processing circuit 116 to generate a luminance color difference signal.

The image data from the moving image signal processing circuit 116 is held in the external memory 110. The image data held by the external memory 110 is output to the compression / decompression circuit 121 via the system controller 105 and converted into a format such as MPEG4 or Motion-JPEG. The converted image data is stored again in the external memory 110 via the system controller 105.

The compressed image data recorded in the external memory 110 is output to the recording medium I / F circuit 126 via the system controller 105 and recorded on the recording medium 112. It is also possible to record on the recording medium 112 without performing compression processing.

Next, AE control in the normal moving image mode will be described. The image converted into a digital image by the A / D conversion unit 104 is input to the AE processing circuit 122. An image from the AE processing circuit 122 is output to the system controller 105, and an AE evaluation value is obtained.

The aperture control unit 120 controls the aperture 102 using the AE evaluation value, and the image sensor control unit 119 performs accumulation time control of the image sensor 103 using the AE evaluation value. By these controls, AE control is performed so that an appropriate exposure value is obtained. The AE gain setting circuit 123 sets an AE gain (for example, 1) in the normal moving image mode, or does not set an AE gain in the normal moving image mode.

Next, display control in the normal moving image mode will be described. The image converted into a digital image by the A / D conversion unit 104 is input to the AE processing circuit 122. Based on the image from the AE processing circuit 122, the system controller 105 obtains an evaluation value for displaying on the display device 111 with appropriate brightness. The display control unit 150 performs display control based on the evaluation value. The display gain setting circuit 125 sets a display gain (for example, 1) in the normal moving image mode, or does not set a display gain in the normal moving image mode.

7. Normal Still Image Mode Next, the normal still image mode shown in FIG. 9 will be described. This mode is a mode for photographing only a still image, and is a mode for reading all pixels without performing addition reading.

The operation in the normal still image mode will be described using the imaging apparatus shown in FIG. 5 as an example. In the normal still image mode, the system controller 105 controls the imaging unit 100, the aperture control unit 120, the imaging element control unit 119, and the control unit 113 in accordance with the setting contents shown in FIG. In this mode, the estimation calculation process by the high resolution processing circuit 127 is not performed. The system controller 105 performs various settings according to instructions from the mode switch 107.

From the image sensor 103, an all-pixel readout signal with no weighting and no superposition shift is read out.

The image used for AE control and the image used for display are images that do not gain up and are the same as the images recorded on the recording medium 112.

The subject image formed on the image sensor 103 is converted into an electrical signal and sequentially read out. The read image is converted into a digital image by the A / D conversion unit 104 and then input to the high resolution processing circuit 127. In response to a control signal (command) from the system controller 105, the high resolution processing circuit 127 is set to OFF (non-operating state). The signal from the high resolution processing circuit 127 is processed by the still image signal processing circuit 117 to generate a luminance color difference signal.

The image data from the still image signal processing circuit 117 is held in the external memory 110. The image data held by the external memory 110 is output to the compression / decompression circuit 121 via the system controller 105 and converted into a format such as RAW or JPEG. The converted image data is stored again in the external memory 110 via the system controller 105.

The compressed image data recorded in the external memory 110 is output to the recording medium I / F circuit 126 via the system controller 105 and recorded on the recording medium 112. It is also possible to record on the recording medium 112 without performing compression processing.

In the normal still image mode, an image is acquired by the same readout control as in the normal moving image mode during AE control, and AE control is performed using the image. Therefore, the AE control is the same control as the AE control in the normal moving image mode. Similarly, the display control is the same control as the display control in the normal moving image mode because the live view image is acquired and displayed by the same readout control as in the normal moving image mode.

8). Movie Still Image Fusion Movie 1 Mode Next, the movie still image fusion movie 1 mode shown in FIG. 9 will be described. This mode is one of the fused moving image modes, in which a fused moving image for acquiring a still image is recorded as a moving image, and a still image is not estimated. In this mode, pixel shift readout is not performed. Note that it is possible to estimate and acquire a high-resolution still image from the fused video shot in this mode after the end of shooting.

The operation in the moving image still image fusion moving image 1 mode will be described using the image pickup apparatus shown in FIG. 5 as an example. In this mode, the system controller 105 controls the imaging unit 100, the aperture control unit 120, the imaging element control unit 119, and the control unit 113 in accordance with the mode setting content shown in FIG. The system controller 105 performs various settings according to instructions from the mode switch 107.

From the image sensor 103, a weighted pixel addition signal with a superimposed shift is read out. The addition reading is performed by a method described later with reference to FIG. For example, the weighting of the pixel value is realized by a method of changing the gain for each pixel. Specifically, when each pixel has an A / D conversion circuit, weighting is performed during A / D conversion. Alternatively, the pixel readout circuit may be weighted in an analog manner by giving a gain, or may be weighted by digital processing after A / D conversion.

The subject image formed on the image sensor 103 is converted into an electrical signal and sequentially read out. The read image is converted into a digital image by the A / D conversion unit 104 and then input to the electronic image stabilization circuit 114. The image subjected to the image stabilization process is processed by the moving image signal processing circuit 116 to generate a luminance color difference signal.

The image data from the moving image signal processing circuit 116 is held in the external memory 110. The image data held by the external memory 110 is output to the compression / decompression circuit 121 via the system controller 105 and converted into a format such as MPEG4 or Motion-JPEG. The converted image data is stored again in the external memory 110 via the system controller 105.

The compressed image data recorded in the external memory 110 is output to the recording medium I / F circuit 126 via the system controller 105 and recorded on the recording medium 112. It is also possible to record on the recording medium 112 without performing compression processing.

Next, AE control in the moving image still image fusion moving image 1 mode will be described. The image converted into a digital image by the A / D conversion unit 104 is input to the AE processing circuit 122. An image from the AE processing circuit 122 is output to the system controller 105, and an AE evaluation value is obtained.

The aperture control unit 120 controls the aperture 102 using the AE evaluation value, and the image sensor control unit 119 performs accumulation time control of the image sensor 103 using the AE evaluation value. By these controls, AE control is performed so that an appropriate exposure value is obtained.

In this AE control, the AE gain setting circuit 123 sets the AE gain. That is, the AE image in the normal moving image mode is a pixel addition signal without weighting and without a superimposition shift, whereas the AE image in the moving image still image fusion moving image 1 mode is a pixel addition signal with weighting and a superimposition shift. Therefore, as described above, the range (value) of the signal after addition differs between the weighted addition signal and the weighted addition signal. The AE gain setting circuit 123 sets the AE gain (for example, 1.78), thereby aligning the pixel addition signal ranges in the normal moving image mode and the moving image still image fusion moving image 1 mode.

Next, display control in the normal moving image mode will be described. The image converted into a digital image by the A / D conversion unit 104 is input to the AE processing circuit 122. A coefficient having the same value as the coefficient input to the AE gain setting circuit 123 set by the coefficient setting unit 130 in the system controller 105 or a coefficient suitable for the display device 111 is input to the display gain setting circuit 125. . The display gain setting circuit 125 sets the display gain to match the characteristics of the display device 111 using the same value as the coefficient input to the AE gain setting circuit 123. Then, the display device control circuit 124 performs control to display a moving image on the display device 111 with appropriate brightness by performing display control using the display gain.

In this display control, the display gain setting circuit 125 sets the display gain. That is, as described above, the added signal range differs between the weighted addition signal and the unweighted addition signal. The display gain setting circuit 125 sets the display gain (for example, 1.78), thereby aligning the pixel addition signal ranges in the normal moving image mode and the moving image still image fusion moving image 1 mode.

9. Movie Still Image Fusion Movie 2 Mode Next, the movie still image fusion movie 2 mode shown in FIG. 9 will be described. This mode is one of the fused video modes, in which the fused video is recorded as a video and no still image is estimated. In this mode, pixel shift readout is performed. Note that it is possible to estimate and acquire a high-resolution still image from the fused video shot in this mode after the end of shooting. Here, description of operations similar to those described in the moving image still image fusion moving image 1 mode will be omitted as appropriate.

The operation in the moving image still image fusion moving image 2 mode will be described using the image pickup apparatus shown in FIG. 5 as an example. In this mode, the system controller 105 controls the imaging unit 100, the aperture control unit 120, the imaging element control unit 119, and the control unit 113 in accordance with the mode setting content shown in FIG. The system controller 105 performs various settings according to instructions from the mode switch 107.

From the image sensor 103, a weighted pixel addition signal with a superimposed shift is read out. The addition reading is performed by a method described later with reference to FIG. The weighted addition is realized in the same manner as the method described in the moving image still image fusion moving image 1 mode.

AE control such as AE gain setting and display control such as display gain setting are performed in the same manner as described in the video still image fusion video 1 mode.

10. Moving Image Still Image Fusion Still Image 1 Mode Next, the moving image still image fusion still image 1 mode shown in FIG. 9 will be described. This mode is one of the fused video modes, in which a fused video is shot by pixel shift readout, and a high-resolution still image is acquired from the fused video. In addition, about the operation | movement similar to the operation | movement demonstrated in the moving image still image fusion moving image 1 mode, description is abbreviate | omitted suitably.

In this mode, high-resolution still image processing is performed by the high-resolution processing circuit 127. That is, the high resolution processing circuit 127 is turned on (operating state) by the control signal from the system controller 105. The image from the A / D conversion unit 104 is temporarily stored in the frame memory 128 and subjected to high resolution processing by the high resolution processing circuit 127. For example, the process of estimating a still image is performed by the method described later with reference to FIGS. Alternatively, the process may be performed by other methods such as a known super-resolution process. The image from the high resolution processing circuit 127 is processed by the still image signal processing circuit 117 to generate a luminance color difference signal.

The image data from the still image signal processing circuit 117 is held in the external memory 110. The image data held by the external memory 110 is output to the compression / decompression circuit 121 via the system controller 105 and converted into a format such as RAW or JPEG. The converted image data is stored again in the external memory 110 via the system controller 105.

The compressed image data recorded in the external memory 110 is output to the recording medium I / F circuit 126 via the system controller 105 and recorded on the recording medium 112. It is also possible to record on the recording medium 112 without performing compression processing.

AE control such as AE gain setting and display control such as display gain setting are performed in the same manner as described in the video still image fusion video 1 mode.

11. Moving Image Still Image Fusion Still Image 2 Mode Next, the moving image still image fusion still image 2 mode shown in FIG. 9 will be described. This mode is one of the fused video modes, and is a mode in which a fused video is shot without pixel shift and a high resolution still image is acquired from the fused video. In addition, about the operation | movement similar to the operation | movement demonstrated in the moving image still image fusion moving image 2 mode, description is abbreviate | omitted suitably.

In this mode, high-resolution still image processing is performed by the high-resolution processing circuit 127. That is, the high resolution processing circuit 127 is turned on (operating state) by the control signal from the system controller 105. The image from the A / D conversion unit 104 is temporarily stored in the frame memory 128 and subjected to high resolution processing by the high resolution processing circuit 127. For example, the process of estimating a still image is performed by the method described later with reference to FIGS. Alternatively, the process may be performed by other methods such as a known super-resolution process. In this mode, since pixel shift is not performed, a pixel value corresponding to the pixel shift is obtained by performing interpolation processing on the 4-pixel addition value captured in each frame. Alternatively, in this mode, interpolation processing is not performed, and high resolution processing may be performed by a technique such as edge enhancement. The image from the high resolution processing circuit 127 is processed by the still image signal processing circuit 117 to generate a luminance color difference signal.

AE control such as AE gain setting and display control such as display gain setting are performed in the same manner as described in the video still image fusion video 1 mode.

12 Next, a method for performing addition reading from the image sensor will be described in detail. Note that the frame used in the following description is, for example, the timing at which one (one) image is captured by the image sensor or the timing at which one image is processed in image processing. Alternatively, one image in the image data is also referred to as a frame as appropriate.

FIG. 10 shows an explanatory diagram in the case where pixel shift is performed at one pixel pitch in each frame. This read control is performed in the above-described moving image still image fusion moving image 1 mode and moving image still image fusion still image 1 mode. Hereinafter, a case where the image sensor is a Bayer array color image sensor will be described as an example.

As shown in FIG. 10, in the nth frame fn (n is a natural number), the 4-pixel addition value shown in the following equation (8) is read. Here, W1, W2, W3 and W4 are weighting coefficients shown in the above equation (1). GRij and GBij (i and j are natural numbers) represent green pixel values, Rij represents a red pixel value, and Bij represents a blue pixel value. Further, grij and grij represent a green 4-pixel addition value, rij represents a red 4-pixel addition value, and bij represents a blue 4-pixel addition value.
gr11 = W1 * GR11 + W2 * GR13 +
W3 * GR31 + W4 * GR33,
r12 = W1 * R12 + W2 * R14 +
W3 * R32 + W4 * R34,
b21 = W1 * B21 + W2 * B23 +
W3 * B41 + W4 * B43,
gb22 = W1 * GB22 + W2 * GB24 +
W3 * GB42 + W4 * GB44 (8)

In the (n + 1) th frame fn + 1, when viewed from the same color pixel, the pixel is shifted in the horizontal direction by one pixel, and the four-pixel addition value shown in the following equation (9) is read out.
gr13 = W1 * GR13 + W2 * GR15 +
W3 * GR33 + W4 * GR35,
r14 = W1 * R14 + W2 * R16 +
W3 * R34 + W4 * R36,
b23 = W1 * B23 + W2 * B25 +
W3 * B43 + W4 * B45,
gb24 = W1 * GB24 + W2 * GB26 +
W3 * GB44 + W4 * GB46 (9)

In the (n + 2) th frame fn + 2, when viewed from the same color pixel, the pixel is shifted in the horizontal direction and the vertical direction by one pixel, and the 4-pixel addition value shown in the following equation (10) is read out.
gr33 = W1 * GR33 + W2 * GR35 +
W3 * GR53 + W4 * GR55,
r34 = W1 * R34 + W2 * R36 +
W3 * R54 + W4 * R56,
b43 = W1 * B43 + W2 * B45 +
W3 * B63 + W4 * B65,
gb44 = W1 * GB44 + W2 * GB46 +
W3 * GB64 + W4 * GB66 (10)

In the (n + 3) th frame fn + 3, when viewed from the same color pixel, the pixel is shifted in the vertical direction by one pixel, and the four-pixel addition value shown in the following equation (11) is read out.
gr31 = W1 * GR31 + W2 * GR33 +
W3 * GR51 + W4 * GR53,
r32 = W1 * R32 + W2 * R34 +
W3 * R52 + W4 * R54,
b41 = W1 * B41 + W2 * B43 +
W3 * B61 + W4 * B63,
gb42 = W1 * GB42 + W2 * GB44 +
W3 * GB62 + W4 * GB64 (11)

In addition, although a part of pixels of the image sensor have been described above, the same applies to the readout control of other pixels. For example, in the frame fn, gr15, gr19, etc. are read out similarly to gr11.

FIG. 11 is an explanatory diagram in the case of r = 1 in the above addition read control. From the above equation (1), when r = 1, W1 = W2 = W3 = W4 = 1. As shown in FIG. 11, for example, it is assumed that the pixel values of all the pixels of the image sensor are 256. In this case, 1024 is read as each 4-pixel addition value from the above equations (8) to (11).

FIG. 12 shows an explanatory diagram in the case of r = 2 in the above addition read control. From the above equation (1), when r = 2, W1 = 1, W2 = W3 = 1/2, and W4 = 1/4. As shown in FIG. 12, for example, if the pixel values of all the pixels of the image sensor are 256, for example, in the frame fn, the weighted pixel values are W1 * GR11 = 256, W2 * GR13 = W3 * GR31. = 128, W4 * GR33 = 64 etc. In this case, 576 is read as each 4-pixel addition value from the above equations (8) to (11).

13. Addition Read Control when Pixel Shift is not Performed FIG. 13 is an explanatory diagram showing the addition read control when pixel shift is not performed. This reading control is performed in the above-described normal moving image mode, moving image still image fusion moving image 2 mode, and moving image still image fusion still image 2 mode.

As shown in FIG. 13, the 4-pixel addition value grij shown in the following equation (12), the 4-pixel addition value rij shown in the following equation (13), the 4-pixel addition value bij shown in the following equation (14), and the following equation (15 4 pixel added value gbij shown in FIG.
gr11 = W1 * GR11 + W2 * GR13 +
W3 * GR31 + W4 * GR33,
gr13 = W1 * GR15 + W2 * GR17 +
W3 * GR35 + W4 * GR37,
gr31 = W1 * GR51 + W2 * GR53 +
W3 * GR71 + W4 * GR73,
gr33 = W1 * GR55 + W2 * GR57 +
W3 * GR75 + W4 * GR77 (12)

r12 = W1 * R12 + W2 * R14 +
W3 * R32 + W4 * R34,
r14 = W1 * R16 + W2 * R18 +
W3 * R36 + W4 * R38,
r32 = W1 * R52 + W2 * R54 +
W3 * R72 + W4 * R74,
r34 = W1 * R56 + W2 * R58 +
W3 * R76 + W4 * R78 (13)

b21 = W1 * B21 + W2 * B23 +
W3 * B41 + W4 * B43,
b23 = W1 * B25 + W2 * B27 +
W3 * B45 + W4 * B47,
b41 = W1 * B61 + W2 * B63 +
W3 * B81 + W4 * B83,
b43 = W1 * B65 + W2 * B67 +
W3 * B85 + W4 * B87 (14)

gb22 = W1 * GB22 + W2 * GB24 +
W3 * GB42 + W4 * GB44,
gb24 = W1 * GB26 + W2 * GB28 +
W3 * GB46 + W4 * GB48,
gb42 = W1 * GB62 + W2 * GB64 +
W3 * GB82 + W4 * GB84,
gb44 = W1 * GB66 + W2 * GB68 +
W3 * GB86 + W4 * GB88 (15)

FIG. 14A shows an explanatory diagram in the case of r = 1 in the above addition read control. From the above equation (1), when r = 1, W1 = W2 = W3 = W4 = 1. As shown in FIG. 14A, for example, it is assumed that the pixel values of all the pixels of the image sensor are 256. In this case, 1024 is read as each 4-pixel addition value from the above equations (12) to (15).

FIG. 14B shows an explanatory diagram in the case of r = 2 in the above addition read control. From the above equation (1), when r = 2, W1 = 1, W2 = W3 = 1/2, and W4 = 1/4. As shown in FIG. 14B, for example, if the pixel values of all the pixels of the image sensor are 256, the weighted pixel values are, for example, W1 * GR11 = 256, W2 * GR13 = W3 * GR31 = 128, W4 * GR33 = 64, etc. In this case, 576 is read as each 4-pixel addition value from the above equations (12) to (15).

14 Estimation Process A process for estimating a high-resolution image from the weighted addition pixel value will be described with reference to FIGS.

The light receiving unit (pixel group) used in the following description represents an area on the image sensor including a plurality of pixels to be added and read, and pixel values of a plurality of pixels included in the light receiving unit are weighted and added. Thus, the added pixel value is acquired.

In the following description, when the pixels of the image sensor are arranged in an orthogonal two-axis coordinate system, a direction along one axis is referred to as a horizontal direction, and a direction along the other axis is referred to as a vertical direction. . For example, the horizontal direction is the horizontal scanning direction in the imaging operation. In the following description, as to the direction on the image data, the direction along one of the two orthogonal axes is referred to as a horizontal direction, and the direction along the other axis is referred to as a vertical direction.

As shown in FIG. 15, the weighting factors for addition reading are c ₁ , c ₂ , c ₃ , and c ₄ . Assuming that c ₁ = 1, the weighting coefficient takes the ratio relationship rule shown in the following equation (16) (r is a real number where r ≧ 1).
c ₁ = 1, c ₂ = 1 / r, c ₃ = 1 / r, c ₄ = 1 / r ² (16)

In the following, in order to simplify the explanation, r = 2 is assumed and the following equation (17) is assumed.
c ₁ = 1, c ₂ = 1/2, c ₃ = 1/2, c ₄ = ¼ (17)

FIG. 16A is an explanatory diagram of light reception units. v _ij is an estimated pixel value estimated from the added pixel value, and is a pixel value corresponding to each pixel of the image sensor. The light reception unit is set for every four pixels of v _{ij, and} the 4-pixel addition value a _ij is acquired by reading from each light reception unit. Adjacent light receiving units have overlapping regions. For _{example, a 00} and _{a 10 are} superimposed in v _{10, v 01.}

For example, in the pixel shift mode, 4-pixel addition values a ₀₀ , a ₁₀ , a ₀₁ , and a ₁₁ are read in frames fn to fn + 3, respectively. In the mode in which no pixel shift is performed, the 4-pixel addition values a ₀₀ , a ₂₀ ,... Are read, and the 4-pixel addition values a ₁₀ , a ₀₁ , a ₁₁ are the surrounding 4-pixel addition values a _00. , A ₂₀ ,...

FIG. 16B illustrates an intermediate pixel value (intermediate estimated pixel value). In the estimation process of the present embodiment obtains the intermediate pixel value b _ij is first performed in the horizontal direction of the high-resolution, determining an estimated pixel value v _ij by Kokai Zoka the b _ij in the vertical direction. The intermediate pixel value b _ij corresponds to v _ij and v _{i (j + 1)} . _Bij adjacent to each other in the vertical direction has an overlapping region. For _{example, b 00} and _{b 01 are} superimposed in _{v 01.} In this embodiment, b _ij may be obtained by increasing the resolution in the vertical direction, and v _ij may be obtained by increasing the resolution in the horizontal direction.

As shown in FIG. 17, paying attention to the first horizontal row detected by weighted pixel addition superimposed shift sampling, the weighted pixel addition values are set to a ₀₀ , a ₁₀ , and a _{20 in the} order of shift. At this time, the following expression (18) holds.
a ₀₀ = c ₁ v ₀₀ + c ₂ v ₀₁ + c ₃ v ₁₀ + c ₄ v ₁₁
a ₁₀ = c ₁ v ₁₀ + c ₂ v ₁₁ + c ₃ v ₂₀ + c ₄ v ₂₁ (18)

Further, b ₀₀ , b ₁₀ , and b ₂₀ are defined as shown in the following expression (19), and the above expression (17) is substituted.
b ₀₀ = c ₁ v ₀₀ + c ₂ v ₀₁ = v ₀₀ + (1/2) v ₀₁
b ₁₀ = c ₁ v ₁₀ + c ₂ v ₁₁ = v ₁₀ + (1/2) v _{11
b 20 = c 1 v 20 +} c 2 v 21 = v 20 + (1/2) v 21 (19)

Next, when the above equation (18) is transformed using the above equations (17) and (19), the following equation (20) is established.
a ₀₀ = v ₀₀ + (1/2) v ₀₁ + (1/2) v ₁₀ + (1/4) v ₁₁
= B ₀₀ + (1/2) b ₁₀ ,
_{a 10 = v 10 + (1/2} ) v 11 + (1/2) v 20 + (1/4) v 21
= B ₁₀ + (1/2) b ₂₀ (20)

In the above equation (20), when a ₀₀ and a ₁₀ are multiplied by a predetermined coefficient (predetermined weighting coefficient) to obtain a difference δi ₀ and transformed using the above equation (19), the following equation (21) is established.
δi ₀ = a ₁₀ -2a _{00
= (1/2) v 20 + (} 1/4) v 21 - (2v 00 + v 01)
= (1/2) b ₂₀ -2b ₀₀ (21)

If b ₀₀ is an unknown number, intermediate pixel values b ₁₀ and b ₂₀ can be obtained as a function of b ₀₀ as shown in the following equation (22).
b ₀₀ = (unknown number),
b ₁₀ = 2 (a ₀₀ -b ₀₀ ),
b ₂₀ = 4b ₀₀ + 2δi ₀ = 4b ₀₀ +2 (a ₁₀ −2a ₀₀ ) (22)

In this way, a high-definition combination pattern of intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } is obtained with b ₀₀ as an unknown (initial variable). Similarly, combination patterns of intermediate pixel values {b ₀₁ , b ₁₁ , b ₂₁ }, {b ₀₂ , b ₁₂ , b ₂₂ } are obtained with b ₀₁ and b ₀₂ as unknowns in the second and third lines. It is done.

Next, a description will be given of a method of obtaining the unknown _{b 00.} As shown in FIG. 18, a pattern {a ₀₀ , a ₁₀ } based on sampling pixel values detected by weighted superimposition shift sampling is compared with a pattern based on intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ }. Then, an unknown number b ₀₀ that minimizes the error E is derived and set as the intermediate pixel value b ₀₀ .

At this time, as shown in the above equation (20), the sampling pixel values {a ₀₀ , a ₁₀ } are added values of adjacent values by different weighting of the intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ }. Therefore, even if these are simply compared, a correct estimated value cannot be obtained. Therefore, as shown in FIG. 18, the intermediate pixel values are weighted for comparison. Specifically, the intermediate pixel values weighted _{{b ij, b (i +} 1) j} _is, the use of it is _{c 3 = c 1/2,} c 4 = c 2/2, the following equation (23) You can see that it holds.
a _ij = b _ij + (1/2) b _{(i + 1) j} (23)

Considering the weighting by the above equation (23), the evaluation function Ej shown in the following equation (24) is obtained. Then, the similarity between the pattern {a ₀₀ , a ₁₀ } and the intermediate estimated pixel value {b ₀₀ , b ₁₀ , b ₂₀ } is evaluated using this evaluation function Ej.

Using the above equation (22), the evaluation function Ej is represented by a function with b ₀₀ as an initial variable. Therefore, as shown in FIG. 19, an unknown b ₀₀ (= α) that minimizes Ej is obtained, and the value of b ₀₀ can be determined. Then, the estimated value of b ₀₀ is substituted into the above equation (22) to obtain b ₁₀ and b ₂₀ . Since the range of values that b ₀₀ can take is 0 ≦ b ₀₀ ≦ a ₀₀ , the minimum value of the evaluation function Ej may be obtained within this range. Similarly, in the second and third lines, a combination pattern of intermediate pixel values {b ₀₁ , b ₁₁ , b ₂₁ } and {b ₀₂ , b ₁₂ , b ₂₂ } is obtained with b ₀₁ and b ₀₂ as unknowns. It is done.

_The processing for estimating the _{v ij} from _{b ij} is carried out in the same manner as method of estimating the _{b ij} from above _{a ij.} That is, the relational expression of v ₀₀ , v ₀₁ , v ₀₂ is obtained using the difference value of b ₀₀ , b ₀₁ with v ₀₀ as an unknown. Next, an error evaluation function of {v ₀₀ , v ₀₁ , v ₀₂ } and {b ₀₀ , b ₀₁ } is obtained, v ₀₀ that minimizes the evaluation function is obtained, and the obtained v ₀₀ is substituted into the relational expression. Then, v ₀₁ and v ₀₂ are obtained.

As described above, when shooting a still image during moving image shooting, there is a problem that it is difficult to achieve both a high frame rate moving image and a high-resolution still image. In addition, since the super-resolution processing is a heavy load processing, there is a problem that the scale of the processing circuit increases.

In this regard, according to the present embodiment, the light receiving unit is set for each of the plurality of pixels of the image sensor, and the pixel values of the plurality of pixels included in the light receiving unit are weighted and added to read as an added pixel value (light receiving value). The low resolution frame image is obtained. Then, the acquired low-resolution frame image is stored, and the pixel value of each pixel included in the light receiving unit is estimated based on the plurality of stored low-resolution frame images. Based on the estimated pixel value, a high-resolution frame image having a higher resolution than the low-resolution frame image is output. At this time, the low-resolution frame image is acquired by reading the added pixel value while sequentially shifting the pixels while superimposing the light receiving units. Then, the pixel value of each pixel included in the light reception unit is estimated based on a plurality of added pixel values obtained by sequentially shifting the light reception unit.

For example, as described above with reference to FIG. 16A, the light receiving unit is set for every four pixels. In the first frame, the additional pixel values a ₀₀ , a _{20 and the} like are added and read, and a low resolution frame image by a ₀₀ , a _{20 and the} like is acquired. Then, a low resolution frame image by a ₁₀ , a _30, etc., a low resolution frame image by a ₁₁ , a _31, etc., and a low resolution frame image by a ₀₁ , a _21, etc. are sequentially acquired. For example, the light receiving units for acquiring a ₀₀ , a ₁₀ , a ₁₁ , and a ₀₁ are shifted by one pixel and overlapped by two pixels. These images are stored in, for example, the frame memory 128 (storage unit) shown in FIG. Then, the estimated pixel value v _ij is estimated by the high resolution processing circuit 127 (estimation calculation unit). The still image signal processing circuit 117 (image output unit) processes v _{ij and} outputs a high resolution image corresponding to the resolution of the image sensor.

This makes it possible to obtain high-resolution images from moving images with simple processing. For example, the estimation process can be simplified using the above-described estimation of the intermediate pixel value. In addition, since the high-resolution still image can be generated at any timing of the low-resolution moving image, the user can easily obtain the high-resolution still image at the decisive moment. In addition, by capturing a low-resolution moving image at the time of shooting, it is possible to capture at a high frame rate and acquire a high-resolution still image as necessary.

More specifically, in the present embodiment, as shown in FIG. 16 (A), the light receiving unit is sequentially set to a first position a ₀₀ second position a ₁₀ follows. These light receiving units are overlapped in a region including v ₁₀ and v ₁₁ . Then, as described above with reference to FIG. 17, the difference value δi ₀ of the added pixel values obtained from these light receiving units is obtained. As shown in FIG. 16B, the first intermediate pixel value b ₀₀ is the light reception value of the first light receiving regions v ₀₀ and v _{01 obtained} by removing the overlapping regions v ₁₀ and v ₁₁ from the light receiving unit a _00. . The second intermediate pixel value _{b 20} is a received-light value in the second light receiving region _{v 20, v 21} excluding the overlapping area _{v 10, v 11} from the light receiving unit _{a 10.} Then, as shown in the above equation _(22), equation of _{b 00} and _{b 20} are represented by using a difference value .delta.i _0. The first and second intermediate pixel values b ₀₀ and b ₂₀ are estimated using the relational expression, and the pixel value of each pixel of the light receiving unit is obtained using the estimated first intermediate pixel value b ₀₀ .

In this way, it is possible to simplify the estimation process of the high-resolution image by once estimating the intermediate pixel value from the addition pixel value subjected to the superposition shift and obtaining the estimation pixel value from the intermediate pixel value subjected to the superposition shift. . For example, complicated processing such as a repetitive calculation of a two-dimensional filter (for example, Japanese Patent Laid-Open No. 2009-124621) and a search for a part suitable for setting an initial value (for example, Japanese Patent Laid-Open No. 2008-243037) becomes unnecessary. .

Further, in the present embodiment, as described above with reference to FIG. 18, successive intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } including the intermediate pixel values b ₀₀ and b ₂₀ are converted into intermediate pixel values. A pattern. The relational expression between the intermediate pixel values is expressed using the added pixel values a ₀₀ and a ₁₀ . Further, the addition pixel value pattern addition pixel values _{a 00, a} summing pixel values successive containing _{10 {a 00, a 10}} . Then, the similarity is evaluated by comparing the intermediate pixel value pattern with the added pixel value pattern, and based on the evaluation result, the intermediate pixel values b ₀₀ , b ₁₀ , and b ₂₀ are determined so that the similarity becomes the highest. Is done.

In this way, the intermediate pixel value can be estimated based on a plurality of added pixel values acquired by pixel shifting while superimposing the light receiving units.

More specifically, in the present embodiment, as shown in the above formulas (22) and (24), the intermediate pixel value pattern {b ₀₀ , b ₁₀ , b ₂₀ } represented by the relational expression between the intermediate pixel values. And an evaluation function Ej representing an error between the added pixel value pattern {a ₀₀ , a ₁₀ }. Then, as shown in FIG. 19, the intermediate pixel values b ₀₀ , b ₁₀ , and b ₂₀ are determined so that the value of the evaluation function Ej is minimized.

In this way, the value of the intermediate pixel value can be estimated by expressing the error by the evaluation function and obtaining the intermediate pixel value corresponding to the minimum value of the evaluation function. For example, as described above, the initial value of the intermediate pixel estimation can be set with a simple process by obtaining the unknown using the least square method.

Here, in the above-described embodiment, each part of the exposure control information output unit 140 and the display control unit 150 is configured by hardware. However, the present invention is not limited to this. For example, the CPU may be configured to perform processing of each unit, and may be realized as software by the CPU executing a program. In this case, the CPU executes, for example, the processing of the flowcharts shown in FIGS.

In the above embodiment, each unit constituting the still image processing unit 141 is configured by hardware. However, the present invention is not limited to this. For example, when a known computer system such as a personal computer is used as an image processing apparatus and a program for realizing processing performed by each unit of the still image processing unit 141 is executed by the CPU of the computer system, each unit of the still image processing unit 141 The processing performed by may be implemented as software.

Although the present embodiment has been described in detail as described above, it will be readily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, in the specification or the drawings, terms (AE control, 4-pixel addition values, etc.) described at least once together with different terms (exposure control, addition pixel value, etc.) in a broader sense or synonymous It can also be replaced with the different terminology. Also, the configuration and operation of the exposure control information output unit, display control unit, still image processing unit, control unit, imaging control unit, imaging unit, imaging device, etc. are not limited to those described in this embodiment, and various Can be implemented.

100 imaging unit, 101 imaging lens, 102 aperture, 103 imaging device,
104 A / D converter, 105 system controller,
106 User I / F section, 107 Mode switch, 108 Movie switch,
109 still image switch, 110 external memory, 111 display device,
112 recording medium, 113 control unit, 114 electronic image stabilization circuit,
115 line memory, 116 video signal processing circuit,
117 still image signal processing circuit, 118 imaging control unit,
119 Image sensor control unit, 120 Aperture control unit, 121 Compression / decompression circuit,
122 AE processing circuit, 123 AE gain setting circuit,
124 display device control circuit, 125 display gain setting circuit,
126 recording medium I / F circuit, 127 high resolution processing circuit,
128 frame memory, 130 coefficient setting unit,
140 exposure control information output unit, 141 still image processing unit,
142 video processing unit, 150 display control unit, 160 readout control unit,
161 exposure control unit,
a _ij pixel addition value, b _ij intermediate pixel value, δi ₀ difference value, Ej evaluation function,
v _ij estimated pixel value, fn frame, W1 to W4 weighting coefficient

Claims (10)

An image sensor for capturing a subject image;
A read control unit that weights and adds pixel values of a plurality of pixels of the image sensor and reads them as added pixel values;
A coefficient setting unit for setting a weighting coefficient in the weighted addition;
An exposure control information output unit that outputs exposure control information for performing exposure control of the imaging unit based on the weighting coefficient;
An imaging apparatus comprising: In claim 1,
The coefficient setting unit
In the first imaging mode, a first weighting factor is set,
In the second imaging mode, a second weighting factor is set,
The exposure control information output unit
A ratio of the sum of the first weighting coefficients and the sum of the second weighting coefficients is obtained as a weighting coefficient ratio, and the exposure control information is output using a photometric evaluation value based on the weighting coefficient ratio. An imaging device. In claim 2,
The coefficient setting unit
In the first imaging mode, a coefficient having the same value is set as the first weighting coefficient for each pixel to be weighted and added,
In the second mode, setting the second weighting factor different from the first weighting factor,
The exposure control information output unit
In the first imaging mode, a photometric evaluation value is obtained from the added pixel value, the exposure control information is output using the obtained photometric evaluation value,
In the second imaging mode, obtaining the photometric evaluation value from a pixel value obtained by multiplying the added pixel value by the weighting coefficient ratio, and outputting the exposure control information using the obtained photometric evaluation value. An imaging device that is characterized. In claim 1,
An imaging apparatus comprising: a display control unit configured to adjust a luminance of a display image based on the weighting coefficient and perform control to display the adjusted display image. In claim 4,
The coefficient setting unit
In the first imaging mode, a first weighting factor is set,
In the second imaging mode, a second weighting factor is set,
The display control unit
An imaging apparatus, wherein a ratio between a sum of the first weighting coefficients and a sum of the second weighting coefficients is obtained as a weighting coefficient ratio, and brightness of the display image is adjusted based on the weighting coefficient ratio. In claim 5,
The coefficient setting unit
In the first imaging mode, a coefficient having the same value is set as the first weighting coefficient for each pixel to be weighted and added,
In the second mode, setting the second weighting factor different from the first weighting factor,
The display control unit
In the first imaging mode, control is performed to display the display image by the added pixel value weighted and added by the first weighting coefficient,
In the second imaging mode, control for multiplying the added pixel value weighted and added by the second weighting factor by the weighting factor ratio and displaying the display image by the added pixel value after multiplication An imaging apparatus characterized by performing In any one of Claims 1 thru | or 6.
A storage unit for storing an image based on the added pixel value as a low-resolution frame image;
Based on a plurality of low-resolution frame images stored in the storage unit, an estimation calculation unit that estimates a pixel value of each pixel included in the light receiving unit;
An image output unit that outputs a high-resolution frame image having a higher resolution than the low-resolution frame image based on the pixel value estimated by the estimation calculation unit;
Including
The read control unit
A light receiving unit, which is a unit for obtaining the added pixel value, is set for each of a plurality of pixels of the image sensor, pixel values of a plurality of pixels included in the light receiving unit are weighted and added, and the light receiving units are sequentially superimposed. Read the added pixel value while shifting the pixel,
The estimation calculation unit includes:
An imaging apparatus, wherein a pixel value of each pixel included in the light receiving unit is estimated based on a plurality of added pixel values obtained by sequentially shifting the light receiving unit. In claim 7,
By the pixel shift, the light receiving unit is sequentially set to a first position and a second position next to the first position, and the light receiving unit of the first position and the light receiving unit of the second position. Are superimposed,
The estimation calculation unit includes:
Obtaining a difference value between the added pixel values of the first and second positions;
A first intermediate pixel value that is a light reception value of the first light receiving region obtained by removing the overlap region from the light reception unit of the first position, and a second value obtained by removing the overlap region from the light reception unit of the second position. A relational expression with the second intermediate pixel value that is a light reception value of the light receiving region is expressed using the difference value,
The first and second intermediate pixel values are estimated using the relational expression, and the pixel value of each pixel included in the light receiving unit is obtained using the estimated first intermediate pixel value. Imaging device. In claim 8,
The estimation calculation unit includes:
When successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern, a relational expression between intermediate pixel values included in the intermediate pixel value pattern is expressed as the first and second It is expressed using the added pixel value of the position of
The intermediate pixel value pattern and the addition pixel represented by the relational expression between the intermediate pixel values when successive addition pixel values including the addition pixel values of the first and second positions are used as the addition pixel value pattern. Compare with value patterns to evaluate similarity,
An imaging device, wherein an intermediate pixel value included in the intermediate pixel value pattern is determined based on the similarity evaluation result so that the similarity becomes the highest. In claim 9,
The estimation calculation unit includes:
An evaluation function representing an error between the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the addition pixel value pattern is obtained, and the intermediate pixel value pattern is set so that the value of the evaluation function is minimized. An image pickup apparatus that determines an intermediate pixel value to be included.

Images