JP2012231379A - Imaging apparatus and image generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus which is capable of capturing a high-resolution image from motion picture data of which the data amount has been reduced, through simple processing, and an image generation method or the like.SOLUTION: An imaging apparatus comprises an image capturing section, an inter-frame added image generating section, an inter-frame differential image generating section, an inter-frame added image reproducing section, an inter-frame estimate arithmetic section, and an image output section. The image capturing section captures picked-up images fto f. The inter-frame added image generating section generates an inter-frame added image Bby weight-adding pixel values of fand fand generates an inter-frame added image Bby weight-adding pixel values of fand f. The inter-frame differential image generating section generates a differential between Band Bas an inter-frame differential image DT. The inter-frame added image reproducing section reproduces Bon the basis of Band DT. The inter-frame estimate arithmetic section estimates fto fon the basis of Band B. The image output section outputs a high-resolution image based on the estimated image.

Description

  The present invention relates to an imaging apparatus, an image generation method, and the like.

  Some recent digital cameras and video cameras can be used by switching between a still image shooting mode and a moving image shooting mode. For example, there is one that can shoot a still image with a higher resolution than a moving image by a user operating a button during moving image shooting.

JP 2009-124621 A JP 2008-243037 A

  However, the method of switching between the still image shooting mode and the moving image shooting mode has a problem that when the user notices a photo opportunity, a decisive moment is often already missed.

  The present inventor considers generating a high-resolution still image at an arbitrary timing from moving image data with a reduced amount of data in order to realize shooting at the decisive moment. For example, Patent Documents 1 and 2 disclose a technique for synthesizing a high-resolution image from a low-resolution image acquired by pixel shift. However, this method requires imaging by pixel shift, which complicates the camera configuration. In addition, there is a problem that the load of high resolution processing is large and pixel value estimation may be difficult.

  According to some aspects of the present invention, it is possible to provide an imaging device, an image generation method, and the like that can acquire a high-resolution image by simple processing from moving image data with a reduced data amount.

  One aspect of the present invention is an image acquisition unit that acquires the first to third captured images in the first to third frames, the pixel value of the first captured image, and the pixel value of the second captured image. Are added to each other to generate a first inter-frame added image, and the pixel values of the second captured image and the pixel values of the third captured image are weighted to generate a second inter-frame added image. An inter-frame addition image generation unit; an inter-frame difference image generation unit that generates a difference between the first inter-frame addition image and the second inter-frame addition image as an inter-frame difference image; and Based on the addition image and the inter-frame difference image, an inter-frame addition image reproduction unit that reproduces the second inter-frame addition image, the first inter-frame addition image, and the reproduced second frame Based on the addition image An inter-frame estimation calculation unit that estimates pixel values of the first to third captured images, an image output unit that outputs a high-resolution image based on the pixel values estimated by the inter-frame estimation calculation unit, This relates to an imaging device including

  According to one aspect of the present invention, first to third captured images are acquired, and first and second inter-frame added images are generated by weighted addition of pixel values between frames, and the difference between the frames A difference image is generated. Then, the second inter-frame addition image is reproduced from the first inter-frame addition image and the inter-frame difference image, and the first to third captured images are estimated from the first and second inter-frame addition images. A high resolution image based on the captured image is output.

  Thereby, for example, by applying an estimation process described later, a captured image can be restored from the inter-frame addition image with a simple process. Since moving image shooting and still image shooting are not switched, a high-resolution still image at an arbitrary timing can be acquired from moving image data. Further, by obtaining the inter-frame addition image, the image data can be compressed at a high compression rate.

  In the aspect of the invention, the inter-frame estimation calculation unit obtains a difference value between a pixel value of the first inter-frame addition image and a pixel value of the second inter-frame addition image, and The relational expression between the pixel value of the captured image and the pixel value of the third captured image is expressed using the difference value, and the pixel values of the first to third captured images are expressed using the relational expression. It may be estimated.

  In the aspect of the invention, the inter-frame estimation calculation unit uses the pixel values of the first and second inter-frame addition images as the relational expression between the pixel values of the first to third captured images. Comparing the pixel values of the first to third captured images represented by the relational expressions with the pixel values of the first and second inter-frame addition images, and evaluating the similarity, Based on the similarity evaluation result, the pixel values of the first to third captured images may be estimated so that the similarity is the highest.

  In this way, the relational expression of the pixel value of the captured image can be expressed using the pixel value superimposed and added between the frames, and the pixel value of the captured image can be restored based on the relational expression. Thereby, the restoration estimation process for the inter-frame addition image can be simplified.

  In one embodiment of the present invention, the first inter-frame addition image or the inter-frame difference image is received as an input image, and an addition unit, which is a unit for obtaining an addition pixel value, is obtained for each of the plurality of pixels of the input image. An intra-frame addition image generation unit configured to weight and add pixel values included in the addition unit while sequentially shifting the addition units, and generating first to n-th intra-frame addition images; An average image generating unit that generates an average image of the n-th intra-frame addition image; an m-th intra-frame addition image (m is a natural number equal to or less than n) of the first to n-th intra-frame addition images; An intra-frame difference image generating unit that generates a difference from the average image as an m-th intra-frame difference image, a data compression unit that compresses the average image and the m-th intra-frame difference image, and the compressed A compressed data expansion unit that expands the average image and the m-th intra-frame difference image; and the first to n-th intra-frame addition images based on the expanded average image and the m-th intra-frame difference image May be included, and an intra-frame estimation calculation unit that estimates a pixel value of the input image based on the reproduced first to n-th intra-frame addition images.

  In this way, after performing addition between frames, addition within the frame is performed, and an average image and an intra-frame difference image can be generated and compressed from the intra-frame addition image. Thereby, the compression rate can be further improved. That is, since the intra-frame difference image is a difference between the average image and the intra-frame addition image, the entropy of the pixel value can be made smaller than that of the input image. Therefore, the intra-frame difference image can be compressed at a high compression rate by, for example, entropy encoding.

  In one aspect of the present invention, the intra-frame addition image generation unit sequentially shifts the addition unit one pixel at a time horizontally or vertically to set the first to n-th positions. The first to nth intra-frame addition images are acquired at the respective positions, and the addition unit of the mth position and the (m + 1) th position among the first to nth positions includes a common pixel. But you can.

  In this way, it is possible to obtain the added pixel value by sequentially shifting the addition unit while including the common pixels, and to construct the first to nth frame added images by the added pixel value.

  In the aspect of the invention, the data compression unit may reversibly compress the first to nth intra-frame difference images by entropy coding.

  In this way, it is possible to compress at a high compression rate by entropy encoding the intra-frame difference image having entropy smaller than the input image as described above.

  In the aspect of the invention, when the addition unit set in the first position and the addition unit set in the second position next to the first position overlap, the estimation calculation unit Is a difference value between the added pixel value of the first position and the added pixel value of the second position, and the added pixel value of the first area excluding the overlapping area from the addition unit of the first position. A relational expression between the first intermediate pixel value that is and the second intermediate pixel value that is the addition pixel value of the second region excluding the overlap region from the addition unit of the second position, and the difference value The first intermediate pixel value is estimated using the relational expression, and the pixel value of each pixel included in the addition unit is obtained using the estimated first intermediate pixel value. May be.

  In this way, the intermediate pixel value is estimated from the first to nth intraframe addition images that are sequentially pixel-shifted while the addition unit is superimposed, and the final estimated pixel value is obtained from the estimated intermediate pixel value. Can do. Thereby, the restoration estimation process for the intra-frame addition image can be simplified.

  In the aspect of the invention, the estimation calculation unit may include an intermediate included in the intermediate pixel value pattern when successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern. A relational expression between pixel values is represented using the addition pixel value, the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values is compared with the addition pixel value, and similarity is evaluated, Based on the similarity evaluation result, an intermediate pixel value included in the intermediate pixel value pattern may be determined so that the similarity is the highest.

  In this way, an intermediate pixel value can be obtained by estimation based on a plurality of added pixel values acquired by pixel shifting while the addition unit is superimposed.

  In one aspect of the present invention, an addition unit, which is a unit for obtaining an added pixel value in response to an x-th captured image (x is a natural number of 3 or less) among the first to third captured images. A frame that is set for each of a plurality of pixels of the x-th captured image and generates the first to nth intra-frame addition images by weighting and adding pixel values included in the addition units while sequentially shifting the addition units. An inner addition image generation unit; an average image generation unit that generates an average image of the first to nth intraframe addition images; and an mth intraframe addition among the first to nth intraframe addition images. An intra-frame difference image generation unit that generates a difference between the image and the average image as an m-th intra-frame difference image, and the inter-frame addition image generation unit includes the m-th intra-frame difference image or The average image is represented by first to third. Receiving the x-th input image of the input images, generating the first inter-frame addition image based on the first and second input images, and generating the first inter-frame image based on the second and third input images. Two inter-frame addition images may be generated.

  In this way, it is possible to generate the inter-frame difference image for each of the average image and the intra-frame difference image and compress it by performing the intra-frame addition after performing the intra-frame addition. Even in such a processing order, it is possible to obtain an entropy reduction effect by the intra-frame difference image, an effect of reducing a change in the pixel value by the inter-frame addition image, and an entropy reduction effect by the inter-frame difference image.

  In another aspect of the present invention, the first to third captured images in the first to third frames are acquired, and the pixel values of the first captured image and the pixel values of the second captured image are weighted. Adding to generate a first inter-frame added image, weighting and adding the pixel value of the second captured image and the pixel value of the third captured image to generate a second inter-frame added image, A difference between the first inter-frame addition image and the second inter-frame addition image is generated as an inter-frame difference image. Based on the first inter-frame addition image and the inter-frame difference image, the second Reproducing the inter-frame addition image, estimating the pixel values of the first to third captured images based on the first inter-frame addition image and the reproduced second inter-frame addition image, Pixel estimated by the inter-frame estimation calculation unit Related to Kokai image generation method for outputting an image image-based.

Explanatory drawing about the 1st data compression method. 1 is a first configuration example of an imaging apparatus. Explanatory drawing about the 2nd data compression method. An example of a Huffman code table. 2 shows a second configuration example of an imaging apparatus. 7 is a data configuration example in the second configuration example of the imaging apparatus. 7 is a data configuration example in the second configuration example of the imaging apparatus. 3rd structural example of an imaging device. Explanatory drawing about the 4th data compression method. Explanatory drawing about the 1st modification of a data compression method. Explanatory drawing about the 1st modification of a data compression method. Explanatory drawing about the 2nd modification of a data compression method. Explanatory drawing about the 2nd modification of a data compression method. Explanatory drawing about the pixel value of the addition image between frames. Explanatory drawing about a restoration estimation process. Explanatory drawing about a restoration estimation process. Explanatory drawing about a restoration estimation process. 18A and 18B are explanatory diagrams of the pixel value and intermediate pixel value of the intra-frame added image.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. Outline of this embodiment Digital camera and video camera products have a digital camera that mainly shoots still images with a video shooting function, or a video camera that mainly shoots video with a still image shooting function. There is something. If such a device is used, there is the convenience that a still image and a moving image can be shot with a single device.

  However, with this method, there is a problem that it is difficult to obtain a high-quality still image without missing the shutter chance that many people demand. For example, in a method of instantaneously switching to a mode for shooting a high-quality still image during moving image shooting, there is a problem that the moving image is interrupted or a decisive moment is already missed when the user notices.

  In order to obtain a still image at a decisive moment, a method of acquiring a high-resolution moving image with a high frame rate and extracting a still image at an arbitrary timing from the high-resolution moving image can be considered. However, this method has a problem that the amount of data increases in order to capture a high-resolution moving image.

In this embodiment, as shown in FIG. 1, the captured image _f 1 ~f ₄ and pixel addition between frames acquires added image _B 1 .about.B _3, inter-frame and the added image _B 1 .about.B ₃ The difference is taken and difference images DT ₂ and DT ₃ are obtained. Since the added images B _{1 to} B ₃ are added between frames, the change in pixel value between frames is smaller than that of the captured images f _{1 to} f ₄ . Therefore, it is considered that the difference images DT ₂ and DT ₃ are unevenly distributed in the vicinity of zero as compared with the difference between the captured images f _{1 to} f ₄ . Thereby, the compression rate can be improved by, for example, entropy encoding the images B ₁ , DT ₂ , and DT ₃ .

  As a technique for obtaining a high-resolution image from an image obtained by adding pixel values in the time axis direction, a technique for obtaining a high-resolution image from an image obtained by adding pixel values in the spatial direction can be applied. For example, a method of performing so-called super-resolution processing on a low-resolution image captured by pixel shift can be considered. In this method, addition reading is performed while sequentially shifting the position, and a high-definition image is temporarily assumed based on the plurality of position-shifted images. Then, the assumed image is degraded to generate a low resolution image, which is compared with the original low resolution image, and the high definition image is estimated so that the difference is minimized. As this super-resolution processing, an ML (Maximum-Likelihood) method, a MAP (Maximum A Postterior) method, a POCS (Projection Onto Convex Set) method, an IBP (Iterative Back Projection) method, and the like are known.

  For example, in Patent Document 1 described above, low-resolution images that have been pixel-shifted during moving image shooting are sequentially captured in time series, and a plurality of low-resolution images are combined to assume a high-resolution image. A technique for performing the above-described super-resolution processing on a high-resolution image and estimating a high-resolution image with high likelihood is disclosed.

  However, this technique uses a general super-resolution process in which the estimation accuracy is increased by iterative calculations that frequently use a two-dimensional filter. For this reason, the scale of processing becomes very large and the processing time increases, and there is a problem that it is difficult to apply to a device such as a digital camera with processing capacity and cost restrictions.

  In Patent Document 2 described above, a plurality of pixel-shifted low-resolution images are captured, and temporary pixels constituting the high-resolution image to be obtained are set as sub-pixels. A technique for estimating the pixel value of the sub-pixel so as to match the pixel value of the low-resolution image that has been made is disclosed. In this method, initial values of a plurality of sub-pixels are set, pixel values of sub-pixels excluding sub-pixels to be calculated are subtracted from pixel values of the low-resolution image, and pixel values are sequentially obtained for adjacent pixels. Apply.

  However, with this method, there is a problem that the estimation error becomes very large unless the initial value is specified. In this method, an area satisfying a predetermined condition is found from an image in order to set an initial value. Therefore, if an appropriate area cannot be found from the captured image, it is difficult to estimate the initial value. In addition, it is necessary to search for an appropriate area for setting the initial value.

In this regard, in the present embodiment, as will be described later with reference to FIG. 1, the added images B _{1 to} B ₃ are reproduced from the images B ₁ , DT ₂ , and DT _3, and the captured image f ₁ from the added images B _{1 to} B _3. to restore the ~f _4. The pixel values v _ij (1) ′ to v _ij (3) ′ of the addition images B _{1 to} B ₃ are superimposed and added, and the pixel values of adjacent frames include pixels of a common captured image. . As will be described later with reference to FIGS. 14 to 17, the captured images f _{1 to} f ₄ can be restored by a simple process by using the pixel values subjected to the superposition and addition. Further, since the captured images f _{1 to} f ₄ can be restored from the compressed data, a time resolution equivalent to that of the original captured image can be obtained.

2. First Data Compression Method Next, this embodiment will be described in detail. First, a method for generating an added image and a method for restoring a captured image will be described. In the following description, the case where the image sensor is an RGB Bayer array will be described as an example. However, the present embodiment is not limited to this, and may be, for example, an image sensor of a complementary color filter. In the following, a case where addition between two frames is performed will be described as an example. However, the present embodiment is not limited to this, and addition may be performed with other number of frames.

  Here, the frame is, for example, a timing at which an image is captured by an image sensor or a timing at which one captured image is processed in image processing. Alternatively, one image such as an added image or a difference image in the image data is also referred to as a frame as appropriate.

As shown in FIG. 1, in a frame f _T (T is a natural number), an RGB Bayer array captured image (hereinafter referred to as a captured frame image) is acquired by reading all pixels. The pixel values at the same position of two consecutive frames of the captured image f _T, 1 frame of the captured image is weighted addition so as to overlap, to produce a sum image B _T. In the following description, the captured frame images {f ₁ , f ₂ , f ₃ } will be described as an example.

Arbitrary pixel values of the photographed frame images {f ₁ , f ₂ , f ₃ } are assumed to be {v _ij (1), v _ij (2), v _ij (3)}, respectively. Each pixel value {v _ij (1) ′, v _ij (2) ′} of the inter-frame addition image {B ₁ , B ₂ } is obtained by the following equation (1).
v _ij (1) ′ = 1 · v _ij (1) + (1 / r) · v _ij (2)
v _ij (2) ′ = 1 · v _ij (2) + (1 / r) · v _ij (3) (1)

Here, r is a weighting coefficient and satisfies 1 ≦ | r |. v _ij (T) represents the pixel value of the address (i, j) of the captured frame image f _T.

  As described above, by applying the restoration estimation process described later with reference to FIGS. 14 to 17 to the added image data recorded by weighted addition between frames, the original captured image before addition is restored by estimation. it can.

  Next, a method for reducing data using inter-frame differences based on the above-described addition image and restoration estimation processing will be described.

As shown in FIG. 1, a difference image DT ₂ between the addition images B ₁ and B ₂ is obtained. Specifically, as shown in the following expression (2), the difference pixel value dt _ij of the pixel values {v _ij (1) ′, v _ij (2) ′} at arbitrary same positions in the addition images B ₁ and B _2. Find (2). Then, recording the added image _{B 1} and the difference image DT _2.
dt _ij (2) = v _ij (1) ′ − v _ij (2) ′ (2)

When restoring the captured frame image, from the recorded added image _{B 1} of the pixel value _v ij (1) 'and the pixel value _dt ij of the difference image DT ₂ (2), adding the above formula (2) Image _{B 2} For the pixel value v _ij (2) ′. Then, the pixel values v _ij (1) to v _ij (3) of the captured images f _{1 to} f ₃ are restored from the pixel values v _ij (1) ′ and v _ij (2) ′ of the added images B ₁ and B _2. .

  Now, between the temporally adjacent frames, the mutual correlation is high when the movement of the subject to be imaged is small, and the mutual correlation is low when the movement is large. Therefore, since the difference pixel value between frames is greatly influenced by the movement of the subject, it is difficult to stably reduce data using the difference value.

  In this regard, according to the present embodiment, weighted addition is performed on pixels at the same position between frames. As a result, a kind of addition filter effect is obtained, and an effect of gradual change of the difference value is born. Since the change in the difference value becomes gradual, data can be reduced more stably than when the captured image is directly compressed.

  Next, consider the point of restoring the captured frame image from the added image. In general, in order to return the value obtained by simple addition to the original value before addition, it is sufficient to solve the equations sequentially once the initial value is determined. However, if any deterioration occurs in the difference value, an error ripple occurs, which is not desirable.

  In this regard, according to the present embodiment, a captured image of 3 frames is estimated from an added image of 2 frames in one restoration process. Since each restoration process does not use the result of the other restoration process, an error does not spread. That is, in the weighted addition restoration estimation process, the influence of errors can be kept between partial frames, so that deterioration as an image can be suppressed to a smaller value.

In the above description, the frame {f ₁ , f ₂ , f ₃ } has been described as an example, but it goes without saying that the frame can be extended to an arbitrary frame f _T. That is, the pixel value v _ij (T) ′ of the added image B _T is expressed by the following equation (3), and the pixel value dt _ij (T + 1) of the difference image DT _{T + 1} is expressed by the following equation (4).
v _ij (T) ′ = 1 · v _ij (T) + (1 / r) · v _ij (T + 1) (3)
dt _ij (T + 1) = v _ij (T) ′ − v _ij (T + 1) ′ (4)

In one restoration estimation process, pixel values {v _ij (T), v _ij (T + 1), v _ij (T + 2)} are obtained. Therefore, when the frame image to be processed is shifted, the pixel value of the same pixel is obtained a plurality of times. Therefore, only one value (for example, the pixel value v _ij (T) in the restoration of f _{T to} f _{T + 2} ) may be adopted as the restoration estimation data in one restoration process.

  Here, in the above description, the weighted addition of two adjacent frames has been described as an example, but the present embodiment is not limited to this. For example, two weighted addition values of two adjacent frames may be further weighted and added to obtain an apparent value of four adjacent frames. Needless to say, it is easy to apply the data based on the difference value of the added values of the adjacent four frames as an application of this method.

3. Imaging Device FIG. 2 shows a first configuration example of an imaging device as a configuration example in the case of performing the above compression. The imaging apparatus includes an imaging unit 10 that performs imaging and data compression processing, and an image processing unit 20 that performs restoration processing of a high-definition image. The image processing unit 20 may be built in the camera body or may be configured by an external information processing apparatus such as a PC.

  Specifically, the imaging unit 10 includes a lens 110, an imaging element 120 (imaging sensor), an inter-frame addition image generation unit 122, an inter-frame difference image generation unit 124, and a data recording unit 155.

  The lens 110 forms an image of the subject 100. The image sensor 120 captures the formed subject image. An analog signal obtained by imaging is converted into a digital signal by an A / D converter (not shown).

As described above, the inter-frame addition image generation unit 122 weights and adds the pixel values of the captured frame images f _{1 to} f ₃ between frames to generate an inter-frame addition image {B ₁ , B ₂ }. The inter-frame difference image generation unit 124 uses the inter-frame addition image B ₁ as a reference image, takes a difference between the reference image B ₁ and the inter-frame addition image B ₂ , and generates an inter-frame difference image DT ₂ . Data recording unit 155 records the reference image _{B 1} and the inter-frame difference image DT _2.

  The image processing unit 20 includes an inter-frame addition image reproduction unit 214, an inter-frame estimation calculation unit 234, a high-definition still image generation unit 240, a high-definition moving image generation unit 250, a standard moving image generation unit 260, an image output unit 290, and an image selection unit. 295.

The inter-frame addition image reproduction unit 214 reproduces the inter-frame addition image {B ₁ , B ₂ } from the recorded reference image B ₁ and inter-frame difference image DT ₂ . The inter-frame estimation calculation unit 234 restores the captured frame images f _{1 to} f ₃ by estimation based on the inter-frame addition image {B ₁ , B ₂ }. The restored image is a Bayer array RAW image. This estimation calculation will be described later with reference to FIG.

  The high-definition still image generation unit 240 performs demosaicing processing on the restored image with the Bayer arrangement, and performs image processing such as gradation correction processing on the image to generate a high-definition still image. At this time, a still image at the timing selected by the image selection unit 295 is generated. The timing is selected according to the user's instruction, and the user selects the timing by looking at the output moving image of the image output unit 290, for example.

  The high-definition moving image generation unit 250 performs demosaicing processing on the restored moving image having the Bayer arrangement, performs image processing such as gradation correction processing on the moving image, and generates a high-definition moving image. The standard moving image generation unit 260 downsamples the high-definition moving image, and generates, for example, a moving image having the number of high-definition pixels as the standard moving image.

  The image output unit 290 outputs a high-definition still image, a high-definition moving image, and a standard moving image to, for example, a display device or a printer.

  According to the above embodiment, as illustrated in FIG. 2, the imaging apparatus includes an image acquisition unit (for example, the imaging device 120), the inter-frame addition image generation unit 122, the inter-frame difference image generation unit 124, and the inter-frame addition image reproduction. A unit 214, an inter-frame estimation calculation unit 234, and an image output unit 290.

As illustrated in FIG. 1, the image acquisition unit acquires first to third captured images f _{1 to} f ₃ in the first to third frames. Interframe adding image generation unit 122, between the first frame by weighted addition of pixel values of the first captured image _{f 1 v} ij (1) and the second pixel value _v ij of the captured image _{f 2} (2) added image _{B 1} generates, between the second pixel value of the captured image _{f 2} of _v ij (2) and the third pixel value _v ij (3) adding the weighted with the second captured image _{f 3} of the frame addition generating an image _{B 2.} The inter-frame difference image generation unit 124 generates a difference between the first inter-frame addition image B ₁ and the second inter-frame addition image B ₂ as an inter-frame difference image DT ₂ .

As shown in FIG. 2, the inter-frame addition image reproduction unit 214 reproduces the second inter-frame addition image B ₂ based on the _first inter-frame addition image B ₁ and the inter-frame difference image DT ₂ . The inter-frame estimation calculation unit 234 calculates the first to third captured images f _{1 to} f ₃ based on the _first inter-frame addition image B ₁ and the reproduced second inter-frame addition image B ₂ . estimating a pixel value _{v ij (1) ~v ij (} 3). The image output unit 290 outputs a high resolution image based on the estimated pixel values v _ij (1) to v _ij (3).

This makes it possible to compress efficiently captured image f _T. That is, by adding the pixel value of the captured image between frames, the temporal change of the pixel value can be moderated. Thereby, as described above, the entropy of the inter-frame difference image can be lowered, and for example, a high compression rate can be realized by entropy coding.

Further, by applying the restoring estimation process example described below, it is possible to reproduce the captured image f _T a simple process. As a result, the image before the addition between frames is re-established, and thus a restored image with a high temporal resolution with little blurring can be obtained. Further, it is possible to take out the high-resolution still image of an arbitrary timing from the image data of a high compression ratio to restore the captured image f _T.

4). Second Data Compression Method Next, a method of reducing data by adding pixels in the above-described reference image B ₁ and difference image DT _T (T = 2, 3,...) Within a frame (inside an image) will be described. To do. In the following is described taking the process for any difference image DT _T as an example, it is the same for the reference image B _1. Although the case where 4-pixel addition is performed will be described as an example, the present embodiment is not limited to this, and other pixel numbers may be added. Hereinafter, (T) representing the frame number is omitted from the pixel values v _ij (T), dt _ij (T), and a _ij (T), and is denoted as v _ij , dt _ij , and a _ij .

As shown in FIG. 3, to set the configured summation unit (summed as pixel group) in the 4 pixels in the difference image DT _T, it adds by weighting the pixel value of the summation unit. At this time, the addition unit is shifted horizontally or vertically while superimposing one pixel, and four pixel addition images A _{1 to} A ₄ are generated. This process is performed for each difference image.

The 4-pixel addition value a _ij is expressed by the following equation (5). The addition pixel values constituting the pixel addition images A _{1 to} A ₄ are {a _ij , a _{(i + 1) j} , a _{(i + 1) (j + 1)} , a _{i (j + 1)} }, respectively.
a _ij = dt _ij + (1 / r) dt _{(i + 1) j} +
(1 / r) dt _{i (j + 1)} + (1 / r ² ) dt _{(i + 1) (j + 1)}
(5)

Here, r is a weighting parameter, and 1 ≦ r. The _{dt ij} is a pixel value of the address (i, j) in the difference image DT _T.

Next, the addition positions of the generated pixel addition images A _{1 to} A ₄ are matched, and an average image M is generated by superimposing the four pixel addition images and taking the addition average of the values at the same position. . That is, the 4-pixel addition values of the pixel addition images A _{1 to} A ₄ are respectively represented by {a ⁽¹⁾ _ij , a ⁽²⁾ _{(i + 1) j} , a ⁽³⁾ _{(i + 1) (j + 1)} , a ⁽⁴⁾ _{i ( j + 1)} }, the pixel value a ^M _ij of the average image M is expressed by the following equation (6).
a ^M _ij = [a ⁽¹⁾ _ij + a ⁽²⁾ _{(i + 1) j} +
a ⁽³⁾ _{i (j + 1)} + a ⁽⁴⁾ _{(i + 1) (j + 1)} ] / 4 (6)

Using the generated average image M as a reference, the difference (projection to the direction vector (1, −1)) between the average image M and the pixel addition images A _{1 to} A ₄ is set as difference images D _{1 to} D ₄ , respectively. The difference images D _{1 to} D ₄ and the average image M are combined to obtain fused image data F (M, D _{1 to} D ₄ ). If the 4-pixel addition difference values constituting the difference images D _{1 to} D ₄ are represented as {a ^D1 _ij , a ^D2 _ij , a ^D3 _ij , a ^D4 _ij }, they can be expressed as the following expression (7).
a ^D1 _ij = a ^M _ij −a ⁽¹⁾ _ij ,
a ^D2 _ij = a ^M _ij −a ⁽²⁾ _{(i + 1) j} ,
a ^D3 _ij = a ^M _ij −a ⁽³⁾ _{(i + 1) (j + 1)} ,
a ^D4 _ij = a ^M _ij −a ⁽⁴⁾ _{i (j + 1)} (7)

In this way, the still image fusion is fused to the moving image data _{F (M, D 1 ~D 4} ) generates, may be recorded it, the recorded F _(M, D 1 ~D ₄ ) In the subsequent stage, "high-definition still image data" or "moving image data" can be appropriately generated.

In this example, the difference (that is, projection onto the direction vector (1, −1)) is taken, but the present embodiment is not limited to this. For example, in accordance with the correlation distribution between the pixel values of the average image M and the pixel values of the pixel addition images A _{1 to} A ₄ , the principal component axis (straight line with the largest variance) is obtained, and the axis orthogonal to the principal component axis is obtained. It goes without saying that the data can be further compressed by using the projected value.

5. Entropy Coding Next, a method for compressing the above-described average image M and intra-frame difference images D _{1 to} D ₄ will be described.

In the pixel value data of the average image M and the intra-frame difference images D _{1 to} D ₄ , since the occurrence frequency distribution is significantly biased, entropy coding (eg, Huffman code, LZH, etc.) is effective as data compression. Entropy coding is a compression technique in which the occurrence probability of a pixel value is obtained and a short code length is assigned from the one with the highest occurrence probability.

For example, it is appropriate to assign the shortest code length to the average value of the pixel value data of the average image M, and to make the code length longer as the value is farther from the average value. That is, since the pixel value data of the intra-frame difference images D _{1 to} D ₄ has an average of zero and is distributed with the occurrence probability of zero as a peak, the shortest code may be assigned to the zero value. . It is also effective to perform nonlinear quantization on the pixel value data of the average image M and the intra-frame difference images D _{1 to} D ₄ according to the probability density.

On the other hand, in the average image M, the deviation of the occurrence frequency distribution becomes remarkable because it is certainly the pixel value data obtained by addition averaging, but the range that the pixel addition value can take is the pixel value data of the intra-frame difference images D _{1 to} D _4. When compared, it becomes quite large. For this reason, application of an irreversible compression method (JPEG, MPEG, etc.) that can generally be expected to have a high compression rate is effective as data compression.

Therefore, in the average image M, irreversible high compression is applied by reducing the number of pixels, and in the intra-frame difference images D _{1 to} D ₄ , high-speed entropy encoding using characteristics that exhibit a similar occurrence distribution bias is performed. It is possible to apply.

  Next, an example of applying the simplest Huffman code is shown. If one Huffman code table is generated based on the pixel value distribution of a large number of images, the application range becomes wide and the compression effect becomes image-dependent. However, it is not effective to generate the Huffman table for each image or image range every time.

ａ’^Ｍ _ｉｊ＝ａ^Ｍ _ｉｊ−α^Ｍ _ｉｊ （９） Therefore, when applied to the above average image M, an average value α ^M _ij is obtained for each predetermined image area by the following equation (8), and the difference between the average value α ^M _ij and each added average value {a ^M _ij }. The value {a ′ ^M _ij } is obtained by the following equation (9). Then, the difference value {a ′ ^M _ij } is used as the data of the average image M again, and Huffman coding may be applied. Thus, the difference value {a ′ ^M _ij } can form an occurrence distribution of values universally centered on zero. M in the fusion image data F (M, D _{1 to} D ₄ ) is considered to be replaced with M = {α ^M _ij , a ′ ^M _ij }.

a ′ ^M _ij = a ^M _ij −α ^M _ij (9)

Here, (h0, v0) is the start point of the area where the average value α ^M _ij is calculated, and (h, v) is the end point of the area where the average value α ^M _ij is calculated. That is, in the above formula (8), h0 ≦ i ≦ h and v0 ≦ j ≦ v are satisfied. For example, the area for calculating the average value α ^M _ij is the entire average image M. The area for calculating the average value α ^M _ij and the number of average values a ^M _ij used for calculating the average value α ^M _ij may be determined by evaluating the balance with the data compression rate.

Since the difference value {a ′ ^M _ij } and the 4-pixel addition difference value {a ^D1 _ij , a ^D2 _ij , a ^D3 _ij , a ^D4 _ij } are themselves composed of a value of zero on average, FIG. The code assignment shown in FIG. In this way, the Huffman code table can be a fixed type.

Note that, as the above-described average value α ^M _ij , the fixed value m _AVR shown in the following equation (10) may be applied, and may be handled as look-ahead information without being treated as fluctuation data. In this case, the average value m _AVR need not be recorded as data. M in the fusion image data F (M, D _{1 to} D ₄ ) is considered to be replaced with M = {m _AVR , a ′ ^M _ij }. The following equation (10) is applied to the case of the weighted 4-pixel addition value defined by the above equation (5), and it should be noted that an appropriate value is appropriately set depending on the number of added pixels and the weighting method. is necessary.
m _AVR = v _max · {1+ (1 / r) + (1 / r) + (1 / r ² )} / 2
(10)

Here, r is a weighting parameter, and 1 ≦ r. v _max is a maximum value that defines the pixel value.

6). High-definition still image reproduction method A method for reconstructing and estimating the inter-frame difference image DT _T (or the reference image B ₁ ) from the above-described F (M, D _{1 to} D ₄ ) will be described.

When the above equation (7) is transformed, the following equation (11) is obtained. As shown in the following formula (11), the 4-pixel addition value {a ^M _ij } constituting the average image M and the 4-pixel addition values {a ^D1 _ij , a ^D2 constituting the intra-frame difference images D _{1 to} D _{4. ij} , a ^D3 _ij , a ^D4 _ij }, the four-pixel addition values {a ⁽¹⁾ _ij , a ⁽²⁾ _{(i + 1) j} , a ⁽³⁾ _{(i + 1) of} the pixel addition images A _{1 to} A _{4 (J + 1)} , a ⁽⁴⁾ _{i (j + 1)} } is obtained.
a ⁽¹⁾ _ij = a ^M _ij −a ^D1 _ij ,
a ⁽²⁾ _{(i + 1) j} = a ^M _ij −a ^D2 _ij ,
a ⁽³⁾ _{(i + 1) (j + 1)} = a ^M _ij −a ^D3 _ij ,
a ⁽⁴⁾ _{i (j + 1)} = a ^M _ij −a ^D4 _ij (11)

The obtained 4-pixel addition value {a ⁽¹⁾ _ij , a ⁽²⁾ _{(i + 1) j} , a ⁽³⁾ _{(i + 1) (j + 1)} , a ⁽⁴⁾ _{i (j + 1)} } is one high definition This is image data composed of an added value of four pixels obtained by superimposing and shifting an image horizontally or vertically one pixel at a time. The pixel value of the original reference image B ₁ or the difference image DT ₂ is obtained by applying this 4-pixel addition value to the restoration estimation process described later with reference to FIG.

Next, the inter-frame addition image B ₂ is reproduced using the restored reference image B ₁ and the inter-frame difference image DT ₂ . Specifically, as shown in the following formula (12), the pixel value v _ij (1) ′ of the reference image B ₁ is used as a reference value, and the pixel value v _ij (2) ′ of the added image B ₂ is obtained.
v _ij (2) ′ = v _ij (1) ′ − dt _ij (2) (12)

For the pixel values {v _ij (1) ′, v _ij (2) ′} of the inter-frame addition image {B ₁ , B ₂ } obtained in this way, the restoration estimation process described later with reference to FIG. Apply and estimate the pixel values {v _ij (1), v _ij (2), v _ij (3)} of the captured frame images {f ₁ , f ₂ , f ₃ } before inter-frame addition. In the above embodiment, the frame image to be handled is described as being limited to the image frames {f ₁ , f ₂ , f ₃ }, but it goes without saying that this concept can be applied to the subsequent frames one after another.

7). Second Configuration Example FIG. 5 of the imaging device, and generates the reference image B ₁ and the inter-frame difference image DT _T, as described above, in the case of further generating an average image M and frame difference image D ₁ to D ₄ As a configuration example, a second configuration example of the imaging apparatus is shown. The imaging device includes an imaging unit 10 and an image processing unit 20. In the following description, the same components as those described above with reference to FIG. 2 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The imaging unit 10 includes a lens 110, an image sensor 120, an inter-frame addition image generation unit 122 (first addition image generation unit), an inter-frame difference image generation unit 124 (first difference image generation unit), and an intra-frame addition image. A generation unit 130 (second addition image generation unit), an intra-frame difference image generation unit 141 (second difference image generation unit), an average image generation unit 142, an entropy encoding unit 143, and a compressed data recording unit 150 are included.

Frame adding image generation unit 130, as described above, it adds the pixel values of the reference image B ₁ or the inter-frame difference image DT _T while shifting pixels, to generate a frame added image A ₁ to A _4. The average image generation unit 142 generates an average image M from the intra-frame addition images A _{1 to} A ₄ . The intra-frame difference image generation unit 141 generates intra-frame difference images D _{1 to} D ₄ from the intra-frame addition images A _{1 to} A ₄ and the average image M. The entropy encoding unit 143 compresses the difference images D _{1 to} D ₄ by lossless compression such as the above-described entropy encoding. The data compression unit 144 compresses the average image M by irreversible compression such as M-JPEG or JPEG-XR. The compressed data recording unit 150 records data of the compressed average image M and intra-frame difference images D _{1 to} D ₄ .

  The image processing unit 20 includes a compressed data decompression unit 200, an intra-frame addition image reproduction unit 210 (second addition image reproduction unit), an inter-frame addition image reproduction unit 214 (first addition image reproduction unit), and an intra-frame estimation calculation. Unit 230 (second estimation calculation unit), inter-frame estimation calculation unit 234 (first estimation calculation unit), high-definition still image generation unit 240, high-definition moving image generation unit 250, standard moving image generation unit 260, image output unit 290 and an image selection unit 295.

The compressed data decompression unit 200 performs a process of decompressing the compressed average image M and intra-frame difference images D _{1 to} D ₄ . The intra-frame addition image reproduction unit 210 performs processing for reproducing the intra-frame addition images A _{1 to} A ₄ from the average image M and the intra-frame difference images D _{1 to} D ₄ . Frame estimation calculation section 230, based on the frame in addition image _A 1 to A _4, is restored by estimating the reference image _{B 1} and the inter-frame difference image DT _T. The restored image is a Bayer array RAW image.

6 and 7 show data configuration examples in the second configuration example. As shown in FIG. 6, the captured image _{f T} is, P × Q pixels (P, Q is a natural number) formed by the pixel value _v 00 of the _{(T) ~v PQ (T)} . The inter-frame difference image DT _{T is} also composed of pixel values dt ₀₀ (T) to dt _PQ (T) of P × Q pixels. Reference image _{B 1} is the same.

As shown in FIG. 7, the intra-frame addition images A ₁ (DT _T ) to A ₄ (DT _T ) are configured with pixel values a ₀₀ (T) to a _pq (T) of p × q pixels, respectively. Here, p = P / 2 and q = Q / 2. The same applies to the intra-frame addition images A ₁ (B ₁ ) to A ₄ (B ₁ ). The average image M (DT _T ) and the intra-frame difference images D ₁ (DT _T ) to D ₄ (DT _T ) are also configured with pixel values of p × q pixels. The same applies to the average image M (B ₁ ) and the intra-frame difference images D ₁ (B ₁ ) to D ₄ (B ₁ ).

  According to the above embodiment, as illustrated in FIG. 5, the imaging device includes an intra-frame addition image generation unit 130, an average image generation unit 142, an intra-frame difference image generation unit 141, and a data compression unit (for example, an entropy encoding unit). 143), a compressed data decompression unit 200, an intra-frame addition image reproduction unit 210, and an intra-frame estimation calculation unit 230.

As illustrated in FIG. 3, the intra-frame addition image generation unit 130 is a unit that receives the first inter-frame addition image B ₁ or the inter-frame difference image DT ₂ as an input image and acquires the addition pixel value a _ij. The addition unit is set for each of a plurality of pixels of the input image (for example, every 4 pixels), and the pixel value dt _ij included in the addition unit is weighted and added (the above equation (5)) while sequentially shifting the addition unit. First to fourth (first to nth in a broad sense) intra-frame addition images A _{1 to} A ₄ are generated. As described in the above equation (6), the average image generation unit 142 generates the average image M of the intra-frame addition images A _{1 to} A ₄ . As described in Expression (7) above, the intra-frame difference image generation unit 141 calculates the difference between the m-th intra-frame addition image A _m (m is a natural number equal to or less than n) and the average image M as the m-th frame. generating as an internal difference image _{D m.} Data compression unit compresses the average image M and frame difference image D _m.

As shown in FIG. 5, the compressed data recording unit 150 decompresses the compressed average image M and frame difference image D _m. The intra-frame addition image reproduction unit 210 reproduces the intra-frame addition images A _{1 to} A ₄ based on the expanded average image M and intra-frame difference image D _m . The intra-frame estimation calculation unit 230 estimates the pixel value of the input image based on the reproduced intra-frame addition images A _{1 to} A ₄ .

In this way, intra-frame addition is further performed on the inter-frame addition image B ₁ and the inter-frame difference image DT ₂ , and the average image M of the intra-frame addition images A _{1 to} A ₄ and the intra-frame difference image D _1. to D ₄ can be produced by compressing. Thereby, the compression rate can be further improved. That is, although the input image B ₁ (or DT ₂ ) and the four added images A _{1 to} A ₄ have the same number of pixels, the entropy of the pixel value of the difference image D _m can be made smaller than that of the input image. The compression rate can be improved.

Further, according to the present embodiment, it is possible to reproduce the input image from the compressed data with a simple process. That is, the intra-frame addition images A _{1 to} A ₄ are image data obtained by superposition shift addition, and a restoration estimation process described later can be applied. This restoration estimation process can simplify the process of estimating a high-resolution image from a low-resolution image as compared with Patent Documents 1 and 2 described above. Further, since the reconstructed input image can further recover the captured image f _T, it is possible to take out the high-resolution still image of an arbitrary timing from the image data of a high compression ratio.

In the present embodiment, as shown in FIG. 3, the intra-frame addition image generation unit 130 sequentially shifts the addition unit one pixel at a time horizontally or vertically (i-axis direction or j-axis direction). 4 positions (for example, coordinates (0, 0), (1, 0), (1, 1), (0, 1)), and in-frame addition images A _{1 at the first} to fourth positions, respectively. to get the ~A _4. The addition unit of the m-th position and the m + 1-th position (for example, (0, 0), (1, 0)) includes a common pixel (v ₁₀ , v ₁₁ ).

  In this way, it is possible to perform superposition shift addition for obtaining an added pixel value by sequentially shifting the addition unit while including a common pixel.

In the present embodiment, as shown in the above equation (6), the average image generation unit 142 calculates the average value a ^M _ij of the added pixel values of the first to fourth positions as the pixel value of the average image. As shown in the above equation (7), the intra-frame difference image generation unit 141 calculates a difference value (a between the pixel value a ^M _ij of the average image and the added pixel value (for example, a ⁽¹⁾ _ij ) at the m-th position. ^D1 _ij ) is obtained as the pixel value (a ^D1 _ij ) of the mth difference image.

In the present embodiment, as described with reference to FIG. 3 and the like, the data compression unit reversibly compresses the _mth intra-frame difference image _Dm by entropy coding.

In this way, it is possible to compress the frame difference image D _m entropy as described above is reduced with a high compression ratio, it is possible to improve the compression ratio of the image data. Thereby, the capacity of the storage for recording the compressed data can be reduced, and the communication load in data communication can be reduced.

8). Third Data Compression Method In the above embodiment, the addition within the frame is performed after the addition between the frames. However, the addition between the frames may be performed after the addition within the frame. FIG. 8 shows a third configuration example of the imaging apparatus as a configuration example in such a case.

  The imaging device includes an imaging unit 10 and an image processing unit 20. In the following description, the same components as those described above with reference to FIG. 2 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The imaging unit 10 includes a lens 110, an image sensor 120, an inter-frame addition image generation unit 122, an inter-frame difference image generation unit 124, an intra-frame addition image generation unit 130, an intra-frame difference image generation unit 141, an average image generation unit 142, An entropy encoding unit 143 and a compressed data recording unit 150 are included. The components of the image processing unit 20 are the same as those of the image processing unit 20 in FIG.

The intra-frame addition image generation unit 130 performs superposition shift weighting addition in each frame in each frame of the captured images f _T (T is a natural number) obtained continuously, and adds the intra-frame addition image {A ₁ (T). , A ₂ (T), A ₃ (T), A ₄ (T)}. The average image generation unit 142 obtains an average image M (T) of the intra-frame addition image. The intra-frame difference image generation unit 141 generates an intra-frame difference image {D ₁ (T), D ₂ (T), D ₃ (T), D ₄ (T)} that is a difference between the intra-frame addition image and the average image. Ask.

The inter-frame addition image generation unit 122 performs inter-frame processing for each of the average image M (T) and the intra-frame difference image {D ₁ (T), D ₂ (T), D ₃ (T), D ₄ (T)}. Are added, and an inter-frame addition image {B _T } is generated. That is, when expressed in image units, the weighted addition is expressed by the following equation (13).
B _T (M) = 1 · M (T) + (1 / r) · M (T + 1),
B _T (D _m ) = 1 · D _m (T) + (1 / r) · D _m (T + 1) (13)

  Here, r is a weighting coefficient. m is a natural number of 4 or less.

The inter-frame difference image generation unit 124 obtains the difference image DT _T by sequentially subtracting {B _T } with the inter-frame addition image {B ₁ } as a reference, as shown in the following equation (14).
B ₁ = (reference value),
DT _{T + 1} = B _T −B _{T + 1} (14)

The entropy encoding unit 143 performs entropy encoding (reversible compression) on the added image {B ₁ } and the difference image DT _T to generate recording data.

The compressed data decompression unit 200 performs a process of decompressing the compressed added image {B ₁ } and the difference image DT _T. The inter-frame addition image reproduction unit 214 reproduces the addition image {B ₁ , B ₂ } from the expanded reference image B ₁ and the difference image DT ₂ . The inter-frame estimation calculation unit 234 restores the average image M (T) and the difference images D ₁ (T) to D ₄ (T) by estimation based on the added image {B ₁ , B ₂ }.

The intra-frame addition image reproduction unit 210 performs a process of reproducing the addition images A _{1 to} A ₄ from the average image M (T) and the difference images D ₁ (T) to D ₄ (T). Frame estimation calculation section 230, based on the added image _A 1 to A _4, is restored by estimating the captured image _{f T.} The restored image is a Bayer array RAW image.

According to the above embodiments, frame adding image generation unit 130 _(the x 3 following a natural number) the captured image f _x of the x receives the captured image addition unit is a unit for acquiring an addition pixel value set for each of a plurality of pixels of f _x, frame added image a ₁ of the first to n by weighted addition of pixel values included in the summation unit while sequentially pixel shifting summation unit _(x) ~A _{4 (} x). The average image generation unit 142 generates an average image M (x) of the intra-frame addition images A ₁ (x) to A ₄ (x). The intra-frame difference image generation unit 141 generates a difference between the m-th intra-frame addition image A _m (x) and the average image M (x) as an m-th intra-frame difference image D _m (x).

The inter-frame addition image generation unit 122 receives the intra-frame difference image D _m (x) or the average image M (x) as the x-th input image, and receives the first and second input images (for example, D _m (1)). , D _m (2)) to generate a _first inter-frame addition image B ₁ and a second inter-frame based on the second and third input images (D _m (2), D _m (3)) generating a sum image _{B 2.}

  In this way, it is possible to perform data compression by performing intra-frame addition after performing intra-frame addition on the captured image. Even in such a processing order, it is possible to obtain an entropy reduction effect by the intra-frame difference image, an effect of reducing a change in the pixel value by the inter-frame addition image, and an entropy reduction effect by the inter-frame difference image. Further, the captured image can be restored by the restoration estimation process, and a high-definition still image at an arbitrary timing can be obtained.

9. Fourth Data Compression Method In the above embodiment, the inter-frame difference image is generated based on the difference between the two frames. However, the inter-frame difference image may be generated based on the difference between the reference frame and the previous and subsequent frames. A method for performing data compression based on the difference between the reference frame and the preceding and following frames will be described with reference to FIG.

As shown in FIG. 9, first, an inter-frame addition image B _T (T is a natural number) is generated in the same manner as described above. In this inter-frame addition image _{B T,} for example, _B (the k 0 or an integer) _{3k +} 2 the reference image and. Then, the difference between the reference image B _{3k + 2} and the inter-frame addition images B _{3k + 1} and B _{3k + 3} before and after the difference is obtained to obtain difference images DT _{3k + 1} and DT _{3k + 3} . The image difference is obtained by obtaining a difference value between pixel values at the same position.

Next, the combination data {B _{3k + 2} , DT _{3k + 1} , DT _{3k + 3} } of the reference image and the difference image are used as data units, each is subjected to lossless compression coding (for example, entropy coding), and sequentially recorded as fusion compressed data.

To reproduce the original image f _T may be performed reverse process to the above. That is, the lossless compression code is decoded with respect to the recorded fusion compressed data, and combination data {B _{3k + 2} , DT _{3k + 1} , DT _{3k + 3} } is generated. Then, the inter-frame addition image _BT is reproduced from the combination data. Applying the image reconstruction estimation method with respect to the frame between the addition image B _T, obtaining a captured frame image f _T.

10. First Modified Example of Data Compression Method Next, a first modified example of the data compression method will be described. As shown in FIG. 10, first, in each frame of the captured frame image f _T (T is a natural number), the above-described superposition shift 4-pixel addition is performed in FIG. 3 and the like, and the intra-frame addition image f _T ′ = {A ₁ generating a to a _4}.

Next, the inter-frame difference process described above with reference to FIG. 1 and the like is performed in the time axis direction of the intra-frame addition image f _T ′. That is, the reference image and _{B 1} = _f 1 ', to generate a difference image DT _T between frames by difference of adjacent frames. Although representing a reference image B ₁ and the inter-frame difference image DT _T for simplicity in FIG. 10 in one image respectively, as shown in fact following formulas (15), each of the four images Consists of.
A _k ′ (T) = A _k (T) −A _k (T + 1) (15)

  Here, the difference of the above equation (15) is performed by subtraction of pixel values at the same position. k is a natural number of 4 or less.

  The above equation (15) corresponds to setting the weighting coefficient to r = −1 in the above-described interframe addition in FIG.

Next, as shown in FIG. 11, an average image M of the _four images A ₁ ′ (T) to A ₄ ′ (T) constituting the reference image B ₁ and the inter-frame difference image DT _T is generated. Difference images D _{1 to} D ₄ between the average image M and the four images A ₁ ′ (T) to A ₄ ′ (T) are generated. The average image M and the difference images D _{1 to} D ₄ are generated in the same manner as described above with reference to FIG. Then, the average image M and the difference images D _{1 to} D ₄ are subjected to lossless compression processing to generate data-compressed recording data.

When restoring the captured frame image f _T from the recording data, performs decompression processing on the compressed recorded data first, to reproduce the average image M and the difference image D ₁ to D _4. Next, determine the reference image _{B 1} and the inter-frame difference image DT _T from the average image M and the difference image _D 1 to D _4. Next, the intra-frame addition image f _T ′ = {A _{1 to} A ₄ } is reproduced from the reference image B ₁ and the difference image DT _T. This intra-frame addition image f _T ′ is an addition pixel value that is superimposed and shifted horizontally or vertically, and by applying a restoration estimation process described later with reference to FIG. Play of the shooting frame image f _T.

11. In the second variant above modification of the data compression technique has been generated an average image M and the difference image D ₁ to D ₄ from taking a difference between frames, the average image M and the difference image D ₁ to D ₄ An inter-frame difference may be taken after generation. Such a second modification will be described with reference to FIGS.

As shown in FIG. 12, first, in each frame of the captured frame image f _T (T is a natural number), the above-described superposition shift 4-pixel addition is performed in FIG. 3 and the like, and the intra-frame addition image f _T ′ = {A ₁ generating a to a _4}.

Next, an average image M (T) of the intra-frame addition image f _T ′ = {A _{1 to} A ₄ } is generated, and the difference between the average image M (T) and the intra-frame addition image {A _{1 to} A ₄ } And generate the intra-frame difference images {D ₁ (T) to D ₄ (T)}. In FIG. 12, the intra-frame difference image is represented by D _m (T) (m is a natural number of 4 or less).

Next, as shown in FIG. 13, using the average image M (1) as a reference image, a difference image DM (T) = M (T + 1) −M (T) between frames of the average image M (T) is generated. . Also, intraframe difference image _D m (1) is a reference image, the difference image _DT m between frames of intraframe difference image _{D m (T) (T)} = D m (T + 1) generate _-D m (T) To do.

Next, a reversible compression process is performed on the reference image M (1) and the difference image DM (T) of the average image, and the reference image D _m (1) and the difference image DT _m (T) of the intra-frame difference image, Generate recorded data.

When restoring the captured frame image f _T from the recording data, first, the compressed recording data is expanded, and the reference image M (1), the difference image DM (T), the reference image D _m (1), The difference image DT _m (T) is reproduced. Next, the average image M (T) is reproduced from the reference image M (1) and the difference image DM (T), and the intra-frame difference image D _m (from the reference image D _m (1) and the difference image DT _m (T) is reproduced. T) is played back. Next, an intra-frame addition image {A _{1 to} A ₄ } is reproduced from the average image M (T) and the intra-frame difference image D _m (T). Next, restoration estimation processing described later with reference to FIG. 18A and the like is applied to the intra-frame addition images {A _{1 to} A ₄ } to restore the original captured frame image f _T.

12 Next, a method for estimating the pixel value v _ij (T) of the captured frame image f _T before the addition based on the inter-frame addition image B _T described above with reference to FIG. 1 and the like will be described. .

In the following, a case where the pixel values {v _ij (1), v _ij (2), v _ij (3)} (i, j are integers of 0 or more) of the frames f _{1 to} f ₃ will be described as an example. However, the same applies to the pixel values of other frames. In addition, although a case where weighted addition of pixel values of two frames is described as an example, the present invention is not limited to this, and pixel values of three or more frames may be weighted and added.

The added pixel values {b _ij (1), b _ij (2)} shown in FIG. 14 are the pixel values {v _ij (1) ′, v _ij () of the inter-frame added images B ₁ and B ₂ described in FIG. 2) corresponds to '}. In the estimation processing, final estimated pixel values v _ij (1) to v _ij (3) are estimated using the added pixel values {b _ij (1), b _ij (2)}.

For the sake of simplicity, for example, when the weighting factor r = 2, the added pixel value {b _ij (1), b _ij (2)} is expressed by the following equation (16).
b _ij (1) = v _ij (1) + (1/2) · v _ij (2)
b _ij (2) = v _ij (2) + (1/2) · v _ij (3) (16)

As shown in FIG. 15, in the above equation (16), when {b _ij (1), b _ij (2)} is multiplied by a predetermined weighting factor to obtain the difference δi ₀ , the following equation (17) is established.
δi ₀ = b _ij (2) -2b _ij (1)
= (1/2) v _ij (3) -2v _ij (1) (17)

  15 to 17, the pixel value suffix ij is omitted for simplicity.

When v _ij (1) is an unknown number (initial variable), the pixel value {v _ij (2), v _ij (3)} is obtained as a function of v _ij (1) as shown in the following equation (18). Can do. In this way, a combination pattern of pixel values {v _ij (1), v _ij (2), v _ij (3)} not subjected to frame addition is obtained using v _ij (1) as an unknown.
v _ij (1) = (unknown number),
v _ij (2) = 2 (b _ij (1) −v _ij (1)),
v _ij (3) = 4 v _ij (1) + 2δi ₀
= 4v _ij (1) +2 (b _ij (2) -2b _ij (1))
(18)

Next, a method for _obtaining the unknown number v _ij (1) will be described. As shown in FIG. 16, the pattern of added pixel values {b _ij (1), b _ij (2)} and the pattern of estimated pixel values {v _ij (1), v _ij (2), v _ij (3)} Compare Then, an unknown number v _ij (1) that minimizes the error E is derived and set as the estimated pixel value v _ij (1).

Specifically, the relationship of the following formula (19) is established between the added pixel value {b _ij (T)} and the estimated pixel value {v _ij (T), v _ij (T + 1)}. Considering the weighting by the following equation (19), the evaluation function E shown in the following equation (20) is obtained. Then, with this evaluation function E, the similarity between the pattern {b _ij (1), b _ij (2)} and the pattern {v _ij (1), v _ij (2), v _ij (3)} is evaluated.
b _ij (T) = v _ij (T) + (1/2) v _ij (T + 1) (19)

As shown in FIG. 17, an unknown v _ij (1) (= α) that minimizes the evaluation function E is obtained, and the value of v _ij (1) can be determined. Then, the estimated value of v _ij (1) is substituted into the above equation (18) to obtain {v _ij (2), v _ij (3)}.

According to the above embodiment, as shown in the above equation (17), the inter-frame estimation calculation unit 234 calculates the pixel value b _ij (1) of the first inter-frame addition image and the second inter-frame addition image. A difference value δi ₀ from the pixel value b _ij (2) is obtained. As shown in the above equation (18), a relational expression between the pixel value v _ij (1) of the first captured image and the pixel value v _ij (3) of the third captured image is expressed using the difference value δi _0. Represent. Then, the pixel values v _ij (1) to v _ij (3) of the first to third captured images are estimated using the relational expression.

More specifically, as shown in the above equation (18), the inter-frame estimation calculation unit 234 adds the relational expression between the pixel values v _ij (1) to v _ij (3) of the captured image to the inter-frame addition. The pixel value {b _ij (1), b _ij (2)} of the image is used. As described in FIG. 16, the pixel values v _ij (1) to v _ij (3) of the captured image represented by the relational expression and the pixel values {b _ij (1), b _ij ( 2)} and the similarity is evaluated. As described with reference to FIG. 17, based on the similarity evaluation result, the pixel values v _ij (1) to v _ij (3) of the captured image are estimated so that the similarity becomes the highest.

  In this way, the relational expression of the pixel value of the captured image can be expressed using the pixel value superimposed and added between the frames, and the pixel value of the captured image can be restored based on the relational expression. Thereby, the restoration estimation process in the time axis direction can be simplified. For example, complicated processing such as iterative calculation of a two-dimensional filter (Patent Document 1) and searching for a part suitable for setting an initial value (Patent Document 2) becomes unnecessary.

13. Next, based on the intra-frame addition images A _{1 to} A ₄ described above with reference to FIG. 3 and the like, the pixel value v _ij (1) ′ of the reference image B ₁ before the addition or the inter-frame difference A method for estimating the pixel value dt _ij (T) of the image DT _T will be described.

FIGS. 18A and 18B are explanatory diagrams of the estimated pixel value and the intermediate pixel value. Summing the pixel values shown in FIG. _{18 (A) {a 00,} a 10, a 11, a 01} is added pixel value of the added image _A 1 to A ₄ described in FIG. _{^{3 {a (1) 00,}} a ( ²⁾ Corresponds to ₁₀ , a ⁽³⁾ ₁₁ , a ⁽⁴⁾ ₀₁ }. In the estimation process, final estimated pixel values v _{00 to} v ₂₂ are estimated using the added pixel values. The estimated pixel value v _ij corresponds to the pixel value v _ij (1) ′ of the reference image B ₁ described in FIG. 3 or the pixel value dt _ij (T) of the inter-frame difference image DT _T.

As shown in FIG. 18B, first, intermediate pixel values b _{00 to} b ₂₁ are estimated from the added pixel values a _{00 to} a ₁₁ . Intermediate pixel value corresponds to 2-pixel sum values, for example, _{b 00} corresponds to the sum of the pixel values _{v 00} and _{v 01.} The final pixel values v _{00 to} v ₂₂ are estimated from these intermediate pixel values b _{00 to} b ₂₁ .

The intermediate pixel values b _{00 to} b ₂₁ can be obtained in the same manner as the above estimation process. That is, if the above equation (16) is replaced with the following equation (21), the subsequent processing is the same.
a ₀₀ = b ₀₀ + (1/2) b ₁₀ ,
a ₁₀ = b ₁₀ + (1/2) b ₂₀ (21)

The process of estimating the estimated pixel values v _{00 to} v ₂₂ from the obtained intermediate pixel values b _{00 to} b ₂₁ can also be performed in the same manner as the above-described estimation process. That is, if the above equation (16) is replaced with the following equation (22), the subsequent processing is the same.
b ₀₀ = v ₀₀ + (1/2) v ₀₁ ,
b ₀₁ = v ₀₁ + (1/2) v ₀₂ (22)

According to the above embodiment, as shown in FIG. 18A, the addition unit (eg, a ₀₀ ) set at the first position and the second position after the first position are set. The addition unit (for example, a ₁₀ ) is overlapped. In the same manner as the above equation (17), the intra-frame estimation calculation unit 230 obtains a difference value δi ₀ between the added pixel values a ₀₀ and a ₁₀ at the first and second positions. As shown in FIG. 18B, the first intermediate pixel value b ₀₀ is an addition of the first area (v ₀₀ , v ₀₁ ) obtained by removing the overlapping area (v ₁₀ , v ₁₁ ) from the addition unit a _00. It is a pixel value. The second intermediate pixel value b ₂₀ is an addition pixel value of the second region (v ₂₀ , v ₂₁ ) obtained by removing the overlap region (v ₁₀ , v ₁₁ ) from the addition unit a ₁₀ . Similar to the above equation (18), the relational expression of the first and second intermediate pixel values b ₀₀ and b ₂₀ is expressed using the difference value δi ₀ . Then, the first and second intermediate pixel values b ₀₀ and b ₂₀ are estimated using the relational expression. Using the estimated first intermediate pixel value b ₀₀ , pixel values (v ₀₀ , v ₁₀ , v ₁₁ , v ₀₁ ) of each pixel included in the addition unit are obtained.

  In this way, the intermediate pixel value is once estimated from the added pixel value that is superimposed and shifted in the frame, and the estimated pixel value is obtained from the superimposed and shifted intermediate pixel value, thereby performing the estimation processing of the high resolution image. It can be simplified. For example, complicated processing such as iterative calculation of a two-dimensional filter (Patent Document 1) and searching for a part suitable for setting an initial value (Patent Document 2) becomes unnecessary.

Here, superimposing means having an area where the addition unit and the addition unit overlap. For example, as shown in FIG. 18A, the addition unit a ₀₀ and the addition unit a ₁₀ are two estimated pixels v _10. is to share the _{v 11.}

  Further, the position of the addition unit is the position and coordinates of the addition unit in the captured image, or the position and coordinates of the addition unit on the estimated pixel value data (image data) in the estimation process. The next position is a position shifted from the original position by a pixel, and is a position where the position and coordinates do not coincide with the original position.

In this embodiment, continuous intermediate pixel values including the first and second intermediate pixel values (for example, b ₀₀ , b ₂₀ ) are defined as intermediate pixel value patterns ({b ₀₀ , b ₁₀ , b ₂₀ }). Similarly to the above equation (18), the intra-frame estimation calculation unit 230 represents the relational expression between the intermediate pixel values included in the intermediate pixel value pattern using the added pixel values a ₀₀ and a ₁₀ . Similar to the method described in FIG. 17, the similarity is evaluated by comparing the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the added pixel value. Based on the similarity evaluation result, the intermediate pixel values b ₀₀ , b ₁₀ , and b ₂₀ included in the intermediate pixel value pattern are determined so that the similarity is the highest.

  In this way, the intermediate pixel value can be estimated based on a plurality of added pixel values acquired by pixel shifting while the addition unit is superimposed.

  Here, the intermediate pixel value pattern is a data string (a set of data) of intermediate pixel values in a range used for estimation processing. The addition pixel value pattern is a data string of addition pixel values in a range used for the estimation process.

Further, in the present embodiment, in the same manner as the above equation (20), the intra-frame estimation calculation unit 230 performs an intermediate pixel value pattern ({b ₀₀ , b ₁₀ , b ₂₀₎ represented by a relational expression between intermediate pixel values. }) And an evaluation function E representing an error between the added pixel values (a ₀₀ , a ₁₀ ). Intermediate pixel values b ₀₀ , b ₁₀ , and b ₂₀ included in the intermediate pixel value pattern are determined so that the value of the evaluation function E is minimized.

  In this way, the value of the intermediate pixel value can be estimated by expressing the error by the evaluation function and obtaining the intermediate pixel value corresponding to the minimum value of the evaluation function. For example, as described above, the initial value of the intermediate pixel estimation can be set with a simple process by obtaining the unknown using the least square method. For example, searching for an image portion suitable for initial value setting (Patent Document 2) is unnecessary.

In the present embodiment, as shown in the above equation (5), an addition pixel value (a ₀₀ ) obtained by weighting and adding each pixel value (for example, v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ ) of the addition unit is used. get. Based on the obtained addition pixel value (a ₀₀ , a ₁₀ ) of the addition unit, the pixel value (v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ ) of each pixel of the addition unit is estimated.

  In this way, each pixel value in the addition unit is weighted and added to obtain an intra-frame addition image, and the pixel value of the high-resolution image can be estimated from the obtained intra-frame addition image. Thereby, in the estimation process, the reproducibility of the high-frequency component of the subject can be improved. That is, when the pixel values of the addition unit are simply added, a rectangular window function is convoluted for imaging. On the other hand, when the pixel value of the addition unit is weighted and added, a window function containing more high frequency components than the rectangle is convoluted for imaging. Therefore, it is possible to acquire an added image that includes more high-frequency components of the subject, and to improve the reproducibility of the high-frequency components in the estimated image.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configurations and operations of the imaging unit, the image processing unit, the imaging device, and the like are not limited to those described in the present embodiment, and various modifications can be made.

10 imaging unit, 20 image processing unit, 100 subject, 110 lens,
120 image sensors, 122 inter-frame addition image generation unit,
124 inter-frame difference image generation unit, 130 intra-frame addition image generation unit,
141 intra-frame difference image generation unit, 142 average image generation unit,
143 Entropy encoding unit, 144 data compression unit,
150 compressed data recording unit, 155 data recording unit, 200 compressed data decompression unit,
210 intra-frame addition image reproduction unit, 214 inter-frame addition image reproduction unit,
230 intra-frame estimation calculation unit, 234 inter-frame estimation calculation unit,
240 high-definition still image generator, 250 high-definition video generator,
260 standard video generation unit, 290 image output unit, 295 image selection unit,
A ₁ -A ₄ intra-frame addition images, B ₁ -B ₃ inter-frame addition images,
D _{1 to} D ₄ intra-frame difference images, DT ₂ , DT ₃ inter-frame difference images,
E evaluation function, a _ij addition pixel value, b _ij intermediate pixel value, dt _ij difference pixel value,
f _{1 to} f ₃ captured frame images, m _AVR average value, r weighting coefficient,
v _ij pixel value, v _ij (1) unknown, α ^M _ij average value, δi ₀ difference value

Claims (10)

An image acquisition unit for acquiring the first to third captured images in the first to third frames;
The pixel value of the first captured image and the pixel value of the second captured image are weighted and added to generate a first inter-frame added image, and the pixel value of the second captured image and the third captured image An inter-frame addition image generation unit that generates a second inter-frame addition image by weighted addition of pixel values of the image;
An inter-frame difference image generation unit that generates a difference between the first inter-frame addition image and the second inter-frame addition image as an inter-frame difference image;
An inter-frame addition image reproduction unit that reproduces the second inter-frame addition image based on the first inter-frame addition image and the inter-frame difference image;
Based on the first inter-frame addition image and the reproduced second inter-frame addition image, an inter-frame estimation calculation unit that estimates pixel values of the first to third captured images;
An image output unit that outputs a high-resolution image based on the pixel value estimated by the inter-frame estimation calculation unit;
An imaging apparatus comprising: In claim 1,
The inter-frame estimation calculation unit is
Obtaining a difference value between a pixel value of the first inter-frame addition image and a pixel value of the second inter-frame addition image;
A relational expression between the pixel value of the first captured image and the pixel value of the third captured image is expressed using the difference value,
An image pickup apparatus that estimates pixel values of the first to third picked-up images using the relational expression. In claim 2,
The inter-frame estimation calculation unit is
The relational expression between the pixel values of the first to third captured images is expressed using the pixel values of the first and second inter-frame addition images,
The pixel values of the first to third captured images represented by the relational expression are compared with the pixel values of the first and second inter-frame addition images to evaluate similarity,
An imaging apparatus, wherein pixel values of the first to third captured images are estimated so that the similarity is the highest based on the similarity evaluation result. In any one of Claims 1 thru | or 3,
The first inter-frame addition image or the inter-frame difference image is received as an input image, an addition unit, which is a unit for obtaining an addition pixel value, is set for each of the plurality of pixels of the input image, and the addition unit is sequentially An intra-frame addition image generation unit that generates a first to n-th intra-frame addition image by weighted addition of pixel values included in the addition unit while performing pixel shift;
An average image generating unit for generating an average image of the first to n-th intraframe addition images;
A frame that generates a difference between the mth intraframe addition image (m is a natural number equal to or less than n) of the first to nth intraframe addition images and the average image as an mth intraframe difference image. An internal difference image generation unit;
A data compression unit for compressing the average image and the mth intra-frame difference image;
A compressed data decompression unit for decompressing the compressed average image and the mth intra-frame difference image;
An intra-frame addition image reproduction unit for reproducing the first to n-th intra-frame addition images based on the expanded average image and the m-th intra-frame difference image;
An intra-frame estimation calculation unit that estimates a pixel value of the input image based on the reproduced first to n-th intra-frame addition images;
An imaging apparatus comprising: In claim 4,
The intra-frame addition image generation unit
The addition unit is sequentially shifted one pixel at a time horizontally or vertically to set the first to n-th positions, and the first to n-th intra-frame addition images are obtained at the first to n-th positions, respectively. And
The imaging apparatus, wherein the addition unit of the m-th position and the (m + 1) -th position among the first to n-th positions includes a common pixel. In claim 4 or 5,
The data compression unit
An imaging apparatus, wherein the first to n-th frame difference images are reversibly compressed by entropy coding. In any one of Claims 4 thru | or 6.
When the addition unit set in the first position and the addition unit set in the second position next to the first position overlap,
The estimation calculation unit includes:
A difference value between the added pixel value of the first position and the added pixel value of the second position;
A first intermediate pixel value that is an addition pixel value of the first area obtained by removing the overlap area from the addition unit of the first position, and a second that is obtained by removing the overlap area from the addition unit of the second position. A relational expression with the second intermediate pixel value that is the added pixel value of the region is expressed using the difference value,
The first and second intermediate pixel values are estimated using the relational expression, and the pixel value of each pixel included in the addition unit is obtained using the estimated first intermediate pixel value. Imaging device. In claim 7,
The estimation calculation unit includes:
When successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern, a relational expression between intermediate pixel values included in the intermediate pixel value pattern is obtained using the added pixel value. Represent,
Comparing the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the added pixel value to evaluate similarity;
An image pickup apparatus, comprising: determining an intermediate pixel value included in the intermediate pixel value pattern based on the similarity evaluation result so that the similarity becomes the highest. In any one of Claims 1 thru | or 3,
Receiving the x-th picked-up image (x is a natural number of 3 or less) among the first to third picked-up images, an addition unit, which is a unit for obtaining an added pixel value, is a plurality of the x-th picked-up images. An intra-frame addition image generation unit that sets for each pixel and generates the first to n-th intra-frame addition images by weighted addition of pixel values included in the addition units while sequentially shifting the addition units.
An average image generating unit for generating an average image of the first to n-th intraframe addition images;
An intra-frame difference image generation unit that generates a difference between the m-th intra-frame addition image of the first to n-th intra-frame addition images and the average image as an m-th intra-frame difference image;
Including
The inter-frame addition image generation unit
The first frame based on the first and second input images by receiving the mth intra-frame difference image or the average image as the xth input image of the first to third input images. An imaging apparatus that generates an inter-frame addition image and generates the second inter-frame addition image based on the second and third input images. Obtaining the first to third captured images in the first to third frames;
The pixel value of the first captured image and the pixel value of the second captured image are weighted and added to generate a first inter-frame added image, and the pixel value of the second captured image and the third captured image Weighted addition of pixel values of the image to generate a second inter-frame addition image,
A difference between the first inter-frame addition image and the second inter-frame addition image is generated as an inter-frame difference image;
Based on the first inter-frame addition image and the inter-frame difference image, reproduce the second inter-frame addition image,
Based on the first inter-frame addition image and the reproduced second inter-frame addition image, the pixel values of the first to third captured images are estimated,
A method for generating an image, comprising: outputting a high-resolution image based on a pixel value estimated by the inter-frame estimation calculation unit.

Images