JP2000244851A - Image processing apparatus, method, and computer-readable storage medium - Google Patents

Abstract

(57) [Problem] To create a single high-resolution still image by synthesizing a plurality of frames of a moving image so that errors can be reduced and good synthesis can be performed even when there are unclear frames. To SOLUTION: A selection unit 102 selects one reference still image based on edge information or the like from continuous (n + 1) pieces of moving image information of m to (m + n) frames stored in a storage unit 101. Then, the arrangement unit 105 arranges the selected reference still image in the memory. Next, the motion vector calculation unit 104
Calculates a motion vector for each of the n still images other than the reference still image with respect to the reference still image. Arrangement part 1
In step 07, the n still images are arranged at positions different from the arrangement points of the reference still images in the memory based on the calculation results. The synthesizing unit 108 sets the (n +
1) One image is generated by synthesizing one image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image output apparatus, such as a printer, which enlarges and resizes input image information, and communication between models having different resolutions. TECHNICAL FIELD The present invention relates to an image processing apparatus and method suitable for use in converting a resolution into a plurality of images, and a computer-readable storage medium used in the apparatus and method.

[0002]

2. Description of the Related Art Conventionally, various methods have been proposed for converting the resolution of low-resolution information of an input image into high-resolution information. In these conventional methods, the types of target images (for example, a multi-valued image having gradation information for each pixel, a binary image binarized by a pseudo halftone, and a binary image binarized by a fixed threshold) are used. Value image, character image, etc.)
The conversion processing method is different.

As a conventional interpolation method, as shown in FIG. 12, a nearest interpolation method in which the same pixel values closest to the interpolation point are arranged, and four points (4) surrounding the interpolation point as shown in FIG. In general, a bilinear interpolation method or the like that determines the pixel value E by the following calculation based on the distances of the pixel values of the points A, B, C, and D) is used. E = (1−i) · (1−j) · A + i · (1−j) · B + (1−i) · j · C + i · j · D (1) (However, distance between pixels Is assumed to be 1, it is assumed that there is a distance of i in the horizontal direction and j in the vertical direction from A (i ≦ 1, j ≦
1)).

As a means for converting a sampled discrete signal into a continuous signal, as represented by the sampling theorem for a long time, reproduction by passing through an ideal low-pass filter that can be expressed by a SINC function Can be. Since it takes a long processing time to calculate the SINC function, there is a method of approximating an interpolation function represented by the SINC function and calculating an interpolation value only by a simple product-sum operation.

According to "Image Analysis Handbook: Mikio Takagi and Hirohisa Shimoda, University of Tokyo Press", cubic convolution interpolation method (Cubic Convolution interpolation)
polation), approximation of the interpolation function can be realized. The image data to be obtained is interpolated using the cubic convolution function represented by the following equation using the image data of 16 observation points around the point to be interpolated.

[0006]

(Equation 1)

[0007]

(Equation 2)

[0008]

(Equation 3)

In the expression (2), Pn to P44 indicate peripheral pixel values, and the arrangement is shown in FIG.

However, in the above three types of conventional examples, blurring due to interpolation and a block-shaped jaggy depending on the input low resolution occurred at the time of interpolation, and high-resolution high-resolution information could not be created. In view of the above, the present applicant has proposed a method capable of performing resolution conversion without interpolating blur due to interpolation processing in generating high-resolution information from low-resolution information and without causing jaggies, as disclosed in Japanese Patent Application Laid-Open No. 7-93531. And JP-A-7-107268 and JP-A-7-105359.

The basic idea of these proposals is that the resolution-dependent component is removed from the input original information, and the number of pixels is increased to the output resolution in a state where the resolution-dependent component is removed. This is a method of guessing and creating information that is appropriate for. As means for removing the dependency on the input resolution, smoothing by LPF and increasing the number of pixels can be realized by linear interpolation. The estimation of the high-resolution information is performed by simply binarizing the information after the interpolation and performing different processing on the pixel classified as “1” and the pixel classified as “0”, thereby outputting the pixel to be output. Calculate the value.

Further, as proposed in Japanese Patent Application Laid-Open No. 9-252400, there is a method of creating a good edge while maintaining the continuity of pixel values. In this publication, a pixel at m points (m ≧ 1) (where the pixel value at an observation point n among the m points is P (n)) is detected from the peripheral pixels of the low resolution target pixel, and a plurality of target pixels are detected. The output value h (k) is calculated by the following equation based on the interpolation value C (k) at each interpolation point k interpolated into pixels.

[0013]

(Equation 4)

[0014]

However, the above-mentioned prior art has the following drawbacks. That is, no matter how much high-resolution information is created, there is a limit to high image quality. Naturally, as is clear from the sampling theorem, since information of the input resolution equal to or higher than the Nyquist limit does not exist in the input image, creation of information equal to or higher than the Nyquist frequency is based on estimation.

Therefore, a CG image which is not so complicated,
It is easy to convert flat artificial images such as illustration images and animation images to jaggy-less,
It is difficult to achieve high image quality by estimating information beyond the Nyquist limit of a natural image. That is, no matter what method is used, the quality of an image obtained by inputting low-resolution information and converting it to high resolution is clearly lower than that of an image originally inputting high-resolution information.

On the other hand, in recent years, with the spread of digital video cameras and the like, means for inputting captured moving images to a computer in a continuous frame unit have been increasing. However, although the output resolution of the printer is increasing year by year, the input resolution of the imaging system is increasing, but it is still lower than the printer resolution.

Therefore, as described in the prior art,
Instead of creating one high-resolution still image from one low-resolution still image, one high-resolution still image is created from a plurality of continuous low-resolution still images captured from a moving image. A technique is proposed according to the invention. Conventionally, a method of creating a panoramic image of a wider range from a plurality of still images is described in "Synthesis of Background Image in Consideration of Panning of Moving Image: Yoshizawa, Hanamura, Tominaga, Shingaku Spring Full-Size Proceedings 7-51 (1990) )), And “Method of generating panoramic image by divided imaging: Nakamura, Kaneko, Hayashi, Shingaku Spring Full-Size Preliminary Proceedings, 7-165 (1991)” and the like.

However, instead of creating a panoramic image whose imaging range is larger than that of a single still image, the imaging range is the same, information of a plurality of still images is synthesized, and the resolution of the image is increased by interpolation. There are few proposals for techniques to improve.

As a technique for creating a high-resolution still image from a low-resolution moving image, Japanese Patent Application Laid-Open No. 5-260264 proposes a technique. In this proposal, continuous images are compared with each other, and parameters of affine transformation and translation are detected based on a difference between the two types of images to synthesize the two types of images. The second embodiment of the above proposal describes an example in which synthesis is used for interpolation.

However, the above proposal has the following problems. That is, the method described in the second embodiment of the above publication compares the continuous images enlarged by the above-described interpolation method shown in FIGS. Are determined and synthesized. However, since the interpolation calculation itself does not create new high-resolution information, it is difficult to accurately determine coordinates to be combined.

Interpolating means interpolating between pixels. In the above method, there is no information between pixels of the input resolution when comparing continuous images. In other words, assuming that two types of images are image A and image B,
It is difficult to determine where to interpolate the pixels of the image B between the pixels of the image A by simply comparing the enlarged images.

This is because the minimum unit of the vector amount of the motion vector is a pixel unit, and there is no resolution finer than the distance between pixels. That is, if the resolution of the vector does not have an accuracy equal to or less than the pixel-to-pixel accuracy, the effect of interpolating using a plurality of still images is diminished, and one low-resolution still image is replaced by one high-resolution still image described in the conventional example. The image quality is almost the same as in the case of creating a still image.

"Basic technology of international standard image coding: Fumitaka Ono, Hiroshi Watanabe" describes various motion detection methods. However, any of the methods described in this description is a detection method for the purpose of motion compensation, which is different from the object of the present invention in which one image is created from a plurality of images, and therefore, fine detection accuracy is unnecessary. Therefore, even if these techniques are used, it is difficult to successfully combine a plurality of images.

Then, the present applicant utilizes orthogonal transformation to
A calculation method of vectors with finer resolution than the distance between pixels was proposed. According to this method, a plurality of still images are not independent from each other, and it is possible to associate spatial coordinates between images. However, even if the resolution of the motion vector is calculated finer than the pixel-to-pixel distance, only the relative position between the images can be correctly grasped, and many problems still remain.

One of the problems is that an error at the time of combining causes deterioration of image quality. That is, as the number of images to be synthesized increases, errors in motion vectors accumulate.

FIG. 15 is a diagram showing a conventional order of calculating motion vectors. In the figure, an example will be described in which images for four consecutive frames from the time m frame to the (m + 3) frame are synthesized. 1501, 1502,
Reference numerals 1503 and 1504 denote image information of the m-th frame, image information of the (m + 1) frame, image information of the (m + 2) frame, and image information of the (m + 3) frame, respectively.

The order of calculating the motion vector is as follows: first, the m-th frame and the (m + 1) -th frame, and then (m
(+1) th frame and (m + 2) th frame, then (m
(+2) th frame and (m + 3) th frame are calculated three times. That is, similarly to the motion compensation of the conventional moving picture coding, this is a method of sequentially calculating the motion amount after one frame progresses. In this case, a frame serving as a reference for calculating a motion vector (referred to as a reference frame) always has a temporal difference of only one frame from a target frame (referred to as a target frame).

However, even if this method is optimal for the purpose of motion compensation, there are some problems in synthesizing a plurality of images as the object of the present invention. One of them is accumulation of the above error. In other words, if the movement amount for one frame is not accurately calculated, a new movement amount is calculated for an image in which an error has occurred in temporally subsequent motion vector calculation. As the number of images increases, the accumulated error increases, which may result in a large difference from the original arrangement position.

Another problem is to cope with a case where a motion vector is difficult to calculate in the middle of a plurality of continuous images. Naturally, in a plurality of continuous images, unclear image frames also exist due to blurring on the input device side such as a video camera or the movement of a target object during imaging. In this case, if the conventional method as shown in FIG. 15 is used, there is a possibility that a motion vector may be largely incorrect as a result. The error is not eliminated.

The problem of the order of calculating the motion vector is, for example, a problem that occurs in a video camera that is already on the market at present by using a technique of shifting pixels of a CCD to achieve high resolution. It does not matter if you combine them. This is because the “shift amount” corresponding to the relative position of each image is controlled on the device side. But,
In the embodiment of the present invention described later, the relative position of each image is not controlled at all. Therefore, the order in which the motion vectors are calculated has a significant factor in improving image quality.

The inventor of the present invention has proposed a method of processing a plurality of images by processing image data so as to conform to a reference frame rather than simply arranging pixel values of a plurality of frames. Suggested how to. In a method of processing image data, a synthesized image having completely different image quality is obtained depending on which frame is used as a reference frame. In the method of combining after data processing, not only when the number of combined frames is three or more, but also when two, the image quality differs depending on which image is used as the reference frame. That is, conventionally, when combining a plurality of images,
A good frame control method has not been proposed that compares which images are compared to determine the amount of motion and leads to synthesis.

Accordingly, the present invention proposes a frame control method in a case where a plurality of images are combined to obtain one high-resolution image.

[0033]

In order to achieve the above object, in an image processing apparatus according to the present invention, a continuous (n +) frame from an m-th frame to an (m + n) -th frame is provided.
1) selecting means for selecting a first still image from pieces of moving image information (m and n are arbitrary natural numbers); and (n + 1)
Storage means for arranging and storing a plurality of moving images, first arranging means for arranging the selected first still image in the storage means, and n images other than the first still image Calculating means for calculating a motion vector with respect to the first still image for each of the still images, and n based on the calculation result, at a position different from the arrangement position of the first still image in the storage means. A second arranging unit for arranging the still images and a synthesizing unit for synthesizing the (n + 1) images after the arrangement to obtain one image are provided.

In the image processing method according to the present invention, a selection procedure for selecting a first still image from (n + 1) consecutive pieces of moving image information from the m-th frame to the (m + n) -th frame is provided. An arranging means for arranging the selected first still image in the storage means, and an operation procedure for calculating a motion vector for the first still image for each of n still images other than the first still image. And an arrangement procedure for respectively arranging the n still images in a position different from the arrangement position of the first still image in the storage means based on the operation result, and (n + 1) after the arrangement. And a synthesizing procedure for synthesizing one image to obtain one image.

Further, in the storage medium according to the present invention, a selection process of selecting a first still image from (n + 1) consecutive pieces of moving image information from the m-th frame to the (m + n) -th frame, An arranging process of arranging the selected first still image in the storage means, and an arithmetic process of calculating a motion vector for each of the n still images other than the first still image with respect to the first still image. An arrangement process of arranging the n still images at positions different from the arrangement positions of the first still images in the storage means on the basis of the calculation results; and (n + 1) images after the arrangement And a synthesizing process for synthesizing these images to obtain one image.

[0036]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing an image processing apparatus according to an embodiment of the present invention. Note that the image processing apparatus according to the present embodiment is mainly used for image processing such as a printer or a video printer which is connected to an analog video camera or a digital video camera for capturing a moving image, or directly or via a computer. Although it is efficient to have it inside the output device, it is built in as an image processing device that becomes an intermediate adapter when connecting the video camera and the printer, or application software in the host computer, or printer driver software for outputting to the printer It is also possible.

The configuration and operation of the image processing apparatus shown in FIG. 1 will be described. In FIG. 1, reference numeral 100 denotes an input terminal to which a moving image captured by a video camera is input. In this embodiment, an example will be described in which an image captured by a digital video camera is transmitted to a computer and converted to a resolution equivalent to a printer by application software in the computer.

A moving image captured by digital video is reproduced from a recording medium, and a user sends a command to capture an image in a scene desired by the user. In synchronization with this fetch instruction, image information of a plurality of continuous frames immediately after the fetch instruction is stored in the storage unit 101 in the computer. Now, the fetch instruction is set to the m-th frame at the time, and (m
It is assumed that (n + 1) image information up to the (+ n) th frame is stored.

Reference numeral 102 denotes a selection unit which stores (n +
1) It is determined which image at which time is to be set as the reference frame from the image information of the frame. Now, the reference frame set based on the determination result is temporarily set to a frame G. Reference numeral 103 denotes a frame control unit, which is a unit for selecting two types of images to be processed. The two types of images are a frame G, one of which is a reference frame, and the other is one of n frames stored other than the frame G (referred to as a frame H).

Reference numeral 104 denotes a motion vector calculation unit, which is a means for measuring, as a vector, the movement amount of the partial movement based on the difference between the two images of the frame G and the frame H.

Reference numeral 105 denotes an arrangement processing unit A, which is means for arranging a captured image of the frame G in a memory. This memory has an address space of (number of input frame pixels) × (vertical magnification) × (horizontal magnification) or more. Therefore, the arrangement processing unit A executes the arrangement of the pixels corresponding to the predetermined enlargement ratio. For example, when the enlargement ratio is double in both the horizontal and vertical directions, the pixels of the frame G are arranged every other pixel in both the vertical and horizontal directions.

Reference numeral 106 denotes a data processing unit, and a frame G
Means for processing the pixel values of the frame H so as to match the image information of the frame H well.

Reference numeral 107 denotes an arrangement processing unit B, which is means for arranging in the same memory as the arrangement processing unit A 105 in accordance with the relative vector amount between the frame G and the frame H calculated by the frame control unit 103. Number of stored frames is 3
If there are more frames, the process returns to the frame control unit 103 again, and the same processing as described above is repeated for a new frame. However, in the second and subsequent processes, the frame G
Is fixed, and the image information of the frame G has already been arranged, so there is no need to newly arrange it. Only the frame H is updated to a new frame, and the above-described processing steps of motion vector calculation, data processing, and arrangement are executed.

Reference numeral 108 denotes a synthesizing unit, which synthesizes a plurality of images arranged in the same memory into one image information.
Reference numeral 109 denotes an interpolation operation unit that calculates information on interpolation points that are not embedded by interpolation when information on interpolation points up to a desired resolution is not yet embedded in the synthesized image.
Reference numeral 110 denotes an output terminal, to which image information having a higher resolution is transmitted to a printer or the like.

FIG. 2 shows a configuration of the selection unit 102 which is a feature of the present embodiment. As an example, it is assumed that the storage unit 101 stores a total of four continuous images from the image in the m-th frame to the image in the (m + 3) -th frame. 20
Reference numerals 1, 202, 203, and 204 denote edge extraction units, respectively, which are means for extracting edge information included in an image based on the stored four pieces of image information.

FIG. 3 shows an example of a general Laplacian edge extraction filter. Now, the pixel value at the coordinates (x, y) on the image of the (m + s) th frame (where 0 ≦ s ≦ 3) is f _s (x, y), and the value after the edge extraction processing is k
_{Assuming that s} (x, y), the following product-sum operation is performed in the edge extraction filter of FIG.

[0047] _{k s (x, y) =} f s (x-1, y-1) + f s (x, y-1) + f s (x + 1, y-1) + f s (x-1, y) _{-8f s (x, y) +} f s (x + 1, y) + f s (x-1, y + 1) + f s (x, y + 1) + f s (x + 1, y + 1) ···· (6)

In FIG. 2, 205, 206, 207,
Reference numeral 208 denotes an edge strength evaluation unit, which is means for integrating the edge strengths extracted by the edge extraction units 201 to 204 over the entire image. Assuming that the number of vertical pixels in the entire image is V and the number of horizontal pixels is H, the edge strength Ps of the (m + s) frame is calculated as follows.

[0049]

(Equation 5)

Reference numeral 209 denotes a maximum edge strength determining unit which determines a frame s in which Ps obtained by the edge strength evaluating units 205 to 208 is the maximum as a reference frame.
That is, a unique evaluation function called edge strength is set, and a frame evaluated as having the highest edge strength over the entire image is selected from a plurality of stored images.

Setting an image having a large edge strength as a reference frame is advantageous both when calculating a motion vector, which will be described later, and when processing data for another target frame. The evaluation based on the edge intensity can also be assumed to be an image whose focus is sharpest among a plurality of stored images. Therefore, the image of the target frame other than the reference frame plays a role of adding an added value to the image quality of the reference frame, and the image quality improvement of at least the reference frame alone is guaranteed.

Although the configuration of FIG. 2 has been described with respect to an example in which all frames are processed in parallel for ease of explanation, it is needless to say that the edge extraction unit and the edge strength evaluation unit are single.
A configuration for processing vertically may be used. Further, in Formula _{(7) k s' (x} , y) calculation of k _s (x, y) had used the absolute value of the course, by using the square of k _s (x, y) calculation It is also possible.

Next, the motion vector calculation unit 104 will be described. Various methods have been proposed for calculating a motion vector since ancient times. However, in the conventional method, since there is no resolution of a vector equal to or less than the distance between pixels, synthesis,
It is not suitable for use in performing interpolation to convert a low-resolution moving image into a high-resolution still image.

FIG. 4 is a detailed block diagram of the motion vector calculation unit 104 according to the present embodiment. Storage unit 10 of FIG.
The two types of images transmitted from 1 to the motion vector calculation unit 104 are a frame G as a reference frame and a frame H as a target frame.

In FIG. 4, a blocking unit 401 blocks image information of a frame H in units of N × N pixels. There are various possible values of N, but N = 8 is assumed as an example. Now, the created target block of 8 × 8 pixels is temporarily referred to as a block A. Next, the orthogonal transform unit 402 calculates the orthogonal transform of the block A. The type of orthogonal transform is not limited, but Hadamard transform, which can be easily operated at high speed, and JPEG (Joint Photographi)
c DC used in Expert Group)
T (discrete cosine transform) and the like are common. Now DC
T as an example, the transform coefficient of a two-dimensional DCT of N × N pixels is

[0056]

(Equation 6)

Is obtained by

On the other hand, the reference frame G is divided into blocks of M × M ′ pixels by the M × M ′ blocking unit 403. At this time, the block in units of M × M ′ pixels includes a block of N × N pixels having the same coordinates as the block A in the frame H, and the magnitude relation is M ≧ N and M ′ ≧ N (where,
M = M ′ = N (except the case). Now, M = M '=
Assume 20. That is, 2 including the same coordinates as block A
A block of 0 × 20 is prepared in the frame G.

Next, the N × N blocking unit 404
Within a block of 20 × 20 pixels, a block of N × N pixels of the same size as block A is created. The creation of the block may start from the same coordinates as block A,
Further, it may be the first time from the end of the M × M ′ block. Now, a block of N × N pixels created in the frame G is temporarily referred to as a block B.

The orthogonal transform unit 405 generates the block B
Are orthogonally transformed in the same manner as in the block A. Naturally, the orthogonal transformations of the orthogonal transformation units 402 and 405 must be the same transformation means. The orthogonal transform coefficient evaluator 406 includes a block A,
Based on the orthogonal transform coefficients of the block B, the similarity of the transform coefficients is evaluated. The evaluation of similarity is based on the DC (direct current) component of the block and the AC (alternating current) component mainly in the low frequency range, and multiplies the difference between the respective coefficients by a weighting coefficient corresponding to the component. Evaluate by sum.

For the sake of simplicity, the coordinates of the block will be managed by the coordinates of the upper left pixel forming the block (hereinafter, the coordinates of this pixel will be referred to as the origin of the block). That is, as shown in FIG. 5, if the origin of block B (corresponding to the hatched pixel) is (a, b), the similarity evaluation function between block A and block B is:

[0062]

(Equation 7)

Is calculated.

The higher the frequency, the lower the correlation of the transform coefficients between adjacent blocks. Therefore, the higher the frequency, the smaller the value of the weighting coefficient W (u, v) is set. Since the transform coefficients in the low-frequency range between blocks whose coordinates are spatially close to each other have a very high correlation, the expression (9) evaluates the blocks by replacing the spatial positional relationship between the blocks with the similarity of the transform coefficients. . Further, although the absolute value is used in the equation (9), the same evaluation can be performed by the square of the difference.

Next, the block controller 407 moves the origin (a, b) of the block B by one pixel, creates a new block, and repeats the same processing. That is, N = 8, M =
Taking M ′ = 20 as an example, a block of 8 × 8 pixels is 20
Since 13 × 13 pixels can be created in a block of × 20 pixels, similarity is repeatedly calculated for the number of blocks.

When the scanning of all the blocks B in the frame G is completed, the above-mentioned evaluation function R (a, b) is obtained.
Are determined at which coordinates (a ′, b ′) are minimized. That is, since the similarity R (a, b) can be regarded as an error component between the blocks AB, the block B when the minimum value of R (a, b) is taken
(The block at this time is referred to as a block B ′) is regarded as a block that is spatially closest to the block A, and is determined to be a destination to which the block A has moved.

However, only in this case, as in the conventional example, the resolution of the motion vector is on a pixel-by-pixel basis, and it is not possible to judge a vector smaller than the pixel-to-pixel distance. Therefore, in the present embodiment,
A motion vector is estimated with a resolution shorter than the distance between pixels.

The method for estimating the vector will be described below. In the above-described method, the origin of the block A, which is the target block, on the frame H, which is the target frame, is set to (a0, b0).
Further, the origin of the block B ′ of the frame G that takes the minimum value of R (a, b) is (a ′, b ′).
In the conversion coefficient evaluation unit 406, the search for the block B 'is a rough pixel-by-pixel search, but this time, a small distance is narrowly estimated around the block B'.

That is, the conversion coefficient evaluation unit 406 first
Searching for the block B 'that seems to be the closest in space, estimating the amount of slight deviation from the block B' found next,
A different assessment of the two-stage composition will be performed.

FIG. 6 is a flowchart showing an operation procedure of the second-stage estimation. Step S601 (hereinafter, referred to as step S601)
The step (omitted) compares the evaluation function result of the equation (7) between the block created one pixel left of the block B ′ and the block created one pixel right of the block B ′. That is, since the origin of the block B ′ is (a ′, b ′), R (a ′ + 1, b ′)
And the magnitude of R (a'-1, b ') are evaluated. This R
Since (a '+ 1, b') and R (a'-1, b ') are calculated at the time of the first-stage similarity evaluation, it is preferable to store and hold the calculation results. .

In S601, if R (a '+ 1,
If b ′) is evaluated to be small, the process moves to S602, and if b ′) is evaluated to be no, the process moves to S603. Next, in S602, a block composed of the origin R (a '+ 1, b') is set as a block C. In S603, the origin R (a'-
A block composed of (1, b ′) is set as a block C. At the same time, the variable c is set to c = 1 in S602, and c = −1 in S603.

Next, in S604, the evaluation function results of the equation (7) of the block created one pixel above the block B 'and the block created one pixel below the block B' are compared. That is, since the origin of the block B 'is (a', b '), the magnitudes of R (a', b '+ 1) and R (a', b'-1) are evaluated. Since the similarity evaluation function is also calculated at the time of the first-stage similarity evaluation, it is preferable to store and hold the calculation result.

At S604, if R (a ', b'
If +1) is evaluated to be small, the process proceeds to S605, and if not, the process proceeds to S606. Next, in S605, a block composed of the origin R (a ', b' + 1) is set as a block D, and in S606, the origin R (a ', b' + 1) is set.
The block composed of b′-1) is set as a block D. At the same time, the variable d is set to d = 1 in S605, and d = −1 in S606.

Next, in step S607, the orthogonal transform of the block A is performed.
F, which is the horizontal AC fundamental wave component in the coefficient_A(1,
0) and the orthogonal transform coefficients of block B ′ and block C
F, which is the AC fundamental wave component in the horizontal direction_B’(1,
0), F_CEvaluate the magnitude relationship of the three values (1, 0)
You. That is, F_AThe value of (1,0) is F_B’(1,0)
Value and F _CJudge whether it exists between the value of (1, 0)
are doing. If it exists, go to S608, if not
Move to S609.

In step S608, the variable Vx is calculated by the following equation (1).
0). Vx = The _{(F A (1,0) -F B} '(1,0)) / (F C (1,0) -F B' (1,0)) ···· (10), in S609 , The variable Vx is set to Vx = 0.

Similarly, in S610, F is a vertical AC fundamental wave component in the orthogonal transform coefficient of block A.
_A (0, 1) and F _B ′, which is the horizontal AC fundamental wave component in the orthogonal transform coefficients of block B ′ and block D
Evaluate three kinds of magnitude relations of (0, 1) and F _D (0, 1). That is, it is determined whether the value of F _A (0,1) exists between the value of F _B ′ (0,1) and the value of F _D (0,1). If it exists, the process moves to S611; otherwise, the process moves to S612.

In S611, the variable Vy is calculated by the following equation (1).
It is calculated in 1). Vy = The _{(F A (0,1) -F B} '(0,1)) / (F D (0,1) -F B' (0,1)) ···· (11), in S612 , The variable Vy is set to Vy = 0.

In step S613, a block determined to have truly moved from block A (referred to as block B ″) based on Vx and Vy calculated by equations (10) and (11).
Is set as follows, and the processing ends.

[0079]

(Equation 8)

That is, the motion vector from block A to block B ′ is

[0081]

(Equation 9)

Thus, the terms c × Vx and d × Vy in equation (12) are vector components having higher resolution than the distance between pixels.

The motion vector calculation unit 104 has been described above. As described above, the motion vector calculation unit 104 of this embodiment is a two-stage process. First, a search is made for a block that is considered to be spatially closest, and then, a small deviation amount from the obtained block is estimated. In the above-described embodiment, the estimation is performed using the orthogonal transform coefficient in both the two stages. However, due to simplification and speeding up of the processing, the first-stage block search is performed without using the orthogonal transform coefficient. Alternatively, a method of estimating by comparing the pixel values within may be used.

Next, the data processing section 106 will be described with reference to FIG. 7, a coordinate management unit 701 includes:
According to the vector calculated by the motion vector calculation unit 104, the position of the block of the frame H which is the target frame corresponds to the position of the frame G which is the reference frame. From the coordinate management unit 701, an address at which the evaluation function of Expression (9) is the smallest is output.

The N × N blocking unit 702 outputs the frame H
Is divided into blocks in units of N × N pixels. This means does not need to be performed inside the data processing unit 106 again as long as it holds the pixel value information of the block (referred to as the block of interest) used inside the motion vector calculation unit 104 in the preceding stage.

Similarly, the N × N blocking unit 703 executes blocking of the frame G in units of N × N pixels based on the address received from the coordinate management unit 701. This means is also a block in which the evaluation function is minimized (referred to as a minimum error block) and a block located around the minimum error block (a peripheral block and a peripheral block) among the blocks created and evaluated in the motion vector calculation unit 104 in the preceding stage. ), The data processing unit 10 is renewed.
6 need not be performed internally.

Now, the block of interest on the frame H is shifted to the block A, the block having the minimum error with respect to the block A on the frame G is shifted to the block B ', and the block B' is shifted by one pixel on the left and right in the horizontal direction. 2
Among the blocks of the kind, the block evaluated as having a small evaluation function is divided into blocks by shifting vertically by one pixel in the vertical direction with reference to the block C and similarly to the block B ′.
A block evaluated as having a small evaluation function among the seed blocks is referred to as a block D.

Further, the x coordinate of the origin of the block C, and
A block whose origin is the y coordinate of the origin of the block D is referred to as a block E. Block E is shifted from block B 'by one pixel both horizontally and vertically.

The average value calculation unit 704 is means for calculating the average value of the pixel values in the block A which is the target block. If the origin coordinate of block A is (a0, b0),
The average value T _A of the block A is calculated as follows.

[0090]

(Equation 10)

The average value separating section 705 is a means for separating the pixels in the block _A by subtracting the calculated average value TA from each pixel. If the value after separation is g _H (x, y), it is calculated by the following equation (15). _{g H (x, y) =} f H (x, y) -T A ···· (15)

On the other hand, the average value calculation unit
Of the blocks B ′, C, D, and E of FIG. 'The origin coordinates (a' block B, b ') When the mean value T _B of each _{block', T C, T D,} T E each of which is calculated as follows.

[0093]

[Equation 11]

However, as described in the description of the flowchart in FIG. 6, c and d indicate the similarity of the block shifted rightward to the block A when the blocks are shifted left and right by one pixel. C = 1 when the evaluation function result shown is evaluated as being small, and c = −1 when the block shifted to the left is evaluated as having a small evaluation function result.
Similarly, in the vertical comparison, d = 1 in the downward direction and d = -1 in the upward direction.

Further, since the blocks B ′, C, D, and E have a large block overlap, only one of the four blocks is averaged, and the remaining three blocks are averaged. The average value may be calculated by adding or subtracting only the non-overlapping pixels of the block from the calculated average value of the block.

Next, in the average value replacing section 707, the following calculation is performed. _{h H (x, y) =} g H (x, y) + (1-Vx ') · (1-Vy') · T B + Vx '· (1-Vy') · T C + (1-Vx ') · Vy' · T _D + Vx '· Vy' · T _E (20)

Here, Vx 'and Vy' indicate distances from the origin (a ', b') of the block B 'to the interpolation point. That is, there are very few cases where the coordinates of Vx and Vy calculated by the above-described equations (10) and (11) completely match the desired interpolation points. Actually, the calculated Vx, V
Based on the value of y, the closest interpolation point V
It will be interpolated on x 'and Vy'.

FIG. 8 shows an example of the positional relationship between Vx, Vy, Vx ', and Vy'. A mark is a sample point of the frame G, and a cross is the formula (10) and formula (11) from the origin coordinates (a ', b').
The points separated by Vx and Vy calculated by and the circles are interpolation points to be truly interpolated in order to increase the resolution. Now, when c = 1 and d = 1, the coordinates of this interpolation point are
(A '+ Vx', b '+ Vy'). This interpolation point becomes the origin of block A and is the arrangement point.

Equation (20) means that the average value of block A is replaced with the average value of blocks B ′, C, D and E. Moreover, the average value to be replaced is a linear operation of the average value of four blocks depending on the interpolation position of block A. In other words, the DC component of block A is changed so as to conform to blocks B ′, C, D, and E on the reference frame, and only the AC component of block A is used.

Although the data processing section 106 has been described above, the present embodiment is not limited to the above-described example. Since the block B ', block C, block D, and block E have a large overlap, the calculated average values may not have much difference. In that case, a simple method of adding only the average value T _B ′ of the block B ′ to g _H (x, y) is also possible.

FIG. 9 shows the operation procedure of the repetition processing particularly when three or more frames are used, including the processing up to the motion vector calculation and arrangement centering on the frame control unit 106 described above. It is a flowchart shown.
First, in step S901, the edge strength is evaluated for each of the (n + 1) frames from the stored mth frame to the (m + n) th frame. Then, in S902, they are compared with each other.

Subsequently, in step S903, the (m + p) th frame having the maximum edge strength is set as the frame G. This is the reference frame. Next, in step S904, the variables s and q are initialized to zero. Next, in S905,
It is determined whether the variable s is equal to p. This is to determine whether the frame currently being processed is a reference frame or not.

If the frame s to be processed is not the reference frame, it is determined in step S906 whether or not q is equal to zero. This is to determine whether or not the number of repetitions currently being processed is the first time. If q
Is equal to 0, the frame G is arranged in S907, and the variable q is counted up in S908. if,
If NO is determined in S906, the process is determined to be the second or later, and since the frame G, which is the reference frame, has already been arranged, S907 and S908 jump.

Subsequently, in step S909, frames G and (m
+ S) A motion vector is calculated between the frame (the frame H). Next, after processing the data of the frame H in S910, the arrangement is performed in S911. S91
After counting up the variable s at 2, in S913,
It is determined whether or not the number of repetitions is n. If no, it is determined that a frame that has not been processed is stored, and the process returns to S905, and the same processing steps are repeated for other frames. When the arrangement is completed for all the stored frames, the frames are combined into one piece of image information, and the processing is completed.

The series of processing according to the present embodiment has been described above. The feature of the present invention resides in the selection unit 102. Therefore, the motion vector calculation unit 104, the data processing unit 106,
The contents of the arrangement processing unit 107 and the like are not limited. Naturally, the motion vector calculation can be performed by a method that does not use the orthogonal transformation, and a configuration in which the pixel values of each target frame are simply arranged without processing the data of the target frame is also conceivable.

Further, the evaluation function of the edge strength in the equation (7) is not limited to this. The following equation (21) is also conceivable as a modified example of equation (7).

[0107]

(Equation 12)

In this case, it means that the number of pixels whose value after the edge extraction filter is equal to or larger than a certain threshold value is counted. Equation (21) can sufficiently grasp the edge strength of the entire image. The edge extraction filter is also shown in FIG.
However, the present invention is not limited to this, and a filter having higher noise resistance may be used.

In addition, a method that does not use an edge extraction filter for the evaluation of the edge strength, for example, a method that makes a determination based on a transform coefficient of a high-frequency component of orthogonal transform may be considered. In that case, which frame has the higher power in the high frequency range is evaluated, and the frame determined to have the highest power is set as the reference frame.

Further, in the present embodiment, the edge information is used for the feature amount of the image. However, the present invention is not limited to this, and the evaluation may be performed using the feature amount of another image.

FIG. 10 is a flowchart showing an operation procedure according to the second embodiment of the present invention. This embodiment is different from the first embodiment only in the selection method by the selection unit 102 in FIG. 1, and the other units are the same. FIG.
Is an example of creating one high-resolution still image based on (n + 1) image information from the m-th frame to the (m + n) -th frame.

S1001 shows a division step, in which the value of n is set to 2
Substitute the integer part obtained by division with p as p. In actual processing, it can be realized by a bit shift. Then, S1
In 002, the (m + p) th frame is set as a frame G which is a reference frame. Next, in S1003, the variables s and q are initialized to 0. Next, in S1004,
It is determined whether the variable s is equal to p. This is to determine whether the frame currently being processed is a reference frame or not.

If the frame S to be processed is not the reference frame, it is determined in step S1005 whether q is 0 or not. This is to determine whether the number of times of processing is the first time. If q is 0, S100
In 6, the frame G is arranged, and in S1007, the variable q is counted up. If it is determined to be NO in S1005, it is determined that the process is the second or later, and the frame G that is the reference frame has already been arranged.
S1007 is jumped.

Subsequently, in step S1008, between the frame G and the (m + s) th frame (referred to as frame H),
After calculating a motion vector and processing the data of the frame H in S1009, it is arranged in S1010. S101
In step 1, after counting up the variable s, it is determined whether or not the number of repetitions is n. If a frame that has not been processed is stored, the process proceeds to S1004.
And the same processing steps are repeated for other frames.

When the arrangement is completed for all the stored frames, the images are combined as a single image, and the processing is completed. As described above, the present embodiment is characterized in that the selection of the reference frame is determined based on the input frame order.

FIG. 11 is a diagram showing how a reference frame is determined when five frames are stored. The frame indicated by oblique lines is the reference frame. If the number of stored images is 5 frames, n = 4, so by dividing by 2, p
= 2, and the intermediate frame of the (m + 2) th frame is set as the reference frame. This reference frame is compared with each of the other four frames and processed.

If the number of stored frames is even,
Since the result of dividing n by 2 is a non-integer, it cannot be correctly intermediate, but frames before and after the intermediate may be set as reference frames (however, in the flowchart of FIG. ). That is, in the embodiment of the flowchart of FIG. 9 described above, the method of setting the selection of the reference frame based on the “image feature” is used. The edge strength was evaluated as an evaluation function in which the feature amount of the image can be most remarkably expressed. Certainly, if the image is selected based on the characteristics of the image, there is a possibility that an image optimal in image quality can be set as the reference frame.

However, since continuous images are handled, it is not always optimal in terms of time. Therefore, in the embodiment of the flowchart of FIG. 10, the selection is made with emphasis on “temporal image correlation”. The use of an intermediate image on the time axis can be assumed to be a central image having the highest image correlation when considering the continuity of the image when compared with each frame in the stored image. That is, the difference between each frame and the reference image can be reduced because the time lag is minimal.

The embodiment of the present invention has been described above.
A compromise between the flowcharts of FIGS. 9 and 10 is also conceivable. That is, it is also possible to create a new evaluation function in consideration of the image feature amount and the position on the time axis, and determine the reference frame. In this case, even if the optimal frame on the time axis is unclear in image quality, it is possible to select an overall optimal image.

Next, a storage medium according to another embodiment of the present invention will be described. The object of the present invention can be achieved by a hardware configuration, and can also be achieved by a computer system including a CPU and a memory. When configuring with a computer system,
The memory constitutes a storage medium according to the present invention. That is,
A storage medium storing software program codes for executing the operations described in each of the above-described embodiments is used in a system or an apparatus, and a CPU of the system or apparatus reads out the program code stored in the storage medium. , The object of the present invention can be achieved.

The storage media include ROM, R
Semiconductor memory such as AM, optical disk, magneto-optical disk,
A magnetic medium or the like may be used, and these may be configured and used in a CD-ROM, a floppy disk, a magnetic medium, a magnetic card, a nonvolatile memory card, or the like.

Therefore, the storage medium may be used in a system or apparatus other than the system or apparatus shown in FIG. 1 or the like, and the system or computer may read out and execute the program code stored in the storage medium. In addition, the same functions as those of the above embodiments can be realized, the same effects can be obtained, and the object of the present invention can be achieved.

An OS running on a computer
When performing part or all of the processing, or after the program code read from the storage medium is written to a memory provided in an extended function board or an extended function unit connected to the computer, Even when the CPU or the like provided in the above-mentioned extended function board or extended function unit performs a part or all of the processing based on the instruction of the program code, the same functions as those of the above embodiments can be realized and the same functions can be realized. Effect can be obtained,
The object of the present invention can be achieved.

[0124]

As described above, according to the present invention,
By setting a single reference frame from among the stored multiple frames as a reference for comparison with each frame based on the image feature amount and temporal correlation, accumulation of errors during vector calculation occurs. In addition, even if there is an unclear frame, good synthesis can be performed without any problem.

Further, according to the present invention, image information which has been significantly improved in image quality as compared with the conventionally proposed interpolation and interpolation techniques for creating a high-resolution still image from a single low-resolution still image. Can be created.

Further, according to the present invention, since one piece of high-resolution still image information can be easily created from low-resolution still image information captured by a video camera, communication between models having different input / output resolutions, It is possible to provide a video camera, a printer, and the like that output a high-quality image by enlarging and changing the magnification.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an image processing apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a selection unit of FIG. 1;

FIG. 3 is a configuration diagram illustrating an example of an edge extraction filter.

FIG. 4 is a block diagram illustrating a motion vector calculator of FIG. 1;

FIG. 5 is a configuration diagram illustrating a motion vector.

FIG. 6 is a flowchart illustrating an operation procedure of a transform coefficient evaluation unit in FIG. 4;

FIG. 7 is a block diagram showing a data processing unit of FIG. 1;

FIG. 8 is a configuration diagram illustrating an arrangement position in a block.

FIG. 9 is a flowchart illustrating a series of processes including a selection unit according to the first embodiment of the present invention.

FIG. 10 is a flowchart illustrating a series of processes including a selection unit according to the second embodiment of the present invention.

FIG. 11 is a configuration diagram illustrating a reference frame.

FIG. 12 is a configuration diagram illustrating a conventional nearest neighbor interpolation method.

FIG. 13 is a configuration diagram illustrating a conventional bilinear interpolation method.

FIG. 14 is a configuration diagram illustrating a conventional cubic convolution interpolation method.

FIG. 15 is a configuration diagram illustrating a comparative frame for calculating a conventional motion vector.

[Explanation of symbols]

 102 selection unit 103 frame control unit 104 motion vector calculation unit 105, 107 placement processing unit 106 data processing unit 108 synthesis unit 109 interpolation unit 201-204 edge extraction unit 205-208 edge strength evaluation unit 209 maximum edge strength determination unit

Claims (8)

[Claims] 1. A selecting means for selecting first image information from (n + 1) (m and n are arbitrary natural numbers) consecutive image information from an m-th frame to an (m + n) -th frame. Storage means for arranging and storing the (n + 1) pieces of image information; and calculating means for calculating a motion vector for the first image information for each of the n pieces of image information other than the first image information. A arranging means for arranging the (n + 1) image information based on the calculation result; and a synthesizing means for synthesizing the (n + 1) images after the arranging to form one image. An image processing apparatus characterized by the above-mentioned. 2. The image processing apparatus according to claim 1, wherein the selecting unit includes an evaluating unit that evaluates a feature amount of each of the (n + 1) images, and selects one of the (n + 1) images based on the evaluation result. The image processing apparatus according to claim 1, wherein: 3. The image processing apparatus according to claim 2, wherein the feature amount is edge information of an image. 4. The apparatus according to claim 2, wherein said evaluation means has a filter means for extracting an edge of the image, and performs the evaluation based on the edge extraction information after the filtering.
The image processing apparatus according to any one of the preceding claims. 5. The image processing apparatus according to claim 1, wherein the selection unit selects the image based on the input order of the (n + 1) images. 6. The image processing apparatus according to claim 5, wherein said selecting means selects a frame corresponding to an intermediate time in the input order. 7. A selection procedure for selecting first image information from (n + 1) consecutive (m and n are arbitrary natural numbers) image information from the mth frame to the (m + n) th frame, An operation procedure for calculating a motion vector for the first image information for each of the n pieces of image information other than the first image information, and an arrangement procedure for arranging the (n + 1) image information based on the operation result And a combining procedure for combining the (n + 1) images after the arrangement to form one image. 8. A selection process for selecting first image information from (n + 1) (m and n are arbitrary natural numbers) continuous image information from the m-th frame to the (m + n) -th frame; An operation process of calculating a motion vector for the first image information for each of the n pieces of image information other than the first image information, and an arrangement process of arranging the (n + 1) image information based on the operation result And a computer-readable storage medium storing a program for performing a combining process of combining the (n + 1) images after the arrangement to form one image.