JP2012010297A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

【Task】
In the super-resolution technique using a plurality of frames when a video is switched or changed at high speed, there are cases where a lot of errors are included and the accuracy of the video is lacking.
[Solution]
The image processing apparatus includes: a first image generating unit that performs intra-frame super-resolution processing on an input frame that is continuously input in time series to generate a first high-resolution frame image; and a plurality of the input frames. Second image generation means for generating a second high-resolution frame image by using; detection means for detecting a change point between frames of the input frames that are continuously input; and If detected, the first high-resolution frame image is output, and if the detection means does not detect a change point, the output means for outputting the second high-resolution frame image;
It is characterized by providing.
[Selection] Figure 1

Description

  Embodiments described herein relate generally to an image processing apparatus, an image processing method, and an image processing program.

  Surveillance systems using an imaging device have been developed for monitoring buildings, station premises, airports, streets, and the like.

  When using the zoom and pan functions of the imaging device or when switching the imaging device used by the switcher during this monitoring system operation, scene switching is included when displaying pre-recorded video For example, if a super-resolution technique using a plurality of frames is to be used at the time of switching or changing the displayed video, super-resolution processing is performed based on discontinuous frames, and errors are likely to be included.

JP 2009-217658 A

  In view of the above circumstances, there are provided an image processing apparatus, an image processing method, and an image processing program that realize high resolution of an output frame with few errors while performing super-resolution processing on all output images. In the embodiment, when a video change or switching is detected, a plurality of frames used for the super-resolution processing are selected so that both frames before and after the video change and switching are not included. In this way, an image processing apparatus, an image processing method, and an image processing program that prevent pixel deterioration due to video change or switching are provided.

In order to achieve the above object, the image processing apparatus includes:
First image generation means for performing intra-frame super-resolution processing on an input frame that is continuously input in time series and generating a first high-resolution frame image;
Second image generation means for generating a second high-resolution frame image using a plurality of the input frames;
Detecting means for detecting a change point between frames of the input frames that are continuously input;
Output means for outputting the first high-resolution frame image when the detection means detects a change point, and outputting the second high-resolution frame image when the detection means does not detect the change point; ,
It is characterized by providing.

In addition, this image processing method
Intra-frame super-resolution processing is performed on input frames that are continuously input in time series to generate a first high-resolution frame image, and a plurality of the input frames are used to generate a second high-resolution frame image.
Detecting a change point between frames of the input frames that are continuously input;
The second high-resolution frame image is output when the detection unit detects a change point, and the first high-resolution frame image is output when the detection unit does not detect the change point. And

In addition, this image processing program
A first generation process for generating a first high-resolution frame image by performing intra-frame super-resolution processing on an input frame that is continuously input in time series;
A second generation process for generating a second high-resolution frame image using a plurality of the input frames;
A detection process for detecting a change point between frames of the input frames that are continuously input;
An output process for outputting the second high-resolution frame image when the detection unit detects a change point, and outputting the first high-resolution frame image when the detection unit does not detect the change point; ,
Is operated on an image processing apparatus.

1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment. The block diagram which shows the modification of the image processing apparatus which concerns on 1st Embodiment. The block diagram which shows the modification of the image processing apparatus which concerns on 1st Embodiment. The flowchart which shows the operation | movement which concerns on 1st Embodiment. The figure for demonstrating the operation | movement which concerns on 1st Embodiment. The figure for demonstrating the operation | movement which concerns on 1st Embodiment. The flowchart which shows the operation | movement which concerns on 2nd Embodiment. The figure for demonstrating the operation | movement which concerns on 2nd Embodiment. The flowchart which shows the operation | movement which concerns on 3rd Embodiment. The figure for demonstrating the operation | movement which concerns on 3rd Embodiment. The block diagram which shows the structure of the image processing apparatus which concerns on 4th Embodiment. The flowchart which shows the operation | movement which concerns on 4th Embodiment. The figure for demonstrating the operation | movement which concerns on 4th Embodiment. The figure for demonstrating the operation | movement which concerns on 4th Embodiment. The figure for demonstrating the operation | movement which concerns on 4th Embodiment. The flowchart which shows the operation | movement which concerns on 5th Embodiment. The figure for demonstrating the operation | movement which concerns on 5th Embodiment. The figure for demonstrating the operation | movement which concerns on 5th Embodiment. FIG. 10 is a block diagram illustrating a configuration of an image processing apparatus according to a sixth embodiment. The flowchart which shows the operation | movement which concerns on 6th Embodiment. The figure for demonstrating the operation | movement which concerns on 6th Embodiment. The figure for demonstrating the operation | movement which concerns on 6th Embodiment. The figure for demonstrating the operation | movement which concerns on 6th Embodiment. The flowchart which shows the operation | movement which concerns on 7th Embodiment. The figure for demonstrating the operation | movement which concerns on 7th Embodiment. The block diagram which shows the structure of the image processing apparatus which concerns on 8th Embodiment. The flowchart which shows the operation | movement which concerns on 8th Embodiment. The figure for demonstrating the operation | movement which concerns on 8th Embodiment. The figure for demonstrating the modification of a super-resolution process.

  Each embodiment will be described below with reference to FIGS.

(First embodiment)
A first embodiment will be described with reference to FIGS. 1 to 6.

  As shown in FIG. 1, the image processing apparatus according to the present embodiment includes an imaging apparatus 10, an image processing apparatus 20, and an image receiving apparatus 30.

  The imaging device 10 is a device that images a subject. It refers to a digital camera, an analog camera, or the like, for example, a PTZ camera (integrated pan / tilt zoom lens), and can perform high-speed pan / zoom. Further, as shown in FIG. 2, the imaging device 10 may include a plurality of imaging devices 10 a to 10 n. When there are a plurality of imaging devices, the switching of the camera and the like is further performed by a remote operation or the like. An input frame is created from the image captured by the imaging device 10 and the created input frame is sent to the image processing device 20 in time series. As described above, in this embodiment, the input frame is described as being input to the image processing apparatus 20 in the order of f1, f2, f3, f4, f5, f6, and f7 with the passage of time.

  Further, as shown in FIG. 3, a video storage device 12 may be used instead of the imaging device 10. The video storage device 12 sequentially sends the stored input frames to the image processing device 20. When the video storage device 12 is used, information such as the number of change points can be held in advance. Therefore, it is possible to perform delay processing in consideration of the presence of a change point in advance and easily adjust the output interval of output frames, thereby further reducing image disturbance.

  The imaging device control unit 11 operates the imaging device 10. For example, when the imaging device is a PTZ camera, panning / zooming or the like is instructed to the imaging device 10 by manual or automatic remote operation. In the case where there are a plurality of imaging devices, it also serves as a switcher for changing the camera that sends the input frames of the imaging devices 10a... 10n to the image processing device 20.

  The image processing device 20 is a device that includes a multi-frame super-resolution processing unit 21, an intra-frame super-resolution processing unit 22, a change point detection unit 23, a storage unit 24, and an output frame selection unit 25. The image processing device 20 receives an input frame sent from the imaging device 10 or the video storage device 12 and performs super-resolution processing. The output frame subjected to the super-resolution processing is sent to the image receiving device 30. In addition, the image processing apparatus 20 performs arithmetic processing and storage operations necessary for image processing and the like.

  Super-resolution processing is a type of high-resolution conversion processing and refers to processing for converting a low-resolution frame image into a high-resolution frame image.

  The multi-frame super-resolution processing unit 21 performs super-resolution processing using a plurality of input frames (two or three or more frames). In the present embodiment, super-resolution processing is performed using a plurality of frame images sent from the imaging device 10 or the like, and a frame image with a higher resolution is generated and output. Note that the position of the reference frame used for the multi-frame super-resolution processing may be set in advance, or may be changed or instructed by the function of the image processing apparatus 20.

  A case where super-resolution processing is performed using three or more frames will be described. For example, assume that input frames f1 to f3 are input. At this time, super resolution processing is first performed from f1 and f2, and a frame f1-2 at an intermediate stage is created. Super-resolution processing is performed from f1-2 and f3. These intermediate steps f1-2 are also sent to the storage unit 24 for storage. In addition, after creating f1-3, f2-3, etc., super resolution processing combined with f2, f1 may be used.

  This multi-frame super-resolution processing is performed by two processes. The first process is an alignment process. A case where super-resolution processing is performed using two frame images will be described. One of a plurality of frames used for super-resolution processing is used as a reference frame, and a motion vector with another input frame is obtained. When obtaining a motion vector, subpixels (decimal point pixels) are used. First, a motion vector with integer pixel precision is obtained by a region-based template matching method (search range is arbitrary). The estimated value of the motion vector with sub-pixel accuracy is calculated and obtained for the result by parabolic fitting or equiangular straight line fitting. Another process is a reconstruction process. A high-resolution image is reconstructed by reconstructing (e.g., superimposing) the control block of the reference frame with the corresponding block in another input frame. In the present embodiment, such a process will be described, but a process of converting from a low resolution frame to a high resolution frame using a plurality of other frames may be used.

  The intra-frame super-resolution processing unit 22 performs super-resolution processing using one frame image. In the present embodiment, an output frame that has been increased in resolution by super-resolution processing using association in one frame sent from the imaging device 10 or the like is output. When performing in parallel with multi-frame super-resolution processing, intra-frame super-resolution processing is performed on the same frame as the reference frame used for multi-frame super-resolution processing.

  This intra-frame super-resolution processing uses, for example, the nature (self-congruity) of a natural image that edge portions such as the contour of a subject often have patterns with similar brightness changes along the edges. A line whose luminance pattern is similar to the line of interest (vertical or horizontal pixel column) in the input frame is detected from other adjacent lines. Next, pixels in other lines are copied to corresponding positions in the line of interest. A high resolution image is generated based on the copied pixels.

  Thus, based on the information in one frame image, the number of pixels (pixels) of the output frame is increased to create a high-resolution frame image.

The change point detection unit 23 detects a change or switching (change point) of the video. Specifically, a change point (change point) of a certain level or more between two input frames that are continuously input is detected. For example, it is analyzed as an inter-frame difference, and when a difference greater than a predetermined value (threshold value) is detected, it is detected as a change point if there is a change. In addition, for example, a method for analyzing and discriminating a difference between two frames by a surface luminance method, a TM method, a frequency distribution method, a CTM method, a chromatic frequency distribution method, a χ ² test method, a divided χ ² test method, Changes in image features such as vectors (motion amount) are detected. If these changes are above a certain level, it is detected that there has been a change.

  The storage unit 24 is a storage medium that stores output frame images. In the present embodiment, a high-resolution frame image (output frame) that has been super-resolution processed by the multi-frame super-resolution processing unit 21 and the intra-frame resolution processing unit 22 and sent is stored. In response to a read command from the output frame selection unit 25, an output frame is output to the image receiving device 30. As storage media for data, various storage media such as generally used disk devices, optical disk devices, and semiconductor memories can be used. These storage media may be used so as to be temporarily stored. In this embodiment, the frames subjected to the super-resolution processing are stored, but may be used when temporarily storing other input frames.

  The output frame selection unit 25 selects a frame to be output from the high resolution frame image stored in the storage unit 24. In this embodiment, when the change point detection unit 23 detects a change point, the storage unit 24 instructs the output of an output frame that has been subjected to the intra-frame super-resolution processing corresponding to the reference frame. When the change point detection unit 23 does not detect the change point, the storage unit 24 instructs the output of the output frame that has been subjected to the multi-frame super-resolution processing corresponding to the reference frame.

  The image receiving device 30 receives and displays the image that has been super-resolution processed by the image processing device 20. The image receiving device may be any device that can display an image. For example, a liquid crystal display such as a mobile phone or a personal computer screen, a television screen, a plasma display, an organic EL display, or other various image / video display devices may be used. Can do.

Next, the operation of the first embodiment will be described with reference to FIGS. 4, 5, and 6.

  FIG. 4 is a diagram illustrating the operation of the first embodiment.

  The imaging device 10 captures a low-resolution image of the subject, frames the low-resolution image, and sends it to the image processing device 20 as an input frame.

  The sent input frame is input to the image processing apparatus 20 (step S101).

  The image processing apparatus 20 causes the multi-frame super-resolution processing unit 21 and the intra-frame super-resolution processing unit 22 to process the received low-resolution image. Further, the change point detection unit 23 is caused to detect change points between input frames. At this time, the input frame may be stored in the image processing apparatus and used for multiple readings.

  The multi-frame super-resolution processing unit 21 performs super-resolution processing using a plurality of input frames (two or three or more frames). For example, multi-frame super-resolution processing using two input frames will be described. In the present embodiment, two input frames are described for the sake of simplicity, but super-resolution processing is similarly performed for three or more frames.

  As shown in FIG. 5, it is assumed that the reference frame is set to use the later frame of the two input frames sent. Input frames f1 to f7 are sequentially sent from the imaging device 10 and the video storage device 12. Using this input frame f1 as a reference frame, multi-frame super-resolution processing is performed using f1 and a frame immediately before f1 (not shown), and the output frame F1 is temporarily stored or stored in the image storage unit 24. Next, the reference frame is set to f2, and multi-frame super-resolution processing is performed using f1 and f2, and the output frame F2 is sent to the storage unit 24 and stored therein. This operation is repeated.

  In addition, as shown in FIG. 6, a case will be described in which the previous frame is set to be used as a reference frame among the two frames used for the super-resolution processing. Input frames f1 to f7 are continuously sent from the imaging device 10 and the video storage device 12, and multi-frame super-resolution processing is performed using f1 and f2 with f1 as a reference frame. The output frame F1 that has been subjected to the super-resolution processing is sent to and stored in the storage unit 24 (steps S102 and S103).

  The intra-frame super-resolution processing unit 22 performs intra-frame super-resolution processing in parallel with the super-resolution processing of the multi-frame super-resolution processing unit 21. For example, as shown in FIGS. 5 and 6, when input frames f1 to f7 are sent from the imaging device 10 or the like, intra-frame super-resolution processing is performed on each frame of f1 to f7, and the storage unit 24 stores the frames. Temporarily stored or stored (step S104, step S105).

  The change point detection unit 23 detects a change (change point) of two or more input frames that are continuously input. For example, it is analyzed as an inter-frame difference, and when a difference greater than a predetermined value (threshold value) is detected, it is detected as a change point if there is a change. In FIG. 5, f1 and f2, f2 and f3, f3 and f4, f4 and f5, f5 and f6, and inter-frame differences between f6 and f7, etc., are detected by detecting the change point. (Step S106).

  If the change point is not detected by the change point detection unit 23 (NO in step S106), the output frame selection unit 25 sequentially reads out the multi-frame super resolution processed images stored in the storage unit 24 and outputs them to the image receiving device 30. To do. In FIG. 5, since no change point is detected except for between the input frames f4 and f5, output frames subjected to the multi-frame super-resolution processing are output for the output frames F1 to F4, F6, and F7. In FIG. 6, the output frames F1 to F3 and F5 to F7 are subjected to a multi-frame super-resolution process and output. (Step S107).

  On the other hand, when the change point is detected by the change point detection unit 23 (YES in step S106), the output frame selection unit 25 receives the intra-frame super-resolution processed image instead of outputting the multi-frame super-resolution processed image. Output to 30. In FIG. 5, since a change point is detected between the input frames f4 and f5, an output frame F5 subjected to intra-frame super-resolution processing is output instead of the multi-frame super-resolution processing image using the input frame f5 as a reference. Is done. In FIG. 6, a change point is detected between the input frames f4 and f5, and a multi-frame super-resolution processed image with the input frame f4 as a reference is not output. Instead, the output frame F4 subjected to the intra-frame super-resolution processing is output and transmitted to the image receiving device 30 (step S108).

As described above, when there is a change point, by outputting the intra-frame super-resolution processed image instead of the multi-frame super-resolution processed image, the output frames obtained by performing the super-resolution processing on all the input frames are output. Can be output. Since super-resolution processing is performed on all input frames, high-precision high-resolution processing or video can be provided. In addition, instead of performing super-resolution processing after detecting whether there is a change point in advance, it is possible to quickly provide an output frame that has undergone super-resolution processing because it performs processing in parallel. In addition, high-quality images can be provided.

(Second Embodiment)
The aspect of 2nd Embodiment is demonstrated referring FIG. 7, FIG.

  The difference from the first embodiment is mainly in the multi-frame super-resolution processing unit 21 and the output frame selection unit 25. In addition, the frame image data and the like stored in the storage unit 24 differ according to these differences.

  The multi-frame super-resolution processing unit 21 according to the second embodiment performs different super-resolution processing depending on the set position of the reference frame. In the present embodiment, a case will be described in which the last frame of three frames is used as a reference frame as shown in FIG.

  As shown in FIG. 8, input frames f1 to f3 are input. In this case, although super resolution processing is performed using f1 to f3 with f3 as a reference frame, at least super resolution processing using f2 and f3 is performed, and the created f2-3 is stored in the storage unit 24. . In the present embodiment, it is preferable to perform three super-resolution processes using f2-3 and f1. However, after creating f1-2 separately, super-resolution processing may be performed using f1-2 and f3. Next, the same processing is performed for f2 to f4 and thereafter, and the same processing is performed thereafter. Although super-resolution processing using three frames has been described, the same principle can be used even with four or more frames.

  The output frame selection unit 25 of the second embodiment further includes a determination function as to whether or not a change point has been detected in the detection process of the previous change point detection unit 23. If the change point detection means 23 does not detect a change point and it is determined that a change point has been detected in the previous detection process in the determination by the determination function of the output frame selection unit 25, a predetermined number of frames are detected. Instead of the used super-resolution processed image, a multi-frame super-resolution image using fewer frames than the specified number stored in the storage unit 24 is selected and output. If the change point detection means 23 does not detect a change point and it is determined that a change point is not detected in the previous detection process in the determination by the determination function of the output frame selection unit 25, the specified number of frames Outputs super-resolution processed images using. If no change point is detected, an in-frame resolution processed image is output. Other functions are the same as those in the first embodiment.

  Specifically, the case where the prescribed number of frames used for a plurality of frames is three will be described with reference to FIG. As shown in FIG. 8, input frames f1 to f7 are input. Here, there is a change point between f4 and f5. Therefore, when f3, f4, and f5 are input, the output frame F5 corresponding to f5 outputs an output frame that has been subjected to intra-frame super-resolution processing. Next, when f4, f5, and f6 are input, a super-resolution image using two frames f5 and f6, which is one less than the specified number, is selected and output with f6 as a reference.

The operation of the second embodiment will be described with reference to FIGS.

  First, a plurality of input frames are continuously input to the image processing apparatus 20 (step S201).

  The image processing apparatus 20 causes the multi-frame super-resolution processing unit 21 and the intra-frame super-resolution processing unit 22 to process the received low-resolution image. Further, the change point detection unit 23 is caused to detect change points between input frames.

  The multi-frame super-resolution processing unit 21 performs different super-resolution processing depending on the set position of the reference frame. For example, as shown in FIG. 8, a case will be described in which the last frame is used as a reference frame among three plural frames.

  As shown in FIG. 8, input frames f1 to f3 are input. In this case, super resolution processing is performed using f1 to f3 using f3 as a reference frame. At this time, super resolution processing using f2 and f3 is performed, and the created f2-3 is stored in the storage unit 24. Remember. F3 (f1-2-3) created by further performing three super-resolution processing using f2-3 and f1 is stored in the storage unit 24. Next, after two input frame super-resolution processes are performed in the same manner for f2 to f4 and thereafter, a super-resolution processed image using three input frames using the two super-resolution processed images. Is created. With this operation, a super-resolution processed image using the reference frame and the input frame immediately before the reference frame, and a super-resolution processed image using the reference frame and the input frame before and two frames before the reference frame are stored. (Step S202, step S203).

  The intra-frame super-resolution processing unit 22 performs intra-frame super-resolution processing in parallel with the super-resolution processing of the multi-frame super-resolution processing unit 21. For example, as shown in FIG. 7, when input frames f1 to f7 are sent from the imaging device 10 or the like, intra-frame super-resolution processing is performed on each frame of f1 to f7 and is temporarily stored in the storage unit 24. Stored (step S204, step S205).

  The change point detection unit 23 detects a change (change point) of two or more input frames that are successively input. For example, it is analyzed as an inter-frame difference, and when a difference greater than a predetermined value (threshold value) is detected, it is detected as a change point if there is a change.

  In FIG. 7, f1 and f2, f2 and f3, f3 and f4, f4 and f5, f5 and f6, f6 and f7, and the difference between frames of the input frames inputted as f1 to f7 are compared to detect a change point. (Step S206).

  When the change point is detected by the change point detection unit 23 (step S206—YES), the intra-frame super-resolution processed image is output. In FIG. 7 of the present embodiment, the intra-frame super-resolution processed image using the input frame f5 is output as the output frame F5 (step S210).

  When the change point is not detected by the change point detection unit 23 (NO in step S206), the output frame selection unit 25 further determines whether or not the change point is performed in the previous detection process. If a change point is detected and a change point is detected in the previous detection process in the determination by the determination function (YES in step S207), the number of frames is smaller than the specified number stored in the storage unit 24. A multi-frame super-resolution image using is output. In the present embodiment, no change point is detected between f5 and f6, but a change point is detected between f4 and f5 performed in the previous detection process. Therefore, the output frame F6 uses f6 as a reference and f5. And a super-resolution processed image using the two images f6 (step S208).

  When the change points are not continuously detected by the change point detection unit 23 and the output frame selection unit 25, a prescribed number, that is, three frames subjected to super-resolution processing are output. In this embodiment, it is F1-F4 and F7 (step S209).

As described above, when there is a change point, by outputting the intra-frame super-resolution processed image instead of the multi-frame super-resolution processed image, the output frames obtained by performing the super-resolution processing on all the input frames are output. Can be output. Since super-resolution processing is performed on all input frames, high-resolution processing with little image degradation can be realized.

  In addition, when super-resolution processing is performed using three or more input frames, even if a change point is detected, the next input after detecting the change point (that is, two images counted from the change point) is input. After a frame is input, a super-resolution processed image using two or more frames can be output, and a larger number of multi-frame super-resolution processed images can be used as output frames. This makes it possible to output a frame with higher resolution.

(Third embodiment)
The aspect of 3rd Embodiment is demonstrated referring FIG. 9, FIG.

  The difference from the second embodiment is mainly in the output frame selection unit 25. In addition, the frame image data and the like stored in the storage unit 24 differ according to these differences.

  The multi-frame super-resolution processing unit 21 of the third embodiment performs different super-resolution processing depending on the set position of the reference frame. In the present embodiment, as shown in FIG. 10, a case where the first frame of three frames is used as a reference frame will be described.

  As shown in FIG. 10, input frames f1 to f3 are input. In this case, super resolution processing is performed using f1 to f3 using f1 as a reference frame. At least super resolution processing using f1 and f2 is performed, and the created f1-2 is stored in the storage unit 24. . In the present embodiment, it is preferable to perform three super-resolution processes using f1-2 and f3. Next, the same processing is performed for f2 to f4 and thereafter, and the same processing is performed thereafter. Although super-resolution processing using three frames has been described, the same principle can be used even with four or more frames.

  Further, the output frame selection unit 25 of the present embodiment has a determination function as to whether or not a change point has been detected in the detection process of the previous change point detection means 23 as in the second embodiment. When the change point detection unit 23 detects the change point and the determination by the determination function of the output frame selection unit 25 determines that the change point has been detected in the previous detection process, the change point detection unit 23 stores the change point in the storage unit 24. The processed intra-frame super-resolution image is output. When the change point detection unit 23 detects the change point and the determination by the determination function of the output frame selection unit 25 determines that the change point is not detected in the previous detection process, the change point detection unit 23 stores the change point in the storage unit 24. A multi-frame super-resolution image using frames smaller than the specified number is selected and output.

  Specifically, the case where the prescribed number of frames used for a plurality of frames is three will be described with reference to FIG. As shown in FIG. 10, input frames f1 to f7 are input. Here, there is a change point between f4 and f5. Therefore, when f3, f4, and f5 are input, the output frame F3 corresponding to f3 selects and outputs a super-resolution processed image that uses two frames, f3 and f4, which are one fewer than the specified number with f3 as a reference. Let Next, when f4, f5, and f6 are input, an output frame that has been subjected to intra-frame super-resolution processing with f4 as a reference is output.

The operation of the third embodiment will be described with reference to FIGS.

  First, a plurality of input frames are continuously input to the image processing apparatus 20 (step S301).

  The image processing apparatus 20 causes the multi-frame super-resolution processing unit 21 and the intra-frame super-resolution processing unit 22 to process the received low-resolution image. Further, the change point detection unit 23 is caused to detect change points between input frames.

  The multi-frame super-resolution processing unit 21 performs different super-resolution processing depending on the set position of the reference frame. In the present embodiment, a case will be described in which the last frame of three frames is used as a reference frame as shown in FIG.

  As shown in FIG. 10, input frames f1 to f3 are input. In this case, super resolution processing is performed using f1 to f3 using f1 as a reference frame. At this time, super resolution processing using f1 and f2 is performed, and the created f1-2 is stored in the storage unit 24. Remember. F3 (f1-2-3) created by further performing three super-resolution processes using f1-2 and f3 is stored in the storage unit 24. Next, after two input frame super-resolution processes are similarly performed for f2 to f4 and thereafter, a super-resolution process image using three input frames is obtained using the two super-resolution process images. Creation is done. With this operation, a super-resolution processed image using the reference frame and the input frame after the reference frame, and a super-resolution processed image using the reference frame and the input frame after the reference frame are stored. (Step S302, step S303).

  The intra-frame super-resolution processing unit 22 performs intra-frame super-resolution processing in parallel with the super-resolution processing of the multi-frame super-resolution processing unit 21. For example, as shown in FIG. 10, when input frames f1 to f7 are sent from the imaging device 10 or the like, intra-frame super-resolution processing is performed on each of the frames f1 to f7 and is temporarily stored in the storage unit 24. Stored (step S304, step S305).

  The change point detection unit 23 detects a change (change point) of two or more input frames that are successively input. For example, it is analyzed as an inter-frame difference, and when a difference greater than a predetermined value (threshold value) is detected, it is detected as a change point if there is a change.

  In FIG. 10, f1 and f2, f2 and f3, f3 and f4, f4 and f5, f5 and f6, f6 and f7, and the difference between frames of the input frames input as f1 to f7 are compared to detect the change point. (Step S306).

  If no change point is detected by the change point detection unit 23 (YES in step S306), a prescribed number, that is, three frames subjected to super-resolution processing are output. In FIG. 7 of the present embodiment, the intra-frame super-resolution processed image using the input frame f5 is output as the output frame F5 (step S308).

  When the change point is detected by the change point detection unit 23 (YES in step S306), the output frame selection unit 25 further determines whether or not the change point is performed in the previous detection process. When the change point detection unit 23 detects the change point, and it is determined that the change point is detected in the previous detection process in the determination by the determination function of the output frame selection unit 25 (step S307—YES), the storage is performed. The intra-frame super-resolution processed image stored in the unit 24 is output. In the present embodiment, no change point is detected between f5 and f6, but a change point is detected between f4 and f5 performed in the previous detection process. Therefore, the output frame F6 uses f6 as a reference and f5. And a super-resolution processed image using the two images f6 (step S310).

  When the change point is detected by the change point detection unit 23 and the change point is not detected by the output frame selection unit 25, a multi-frame super-resolution image using fewer frames than the specified number is output. In this embodiment, it is F3 (step S309).

As described above, when there is a change point, by outputting the intra-frame super-resolution processed image instead of the multi-frame super-resolution processed image, the output frames obtained by performing the super-resolution processing on all the input frames are output. Can be output. Since super-resolution processing is performed on all input frames, high-resolution processing with little image degradation can be realized.

  In addition, when super-resolution processing is performed using three or more input frames, even if a change point is detected, the next input after detecting the change point (that is, two images counted from the change point) is input. After a frame is input, a super-resolution processed image using two or more frames can be output, and a larger number of multi-frame super-resolution processed images can be used as output frames. This makes it possible to output a frame with higher resolution.

(Fourth Embodiment) Claim 4
A fourth embodiment will be described with reference to FIGS.

  The differences from the first, second, and third embodiments are mainly in the multi-frame super-resolution processing unit 21 and the output frame selection unit 25. In addition, the frame data stored in the storage unit 24 is different. In FIG. 11, the video storage device 12 is described, but as in the first embodiment, a configuration including one imaging device, a plurality of imaging devices, and an imaging device control unit that controls these imaging devices. It may be.

  The multi-frame super-resolution processing unit 21 of the fourth embodiment performs multi-frame super-resolution processing by changing the reference frame while keeping the plurality of input frames used for the input frame the same. The frame that has been subjected to the multi-frame super-resolution processing is temporarily stored or stored in the storage unit 24.

  Specifically, when performing multi-frame super-resolution processing using two input frames f1 and f2, super-resolution processing using f1 and f2 with f1 as a reference frame, and f1 with f2 as a reference frame, Super resolution processing using f2 is performed. These processes may be performed in parallel, or may be performed by appropriately dividing according to the processing capability.

  The output frame selection unit 25 of the fourth embodiment includes a determination function as to whether or not a change point has been detected in the detection process of the previous change point detection unit 23. Further, a delay adjustment function for adjusting a delay time at a video change point may be provided.

  When a change point is not detected by the change point detection unit 23 and a change point is detected in the previous detection process in the determination by the determination function, a normal reference frame set in advance is used. Instead of the super-resolution processed image, a super-resolution processed image in which the reference frame position is changed is output. When there are a plurality of super-resolution processed images with the reference frame position changed, the frames at the reference frame position are output in order from the earliest.

  When a change point is not detected by the change point detection unit 23 and when a change point is not detected in the previous detection process in the determination by the determination function, a normal reference frame set in advance is used. The super-resolution processed image is output.

  The delay adjustment function adjusts the output interval of the output frames, and is not necessarily provided in the output frame selection unit 25. The delay adjustment function may be provided in the image processing device 20, and the image receiving device 30, the image processing device 20, and the like. You may provide as another apparatus between. As a function, it detects that there is a change point in a frame that is input in advance, and performs processing such as delaying the output time for several frames in advance. Other functions are the same as those in the first embodiment.

  Specifically, the operation of the image processing apparatus 20 will be described with reference to FIGS. 13 to 15 by taking as an example a case where the prescribed number of frames used for a plurality of frames is two. The reference used for so-called normal reference is the latter frame of the two frames. As shown in FIG. 13, input frames f1 to f7 are input. The input frame is subjected to super-resolution processing using f1 and f2 with f2 as a reference, and F2 is output. This is repeated for f2 to f7. Here, there is a change point between f4 and f5. Therefore, the super-resolution processed image generated when f4 and f5 are input is not output. Thereafter, when f5 and f6 are input, since the previous change point is detected between f4 and f5, the output of the super-resolution processed image using f5 and f6 is instructed with f5 as a reference, and then An instruction to output a super-resolution processed image using f5 and f6 with f6 as a reference is given.

  When these are output, as shown in FIG. 13, if the output is performed as it is, there is no frame to be output between the output frames F4 and F5. Therefore, by delaying the output time of the output frame for a predetermined time in advance, the delay processing by the delay processing function is used so that the output frames F4 and F5 are not lost as shown in FIG. I won't let you. The delay processing function delays the output timing of the output frame in advance in order to prevent the output frame from being lost. If the output frame between F4 and F5 shown in FIG. 13 is not output, the output frame timing is advanced. The function to switch.

  Further, as shown in FIG. 15, the normal reference is the previous frame of the two frames. The super-resolution processed image generated when f4 and f5 are input is only stored or deleted and not output. After that, when f5 and f6 are input, since the previous change point is detected between f4 and f5, the output of the super resolution processing image using f3 and f4 is instructed with f4 as a reference, and then An instruction to output a super-resolution processed image using f5 and f6 with f5 as a reference is given.

Next, the operation of the fourth embodiment will be described.

First, a plurality of input frames are continuously input to the image processing apparatus 20 (step S401).
The image processing apparatus 20 causes the multi-frame super-resolution processing unit 21 to process the received low-resolution image. Further, the change point detection unit 23 is caused to detect change points between input frames. In the present embodiment, an embodiment in which two input frames are used will be described, but three or more frames may be used.

  The reference used for so-called normal reference is the latter frame of the two frames. In this case, the multi-frame super-resolution processing unit 21 performs super-resolution processing with the subsequent frame as a reference (step S402) and stores it in the storage unit 24 (step S403). Also, super-resolution processing is performed using the previous frame as a reference (step S404) and stored (step S405). In the case of a plurality of sheets, super-resolution processing is performed using each input frame as a reference, and each is stored.

Specifically, as shown in FIG. 13, when f1 to f7 are input, f1 is used as a reference, super resolution processing using f1 and f2 is performed and stored (steps S402 and S403), and f2 is used as a reference. Super-resolution processing using f1 and f2 is performed and stored (steps S404 and S405). This is repeated for f2 to f7.

  The change point detection unit 23 detects a change (change point) of a certain amount or more between two input frames that are successively input (step S406).

  If a change point is detected, the process ends. In FIG. 13, since a change point is detected between f4 and f5, a super-resolution processed image using f4 and f5 with f5 as a reference is not output (step S406—YES).

  If a change point has not been detected (step S406—NO), it is further detected whether there has been a change point in the previous change point detection process or whether an output frame has been output (step S407).

  If the previous change point has not been detected (NO in step S407), a super-resolution processed image using a normal reference is output. In FIG. 13, F1 to F4 and F7.

  When the previous change point is detected (step S407-YES), the super resolution processed image with the reference changed is output (step S408), and then the super resolution processed image using the normal reference is output (step S409). ).

  When these are output, as shown in FIG. 13, if the output is performed as it is, there is no frame to be output between the output frames F4 and F5. Therefore, by delaying the output time of the output frame for a predetermined time in advance, the delay processing by the delay processing function is used so that the output frames F4 and F5 are not lost as shown in FIG. I won't let you.

  Further, as shown in FIG. 15, the normal reference is the previous frame of the two frames. The super-resolution processed image generated when f4 and f5 are input is only stored or deleted and not output. Thereafter, when f5 and f6 are input, since the previous change point is detected between f4 and f5, the output of the super-resolution processed image using f3 and f4 is instructed using f4 as a reference (step S308). Next, an instruction to output a super-resolution processed image using f5 and f6 with f5 as a reference is given (step S309).

As described above, a frame that has undergone super-resolution processing across the change points is not output, and an output frame that has been subjected to super-resolution processing can be output using all input frames as reference images. In addition, since multiple frame super-resolution processing is performed on all input frames, output frames having the same number of output pixels can be output. Further, since the super-resolution processing using only the multi-frame super-resolution processing is used, it is possible to output an output frame with higher accuracy.

(Fifth embodiment)
A fifth embodiment will be described with reference to FIGS.

  This embodiment is a modification of the fourth embodiment.

  The difference from the fourth embodiment is mainly in the output frame selection unit 25. In addition, the frame image data and the like stored in the storage unit 24 differ according to these differences.

  The multi-frame super-resolution processing unit 21 according to the fifth embodiment performs different super-resolution processing depending on the set position of the reference frame. In the present embodiment, as shown in FIG. 17, the previous frame is used as a reference frame among two plural frames.

Next, the operation of the fifth embodiment will be described with reference to FIG.

  First, a plurality of input frames are continuously input to the image processing apparatus 20 (step S501).

  The image processing apparatus 20 causes the multi-frame super-resolution processing unit 21 to process the received low-resolution image. Further, the change point detection unit 23 is caused to detect change points between input frames.

  The multi-frame super-resolution processing unit 21 performs super-resolution processing using the first frame as a reference, and stores it in the storage unit 24. Even when there are two or four or more images, the same super-resolution processing using a predetermined reference image is performed and stored.

  Specifically, as shown in FIG. 16, when f1 to f7 are input, super resolution processing using f1 and f2 is performed using f1 as a reference and stored (steps 502 and 503). At the same time, f2 is used as a reference, and super-resolution processing using f1 and f2 is performed and stored (steps S504 and S505).

  The change point detection unit 23 detects a change (change point) of two or more input frames that are successively input (a change point) (step S506).

  If no change point is detected (step S506—NO), a super-resolution frame at a normal reference position is created and output. In FIG. 17, it is F1-F3 and F5-F7 (step S507).

  If a change point is detected (step 506-YES), a super-resolution frame with the reference position changed is created and output. In FIG. 17, since a change point is detected between f4 and f5, the super-resolution processed image is not output using f4 and f5 with f4 as a reference, and f3 and f5 are used with f4 stored in the previous process as a reference. The super-resolution processed image is read from the storage unit 24 and output as F4 (step S508).

  FIG. 18 is a processing diagram when FIG. 17 is considered in a different concept. In other words, when a change point is detected, the previously processed super-resolution processing image is output, so that the processing time between F3 and F4 existing in FIG. 17 can be made extremely short. .

In this way, when super-resolution processing with the reference position in front is performed and a change point is detected, a super-resolution processing image with a different reference position that has been performed in the previous processing is output in advance. A super-resolution processed image can be output.

(Sixth embodiment)
The sixth embodiment will be described with reference to FIGS.

  This embodiment is a modification of the fourth embodiment.

  In this embodiment, as shown in FIG. 19, it has the memory | storage part 27 which memorize | stores an input frame.

  In the present embodiment, the storage unit 27 is described separately from the storage unit 24, but may be the same medium.

Next, the operation of the sixth embodiment will be described.

  In the present embodiment, as shown in FIG. 21, an example of super-resolution processing using three input frames and using the second input frame as a reference among the three input frames is shown.

  First, a plurality of input frames are continuously input to the image processing apparatus 20 (step S601).

  The image processing apparatus 20 causes the multi-frame super-resolution processing unit 21 to process the received low-resolution image. Further, the change point detection unit 23 is caused to detect change points between input frames.

  The multi-frame super-resolution processing unit 21 performs super-resolution processing using the second frame as a reference, and stores it in the storage unit 24. Even when there are two or four or more images, the same super-resolution processing using a predetermined reference image is performed and stored.

  Specifically, as shown in FIG. 20, when f1 to f7 are input, super resolution processing using f1, f2, and f3 is performed and stored using f2 as a reference (steps S602 and S603).

  The change point detection unit 23 detects a change (change point) of two or more input frames that are successively input (change point) more than a certain amount (step S604).

  If a change point is detected, the process ends. In FIG. 21, since a change point is detected between f4 and f5, a super-resolution processing image using f3, f4, and f5 with f4 as a reference, and a super-resolution processing using f4, f5, and f6 with f5 as a reference The image is not output (step S604—YES).

  If a change point is not detected (step S604—NO), it is further detected whether there was a change point in the previous change point detection process or whether an output frame was output (step S605).

  When the previous change point has not been detected (step S605-NO), a super-resolution processed image using a normal reference is output. In FIG. 21, it is F1-F3, F6, F7 (step S608).

  When the previous change point is detected (step S605—YES), the multi-frame super-resolution processing unit 21 performs super-resolution processing with the reference changed and outputs (step S606, step S607). Next, a super-resolution processed image using a normal reference is output (step S608).

  In FIG. 21, when f4, f5, and f6 are read and super resolution processing is performed, the change point is detected, and the change point is also detected when the previous f3, f4, and f5 are read. Next, the multi-frame super-resolution processing unit reads the input frame again from the storage unit 27, generates and outputs a super-resolution processed image using f2, f3, and f4 as a reference, and uses f5 as a reference. A super-resolution processed image using f5, f6, and f7 is created and output, and then a super-resolution processed image with the normal reference position image f6 as a reference is output while using the same input frame again (step S606). , Step S607, Step S608).

  When these are output, as shown in FIG. 21, if the output is performed as it is, there is no frame to be output between the output frames F3 and F4. In view of this, a delay process for delaying the output time of the output frame by a predetermined time may be used in advance to prevent the frame from being lost.

Similarly, an example of super-resolution processing using three input frames and using the first input frame of the three input frames as a reference will be described with reference to FIG.

  When f1 to f7 are input, super resolution processing using f1, f2, and f3 is performed using f3 as a reference and stored (steps S602 and S603).

  The change point detection unit 23 detects a change (change point) of two or more input frames that are successively input (change point) more than a certain amount (step S604).

  If a change point is detected, the process ends. In FIG. 22, since a change point is detected between f4 and f5, a super-resolution processing image using f3, f4 and f5 with f5 as a reference, and a super-resolution processing using f4, f5 and f6 with f6 as a reference The image is not output (step S604—YES).

  If a change point is not detected (step S604—NO), it is further detected whether there was a change point in the previous change point detection process or whether an output frame was output (step S605).

  When the previous change point has not been detected (step S605-NO), a super-resolution processed image using a normal reference is output. In FIG. 22, it is F1-F4 and F7 (step S608).

  When the previous change point is detected (step S605—YES), the multi-frame super-resolution processing unit 21 performs super-resolution processing with the reference changed and outputs (step S606, step S607). Next, a super-resolution processed image using a normal reference is output (step S608).

  In FIG. 22, when trying to perform super-resolution processing by reading f4, f5, and f6, the change point is detected, and the change point is also detected when the previous f3, f4, and f5 are read. Next, the multi-frame super-resolution processing unit reads the input frame from the storage unit 27 again, creates and outputs a super-resolution processed image using f5, f6, and f7 as a reference, and uses f6 as a reference. A super-resolution processed image using f5, f6, and f7 is created and output, and then a super-resolution processed image with the normal reference position image f7 as a reference is output while using the same input frame again (step S606). , Step S607, Step S608).

Similarly, an example of super-resolution processing using three input frames and using the first input frame of the three input frames as a reference will be described with reference to FIG.

  When f1 to f7 are input, super resolution processing using f1, f2, and f3 is performed using f1 as a reference and stored (steps S602 and S603).

  The change point detection unit 23 detects a change (change point) of two or more input frames that are successively input (change point) more than a certain amount (step S604).

  If a change point is detected, the process ends. In FIG. 22, since a change point is detected between f4 and f5, a super-resolution processing image using f3, f4, and f5 with f3 as a reference, and a super-resolution processing using f4, f5, and f6 with f4 as a reference The image is not output (step S604—YES).

  If a change point is not detected (step S604—NO), it is further detected whether there was a change point in the previous change point detection process or whether an output frame was output (step S605).

  When the previous change point has not been detected (step S605-NO), a super-resolution processed image using a normal reference is output. In FIG. 23, F1, F2, and F5 to F7 (step S608).

  When the previous change point is detected (step S605—YES), the multi-frame super-resolution processing unit 21 performs super-resolution processing with the reference changed and outputs (step S606, step S607). Next, a super-resolution processed image using a normal reference is output (step S608).

  In FIG. 23, when f4, f5, and f6 are read and super resolution processing is performed, the change point is detected, and the change point is also detected when the previous f3, f4, and f5 are read. Next, the multi-frame super-resolution processing unit reads the input frame again from the storage unit 27, creates and outputs a super-resolution processed image using f2, f3, and f4 as a reference, and uses f4 as a reference. A super-resolution processed image using f2, f3, and f4 is created and output, and then a super-resolution processed image with the normal reference position image f7 as a reference is output while using the same input frame again (step S606). , Step S607, Step S608).

As described above, when the change point is detected twice in succession, the super-resolution processing with the changed reference image is performed and output, so that the multi-frame super-resolution processing unit 21 does not need to perform the processing simultaneously.

(Seventh embodiment)
The seventh embodiment will be described with reference to FIGS. 24 and 25. FIG.

  This embodiment is a modification of the sixth embodiment.

  The difference from the sixth embodiment is mainly in the output frame selection unit 25. In addition, the frame image data and the like stored in the storage unit 24 differ according to these differences.

  The multi-frame super-resolution processing unit 21 of the sixth embodiment performs different super-resolution processing depending on the set position of the reference frame. In the present embodiment, as shown in FIG. 25, the previous frame is used as a reference frame among three frames.

Next, the operation of the seventh embodiment will be described with reference to FIG.

  First, a plurality of input frames are continuously input to the image processing apparatus 20 (step S701).

  The image processing apparatus 20 causes the multi-frame super-resolution processing unit 21 to process the received low-resolution image. Further, the change point detection unit 23 is caused to detect change points between input frames.

  The multi-frame super-resolution processing unit 21 performs super-resolution processing using the first frame as a reference, and stores it in the storage unit 24. Even when there are two or four or more images, the same super-resolution processing using a predetermined reference image is performed and stored.

  Specifically, as shown in FIG. 21, when f1 to f7 are input, super resolution processing using f1, f2, and f3 is performed using f1 as a reference and stored (steps 702 and 703).

  The change point detection unit 23 detects a change (change point) of two or more input frames that are successively input (change point) more than a certain amount (step S704).

  If no change point is detected (step S704—NO), a super-resolution frame at a normal reference position is created and output. In FIG. 25, they are F1, F2, and F5 to F7 (step S705).

  If a change point is detected (step 704-YES), a super-resolution frame with the reference position changed is created and output. In FIG. 25, since a change point is detected between f4 and f5, the super-resolution processed image is not output using f3, f4, and f5 with f3 as a reference, and f2 and f3 are stored with f3 stored in the previous process as a reference. And a super-resolution processed image using f4 and output as F3. Next, super resolution processing using f2, f3, and f4 is performed using f4 as a reference, and output as F4 (steps S707 and 78).

As described above, when the super-resolution processing with the reference position in front is performed and the change point is detected, the super-resolution processing image having a different reference position that has been previously performed in the previous processing as in the fifth embodiment. Can output a super-resolution processed image promptly.

(Eighth embodiment)
The eighth embodiment will be described with reference to FIGS.

  In the present embodiment, basically the same configuration as that of the sixth embodiment is used, but the change point detection unit 23 of the present embodiment detects the leading edge or the end of the video instead of the change point detection. .

Next, the operation of the eighth embodiment will be described.

  In this embodiment, three input frames are used and the reference position is set to be the second.

  First, a plurality of input frames are continuously input to the image processing apparatus 20 (step S801).

  The multi-frame super-resolution processing unit 21 performs super-resolution processing using the second frame as a reference, and stores it in the storage unit 24. Even when there are two or four or more images, the same super-resolution processing using a predetermined reference image is performed and stored.

  Specifically, as shown in FIG. 28, when f1 to f7 are input, super resolution processing using f1, f2, and f3 is performed and stored using f2 as a reference (steps S802 and 803).

  The change point detection unit 23 detects the leading edge or the end of the video (step S804).

  If it is the leading edge of the video (step S804-YES), it is further determined whether the preset reference image is the forefront or the back (step S805).

  If the reference image is the front, the normal reference image stored in step S803 is output from the storage unit 24 (step S808).

  When the reference image does not appear the most, the super-resolution processing with the changed reference position is performed and then output, and then the normal-resolution super-resolution processing image is output (step S806, step S807, and step S808).

  If the leading edge of the video does not appear (step S804-NO), a normal super-resolution processed image is output (step S809).

  Next, it is determined whether it is the last end of the video (step S810).

  If it is not the end of the video, the process is terminated.

  Next, it is determined whether or not the reference position is the last position. If the reference position is the last position (step S811—YES), the process is terminated as it is.

  If the reference position is not the last (step S811-NO), super-resolution processing with the changed reference position is performed and output (steps S812 and S813).

In this way, super-resolution processing with all input frames as a reference can be performed, and there is no missing frame. Super-resolution processing can be performed using the same number for all output frames.

In the first to eighth embodiments, the multi-frame super-resolution process or the intra-frame super-resolution process may be as shown in FIG. 29 in addition to the normal super-resolution process. This will be described below.

  Use transitions (switching effects) to switch camera images or scenes. “Fade-in / Fade-out / Crossfade” is often used as one of the transitions, but when finding the motion vector between the previous and next frames in the alignment process, the entire frame video changes due to the transition, so template matching The accuracy of the motion vector is lowered (matching cannot be determined). Therefore, as shown in FIG. 20, matching is performed by adding templates corresponding to the search range (arbitrary range) in the frame and the expected luminance change (+: brighter, −: darker) within the range. As a result, the accuracy of the motion vector due to the matching is not lowered, and a high-resolution image can be reconstructed by reconstructing (for example, superposing) the reference frame with a reference frame having the same luminance as the reference frame.

  When the imaging device control unit 11 is used, super-resolution processing may be performed in consideration of the amount of change in the captured image due to camera panning by the imaging device control unit.

In these embodiments, an I frame (I picture) output at the time of conversion to MPEG or the like may be used for detection of the change point by the change point detection means 23.

Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 10 ... Imaging device 11 ... Imaging device control part 12 ... Image | video storage device 20 ... Image processing apparatus 21 ... Multiple touch 0 super-resolution processing part 22 ... Intra-frame super-resolution processing part 23 ... Change point detection part 24 ... Memory | storage part 25 ... Output frame selection unit 27 ... storage unit 30 ... image receiving device

Claims (14)

First image generation means for performing intra-frame super-resolution processing on an input frame that is continuously input in time series and generating a first high-resolution frame image;
Second image generation means for generating a second high-resolution frame image using a plurality of the input frames;
Detecting means for detecting a change point between frames of the input frames that are continuously input;
Output means for outputting the first high-resolution frame image when the detection means detects a change point, and outputting the second high-resolution frame image when the detection means does not detect the change point; ,
An image processing apparatus comprising: The second image generation means generates a second high-resolution frame image using the N input frames,
The output means includes
If the detection means detects a change point, the first high-resolution frame image is output,
If the detection means does not detect a change point and if no change point is detected in any detection process before (N-2) times of the detection means, the second high-resolution frame image is output,
If the detection means does not detect the change point and if the change point is detected in any detection process before (N-2) times of the detection means, the second image generation means after the change point The image processing apparatus according to claim 1, wherein a plurality of high-resolution frame images generated using a plurality of input frames input are output. The output means includes
If the detection means does not detect the change point, the second high-resolution frame image is output,
If the detection means detects a change point and if a change point has been detected in the previous detection process of the detection means, the first high-resolution frame image is output,
When the detection unit detects a change point and when no change point is detected in the previous detection process of the detection unit, a plurality of input frames input before the change point by the second image generation unit The image processing apparatus according to claim 1, wherein a plurality of high-resolution frame images generated by using an image are output. Third image generation means for generating a third high-resolution frame image using a plurality of frames as input frames that are continuously input in time series;
A fourth image generating means for generating a fourth super-resolution frame image using a plurality of frames having different reference positions from the third high-resolution frame image;
Detecting means for detecting a change point between frames of the input frames that are continuously input;
When the detection means detects a change point, it does not output an image,
If the detection means does not detect a change point and no change point is detected in the previous detection operation of the detection means, the third high-resolution frame image is output,
If the detection means does not detect a change point and if a change point has been detected in the previous detection process of the detection means, the fourth high resolution frame image is output, and then the third high resolution An image processing apparatus comprising: output means for outputting a frame image. The output means includes
When the detection unit detects a change point, the fourth high-resolution frame image is output, and when the detection unit does not detect the change point, the third high-resolution frame image is output. An image processing apparatus according to claim 4. The output means includes
If the detection means detects a change point, does not output a super-resolution processed image,
If the detection means does not detect a change point and the change point is not detected in the previous detection process of the detection means, the third high-resolution frame image is output,
If the detection means does not detect a change point and if a change point has been detected in the previous detection process of the detection means, a plurality of input frames input after the change point by the fourth image generation means. 5. The image processing apparatus according to claim 4, wherein a plurality of high-resolution frame images are generated and output using the same, and then a third high-resolution frame image using the same input frame is output. The output means includes
When the detection means detects a change point, a plurality of high-resolution frame images are generated and output using a plurality of input frames input after the change point by the fourth image generation means,
The image processing apparatus according to claim 4, wherein the third high-resolution frame image is output when the detection unit does not detect a change point. Setting means for setting a reference frame when generating the second high-resolution frame image;
State-of-the-art detection means to detect the state of the art,
When the leading edge of the video is detected by the leading edge detection means and the reference frame is set to the Nth frame, the (N−1) th input frame from the top frame is the first high resolution frame. The image processing apparatus according to claim 1, wherein the image processing apparatus outputs the image. Setting means for setting a reference position when generating the second high-resolution frame image;
A trailing edge detecting means for detecting the trailing edge of the image;
When the last end of the video is detected by the last end detecting means and the reference frame is set to the Nth frame, the first high frame is applied to the last input frame from the (N + 1) th frame. The image processing apparatus according to claim 1, wherein a resolution frame is output. Setting means for setting a reference frame when generating the third high-resolution frame image;
State-of-the-art detection means to detect the state of the art,
When the leading edge of the video is detected by the leading edge detection means and the reference frame is set to the Nth frame, the (N-1) th frame starts with the reference frame from the topmost frame (N-1). The image processing apparatus according to claim 4, wherein the fourth super-resolution image frame set to the th frame is output. Setting means for setting a reference position in generating the third high-resolution frame image;
A trailing edge detecting means for detecting the trailing edge of the image;
When the last end of the video is detected by the last end detection means and the reference frame is set to the Nth frame, the fourth input frame in which the last input frame from the (N + 1) th frame is set as a reference is used. The image processing apparatus according to claim 4, wherein a high-resolution frame is output. In generating the first high-resolution frame image or the second high-resolution frame image,
The image processing apparatus according to claim 1, wherein super-resolution processing is performed by adding a search range in a frame and a template corresponding to a predetermined luminance change in the search range to perform matching. Intra-frame super-resolution processing is performed on input frames that are continuously input in time series to generate a first high-resolution frame image, and a plurality of the input frames are used to generate a second high-resolution frame image.
Detecting a change point between frames of the input frames that are continuously input;
When the detection means detects a change point, the second high-resolution frame image is output, and when the detection means does not detect the change point, the first high-resolution frame image is output;
An image processing method characterized by the above. A first generation process for generating a first high-resolution frame image by performing intra-frame super-resolution processing on an input frame that is continuously input in time series;
A second generation process for generating a second high-resolution frame image using a plurality of the input frames;
A detection process for detecting a change point between frames of the input frames that are continuously input;
An output process for outputting the second high-resolution frame image when the detection unit detects a change point, and outputting the first high-resolution frame image when the detection unit does not detect the change point; ,
An image processing program for operating an image processing apparatus on an image processing apparatus.

Images