JP2018023034A - Super-resolution frame selection apparatus, super-resolution device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the image quality of a super-resolution image when super-resolution processing a decoded image. A super-resolution frame selection device (11), for each divided block of a residual image, a motion prediction accuracy determination unit (111) that determines whether the prediction accuracy of motion prediction corresponding to the divided block is high or low. For a divided block on which intra prediction has been performed, super-resolution processing by a single frame is instructed, and for a divided block on which inter prediction has been performed, the prediction accuracy of motion prediction by the motion prediction accuracy determination unit. Indicates that the super-resolution processing is performed by a plurality of frames when it is determined to be high, and instructs the super-resolution processing by a single frame when the prediction accuracy of the motion prediction is determined to be low by the motion prediction accuracy determination unit. And a frame selection unit 112 that outputs frame selection information. [Selection diagram] Figure 1

Description

  The present invention relates to a super-resolution frame selection device, a super-resolution device, and their programs used for generating a super-resolution image.

  In general, as a super-resolution technology that converts an input image into a higher-resolution image, single-frame super-resolution processing that uses a single frame and super-resolution that uses multiple frames. Multi-frame super-resolution processing that performs image processing is known.

  For example, Patent Document 1 discloses a single-frame super-resolution process that searches for a similar portion in a single frame and generates a super-resolution high-frequency component that exceeds the sampling frequency of the original image from the spatial high-frequency component. Has been.

  Non-Patent Document 1 discloses multi-frame super-resolution processing using a motion vector of MPEG data. Since an MPEG motion vector does not necessarily have high accuracy, it is determined whether occlusion has occurred, alignment accuracy is determined by detecting the amount of external displacement, and whether the brightness of corresponding pixels is equal. This measures the accuracy of the motion vector.

  Japanese Patent Application Laid-Open No. 2004-228688 discloses a method of performing single frame super-resolution processing when the motion compensation error is large, and performing multi-frame super resolution processing when the motion compensation error is small and the motion speed is slow. It is disclosed.

  On the other hand, Non-Patent Document 2 describes ITU-T H.264. In H.265 / MPEG-H HEVC (hereinafter, simply referred to as “HEVC”), a coding tree unit (CTU: Coding Tree Unit) is hierarchically divided into quadtrees to divide a variable-size coding unit (CU). Coding Unit) is generated, and each CU is further divided into prediction units (PU: Prediction Unit) for inter prediction / intra prediction, and prediction processing is performed for each PU.

JP2015-203952A Japanese Patent No. 5021017

Masayuki Tanaka, Yoichi Yaguchi, Eiji Furukawa, Masatoshi Okutomi, “Robust Super-Resolution Processing Based on Pixel Selection Considering Misalignment”, IEICE Transactions D, Vol. 92-D, nO. 5, pp. 650-660, May2009. Supervised by Satoshi Okubo, “Impress Standard Textbook Series H.265 / HEVC Textbook”, Impress Japan Co., Ltd., October 21, 2013

  In the conventional multi-frame super-resolution processing, when performing the super-resolution processing using the motion prediction information of the encoded signal, the motion prediction accuracy is measured by the motion vector size and the external displacement detection method. . However, the motion prediction accuracy is not necessarily measured by the magnitude of the motion vector, and there is a problem that the use of an external positional deviation amount leads to an increase in calculation cost.

  An object of the present invention made in view of such circumstances is a super-resolution frame selection device, a super-resolution device, and those capable of improving the quality of a super-resolution image when a decoded image is subjected to super-resolution processing. Is to provide a program.

  In order to solve the above-described problem, the super-resolution frame selection device according to the present invention includes a segmentation unit used in decoding processing from a decoding device that decodes encoded data generated by performing motion prediction on an original image. A super-resolution frame selection device that inputs information and a residual image, wherein the division information is information indicating a size of each divided block when performing motion prediction, and the residual image includes an original image and a motion A motion prediction accuracy determination unit that determines whether the prediction accuracy of motion prediction corresponding to each divided block is high or low for each divided block of the residual image, which is a difference image from a prediction image generated by prediction. For a divided block for which intra prediction has been performed, super-resolution processing using a single frame is instructed, and for a divided block for which inter prediction has been performed, the motion prediction accuracy determination unit performs motion prediction. When it is determined that the prediction accuracy is high, super-resolution processing by a plurality of frames is instructed, and when the motion prediction accuracy determination unit determines that the prediction accuracy of motion prediction is low, the super-resolution processing by a single frame A frame selection unit that outputs frame selection information for instructing.

  Furthermore, in the super-resolution frame selection device according to the present invention, the motion prediction accuracy determination unit, when the power per pixel of the residual image of the divided block is equal to or less than a threshold value, the motion corresponding to the divided block It is determined that the prediction accuracy of prediction is high, and when the power per pixel of the residual image of the divided block exceeds a threshold, it is determined that the prediction accuracy of motion prediction corresponding to the divided block is low. To do.

  In order to solve the above problems, a super-resolution apparatus according to the present invention includes a super-resolution process using a single frame or a super-resolution using a plurality of frames based on the super-resolution frame selection apparatus and the frame selection information. And a super-resolution processor that performs image processing.

  Further, in the super-resolution device according to the present invention, the super-resolution image generating unit that generates an image obtained by subtracting the residual image from the decoded image as a super-resolution frame image, the super-resolution processing unit, A decoded image obtained by decoding the original image from the decoding device and motion prediction information indicating a motion vector detected by the motion prediction are input, and the frame selection information instructs a super-resolution process using a single frame. If the decoded image is super-resolved, and the frame selection information indicates super-resolution processing by a plurality of frames, the super-resolved frame image is used for the motion prediction information. And super-resolution processing.

  In order to solve the above problem, a program according to the present invention causes a computer to function as the super-resolution frame selection device.

  In order to solve the above problems, a program according to the present invention causes a computer to function as the super-resolution device.

  According to the present invention, it is possible to improve the quality of a super-resolution image when a decoded image is subjected to super-resolution processing.

It is a block diagram which shows the structural example of the super-resolution apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the structural example of the decoding apparatus from which the super-resolution apparatus which concerns on one Embodiment of this invention acquires information. It is a figure which shows an example of a process of the motion prediction accuracy determination part in the super-resolution apparatus which concerns on one Embodiment of this invention.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration example of a super-resolution apparatus according to an embodiment of the present invention. The super-resolution device 1 acquires division information, motion prediction information, a residual image, and a decoded image from the decoding device 2. First, the decoding device 2 will be described.

  The decoding device 2 is a device that inputs encoded data from an encoding device (not shown) that generates encoded data by performing motion prediction (motion compensation) on the original image, and decodes the encoded data. It is. In an encoding method such as HEVC, inter prediction (interframe prediction) and intra prediction (intraframe prediction) are used as motion prediction. Inter-prediction is a predictive coding method in which a predictive image is generated from reference frames that are temporally changed and a difference between an original image and a predictive image is encoded. Intra prediction is a predictive coding method in which a predicted image is generated from pixels of adjacent blocks that have been encoded for a block to be intraframe-encoded, and a difference between the original image and the predicted image is encoded.

  FIG. 2 is a block diagram illustrating a configuration example of the decoding device 2. In the example illustrated in FIG. 2, the decoding device 2 includes an entropy decoding unit 21, an inverse quantization unit 22, an inverse transformation unit 23, an addition unit 24, a storage unit 25, and a prediction unit 26. In this figure, the flow of data generated by the encoded data and the calculation process of the encoded data is indicated by a solid line arrow, and the flow of information used for the decoding process of the encoded data is indicated by a broken line arrow.

  The entropy decoding unit 21 decodes the encoded data input from the encoding device, and acquires quantization coefficients, division information, and motion prediction information. Then, the quantization coefficient is output to the inverse quantization unit 22, the division information is output to the inverse quantization unit 22, the prediction unit 26, and the super-resolution device 1, and the motion prediction information is output to the prediction unit 26 and the super-resolution device. Output to 1.

  Here, the quantization coefficient means a value obtained by quantizing the transform coefficient of the residual image, which is calculated by the encoding device. A residual image is a difference image between an original image and a predicted image generated by motion prediction.

  The division information is information indicating the size of each divided block when performing motion prediction, which is determined by the encoding device. In HEVC, a divided block means a PU. Further, the division information includes information indicating whether the division block is a block on which inter prediction or intra prediction has been performed.

  The motion prediction information is information indicating a motion vector detected by motion prediction.

  The inverse quantization unit 22 multiplies the quantization coefficient input from the entropy decoding unit 21 by the quantization step to restore the transform coefficient for each block, and outputs the restored transform coefficient to the inverse transform unit 23. The quantization step may be acquired from the encoding device, or may have a quantization table common to the encoding device in advance.

  The inverse transform unit 23 performs inverse transform on the transform coefficient input from the inverse quantization unit 22 to restore a residual image for each block. Then, the residual image is output to the adding unit 24 and the super resolving device 1.

  The addition unit 24 adds the pixel values of the residual image input from the inverse conversion unit 23 and the prediction image input from the prediction unit 26 to generate a decoded image for each block. Output to the resolution apparatus 1.

  The storage unit 25 is a memory that stores the decoded image input from the addition unit 24.

  The prediction unit 26 refers to the decoded decoded image stored in the storage unit 25, performs prediction processing according to the prediction information input from the entropy decoding unit 21, generates a prediction image, and outputs the prediction image to the addition unit 24.

  In this way, the super-resolution device 1 acquires the division information, the motion prediction information, the residual image, and the decoded image that is the result of the decoding process from the general decoding device 2. Returning to FIG. 1 again, the configuration of the super-resolution device 1 will be described.

  In the example illustrated in FIG. 1, the super-resolution device 1 includes a super-resolution frame selection device 11, a super-resolution image generation unit 12, and a super-resolution processing unit 13. The super-resolution frame selection device 11 includes a motion prediction accuracy determination unit 111 and a frame selection unit 112. In this figure, the flow of images is indicated by solid arrows, and the flow of information used for super-resolution processing is indicated by broken arrows.

  The motion prediction accuracy determination unit 111 inputs division information and a residual image, and determines whether the prediction accuracy of motion prediction corresponding to the divided block is high or low for each divided block of the residual image. The motion prediction accuracy determination information indicating the result is output to the frame selection unit 112. The motion prediction accuracy can be determined based on the residual image. For example, when the power per pixel of the residual image of the divided block is less than or equal to the threshold value, it is determined that the prediction accuracy of the motion prediction corresponding to the divided block is high, and the per pixel of the residual image of the divided block If the power exceeds the threshold, it is determined that the prediction accuracy of motion prediction corresponding to the divided block is low.

  FIG. 3 is a diagram illustrating an example of processing of the motion prediction accuracy determination unit 111. As shown in FIG. 3, the sizes of the divided blocks can be different sizes. The motion prediction accuracy determination unit 111 determines motion prediction accuracy for each of the divided blocks having different sizes. Here, the motion prediction accuracy determination information is a 1-bit flag that is “0” when it is determined that the motion prediction accuracy is low, and “1” when it is determined that the motion prediction accuracy is high. Indicates the value of the flag.

  The frame selection unit 112 receives the division information and the motion prediction accuracy determination information, and generates frame selection information for instructing whether to perform single-frame super-resolution processing or multi-frame super-resolution processing for each divided block. The data is output to the super-resolution processor 13. Specifically, super-resolution processing with a single frame is instructed for the divided blocks for which intra prediction has been performed. In addition, for a divided block subjected to inter prediction, when the motion prediction accuracy determination unit 111 determines that the prediction accuracy of motion prediction is high, super-resolution processing with a plurality of frames is instructed, and motion prediction accuracy determination is performed. When it is determined by the unit 111 that the prediction accuracy of motion prediction is low, super resolution processing with a single frame is instructed. The frame selection information can be, for example, a 1-bit flag that is “0” when instructing single-frame super-resolution processing and “1” when instructing multi-frame super-resolution processing.

  The super-resolution image generation unit 12 receives the decoded image and the residual image, generates a super-resolution frame image used for multi-frame super-resolution, and outputs it to the super-resolution processing unit 13. Specifically, the super-resolution image generation unit 12 sets an image obtained by subtracting the residual image from the decoded image as a super-resolution frame image.

  The super-resolution processing unit 13 performs super-resolution processing by a single frame or super-resolution processing by a plurality of frames based on the frame selection information input from the frame selection unit 112. For the super-resolution processing using a single frame, any known method such as the method described in Patent Document 1 can be used. For the super-resolution processing using a plurality of frames, any known method such as the method described in Non-Patent Document 1 can be used.

  Specifically, the super-resolution processing unit 13 receives an input from the decoding device 2 when the frame selection information input from the super-resolution frame selection device 11 indicates a super-resolution process using a single frame. The decoded image is subjected to super-resolution processing. In addition, when the frame selection information input from the super-resolution frame selection device 11 indicates super-resolution processing using a plurality of frames, the super-resolution frame input from the super-resolution image generation unit 12 The image is super-resolution processed using the motion prediction information input from the decoding device 2.

  It should be noted that a computer can be suitably used to cause the above-described super-resolution device 1 to function, and such a computer can store a program describing processing contents for realizing each function of the super-resolution device 1 in the computer. It can be realized by storing the program in a storage unit and reading and executing the program by the CPU of the computer. Similarly, a computer can be suitably used for the super-resolution frame selection device 11, and such a computer stores a program describing processing contents for realizing each function of the super-resolution frame selection device 11. The program can be realized by reading out and executing the program by the CPU of the computer. These programs can be recorded on a computer-readable recording medium.

  Thus, according to the present invention, for each divided block of the residual image, it is determined whether the prediction accuracy of motion prediction corresponding to the divided block is high or low, and the divided block subjected to intra prediction is determined. In this case, a super-resolution process with a single frame is instructed, and a super-resolution process with a plurality of frames is instructed when it is determined that the prediction accuracy of motion prediction is high for a divided block for which inter prediction has been performed. When it is determined that the prediction accuracy of motion prediction is low, super-resolution processing with a single frame is instructed. Therefore, by performing the super-resolution processing based on this instruction, the image quality of the super-resolution image when the decoded image is subjected to the super-resolution processing can be improved.

  The prediction accuracy of motion prediction can be determined accurately by determining whether or not the power per pixel of the residual image of the divided block is not more than a threshold value.

  Note that if super-resolution processing is performed using motion prediction information for an image including an encoding error, the super-resolution image includes the encoding error. For this reason, in the super-resolution processing using a plurality of frames, it is preferable to perform super-resolution processing on an image obtained by subtracting a residual image that is an encoding error from a decoded image using motion prediction information.

  Although the above embodiment has been described as a representative example, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiments can be combined into one, or one constituent block can be divided.

DESCRIPTION OF SYMBOLS 1 Super-resolution apparatus 2 Decoding apparatus 11 Super-resolution frame selection apparatus 12 Super-resolved image generation part 13 Super-resolution processing part 21 Entropy decoding part 22 Inverse quantization part 23 Inverse conversion part 24 Adder part 25 Storage part 26 Prediction Unit 111 motion prediction accuracy determination unit 112 frame selection unit

Claims (6)

A super-resolution frame selection device that inputs division information and a residual image used for decoding processing from a decoding device that decodes encoded data generated by performing motion prediction on an original image,
The division information is information indicating the size of each divided block when performing motion prediction,
The residual image is an image of a difference between an original image and a predicted image generated by motion prediction,
A motion prediction accuracy determination unit that determines, for each divided block of the residual image, whether the prediction accuracy of motion prediction corresponding to the divided block is high or low;
For a divided block for which intra prediction has been performed, super-resolution processing using a single frame is instructed, and for a divided block for which inter prediction has been performed, the prediction accuracy of motion prediction is determined by the motion prediction accuracy determination unit. Instructing super-resolution processing with a plurality of frames when it is determined that the motion prediction accuracy is high, and instructing super-resolution processing with a single frame when the motion prediction accuracy determination unit determines that the prediction accuracy of motion prediction is low A frame selection unit for outputting frame selection information;
A super-resolution frame selection device comprising:   The motion prediction accuracy determination unit determines that the prediction accuracy of motion prediction corresponding to the divided block is high when the power per pixel of the residual image of the divided block is equal to or less than a threshold, and the remaining of the divided block is determined. The super-resolution frame selection device according to claim 1, wherein when the power per pixel of the difference image exceeds a threshold, it is determined that the prediction accuracy of motion prediction corresponding to the divided block is low. . The super-resolution frame selection device according to claim 1 or 2,
A super-resolution processing unit that performs super-resolution processing by a single frame or super-resolution processing by a plurality of frames based on the frame selection information;
A super-resolution device comprising: A super-resolution image generation unit that generates an image obtained by subtracting the residual image from the decoded image as a super-resolution frame image;
The super-resolution processing unit inputs a decoded image obtained by decoding the original image from the decoding device and motion prediction information indicating a motion vector detected by the motion prediction, and the frame selection information is based on a single frame. When super-resolution processing is instructed, the decoded image is super-resolution processed, and when the frame selection information indicates super-resolution processing by a plurality of frames, the super-resolution frame The super-resolution apparatus according to claim 3, wherein the image is subjected to super-resolution processing using the motion prediction information.   A program for causing a computer to function as the super-resolution frame selection device according to claim 1. The program for functioning a computer as a super-resolution apparatus of Claim 3 or 4.

Images