RU2583725C1 - Method and system for image processing - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: invention relates to digital image processing. Image processing method using first and second base patch data, number of patches in first database is equal to number of patches in second database comprises steps of: carried out or low-frequency band-pass filtering original image; filtered image is divided into blocks of same size as size of patches in first data base patches; generating texture for each block, performing following steps: performing pre-processing; for each patch in second data base is calculated as sum of coefficients projections of pixels within block pieces pre-processed image and corresponding pixel values of first patch from database; calculate output pixel texture block as a sum of products of projection coefficients and patches from second database; perform normalisation of texture; perform post-processing of generated texture; and add texture to post-process original pixel values within block.

EFFECT: technical result is high resolution and level of detail of input image.

48 cl, 37 dwg

Description

Technical field

The present invention relates, in General, to the field of digital image and video processing and, in particular, to increasing the resolution of the image and the level of detail of the input image.

State of the art

Display devices with ultra-high definition resolution have a screen resolution that is twice the resolution of most of the existing video content. The video processing pipeline (FIG. 1) for an ultra-high-definition display device typically consists of a receiver 101 that receives a high-definition video signal 102 transmitted to a scaling device 103 that converts it into an ultra-high definition video signal 104 ultimately transmitted to an ultra-high definition display 105 as shown in FIG. 1. Ultra-high-definition video obtained from a scaling device 103 that implements well-known techniques for increasing resolution usually has little representation of high-frequency details compared to ultra-high-definition video captured by an ultra-high-definition video camera. To solve this problem, special techniques have been developed to increase the resolution of the image. In FIG. 2 shows an approach for increasing image resolution using information from several consecutive low-resolution frames for reconstruction in a high-resolution frame. In FIG. Figure 3 shows the same approach as applied to low-resolution content from digital video cameras. In documents US 6434280, US 8306121, US 20130114892, US 7373019 described the approach using multiple low-resolution frames.

No. 6,434,280 describes a super-high resolution (super-resolution) mosaic image forming system that generates a super-high-resolution mosaic image using a plurality of image frames, where each frame represents a portion of a scene for the output image. The described system consists of a mosaic image generator and an ultra-high resolution generator. The first part generates mosaic image data that is represented by a mosaic image from image frames. The second part takes the mosaic image data and image data presented in the frames, and produces a mosaic image with an increased ultra-high resolution. In the process of forming a mosaic image with increased ultra-high resolution, the mosaic image is divided into many patches (image fragments) (each patch is associated with at least one image frame), and an ultra-high resolution operation is performed in connection with this patch to form a mosaic image with increased ultra high resolution.

US 8,306,121 describes another method for generating ultra-high resolution images using a sequence of low resolution images. The method comprises a generator of an estimated high-resolution image, a motion estimation module between the evaluated high-resolution image and comparing low-resolution images from this sequence, a back-projection projector with motion compensation, and a motion-free projector that provides the desired ultra-high resolution image. In addition, the formation system contains input and output interfaces.

US 20130114892 provides for inputting at least two images represented by digital samples and outputting an ultra-high resolution image that has a higher resolution than at least one of the at least two input images. In this method, a distortion model for all input images is first obtained, then each image is recorded on a reference grid for final resolution. After that, each image is deformed into a deformed image according to the distortion model, and distorted coordinates are calculated respectively for digital samples of input images. During this procedure, the contribution weight of each digital sample for each input image is determined, and this weight is applied depending on the distorted coordinates of the digital sample. Final resolution is obtained by combining weighted digital samples at their positions.

US 7373019 proposes a super-high resolution image forming system from an image representing a scene that contains a main image and at least one other image. The system consists of a source generator with an increased ultra-high resolution, an image projector module and an generator module for updating an image rating with an increased ultra-high resolution. The ultra-high resolution source image generator module generates an increased ultra-high resolution image estimate for the scene image, the image projector module selectively uses the deformation, blurring and decimation operators applied to other images to evaluate the image with increased ultra-high resolution, and the image rating update generator with super-high resolution uses the previous high-resolution image estimates for f Framing the final updated assessment of the image with increased ultra-high resolution.

In another known approach, it is proposed to create a high resolution frame using only one low resolution frame. This can be done without additional training or using some pre-trained database.

Methods without additional training can be based on reprojection, for example, as described in JPA 2008-140012, where it is proposed to use the Gauss-Seidel method to calculate the feedback value added to a low-resolution image, the resolution of which is increased by any known method.

US 20130044965 discloses an ultra-high resolution method and system with texture synthesis that is independent of any additional information from the database. All information for upsampling the image is obtained from the image itself. First, the image resolution is increased, then it is determined whether the high resolution image has any smooth areas. The system reinforces the edges of the image with higher resolution, and if a smooth area is detected, bypasses it with the edge. Finally, a texture is generated for enhanced and unimproved images with upsampling. A texture covers smooth areas and is synthesized using an input image as an example of a texture.

Methods including a pre-trained database for storing a texture model can be included in the pipeline for processing an ultra-high definition video display, as shown in FIG. 4. Examples of ultra-high resolution methods using a pre-trained database are described in US 20130156345 and US 6766067.

In US 20130156345, it is proposed to use a plurality of frames captured at a high frame rate to reconstruct a frame with a high resolution. After merging these frames, image quality is improved using a non-linear filter implemented in the form of a pre-trained neural network. The author suggested using this method for a mobile application when shooting enlarged fragments. A neural network is trained for each specific camera model using a test image that includes radial and sinusoidal test tables.

In US 8655108 B2, image scaling is performed using a group of filters based on a low-resolution data set and a high-resolution data set. The filters are in the form of regression coefficients obtained for the subspaces of the neighborhoods of the point of the training images, while the neighborhoods are classified using the CART algorithm (classification and regression tree) in the form of a decision tree, and normalized pixel values from the neighborhood of 3 × 3 low-resolution images are taken as predictors . A regression model is created for each class, and a neighborhood of 2 × 2 high-resolution images is processed as an output.

A prototype of the present invention is the solution described in US 6766067. At the first stage of this algorithm, the initial image interpolation is performed. The resulting image is divided into overlapping patches. After that, all operations are performed for each patch in raster scan order: the system generates a scaled mid-frequency patch using this low-resolution patch, then it models the search vector by pixels of the scaled mid-frequency input patch, as well as by overlapping high-frequency patches that were generated earlier. The system finds in the previously filled database the closest vector to the given search vector and the corresponding high-frequency output patch. At the last stage of the algorithm, the high-frequency output patch is interpolated taking into account the interpolated low-resolution patch and the combination of all the patches into one ultra-high resolution image.

The multi-frame ultra-high resolution methods have the following disadvantages:

- very high sensitivity to accuracy of estimation of subpixel motion;

- very high complexity of the hardware (due to the need to save several frames and perform an accurate assessment of the movement between these frames);

- high sensitivity to image capture and degradation models (such as point spread function, motion blur parameters, noise level).

A known disadvantage of reconstruction-based methods is structural type defects.

In methods such as US 8655108 B2, a very small neighborhood of the input image is considered, which leads to a slight difference in texture in the case of a sufficient number of subspaces (for example, 8, as suggested in the best implementation method).

The well-known training-based method described in US 6766067, allows to provide high quality results, however, it has the following problems:

- for each input patch, only one mid-frequency patch is used as the generated texture, which leads to a linear relationship between the quality of the synthesized texture and the number of records in the database, therefore, very high hardware complexity;

- to suppress block defects, the method must be implemented using overlapping blocks, which increases the complexity of the hardware several times;

- the method can be trained for only one magnification;

- the method always gives exactly the same result for the same patch of low resolution image, which can lead to structural defects if the texture of the original low resolution image is not diverse enough.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention, there is provided a method for processing an image using the first and second database of patches, wherein the number of patches in the first database is equal to the number of patches in the second database, the method comprising the steps of:

- carry out low-pass or band-pass filtering of the original image;

- divide the filtered image into blocks of the same size as the size of the patches in the first database of patches;

- generate a texture for each block, performing the following steps:

- perform pre-processing;

- for each patch in the second database, projection coefficients are calculated as the sum of the product of pixels inside the block of the pre-processed image and the corresponding pixel values of the patch from the first database;

- calculate the pixels of the output texture block as the sum of the products of the projection coefficients and patches from the second database;

- perform texture normalization;

- carry out post-processing of the generated texture; and

- add a post-processed texture to the original pixel values inside the block.

According to one embodiment of the present invention, the first database contains an orthonormal basis of patches obtained from the training patches of the filtered training image, and the second database contains an orthonormal basis of patches obtained from the training patches of a texture image obtained by subtracting the filtered training image from the training image.

According to another embodiment of the present invention, the preprocessing removes the trend, including subtracting the average value of the brightness of the block, or subtracting the linear or quadratic regression model that describes the input patch mentioned here.

According to another embodiment of the present invention, the pre-processing comprises removing a trend, comprising the steps of:

- calculate the first candidate pre-processed block by subtracting from each pixel the average row value;

- calculate the second candidate pre-processed block by subtracting from each pixel the average column value;

- calculate the third candidate of the pre-processed block by subtracting from each pixel the average value for pixels lying on the same main diagonal;

- calculate the fourth candidate pre-processed block by subtracting from each pixel the average value for pixels lying on the same additional diagonal;

- choose from the first, second, third and fourth candidates a pre-processed block of the candidate having the smallest discrepancy or standard deviation, as the target pre-processed block;

- calculate the relationship between the discrepancy or standard deviation of the target pre-processed block and the discrepancy or standard deviation of the source block;

- calculate the mixing coefficient as a decreasing function of this ratio, which takes a value between unity and zero;

- calculate the final values of the pre-processed pixels as the sum of the product of the mixing coefficient and the pixels of the target pre-processed block and the product "one minus the mixing coefficient" (1-mixing coefficient) and the pixels of the original block.

According to another embodiment of the present invention, the pre-processed pixel in the pre-processing is calculated as the sum or product with the pixel from some additional data structure having the same size as the image.

According to another embodiment of the present invention, an additional data structure is formed as a smooth function of several random or pseudo-random (depending on the position of the block) arguments.

According to another embodiment of the present invention, elements of an additional data structure are formed as

, $$D_{xy}=c_{\mathrm{one}}\cdot e^{-\frac{{((x\%p)-X_{\mathrm{one}})}^2+{((y\%q)-Y_{\mathrm{one}})}^2}{{\sigma }_{\mathrm{one}}^2}}+{\mathrm{<figure-callout\; id="2"\; label="c"\; filenames="00000071.png"\; state="\{\{state\}\}">c</figure-callout>}}_2\cdot e^{-\frac{{((x\%p)-X_2)}^2+{((y\%q)-Y_2)}^2}{{\mathrm{<figure-callout\; id="2"\; label="\sigma "\; filenames="00000071.png"\; state="\{\{state\}\}">\sigma </figure-callout>}}_2^2}}$$

,

,

, X₁, Y₁, X₂, Y₂ - псевдослучайные числа, зависящие от положения на строке floor(y/q) и положения в столбце floor(x/p) текущего блока пикселей.where x, y is the position of the pixel on the row and in the column inside the image, p is the block width, q is the block height, and c ₁ , c ₂ ,

,

, X ₁ , Y ₁ , X ₂ , Y ₂ - pseudo-random numbers depending on the position on the floor row (y / q) and the position in the floor column (x / p) of the current block of pixels.

или

, где f(·) - некая неубывающая функция, принимающая положительные значения на правом полуинтервале, и R_ij - значения пикселей текстуры.According to another embodiment of the present invention, texture normalization is calculated as

or

, where f (·) is a certain non-decreasing function that takes positive values on the right half-interval, and R _ij are the pixel values of the texture.

или

, где

или

, где N и M - некие положительные числа, и f() - некая неубывающая функция, принимающая положительные значения на правом полуинтервале, b^k - коэффициенты проекции, и T^k - высокочастотные патчи из высокочастотной базы данных.According to another embodiment of the present invention, texture normalization is calculated as

or

where

or

, where N and M are some positive numbers, and f () is a certain non-decreasing function that takes positive values on the right half-interval, b ^k are the projection coefficients, and T ^k are high-frequency patches from the high-frequency database.

According to another embodiment of the present invention, the post-processing of the generated texture comprises the steps of:

- set the pixel-by-pixel gain for each pixel in the current block to unity;

- for each block, the norm of the generated texture and the norm of the texture generated in adjacent blocks are calculated: upper left, upper, upper right, left, right, lower left, lower right and lower;

- for each neighboring block, which has a texture norm below a predetermined threshold, the gain in the current blocks is updated, containing the stages in which:

- calculate the average brightness value of the neighboring block of the input image;

- calculate the median brightness value of the neighboring block of the input image;

- for each pixel of the current block, if it has an absolute difference from the average brightness value below the first predefined threshold or an absolute difference from the median brightness value below the second predefined threshold, the pixel-by-pixel gain is re-calculated as the minimum value of the previous pixel-by-pixel gain and the relationship between the texture norm of the corresponding neighboring block and the texture norm of the source block;

- recalculate each texture pixel in the current block by multiplying by the corresponding pixel-by-pixel gain.

According to another embodiment of the present invention, the post-processing of the generated texture comprises the steps of:

- set the pixel-by-pixel gain of each pixel in the current block to unity;

- for each block, the norm of the generated texture and the norm of the texture generated in adjacent blocks are calculated: upper left, upper, upper right, left, right, lower left, lower right and lower;

- for each neighboring block, which has a texture norm below a predetermined threshold, the gain in the current block is updated, containing the steps in which:

- calculate the average values of the neighboring block of the input image in each color channel (RGB, YCbCr or other color space);

- calculate the median values of the adjacent input image block in each color channel (RGB, YCbCr or other color space);

- for each pixel of the current block, the first test value is calculated as a certain monotonically increasing function of the three arguments of the absolute values of the differences of the pixel values of the input image and the corresponding average values from the neighboring block in each color channel;

- for each pixel of the current block, the second test value is calculated as a certain monotonically increasing function of the three arguments of the absolute values of the differences in the pixel values of the input image and the corresponding median values from the neighboring block in each color channel;

- for each pixel of the current block, if its first test value is lower than the first predefined threshold or the second test value is lower than the second predefined threshold, the pixel-by-pixel gain is calculated again as the minimum value of the previous pixel-by-pixel gain and the relationship between the texture norm of the corresponding neighboring block and the norm source block textures;

- recalculate each texture pixel in the current block by multiplying by the corresponding pixel-by-pixel gain.

According to another embodiment of the present invention, the post-processing of the generated texture comprises the steps of:

- for each block (which may have a size different from the size of the block used to generate the texture), segmentation into k classes is performed;

- for each class, the average absolute value of the generated texture is calculated;

- for each pixel inside the block, the relation between the average absolute value of the generated texture in some small neighborhood and the average absolute value of the corresponding segment is calculated;

- calculate the pixel-by-pixel gain, which takes a zero or very small value if the ratio is very large, and a unit value if this ratio is less than unity or very close to it;

- recalculate each texture pixel in the current block by multiplying by the corresponding pixel-by-pixel gain.

According to another embodiment of the present invention, said method uses an overlapping arrangement of blocks.

According to another embodiment of the present invention, the second database consists of patches filling only the non-overlapping part of the input block.

According to another embodiment of the present invention, the texture generated for neighboring blocks is multiplied by a certain function of mixing pixel coordinates within the block.

According to another embodiment of the present invention, the texture generated for adjacent blocks is stitched along the minimum energy path.

According to another embodiment of the present invention, the blocks are rectangular in shape.

According to another embodiment of the present invention, all the blocks have the same shape and form planar mosaic fragmentation.

According to another embodiment of the present invention, the input image is divided into blocks of two or more different shapes and / or sizes, and separate first and second databases are stored for each shape and / or block size.

According to a second aspect of the present invention, there is provided a method for processing an image using the first and second database of patches, wherein the number of patches in the first database is equal to the number of patches in the second database, the method comprising the steps of:

- carry out low-pass or band-pass filtering of the input image;

- divide the filtered image into blocks of the same size as the size of the patches in the first database of patches;

- generate a texture for each block, performing the following steps:

- perform pre-processing;

- for each patch in the second database, projection coefficients are calculated as the sum of the product of pixels inside the block of the pre-processed image and the corresponding pixel values of the patch from the first database;

- calculate the pixels of the output texture block as the sum of the projection coefficients and patches from the second database;

- perform texture normalization;

- interpolating the input image using the method of two-dimensional interpolation;

- perform post-processing of the generated normalized texture; and

- get an ultra-high resolution image by adding a post-processed texture to the interpolated image.

According to one embodiment of the present invention, the patch size of the first database is equal to the patch size of the second database, and the interpolation of the input image comprises interpolating the input image to a size equal to the size of the required ultra-high resolution.

According to another embodiment of the present invention, the horizontal patch size from the first database is m times smaller than the horizontal patch size from the second database, and the vertical patch size from the first database is n times smaller than the vertical patch size from the second database, and :

- the interpolation of the input image contains the interpolation of each block of the input image horizontally m times and vertically n times more;

- adding a post-processed texture to the interpolated image comprises adding a post-processed texture to each interpolated block of the input image; and

- the ultra-high resolution image has a horizontal size m times larger than the input image, and a vertical size n times larger than the input image.

According to a third aspect of the present invention, there is provided a method for multiscale image processing using at least two scales, each using the first and second database of patches, the number of patches in the first database being equal to the number of patches in the second database, the method comprising the steps , where:

- create a texture layer for each scale by performing the following steps:

- carry out low-pass or band-pass filtering of the input image;

- divide the filtered image into blocks of the same shape and size as the shape and size of the patches in the first database of a given scale;

- generate a texture for each block, performing the following steps:

- perform pre-processing;

- for each patch in the second database, projection coefficients are calculated as the sum of the product of pixels inside the block of the pre-processed image and the corresponding pixel values of the patch from the first database;

- calculate the pixels of the output texture block as the sum of the projection coefficients and patches from the second database;

- perform texture normalization;

- make up the resulting texture from texture layers generated for multiple scales;

- carry out post-processing of the resulting texture; and

- add a post-processed texture to the original pixel values inside the block.

According to one embodiment of the present invention, the first database of each scale contains an orthonormal basis of patches obtained from the training patches of the filtered training image, and the second database of each scale contains an orthonormal basis of patches obtained from the training patches of a texture image obtained by subtracting the filtered training image from the training images, and filters used to obtain a filtered training image at each scale e represent some lowpass and bandpass filters, which can be different for each scale.

According to another embodiment of the present invention, composing the resulting texture from texture layers generated for at least two scales is calculated as a weighted sum of texture pixels calculated at each scale.

According to another embodiment of the present invention, compiling the resulting texture from texture layers generated for at least two scales comprises the steps of:

- calculate for each block the norm of the generated texture at the smallest scale (a scale that has band-pass or low-pass filtering, supporting the highest frequency range among all other scales);

- calculate the gain as a certain monotonous function of the norm of the block texture at the smallest scale, which takes a small or zero value when the texture norm is small (below the threshold), then gradually increases and takes on some larger predetermined value if the texture norm is above a predetermined threshold;

- recalculate texture layers at each scale, except for the smallest, by multiplying each block of texture by the gain.

According to a fourth aspect of the present invention, there is provided a multi-scale image processing method using at least two scales, each of which uses a first and second patch database, the number of patches in the first database being equal to the number of patches in the second database, the method comprising the steps , where:

- create a texture layer for each scale by performing the following steps:

- carry out low-pass or band-pass filtering of the input image;

- divide the filtered image into blocks of the same shape and size as the shape and size of the patches in the first database of patches of a given scale;

- generate a texture for each block, performing the following steps:

- pre-processing for each patch in the second database is performed by calculating the projection coefficients as the sum of the product of pixels inside the block of the pre-processed image and the corresponding pixel values of the patch from the first database;

- calculate the pixels of the output texture block as the sum of the projection coefficients and patches from the second database;

- perform texture normalization;

- interpolate the input image using the method of two-dimensional interpolation;

- make up the resulting texture from texture layers generated for at least two scales;

- perform post-processing of the resulting texture;

- create an ultra-high resolution image by adding a post-processed texture to the interpolated image.

According to one embodiment of the present invention, the patch size of the first database is equal to the patch size of the second database, and the interpolation of the input image comprises interpolating the input image to a size equal to the size of the required ultra-high resolution image.

According to another embodiment of the present invention, the input image interpolation comprises interpolating each input image block horizontally m times and vertically n times more; adding a post-processed texture to the interpolated image comprises adding a post-processed texture to each interpolated block of the input image; and the super-high resolution image has a horizontal size m times larger than the input image, and a vertical size n times larger than the input image.

According to a fifth aspect of the present invention, there is provided an image processing system comprising: a filtering unit, a projection coefficient calculation unit, a low-frequency texture bank, a high-frequency texture bank and a texture synthesis unit, the system input being connected to the input of the filtering unit, the output of the filtering unit being connected to the first input of the unit calculating projection coefficients, the output of the first texture bank is connected to the second input of the projection coefficient calculating unit, the output of the projection coefficient calculating unit nen with the first input of the texture synthesis block, the output of the second texture bank is connected to the second input of the texture synthesis block, the output of the texture synthesis block is fed to the system output.

, где P_ij - значения пикселей, и $${Е}_{ij}^k$$

- пиксели патчей из первой базы данных; блок синтеза текстуры выполнен с возможностью вычисления значений пикселей текстур как

, где $${Т}_{ij}^k$$

- пиксели патчей из второй базы данных, и выполнения нормализации текстур.According to one embodiment of the present invention, the filtering unit is configured to perform low-pass or band-pass filtering; the first texture bank is configured to store texture patches from the first database; the second texture bank is configured to store texture patches from the second database; the projection coefficient calculation unit is configured to calculate projection coefficients as

where P _ij are the pixel values, and $$E_{ij}^k$$

- pixels of patches from the first database; a texture synthesis unit is configured to calculate texture pixel values as

where $$T_{ij}^k$$

- pixels of patches from the second database, and performing normalization of textures.

The technical result of the present invention is to increase the level of detail of the input image or video stream, or increase the resolution of the input image or video stream. The proposed method for creating ultra-high resolution images uses less sophisticated hardware than in the known methods, due to its following properties:

- requires a smaller texture database size to obtain a texture of similar quality;

- no overlap processing is required due to the special texture database configuration.

The present invention has the following differences from the known solutions:

- the generated texture is formed as a linear combination of texture patches stored in the database of high-frequency patches, where the linear combination coefficients are estimated as the correlation of the patch from the low-resolution image (after low-pass or bandpass filtering) with the low-resolution patches stored in the low-frequency patch database;

- low-resolution patches stored in the database of low-frequency patches form an orthonormal basis;

- the generated texture undergoes a normalization procedure to suppress texture that is too high in high-contrast areas of the image and to enhance texture in low-contrast areas of the image;

- the method may contain different scales, which allows you to adapt to multiple magnifications (for example, × 2, × 4) within the same solution;

- the method allows you to increase the resolution to an arbitrarily selected output size using specially trained databases of low- and high-frequency patches for a given magnification;

- the method can be combined with a special subalgorithm to enhance the diversity of the generated texture, if the texture of the original low-resolution image is not diverse enough.

Brief Description of the Drawings

FIG. 1. The processing pipeline in the ultra-high definition display device.

FIG. 2. Multi-frame ultra-high resolution in the video processing pipeline of an ultra-high definition display device.

FIG. 3. Multi-frame ultra-high resolution in the video processing pipeline of a digital video camera.

FIG. 4. A video processing pipeline including a texture modeling unit in an ultra-high definition display device.

FIG. 5. Another possible video processing pipeline, including a texture modeling unit.

FIG. 6. The database of low-frequency patches (a) and the database of high-frequency patches (b) for the structure in FIG. four.

FIG. 7. The ultra-high resolution image construction scheme for the structure of FIG. four.

FIG. 8. Parallel multiscale ultra-high resolution image construction.

FIG. 9. Consistent multiscale ultra-high resolution image design.

FIG. 10. Texture generation scheme for one image patch.

FIG. 11. Texture generation scheme with pre-processing for one image patch.

FIG. 12. The pixels of the low-frequency patch database (a), the pixels of the high-frequency patch database (b), the pixels of the processed image patch (c) and the pixels of the generated texture (d).

FIG. 13. Pixels of an additional data structure introducing an additional difference.

FIG. 14. The database of low-frequency patches (a) and the database of high-frequency patches (b) for the structure in FIG. 5.

FIG. 15. The ultra-high resolution image construction scheme for the structure of FIG. 5.

FIG. 16. A system for constructing an ultra-high resolution image with a work stream parallel to the scaling device.

FIG. 17. System for constructing ultra-high resolution images with post-processing output of the scaling device.

FIG. 18. A multiscale system for creating ultra-high resolution images with mixing controls based on the pixels of the original image.

FIG. 19. A multiscale system for creating ultra-high resolution images with blending controls based on the generated texture.

FIG. 20. A multiscale system for creating ultra-high resolution images with mixing controls based on projection coefficients.

FIG. 21. An exemplary pixel map forming an auxiliary data structure introducing an additional difference.

FIG. 22. Processing results obtained using the best implementation.

FIG. 23. An implementation option with pre-processing and post-processing.

FIG. 24. Structural defects on the lines: (a) input image, (b) output texture with defects.

FIG. 25. Block pre-processing: (a) a block with a pronounced vertical edge; (b) a block with a pronounced vertical edge after pre-processing; (c) a block with a pronounced horizontal edge; (d) a block with a pronounced horizontal edge after pre-processing; (e) a block with a pronounced main diagonal edge; (f) a block with a pronounced main diagonal edge after preprocessing; (g) a block with a pronounced additional diagonal edge; (h) a block with a pronounced additional diagonal edge after pre-processing.

FIG. 26. An implementation option with pre-processing, post-processing and block gain control.

FIG. 27. Sampling results of a three-scale implementation with a block size of 16 × 16: (a) the original image; (b) the texture generated for the first scale; (c) the texture generated for the second scale; (d) the texture generated for the third scale.

FIG. 28. Three-scale implementation scheme with merge logic.

FIG. 29. Processing results: (a) without merge logic, (b) with merge logic.

FIG. 30. Neighborhood of the original image and the generated texture used to provide protection at the pixel level.

FIG. 31. Protection at the pixel level in detail: (a) 16 blocks of 16 × 16 of the original image, (b) 16 blocks of 16 × 16 of the generated texture, (c) 16 blocks of 16 × 16 of the generated texture after protection at the pixel level.

FIG. 32. Processing results: (a) without protection at the pixel level, (b) a textured image obtained from the original image in Fig. 27a.

FIG. 33. Security logic based on segmentation: (a) source image; (b) a texture generated without segmentation protection logic; (c) texture generated without security logic based on segmentation; (d) K-means segmentation; (e) segmentation obtained by maximizing expectations.

FIG. 34. An embodiment with an overlapping arrangement of blocks: (a) adjacent overlapping blocks; (b) the size of the patches from the low-frequency database; (c) the size of the patches from the high-frequency database filling the non-overlapping part of the input block; (d) other possible patch sizes from the high-frequency database filling the entire input block.

FIG. 35. A possible way to stitch textures generated in neighboring blocks, in the case of overlapping arrangement of blocks.

FIG. 36. An implementation option with hexagonal blocks.

FIG. 37. An implementation option using blocks of two different forms.

Description of Embodiments

The present invention relates, in General, to the field of digital image processing and, in particular, to increasing the resolution or formation of images of ultra-high resolution, based on training. An ultra-high resolution image means an image that has a higher resolution (more pixels) or more image details than the low resolution image used to create this ultra-high resolution image.

The proposed method can be used as a post-processing unit to increase the level of detail of the input video signal. For example, there is very little ultra-high definition content, but ultra-high definition television sets are ubiquitous. Therefore, existing standard and high definition content is displayed on an ultra-high definition display after zooming. Such zoomed-in content has a significantly lower level of detail than authentic ultra-high definition content. The described invention can be used to increase the level of detail by creating an ultra-high resolution image of the type shown in FIG. 4. The receiver 401 receives and decodes a standard or high resolution video signal 402 (low resolution image stream) that is scaled to ultra-high definition in the scaling device 403, and the ultra high definition video signal 404 is processed by the texture modeling unit 405, which increases the level of detail of the video signal. An ultra-high-definition video signal 406 with an increased level of detail (ultra-high resolution image stream) can be displayed on an ultra-high definition display 407.

In FIG. 5 shows how the proposed invention can be used to create an ultra-high resolution image 507 based on a low resolution image 501. A low-resolution image 501 (e.g., a standard or high definition video signal received by a television receiver) is scaled to ultra-high definition in a scaler 502 and is used in a texture modeling unit 503 to create a texture layer 505. The sum of the scaled image 504 and the texture layer 505 calculated in block 506 forms an ultra-high resolution image 507.

The main idea of this method is to find a correspondence between the low-resolution patches from the input image in the low-frequency patch database (Fig. 6-a) and, using this correspondence, reconstruct the high-resolution patch of the output image from the high-frequency patch database (Fig. 6-b). Patches in databases of high-frequency and low-frequency patches can have the same resolution, as shown in FIG. 6. In FIG. 7 shows a general diagram of a method corresponding to that of FIG. 4. The input image 701 low resolution is subjected to low-pass or band-pass filtering 702. The filtered image of the low resolution is divided 703 into non-overlapping or overlapping blocks. Then, for each block, 704 high-resolution texture is generated and 705 added to the original image, scaled to high resolution using some conventional method (for example, bicubic). Texture generation can be accomplished by multiscale processing as shown in FIG. 8-9. FIG. 8 illustrates a parallel multi-scale processing scheme where the input image 801 is processed in texture generation units 802 and 803, which generate two different textures 804 and 805, respectively, which merge and add 806 to the original image. Texture generation blocks 802 and 803 can have different filtering cores, block sizes and overlap parameters, as well as texture databases used to generate the texture. It is clear that, if necessary, you can connect three or more texture generation blocks in a parallel circuit. FIG. 9 illustrates a sequential multi-scale processing scheme where the input image 901 is first processed by the first texture generating unit 902, which creates an intermediate resultant image 903, which is then processed by the second texture generating unit 904, which creates the final resulting image. Texture generation blocks 902 and 904 can have different filtering cores, block size and overlap parameters, as well as texture databases used to generate the texture. It is understood that, if necessary, three or more texture generation units can be combined into a serial circuit. Each texture generation subalgorithm operates as described in FIG. 10. For each patch 1001 of the input image, a set of projection coefficients 1002 is calculated as the correlation coefficients of the patch of the input image and patches from the first database that contains low-frequency patches, then a texture is generated as a linear combination of patches from the second database that contains high-frequency patches 1003, and normalization is carried out 1004.

A texture can also be generated, as shown in FIG. 11. For each patch 1101 of the input image, preliminary processing 1102 is performed, then 1103 a set of projection coefficients is calculated as the correlation coefficients of the patch of the input image and low-resolution patches from the first database that contains low-frequency patches, after which a texture is created by calculating 1104 linear combination of patches from the second database, which contains high-frequency patches, and normalization 1105.

. В блоках 1003 и 1104 пиксели сгенерированной текстуры (фиг. 12-d) вычисляются на основе пикселей патчей из второй базы данных (фиг. 12-b) и коэффициентов проекции как

. Нормализацию 1004 и 1105, соответственно, осуществляют как

или

, где f(·) - некая неубывающая функция, принимающая положительные значения на правом полуинтервале. Нормализация 1004 и 1105 может быть также осуществлена как $${\tilde{R}}_{ij}=\frac{R_{ij}}{n}$$

или $${\tilde{R}}_{ij}=\frac{R_{ij}}{f(n)}$$

, где

или

, N и M - некие положительные числа, и f(·) - некая неубывающая функция, принимающая положительные значения на правом полуинтервале.In blocks 1002 and 1103 for the pixels of the processed image patch (FIG. 12-c), projection coefficients are calculated based on the pixels of the patches from the first database (FIG. 12-a) as

. In blocks 1003 and 1104, the pixels of the generated texture (FIG. 12-d) are calculated based on the pixels of the patches from the second database (FIG. 12-b) and projection factors as

. Normalization 1004 and 1105, respectively, is carried out as

or

, where f (·) is a certain non-decreasing function that takes positive values on the right half-interval. Normalization 1004 and 1105 can also be implemented as $${\tilde{R}}_{ij}=\frac{R_{ij}}{n}$$

or $${\tilde{R}}_{ij}=\frac{R_{ij}}{f(n)}$$

where

or

, N and M are some positive numbers, and f (·) is a certain non-decreasing function that takes positive values on the right half-interval.

Pre-processing 1102 may include, although not necessarily, removing a trend. Removing a trend can consist in simply subtracting the average value of the brightness of the block or subtracting a linear or quadratic regression model that describes the mentioned input patch. Pre-processing may also include some modification of the input patch, aimed at enhancing the difference between the generated texture in case the texture of the original image has a slight difference (similar edge). The pre-processed pixels of the original patch can be calculated as a pixel-by-pixel sum or pixel-by-pixel product with pixels of an additional data structure (Fig. 13), which can be formed as a smooth function of several random or pseudorandom (depending on the position of the block) arguments.

In the case of the circuit shown in FIG. 5, patches from the first database that contains low-frequency patches (Fig. 14-a) are k times smaller than patches from the second database that contains high-frequency patches (Fig. 14-b), where k is the scaling factor. In FIG. 15 is a general diagram of a method corresponding to that shown in FIG. 5. The low-resolution input image 1501 undergoes low-pass or band-pass filtering 1502. The low-resolution filtered image is divided 1503 into non-overlapping or overlapping blocks. Then, 1504 high-resolution texture is generated for each block, and the scale of the block is increased 1505 by a factor of k using any known method, for example, a bilinear or bicubic, or other known method. A texture is added 1506 to each zoomed-in block of the original image to produce an ultra-high resolution image.

In FIG. 16-18 show an ultra-high resolution image creation system. FIG. 16 describes a system for creating an ultra-high resolution image with a workflow that is parallel to the scaling device. The low-resolution image obtained from the receiver 1601 is supplied to a video enhancement pipeline 1602 that is connected to the recording port of the linear memory 1603 and the filtering unit 1604. The read port of the linear memory 1603 is connected to the input of the scaling device 1605, which performs up-sampling of the image by or a known method. The output of the filtering unit 1604 is connected to the first input of the projection coefficient calculator 1607. The low-resolution texture patches from the first database are stored in the low-frequency texture bank 1606, which is connected to the second input of the projection coefficient calculator 1607. The output of the projection coefficient calculator 1607 is connected to the first input of the texture synthesis block 1609. The high-resolution texture patches from the second database are stored in the high-frequency texture bank 1608, which is connected to the second input of the texture synthesis unit 1609. The output of the scaling device 1605 is connected to the first input of the texture input unit 1610. The output of texture synthesis block 1609 is connected to the second input of texture adding block 1610. The output of texture adding unit 1610 is connected to an ultra-high definition display device 1611. Video enhancement conveyor 1602 performs operations typically used to improve video quality in display devices, including, but not limited to, noise reduction, frame rate conversion, interlace to progressive, detail enhancement, and contrast enhancement. The filtering unit 1604 performs low-pass or band-pass filtering according to step 702. The projection coefficient calculator 1607 calculates the projection coefficients as in step 1002. The texture synthesis unit 1609 generates a texture as in step 1003. The texture addition unit 1610 calculates a gain, multiplies it by the generated texture and adds this artwork to the original low resolution image. The gain can be a constant predetermined value or some value calculated based on the local properties of the image.

FIG. 17 depicts a system for creating an ultra-high resolution image with post-processing output of a scaler. The low resolution image obtained from the receiver 1701 is supplied to the video enhancement pipeline 1702. The video enhancement conveyor 1702 performs operations typically used to improve the quality of video in display devices, including, but not limited to, noise reduction, frame rate conversion, interlace to progressive, detail enhancement, and contrast enhancement. The output of the video enhancement conveyor 1702 is connected to the input of the scaling device 1703, which performs upsampling using any conventional method. The output of the scaling device 1703 is connected to the recording port of the linear memory 1704 and the filtering unit 1705. The filtering unit 1705 performs low-pass or bandpass filtering according to step 702. The output of the filtering unit 1705 is connected to the first input of the projection coefficient calculator 1707. The low-resolution texture patches from the first database are stored in the low-frequency texture bank 1706, which is connected to the second input of the projection coefficient calculator 1707. The projection coefficient calculator 1707 calculates the projection coefficients as in step 1002. The output of the projection coefficient calculator 1707 is connected to the first input of the texture synthesis block 1709. The high resolution texture patches from the second database are stored in the high-frequency texture bank 1708, which is connected to the second input of the texture synthesis block 1709. The texture synthesis unit 1709 generates a texture, as in step 1003. The read port of the linear memory 1704 is connected to the first input of the texture input unit 1710. The output of texture synthesis block 1709 is connected to the second input of texture adding block 1710. Block 1710 add texture calculates the gain, multiplies it by the generated texture and adds this product to the original low-resolution image. The gain can be a constant predetermined value or some value calculated based on the local properties of the image. The output of texture adding unit 1710 is connected to an ultra-high definition display device 1711.

FIG. 18 illustrates a multi-scale system for creating ultra-high resolution images. An input video stream 1800 is supplied to the inputs of the first and second texture generation units 1801 and 1802, performing the operations described in steps 801 and 802 of a parallel multiscale ultra-high resolution image construction circuit. The outputs of the first and second texture generation units 1801 and 1802 are connected to the first and second inputs of the texture adding unit 1803. The texture adding unit 1803 comprises a mixing control block 1804, an adder 1805 and multipliers 1806 and 1807. An input video stream is input to the mixing control block, to the mixing control block 1804. The output of the first texture generating unit 1801 is connected to the first input of the multiplier 1806. The output of the second texture generating unit 1802 is connected to the first input of the multiplier 1807. The first output of the mixing control unit 1804 is connected to the second input of the multiplier 1806. The second output of the mixing control unit 1804 is connected to the second input of the multiplier 1807. The output of the multiplier 1806 is connected to the first input of the adder 1805. The output of the multiplier 1807 is connected to the second input of the adder 1805. The video stream 1800 is fed to the third input of the adder 1805. The output of this adder is connected to the output A system for creating ultra-high resolution images with pixel-based blending controls for the original image. Mixing control unit 1804 calculates the proportion of the texture generated by the first texture generation unit 1801 and the proportion of the texture generated by the second texture generation unit 1802, which is transmitted to the system output. This proportion is calculated based on the pixels of the original image, i.e. it may be proportional to the average absolute value of the original pixels filtered by some predefined digital filters (such as Gaussian).

или

, где k₁ - некая предварительно определенная константа, R¹ - прямоугольный блок значений текстуры, сгенерированной первым блоком 1901 генерации текстуры, и f(·) - некая неубывающая функция, принимающая положительные значения на правом полуинтервале. Пропорция текстуры, сгенерированной вторым блоком 1902 генерации текстуры, которая передается на выход системы, может быть вычислена как

или

, где k₂ - некая предварительно определенная константа, R² - прямоугольный блок значений текстуры, сгенерированной вторым блоком 1902 генерации текстуры, и f(·) - некая неубывающая функция, принимающая положительные значения на правом полуинтервале.FIG. 19 illustrates a multi-scale method for creating an ultra-high resolution image with blend control based on the generated texture. An input video stream 1900 is supplied to the inputs of the first and second texture generation units 1901 and 1902, performing the operations described in steps 801 and 802 of a parallel multiscale ultra-high resolution image generating circuit. The outputs of the first and second texture generation units 1901 and 1902 are connected to the first and second inputs of the texture adding unit 1903. The texture adding unit 1903 comprises a mixing control unit 1904, an adder 1905 and multipliers 1906 and 1907. The output of the first texture generating unit 1901 is connected to the first input of the multiplier 1906 and the first input of the mixing control unit 1904. The output of the second texture generation unit 1902 is connected to the first input of the multiplier 1907 and the second input of the mixing control unit 1904. The first output of the mixing control unit 1904 is connected to the second input of the multiplier 1906. The second output of the mixing control unit 1904 is connected to the second input of the multiplier 1907. The output of the multiplier 1906 is connected to the first input of the adder 1905. The output of the multiplier 1907 is connected to the second input of the adder 1905. The video stream 1900 is fed to the third input of the adder 1905. The output of the adder is connected to the output of the system to create an ultra-high resolution image with mixing control based on the generated texture. The mixing control unit 1904 calculates the proportion of the texture generated by the first texture generation unit 1901 and the proportion of the texture generated by the second texture generation unit 1902, which is transmitted to the system output. The proportion of the texture generated by the first texture generation unit 1901, which is transmitted to the system output, can be calculated as

or

, where k ₁ is a certain predefined constant, R ¹ is a rectangular block of texture values generated by the first texture generation unit 1901, and f (·) is a certain non-decreasing function taking positive values on the right half-interval. The proportion of the texture generated by the second texture generation unit 1902, which is transmitted to the system output, can be calculated as

or

, where k ₂ is a certain predetermined constant, R ² is a rectangular block of texture values generated by the second texture generation unit 1902, and f (·) is a certain non-decreasing function taking positive values on the right half-interval.

FIG. 20 illustrates a multi-scale system for creating ultra-high resolution images with mixing controls based on projection coefficients. An input video stream 2000 is supplied to the inputs of the first and second texture generation blocks 2001 and 2002, performing the steps described in steps 801 and 802 of a parallel multiscale ultra-high resolution image generating circuit. The first output of the first texture generation unit 2001 is connected to the first input of the texture adding unit 2003. The second output of the first texture generation unit 2001 is connected to the third input of the texture input unit 2003. The first output of the second texture generating unit 2002 is connected to the second input of the texture adding unit 2003. The second output of the second texture generating unit 2002 is connected to the fourth input of the texture adding unit 2003.

The first texture generation unit 2001 and the second texture generation unit 2002 comprise a filtering unit 2008, a low-frequency texture bank 2009 containing patches from the first database, projection coefficient calculation unit 2010, a high-frequency texture bank 2011 containing patches from the second database, and a synthesis block 2012 textures. The input of each texture generation unit is connected to the input of the filtering unit 2008. The filtering unit 2008 performs low-pass or band-pass filtering according to step 702. The output of the filtering unit 2008 is connected to the first input of the projection coefficient calculation unit 2010. The output of the low-frequency texture bank 2009 is connected to the second input of the projection coefficient calculation unit 2010. The projection coefficient calculator 2010 calculates the projection coefficients as in step 1002. The output of the projection coefficient calculator 2010 is connected to the first input of the texture synthesis unit 2012 and the second output of the texture generation unit. The output of the high-frequency texture bank 2011 is connected to the second input of the texture synthesis unit 2012. The texture synthesis unit 2012 generates a texture, as in step 1003. The output of the texture synthesis unit 2012 is connected to the first output of the texture generation unit.

The texture adding unit 2003 contains a mixing control unit 2004, an adder 2005, and multipliers 2006 and 2007. The first output of the first texture generating unit 2001 is connected to the first input of the multiplier 2006. The first output of the second texture generating unit 2002 is connected to the first input of the multiplier 2007. Second output of the first block The texture generation 2001 is connected to the first input of the blend control unit 2004. The second output of the second texture generation unit 2002 is connected to the second input of the mixing control unit 2004. The first output of the mixing control unit 2004 is connected to the second input of the multiplier 2006. The second output of the mixing control unit 2004 is connected to the second input of the multiplier 2007. The output of the multiplier 2006 is connected to the first input of the adder 2005. The output of the multiplier 2007 is connected to the second input of the adder 2005. Video stream 2000 is fed to the third input of the adder 2005. The output of this adder is connected to the output of the system to obtain an ultra-high resolution image with mixing control based on projection coefficients. The mixing control unit 2004 decides the proportion of the texture generated by the first texture generation unit 2001 and the proportion of the texture generated by the second texture generation unit 2002, which is transmitted to the system output.

или

, где k₁ - некая заранее определенная константа,

или

, где b^k - коэффициенты проекции, вычисленные в блоке 2010 вычисления коэффициентов проекции первого блока 2001 генерации текстуры,

- патчи текстуры высокого разрешения из высокочастотной базы данных, хранящиеся в банке 2011 высокочастотных текстур первого блока 2001 генерации текстуры, N и M - некие положительные числа, и f(·) - некая неубывающая функция, принимающая положительные значения на правом полуинтервале. Пропорцию текстуры, сгенерированной вторым блоком 2002 генерации текстуры, которая передается на выход системы, можно вычислить как

или

, где k₂ - некая предварительно определенная константа,

или

, где b^k - коэффициенты проекции, вычисленные в блоке 2010 вычисления коэффициентов проекции второго блока 2002 генерации текстуры,

- патчи текстуры высокого разрешения из высокочастотной базы данных, хранящейся в банке 2011 высокочастотных текстур второго блока 2002 генерации текстуры, N и M - некие положительные числа, и f(·) - некая неубывающая функция, принимающая положительные значения на правом полуинтервале.The proportion of the texture generated by the first block 2001 of the texture generation, which is transmitted to the system output, can be calculated as

or

where k ₁ is a certain predetermined constant,

or

where b ^k are the projection coefficients calculated in block 2010 calculating the projection coefficients of the first texture generation block 2001,

- high-resolution texture patches from the high-frequency database stored in the 2011 high-frequency texture bank of the first block 2001 of texture generation, N and M are some positive numbers, and f (·) is some non-decreasing function that takes positive values on the right half-interval. The proportion of the texture generated by the second texture generation unit 2002, which is transmitted to the system output, can be calculated as

or

where k ₂ is a certain predetermined constant,

or

where b ^k are the projection coefficients calculated in block 2010 calculating the projection coefficients of the second texture generation block 2002,

- high-resolution texture patches from the high-frequency database stored in the 2011 high-frequency texture bank of the second texture generation unit 2002, N and M are some positive numbers, and f (·) is some non-decreasing function that takes positive values on the right half-interval.

, где D_ij - сумма двух Гауссовых функций

, где c₁, c₂, σ₁, σ₂, X₁, Y₁, X₂, Y₂ - псевдослучайные числа, зависящие от положения текущего блока пикселей на строке и в столбце, и i и j - положение пикселя на строке и в столбце внутри блока. Пример такой дополнительной структуры D данных показан на фиг. 21. Нормализацию в наилучшем варианте реализации выполняют как

, где

- некая предварительно определенная константа. Смешивание текстур, полученных в двух масштабах, осуществляется обратно пропорционально нормам текстуры, чтобы обеспечить равное воздействие текстур, сгенерированных в обоих масштабах, что можно описать формулой

, где

и

- прямоугольные блоки нормированных значений текстур, сгенерированных двумя масштабами алгоритма, соответственно.The best embodiment of the invention can be implemented in the form of a two-scale algorithm, as shown in Fig. 8. The texture generation for each block in the best implementation is according to FIG. 11, where the preprocessing 1102 pixels of each block are calculated as $$P_{ij}^{\text{''}}=P_{ij}\cdot D_{ij}$$

, where D _ij is the sum of two Gaussian functions

, where c ₁ , c ₂ , σ ₁ , σ ₂ , X ₁ , Y ₁ , X ₂ , Y ₂ are pseudorandom numbers depending on the position of the current block of pixels on the row and in the column, and i and j are the position of the pixel on the row and in the column inside the block. An example of such an additional data structure D is shown in FIG. 21. Normalization in the best embodiment is performed as

where

is a certain predefined constant. The mixing of textures obtained on two scales is inversely proportional to the texture norms to ensure equal effect of the textures generated at both scales, which can be described by the formula

where

and

- rectangular blocks of normalized texture values generated by two scales of the algorithm, respectively.

In FIG. 21 shows the simulation results with a block size of 51 × 51, non-overlapping blocks and the number of database records equal to 100.

The first and second databases containing low-frequency and high-frequency patches, respectively, applicable to the texture generation process described above, can be calculated as follows:

- for the reference grayscale image I calculate the low, medium and high frequency components I _low , I _middle , I _high as

I ′ = I⊗K ₁

I _low = I′K ₂

I _middle = I′-I _low

I _high = II ′,

where K ₁ is the smaller Gaussian smoothing core with a smaller standard deviation parameter, and K ₂ is the larger Gaussian smoothing core with a larger standard deviation parameter.

- Choose a set Ω of n random blocks of size p × q from the mid-frequency component I _middle and a set G of blocks from the same position from the high-frequency component I _high :

Ω = {S ₁ , S ₂ , ..., S _n }

G = {G ₁ , G ₂ , ..., G _n }

will be the initial set of mid-frequency components of n randomly selected patches of the same size p × q.

- Stretch each block of pixels S _k into a vector of p · q elements S ′ _k .

- Select m≤n vectors from the set Ω ′ = {S ′ ₁ , S ′ ₂ , ..., S ′ _n } according to the following procedure:

let U ₁ = S ′ ₁

find U ₂ = argmax _{S∈Ω \ {U1}} | SU ₁ |

|, где L_k-1 - линейное подпространство на U₁, U₂, …, U_k-1, и Pr_Lk-1(S) - проекция S на L_k-1.find all other vectors as U _k = argmax _{S∈Ω \ {U1, U2, ..., Uk-1} | $$S-{\mathrm{Pr}}_{L_{k-\mathrm{one}}}(S)$$

|, where L _k-1 is the linear subspace on U ₁ , U ₂ , ..., U _k-1 , and Pr _Lk-1 (S) is the projection of S onto L _k-1 .

, $$U_2=S_{i_2}$$

, …, $$U_m=S_{i_m}$$

. Тогда $$H=\left\{H_1=G_{i_1},H_2=G_{i_2},\mathrm{...,}{H}_m=G_{i_m}\right\}$$

будет набором текстур (высокочастотной базы данных), соответствующим U.- Let the sequence of indices i ₁ , i ₂ , ..., i _m be the numbers of the selected mid-frequency vectors from the previous stage: $$U_{\mathrm{one}}=S_{i_{\mathrm{one}}}$$

, $${\mathrm{<figure-callout\; id="2"\; label="U"\; filenames="00000071.png"\; state="\{\{state\}\}">U</figure-callout>}}_2={\mathrm{<figure-callout\; id="2"\; label="S\; i"\; filenames="00000071.png"\; state="\{\{state\}\}">S\; </figure-callout>}}_{i_{}}$$

, ..., $$U_m=S_{i_m}$$

. Then $$H=\left\{H_{\mathrm{one}}=G_{i_{\mathrm{one}}},{\mathrm{<figure-callout\; id="2"\; label="H"\; filenames="00000071.png"\; state="\{\{state\}\}">H</figure-callout>}}_2={\mathrm{<figure-callout\; id="2"\; label="G\; i"\; filenames="00000071.png"\; state="\{\{state\}\}">G\; </figure-callout>}}_{i_{}},\mathrm{...,}{H}_m=G_{i_m}\right\}$$

will be a texture set (high-frequency database) corresponding to U.

The algorithm can provide sufficient quality with a 51 × 51 block size and a database containing 1,500 or more patches, however, its implementation in the form of SoC (system on a chip) will be prohibitively expensive. To ensure the possibility of hardware implementation, we tested a circuit with a 16 × 16 block and a database of 50 patches. She demonstrated several types of defects; therefore, additional subalgorithms were developed to eliminate them.

вычисляют как разность между исходным пикселем P^ij и средним значением пикселя, стоящего на соответствующей диагонали блокаIn FIG. 23 shows a possible implementation with a 16 × 16 block including preprocessing and post-processing; here, the input image 2301 undergoes low-pass or band-pass filtering 2302. The filtered image is divided 2303 into non-overlapping or overlapping blocks. Each block is pre-processed 2304. Then, for each block, 2305 texture is generated and subjected to post-processing 2306, after which it is added to the original image 2307. The purpose of the pre-processing can be explained as follows: in the presence of pronounced edges, the texture generated for neighboring blocks is exactly or almost the same , which manifests itself in the form of a structural defect along the line (Fig. 24). In the case of a vertical edge, this defect can be eliminated by subtracting the corresponding average column value from each pixel of the block. In the case of a horizontal edge, this defect can be eliminated by subtracting the corresponding average row value from each pixel of the block. In the case of a diagonal edge, you can perform preliminary block processing: each pixel of the pre-processed block

calculated as the difference between the original pixel P ^ij and the average value of the pixel standing on the corresponding diagonal of the block

.

.

In FIG. 25 shows the results of pre-processing for blocks with pronounced vertical, horizontal, main diagonal, additional diagonal edges. It can be seen that the pixel values of the pre-processed blocks are much smaller than the values of the input block, which leads to a significantly lower power of the generated texture. To decide what type of pre-treatment should be used, the following operations can be performed:

1) Calculate the standard deviation S ₀ of the pixel values of the input block P

2) _{Preprocess the} target blocks with a horizontal edge and calculate the preprocessed block P _hor and the standard deviation of the pixel values in the processed block S _hor

3) Pre-process the target blocks with a vertical edge and calculate the pre-processed block P _ver and the standard deviation of the pixel values in the processed block S _ver

4) Pre-process the target blocks with the main diagonal edge and calculate the pre-processed block P _diag1 and the standard deviation of the pixel values in the processed block S _diag1

5) _{Preprocess the} target blocks with an additional diagonal edge and calculate the pre-processed block P _diag2 and the standard deviation of the pixel values in the processed block S _diag2

6) Choose the minimum value of standard deviations S _min = min (S _ver , S _hor , S _diag1 , S _diag2 )

, где7) Calculate the gain k =

where

f (x) =

=(1-k)·P+k·P_min.8) Calculate the final pixel values of the pre-processed block as

= (1-k) · P + k · P _min .

Operations (7) and (8) allow you to vary the degree of pre-processing depending on how high the severity of the edge.

Since there are other directions of the edges, as well as some complex high-contrast objects (such as subtitle letters), the protection techniques described above may not be enough. On Fig shows an example of an algorithm for obtaining ultra-high resolution, including additional protection. Before preprocessing 2605, this algorithm includes calculating 2604 block gain, which is multiplied by the generated texture before adding 2608 to the image. This coefficient can be calculated as a decreasing sawtooth function of the norm of the generated block texture.

, выше некоторого более высокого заданного порога, и норма текстуры, сгенерированной для некоторого соседнего блока n_i=

, i=1 … 7, ниже некоторого более низкого заданного порога, для каждого пикселя с координатами (x, y) текущего блока вычисляется попиксельный коэффициент усиления как q_xy=min(q_xy,

), если абсолютное отличие текущего пикселя от медианного значения значений пикселей в блоке I_i меньше, чем первый заданный порог, или абсолютное отличие текущего пикселя от среднего значения значений пикселей в блоке I_i меньше второго заданного порога. Эту процедуру можно выполнять итеративно для всех соседних блоков, при допущении, что на первой итерации q_xy=1. На фиг. 31 показаны увеличенный фрагмент изображения, содержащий окрестность 4×4 блоков пикселей размером 16×16 (a), первоначально сгенерированная текстура с видимыми блочными дефектами (b), и результат попиксельной обработки, описанной выше (c). На фиг.32 представлено полное изображение текстуры, полученной (a) без и (b) с попиксельной защитой.To reduce the number of patches in the database, an implementation option was developed with a database distributed over a larger number of scales. On Fig shows the results obtained with three scales (database of 10, 20 and 20, respectively) of different roughness. You can notice that the second and third scales create a lot of noise in a flat area (sky, clouds), and the first does not. Therefore, the first scale can be used to control gain at other scales. If the norm of the texture generated by the first scale is small, the amplitude of the texture generated by two other scales should be reduced, for example, by multiplying by a certain decreasing function the texture norm generated by the first scale in the current block. In FIG. 28 shows a system architecture using this kind of merge logic. Fig. 29 shows the difference between the results obtained using a simple summation of the texture generated at three scales (a) and the texture obtained using the merge logic described above (b). In FIG. Figure 29-b shows that in blocks with a non-empty texture that have adjacent blocks with an empty texture, block defects appear. To avoid this problem, pixel-by-pixel protection logic is used, based on the analysis of the texture from blocks adjacent to the current block T ₀ and the original color image from blocks adjacent to the current block I ₀ (Fig. 30). If the norm of the texture generated for the current block is n ₀ =

, above some higher given threshold, and the norm of the texture generated for some neighboring block n _i =

, i = 1 ... 7, below some lower given threshold, for each pixel with coordinates (x, y) of the current block, the pixel-by-pixel gain is calculated as q _xy = min (q _xy ,

), if the absolute difference of the current pixel from the median value of the pixel values in the block I _{i is} less than the first specified threshold, or the absolute difference of the current pixel from the average value of the pixel values in the block I _{i is} less than the second specified threshold. This procedure can be performed iteratively for all neighboring blocks, assuming q _xy = 1 at the first iteration. In FIG. 31 shows an enlarged fragment of an image containing a neighborhood of 4 × 4 pixel blocks of 16 × 16 (a), the initially generated texture with visible block defects (b), and the result of the pixel processing described above (c). On Fig presents a complete image of the texture obtained (a) without and (b) with pixel-by-pixel protection.

In some cases, as shown in FIG. 33- (a), where there are a large number of long thin objects on a flat background, the protection logic described above is not enough, FIG. 33- (b). For this case, protection logic based on segmentation is proposed:

1) For each large block (128 × 128 pixels or more) color-based segmentation is performed into k classes (we used three or five)

сгенерированной текстуры 2) For each k-th class obtained by segmentation, get the average absolute value

generated texture

соответствующего сегмента в случае, если |t|>

и r=

,3) For each pixel inside the block, the ratio r between the absolute value of the generated texture | t | and average absolute value

corresponding segment if | t |>

and r =

,

or

4) if this ratio is very large or very small, they reduce the texture index in the current pixel or keep it as it is. This dependence can be realized as multiplication by a piecewise linear function of the relation r:

.

.

In FIG. 33-c shows the resulting texture obtained after this procedure. The segmentation of step (1) can be performed using the k-means method (Fig. 33-d), maximizing expectations (Fig. 33-e), or another segmentation algorithm.

Although the use of an overlapping arrangement of blocks (Fig. 34) did not show any advantages in terms of quality, sometimes it may be necessary to implement it for sharing hardware if it is used in conjunction with some other way to improve video quality, which requires this kind of operation. In FIG. 34-a shows an overlapping arrangement of 9 adjacent overlapping blocks, and in FIG. 34-b shows the patch size from the first database corresponding to the location in FIG. 34-a. The second database can be configured to fill only the non-overlapping part of the input block, as in FIG. 34-c, or completely filling the input image block, as in FIG. 34-d. In the latter case, the texture generated for the neighboring block can be joined with an overlapping area with some mixing function (such as Gaussian), which takes on large values at the center of the block and lower values at its periphery. The texture generated in adjacent overlapping blocks 3501 and 3502 can also be stitched along the minimum energy path 3503, as shown in FIG. 35. This path can be found, for example, using dynamic programming and to provide a seamless connection of textures from neighboring blocks.

As seen in FIG. 36-37, it is also possible to use blocks of non-rectangular shape or even fragmentation of the image into blocks of two or more different shapes and / or sizes.

Industrial applicability

The proposed method can be applied in household electronic devices that increase the resolution of the image, such as television sets, DVD / BD players and game consoles, etc.

Claims (48)

1. The method of image processing using the first and second database of patches, and the number of patches in the first database is equal to the number of patches in the second database, the method comprising the steps of:
- carry out low-pass or band-pass filtering of the original image;
- divide the filtered image into blocks of the same size as the size of the patches in the first database of patches;
- generate a texture for each block, performing the following steps:
- perform pre-processing;
- for each patch in the second database, projection coefficients are calculated as the sum of the product of pixels inside the block of the pre-processed image and the corresponding pixel values of the patch from the first database;
- calculate the pixels of the output texture block as the sum of the products of the projection coefficients and patches from the second database;
- perform texture normalization;
- carry out post-processing of the generated texture; and
- add a post-processed texture to the original pixel values inside the block. 2. The method according to claim 1, in which the first database contains an orthonormal basis of patches obtained from the training patches of the filtered training image, and the second database contains an orthonormal basis of patches obtained from the training patches of the texture image obtained by subtracting the filtered training image from the training Images. 3. The method of claim 1, wherein the pre-processing comprises removing a trend, including subtracting the average value of the block brightness or subtracting a linear or quadratic regression model describing the input patch mentioned here. 4. The method according to p. 1, in which the preliminary processing contains the removal of the trend, containing stages in which:
- calculate the first candidate pre-processed block by subtracting from each pixel the average row value;
- calculate the second candidate pre-processed block by subtracting from each pixel the average column value;
- calculate the third candidate pre-processed block by subtracting from each pixel the average value of pixels lying on the same main diagonal;
- calculate the fourth candidate pre-processed block by subtracting from each pixel the average value of pixels lying on the same additional diagonal;
- choose from the first, second, third and fourth candidates a pre-processed block of the candidate having the smallest discrepancy or standard deviation, as the target pre-processed block;
- calculate the relationship between the discrepancy or standard deviation of the target pre-processed block and the discrepancy or standard deviation of the source block;
- calculate the mixing coefficient as a decreasing function of this ratio, which takes a value between unity and zero;
- calculate the final values of the pre-processed pixels as the sum of the product of the mixing coefficient and the pixels of the target pre-processed block and the product "one minus the mixing coefficient" and the pixels of the original block. 5. The method of claim 1, wherein the pre-processed pixel as part of the pre-processing is calculated as a sum or a product with a pixel from some additional data structure having the same size as the image. 6. The method of claim 5, wherein the additional data structure is formed as a smooth function of several random or pseudo-random arguments. 7. The method of claim 5, wherein the elements of the additional data structure are formed as

,
where x, y is the position of the pixel on the row and in the column inside the image, p is the block width, q is the block height and c ₁ , c ₂ , σ ₁ , σ ₂ , X ₁ , Y ₁ , X ₂ , Y ₂ are pseudo-random numbers depending on the position on the floor (y / q) row and the position in the floor (x / p) column of the current block of pixels. 8. The method according to p. 1, in which the normalization of the texture is calculated as

or

, where f (·) is a certain non-decreasing function that takes positive values on the right half-interval, and R _ij are the pixel values of the texture. 9. The method according to p. 1, in which the normalization of the texture is calculated as

or

where

or

, where N and M are some positive numbers and f (·) is a certain non-decreasing function that takes positive values on the right half-interval, b ^k are the projection coefficients, and T ^k are the high-frequency patches from the high-frequency database. 10. The method of claim 1, wherein the post-processing of the generated texture comprises the steps of:
- set the pixel-by-pixel gain per unit for each pixel in the current block;
- for each block, the norm of the generated texture and the norm of the texture generated in adjacent blocks are calculated: upper left, upper, upper right, left, right, lower left, lower right and lower;
- for each neighboring block, which has a texture norm below a predetermined threshold, the gain in the current blocks is updated, containing the stages in which:
- calculate the average brightness value of the neighboring block of the input image;
- calculate the median brightness value of the neighboring block of the input image;
- for each pixel of the current block, if it has an absolute difference from the average brightness value below the first predefined threshold or an absolute difference from the median brightness value below the second predefined threshold, the pixel-by-pixel gain is re-calculated as the minimum value of the previous pixel-by-pixel gain and the relationship between the texture norm of the corresponding neighboring block and the texture norm of the source block;
- recalculate each texture pixel in the current block by multiplying by the corresponding pixel-by-pixel gain. 11. The method according to p. 1, in which the post-processing of the generated texture contains the steps in which:
- set the pixel-by-pixel gain for each pixel in the current block to unity;
- for each block, the norm of the generated texture and the norm of the texture generated in adjacent blocks are calculated: upper left, upper, upper right, left, right, lower left, lower right and lower;
- for each neighboring block, which has a texture norm below a predetermined threshold, the gain in the current block is updated, containing the steps in which:
- calculate the average values of the neighboring block of the input image in each color channel;
- calculate the median values of the neighboring input image block in each color channel;
- for each pixel of the current block, the first test value is calculated as a certain monotonically increasing function of the three arguments of the absolute values of the differences of the pixel values of the input image and the corresponding average values from the neighboring block in each color channel;
- for each pixel of the current block, the second test value is calculated as a certain monotonically increasing function of the three arguments of the absolute values of the differences in the pixel values of the input image and the corresponding median values from the neighboring block in each color channel;
- for each pixel of the current block, if its first test value is lower than the first predefined threshold or the second test value is lower than the second predefined threshold, the pixel-by-pixel gain is calculated again as the minimum value of the previous value of the pixel-by-pixel gain and the relationship between the texture norms of the corresponding neighboring block and the norm source block textures;
- recalculate each texture pixel in the current block by multiplying by the corresponding pixel-by-pixel gain. 12. The method according to p. 1, in which the post-processing of the generated texture contains the steps in which:
- segmentation into k classes is performed for each block;
- for each class, the average absolute value of the generated texture is calculated;
- for each pixel inside the block, the relation between the average absolute value of the generated texture in some small neighborhood and the average absolute value of the corresponding segment is calculated;
- calculate the pixel-by-pixel gain, which takes a zero or very small value if the ratio is very large, and a unit value if this ratio is less than unity or very close to it;
- recalculate each texture pixel in the current block by multiplying by the corresponding pixel-by-pixel gain. 13. The method of claim 1, wherein an overlapping arrangement of blocks is used. 14. The method of claim 13, wherein the second database consists of patches filling in only the non-overlapping portion of the input block. 15. The method according to p. 13, in which the texture generated for neighboring blocks is multiplied by a certain function of mixing the coordinates of the pixels inside the block. 16. The method according to p. 13, in which the texture generated for the neighboring blocks is stitched along the path of minimum energy. 17. The method of claim 1, wherein the blocks are rectangular in shape. 18. The method according to p. 1, in which all the blocks have the same shape and form a flat mosaic fragmentation. 19. The method according to claim 1, in which the input image is divided into blocks of two or more different shapes and / or sizes, and for each shape and / or block size, separate first and second databases are stored. 20. An image processing method using the first and second database of patches, the number of patches in the first database being equal to the number of patches in the second database, the method comprising the steps of:
- carry out low-pass or band-pass filtering of the input image;
- divide the filtered image into blocks of the same size as the size of the patches in the first database of patches;
- generate a texture for each block, performing the following steps:
- perform pre-processing;
- for each patch in the second database, projection coefficients are calculated as the sum of the product of pixels inside the block of the pre-processed image and the corresponding pixel values of the patch from the first database;
- calculate the pixels of the output texture block as the sum of the projection coefficients and patches from the second database;
- perform texture normalization;
- interpolating the input image using the method of two-dimensional interpolation;
- perform post-processing of the generated normalized texture; and
- create an ultra-high resolution image by adding a post-processed texture to the interpolated image. 21. The method according to p. 20, in which the patch size of the first database is equal to the size of the patch of the second database, and
interpolating the input image comprises interpolating the input image to a size equal to the size of the required ultra-high resolution. 22. The method according to p. 20, in which the horizontal patch size from the first database is m times smaller than the horizontal patch size from the second database, and the vertical patch size from the first database is n times smaller than the vertical patch size from the second databases, moreover
- the interpolation of the input image contains the interpolation of each block of the input image horizontally m times and vertically n times more;
- adding a post-processed texture to the interpolated image comprises adding a post-processed texture to each interpolated block of the input image; and
- the ultra-high resolution image has a horizontal size m times larger than the input image, and a vertical size n times larger than the input image. 23. A method for multiscale image processing using at least two scales, each of which uses the first and second database of patches, the number of patches in the first database being equal to the number of patches in the second database, the method comprising the steps of:
- create a texture layer for each scale by performing the following steps:
- carry out low-pass or band-pass filtering of the input image;
- divide the filtered image into blocks of the same shape and size as the shape and size of the patches in the first database of a given scale;
- generate a texture for each block, performing the following steps:
- perform pre-processing;
- for each patch in the second database, projection coefficients are calculated as the sum of the product of pixels inside the block of the pre-processed image and the corresponding pixel values of the patch from the first database;
- calculate the pixels of the output texture block as the sum of the projection coefficients and patches from the second database;
- perform texture normalization;
- make up the resulting texture from texture layers generated for multiple scales;
- carry out post-processing of the resulting texture; and
- add a post-processed texture to the original pixel values inside the block. 24. The method according to p. 23, in which the first database of each scale contains an orthonormal basis of patches obtained from the training patches of the filtered training image, and the second database of each scale contains an orthonormal basis of patches obtained from the training patches of a texture image obtained by subtracting the filtered training images from the training image, and the filters used to obtain the filtered training image at each scale are some low-frequency Different and bandpass filters, which may be different for each scale. 25. The method of claim 23, wherein the preprocessing comprises removing a trend, including subtracting the average brightness of the block or subtracting a linear or quadratic regression model describing the patch introduced here. 26. The method according to p. 23, in which the preliminary processing includes removing the trend, containing stages in which:
- calculate the first candidate pre-processed block by subtracting from each pixel the average row value;
- calculate the second candidate pre-processed block by subtracting from each pixel the average column value;
- calculate the third candidate pre-processed block by subtracting from each pixel the average value of pixels lying on the same main diagonal;
- calculate the fourth candidate pre-processed block by subtracting from each pixel the average value of pixels lying on the same additional diagonal;
- choose from the first, second, third and fourth candidates a pre-processed block of the candidate having the smallest discrepancy or standard deviation, as the target pre-processed block;
- calculate the relationship between the discrepancy or standard deviation of the target pre-processed block and the discrepancy or standard deviation of the source block;
- calculate the mixing coefficient as a decreasing function of this ratio, which takes a value between unity and zero;
- calculate the final values of the pre-processed pixels as the sum of the product of the mixing coefficient and the pixels of the target pre-processed block and the product "one minus the mixing coefficient" and the pixels of the original block. 27. The method according to p. 23, in which during pre-processing, the pre-processed pixel is calculated as the sum or product of the pixel from some additional data structure having the same size as the image. 28. The method according to p. 27, in which an additional data structure is formed as a smooth function of several random or pseudo-random arguments. 29. The method of claim 27, wherein the elements of the additional data structure are formed as

,
where x, y is the position of the pixel on the row and in the column inside the image, p is the block width, q is the block height and c ₁ , c ₂ , σ ₁ , σ ₂ , X ₁ , Y ₁ , X ₂ , Y ₂ are pseudo-random numbers depending on the position on the floor (y / q) row and the position in the floor (x / p) column of the current block of pixels. 30. The method according to p. 23, in which the normalization of the texture is calculated as

or

, where f (·) is a certain non-decreasing function that takes positive values on the right half-interval, and R _ij are the pixel values of the texture. 31. The method according to p. 23, in which the normalization of the texture is calculated as

or

where

or

, where N and M are some positive numbers and f (·) is a certain non-decreasing function that takes positive values on the right half-interval, b ^k are the projection coefficients, and T ^k are the high-frequency patches from the high-frequency database. 32. The method according to p. 23, in which when compiling the resulting texture from the texture layers generated for at least two scales, calculate the weighted sum of texture pixels calculated at each scale. 33. The method according to p. 23, in which the compilation of the resulting texture from the texture layers generated for at least two scales, comprises the steps of:
- calculate for each block the norm of the generated texture at the smallest scale, i.e. a scale that has band-pass or low-pass filtering that maintains the highest frequency range among all other scales;
- calculate the gain as a certain monotonous function of the norm of the texture block at the smallest scale, which takes a small or zero value when the texture norm is below the threshold, then gradually increases and takes on some larger predefined value if the texture norm is above a predetermined threshold;
- recalculate the texture layers at each scale, except for the smallest, by multiplying each block of the texture by the gain. 34. The method of claim 23, wherein the post-processing of the generated texture comprises the steps of:
- set the pixel-by-pixel gain for each pixel in the current block to unity;
- for each block, the norm of the generated texture and the norm of the texture generated in adjacent blocks are calculated: upper left, upper, upper right, left, right, lower left, lower right and lower;
- for each neighboring block, which has a texture norm below a predetermined threshold, the gain in the current blocks is updated, containing the stages in which:
- calculate the average brightness value of the neighboring block of the input image;
- calculate the median brightness value of the neighboring block of the input image;
- for each pixel of the current block, if it has an absolute difference from the average brightness value below the first predefined threshold or an absolute difference from the median brightness value below the second predefined threshold, the pixel-by-pixel gain is re-calculated as the minimum value of the previous pixel-by-pixel gain and the relationship between the texture norm of the corresponding neighboring block and the texture norm of the source block;
- recalculate each texture pixel in the current block by multiplying by the corresponding pixel-by-pixel gain. 35. The method of claim 23, wherein the post-processing of the generated texture comprises the steps of:
- set the pixel-by-pixel gain for each pixel in the current block to unity;
- for each block, the norm of the generated texture and the norm of the texture generated in adjacent blocks are calculated: upper left, upper, upper right, left, right, lower left, lower right and lower;
- for each neighboring block, which has a texture norm below a predetermined threshold, the gain in the current block is updated, containing the steps in which:
- calculate the average values of the neighboring block of the input image in each color channel;
- calculate the median values of the neighboring input image block in each color channel;
- for each pixel of the current block, the first test value is calculated as a certain monotonically increasing function of the three arguments of the absolute values of the differences of the pixel values of the input image and the corresponding average values from the neighboring block in each color channel;
- for each pixel of the current block, the second test value is calculated as a certain monotonically increasing function of the three arguments of the absolute values of the differences in the pixel values of the input image and the corresponding median values from the neighboring block in each color channel;
- for each pixel of the current block, if its first test value is lower than the first predefined threshold or the second test value is lower than the second predefined threshold, the pixel-by-pixel gain is calculated again as the minimum value of the previous pixel-by-pixel gain and the relationship between the texture norm of the corresponding neighboring block and the norm source block textures;
- recalculate each texture pixel in the current block by multiplying by the corresponding pixel-by-pixel gain. 36. The method according to p. 23, in which the post-processing of the generated texture contains the steps in which:
- segmentation into k classes is performed for each block;
- for each class, the average absolute value of the generated texture is calculated;
- for each pixel inside the block, the relation between the average absolute value of the generated texture in some small neighborhood and the average absolute value of the corresponding segment is calculated;
- calculate the pixel-by-pixel gain, which takes a zero or very small value if the ratio is very large, and a unit value if this ratio is less than unity or very close to it;
- recalculate each texture pixel in the current block by multiplying by the corresponding pixel-by-pixel gain. 37. The method of claim 23, wherein the overlapping arrangement of the blocks is used. 38. The method according to p. 37, in which the second database consists of patches that fill only the non-overlapping part of the input block. 39. The method according to p. 37, in which the texture generated for neighboring blocks is multiplied by a certain function of mixing the coordinates of the pixels inside the block. 40. The method according to p. 37, in which the texture generated for neighboring blocks, stitched along the path of minimum energy. 41. The method according to p. 23, in which the blocks are rectangular in shape. 42. The method according to p. 23, in which all the blocks have the same shape and form a flat mosaic fragmentation. 43. The method of claim 23, wherein the input image is fragmented into blocks of two or more different shapes and / or sizes and for each shape and / or block size, separate first and second databases are stored. 44. A method for multiscale image processing using at least two scales, each of which uses the first and second database of patches, the number of patches in the first database being equal to the number of patches in the second database, the method comprising the steps of:
- create a texture layer for each scale by performing the following steps:
- carry out low-pass or band-pass filtering of the input image;
- divide the filtered image into blocks of the same shape and size as the shape and size of the patches in the first database of patches of a given scale;
- generate a texture for each block, performing the following steps:
- pre-processing for each patch in the second database is performed, calculating the projection coefficients as the sum of the products of pixels inside the block of the pre-processed image and the corresponding pixel values of the patch from the first database;
- calculate the pixels of the output texture block as the sum of the projection coefficients and patches from the second database;
- perform texture normalization;
- interpolate the input image using the method of two-dimensional interpolation;
- make up the resulting texture from texture layers generated for at least two scales;
- perform post-processing of the resulting texture;
- create an ultra-high resolution image by adding a post-processed texture to the interpolated image. 45. The method of claim 44, wherein the patch size of the first database is equal to the patch size of the second database, wherein
- interpolation of the input image contains interpolation of the input image to a size equal to the size of the desired image ultra-high resolution. 46. The method according to p. 44, in which
- the interpolation of the input image contains the interpolation of each block of the input image horizontally m times and vertically n times more;
- adding a post-processed texture to the interpolated image comprises adding a post-processed texture to each interpolated block of the input image; and
- the ultra-high resolution image has a horizontal size m times larger than the input image, and a vertical size n times larger than the input image. 47. An image processing system comprising: a filtering unit, a projection coefficient calculation unit, a low-frequency texture bank, a high-frequency texture bank and a texture synthesis unit, the system input being connected to the input of the filtering unit, the output of the filtering unit connected to the first input of the projection coefficient calculating unit, the output of the low-frequency texture bank is connected to the second input of the projection coefficient calculation unit, the output of the projection coefficient calculation unit is connected to the first input of the texture synthesis unit, output ankh high texture connected to the second input of the synthesis of the block texture, texture synthesis unit output is fed to the system output. 48. The system of claim 47, wherein the filtering unit is configured to perform low-pass or band-pass filtering; the low-frequency texture bank is configured to store texture patches from the first database; a high-frequency texture bank is configured to store texture patches from a second database; the projection coefficient calculation unit is configured to calculate projection coefficients as

where P _ij are the pixel values and

- pixels of patches from the first database; the texture synthesis unit is configured to calculate texture pixel values as

where

- pixels of patches from the second database, and performing normalization of textures.

Images