KR102452881B1 - Apparatus of processing image and method thereof - Google Patents

Abstract

The present invention relates to an image processing device and method. The image processing device comprises: an input unit for obtaining at least one original image; and a processor for adding noise to at least one still image corresponding to the at least one original image, generating a moving image by combining the at least one still image, performing denoising processing on at least one frame of the moving image, and obtaining the moving image corresponding to the at least one still image based on a denoising process result for at least one frame.

Description

Image processing apparatus and method {APPARATUS OF PROCESSING IMAGE AND METHOD THEREOF}

It relates to an image processing apparatus and method.

In the past, it was common for comics (comics) to be delivered to readers through a publishing process that was printed on a physical medium such as paper and then distributed. Recently, according to the development of information and communication technology, cartoons are distributed based on websites or e-books, and it is becoming common for readers to access the website using a terminal or download an e-book to view and read the distributed cartoons. Recently, the production of Moving Toons, in which characters or main objects in cartoons move like a video, is also active. Moving toons are usually inserted into cartoons consisting of a large number of static images, for example, webtoons, and provide additional dynamic images and sound effects to the readers who read the cartoons, so that readers can enjoy cartoons more vividly. make it possible However, the moving toon not only requires a large number of manpower and cost in its production, but also has a relatively larger size than each cut of the cartoon, so that the image quality is relatively easy to deteriorate compared to the original cartoon. Such low-quality moving toon did not arouse interest in readers, but on the contrary, it also caused the adverse effect of lowering the concentration of the readers on the cartoon.

Republic of Korea Patent Publication No. 10-1791573 ("Super-resolution apparatus and method for video stream using convolutional neural network", Sogang University Industry-University Cooperation Foundation, October 31, 2017) Republic of Korea Patent Publication No. 10-2021-0019780 ("Image processing system and method", LG Display Co., Ltd., 2021.02.23) Japanese Unexamined Patent Publication No. 2011-180798 ("Image processing apparatus and image processing method and program", SONY CORP, 2011.09.15)

An object of the present invention is to provide an image processing apparatus and method capable of generating a high-quality moving image by performing combination and super-resolution on at least one still image.

In order to solve the above problems, an image processing apparatus and method are provided.

The image processing apparatus includes an input unit for acquiring at least one original image, adding noise to at least one still image corresponding to the at least one original image, combining the at least one still image to generate a moving picture, and and a processor configured to perform denoising processing on at least one frame of a moving image, and obtaining a moving image corresponding to the at least one still image based on a denoising processing result for the at least one frame.

The at least one still image may include a cropped still image obtained by cropping a portion of the at least one original image.

The at least one still image may include an extended still image in which a size of the cropped still image is expanded.

The processor is configured to detect a difference between a pixel of a first frame and a pixel of a second frame of the moving picture, wherein a difference exists between a pixel of the first frame and a pixel of the second frame, and the pixel of the second frame If the pixel value of has a value corresponding to black or white, the denoising process is performed by replacing the pixel of the second frame with the pixel value of the corresponding pixel of the first frame to obtain a first denoising image You may.

The processor may obtain a second denoising image by applying at least one filter to the first denoising image.

The processor removes chroma-based grain noise of a very small pixel size from at least one frame among all frames of the first denoised video, or a pixel value in which noise is uniformly present every predetermined number of frames It is also possible to obtain the second denoising image by substituting a pixel value or similar value around the pixel, setting a block value for each frame in the image, or correcting a partially changed color gamut value in the image.

The processor may obtain a final denoising image corresponding to the second denoising image by using a learning algorithm.

The noise added to the at least one still image may include grain noise.

The image processing method includes: obtaining at least one original image; adding noise to at least one still image corresponding to the at least one original image; and generating a moving image by combining the at least one still image. , performing denoising processing on at least one frame of the moving image, and obtaining a moving image corresponding to the at least one still image based on a result of denoising processing for at least one frame. .

The at least one still image may include a cropped still image obtained by cropping a portion of the at least one original image.

The at least one still image may include an extended still image in which a size of the cropped still image is expanded.

In the image processing method, the performing the denoising process on at least one frame of the moving picture includes: detecting a difference between a pixel of a first frame and a pixel of a second frame of the moving picture; and a pixel of the first frame and if there is a difference between the pixels of the second frame and the pixel value of the pixel of the second frame has a value corresponding to black or white, the pixel of the second frame is set to the corresponding pixel of the first frame. The method may include obtaining a first denoising image by substituting a pixel value of .

The denoising may further include obtaining a second denoising image by applying at least one filter to the first denoising image.

The step of obtaining a second denoised image by applying at least one filter to the first denoised image may include: a chroma series having a very small pixel size for at least one frame among all frames of the first denoised moving image. removing grain noise, replacing the value of a pixel in which noise is uniformly in every predetermined number of frames among all frames of the first denoised video with a pixel value or similar value around the pixel, and a block value The method may include at least one of setting for each frame in the image and correcting a partially changed color gamut value in the image.

The denoising may further include obtaining a final denoising image corresponding to the second denoising image using a learning algorithm.

The noise added to the at least one still image may include grain noise.

According to the above-described image processing apparatus and method, it is possible to generate a high-quality moving image corresponding to the still image by performing combination and super-resolution on at least one still image.

According to the image processing apparatus and method described above, by using a still image of a low resolution, for example, SD (Standard Definition) level, a corresponding high resolution, for example, UHD (Ultra High Definition) level still image or video can be obtained. effect can be obtained.

According to the above-described image processing apparatus and method, it is possible to generate a high-quality image more similar to the original image while having relatively less noise than when the original image is simply scaled-up. can

According to the above-described image processing apparatus and method, there is also an advantage in that it is possible to improve the convenience of generating a high-quality video using a cartoon or a comic including a series of still images.

1 is a block diagram of an embodiment of an image processing apparatus.
2 is a view for explaining an example of the operation of the crop unit.
3 is a view for explaining an example of the operation of the size adjustment unit.
4 is a block diagram of an embodiment of a denoising unit.
5 is a diagram for explaining an example of an image before and after adding noise.
6 is a view for explaining an example of the operation of the image combining unit.
7 is a diagram for explaining an example of the operation of the first noise processing unit.
8 is a diagram for explaining an example of an image according to a learning processing result.
9 is a diagram illustrating a peak signal-to-noise ratio (PSNR) between an output result of an image processing apparatus and a simple scale-up result.
10 is a diagram illustrating a Structural Similarity Index Map (SSIM) between an output result of an image processing apparatus and a simple scale-up result.
11 is a flowchart of an image processing method according to an embodiment.

In the following specification, the same reference numerals refer to the same components unless otherwise specified. The term added with 'unit' used below may be implemented in software and/or hardware, and according to an embodiment, one 'unit' may be implemented as one physical or logical part, or a plurality of 'units' may be implemented as one physical or logical part. It may be implemented as one physical or logical part, or one 'unit' may be implemented with a plurality of physical or logical parts. Throughout the specification, when a certain part is connected to another part, it may mean that a certain part and another part are physically connected to each other and/or electrically connected to each other. In addition, when it is said that a part includes another part, it does not exclude another part other than the other part unless otherwise stated, and it means that another part may be further included according to the designer's choice. do. Expressions such as first to Nth (N is a natural number equal to or greater than 1) are for distinguishing at least one part(s) from other part(s), and unless otherwise specified, it does not necessarily mean that they are sequential. Also, the singular expression may include the plural expression unless the context clearly dictates otherwise.

Hereinafter, an embodiment of an image processing apparatus will be described with reference to FIGS. 1 to 10 .

1 is a block diagram of an embodiment of an image processing apparatus.

Referring to FIG. 1 , the image processing apparatus 10 may include a processor 100 provided to perform image processing according to an embodiment, and includes an input unit 11 , a storage unit 15 , and At least one of the output units 19 may be further included.

The input unit 11 receives and obtains one or more images (90, hereinafter, the original image), and transmits the acquired one or more original images 90 to at least one of the storage unit 15 and the processor 100 . have. According to an embodiment, one or more of the original images 90 may include a still image, and more specifically, for example, a live-action photograph, a design, or a cartoon (comics, cartoons or webtoons may be included). and the like. Here, the cartoon may include at least one cut (scene) included in a comic, a cartoon, or a webtoon. Also, two or more original images 90 may be temporally or historically correlated. For example, the two or more original images 90 may include two or more cuts continuously or discontinuously arranged in one cartoon. The input unit 11 may obtain the original image 90 by directly operating the input unit 11 by a user or a designer (hereinafter referred to as a user, etc.) . The input unit 90 may be provided integrally with the image processing apparatus 10 or may be provided to be physically detachable. In addition, the input unit 90 may receive various setting values necessary for the operation of the processor 100 from the user. The input unit 90 may include, for example, a keyboard, a mouse, a tablet input device, a touch screen, a touch pad, a track ball, a track pad, a scanner device, an image capturing module, and/or a microphone. In addition, according to an embodiment, the input unit 90 is a data input/output terminal capable of receiving data from another external device (eg, a portable memory device, etc.) or a communication module connected to another external device through a wired/wireless communication network. (For example, a LAN card, a short-range communication module, a mobile communication module, etc.) and the like may be included. Here, the short-range communication module may include a communication module used to access a short-range communication network, and the short-distance communication network includes, for example, Wi-Fi, Wi-Fi Direct, and Bluetooth. (Bluetooth), Bluetooth Low Energy, CAN communication, Ultra-WideBand (UWB) communication, NFC (Near Field Communication), RFID (Radio-Frequency IDentification) and / or a network implemented based on zigbee communication or the like may be included.

The storage unit 15 temporarily stores a result (eg, the final image 98 ) required for the operation of the processor 100 , generated during the operation of the processor 100 , or finally obtained by the processor 100 . Alternatively, it may be stored non-temporarily. Also, the storage unit 15 may store at least one program (which may be referred to as an app, an application, or software) or various setting values necessary for the at least one program. Here, the program stored in the storage unit 15 may be directly written or modified by a designer such as a programmer and then stored in the storage unit 15, or another physical recording medium (eg, an external memory device or a compact disk (CD)). ), etc.), and/or may be acquired or updated through an electronic software distribution network accessible through a wired/wireless communication network. The storage unit 15 may include, for example, at least one of a main memory device and an auxiliary memory device according to an embodiment. The main memory device may be implemented using a semiconductor storage medium such as ROM and/or RAM. Auxiliary storage devices include flash memory devices (SSD, Solid State Drive, etc.), SD (Secure Digital) card, hard disk drive (HDD, Hard Disc Drive), compact disc, DVD, or laser disc. It may be implemented using at least one storage medium capable of permanently or semi-permanently storing data, such as.

The output unit 19 includes information on whether or not the processor 100 has processed, the processing result of the processor 100 (eg, the final image 98), the image(s) acquired in the processing of the processor 100, The original image 90 acquired by the input unit 11 and/or other data or setting values stored in the storage unit 15, etc. are output to the outside and provided to the user or other device (not shown. For example, a smartphone or tablet PC, etc.). According to an embodiment, the output unit 19 may be provided integrally with the image processing apparatus 10 or may be provided to be physically separated. The output unit 19 may include, for example, a display, a printer device, a speaker device, an image output terminal, a data input/output terminal, and/or a communication module.

The processor 100 may acquire the final image 98 corresponding to the at least one original image 90 by using the at least one original image 90 . Here, the at least one original image 90 includes at least one still image, and the obtained final image 98 uses each still image obtained according to the processing result for each still image 90 as a frame. You can also include built-in videos. In this case, the processor 100 may generate the final image 98 based on the at least one original image 90 by operating according to an operation on the input unit 11 such as a user or a predefined setting, For this purpose, the program stored in the storage unit 15 may be executed. The processor 100 is, for example, a central processing unit (CPU), a graphic processing unit (GPU), a micro controller unit (MCU), an application processor (AP). ), an Electronic Controlling Unit (ECU), a Baseboard Management Controller (BMC), a Micro Processor (Micom), and/or at least one electronic device capable of performing various calculations and control processing. devices and the like. These processing or control devices may be implemented by using, for example, one or more semiconductor chips, circuits, or related components alone or in combination.

The processor 100, according to an embodiment, may include a crop unit 110 , a size adjuster 120 , a denoising unit 130 , and a final image generation unit 140 as shown in FIG. 1 . can At least two of the crop unit 110 , the size adjuster 120 , the denoising unit 130 , and the final image generator 140 may be logically separated or physically separated. When logically separated, at least two of the crop unit 110 , the size adjuster 120 , the denoising unit 130 , and the final image generation unit 140 use one processing unit (eg, a central processing unit). In the case of being physically separated, at least two of the crop unit 110 , the size adjuster 120 , the denoising unit 130 , and the final image generation unit 140 are different processing units (eg, It can be implemented using a central processing unit, a graphic processing unit, etc.). At least one of the cropping unit 110 , the size adjusting unit 120 , the denoising unit 130 , and the final image generating unit 140 may be omitted if necessary. For example, the cropping unit 110 or the size adjusting unit 120 may be omitted, and accordingly, cropping or size expansion may also be omitted.

2 is a view for explaining an example of the operation of the crop unit.

As shown in FIG. 2 , the crop unit 110 crops a portion of at least one original image 90 , and a still image 91 (hereinafter referred to as cropping) of a predetermined ratio corresponding to the at least one original image 90 . still images) can be obtained. Here, the predetermined ratio may be preset by a user or the like, and may include, for example, 4:3, 16:9, or 21:9. According to an exemplary embodiment, the crop unit 110 may crop all the input original images 90 , that is, all still images 90 , or some still images of all the input original images 90 . (90) may be cropped. In the latter case, the still image 90 to be cropped may be selected by a user or the like, or may be selected by the processor 100 according to a predefined setting.

According to an embodiment, the crop unit 110 may perform cropping according to a horizontal length, a vertical length, or a ratio between the still images 90 to be cropped. Specifically, for example, the crop unit 110 removes at least one of an upper portion and a lower portion of the still image 90 if the horizontal length of the still image 90 is smaller than a predetermined ratio with respect to the vertical length, If the vertical length of the still image 90 is smaller than a predetermined ratio with respect to the horizontal length, a part of the still image 90 may be removed by removing at least one of a left part and a right part of the still image 90 . In this case, the crop unit 110 determines two center lines A1 and A2 of the still image 90 , and based on at least one of the determined two center lines A1 and A2, the two center lines A1 , A2), the crop may be performed by cutting at least one of an upper portion, a lower portion, a left portion, and a lower portion separated by a predetermined distance or more. According to another embodiment, the crop unit 110 detects a main subject (B, for example, a person (face of a person) or an animal, etc.) in the still image 90 , and if the main subject B is still If it exists in the image 90 , cropping of the still image 90 is performed by removing at least a portion of an upper part, a lower part, a left part, and a lower part so that the main subject B is present in the cropped still image 91 . can also be done In this case, the crop unit 110 performs cropping so that the detected main subject B is placed at an appropriate position (eg, a center or a third from the top) of the cropped still image 91 . You may. The detection of the main subject B may be performed using a predetermined algorithm (eg, a learning algorithm) prepared in advance for detecting a specific object in the image 90 . For example, the predetermined algorithm is a feature point It may be provided to recognize the main subject B using the detected feature points or edges after first detecting the or edge. Even when the main subject B is detected and cropped, the crop unit 110 may perform the cropping by further reflecting the horizontal length, the vertical length, or a ratio between them of the still image 90 .

The cropped still image 91 may be transmitted to the size adjustment unit 120 .

3 is a view for explaining an example of the operation of the size adjustment unit.

As shown in FIG. 3 , the size adjuster 120 may expand the size of the cropped still image 91 to obtain a larger still image 92 (hereinafter, extended still image). For example, when a cropped still image 91 having a size of 1920*1080 (FHD) is transmitted from the crop unit 110 , the size adjustment unit 120 receives 3840*2160 from the cropped still image 91 in response. An extended still image 92 of (UHD) size may be acquired. According to an embodiment, the expansion of the cropped still image 91 additionally generates one or more pixels corresponding to each pixel (pixel) of the still image 91, and a pixel corresponding to the additionally generated pixel. This can also be done by duplicating and recording values (eg, RGB or CMYK values). For example, the resizing unit 120 adds three pixels corresponding to each pixel of the cropped still image 91, inputs the values of the original pixels to the added three pixels, and arranges them appropriately. An expanded still image 92 whose size is four times larger than that of the cropped still image 91 may be obtained. The expanded still image 92 may be transmitted to the denoising unit 130 .

4 is a block diagram of an embodiment of a denoising unit.

As shown in FIG. 4 , the denoising unit 130 receives at least one extended still image 92 from the size adjuster 120 , and denoises the at least one extended still image 92 . One or more denoised images ( 97 of FIG. 8 ) may be acquired, and the denoised images 97 may be transmitted to the final image generator 140 . According to an embodiment, the denoising unit 130 includes the noise adding unit 131 , the image combining unit 132 , the first noise processing unit 133 , the second noise processing unit 134 , and the learning processing unit 135 . may include In some embodiments, at least one of the noise adding unit 131 , the image combining unit 132 , the first noise processing unit 133 , the second noise processing unit 134 , and the learning processing unit 135 may be omitted.

5 is a diagram for explaining an example of an image before and after adding noise.

As shown in FIG. 5 , the noise adding unit 131 may obtain a still image 93 to which noise is added by adding noise to the extended still image 92 . When the still image 91 is excessively expanded as in the above-described operation of the size adjuster 120 , irregular noise is generated in the expanded still image 92 . The noise adding unit 131 may generally make irregular noise in the image 92 regular by adding and adding arbitrary noise to the extended still image 92 having the irregular noise as described above. In other words, the still image 93 to which the noise is added has approximately regular noise. According to an exemplary embodiment, the noise adding unit 131 may generate noise by a certain ratio (eg, about 5%) of the total number of pixels of the expanded still image 92 in order to prevent the images 92 and 93 from being excessively deformed. It is also possible to obtain a still image 93 to which noise is added by adding ? to the extended still image 92 . According to an embodiment, the noise added to the extended still image 92 may include grain noise. In this case, the noise adding unit 131 may acquire the noise-added still image 93 by applying at least one filter 92a separately provided for adding grain noise to the extended still image 93 . have. As the noise is added, the resolution of the still image 93 to which the noise is added is lower than the resolution of the expanded still image 92 . The still image 93 to which the noise is added may be transmitted to the image combining unit 132 .

6 is a view for explaining an example of the operation of the image combining unit.

The image combining unit 132 receives the still image(s) 93 , 93-1 , 93-2 and 93-3 to which at least one noise is added from the noise adding unit 131 , and receives at least one At least one moving image 94 may be acquired based on the still images 93 , 93 - 1 , 93 - 2 and 93 - 3 to which noise is added. In this case, the image combining unit 132 connects and combines, sequentially or arbitrarily, the still images 93, 93-1, 93-2, and 93-3 to which at least one noise is added to the moving image 94. ), and each of the still images 93 , 93-1 , 93-2 and 93-3 to which noise is added is used as a moving picture frame of the generated moving picture 94 . Here, the still images 93, 93-1, 93-2, and 93-3 to which at least one noise is added are applied to the crop unit 110, the size adjustment unit 120, and the denoising unit 130 described above. Accordingly, crop processing, size adjustment, and noise addition may be performed individually. In this case, the respective noise-added still images 93, 93-1, 93-2, and 93-3 are cropped in the same way, adjusted to the same size, and/or using the same filter 92a. Therefore, noise may be added. In addition, the sequential connection and combination of the still images 93, 93-1, 93-2, and 93-3 to which at least one noise is added follows a predefined story line order or at least one still image 93 , 93-1, 93-2, 93-3) according to the input order of the at least one noise added still images (93, 93-1, 93-2, 93-3) are sequentially arranged can The image combination unit 132 may generate the above-described moving image 94 based on a predetermined moving picture compression standard (codec), for example, in MPEG-2, MPEG-4 (eg, H.264) format. The moving picture 94 may be generated using the same. The generated video 94 may be transmitted to the first noise processing unit 133 .

7 is a diagram for explaining an example of the operation of the first noise processing unit.

Referring to FIG. 7 , the first noise processing unit 133 obtains at least one moving image 94 from the image combining unit 132 , and performs denoising on the at least one moving image 94 to obtain a second One denoised frame 95 - 1 may be obtained or a first denoised video 95 including the first denoised frame 95 - 1 may be obtained. In an embodiment, the denoising of the first noise processing unit 133 may be performed using at least two frames 94 - 1 and 94 - 2 in the moving picture 94 as shown in FIG. 7 . have. Here, the at least two frames 94 - 1 and 94 - 2 may be mutually continuous frames, and/or may be mutually discontinuous frames arbitrarily selected according to a specific pattern from among all frames in the moving picture 94 . .

According to an exemplary embodiment, the first noise processing unit 133 may first perform noise processing with at least one pixel of at least one frame (eg, the first frame 94 - 1 ) and at least one other frame (eg, the first frame 94 - 1 ). A difference between at least one pixel of the second frame 94 - 2 may be detected for each pixel. Here, at least one pixel of the first frame 94 - 1 and at least one pixel of the second frame 94 - 2 may correspond to each other. Also, the second frame 94-2 may be a frame following the first frame 94-1. Next, the first noise processing unit 133 is configured to, when a difference exists between pixels of at least both frames 94 - 1 and 94 - 2 or the difference exceeds a reference value, the corresponding pixel of the second frame 94 - 2 . It may be determined whether the pixel has a value corresponding to black or white. If it is determined that the pixel value of at least one pixel of the second frame 94-2 corresponds to black or white, the first noise processing unit 133 determines that noise exists in the corresponding pixel, and A new second frame 95-2 may be acquired by replacing the pixel value of the corresponding pixel of the second frame 94-2 with the pixel value of the corresponding pixel of the first frame 94-1. As such, when noise processing is performed on all or some of the frames 94-1 and 94-2 in the moving image 94, the first denoised moving image ( 95) is obtained. According to an embodiment, the first noise processing unit 133 extracts all or some of the frames 94-1 and 94-2 of the received video 94, and then processes the noise as described above and removes the noise. It is also possible to obtain the first denoised video 95 by combining the processed frame 95-2(s) or replacing the existing frame 94-2 with the noise-processed frame 95-2. do.

According to an embodiment, the first noise processing unit 133 may obtain the first denoised moving image 95 by performing noise processing using a predetermined filter provided in a moving picture program or the like. In this case, the setting value of the noise level for the moving picture program may be set to -85% to -80%, but is not limited thereto.

Each frame 95 - 2 denoised by the first noise processing unit 133 or the first denoised moving image 95 including them may be transmitted to the second noise processing unit 134 .

The second noise processing unit 134 receives each of the first denoised frames 95-2 or the first denoised video 95, and receives each of the first denoised frames 95-2 or the second noise processing unit 134. Based on the first denoised video 95, a denoised image (96 of FIG. 8 , hereinafter may be a second denoised image, a still image, and/or a moving image) may be acquired. In this case, the second noise processing unit 134 may additionally perform noise processing on all or some of the received frames 95 - 2 or all or some of the frames of the first denoised video 95 . In more detail, for example, pixels that do not form an image may be removed by removing a black and white point (chroma-based grain noise) of a very small pixel size (eg, 1 pixel) existing in at least one frame. , after measuring the noise in units of a predetermined number of frames (eg, 4 frames), remove the uniform noise measured common to all of these frames, and replace the part corresponding to the noise with the pixel value of a neighboring pixel or a similar value You can also set the block value generated in the process of removing noise for each frame (for example, select a frame without damage among the frames in the image 95, learn the frame, This may be performed by correcting the pixel values of a blurred portion of another damaged frame based on the learning result), and/or correcting the partially changed color gamut value during the image expansion process to be the same as the existing image ( For example, when an image is expanded, a rainbow effect occurs at or around the boundary, which may be performed by replacing it with a pixel value of a corresponding position before expansion). In addition to the illustrated bar, the second noise processing unit 134 adds various denoising processes necessary for improving the image quality of the image 95 according to an arbitrary selection of a user or a designer, to the above-described processing, or replaces the above-described processing. can also be done

According to an embodiment, the second noise processing unit 134 may include at least one filter ( function, etc.) to perform the second denoising. Here, the at least one filter is, for example, a denoising filter (in this case, a set value, for example, Despot count is 1, Threshold is 4/5/5, search Radius is set to 2, Similarity radius is set to 4, Intensity is set to 1.2, Block can also be set to 48/48/3), Degrain filter (when the Degrain filter is applied) , e.g. Chroma is selected, soft may be set to 0/0/0), DeBlock filter (setting values are e.g. Quant is 25, the offset may be set to 0/0, and the CQ may be set to 2.0), a Dehalo filter (when applying that filter, for example, repair and SMode) is selected, the repair mode may be set to 17), a Dering filter (a threshold value of 2 may be set), and a Derainbow filter (eg intensity of 200 and diameter of 11) ), dust removal filter (when employing that filter, Threshold to 10/10, Scene Change to 25, Max diff Choose to 50, dupthresh to 64, TRatio to 3 , Luma spatial/temporal is set to 100/20, Chroma spatial/temporal is set to 15/5, Spatial mode is selected, and the number of iterations is 2 ), Sharpen filter (in this case, intensity is set to 100, range sharpening is set to 11, zero is 16, power is 4, damp is 4/48, undershoot ) can be set to 1/2, overshoot is set to 1/2 in unlimited mode, smoothness is set to 20, sharpening mode and HQ mode may be selected for all boundaries), line filters (correspondingWhen applying a filter, for example, Device may be set to -1, Intensity to 1.0, Line Threshold to 2/12, Blur to 2, Depth to 80) and at least of a Stabilizer filter. It may include one filter, but is not limited thereto. In addition, the setting value(s) related to the above-described filter may be determined by a user or a designer, and the setting values determined by the user or designer may be the same as or different from at least one of the above-described example(s). may be

8 is a diagram for explaining an example of an image according to a learning processing result.

The learning processing unit 135, as shown in FIG. 8, receives the second denoised image 96 (possibly including a still image or a moving image) obtained by the second noise processing unit 134, and the obtained second 2 A final denoised image 97 (which may include a still image or a moving image) may be obtained by using the 2 denoised image 96 and at least one learning algorithm 135 - 1 . In this case, according to an embodiment, a corresponding result may be obtained based on at least one learning algorithm 135 - 1 for each of all or some frames of the second denoised image 96 . The at least one learning algorithm 135 - 1 may be trained in advance to output the image 97 with improved image quality, etc. by inputting the second denoised image 96 as an input. For example, the learning algorithm 135 - 1 removes the pixel blur generated around the edge or sets the center point so that the desired part, such as the main subject B, is placed in the center of the image 97 to be acquired. may have been trained in The at least one learning algorithm 135 - 1 is, for example, a deep neural network (DNN), a convolutional neural network (CNN), a multi-layer perceptron, or a recurrent neural network (RNN). : Recurrent Neural Network, Convolutional Recurrent Neural Network (CRNN), Deep Belief Network (DBN), Deep Q-Networks, Long short term memory (LSTM) ), a support vector machine (SVM), a generative adversarial network (GAN), and/or a conditional adversarial neural network (cGAN), either alone or in combination. it may have been

According to the processing result of the learning processing unit 135, it is possible to finally obtain the denoised image 97. The denoised image 97 processed and acquired by the learning processing unit 135 may be transmitted to the final image generating unit 140 .

The final image generator 140 may finally obtain and output the final image 98 (which may include a still image or a moving image) as shown in FIG. 1 by using the finally denoised image 97 . For example, when the final denoised image 97 is one or more still image(s), the final image generating unit 140 renders it based on a predetermined compression standard while connecting and combining the still image(s). may be performed to obtain a final moving image, that is, the final image 98, or the final denoised image 97, which is a moving image, may be compressed according to a predetermined compression standard to obtain the final image 98. The final image 98 is transmitted to the storage unit 15 or the output unit 19, and may be visually and/or aurally output to a user or the like, or may be transmitted to another information processing device.

The above-described image processing apparatus 10 may be implemented using at least one device specially designed to perform all or part of the above-described operations, or by using at least one information processing apparatus alone or in combination. may be implemented. Here, the at least one information processing device is, for example, a desktop computer, a laptop computer, a server hardware device, a scanner device, a printer device, a 3D printer device, a smart phone, a tablet PC, a smart watch, a smart tag, a smart band, a head Head Mounted Display (HMD) Devices, Handheld Game Machines, Personal Digital Assistants (PDAs), Navigation Devices, Smart Keys, Remote Controllers (Remote Controls), Digital Televisions, Set Top Boxes, Digital Media Player Devices, Media Streaming devices, DVD playback devices, compact disc playback devices, sound playback devices (such as artificial intelligence speakers), home appliances (such as refrigerators, fans, air conditioners, ovens or washing machines), manned or unmanned vehicles (such as cars, buses or Vehicles such as two-wheeled vehicles, mobile robots, wireless model vehicles, robotic vacuums, etc.), manned or unmanned aerial vehicles (such as aircraft, helicopters, drones, model airplanes, model helicopters, etc.), medical systems, household, industrial or military robots; It may include, but is not limited to, an industrial or military machine, an electronic blackboard, or an electronic billboard. A designer, a user, or the like may consider and employ, as the image processing device 10 , at least one of devices capable of arithmetic processing and control of information in addition to the above-described information processing device according to circumstances or conditions.

9 is a diagram illustrating a peak signal-to-noise ratio (PSNR) between an output result of an image processing apparatus and a simple scale-up result, where the y-axis means the maximum signal-to-noise ratio, and the x-axis represents the frame. it means. The green line indicates the maximum signal-to-noise ratio for the image processing apparatus 10 described above, and the yellow line indicates the maximum signal-to-noise ratio when only simple scale-up is applied to the image.

Referring to FIG. 9 , the maximum signal-to-noise ratio for the image processing apparatus 10 is maintained without significant change centered on the mid-to-late mid-to-late value of 40 as a result of measurement, and the average value is about 41 However, in the case of simple scale-up, the maximum signal-to-noise ratio varies considerably from about 15 to 30 or more, and the average value thereof is about 18. This means that the noise of the video combined by the processing of the above-described image processing apparatus 10 is greatly improved.

10 is a diagram illustrating a Structural Similarity Index Map (SSIM) between an output result of an image processing apparatus and a simple scale-up result. The y-axis means similarity to the original image, and the x-axis means a frame. . The green line indicates the degree of similarity with the original image of the image processing apparatus 10 described above, and the yellow line indicates the degree of similarity with the original image when only simple scale-up is applied to the image.

Referring to FIG. 10 , the similarity between the image obtained as a result of the processing of the image processing apparatus 10 and the original image is approximately 1 for every frame, and generally represents 0.95 or more, and generally Therefore, the range of change is small. Also, the average of the similarity between the image generated by the image processing apparatus 10 and the original image was 0.98. This means that the above-described image processing process preserves almost 98% of the original image. On the other hand, in the case of a simple scale-up image, the similarity of the image has a value between about 0.7 and 0.95, and the value changes significantly for each frame. Also, the average of the similarity between the simple scale-up image and the original image is 0.87, which is significantly lower than the 0.98 average of the similarity to the image processing apparatus 10 described above. This indicates that when only simple scale-up is performed, the retention rate of the original image varies greatly depending on the frame, and the quality of the image is significantly deteriorated.

Therefore, as shown in FIGS. 9 and 10 , when the above-described image processing apparatus 10 generates a moving image by combining still images, the effect of generating a moving image with as much as possible the same as the original image and very little noise is effective. it can be seen that there is

Hereinafter, various embodiments of an image processing method will be described with reference to FIG. 11 .

11 is a flowchart of an image processing method according to an embodiment.

Referring to FIG. 11 , a plurality of still images may be acquired first ( 200 ). The plurality of still images may include, for example, live-action photos, designs, or cartoons (eg, one or more cuts such as comics, cartoons, or webtoons). The plurality of still images may be temporally or story-wise consecutive according to an embodiment. The acquisition of a plurality of still images may be performed based on a direct input of a user or a designer, transmission from an external storage medium, and/or transmission from a web server using crawling or the like.

At least one still image may be cropped according to a predetermined ratio ( 210 ). Accordingly, at least one cropped image corresponding to each of the at least one still image may be obtained. Cropping may be performed according to a horizontal length, a vertical length, or a ratio between a horizontal length and a vertical length of the still image. In this case, if the horizontal length of the still image is smaller than a predetermined ratio with respect to the vertical length, the crop removes at least one of the upper part and the lower part of the still image, and the vertical length of the still image is greater than the preset ratio with respect to the horizontal length. If it is small, it may be performed by removing at least one of a left part and a right part of the still image. According to an embodiment, the cropping may be performed using at least one center line of the still image. Also, cropping may be performed so that a main subject in the still image is appropriately present in the cropped still image.

At least one cropped still image may be enlarged in size ( 220 ). For example, a cropped still image having a size of 1920*1080 (FHD) may be extended to a size of 3840*2160 (UHD). In this case, the size expansion may be performed through duplication of pixels or the like. The process of enlarging the size 220 may be omitted in some embodiments.

Noise may be added to the at least one cropped still image or the at least one extended still image ( 230 ). According to the addition of noise, the image generally has regular noise. Here, the noise may include grain noise. The addition of noise to the cropped still image or the extended still image may be performed using a predetermined filter (eg, a grain filter).

Still images to which at least one noise is added are connected and combined, and thus at least one moving image may be obtained ( 240 ). Acquisition of the moving image may be performed by sequentially or arbitrarily combining still images to which at least one noise is added. In this case, the still image to which at least one noise is added constitutes each frame of the moving picture. Image acquisition may be performed based on a predetermined video compression standard.

A denoising process is performed based on the generated video ( 260 ). According to an embodiment, the denoising process may be performed through a plurality of processes.

For example, first denoising is performed. In the first denoising process, a difference with respect to at least one pixel of at least two frames (eg, consecutive frames) of a moving picture first generated for denoising may be detected for each pixel. That is, a difference between at least one pixel of the first frame and at least one pixel of a second frame (eg, a frame next to the first frame) corresponding to the at least one pixel of the first frame may be detected. . If there is a difference between corresponding pixels of the first frame and the second frame, and at least one pixel of the second frame has a value corresponding to black to white, at least one pixel of the second frame, A pixel value of a pixel of the first frame corresponding to at least one pixel of the second frame may be recorded. In this case, the existing pixel value of at least one pixel of the second frame is deleted. This denoising process may be performed on all frames or some frames.

Second denoising may be performed on the first denoised frame or a moving picture including them. Accordingly, a second denoising image may be obtained. For example, in the second denoising, chroma-based grain noise having a very small pixel size is removed from at least one frame among all frames of the first denoised moving image, or uniformly existing in every predetermined number of frames. This includes replacing the noise with a pixel value or a similar value around the pixel in which the noise exists, setting the block value for each frame, and/or correcting the partially changed color gamut value during the image expansion process. You may. At least one filter may be used for the second denoising, and the at least one filter may include a denoise filter, a degrain filter, a deblock filter, a dehalo filter, a dering filter, and a derainbow filter, according to an embodiment. , a dust removal filter, a sharpen filter, a line filter, and at least one of a stabilizer filter, but is not limited thereto.

The second denoising-processed image is re-corrected by employing at least one learning algorithm. In this case, the at least one learning algorithm may be trained to use the second denoised image as an input value and finally to use the denoised image as an output value. The learning algorithm-based correction may be used, for example, to remove pixel blur generated around a boundary (edge) or to correct the position of a main subject. The learning algorithm may include at least one learning algorithm that can employ a convolutional neural network or a deep neural network according to a user's or designer's selection.

Finally, a final image (movie) is obtained from the denoised image ( 262 ). Acquisition of the final image may be performed by combining the finally denoised still image and rendering according to a predetermined compression standard, and/or may be performed by rendering the finally denoised moving image according to the predetermined compression standard. have. The final image may be stored in a predetermined storage medium and/or transmitted to a predetermined information processing device (eg, a smart phone or a desktop computer) through a wired/wireless communication network or the like.

The image processing method according to the above-described embodiment may be implemented in the form of a program that can be driven by a computer device. The program may include instructions, libraries, data files and/or data structures alone or in combination, and may be designed and manufactured using machine code or high-level language code. The program may be specially designed to implement the above-described method, or may be implemented using various functions or definitions that are known and available to those skilled in the art of computer software. In addition, here, the computer device may be implemented including a processor or memory that enables the function of the program to be realized, and may further include a communication device if necessary. A program for implementing the above-described image processing method may be recorded in a computer-readable recording medium. The computer-readable recording medium includes, for example, a semiconductor storage medium such as a ROM, RAM or flash memory (eg, a solid state drive (SSD), etc.), a magnetic disk storage medium such as a hard disk or a floppy disk, At least one type of physical storage medium capable of temporarily or non-temporarily storing one or more programs executed in response to a computer call, such as an optical recording medium such as a compact disk or DVD, may be included.

Although the embodiments of the image processing apparatus and the image processing method have been described above, the image processing apparatus or the image processing method is not limited only to the above-described embodiments. Various other devices or methods that can be implemented by a person skilled in the art by modifying and modifying the above-described embodiment based on the above-described embodiment may also be an embodiment of the above-described image processing apparatus or image processing method. For example, the described method(s) are performed in a different order than described, and/or the described component(s) of a system, structure, apparatus, circuit, etc., are combined, connected, or otherwise configured in a different manner than described. Even if they are combined or replaced or substituted by other components or equivalents, the above-described image processing apparatus and/or image processing method may be an embodiment.

10: image processing device 90: image
91: Cropped still image 92: Expanded still image
93: Still image with noise added 94: Movie
95: first denoised image 96: second denoised image
97: final denoised image 98: final image
100: processor 110: crop unit
120: size adjustment unit 130: denoising unit
140: final image generation unit

Claims (16)

an input unit for acquiring at least one original image that is a still image;
At least one cropped still image is obtained from the at least one original image by cropping a portion of the at least one original image, and at least one extended still image is obtained by expanding the size of the at least one cropped still image. acquiring, adding a random noise to the at least one extended still image so that the at least one extended still image has regular noise, and combining the at least one still image to which the random noise is added. a processor for generating a video, performing denoising processing on at least one frame of the video, and obtaining a video corresponding to the at least one still image based on a denoising process result for the at least one frame; do,
The processor is
detecting a difference between a pixel of a first frame and a pixel of a second frame of the video;
If there is a difference between the pixel of the first frame and the pixel of the second frame, and the pixel value of the pixel of the second frame has a value corresponding to black or white, noise is generated in the pixel of the second frame judged to exist,
When it is determined that noise exists in the pixels of the second frame, the pixel values of the second frame are replaced with the pixel values of the pixels of the first frame corresponding to the pixels of the second frame to obtain a first denoised image image processing device. delete delete delete According to claim 1,
The processor may be configured to obtain a second denoised image by applying at least one filter to the first denoised image. 6. The method of claim 5,
The processor removes chroma-based grain noise of a very small pixel size from at least one frame among all frames of the first denoised video, or a pixel value in which noise is uniformly present every predetermined number of frames An image processing apparatus for obtaining the second denoising image by substituting a pixel value or a similar value around the pixel, setting a block value for each frame in the image, or correcting a partially changed color gamut value in the image. 6. The method of claim 5,
The processor is an image processing apparatus for obtaining a final denoising image corresponding to the second denoising image by using a learning algorithm. According to claim 1,
The noise added to the at least one still image includes grain noise. acquiring at least one original image that is a still image;
obtaining at least one cropped still image from the at least one original image by cropping a portion of the at least one original image;
obtaining at least one extended still image by expanding the size of the at least one cropped still image;
adding a random noise to the at least one extended still image so that the at least one extended still image has regular noise;
generating a moving picture by combining the at least one still image to which the random noise is added;
performing denoising processing on at least one frame of the moving picture; and
Including; obtaining a moving picture corresponding to the at least one still image based on the denoising processing result of the at least one frame;
The denoising process on at least one frame of the video includes:
detecting a difference between a pixel of a first frame and a pixel of a second frame of the moving picture;
If there is a difference between the pixel of the first frame and the pixel of the second frame, and the pixel value of the pixel of the second frame has a value corresponding to black or white, noise is generated in the pixel of the second frame determining that it exists; and
if it is determined that noise is present in the pixels of the second frame, replacing the pixels of the second frame with the pixel values of the corresponding pixels of the first frame to obtain a first denoised image; Way. delete delete delete 10. The method of claim 9,
The denoising process on at least one frame of the video includes:
and obtaining a second denoising image by applying at least one filter to the first denoising image. 14. The method of claim 13,
obtaining a second denoising image by applying at least one filter to the first denoising image,
removing chroma-based grain noise having a very small pixel size for at least one frame among all frames of the first denoised video;
replacing a pixel value in which noise is uniformly in every predetermined number of frames among all frames of the first denoised moving picture with a pixel value or similar value around the pixel;
setting a block value for each frame in the image; and
correcting a partially changed color gamut value in an image; An image processing method comprising at least one of 14. The method of claim 13,
The denoising process on at least one frame of the video includes:
and obtaining a final denoising image corresponding to the second denoising image by using a learning algorithm. 10. The method of claim 9,
The noise added to the at least one still image includes grain noise.

Images