KR102794217B1 - High-quality conversion method and apparatus for low-resolution media based on video characteristics - Google Patents

Abstract

The present disclosure relates to a method and device for converting low-quality media into high-quality media based on image characteristics, the method comprising: receiving image content from an external device, classifying the received image content according to a genre category, classifying the classified image content based on at least one of time complexity and space complexity, processing the classified image content based on at least one of resolution, number of frames, sharpness, and color gamut, and improving image quality of the processed image content.

Description

{High-quality conversion method and apparatus for low-resolution media based on video characteristics}

The present disclosure relates to a method and device for converting low-quality media into high-quality media, and more particularly, to a technology for a method and device for converting low-quality media into high-quality media based on image characteristics.

Korean Wave content, represented by K-POP and K-Drama, is now expanding into various forms including K-Movies and taking over the global content market. With the spread of this Korean Wave content, the demand for old, low-quality content produced by broadcasters in the past is also increasing explosively both domestically and internationally. Initially, the demand for old content increased mainly in the Southeast Asian market, but this demand is also increasing explosively in Central and South American and European markets.

With this increase in demand and market changes, there is a need to develop and advance high-definition conversion element technology to improve the utilization of existing content.

However, although various studies are being conducted to improve the performance of element technologies for high-definition conversion of content, they are biased towards studies by conversion element unit using specific public datasets, showing limitations in improving performance, and the development of conversion solutions based on a commercialization model that integrates and applies these element technologies is insufficient, making it difficult to proceed to commercialization at the commercialization level.

The purpose of the present disclosure is to propose a method and device for converting low-quality media into high-quality video content by classifying the time and space complexity of original content into steps and applying an optimal video quality improvement method according to the classified video.

Other objects and advantages of the present disclosure can be understood by the following description, and will be more clearly understood by the embodiments of the present disclosure. In addition, it will be readily apparent that the objects and advantages of the present disclosure can be realized by the means and combinations thereof indicated in the claims.

According to one embodiment of the present disclosure, a method for converting low-quality media into high-quality media based on image characteristics includes the steps of: receiving image content from an external device; classifying the received image content according to a genre category; classifying the classified image content based on at least one of temporal complexity and spatial complexity; processing the classified image content based on at least one of resolution, frame count, sharpness, and color gamut; and improving image quality of the processed image content.

According to one embodiment of the present disclosure, by analyzing the characteristics of a low-quality image, an optimal algorithm is applied to each scene, and element technologies necessary for image quality improvement are applied to each scene, so that an image converted to the optimal quality desired by the user can be obtained, thereby improving user convenience.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art to which the present disclosure belongs from the description below.

FIG. 1 is a diagram illustrating a configuration of a device for converting low-quality media to high-quality media based on image characteristics according to one embodiment of the present disclosure.
FIG. 2 is a diagram illustrating a scenario classification according to time complexity and space complexity according to one embodiment of the present disclosure.
FIG. 3 is a diagram illustrating a flowchart of a method for converting low-quality media into high-quality media based on image characteristics according to an embodiment of the present disclosure.
FIG. 4 is a diagram illustrating a configuration of a device for converting low-quality media to high-quality media based on image characteristics according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

In describing embodiments of the present disclosure, if it is determined that a specific description of a known configuration or function may obscure the gist of the present disclosure, a detailed description thereof will be omitted. In addition, parts in the drawings that are not related to the description of the present disclosure have been omitted, and similar parts have been given similar drawing reference numerals.

In the present disclosure, the components that are distinguished from each other are intended to clearly explain the characteristics of each, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to form a single hardware or software unit, or a single component may be distributed to form a plurality of hardware or software units. Accordingly, even if not mentioned separately, such integrated or distributed embodiments are also included in the scope of the present disclosure.

In the present disclosure, the components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, an embodiment that consists of a subset of the components described in one embodiment is also included in the scope of the present disclosure. In addition, an embodiment that includes other components in addition to the components described in various embodiments is also included in the scope of the present disclosure.

In this disclosure, the terms first, second, etc. are used only for the purpose of distinguishing one component from another component, and do not limit the order or importance between the components unless specifically stated. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

When it is stated that a component of the present disclosure is “connected” or “connected” to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. On the other hand, when it is stated that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Additionally, in the present disclosure, unless one drawing illustrating an embodiment of the present disclosure corresponds to an alternative embodiment with another drawing, the description for each drawing may be applied to different drawings.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

FIG. 1 is a diagram illustrating a configuration of a device for converting low-quality media to high-quality media based on image characteristics according to one embodiment of the present disclosure.

Referring to Fig. 1, the high definition conversion device (100) includes an image characteristic analysis unit (110) and a high definition conversion unit (120).

A high-definition conversion device (100) receives a low-definition image (10) as input and outputs a high-definition image (20).

The image characteristic analysis unit (110) receives a low-quality image (10) as input.

The image characteristic analysis unit (110) analyzes the characteristics of the original image, which is a low-quality image.

The high definition unit (12) improves the image quality by applying a conversion element technology to the image according to the analyzed characteristics. The high definition unit (120) illustrated in Fig. 1 is only one embodiment, and various modifications are possible based on the components illustrated in Fig. 1.

The image characteristic analysis unit (110) includes a genre analysis unit (111) and a spatiotemporal complexity analysis unit (112).

The genre analysis department (111) analyzes and categorizes the content genre of video content and classifies the genre according to categories such as movies, dramas, sports, and news.

The spatiotemporal complexity analysis unit (112) divides video content into scenes, calculates temporal and spatial resolution-based complexity according to each divided scene, and distinguishes them based on a set threshold value.

The high definition unit (120) includes a high resolution unit (121), an HFR unit (High-Frame Rate, 122), an HDR unit (High-Dynamic Range, 123), a WCG unit (Wide-Color Gamut, 124), and an image quality improvement unit (125).

The high definition unit (120) receives the results of the image characteristic analysis unit (110) as input and outputs a high definition image (20).

The high-resolution unit (121) increases the resolution of video content by converting SD (480p) to a maximum 4K (2160p) resolution and HD (1080p) to a maximum 8K (4320p) resolution.

The HFR section (122) converts 60i (interlace method) to 60p (progressive method) or converts 60p to a maximum of 240p to increase the number of frames processed per hour and create a more natural screen.

The HDR section (123) converts SDR (Standard Dynamic Range) images into HDR (High Dynamic Range) images or HLG (High Log Gamma) images to create clearer images.

The WCG department (124) expands the color gamut by converting BT.601 to a maximum of BT.2020 to create a screen that is as close to natural colors as possible.

The image quality improvement unit (125) executes at least one of a compression distortion function, a deblur function, and a denoise function.

The compression distortion improvement function refers to the function that improves the distortion that occurs during the image encoding process. The deblur function refers to the function that reduces the color bleeding of the image. The denoise function removes the noise component of the image.

Specifically, the image characteristic analysis unit (110) classifies the image by analyzing the low-quality content image for at least one of the genre, time complexity, and space complexity of the content image.

At least one of the time complexity and the space complexity can be calculated using the brightness information of the image, and the steps are distinguished by setting an arbitrary cutoff value according to the distribution of the calculated value.

Time complexity( ) can be obtained using the following mathematical formula 1.

Mathematical expression 1 is an operation for calculating time complexity according to frame changes in an image.

Time complexity refers to the value calculated by using the luminance component representing the brightness of the image to calculate the mean absolute difference (MAD) between the current frame and the next frame, and the complexity can be classified as high, medium, or low based on the calculated value and a predefined step cutoff value.

In mathematical expression 1, Yt(i, j) represents the luminance value at point (i, j) of the entire screen.

M and N represent the width and height of the entire screen, respectively.

The width and height may vary depending on the resolution.

For example, in the case of FHD video, the width can be calculated as 1920 pixels and the height as 1080 pixels. Based on the values mentioned above, the time complexity can be obtained by performing calculations for each time unit. Here, the time unit can be seconds (sec).

Next, mathematical expressions 2 to 4 are explained.

Mathematical equations 2 to 4 are operations for calculating spatial complexity, var(t), within a frame of a video.

Spatial complexity refers to the average variance value for the entire image area using the luminance component, which represents the brightness of the image. The complexity can be classified into high, medium, and low according to the step cutoff value defined in advance for the calculated value.

To measure spatial complexity, the luminance component variance of a 16x16 block is used as the basic measurement unit.

Mathematical expressions 2 and 3 are operations for obtaining luminance component variance values for a 16x16 block.

Mathematical expression 4 is an operation to obtain the average variance value for the entire image area using the luminance component variance value of the 16x16 block obtained through Mathematical expressions 2 and 3 and Mathematical expression 2.

In order to obtain spatial complexity, first, the variance value using brightness information is obtained in units of 16x16 blocks as in Equation 2. The spatial complexity can be obtained by calculating the variance average for the entire frame as in Equation 4 using the value calculated using Equation 2.

Space complexity, like time complexity, can be obtained by performing operations for each time unit based on the criteria mentioned above. Here, the time unit can be seconds.

Classify the content image by distinguishing between steps, for example, upper, middle, and lower, for at least one of the time complexity and space complexity derived through mathematical expressions 1 and 4.

FIG. 2 is a diagram illustrating a scenario classification according to time complexity and space complexity according to one embodiment of the present disclosure.

As shown in Fig. 2, when time complexity and space complexity are classified into three levels of high, medium, and low, a total of nine scenarios are required. The unit order and algorithm for the operation of the high definition unit (120) are automatically recommended according to each scenario.

For example, if both time complexity and space complexity are classified as high, the high definition unit (120) selects scenario 1.

The order of the operation algorithm is high-resolution unit (121) -> HFR unit (122) -> HDR unit (123) -> WCG unit (124) -> image quality improvement unit (125), and a method according to the classified time complexity and space complexity is applied within each unit.

If both time complexity and space complexity are classified as low, the high definition unit (120) selects scenario 9.

The processing is done in the order of HDR section (123) -> WCG section (124) -> HFR section (122) -> image quality improvement section (125) -> high-resolution section (121), and similarly, a method according to the classified time complexity and space complexity is applied within each unit.

According to the present invention, not all units within the high definition unit (120) need to operate every time a scenario is executed, and only the necessary units operate to perform image quality improvement.

FIG. 3 is a flowchart illustrating a method for converting low-quality media into high-quality media based on image characteristics according to an embodiment of the present disclosure. The present invention is performed by a device (100) for converting low-quality media into high-quality media based on image characteristics.

Referring to FIG. 3, video content is received from an external device (S310).

The received video content is classified according to genre category (S320).

The classified video content is classified based on at least one of time complexity and space complexity (S330).

The classified video content is processed based on at least one of resolution, frame rate, sharpness, and color range (S340).

Improves the image quality of the processed video content (S350).

FIG. 4 is a diagram illustrating a configuration of a device for converting low-quality media to high-quality media based on image characteristics according to an embodiment of the present disclosure.

Referring to FIG. 4, a device for converting low-quality media into high-quality media includes a device (1600). The device (1600) may include a memory (1602), a processor (1603), a transceiver (1604), and a peripheral device (1601). In addition, as an example, the device (1600) may further include other configurations and is not limited to the above-described embodiment. At this time, as an example, the device (1600) may be a device that operates based on the above-described low-quality media into high-quality media conversion method.

More specifically, the device (1600) of FIG. 4 may be an exemplary hardware/software architecture such as a low-quality media to high-quality media conversion device, a picture quality enhancement device, etc. In this case, as an example, the memory (1602) may be a non-removable memory or a removable memory. In addition, as an example, the peripheral device (1601) may include a display, GPS, or other peripheral devices, and is not limited to the above-described embodiment.

In addition, as an example, the above-described device (1600) may include a communication circuit such as the transceiver (1604), and may perform communication with an external device based thereon.

Also, as an example, the processor (1603) may be at least one of a general-purpose processor, a digital signal processor (DSP), a DSP core, a controller, a microcontroller, ASICs (Application Specific Integrated Circuits), FPGA (Field Programmable Gate Array) circuits, any other type of integrated circuit (IC), and one or more microprocessors associated with a state machine. That is, it may be a hardware/software configuration that performs a control role for controlling the above-described device (1600).

At this time, the processor (1603) may execute computer executable instructions stored in the memory (1602) to perform various essential functions of the node. For example, the processor (1603) may control at least one of signal coding, data processing, power control, input/output processing, and communication operations. In addition, the processor (1603) may control physical layers, MAC layers, and application layers. In addition, for example, the processor (1603) may perform authentication and security procedures in the access layer and/or the application layer, and is not limited to the above-described embodiment.

For example, the processor (1603) can communicate with other devices through the transceiver (1604). For example, the processor (1603) can control the node to communicate with other nodes through the network through the execution of computer-executable instructions. That is, the communication performed in the present invention can be controlled. For example, the other nodes can be NDN servers, content routers, and other devices. For example, the transceiver (1604) can transmit RF signals through the antenna and transmit signals based on various communication networks.

In addition, as an example, MIMO technology, beamforming, etc. can be applied as antenna technology, and are not limited to the above-described embodiment. In addition, a signal transmitted and received through the transceiver (1604) can be modulated and demodulated and controlled by the processor (1603), and are not limited to the above-described embodiment.

The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combinations of two or more.

In addition, various embodiments of the present disclosure can be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, it can be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc. For example, it is obvious that it can be implemented in the form of a program stored on a non-transitory computer-readable medium that can be used at the end or edge, or in the form of a program stored on a non-transitory computer-readable medium that can be used at the edge or in the cloud. In addition, it can be implemented by a combination of various hardware and software.

The scope of the present disclosure includes software or machine-executable instructions (e.g., an operating system, an application, firmware, a program, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and being executable on the device or the computer.

The present disclosure described above is capable of various substitutions, modifications, and changes within a scope that does not depart from the technical spirit of the present disclosure for those with ordinary skill in the art to which the present disclosure pertains, and therefore the scope of the present disclosure is not limited by the above-described embodiments and the attached drawings.

Claims (2)

In a method for converting low-quality media to high-quality media based on image characteristics,
In the image characteristic analysis section, a step of receiving low-quality image content from an external device;
In the above video characteristic analysis unit, a step of classifying the received video content according to genre category;
In the above image characteristic analysis section, a step of classifying the classified image content based on time complexity and space complexity; and
In the high-quality processing unit, a step of processing the classified image content based on at least one of resolution, frame rate, sharpness, and color range to output high-quality image content is included.
The above spatial complexity represents the average variance value for the entire image area using the luminance component, and the average variance value is divided into upper, middle, and lower according to the predefined step division value.
The above time complexity represents the average absolute difference between the current frame and the next frame using the luminance component, and the average absolute difference is divided into upper, middle, and lower according to a predefined step division value.
The step of outputting the above high-definition video content is:
If both the time complexity and the space complexity are classified as “high,” a step of processing the image content in the order of a unit processing the resolution, a unit processing the number of frames, a unit processing the sharpness, and a unit processing the color area is included.
The step of outputting the above high-definition video content is:
If both the time complexity and the space complexity are classified as “low,” the method further includes a step of processing the image content in the order of a unit for processing the sharpness, a unit for processing the color gamut, a unit for processing the number of frames, and a unit for processing the resolution.
How to convert low-quality media to high-quality media.
In a device for converting low-quality media to high-quality media based on image characteristics,
An image characteristic analysis unit that receives low-quality image content from an external device, classifies the received image content according to genre categories, and classifies the classified image content based on time complexity and space complexity; and
Includes a high-quality processing unit that processes the classified video content based on at least one of resolution, frame rate, sharpness, and color range to output high-quality video content.
The above spatial complexity represents the average variance value for the entire image area using the luminance component, and the average variance value is divided into upper, middle, and lower according to the predefined step division value.
The above time complexity represents the average absolute difference between the current frame and the next frame using the luminance component, and the average absolute difference is divided into upper, middle, and lower according to a predefined step division value.
The above high-quality processing unit is configured to process the image content in the order of a unit processing the resolution, a unit processing the number of frames, a unit processing the sharpness, and a unit processing the color area, when both the time complexity and the space complexity are classified as “high”.
The above high-quality processing unit is further configured to process the image content in the order of a unit for processing the sharpness, a unit for processing the color gamut, a unit for processing the number of frames, and a unit for processing the resolution, when both the time complexity and the space complexity are classified as “low.”
A device that converts low-quality media into high-quality media.