KR20200067114A - Apparatus for transmitting image - Google Patents

Abstract

The image transmission apparatus according to an embodiment includes a down-sampling unit for down-sampling an original high-resolution image and converting it into a low-resolution image, and a first coding technique that can be decoded by each of a small user and a small user. Based on the first coding unit for coding the low-resolution image and the input of the low-resolution image converted by the down-sampling unit, a super-resolution unit pre-trained to output a processed high-resolution image having a lower quality than the original high-resolution image. And, based on a second coding technique that can be decoded by the high-end terminal, and a residual map providing unit that calculates a difference between the original high-resolution image and the processed high-resolution image and provides it in the form of a residual map. And a second coding unit for coding the map.

Description

Image transmission device {APPARATUS FOR TRANSMITTING IMAGE}

The present invention relates to an image transmission device.

In normal data communication, distortion is not allowed when performing compression, whereas multimedia signals such as video and audio allow a certain amount of distortion, thereby increasing compression efficiency.

At this time, there is a demand for a technique for allowing a certain amount of distortion and compressing accordingly, considering each case of a high-end terminal (big user) as well as a case of a low-end terminal (small user).

Republic of Korea Patent Publication No. 10-2017-0047489, Publication date May 08, 2017

The problem to be solved by the present invention is to provide a technique related to effectively transmitting an image to terminals having different screen resolutions while having different numbers of antennas.

However, the problem to be solved of the present invention is not limited to those mentioned above, and another problem not to be solved can be clearly understood by a person having ordinary knowledge to which the present invention belongs from the following description. will be.

The image transmitting apparatus according to an embodiment includes a down-sampling unit for down-sampling an original high-resolution image and converting it into a low-resolution image, and a first coding technique that can be decoded by each of a small user and a small user. Based on the first coding unit for coding the low-resolution image and the input of the low-resolution image converted by the down-sampling unit, a super-resolution unit pre-trained to output a processed high-resolution image having a lower quality than the original high-resolution image. And, based on a second coding technique that can be decoded by the high-end terminal, and a residual map providing unit that calculates a difference between the original high-resolution image and the processed high-resolution image and provides it in the form of a residual map. And a second coding unit for coding the map.

In addition, the first coding technique may be an Alamouti technique.

In addition, the second coding technique may be a golden coding technique.

In addition, the super-resolution unit may include a plurality of residual block units configured to include a convolution layer and a ReLU layer but not a batch normalization layer.

In addition, the residual map forming unit may provide the residual map based on a convex optimization technique.

According to an embodiment, optimal performance can be achieved from an end-to-end point of view, and in particular, terminals having different screen resolutions with different numbers of antennas, such as a low-end terminal and a high-end terminal, are in the same area. Even within the image can be effectively delivered to each of these terminals.

1 is a configuration diagram schematically showing a configuration of an image transmission apparatus according to an embodiment.
2 is a configuration diagram schematically showing a configuration of a high-spec terminal according to an embodiment.
3 is a block diagram schematically showing a configuration of a low-spec terminal according to an embodiment.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the person having the scope of the invention, and the present invention is only defined by the scope of the claims.

In describing embodiments of the present invention, when it is determined that a detailed description of known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to a user's or operator's intention or practice. Therefore, the definition should be made based on the contents throughout this specification.

1 is a configuration diagram schematically showing a configuration of an image transmission apparatus 100 according to an embodiment.

Referring to FIG. 1, the image transmitting apparatus 100 includes a down-sampling unit 110, an image source coding unit 120, a first pre-processing unit 130, a first coding unit 140, and a first mux unit 150 ), a super-resolution unit 160, a residual map providing unit 170, a second pre-processing unit 180, a second coding unit 190, and a second mux unit 191. However, the configuration of the image transmission apparatus 100 illustrated in FIG. 1 is merely exemplary. For example, according to an embodiment, the image transmission device 100 may be implemented or configured to be different from that shown in FIG. 1.

First, the image transmission device 100, and components included in each of these image transmission devices 100, a memory storing instructions implemented to perform a function to be described below, and a microcomputer executing instructions stored in the memory It can be implemented by a processor.

First, the image transmission apparatus 100 receives an original high resolution (HR) image 10. Then, the original high resolution image 10 thus received is converted into a low resolution (LR) image 11 and then coded based on Alamouti techniques. Here, the coding technique for the low resolution image is not limited thereto.

In addition, the above-described original high-resolution image 10 received by the image transmission apparatus 100 is converted into a low-quality processed high-resolution image, and thereafter represents a difference between the processed high-resolution image and the original high-resolution image 10 A residual map is generated. The residual map thus generated is coded based on a Golden code technique or the like. Here, the coding technique for the residual map is not limited thereto.

Then, the residual map coded based on the low-resolution image 11 and the golden code technique, which are coded based on the Alamouti technique, as described above, are delivered to an image receiving apparatus not shown in FIG. 1, for example, a low-end terminal or a high-end terminal. .

Here, the low-spec terminal (small user) refers to a terminal having a single antenna and a screen supporting low resolution. On the other hand, a high-spec terminal (big user) refers to a terminal having a plurality of (for example, two) antennas and a screen supporting high resolution.

In the high-end terminal, an image as close as possible to the original high-resolution image, that is, a restored high-resolution image is generated using the low-resolution image 11 and the residual map thus received. Here, in the process of generating the reconstructed high-resolution image, the above-described Alamauti technique or the Golden Code technique may be used.

On the other hand, in the low-end terminal, the image is restored and generated using only the low-resolution image 11 thus received. In this restoration process, the above-described Alamauti technique may be used.

Hereinafter, the image transmission device 100 itself will be described in more detail.

The down sampling unit 110 is configured to down sample the original high-resolution image 10. That is, the original high-resolution image 10 is converted into the low-resolution image 11 by the down-sampling unit 110.

Here, although not shown in FIG. 1, the down-sampling unit 110 may be implemented to further include an anti-aliasing low-pass filter, in which case the original high-resolution image 10 ), as well as anti-aliasing low pass filtering may be performed.

The image source coding unit 120 is configured to perform source coding on the low resolution image 11. Here, since the source coding operation for the image is a known technique, a detailed description thereof will be omitted. In addition, the image source coding unit 120 may not be included in the image transmission apparatus 100 according to an embodiment. In this case, the low-resolution image 11 from the down-sampling unit 110 may be directly transmitted to the super-resolution unit 160 and the first pre-processing unit 130, which will be described later. However, hereinafter, it is assumed that the image source coding unit 120 is included in the image transmission apparatus 100.

The first pre-processing unit 130 is configured to receive the low-resolution image 11 from which the source coding has been performed, from the image source coding unit 120 and perform predetermined pre-processing. The pre-processing may include error detection and correction coding, for example, and may also include constellation symbol mapping. However, the first pre-processing unit 130 may not be included in the image transmission apparatus 100 according to an embodiment. In this case, the low-resolution image 11 from the image source coding unit 120 may be transmitted to the first coding unit 140 to be described later. However, hereinafter, it will be described on the premise that the first pre-processing unit 130 is included in the image transmission apparatus 100.

The first coding unit 140 is configured to code the low-resolution image 11 based on the first coding technique by receiving the low-resolution image 11 on which the aforementioned pre-processing is performed, from the first pre-processing unit 130. Here, the first coding technique may be, for example, the Alamouti technique, but is not limited thereto.

On the other hand, the super-resolution unit 160 may be configured to receive a low-resolution image 11 from which the source coding has been performed, from the image source coding unit 120 to increase the super-resolution, that is, the resolution. . For the result of the super-resolution, the hereinafter referred to as a'processed high-resolution image'.

Here, the super-resolution unit 160 may be pre-trained to output a processed high-resolution image when a low-resolution image is input. Hereinafter, the process of learning the super-resolution unit 160 will be described in more detail.

First, learning data for training the super-resolution unit 160 is prepared. The training data can be, for example, the well-known DIV2K dataset. The DIV2K dataset contains multiple original high-resolution images.

Then, although not shown in FIG. 1, each of the plurality of original high-resolution images included in the DIV2K data set is down-sampled by a separate down-sampling module. At this time, according to an embodiment, each of the plurality of original high-resolution images may pass through an anti-aliasing low-pass filter. Thereafter, each of the plurality of original high-resolution images downsampled may be compressed by a separate compression module according to various different compression rates. For example, the compression ratio at this time may be 0.05, 0.1, 0.15, ... 2.0 bpp, but is not limited thereto. As a result, a plurality of low-resolution images compressed with different compression rates can be provided.

Subsequently, the super-resolution unit 160 may be trained by using a plurality of original high-resolution images prepared as described above as separate answers and inputting a plurality of low-resolution images compressed at different compression rates as inputs by separate learning modules. have. Here,'learning' may be machine learning, for example, deep learning, but is not limited thereto.

At this time, the low-resolution images used for learning may be compressed with different compression rates as described above. For example, when there are 100 original high-resolution images, 100 low-resolution images compressed to 2.0 bpp, 100 low-resolution images compressed to 1.95 bpp, and 100 low-resolution images compressed to 0.05 bpp can be provided. have.

In addition, in the learning process, learning may be performed on the super-resolution unit 160 by 300 epoch with a low-resolution image compressed to 2.0 bpp and an original high-resolution image corresponding thereto, and compressed into another value bpp when the training is finished. Learning may be performed on the super-resolution unit 160 as much as 150 epochs with the low-resolution image and the corresponding high-resolution image. Here, the compression rate for the low-resolution image used when learning is performed and the values of the epochs mentioned in each are only exemplary.

Meanwhile, the super-resolution unit 160 may include a plurality of residual block units. At this time, each of the plurality of residual block units may be configured to include a convolutional layer and a ReLU layer, but may be configured not to include a batch normalization layer. By configuring each of the plurality of residual block portions in this way, the memory footprint can be reduced and performance can be further improved.

The residual map providing unit 170 is configured to provide a residual map. The residual map refers to a map in the form of a matrix, which shows the difference between the above-described processed high-resolution image and the above-described original high-resolution image 10. The residual map may be provided based on a convex optimization technique, but is not limited thereto. Here, since the convex optimization itself is already a known technique, a description thereof will be omitted, but in one embodiment, a limitation on the residual map such that a restored high-resolution image closer to the original high-resolution image can be generated by the residual map. (constraint) is introduced.

To look at these limitations, first, the original high-resolution image is U ₀ , the processed high-resolution image is U ₁ , the residual map is R, and each of these U ₀ , U ₁ and R is a matrix of N _HR x N _HR Let's assume. Here, N _HR x N _HR may mean resolutions for U ₀ , U _1, and R, respectively. In this case, if the total number of elements contained in R as N _HR ^2, elements with non-zero values of these elements are up to L may be not only information of (L is a natural number less than N _HR ²⁾ dog. At this time, the value of L may have a value based on a compression rate or a bit budget.

The second pre-processing unit 180 is configured to receive the residual map from the residual map providing unit 170 and perform predetermined pre-processing. The pre-processing may include error detection and correction coding, for example, and may also include constellation symbol mapping. However, the second pre-processing unit 180 may not be included in the image transmission apparatus 100 according to an embodiment. However, hereinafter, it will be described on the premise that the first pre-processing unit 130 is included in the image transmission apparatus 100.

The second coding unit 190 is configured to receive the residual map on which the above-described pre-processing is performed from the second pre-processing unit 180 and code the residual map based on the second coding technique. Here, the second coding technique may be, for example, a golden coding technique, but is not limited thereto.

The first MUX unit 150 receives the low-resolution image 11 that is coded based on the first coding technique from the first coding unit 140, and also uses the second coding technique from the second coding unit 190. After receiving the coded residual map, it is configured to perform a multiplexing operation on those received. The result of the multiplexing operation may be transmitted to the image receiving device through the antenna shown in FIG. 1.

Similarly, the second MUX unit 191 receives the low-resolution image 11 coded based on the first coding technique from the first coding unit 140, and also receives the second coding technique from the second coding unit 190. After receiving the residual map coded on the basis, it is configured to perform a multiplexing operation on those received. The result of the multiplexing operation may be transmitted to the image receiving device through the antenna shown in FIG. 1.

That is, according to one embodiment, in the image transmission apparatus 100, a low-resolution image is generated based on the original high-resolution image, and a low-quality processed high-resolution image is generated based on the generated low-resolution image, and the low-quality processed high-resolution image and A residual map indicating differences between the original high-resolution images described above is generated. In addition, the residual map and the low-resolution image generated as described above are respectively coded based on a predetermined coding technique and transmitted to the image receiving apparatus.

Hereinafter, the image receiving apparatus will be described. As described above, the image receiving device may have a high-end terminal and a low-end terminal, and hereinafter, a high-end terminal will be described first.

2 is a configuration diagram schematically showing a configuration for a high-spec terminal 200 according to an embodiment.

Referring to FIG. 2, the high-end terminal 200 includes a first demux unit 210, a first decoding unit 220, a first post-processing unit 230, an image source decoding unit 240, and a super-resolution unit 250 ), a second demux unit 211, a second decoding unit 260, and a second post-processing unit 270. However, the configuration for the high-spec terminal 200 shown in FIG. 2 is only exemplary. For example, according to an embodiment, the high specification terminal 200 may be implemented or configured to be different from that shown in FIG. 2.

First, the high-spec terminal 200, and components included in each of these high-spec terminals 200, are stored in a memory storing instructions implemented to perform a function to be described below, and a microprocessor executing instructions stored in the memory. It can be implemented by

First, the high-spec terminal 200 receives the two kinds of data described above from the image transmission device 100 through the two antennas shown in FIG. 2. One is a residual map and the other is a low-resolution image. Here, the residual map is a state coded by the first coding technique as described above, and a low-resolution image is a state coded by the second coding technique.

The first demux unit 210 and the second demux unit 211 are the residual map coded by the first coding technique to the first decoding unit 220 and the second decoding unit 260, respectively, and the second coding A low-resolution image coded by the technique is provided.

Then, the first decoding unit 220 performs decoding based on the received data. In decoding, the Alamouti decoding technique may be employed, but is not limited thereto.

The result decoded by the first decoding unit 220 is transmitted to the first post-processing unit 230. The first post-processing unit 230 may be configured to perform constellation symbol mapping, and may also be configured to detect and correct errors.

The result post-processed by the first post-processing unit 230 is transmitted to the image source decoding unit 240. The image source decoding unit 240 may be configured to output a low-resolution image by, for example, performing source decoding on a result received from the first post-processing unit 230.

Meanwhile, the second decoding unit 260 decodes the ones received from the first demux unit 210 and the second demux unit 211. When decoding, a golden coding technique may be employed, but is not limited thereto.

The result decoded by the second decoding unit 260 is transmitted to the second post-processing unit 270. The second post-processing unit 270 may be configured to perform constellation symbol mapping, and may also be configured to detect and correct errors.

The super-resolution unit 250 generates a reconstructed high-resolution image 21 based on the residual map, which is the result from the low-resolution image 20 and the second post-processing unit 270, the output from the image source decoding unit 240. . Here, the process of generating the reconstructed high-resolution image by the super-resolution unit 250 is as shown in Equation 1 below.

[Equation 1]

U ₂ = U ₁ +GR

Here, U2 represents a reconstructed high-resolution image generated by the super-resolution unit 250, U ₁ represents a processed high-resolution image processed from a low-resolution image, R represents a residual map, and G represents a predefined two-dimensional convolution kernel. . Here, G may be a Gaussian convolution filter such as Equation 2 below, but is not limited thereto.

Hereinafter, with reference to FIGS. 1 and 2, after the original high-resolution image is input to the image transmission apparatus 100, the process of outputting the restored high-resolution image from the high-end terminal 200 will be described in the end.

The original high-resolution image 10 is down-sampled by the down-sampling unit 110. At this time, as described above, the original high-resolution image 10 may pass through the anti-aliasing low-pass filter according to the embodiment.

The down-sampling unit 110 outputs the low-resolution image 11 in response to the input of the original high-resolution image 10. Then, the image source coding unit 120 performs predetermined source coding on a result output from the down sampling unit 110.

Then, the first pre-processing unit 130 performs a predetermined pre-processing on the output from the image source coding unit 120, and the first coding unit 140 removes the output from the first pre-processing unit 130. 1 Coding based on coding technique.

Meanwhile, the output from the image source coding unit 120 is also transmitted to the super-resolution unit 160. Then, the super-resolution unit 160 outputs a high-resolution image of low quality in response to this. Then, the residual map providing unit 170 provides a residual map representing the difference in matrix form, based on the difference between the original high-resolution image 10 and the super-resolution image 160 and a low-quality high-resolution image.

The second pre-processing unit 180 performs predetermined pre-processing on the residual map, and the second coding unit 190 pre-processes the result of the second pre-processing unit 180 based on the second coding technique.

Then, the first MUX unit 150 receives the result of the first coding unit 140 and the result of the second coding unit 190 to perform a multiplexing operation, and the second MUX unit 191 also receives the first coding unit. The multiplexing operation is performed by receiving the result of 140 and the result of the second coding unit 190. The multiplexed results are transmitted to the high-spec terminal 200 through the antenna shown in FIG. 1, respectively.

Next, the low-end terminal will be described. 3 is a configuration diagram schematically showing a configuration for a low-spec terminal 300 according to an embodiment.

Referring to FIG. 3, the low-end terminal 300 includes a first demux unit 310, a first decoding unit 320, a first post-processing unit 330, and an image source decoding unit 340. However, the configuration of the low-spec terminal 300 shown in FIG. 3 is only exemplary. For example, depending on the embodiment, the low specification terminal 300 may be implemented or configured to be different from that shown in FIG. 3.

First, the low specification terminal 300, and the components included in each of these low specification terminals 300, are stored in a memory storing instructions implemented to perform a function to be described below, and a microprocessor executing instructions stored in the memory. It can be implemented by

First, the low-spec terminal 300 receives a low-resolution image from the image transmission device 100 through one antenna shown in FIG. 3. Here, the low-resolution image is coded by the second coding technique described above.

The first demux unit 310 provides the coded low-resolution image coded by the second coding technique to the first decoding unit 320.

Then, the first decoding unit 320 performs decoding based on the received data. In decoding, the Alamouti decoding technique may be employed, but is not limited thereto.

The result decoded by the first decoding unit 320 is transmitted to the first post-processing unit 330. The first post-processing unit 330 may be configured to perform constellation symbol mapping, and may also be configured to detect and correct errors.

The result post-processed by the first post-processing unit 330 is transmitted to the image source decoding unit 340. The image source decoding unit 340 may be configured to output a low-resolution image by performing source decoding, for example, on a result received from the first post-processing unit 330.

As described above, according to an embodiment, it is possible to exert optimal performance when viewed from an end-to-end perspective, and in particular, terminals having different screen resolutions while having different numbers of antennas, for example, low specification terminals Even if the high-end terminal and the high-end terminal are in the same area, images can be effectively delivered to each of these terminals.

On the other hand, according to one embodiment, the image transmission method may be performed by the image transmission apparatus 100 according to an embodiment, and the steps that may be included in each of these methods, instructions programmed to perform these steps It may be stored in a computer-readable recording medium for storing the.

The combination of each block in the block diagram and each step of the flowchart attached to the present invention may be performed by computer program instructions. These computer program instructions may be mounted on a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, so that instructions executed through a processor of a computer or other programmable data processing equipment may be used in each block or flowchart of the block diagram. In each step, means are created to perform the functions described. These computer program instructions can also be stored in computer readable or computer readable memory that can be oriented to a computer or other programmable data processing equipment to implement a function in a particular way, so that computer readable or computer readable memory The instructions stored in it are also possible to produce an article of manufacture containing instructions means for performing the functions described in each step of each block or flowchart of the block diagram. Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so a series of operational steps are performed on a computer or other programmable data processing equipment to create a process that is executed by the computer to generate a computer or other programmable data. It is also possible for instructions to perform processing equipment to provide steps for performing the functions described in each block of the block diagram and in each step of the flowchart.

Further, each block or each step can represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function(s). It should also be noted that in some alternative embodiments it is possible that the functions mentioned in blocks or steps occur out of order. For example, two blocks or steps shown in succession may in fact be executed substantially simultaneously, or it is also possible that the blocks or steps are sometimes performed in reverse order depending on the corresponding function.

The above description is merely illustrative of the technical idea, and those of ordinary skill in the art to which the present invention pertains will be capable of various modifications and variations without departing from essential characteristics. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit, but to explain, and the scope of the technical spirit is not limited by these embodiments. The scope of protection should be interpreted by the claims below, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of rights.

Claims (5)

A down-sampling unit that down-samples the original high-resolution image and converts it into a low-resolution image;
A first coding unit that codes the low-resolution image based on a first coding technique that can be decoded by each of a low-end terminal (small user) and a high-end terminal (big user),
When the low-resolution image converted by the down-sampling unit is received, a super-resolution unit pre-trained to output a processed high-resolution image having a lower quality than the original high-resolution image;
A residual map providing unit calculating a difference between the original high-resolution image and the processed high-resolution image and providing a residual map in the form of a residual map;
And a second coding unit that codes the residual map based on a second coding technique that is decodable by the high-end terminal.
Image transmission device. According to claim 1,
The first coding technique,
Alamouti technique
Image transmission device. According to claim 1,
The second coding technique,
Golden coding technique
Image transmission device. According to claim 1,
The super-resolution section,
Containing a convolutional layer and a ReLU layer, but not including a batch normalization layer.
Image transmission device. According to claim 1,
The residual map forming unit,
Providing the residual map based on a convex optimization technique
Image transmission device.

Images