CN117474816B - High dynamic range image tone mapping method, system and readable storage medium - Google Patents
High dynamic range image tone mapping method, system and readable storage medium Download PDFInfo
- Publication number
- CN117474816B CN117474816B CN202311799099.1A CN202311799099A CN117474816B CN 117474816 B CN117474816 B CN 117474816B CN 202311799099 A CN202311799099 A CN 202311799099A CN 117474816 B CN117474816 B CN 117474816B
- Authority
- CN
- China
- Prior art keywords
- dynamic range
- image
- range image
- high dynamic
- loss function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/088—Non-supervised learning, e.g. competitive learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20208—High dynamic range [HDR] image processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
本发明公开了一种高动态范围图像色调映射方法、系统及可读存储介质。所述高动态范围图像色调映射方法包括：通过全局映射将高动态范围图像压缩为低动态范围图像；获取增强神经网络；对低动态范围图像进行在线训练，提取增强因子；通过增强因子进行图像增强，获得映射图像；其中的损失函数选用非线性损失函数，且包括体现迭代图像的图像参数与统计理想值的差异性的第一类损失函数，以及体现迭代图像与高动态范围图像的差异性的第二类损失函数。本发明所达成了由高动态范围图像向低动态范围的显示图像的高质量映射，解决了高动态显示受限的技术难题；通过结合传统的全局映射与基于学习的图像增强，降低了学习的难度和计算代价。
The invention discloses a high dynamic range image tone mapping method, system and readable storage medium. The high dynamic range image tone mapping method includes: compressing the high dynamic range image into a low dynamic range image through global mapping; obtaining an enhanced neural network; performing online training on the low dynamic range image to extract an enhancement factor; and performing image enhancement through the enhancement factor. , to obtain the mapping image; the loss function is a nonlinear loss function, and includes the first type of loss function that reflects the difference between the image parameters of the iterative image and the statistical ideal value, and the first type of loss function that reflects the difference between the iterative image and the high dynamic range image. The second type of loss function. The present invention achieves high-quality mapping from a high dynamic range image to a low dynamic range display image, solving the technical problem of limited high dynamic range display; by combining traditional global mapping and learning-based image enhancement, the learning time is reduced. Difficulty and computational cost.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a high dynamic range image tone mapping method, system, and readable storage medium.
Background
Real scenes tend to have a wide dynamic range and conventional Low Dynamic (LDR) images cannot reproduce the appearance of the real world. In order to meet the requirement of high-quality images, multiple exposure fusion technology is adopted to fuse image information with different exposure levels into a single High Dynamic Range (HDR) image, and the HDR image with a higher and wider dynamic range contains richer scene information, however, due to high cost, the dynamic range supported by many display devices is limited at present, all details of the HDR image are difficult to present, and serious distortion phenomenon occurs in local areas.
In order to solve the problem of the visualization of HDR images, different tone mapping methods have been proposed in the past. Early conventional tone mapping methods had problems with local contrast loss or artifacts. With the rapid development of deep learning, a great breakthrough is made by applying the technical means of deep learning to the field of image processing.
Unlike other image processing problems, however, tone mapping problems do not have ground truth values, and it is difficult to form large amounts of tagged data, which is often required by most deep learning algorithms to learn the mapping relationship of input and output images. Some studies use the best results of multiple conventional tone mapping methods as labels, but the performance of the network is limited by the quality of labels generated by conventional methods.
Therefore, how to avoid the limitation of labels to implement tone mapping of high dynamic range images by using deep learning method is a technical problem to be solved in the art.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a high dynamic range image tone mapping method, a system and a readable storage medium, which solve the technical problems that tone mapping in the field depends on labeled data, a large amount of labeled data is difficult to obtain and the quality of labels is difficult to control.
In order to achieve the purpose of the invention, the technical scheme adopted by the invention comprises the following steps:
in a first aspect, the present invention provides a high dynamic range image tone mapping method based on deep learning, comprising:
providing a high dynamic range image, and compressing the high dynamic range image into a low dynamic range image through global mapping;
acquiring an enhanced neural network under the encoder-decoder structure, wherein the enhanced neural network takes jump connection as a connection mode;
performing on-line training on the low dynamic range image through the enhanced neural network, and extracting an enhancement factor;
image enhancement is carried out on the low dynamic range image through enhancement factors, and a mapping image is obtained;
the nonlinear loss function is selected as the loss function of the online training, and comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of the iterative image formed in the online training process and a second type of loss function reflecting the difference between the iterative image and the high dynamic range image.
Further, the first class of loss functions includes an intensity loss function and/or a contrast loss function.
Further, the second class of loss functions includes any one or a combination of two or more of hue loss functions, saturation loss functions and structure loss functions.
In a second aspect, the present invention also provides a high dynamic range image tone mapping system based on deep learning, comprising:
the global mapping module is used for compressing the high dynamic range image into a low dynamic range image through global mapping;
the network construction module is used for constructing an enhanced neural network under the encoder-decoder structure, and the enhanced neural network takes jump connection as a connection mode;
the on-line training module is used for on-line training the low dynamic range image through the enhanced neural network and extracting an enhanced factor;
the image enhancement module is used for enhancing the image of the low dynamic range image through the enhancement factors to obtain a mapping image;
the nonlinear loss function is selected as the loss function of the online training, and comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of the iterative image formed in the online training process and a second type of loss function reflecting the difference between the iterative image and the high dynamic range image.
In a third aspect, the present invention also provides a readable storage medium having stored therein a computer program which when executed performs the steps of the high dynamic range image tone mapping method described above.
Based on the technical scheme, compared with the prior art, the invention has the beneficial effects that:
aiming at the problem that tone mapping does not have ground truth value and deep learning needs a large number of real labels, the tone mapping method and system provided by the invention designs a group of non-reference loss functions which can be used for tone mapping tasks, effectively guides training of an enhanced neural network under the condition that no matched labels exist, realizes supervision of tone mapping directly through high dynamic range images, finally achieves high-quality mapping from high dynamic range images to low dynamic range display images, and effectively solves the technical problem that high dynamic range images are limited in display; on the other hand, tone mapping is converted into global mapping and image enhancement, and the learning difficulty and the calculation cost are reduced by combining the traditional global mapping and learning-based image enhancement.
The above description is only an overview of the technical solutions of the present invention, and in order to enable those skilled in the art to more clearly understand the technical means of the present application, the present invention may be implemented according to the content of the specification, the following description is given of the preferred embodiments of the present invention with reference to the accompanying drawings.
Drawings
FIG. 1 is a flow chart of a tone mapping method for high dynamic range images according to an exemplary embodiment of the present invention;
FIG. 2 is a diagram showing an exemplary implementation of a tone mapping method for high dynamic range images according to an exemplary embodiment of the present invention;
FIG. 3 is a schematic diagram of an enhanced neural network according to an exemplary embodiment of the present invention;
FIG. 4 is a schematic diagram showing the composition and structure of a tone mapping system for high dynamic range images according to an exemplary embodiment of the present invention
FIG. 5 is a graph showing an example of the effect of different loss functions on image quality in a high dynamic range image tone mapping method according to an exemplary embodiment of the present invention.
Detailed Description
In view of the shortcomings in the prior art, the inventor of the present invention has long studied and practiced in a large number of ways to propose the technical scheme of the present invention. The technical scheme, the implementation process, the principle and the like are further explained as follows.
Specifically, HDR (High-Dynamic Range) is simply a processing technology for improving brightness and contrast of images, and can lighten details of each dark part, darken the dark part, enrich more detail colors, and enable films and pictures to show excellent effects. Let the user more closely look at the visual perception in real environment when watching, this is what HDR exists. The traditional SDR (standard contrast) has the highest brightness of only 100nit, the part higher than 100nit in the picture is distorted (lost), the lowest debugging is 0.1nit, and the part lower than 0.1nit in the picture is lost. The HDR technology is developed, the highest brightness reaches thousands nit, the lowest brightness reaches 0.0005nit, the details of the parts with the brightness higher than 100nit and lower than 0.1nit in the picture are greatly expanded, and meanwhile, the whole picture is more transparent and clear and has rich details. More abundant scene information results in higher dynamic range for HDR, but some real-world devices do not support such high dynamic range displays, require tone mapping to adjust their dynamic range, and maintain as rich a level of detail as possible. In order to enable HDR to be applied in most current dynamic range limited devices, the present invention proposes a high dynamic range image tone mapping method based on deep learning.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced otherwise than as described herein, and therefore the scope of the present invention is not limited to the specific embodiments disclosed below.
Moreover, relational terms such as "first" and "second", and the like, may be used solely to distinguish one from another component or method step having the same name, without necessarily requiring or implying any actual such relationship or order between such components or method steps.
Referring to fig. 1, an embodiment of the present invention provides a high dynamic range image tone mapping method based on deep learning, which includes the steps of:
s1, providing a high dynamic range image, and compressing the high dynamic range image into a low dynamic range image through global mapping.
S2, acquiring an enhanced neural network under the encoder-decoder structure, wherein the enhanced neural network takes jump connection as a connection mode.
S3, performing online training on the low dynamic range image through the enhanced neural network, and extracting an enhanced factor.
S4, performing image enhancement on the low dynamic range image through the enhancement factors to obtain a mapping image.
In the step S3, the loss function of the online training is a nonlinear loss function, and the nonlinear loss function includes a first type of loss function that reflects a difference between an image parameter and a statistical ideal value of an iterative image formed in the online training process, and a second type of loss function that reflects a difference between the iterative image and the high dynamic range image.
The prior art is mainly based on supervised learning methods for tone mapping, and therefore often requires a large number of labeled datasets for training. The technical scheme provided by the invention designs a series of non-reference loss functions by taking the contrast, intensity, saturation, hue and brightness structure of the LDR image into consideration and utilizing the statistical information of the LDR image dataset and a given HDR image based on unsupervised consideration; based on the loss function, the mapping method provided by the invention realizes tone mapping of the HDR image in an on-line training mode, no tag data is needed, various problems caused by the tag data are avoided, and unsupervised tone mapping is realized.
And in particular in some embodiments the statistical ideal may comprise, for example, average parameter information obtained from existing image dataset statistics.
As an example, some of the statistical ideal values and intensities in the first class of loss functions are based on the statistical information of a public dataset Imagenet dataset, i.e. the mean, variance, etc. of the dataset image. Of course, even if statistics are made from other common data sets, it is possible to count the image average parameters of the higher quality image data collected by itself.
In some embodiments, the first class of loss functions includes an intensity loss function and/or a contrast loss function.
In some embodiments, the second class of loss functions includes any one or a combination of two or more of hue loss functions, saturation loss functions, structure loss functions.
In some embodiments, the intensity loss function may be expressed, for example, as:
；
wherein,as strength lossFailure of value, ->For the average intensity of the color channels of the iterative image, < >>The average ideal intensity of the color channels is obtained for statistics.
In some embodiments, the contrast loss function may be expressed, for example, as:
；
wherein,for contrast loss value, +.>For the standard deviation of the color channels of the iterative image,the mean ideal standard deviation of the color channels obtained for statistics.
In some embodiments, the hue loss function may be expressed, for example, as:
；
wherein,is hue loss value, N is total pixel point, i is constant with value of 1-N, < ->For the red-green component of the high dynamic range image in IPT space, +.>For yellow Lan Fenliang of the high dynamic range image in IPT space,for the red-green component of the iterative image in IPT space, +.>For the yellow-blue component of the iterative image in IPT space.
In some embodiments, the saturation loss function may be expressed, for example, as:
；
wherein,for saturation loss value, +_>For the saturation of the ith pixel point in the high dynamic range image +.>And the saturation of the ith pixel point in the iterative image.
In some embodiments, the structural loss function may be expressed, for example, as:
；
；
；
wherein,for the structural loss value, +.>Is a multi-scale structural similarity index>For the low dynamic range image, +.>For the iterative image;Is the weight of the nth scale, sigma x 、σ y Local standard deviation, sigma, between corresponding blocks in the high dynamic range image and the iterative image, respectively xy Cross-correlation coefficients for the high dynamic range image and the iterative image>Is->Global mapped version, C 1 And C 2 Are all stable constants.
With respect to the utilization of a plurality of loss functions, in some embodiments, a plurality of the first type of loss functions and the second type of loss functions are added according to respective weights as the nonlinear loss functions. A specific embodiment may be, for example, a weighted average method, for example, sequentially giving different weights V1-V5 to the intensity loss function, the contrast loss function, the hue loss function, the saturation loss function, and the structure loss function, and then performing an addition operation to obtain a final total nonlinear loss function, so as to guide the online training.
Based on the practical experience of the inventor, in order to achieve better mapping quality and efficiency, the weight values of different loss functions are different, specifically, the weight range of the intensity loss function is generally set to 40-60, the weight range of the contrast loss function is generally set to 1-3, the weight range of the hue loss function is generally set to 80-120, the weight range of the saturation loss function is generally set to 1-3, and the weight range of the structure loss function is generally set to 1-3. Of course, scaling up and down the weight values in equal proportion is still equivalent to the range of values using the weight values.
Of course, the method is not limited to the above-mentioned range, but generally follows a similar rule of magnitude, as shown in fig. 5, the intensity loss and the contrast loss affect the brightness and the contrast, the saturation loss and the hue loss affect the true color of the image, and the structural loss results in loss of details, so based on experience, the invention summarizes and derives the weight range of each loss function applicable to the technical scheme provided by the embodiment of the invention.
While details regarding the remaining steps of the solution provided by the present invention, in some embodiments, the global mapping includes sequentially performing luminance compression and color restoration on the high dynamic range image.
The luminance compression is expressed as:
；
；
wherein,is an asymmetric parameter>For the brightness of the low dynamic range image, < >>For the brightness of the high dynamic range image, < >>For the average luminance of the high dynamic range image, +.>For the maximum brightness of the high dynamic range image,to be the instituteMinimum brightness of the high dynamic range image.
The color recovery is expressed as:
；
wherein,color channel intensity for color restored low dynamic range image, +.>Color channel intensity for high dynamic range images.
In some embodiments, the convolutional layer in the enhanced neural network adopts mixed hole convolution, the output layer in the active layer adopts sigmod as an activation function, and the rest layers adopt LeakyReLU as an activation function.
In some embodiments, the image enhancement employs pixel-level recursive enhancement expressed as:
；
wherein,for the enhancement factor, < >>For the output of the n-th step of recursion enhancement, n is the number of recursions, and when the number of recursions is 1,/is>Is the low dynamic range image input.
Of course, the manner of enhancement should be consistent with the final image enhancement when training is performed.
Corresponding to the above-mentioned high dynamic range image tone mapping method, the embodiment of the invention also provides a high dynamic range image tone mapping system based on deep learning, which comprises:
and the global mapping module is used for compressing the high dynamic range image into a low dynamic range image through global mapping.
And the network construction module is used for constructing an enhanced neural network under the encoder-decoder structure, and the enhanced neural network takes jump connection as a connection mode.
And the online training module is used for carrying out online training on the low dynamic range image through the enhanced neural network and extracting an enhanced factor.
And the image enhancement module is used for carrying out image enhancement on the low dynamic range image through the enhancement factors to obtain a mapping image.
The nonlinear loss function is selected as the loss function of the online training, and comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of the iterative image formed in the online training process and a second type of loss function reflecting the difference between the iterative image and the high dynamic range image.
Correspondingly, the embodiment of the invention also provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed to execute the steps of the high dynamic range image tone mapping method.
The technical scheme of the invention is further described in detail below through a plurality of embodiments and with reference to the accompanying drawings. However, the examples are chosen to illustrate the invention only and are not intended to limit the scope of the invention.
Example 1
The present embodiment illustrates a process of a tone mapping method for a high dynamic range image, specifically as follows:
s1: the image is compressed from a high dynamic range image to a low dynamic range image by global mapping.
S2: an enhanced neural network under the encoder-decoder structure is constructed and takes a jump connection as a connection mode.
S3: the low dynamic range image is trained online through the enhanced neural network to extract the enhancement factors.
S4: the low dynamic range image is pixel-level enhanced by an enhancement factor.
Considering that the brightness at the same pixel point between the low dynamic range image and the high dynamic range image always has a linear relationship, in this embodiment, a compression object for global mapping is selected by using a brightness channel, and the compression is implemented through an asymmetric omentum response model, and the specific expression is as follows:
；
；
in the method, in the process of the invention,is an asymmetric parameter>For the brightness of low dynamic range images, +.>For the brightness of high dynamic range images, +.>For the average luminance of the high dynamic range image, +.>Maximum brightness for high dynamic range image, +.>Is the minimum brightness of the high dynamic range image.
However, in view of the calculation of luminance channels (taking a high dynamic range image as an example,) Is calculated according to the channel intensity of each color channel, thus, the brightness is switched onAfter the channels are compressed, the channels of each color are required to be restored, and the restoration formula is as follows:
；
in the method, in the process of the invention,color channel intensity for low dynamic range image after color recovery,/->Color channel intensity for high dynamic range images.
Further, as shown in fig. 3, in order to improve the receptive field of the enhancement neural network, in this embodiment, it is proposed that the encoder-decoder enhancement neural network constructed uses mixed-hole convolution instead of the common convolution layer, and the convolution kernels are all 3×3 in size and 1 in step length. Besides, the active layer uses the sigmod as an active function except the output layer, and the other layers use the LeakyReLU as an active function. Meanwhile, in order to avoid the problem of gradient disappearance, a jump connection mode is adopted among layers.
To solve the problem of tag loss, the online training in this embodiment is performed under the guidance of a series of nonlinear loss functions, which considers the constraint on training by each loss function, and ensures the reliability and quality of training even without tags. Specific loss functions include an intensity loss function, a contrast loss function, a hue loss function, a saturation loss function, and a structure loss function.
Wherein the intensity loss function is expressed as the following formula:
；
in the method, in the process of the invention,for strength loss->For the average intensity of the color channels of the iterative image formed in the training,is the ideal intensity of the color channel.
The contrast loss function is expressed as the following formula:
；
in the method, in the process of the invention,for contrast loss +.>To the standard deviation of the color channels of the iterative images formed in the training,is an ideal value for the color channel.
The hue loss function is expressed as the following formula:
；
in the method, in the process of the invention,for hue loss, N is the total number of pixels, i is a constant with a value of 1, N, P is the red-green component in IPT space, and T is yellow Lan Fenliang in IPT space (where subscript ldr represents the iterative image formed in training, rather than the low dynamic range image obtained via global mapping).
The saturation loss function is expressed as the following formula:
；
in the method, in the process of the invention,for saturation loss, +_>Saturation for the i-th pixel in the high dynamic range image, < >>Saturation for the i-th pixel in the low dynamic range image (similarly, the subscript ldr represents the iterative image formed in the training, rather than the low dynamic range image obtained via global mapping).
The structural loss function is expressed as the following formula:
；
in the method, in the process of the invention,for structural loss->Is a multi-scale structural similarity index.
；
Is the weight of the nth scale; (the value of n can be set to 5 in general, but can be adjusted up and down
；
Wherein sigma x 、σ y Sum sigma xy The local standard deviation and cross-correlation between corresponding blocks in the HDR and LDR images respectively,is->Reflecting the importance of signal strength, C 1 、C 2 Is a stable constant.
The multiple loss functions are added and integrated according to a certain weight, and the weighted average obtains a total nonlinear loss function to guide the training, specifically, the weight of the intensity loss function 50, the weight of the contrast loss function 1, the weight of the hue loss function 100, the weight of the saturation loss function 1 and the weight of the structure loss function 1 are given in the embodiment.
Finally, the enhancement factors obtained through training carry out pixel-level enhancement under the recursion enhancement on the low dynamic range image, so that the high dynamic range image is finally converted into a high-quality low dynamic range image, and the high dynamic range image is suitable for HDR image presentation of dynamic range limited equipment. The specific formula expression for pixel enhancement is as follows:
；
in the method, in the process of the invention,to enhance the factor->For the output of the n-th recursion enhancement, n is the number of recursions, when the number of recursions is 1 (i.e. the first round of recursion),>is an input low dynamic range image.
In the specific implementation process, referring to fig. 2, taking a high dynamic range image as an example, the display effect is as shown in the first image in fig. 2, and is limited by the dynamic range of the display device, so that details cannot be clearly presented; the second image is obtained through global mapping, so that the dynamic range is reduced, the whole is clearer, and a large number of defects still exist in local details; and the enhancement factors extracted through training are shown as a third image, the second image is enhanced in pixel level by utilizing the enhancement factors, and a final mapping image shown as a fourth chapter image is obtained, so that the local details are full and clear, and the display effect is better.
Example 2
For better understanding of the technical content of the present invention, the present embodiment illustrates the present invention by way of a system structure, as shown in fig. 4, a high dynamic range image tone mapping system based on deep learning, comprising:
and the global mapping module is used for mapping the image from the high dynamic range image to the low dynamic range image.
And the network construction module is used for constructing an enhancement network under the encoder-decoder structure and takes jump connection as a connection mode.
And the online training module is used for carrying out online training on the low dynamic range image through the enhancement network so as to extract the enhancement factors.
And the pixel enhancement module is used for carrying out pixel-level enhancement on the low dynamic range image through the enhancement factors.
Further, in the global mapping module, the global mapping is specifically to compress the brightness channel and then perform color recovery.
Further, in the pixel enhancement module, pixel-level enhancement is achieved by recursive enhancement.
Based on the above embodiments, it can be clear that, according to the high dynamic range image tone mapping method and system provided by the embodiments of the present invention, a set of non-reference loss functions that can be used for tone mapping tasks are designed, training of an enhanced neural network is effectively guided without pairing tags, supervision of tone mapping directly through high dynamic range images is achieved, and finally high quality mapping from high dynamic range images to low dynamic range display images is achieved, and the technical problem that high dynamic range images are limited in display is effectively solved; on the other hand, tone mapping is converted into global mapping and image enhancement, and the learning difficulty and the calculation cost are reduced by combining the traditional global mapping and learning-based image enhancement.
It should be understood that the above embodiments are merely for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the present invention and implement the same according to the present invention without limiting the scope of the present invention. All equivalent changes or modifications made in accordance with the spirit of the present invention should be construed to be included in the scope of the present invention.
It should be noted that all directional indicators (such as up, down, left, right, front, and rear … …) in the embodiments of the present invention are merely used to explain the relative positional relationship, movement, etc. between the components in a particular posture (as shown in the drawings), and if the particular posture is changed, the directional indicator is changed accordingly.
Furthermore, descriptions such as those referred to herein as "first," "second," "a," and the like are provided for descriptive purposes only and are not to be construed as indicating or implying a relative importance or an implicit indication of the number of features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.
In the present invention, unless specifically stated and limited otherwise, the terms "connected," "affixed," and the like are to be construed broadly, and for example, "affixed" may be a fixed connection, a removable connection, or an integral body; can be mechanically or electrically connected; either directly or indirectly, through intermediaries, or both, may be in communication with each other or in interaction with each other, unless expressly defined otherwise. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
In addition, the technical solutions of the embodiments of the present invention may be combined with each other, but it is necessary to be based on the fact that those skilled in the art can implement the technical solutions, and when the technical solutions are contradictory or cannot be implemented, the combination of the technical solutions should be considered as not existing, and not falling within the scope of protection claimed by the present invention.
Claims (6)
1. A high dynamic range image tone mapping method based on deep learning, comprising:
providing a high dynamic range image, and compressing the high dynamic range image into a low dynamic range image through global mapping;
acquiring an enhanced neural network under the encoder-decoder structure, wherein the enhanced neural network takes jump connection as a connection mode;
performing on-line training on the low dynamic range image through the enhanced neural network, and extracting an enhancement factor;
image enhancement is carried out on the low dynamic range image through enhancement factors, and a mapping image is obtained;
the method comprises the steps that a nonlinear loss function is selected as a loss function of online training, and the nonlinear loss function comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of iterative images formed in the online training process and a second type of loss function reflecting the difference between the iterative images and the high dynamic range images;
the first class of loss functions includes an intensity loss function and a contrast loss function; the second class of loss functions comprises hue loss functions, saturation loss functions and structure loss functions;
the intensity loss function is expressed as:
；
wherein,for the intensity loss value, +.>Is saidAverage intensity of color channels of iterative image, +.>The average ideal intensity of the color channels obtained for statistics;
the contrast loss function is expressed as:
；
wherein,for contrast loss value, +.>Standard deviation for color channel of the iterative image, +.>The average ideal standard deviation of the color channels obtained for statistics;
the hue loss function is expressed as:
；
wherein,is hue loss value, N is total pixel point, i is constant with value of 1-N, < ->For the red-green component of the high dynamic range image in IPT space, +.>Yellow Lan Fenliang in IPT space for the high dynamic range image, +.>For the red-green component of the iterative image in IPT space, +.>Yellow Lan Fenliang in IPT space for the iterative image;
the saturation loss function is expressed as:
；
wherein,for saturation loss value, +_>For the saturation of the ith pixel point in the high dynamic range image +.>The saturation of the ith pixel point in the iterative image is obtained;
the structural loss function is expressed as:
；
；
；
wherein,for the structural loss value, +.>Is a multi-scale structural similarity index>For the low dynamic range image, +.>For the iterative image;Is the weight of the q-th scale, sigma x 、σ y Local standard deviation, sigma, between corresponding blocks in the high dynamic range image and the iterative image, respectively xy For the cross-correlation coefficients of the high dynamic range image and the iterative image, < >>Is->Global mapped version, C 1 And C 2 Are all stable constants;
adding the first class of loss functions and the second class of loss functions according to corresponding weights to serve as the nonlinear loss functions;
the image enhancement employs pixel-level recursive enhancement expressed as:
；
wherein,for the enhancement factor, < >>For the output of the mth step of recursion enhancement, m is the number of recursions, and when the number of recursions is 1, +.>Is the low dynamic range image input.
2. The high dynamic range image tone mapping method of claim 1, wherein said statistical idealities comprise average parameter information obtained from existing image dataset statistics.
3. The high dynamic range image tone mapping method according to claim 1, wherein the global mapping comprises sequentially performing luminance compression and color recovery on the high dynamic range image;
the luminance compression is expressed as:
；
；
wherein,is an asymmetric parameter>For the brightness of the low dynamic range image, < >>For the brightness of the high dynamic range image, < >>For the average luminance of the high dynamic range image, +.>For the maximum brightness of the high dynamic range image, < >>Minimum brightness for the high dynamic range image;
the color recovery is expressed as:
；
wherein,color channel intensity for color restored low dynamic range image, +.>Color channel intensity for high dynamic range images.
4. The high dynamic range image tone mapping method of claim 1, wherein the convolutional layer in the enhanced neural network employs mixed-hole convolutional, the output layer in the active layer employs sigmod as the active function, and the remaining layers employ LeakyReLU as the active function.
5. A deep learning-based high dynamic range image tone mapping system for implementing the high dynamic range image tone mapping method of any of claims 1-4, comprising:
the global mapping module is used for compressing the high dynamic range image into a low dynamic range image through global mapping;
the network construction module is used for constructing an enhanced neural network under the encoder-decoder structure, and the enhanced neural network takes jump connection as a connection mode;
the on-line training module is used for on-line training the low dynamic range image through the enhanced neural network and extracting an enhanced factor;
the image enhancement module is used for enhancing the image of the low dynamic range image through the enhancement factors to obtain a mapping image;
the nonlinear loss function is selected as the loss function of the online training, and comprises a first type of loss function reflecting the difference between the image parameters and the statistical ideal values of the iterative image formed in the online training process and a second type of loss function reflecting the difference between the iterative image and the high dynamic range image.
6. A readable storage medium, characterized in that the readable storage medium has stored therein a computer program which, when executed, performs the steps of the high dynamic range image tone mapping method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311799099.1A CN117474816B (en) | 2023-12-26 | 2023-12-26 | High dynamic range image tone mapping method, system and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311799099.1A CN117474816B (en) | 2023-12-26 | 2023-12-26 | High dynamic range image tone mapping method, system and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117474816A CN117474816A (en) | 2024-01-30 |
| CN117474816B true CN117474816B (en) | 2024-03-12 |
Family
ID=89625930
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311799099.1A Active CN117474816B (en) | 2023-12-26 | 2023-12-26 | High dynamic range image tone mapping method, system and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117474816B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119671881A (en) * | 2024-12-16 | 2025-03-21 | 湖南大学 | Image tone mapping method and system based on unpaired learning of generative adversarial network |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015180854A1 (en) * | 2014-05-28 | 2015-12-03 | Koninklijke Philips N.V. | Methods and apparatuses for encoding an hdr images, and methods and apparatuses for use of such encoded images |
| CN108198152A (en) * | 2018-02-07 | 2018-06-22 | 广东欧珀移动通信有限公司 | Image processing method and device, electronic equipment, computer readable storage medium |
| CN108681991A (en) * | 2018-04-04 | 2018-10-19 | 上海交通大学 | Based on the high dynamic range negative tone mapping method and system for generating confrontation network |
| CN109410126A (en) * | 2017-08-30 | 2019-03-01 | 中山大学 | A kind of tone mapping method of details enhancing and the adaptive high dynamic range images of brightness |
| CN110163808A (en) * | 2019-03-28 | 2019-08-23 | 西安电子科技大学 | A kind of single frames high dynamic imaging method based on convolutional neural networks |
| CN110197463A (en) * | 2019-04-25 | 2019-09-03 | 深圳大学 | High dynamic range image tone mapping method and its system based on deep learning |
| CN110796595A (en) * | 2019-10-31 | 2020-02-14 | 北京大学深圳研究生院 | Tone mapping method and device and electronic equipment |
| CN111709900A (en) * | 2019-10-21 | 2020-09-25 | 上海大学 | A High Dynamic Range Image Reconstruction Method Based on Global Feature Guidance |
| CN112837229A (en) * | 2019-11-25 | 2021-05-25 | 英特尔公司 | Unsupervised training of neural networks for high dynamic range image compression |
| CN113095991A (en) * | 2021-04-15 | 2021-07-09 | 电子科技大学 | High dynamic range image steganography and recovery method based on deep learning |
| CN113822830A (en) * | 2021-08-30 | 2021-12-21 | 天津大学 | Multi-exposure image fusion method based on depth perception enhancement |
| CN115526803A (en) * | 2022-10-14 | 2022-12-27 | 中国石油大学(华东) | Non-uniform illumination image enhancement method, system, storage medium and device |
| CN115619682A (en) * | 2022-11-04 | 2023-01-17 | 南京大学 | Denoising tone mapping method and device based on deep learning |
| CN116563183A (en) * | 2023-04-06 | 2023-08-08 | 北京理工大学 | High dynamic range image reconstruction method and system based on a single RAW image |
-
2023
- 2023-12-26 CN CN202311799099.1A patent/CN117474816B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015180854A1 (en) * | 2014-05-28 | 2015-12-03 | Koninklijke Philips N.V. | Methods and apparatuses for encoding an hdr images, and methods and apparatuses for use of such encoded images |
| CN109410126A (en) * | 2017-08-30 | 2019-03-01 | 中山大学 | A kind of tone mapping method of details enhancing and the adaptive high dynamic range images of brightness |
| CN108198152A (en) * | 2018-02-07 | 2018-06-22 | 广东欧珀移动通信有限公司 | Image processing method and device, electronic equipment, computer readable storage medium |
| CN108681991A (en) * | 2018-04-04 | 2018-10-19 | 上海交通大学 | Based on the high dynamic range negative tone mapping method and system for generating confrontation network |
| CN110163808A (en) * | 2019-03-28 | 2019-08-23 | 西安电子科技大学 | A kind of single frames high dynamic imaging method based on convolutional neural networks |
| CN110197463A (en) * | 2019-04-25 | 2019-09-03 | 深圳大学 | High dynamic range image tone mapping method and its system based on deep learning |
| CN111709900A (en) * | 2019-10-21 | 2020-09-25 | 上海大学 | A High Dynamic Range Image Reconstruction Method Based on Global Feature Guidance |
| CN110796595A (en) * | 2019-10-31 | 2020-02-14 | 北京大学深圳研究生院 | Tone mapping method and device and electronic equipment |
| CN112837229A (en) * | 2019-11-25 | 2021-05-25 | 英特尔公司 | Unsupervised training of neural networks for high dynamic range image compression |
| CN113095991A (en) * | 2021-04-15 | 2021-07-09 | 电子科技大学 | High dynamic range image steganography and recovery method based on deep learning |
| CN113822830A (en) * | 2021-08-30 | 2021-12-21 | 天津大学 | Multi-exposure image fusion method based on depth perception enhancement |
| CN115526803A (en) * | 2022-10-14 | 2022-12-27 | 中国石油大学(华东) | Non-uniform illumination image enhancement method, system, storage medium and device |
| CN115619682A (en) * | 2022-11-04 | 2023-01-17 | 南京大学 | Denoising tone mapping method and device based on deep learning |
| CN116563183A (en) * | 2023-04-06 | 2023-08-08 | 北京理工大学 | High dynamic range image reconstruction method and system based on a single RAW image |
Non-Patent Citations (4)
| Title |
|---|
| High Dynamic Range Image Tone Mapping: Literature review and performance benchmark;Xueyu Han;Digital Signal Processing;20230529;1-20 * |
| Real-time self-supervised tone curve estimation for HDR image;Xiyu Chen 等;Computers & Graphics;20230722;461-471 * |
| 基于亮度自适应分段的高动态图像色调映射算法;刘颖;王倩;刘卫华;;电视技术;20180105(01);24-30 * |
| 高动态范围图像色调映射算法研究;贾爱秀;中国优秀硕士学位论文全文数据库 信息科技辑;20200715(第07期);I138-612 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117474816A (en) | 2024-01-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10970830B2 (en) | Image style conversion method, apparatus and device | |
| CN113628152B (en) | A dark-light image enhancement method based on selective fusion of multi-scale features | |
| CN114067018B (en) | An infrared image colorization method based on dilated residual generative adversarial network | |
| CN108830813A (en) | A kind of image super-resolution Enhancement Method of knowledge based distillation | |
| CN109671023A (en) | A kind of secondary method for reconstructing of face image super-resolution | |
| CN111951164B (en) | An image super-resolution reconstruction network structure and image reconstruction effect analysis method | |
| CN107016415B (en) | A Color Image Semantic Classification Method Based on Fully Convolutional Networks | |
| CN112801904B (en) | Hybrid degraded image enhancement method based on convolutional neural network | |
| CN111681177A (en) | Video processing method and device, computer readable storage medium and electronic equipment | |
| CN109920012A (en) | Image coloring system and method based on convolutional neural network | |
| CN115393227A (en) | Self-adaptive enhancing method and system for low-light-level full-color video image based on deep learning | |
| CN115063318A (en) | Low-light image enhancement method and related equipment based on adaptive frequency decomposition | |
| CN111783494A (en) | Recovery method of damaged two-dimensional code with convolutional autoencoder combined with binary segmentation | |
| CN120014256A (en) | Image semi-supervised semantic segmentation method and system based on pixel-level correction | |
| CN117635478B (en) | Low-light image enhancement method based on spatial channel attention | |
| CN116958534A (en) | An image processing method, image processing model training method and related devices | |
| CN113592765A (en) | Image processing method, device, equipment and storage medium | |
| CN117474816B (en) | High dynamic range image tone mapping method, system and readable storage medium | |
| CN111079864A (en) | Short video classification method and system based on optimized video key frame extraction | |
| CN116596793A (en) | A low-light image enhancement method based on U-shaped network and attention mechanism | |
| US20210004699A1 (en) | Learning apparatus, inferring apparatus, learning method, program, and inferring method | |
| CN112950596A (en) | Tone mapping omnidirectional image quality evaluation method based on multi-region and multi-layer | |
| CN117522903A (en) | SF-Unet model-based high-resolution cultivated land remote sensing image segmentation method | |
| US12309536B2 (en) | Image processing method and apparatus, computer device, and storage medium | |
| CN110489584B (en) | Image classification method and system based on densely connected MobileNets model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |