TWI497448B - Method and system for image enhancement - Google Patents

Description

Image enhancement method and system thereof

The present invention relates to an image processing method and system thereof, and more particularly to an image enhancement method and system thereof.

For the inheritance and promotion of art, the work of art restoration has received more and more attention. With the rapid development of science and technology, the use of scientific instruments combined with the analysis of materials and chemistry to repair damaged paintings and other works of art can make the restored color layer rich, the color tone becomes clear, and the brush strokes rejuvenate.

However, the art restoration mentioned above is more expensive and time consuming, and there is a risk of damaging the painting. In addition, traditional physical images such as general photos, posters, and graphic creations are also depleted with external environmental factors such as time, temperature, humidity, and lighting. Insufficient color performance of such entity images can only be re-created and is not economical. For solid images that cannot be repaired, they can only be let go. Therefore, how to quickly repair and enhance physical images at low cost has become an urgent problem to be solved.

The present invention provides an image enhancement method and an image enhancement system that can enhance a solid image.

The present invention provides an image enhancement method comprising the following steps. First, the physical image is captured to produce a to-be-processed image. Next, the image to be processed is subjected to an enhancement correction process to generate a projected image. The projected image is then overlaid onto the solid image to enhance the solid image.

In an embodiment of the invention, before the step of performing an enhancement correction process on the image to be processed to generate a projected image, the image enhancement method further includes the following steps. Capture an image of the calibrator to produce a corrected image. The corrected image is processed to produce a plurality of image capture correction parameters of the corrected image. The corrected image is subjected to an inverse twist operation according to the image capturing correction parameter to generate an inverse distortion corrected image. The anti-warp corrected image is projected onto the plane, and the inverse distortion corrected image projected on the plane is captured to generate a projected corrected image. According to the image capturing correction parameter, the projection corrected image is subjected to an inverse twisting operation to generate an inverse twisted projection corrected image. A set of affine matrix parameters are obtained based on the anti-warp corrected image and the anti-warp projection corrected image.

In an embodiment of the present invention, the step of performing an enhancement correction process on the image to be processed to generate a projected image includes performing an image enhancement operation on the image to be processed according to the image enhancement algorithm to generate an enhanced image; Like the correction parameters, the enhanced image is subjected to an inverse distortion operation to generate an inverse distortion enhanced image; and the inverse distortion enhanced image is affine-calculated according to the affine matrix parameter to generate a projected image.

In an embodiment of the invention, after the step of overlaying the projected image onto the solid image to enhance the solid image, the image enhancement method further comprises the following steps. The pixels of the overlay image projected onto the superimposed image on the solid image are analyzed to determine whether the pixel of the superimposed image conforms to the desired data. When the pixels of the superimposed image do not meet the desired data, the physical image is captured to generate a to-be-processed image, and the image to be processed is subjected to an enhancement correction process to reproduce the projected image and to nest the projected image to the solid image. Reproduce the overlay image.

In an embodiment of the present invention, when the pixel of the processed image does not conform to the desired data, the image enhancement method further includes the following steps before the step of capturing the physical image to regenerate the image to be processed. Determine if the brightness of the projected image has reached saturation. When the brightness of the projected image has reached saturation, the projected image is nested to the solid image.

The present invention provides an image enhancement system, including: an image capture device, an image processing device, and a projection device, wherein the image processing device is coupled to the image capture device and the projection device. The image capturing device is configured to capture a physical image to generate a to-be-processed image. The image processing device is configured to perform an enhancement correction process on the image to be processed to generate a projected image. The projection device is used to nest the projected image onto the solid image to enhance the solid image.

In an embodiment of the invention, the image capturing device further captures an image of the calibrator to generate a corrected image. The image processing device processes the corrected image to generate a plurality of image capturing correction parameters of the corrected image, and then performs an inverse twisting operation on the corrected image according to the image capturing correction parameter to generate an inverse distortion correction image. For example, according to the image capturing correction parameter, the corrected image is subjected to an inverse twisting operation to generate an inverse distortion corrected image. The projection device described above projects the anti-warp corrected image onto the plane and captures the inverse distortion corrected image projected on the plane to generate a projected corrected image. The image processing device described above obtains a set of affine matrix parameters according to the anti-warp corrected image and the anti-twist projection corrected image.

In an embodiment of the present invention, the image processing apparatus performs an image enhancement operation on the image to be processed according to the image enhancement algorithm to generate an enhanced image, and performs an inverse distortion operation on the enhanced image according to the image capturing correction parameter. To generate an anti-twist enhancement image, and to perform an affine operation on the inverse distortion enhancement image according to the affine matrix parameters to generate a projection image.

In an embodiment of the invention, the image processing device analyzes the pixels of the superimposed image on the solid image by the overlaid projection image to determine whether the pixel of the superimposed image conforms to the desired data. When the image processing device determines that the pixel of the superimposed image does not meet the desired data, the image capturing device captures the physical image to regenerate the image to be processed, and the image processing device performs an intensification correction process to regenerate the projected image, and the projection device will The projected image is nested onto the solid image to reproduce the overlay image.

In an embodiment of the invention, the image processing apparatus further determines whether the brightness of the projected image has reached saturation. When the image processing device determines that the brightness of the projected image has reached saturation, the projection device nests the projected image to the solid image.

Based on the above, the image enhancement method and the image enhancement system provided by the present invention, the image capture device can capture the physical image to generate a to-be-processed image. Image area In addition to performing image enhancement on the image to be processed, the device performs image correction on the image distortion caused by the image capturing device and the projection device to generate a projected image. In addition, using the concept of augmented projection, the projection device nests the projected image onto the solid image for the purpose of solid image enhancement.

The above described features and advantages of the invention will be apparent from the following description.

100‧‧‧Image Enhancement System

110‧‧‧Image capture device

120‧‧‧Image processing device

130‧‧‧Projection device

A‧‧‧ entity image

S201~S205‧‧‧Process of image enhancement method

S301~S311‧‧‧ Flow of calibration parameter calculation method

S401~S413‧‧‧Process of image enhancement method

FIG. 1 is a schematic diagram of an image enhancement system according to an embodiment of the invention.

2 is a flow chart of an image enhancement method according to an embodiment of the invention.

FIG. 3 is a flow chart of a method for calculating a correction parameter according to an embodiment of the invention.

FIG. 4 is a flow chart of an image enhancement method according to an embodiment of the invention.

FIG. 1 is a schematic diagram of an image enhancement system according to an embodiment of the invention, but is for convenience of description and is not intended to limit the present invention. First, Figure 1 first introduces all the components and configuration relationships of the image enhancement system. The detailed functions will be disclosed in conjunction with Figure 2.

The image enhancement device 100 includes an image capture device 110 , an image processing device 120 , and a projection device 130 . The image processing device 120 is coupled to the image capture device 110 and the projection device 130 . In this embodiment, the image enhancement system 100 can perform image enhancement on the physical image A to increase the performance of the solid image A contrast and color. The physical image A can be a dynamic or static image presented by a photo, a poster, a painting, an e-book, a digital photo frame, or a display, and is not limited herein.

In the embodiment, the image capturing device 110 may be a camera using a charge coupled device (CCD) lens, but the invention is not limited thereto. The image capturing device 110 can capture the image of the physical image A, so that the image processing device 120 and the projection device 130 perform image enhancement processing on the physical image A.

In the embodiment, the image processing device 120 can be a personal computer, a notebook computer, a smart phone, or a tablet computer, and the invention is not limited thereto. The image processing device 120 includes a memory and a processor. The memory is used to store images captured by the image capture device 110, and the processor is used to process images stored in the memory. In addition, the image processing device 120 can obtain the image captured by the image capturing device 110 by means of wired transmission or wireless transmission.

In the present embodiment, the projection device 130 may be a video projector such as a mini projector, but the present invention is not limited thereto. In other embodiments, a device of a more suitable specification may be employed under consideration of different resolutions and different brightness comparisons. In addition, the projection device 130 can receive the image processed by the image processing device 120 by means of wired transmission or wireless transmission, and enhance the physical image A by using the concept of augmented projection.

In another embodiment, the image capturing device 110, the image processing device 120, and the projection device 130 can be integrated into the same device, such as a personal computer with a camera, a photographing, and a projection function, a notebook computer, a smart phone, and a tablet. Computer, the invention is not limited thereto.

2 is a flow chart of an image enhancement method according to an embodiment of the invention, and the image enhancement method of FIG. 2 can be implemented by the components of the image enhancement system 100 of FIG.

Referring to FIG. 1 and FIG. 2 simultaneously, first, the image capturing device 110 captures the physical image A to generate a to-be-processed image (step S201). The "image to be processed" here is the original image generated by the image capturing device 110 after capturing the solid image A.

Next, the image processing device 120 performs a correction enhancement process on the image to be processed to generate a projected image (step S203). That is, after receiving the image to be processed from the image capturing device 110, the image processing device 120 corrects the image distortion caused by the image capturing device 110 and the projection device 130 in addition to the image enhancement processing for the image to be processed. . The detailed description will be disclosed together in the subsequent embodiments.

Thereafter, the projection device 130 nests the projected image onto the solid image A to enhance the solid image A (step S205). In detail, the projection image processed by the projection device 130 from the image processing device 120 uses the concept of augmented projection to nest the projected image onto the solid image A, so that the projected image combined after being projected onto the solid image A is further It has substantial meaning.

In this embodiment, the distance between the image capturing device 110 and the projection device 130 in the image enhancement system 100 and the physical image A may be based on, for example, the size of the physical image A and the CCD lens aperture of the image capturing module 110. The size, and the resolution of the imagery have changed. After the components in the image intensifying system 100 are adjusted, the image capturing device 110 and the projection device 130 perform a correction parameter calculation program before the image processing device 120 processes the image to be processed, and the calibration parameter calculation program is performed. The generated correction parameters are available to the image processing device 120 to perform step S203, so that the projected image projected by the projection device 130 in the subsequent step can correspond to the position of the physical image A.

FIG. 3 is a flow chart of a method for calculating a correction parameter according to an embodiment of the invention.

Referring to FIG. 1 and FIG. 3 simultaneously, first, the image capturing device 110 captures an image of the calibrated object to generate a corrected image (step S301). In detail, in the present embodiment, the image enhancement system 100 performs a correction parameter calculation program for the image capturing device 110 and the projection device 130 using the calibrator and the tablet. In this embodiment, the calibrator is a slab having a reference pattern, and the reference pattern may be, for example, a checkerboard. In addition, the calibrator further includes at least four markers attached to the surface. In the present embodiment, the number of feature points is four, and the four feature points are arranged in a square form at the periphery of the reference pattern of the calibrator. After the image capturing device 110 captures the image of the calibrator, a corrected image having four feature points is generated.

Next, the image processing device 120 processes the corrected image to generate a plurality of image capturing correction parameters of the corrected image (step S303). The main purpose of this step The image capturing device 110 first corrects the deformation of the captured image, and the image capturing parameter is the intrinsic parameters and the extrinsic parameters of the image capturing device 110. Internal parameters can be used to describe the conversion relationship between camera coordinates and image coordinates, that is, the camera coordinates are projected onto the projective plane using the pinhole imaging principle. For example, the internal parameters include parameters such as focal length, image center, principal point, and lens distortion coefficient, that is, geometry and optics inside the image capturing device 110. characteristic. On the other hand, external parameters are used to describe the conversion relationship between the three-dimensional world coordinate and the three-dimensional camera coordinates. For example, the position and the shooting direction of the image capturing device 110 in the three-dimensional coordinates include a rotation matrix and a translation vector, and the parameters related to the image capturing device 110 placement position and the shooting direction.

Next, the image processing device 120 performs an inverse twisting operation on the corrected image according to the image capturing correction parameter to generate an anti-warping corrected image (step S305). For example, the image capturing correction parameter in this embodiment may be an internal parameter K and a distortion parameter k. The internal parameter K is a 3×3 matrix representing the relationship of the camera coordinates to the image coordinates, and the distortion parameter k is usually represented by a polynomial for describing the barrel or pincushion deformation caused by the lens. That is to say, after the image processing device 120 performs the inverse distortion operation on the corrected image by using the image capturing correction parameter, the generated reverse distortion corrected image can correct the image back to the approximate stitch. Mathematical model of hole projection.

Next, the projection device 130 projects the anti-warping corrected image onto the plane, and the image capturing device 110 captures the anti-warping corrected image projected on the plane to generate a projected corrected image (step S307). The main purpose of this step is to obtain the parameters of the keystone distortion produced by the projection device 130 for the anti-warp corrected image in a subsequent step.

Next, the image processing device 120 performs an inverse twisting operation on the projection corrected image again according to the image capturing correction parameter to generate an inverse twist projection corrected image (step S309). The image processing device 120 further acquires a plurality of affine matrix parameters based on the inverse distortion corrected image and the inverse distortion projection corrected image (step S311). The affine matrix parameters are mainly calculated by the geometric distribution of the four feature points in the anti-warping corrected image and the anti-warping projection corrected image to obtain the difference between the images before and after the projection.

In detail, after the image capturing device 110 obtains the projected corrected image, the image processing device 120 may first perform the inverse twisting operation on the projected corrected image to correct the image back to the mathematical model of the approximate pinhole projection. After obtaining the inverse distortion projection corrected image, the image processing device 120 can obtain the coordinate correspondence relationship between the four feature points in the reverse distortion corrected image and the reverse distortion projection corrected image. The coordinate correspondence can be described by an affine transformation matrix, and the above affine matrix parameters are elements within the affine matrix.

In the present embodiment, the four feature points are arranged in a square form. When performing the affine operation, the image processing apparatus 120 must affix the pixels belonging to the square in the inverse warped projection corrected image to the square plane of the inverse distortion corrected image. The upper side of the square in the anti-warp corrected image is overlapped with the squared edge in the anti-warp corrected image. In other embodiments, the coordinate correspondence may also be based on, for example, a projection transformation matrix, a similarity transformation matrix, and a Euclid according to the degree of deformation of the projected image by the projection device 130. A matrix of other degrees of freedom, such as an Euclidean transformation matrix, is used for correction.

The image processing device 120 can correct the image to be processed according to the image capturing correction parameter and the affine matrix parameter, so that the projected image and the solid image A are accurately aligned.

FIG. 4 is a flow chart of an image enhancement method according to an embodiment of the invention.

Referring to FIG. 1 and FIG. 4, after the image capturing device 110 captures the physical image A to generate a to-be-processed image (step S401), the image processing device 120 will process the image enhancement algorithm according to an image enhancement algorithm. The image is subjected to an image enhancement operation to generate a fortified image (step S403). The image enhancement algorithm here can be edge detection, contrast enhancement, and color enhancement. In another embodiment, the image enhancement algorithm may be an image stylization of the entity image A by means of special effects. For example, the image stylization here may be converting the solid image A into a watercolor-style image, a mosaic tile effect image, or a charcoal drawing style image, etc., and the present invention is not limited thereto.

Taking the Canny edge-detection algorithm as an example of image enhancement, it uses a Gaussian filter to smooth the pixels of the image to be processed, and then uses a Gaussian function. The first derivative function of the Gaussian distribution function) is used to reduce the interference caused by noise and find out the wide-width features of the image to be processed. Because the Canny edge detection algorithm has accurate edge detection, positioning capability, and single response, the image processing apparatus 120 of this embodiment uses this algorithm to perform image enhancement on the image to be processed.

The image processing device 120 finds out the sleek features of the image to be processed, and considers the superposition operation of the solid image A after the ray is projected. In one embodiment, the image processing device 120 can set the degree of image enhancement according to the ambient light source; in another embodiment, the degree of image enhancement can also be preset according to the needs of the user.

Next, the image processing device 120 performs an inverse twist operation on the enhanced image based on the image capturing correction parameter to generate an inverse twist enhanced image (step S405). This step mainly converts the enhanced image into an anti-twist enhanced image without deformation effect according to the image capturing correction parameter of FIG. 3, wherein the anti-twist enhanced image is a mode conforming to the ideal pinhole imaging.

Next, the image processing device 120 performs an affine operation on the inverse distortion enhanced image based on the affine matrix parameters to generate a projected image (step S407). This step mainly converts the anti-twist enhanced image into a projected image according to the affine matrix parameter of FIG. 3, so that the coordinates of the projected image completely correspond to the coordinates of the solid image A to correct the trapezoidal distortion caused by the projection device 130.

It should be particularly noted that step S403, step S405, and step S407 are the enhancement correction procedures described in step S203 of FIG. In another embodiment, the image enhancement system 100 may perform the step S403 after performing the step S405 and the step S407, that is, the step of performing image enhancement after the step of performing image correction, and the invention is not limited thereto.

Thereafter, the projection device 130 nests the projected image onto the solid image A to generate a superimposed image (step S409). Compared with the physical image A, the Canny edge detection algorithm is taken as an example of image enhancement, and the widened lines of the superimposed image will be more obvious. For another example, if the solid image A is a static color painting, the content is a red apple with a color that is not bright enough. In contrast to the contrast enhancement or color enhancement algorithm as an image enhancement method, the apple color in the superimposed image will be more than the entity. The apple in image A is redder and more vivid. In addition, if the physical image A is a landscape image, and the solid image A image is stylized as an image enhancement method in a special effect, the superimposed image will become a mosaic tile effect or a different style of image such as watercolor.

It is worth mentioning that in the present embodiment, the overall contrast, chroma, and the like of the superimposed image can be further adjusted. In other words, the image processing device 120 can further analyze the pixels of the superimposed image to determine whether the pixels of the superimposed image conform to the desired data (step S411). In this embodiment, when the pixels of the superimposed image do not meet the desired data, the image capturing device 110 will retrieve the physical image A, and the image processing device 120 will regenerate the new projected image. In other words, the image enhancement system 100 when the pixels of the superimposed image do not meet the desired data Steps S401 to S409 are re-executed. When the pixels of the superimposed image meet the desired data, the original projected image is used, and at this time, the image processing system 100 ends the flow of the image intensifying method.

In detail, in step S411, the image capturing device 110 can capture the superimposed image, and the image processing device 110 analyzes the pixels of the superimposed image. For example, the image processing device 120 may calculate a comparison value according to a histogram of all pixels of the superimposed image, and then determine whether the comparison value is higher than a preset expected value. When the comparison value does not meet the preset expected value, the image capturing device 110 first retrieves the physical image A to regenerate a to-be-processed image. The image processing device 120 may perform a reinforcement correction process by gradually increasing the contrast degree to generate a new projection image until the contrast value of the new projection image and the solid image in the superimposed image generated in step S411 satisfies a preset expected value. The so-called "stepwise accumulation of contrast" may, for example, be a fine adjustment of the contrast value such that the dark pixels are darker and the bright pixels are brighter. In another embodiment, the image processing device also analyzes the chroma values or other characteristics of all pixels of the superimposed image, and the present invention is not limited thereto.

However, in the present embodiment, since the dynamic range of the projection device 130 is insufficient, it is possible that the above-described contrast value cannot satisfy the preset desired value. In this case, the image processing device 120 can further perform a determination mechanism to determine whether the projected image has reached a state in which the brightness is saturated (step S413). The brightness saturation here is also the maximum brightness that the projection device 130 can project the projected image. When the image processing device 120 determines that the brightness of the projected image has reached saturation, the image processing device 120 is no longer generated even if the contrast value cannot meet the preset desired value. New projected image. That is to say, in this case, the projection device 130 will still use the original projection image to be nested to the solid image A.

In summary, the image enhancement method and the image enhancement system provided by the present invention can instantly capture any static or dynamic physical image to generate a to-be-processed image. In addition to image enhancement for the image to be processed, the image processing device performs image correction on the image deformation caused by the image capturing device and the projection device to generate a projected image. Then, using the concept of augmented projection, the projection device nests the projected image onto the solid image to achieve the purpose of real-time physical image enhancement.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S201~S205‧‧‧Process of image enhancement method

Claims (10)

An image enhancement method includes: capturing an image of a subject to generate a to-be-processed image, wherein the subject is a physical image; and performing an image on the image to be processed according to an image enhancement algorithm and a plurality of image enhancement parameters Energizing the operation to generate a strengthened image; performing an inverse twisting operation and an affine operation on the enhanced image according to the plurality of image capturing correction parameters and the plurality of affine matrix parameters to generate a projected image; and nesting the projection And capturing an image of the main body of the projected image to generate a superimposed image; determining whether the pixel of the superimposed image conforms to a desired data; and if not, determining whether the brightness of the projected image is Saturated, wherein: when the brightness of the projected image is not saturated, a new projected image is generated, and the new projected image is nested to the subject. The image enhancement method of claim 1, wherein the enhanced image is subjected to the inverse distortion operation and the affine operation according to the image capturing correction parameter and the affine matrix parameter to generate the projection Before the step of image, the image enhancement method further comprises: capturing an image of the calibration object to generate a calibration image; processing the calibration image to generate the image capturing correction parameter of the corrected image; Performing the inverse distortion operation on the corrected image like a correction parameter, Generating an anti-warp corrected image; projecting the anti-warp corrected image onto a plane, and capturing the anti-warp corrected image projected on the plane to generate a projected corrected image; according to the image capturing correction parameter, Projecting the corrected image to perform the inverse twisting operation to generate an inversely twisted projected corrected image; and obtaining the affine matrix parameter based on the inversely twisted corrected image and the inversely warped corrected corrected image. The image enhancement method of claim 2, wherein the enhanced image is subjected to the inverse distortion operation and the affine operation according to the image capturing correction parameter and the affine matrix parameter to generate the projected image. The step of: performing an image enhancement operation on the image to be processed according to an image enhancement algorithm and a plurality of image enhancement parameters to generate a strengthened image; and performing the inverse distortion on the enhanced image according to the image capturing correction parameter Computing to generate an anti-twist enhancement image; and performing the affine operation on the inverse distortion enhancement image based on the affine matrix parameter to generate the projection image. The image enhancement method of claim 1, wherein when the pixel of the superimposed image does not conform to the desired data, the image of the subject is captured to regenerate the image to be processed; and the image to be processed is performed. The image enhancement operation is performed to regenerate the enhanced image; the enhanced image is subjected to the inverse distortion operation and the affine operation to reproduce Generating the projected image; and nesting the projected image to the subject to reproduce the superimposed image. The image enhancement method of claim 1, wherein when the brightness of the projected image has reached saturation, the projected image is nested to the subject. An image enhancement system includes: an image capture device that captures a subject to generate a to-be-processed image, wherein the subject is a physical image; an image processing device coupled to the image capture device, according to an image The enhanced algorithm and the plurality of image enhancement parameters are subjected to an image enhancement operation to generate an enhanced image, and the enhanced image is inversed according to the plurality of image capturing correction parameters and the plurality of affine matrix parameters. a twisting operation and an affine operation to generate a projected image; and a projection device coupled to the image processing device to nest the projected image onto the body, wherein the image capturing device captures the projected image An image of the subject of the image to generate a superimposed image, the image processing device determining whether the pixel of the superimposed image conforms to a desired data, and if not, determining whether the brightness of the projected image has reached saturation, when the brightness of the projected image is When the saturation is not reached, a new projected image is generated, and the projection device nests the new projected image to the subject. The image enhancement system of claim 6, wherein the image capture device captures an image of the calibration object to generate a corrected image. The processing device processes the corrected image to generate the image capturing correction parameter of the corrected image, and then performs the inverse distortion operation on the corrected image according to the image capturing correction parameter to generate an anti-warping corrected image. Performing the inverse distortion operation on the corrected image according to the image capturing correction parameter to generate an inverse distortion corrected image, the projection device projecting the inverse distortion corrected image on a plane, and capturing the inverse projected on the plane The corrected image is warped to generate a projected corrected image, and the image processing device obtains the affine matrix parameter based on the reverse distortion corrected image and the inverse warped corrected image. The image enhancement device of claim 7, wherein the image processing device performs an image enhancement operation on the image to be processed according to an image enhancement algorithm to generate an enhanced image, and corrects the image according to the image. And performing the inverse twisting operation on the enhanced image to generate an inverse distortion enhanced image, and performing the affine operation on the inverse distortion enhanced image according to the affine matrix parameter to generate the projected image. The image enhancement system of claim 6, wherein when the image processing device determines that the pixel of the superimposed image does not conform to the desired data: the image capturing device captures an image of the main body to reproduce the image The image processing device performs the image enhancement operation on the image to be processed to regenerate the enhanced image, and performs the inverse distortion operation and the affine operation on the enhanced image to reproduce the projected image; The projection device nests the projected image to the body to reproduce the overlay image. The image enhancement system of claim 9, wherein the projection device nests the projection image to the main body when the image processing device determines that the brightness of the projected image has reached saturation.