WO2019071613A1

WO2019071613A1 - Image processing method and device

Info

Publication number: WO2019071613A1
Application number: PCT/CN2017/106176
Authority: WO
Inventors: 王军; 杜成; 敖欢欢; 徐荣跃
Original assignee: 华为技术有限公司
Priority date: 2017-10-13
Filing date: 2017-10-13
Publication date: 2019-04-18
Also published as: CN110121882A; CN110121882B

Abstract

An image processing method and device used for reducing motion blur, the method comprising: a terminal, while previewing an image frame acquired by means of a camera, performs a first adjustment on initial exposure parameters of the camera at preview state when a captured target object moves relative to the terminal, the first adjustment comprising reduction of the Initial exposure time and increasing the initial exposure gain; the terminal, when receiving a photographing instruction, performs a second adjustment on the exposure parameters after the first adjustment, generates a first exposure frame according to the exposure parameters after the first adjustment, and generates at least two second exposure frames according to the exposure parameters after the second adjustment, wherein the second adjustment comprises reduction of the exposure time after the first adjustment and increasing the exposure gain after the first adjustment; and the terminal fuses the first exposure frame and the at least two second exposure frames so as to output a fused image.

Description

Image processing method and device

Technical field

The present application relates to the field of image processing technologies, and in particular, to an image processing method and apparatus.

Background technique

Some terminals with camera functions will have different degrees of motion blur when shooting moving objects. If the target moving speed is large, more serious smear will occur. However, sports scenes such as human motion, object motion, and flower shake are important scenes for users to pay attention to. In order to improve the clarity of moving objects, professional photographers often have expensive large apertures and set the appropriate aperture value and shutter speed to capture moving targets. But for mobile phone users who are taking pictures anytime, anywhere, the mobile phone camera aperture is fixed and cannot be extended by itself. It cannot support adjusting the aperture with high shutter speed for shooting. Secondly, even if the camera camera aperture is adjustable, the average user also There is not enough prior knowledge to set the appropriate aperture shutter and other camera parameters in time to shoot.

In recent years, mobile phone image post-processing computing power has improved significantly, and improving the cost of mobile phone camera hardware is high and difficult. So some software methods have been developed and applied. The existing software method for taking pictures against motion blur mainly uses a strategy based on motion detection to reduce the exposure time. This method first performs motion detection, automatically increases the shutter speed when detecting camera or object motion, and increases the shutter speed to reduce the exposure time. In the image imaging stage, the motion blur will decrease with the decrease of the exposure time, but reducing the exposure time will reduce the overall brightness of the image. Therefore, it is necessary to increase the exposure gain in the same proportion while maintaining the overall brightness of the image while reducing the exposure time.

However, the above method of reducing the exposure time and increasing the exposure gain in the same proportion causes the image noise in the dark scene to be excessively large, and the overall quality of the image is lowered. If the image noise level is guaranteed, the intensity of the exposure time is limited, and the anti-motion blur ability is weakened. In summary, reducing the exposure time and increasing the exposure gain are mutually constrained, resulting in insufficient terminal blur reduction motion blur.

Summary of the invention

The present application provides an image processing method and apparatus for solving the problem that when the motion blur is reduced by reducing the exposure time, the strength of the exposure time is limited by weighing the noise level, and the terminal has insufficient ability to reduce the motion blur.

In one aspect, an image processing method is provided. The method is: when previewing an image frame collected by a camera in a preview stage, when the captured target object and the terminal have relative motion, the method is as described in the preview state. Adjusting the initial exposure parameters of the camera, referred to herein as a first adjustment, the first adjustment includes reducing an initial exposure time and increasing an initial exposure gain; the terminal after receiving the shooting instruction, the first adjusted The exposure parameter is adjusted again, referred to herein as a second adjustment, the second adjustment includes decreasing the exposure time after the first adjustment and increasing the exposure gain after the first adjustment, the terminal according to the first adjusted The exposure parameter generates a first exposure frame, and generates at least two second exposure frames according to the second adjusted exposure parameter, the terminal fused the first exposure frame and the at least two second exposure frames , output the merged image. This allows a higher shutter speed to be achieved by reducing the exposure time twice through the imaging phase. By reducing the exposure time during the preview process, the delay of the imaging can be effectively shortened, and the exposure time can be reduced again after the shooting command is issued, which can more effectively reduce the motion blur, by performing the multi-frame short frame in the post-image processing section. Domain multi-frame noise reduction processing can help reduce exposure gain The noise brought by it has stronger anti-motion blur and better anti-motion blur effect.

Among them, when the terminal is in the preview state, the camera will generate initial exposure parameters under automatic exposure. The initial exposure time includes the initial exposure time and the initial exposure gain.

In a possible design, when the terminal meets certain conditions, the first adjusted exposure parameter is second adjusted. When the condition is not met, only the first adjustment is performed, according to the first adjustment. The exposure parameter is directly out of the frame, and the second adjustment is not performed. The condition may be any one of the following, or may be other conditions. The condition may be that the terminal adjusts the first adjusted exposure parameter when the rate of the relative motion of the target object and the terminal is greater than a set rate threshold; or: the terminal When the initial exposure gain of the camera is greater than the set gain threshold in the preview state, the first adjusted exposure parameter is second adjusted; or: the terminal is in the preview state of the camera When the brightness value LV is less than the set brightness threshold, the first adjusted exposure parameter is second adjusted. In this way, by selecting whether to perform the second adjustment according to the exposure gain and the LV value, different strategies for reducing motion blur can be flexibly selected to achieve targeted processing of different ambient brightness scenes.

In a possible design, if the terminal determines that the relative motion rate of the target object and the terminal is not greater than the set rate threshold, only the first adjustment of the initial exposure parameter of the camera in the preview state may be performed, and the imaging process adopts the first adjustment. The subsequent exposure parameter generates an exposure frame, and outputs an image according to the generated exposure frame, that is, a process without second adjustment

In a possible design, the terminal determines a ratio of decreasing the exposure time corresponding to the rate of the relative motion according to a relationship between a preset motion rate and a ratio of decreasing the exposure time, and determining to increase the exposure gain. The first adjustment of the initial exposure parameter of the camera in the preview state is performed according to the determined ratio of decreasing the exposure time and increasing the value of the exposure gain. In this way, by adjusting the value of the movement rate, the value of the exposure time is adjusted according to the ratio, and the ratio of the reduction of the exposure time can be correlated with the movement rate, and a suitable ratio of reducing the exposure time is selected for the current movement rate to prevent the reduction of the low. Exposure time brings a greater level of noise.

In one possible design, a maximum exposure gain threshold is set, and when the exposure parameter of the preview state is first adjusted, the increased exposure gain needs to be less than the set maximum exposure gain threshold. The setting of this method can control the noise level of the preview image by limiting the upper limit of the exposure gain.

In a possible design, the terminal performs time domain multi-frame noise reduction fusion on the at least two second exposure frames before fusing the first exposure frame and the at least two second exposure frames. Processed to get a short frame. Since the second exposure frame is generated with a shorter exposure time, a larger exposure gain is increased in proportion, which also brings more noise. With such a processing method, it can help to reduce the second exposure frame. Noise level.

In a possible design, the terminal uses the short frame as a reference frame, performs image registration and ghost detection on the first exposure frame and the short frame, and images the image according to the result of ghost detection. The first exposure frame after registration is subjected to de-ghost processing to obtain a long frame after ghosting; according to the result of the ghost detection, the ghost region of the long frame is used as a fusion reference, and the long The frame and the short frame are subjected to frequency domain fusion. Specifically, in the ghost region, the short frame pixel fusion weight is greater than the long frame pixel fusion weight. In other regions, the long frame pixel fusion weight is greater than the short frame pixel fusion weight, and the ghost region is the motion region, that is, the target object and the terminal have relative motion. In some areas, the non-moving area may be considered to be relatively stationary or approximately stationary with the terminal, the moving area is a dithered flower, and the non-moving area is a background such as a sky ground. In this way, the brightness information of the long exposure frame can be retained, so that the motion area and the non-motion area of the captured image are more clear, and the problem that the terminal reduces the motion blur due to the limitation of the noise level by reducing the intensity of the exposure time can be effectively avoided. .

In a second aspect, there is provided an image processing apparatus having the functionality to implement any of the possible in-design methods of the first aspect and the first aspect described above. The function can be implemented by hardware, or can be executed by hardware. Implementation. The hardware or software includes one or more modules corresponding to the functions described above.

In one possible design, the device can be a chip or an integrated circuit.

In one possible design, the apparatus includes a camera for acquiring image frames, a processor for executing a set of programs, and the apparatus for performing the first aspect and the first aspect described above when the program is executed The method described in any of the possible designs.

In one possible design, the apparatus also includes a memory for storing code executed by the processor.

In a third aspect, a computer storage medium is provided, stored with a computer program comprising instructions for performing the method of any of the first aspect and the first aspect of the first aspect.

In a fourth aspect, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method described in any of the first aspect and the first aspect of the first aspect.

DRAWINGS

1 is a schematic structural diagram of a terminal hardware in an embodiment of the present application;

2 is a schematic flowchart of an image processing method according to an embodiment of the present application;

3a is a schematic diagram showing the relationship between the ratio of reducing the exposure time and the exposure gain in the embodiment of the present application;

FIG. 3b is a schematic diagram showing the relationship between the motion rate level and the reduced exposure time in the embodiment of the present application; FIG.

4 is a schematic diagram of multi-frame short frame noise reduction and long and short frame fusion in the embodiment of the present application;

FIG. 5 is a schematic flowchart of removing ghost images in a long exposure frame according to an embodiment of the present application; FIG.

6 is a schematic flowchart of frequency domain fusion of long and short frames in an embodiment of the present application;

FIG. 7 is a schematic flowchart of an image processing method in an application scenario according to an embodiment of the present application;

FIG. 8 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

FIG. 9 is a second schematic structural diagram of an image processing apparatus according to an embodiment of the present application.

Detailed ways

The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

The terminal involved in the embodiment of the present application may be any electronic device having a photographing function, including but not limited to a personal computer, a server computer, a handheld or laptop device, a mobile phone, a tablet computer, a personal digital assistant, and a media player. , consumer electronics, small computers, large computers.

Referring to FIG. 1 , it is a schematic diagram of a hardware structure of a terminal applied to an embodiment of the present application. As shown in FIG. 1, the terminal 100 includes a display device 110, a processor 120, and a memory 130. The memory 130 can be used to store software programs and data, and the processor 120 executes various functional applications and data processing of the terminal 100 by running software programs and data stored in the memory 130. The memory 130 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function, and the like; the storage data area may store data created according to the use of the terminal 100, such as audio. Data, phone book, exchangeable image file EXIF. Moreover, memory 130 can include high speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. The processor 120 is a control center of the terminal 100, and connects various parts of the entire terminal by various interfaces and lines, and executes various functions and processing data of the terminal 100 by running or executing software programs and/or data stored in the memory 130. Therefore, the terminal is monitored as a whole. The processor 120 may include one or more general purpose processors, and may also include one or more digital signal processors (digital signal The processor (DSP) is configured to perform related operations to implement the technical solutions provided by the embodiments of the present application. Specifically, the processor 120 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. The processor 120 may further include a hardware chip. The hardware chip may be an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a general array logic (GAL), or any combination thereof. The memory 130 may include a volatile memory such as a random-access memory (RAM); the memory 130 may also include a non-volatile memory such as a flash memory (flash) Memory), hard disk drive (HDD) or solid state drive (SSD); the memory 130 may also include a combination of the above types of memories.

The terminal 100 may further include an input device 140 for receiving input digital information, character information or contact touch/contactless gestures, and generating signal inputs related to user settings and function control of the terminal 100, and the like. The input device 140 can include a touch panel 141. The touch panel 141, also referred to as a touch screen, collects touch operations on or near the user and drives the corresponding connection device according to a preset program. The touch panel 141 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 141, the input device 140 may further include other input devices 142, which may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, switch buttons, etc.), trackballs, mice, joysticks, and the like. One or more of them. The display device 110 includes a display panel 111 for displaying information input by the user or information provided to the user and various menu interfaces of the terminal device 100. Optionally, the display panel can be configured by using a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

In addition to the above, the terminal 100 may further include a power source 150 for powering other modules and a camera 160 for taking photos or videos. The terminal 100 may also include one or more sensors 170, such as an acceleration sensor, a light sensor, a GPS sensor, an infrared sensor, a laser sensor, a position sensor or a lens pointing angle sensor, and the like. The terminal 100 may further include a radio frequency (RF) circuit 180 for performing network communication with the wireless network device, and may further include a WiFi module 190 for performing WiFi communication with other devices.

The image processing method provided by the embodiment of the present application can be executed by the terminal 100 based on the hardware structure of the terminal shown in FIG. 1 .

After the camera is turned on, the terminal enters a preview state. During previewing the image frame acquired by the camera, the captured target object may be in a moving state, for example, human body motion, object motion, flower and grass shake, and the like. In addition, the photographed target object may be in a stationary state, but the terminal is in a motion state, for example, the user's hand-held terminal is photographed on a traveling train, or the user's hand-held terminal is photographed while exercising, and the like. Regardless of the above state, it can be considered that there is a relative motion relationship between the captured target object and the terminal. In the application scenario where the captured target object and the terminal have relative motion, there may be a motion blur problem in the image captured by the terminal. The method provided by the embodiment of the present application can help solve the motion blur problem in this application scenario.

The flow of the image processing method provided by the embodiment of the present application is further described in detail below with reference to the accompanying drawings.

As shown in FIG. 2, the flow of the image processing method provided by the embodiment of the present application is as follows.

Step 201: During previewing the image frame acquired by the camera, when the captured target object and the terminal have relative motion, the first adjustment of the initial exposure parameter of the camera in the preview state is performed.

Wherein the first adjustment comprises reducing the initial exposure time and increasing the initial exposure gain. In practical applications, the end In the preview state, the camera will generate initial exposure parameters under automatic exposure (AE). The initial exposure time includes the initial exposure time and the initial exposure gain.

Step 202: After receiving the shooting instruction, the terminal performs a second adjustment on the first adjusted exposure parameter, and generates a first exposure frame according to the first adjusted exposure parameter, and generates at least two according to the second adjusted exposure parameter. And a second exposure frame, wherein the second adjustment comprises decreasing the exposure time after the first adjustment and increasing the exposure gain after the first adjustment.

Since the second adjusted exposure time is shorter than the first adjusted exposure time, in the embodiment of the present application, the first exposure frame generated according to the first adjusted exposure parameter may also be referred to as a long exposure frame, according to The second exposure frame generated by the second adjusted exposure parameter may also be referred to as a short exposure frame.

Step 203: The terminal combines the first exposure frame and the at least two second exposure frames to output the fused image.

It should be noted that, in the embodiment of the present application, when a certain condition is satisfied, the exposure parameter needs to be adjusted twice, and the adjustment made to the initial exposure parameter of the camera in the preview state is referred to as the first adjustment. The adjustment made to the first adjusted exposure parameter after receiving the shooting instruction input by the user is referred to as a second adjustment. The exposure parameters involved in the embodiments of the present application include at least an exposure time and an exposure gain. The first adjustment refers to lowering the exposure time and increasing the exposure gain based on the exposure parameter of the camera automatic exposure in the preview state; the second adjustment refers to lowering the exposure time and increasing the exposure gain based on the first adjusted exposure parameter. . Optionally, during the first adjustment and the second adjustment, the ratio of decreasing the exposure time is the same as increasing the exposure gain, or the ratio of increasing the exposure gain is obtained according to the ratio of decreasing the exposure time. The photographing involved in the embodiments of the present application may be, but is not limited to, a process of photographing using a camera, such as photographing, photographing, and the like.

It can be seen that the embodiment of the present application can obtain a shorter exposure time by reducing the exposure time twice, and the motion blur will decrease in proportion with the decrease of the exposure time. Therefore, the shorter exposure time contributes to more effective reduction motion. blurry. In order to weaken the noise problem caused by reducing the exposure time, the embodiment of the present application adopts a multi-frame fusion method to generate a long exposure frame according to the first adjusted exposure parameter in the imaging stage, and generate a short exposure frame according to the second adjusted exposure parameter. Since long exposure frames employ longer exposure times, long exposure frames have the advantage of higher brightness, and short exposure frames have the advantage of being sharper in the motion area. The combination of the long exposure frame and the short exposure frame preserves the advantages of the long exposure frame and the short exposure frame during the fusion process, and can effectively output a clearer image of the motion region.

The following is a detailed description of the image processing method shown in Figure 2, and some specific explanations for the achievement of the beneficial effects.

In step 202, the series of actions performed by the terminal is performed during the process in which the terminal receives the user-triggered shooting instruction to output imaging. The terminal generates at least one long exposure frame and at least two short exposure frames in the imaging process. The embodiment of the present application introduces an example of generating one long exposure frame and three short exposure frames. Optionally, the terminal can also generate four or five short exposure frames in the imaging process, but three short exposure frames are a possible implementation manner due to factors such as calculation amount.

Specifically, after the terminal starts the camera, the preview image frame is motion-detected to determine whether there is relative motion between the captured target object and the terminal. Optionally, the embodiment of the present application may perform motion detection on the preview image frame by using any motion detection method in the prior art. For example, in general, the motion detection method generally performs down-sampling of two image frames before and after, and meshes the downsampled image frames. One image frame may be meshed to include multiple image grid blocks. By performing pattern matching on the images in the corresponding image grid blocks of the two image frames before and after, and calculating the correlation, it is determined whether there is motion in the image in the two frame grids, and if the calculated correlation is large, there is no motion; the correlation is small. Indicates that the image in the two-frame grid has changed relatively and there is motion. Two image frames before and after with the current frame and adjacent frames The expression determines the image grid block that is the most similar between the adjacent frame and the current frame, and the distance between the two image grid blocks obtained by the matching measures the motion speed, and the overall motion speed of the target in the image needs to calculate the total image grid block. The average speed is obtained.

If it is detected that the photographed target object is relatively stationary with the terminal, or is relatively stationary, it indicates that the terminal is currently shooting a still image, and the photographing may be performed according to the normal exposure parameter of the AE convergence. If it is detected that there is relative motion between the captured target object and the terminal, the exposure parameters need to be adjusted. In a possible implementation manner, if the terminal determines that the rate of relative motion of the target object and the terminal is greater than a set rate threshold, performing second adjustment on the first adjusted exposure parameter; if the terminal determines relative motion of the target object and the terminal If the rate is not greater than the set rate threshold, only the initial adjustment of the initial exposure parameter of the camera in the preview state may be performed. The imaging process uses the first adjusted exposure parameter to generate an exposure frame, and outputs an image according to the generated exposure frame, that is, There is no process of second adjustment. Wherein, the set rate threshold is an empirical value.

Alternatively, the condition for whether or not to perform the second adjustment may be equivalently replaced with the following two or other conditions.

Condition 1: When the initial exposure gain of the camera is greater than the set gain threshold in the preview state, the first adjusted exposure parameter is second adjusted, and when the exposure gain in the process of determining the preview image is not greater than the set gain threshold The imaging process uses the first adjusted exposure parameter to generate an exposure frame, and outputs an image according to the generated exposure frame, that is, there is no second adjustment process. Wherein, the set gain threshold is an empirical value, for example, the set gain threshold is 800.

Condition 2: In the preview state, when the light value (LV) of the camera is less than the set brightness threshold, the first adjusted exposure parameter is second adjusted, and the brightness value LV in the process of determining the preview image is not less than When the brightness threshold is set, the imaging process uses the first adjusted exposure parameter to generate an exposure frame, and outputs an image according to the generated exposure frame, that is, there is no second adjustment process. Among them, the LV can be considered as the ambient brightness during the process of previewing the image frame acquired by the camera. Set the brightness threshold to the empirical value. For example, set the brightness threshold to 40 or 20.

In this way, by selecting whether to perform the second adjustment according to the exposure gain and the LV value, different strategies for reducing motion blur can be flexibly selected to achieve targeted processing of different ambient brightness scenes. For example, the general mid-highlight scene selection only performs the first adjustment of the initial exposure parameter of the camera in the preview state, and the frame is framed according to the first adjusted exposure parameter; the second adjustment is performed after the first adjustment is performed in the dark scene, and the combination is performed. Image post processing fuses long and short frames. Of course, it is also possible to select the second adjustment after performing the first adjustment in most of the brightness environments, and combine the image post-processing to merge the long and short frames.

Optionally, when the terminal adjusts the initial exposure parameter of the camera in the preview state, the ratio of decreasing the exposure time may be related to the rate of relative motion of the target object and the terminal obtained by the motion detection. Specifically, the terminal determines a proportional value of the reduced exposure time corresponding to the rate of the relative motion according to a relationship between the preset motion rate and a ratio of decreasing the exposure time, and determines the increase according to the determined ratio of decreasing the exposure time. The value of the exposure gain, in general, increases the ratio of the exposure gain to the same ratio as the exposure time. The initial exposure parameter of the camera in the preview state is adjusted according to the determined ratio of decreasing the exposure time and increasing the value of the exposure gain. The relationship between the motion rate and the ratio of decreasing the exposure time may be a linear relationship, that is, as the motion rate increases, the ratio of the exposure time decreases, and the value of the exposure gain increases. However, since the increase of the exposure gain can bring noise, therefore, as shown in FIG. 3a, the embodiment of the present application sets a maximum exposure gain threshold, and when the exposure parameter of the camera is adjusted in the preview state, the increase is performed. The value of the exposure gain needs to be less than the set maximum exposure gain threshold. If the value after increasing the exposure gain in proportion to decreasing the exposure time is greater than the maximum exposure gain threshold, the ratio of decreasing the exposure time needs to be reduced, at least to such an extent that the increased exposure gain is less than the set maximum exposure gain. Threshold. For example, the maximum exposure gain threshold can be 700, 1000. This method can be set by limiting exposure The upper limit of the benefit is to control the noise level of the preview image.

If the condition of the second motion is determined by whether the rate of the relative motion is greater than the set rate threshold, the ratio of the relative motion rate and the decrease of the exposure time satisfies a certain relationship, and the ratio of the exposure time is substantially reduced. The rate of motion is linear and then invariant. For example, as shown in Figure 3b, a schematic diagram of the first adjustment of the exposure parameters by rate level in the preview state is characterized. It is assumed that the rate of relative motion is divided into several levels, and the different levels correspond to different ratios of decreasing exposure time. When the rate of relative motion is 0, it means shooting a still image, and imaging according to the normal exposure parameter of AE convergence; when the rate of relative motion is 1, according to the relationship line, the ratio of exposure time is reduced according to the ratio 1, and Increasing the exposure gain in the same proportion, and imaging according to the adjusted exposure parameters; when the relative motion rate level is 2, according to the relationship graph, the ratio of the exposure time is reduced according to the ratio 2, and the exposure gain is increased in the same proportion, according to the adjustment After the exposure parameter is imaged; when the relative motion rate level is 3, the motion rate at this time is greater than the set rate threshold, and the ratio of the exposure time is decreased according to the ratio 2 in the preview phase, and the exposure gain is increased in the same proportion, according to The adjusted exposure parameter is imaged; when the relative motion rate level is greater than 3, the motion rate at this time is greater than the set rate threshold, and the ratio of the exposure time is decreased according to the ratio 2 in the preview phase, and the exposure gain is increased in proportion. Imaging is performed according to the adjusted exposure parameters.

The following focuses on how to fuse at least two short exposure frames and at least one long exposure frame during the image post-processing stage. For convenience of description, three short exposure frames and one long exposure frame are generated by merging three short exposure frames and one long exposure frame in the imaging stage, and the fused images are output.

As shown in FIG. 4, the process of the fusion is roughly divided into: after the image signal processing unit (ISP) of the terminal outputs three short exposure frames and one long exposure frame, after step 401 and step 402, the final output fusion is performed. As a result, the fused image is output.

Step 401: Perform time domain multi-frame noise reduction fusion processing on the three short exposure frames, and refer to a frame obtained by the time domain multi-frame noise reduction fusion processing as a short frame.

The short frame here refers to the result of the noise reduction fusion process of three short exposure frames, and in particular, may be one of three short exposure frames.

Step 402: Combine a short frame obtained in step 401 with a long exposure frame.

Specifically, in the case of fixing the sensitivity and exposure time specified by the international standards organization (ISO), the same scene is shot at least twice, since the noise is randomly fluctuated between frames, the signal It is fixed. If the noise variance of each frame is σ ² , after the N frame averaging, the noise variance of the resulting frame is reduced to σ ² /N. For every half of the noise standard deviation, the signal-to-noise ratio is increased by 6dB. Therefore, in theory, the 4-frame time domain noise reduction can be improved by 6dB. In the embodiment of the present application, three-frame short exposure frame fusion noise reduction is used. The three-frame short exposure frame is represented by Frame0, Frame1, and Frame2.

If a hand-held jitter occurs during the shooting of multiple frames or an object in the scene moves, the time domain average may be misaligned, introducing ghosts or blurring, so image registration and ghosting need to be added before multi-frame time domain averaging. Detection.

Thus, step 401 can include several specific implementation steps. Image registration, ghost detection, time domain multi-frame noise reduction fusion. among them:

Image registration: Feature extraction is performed on the input reference frame Frame0 and the to-be-registered frame Frame1, respectively, and a series of feature points are obtained, and each feature point is characterized. According to the feature description, the feature points of the two frames of images are matched, and a series of feature point pairs are obtained. The transformation matrix of the two frames of images is obtained by matching the pair of feature points obtained, that is, the projection transformation is a matrix H of 3×3. Frame1 obtains an image aligned with Frame0 through H matrix transformation, which can be called the registration result of Frame1. Similarly, the same method to get the image of Frame2 and Frame0 alignment, can be called the registration of Frame2 result.

Ghost detection: The reference frame Frame0 is compared with the registration results of Frame1 and Frame2, respectively, to obtain the difference value of each pixel, that is, the diff diagram. Gaussian smoothing of the diff image to remove noise effects. The diff is compared with the corresponding ghost threshold to determine whether it is a ghost point. Eliminate isolated points (noise) with corrosion expansion and get the final ghost mask, the ghost shadow.

Time domain multi-frame noise reduction fusion: Determine whether the ghost area of ghost detection is less than 1/3 of the full image. If no, the data of Frame0 is directly output, that is, the process of merging is omitted, and Frame0 is directly used as the short frame to participate in the subsequent long and short frame fusion. If yes, the ghost image is weighted according to the ghost Mask to remove the ghost, and the fusion result is output, wherein the ghost frame reference frame pixel fusion weight is greater than the corresponding pixel of the other registration frame, that is, in the ghost region for reference The pixel of the frame is the result of the fusion.

Step 402 can include several specific implementation steps. The short frame and the long exposure frame obtained in step 401 are subjected to image registration, ghost detection and removal, and the long frame and the short frame from which the ghost is removed are subjected to frequency domain fusion.

Similarly, in order to avoid the fusion error caused by the ghosting of the motion region between the long exposure frame and the short frame, the same image registration and ghost detection method in the process of multiframe short frame noise reduction is used before the long and short frame fusion, and the ghost is removed. The influence of the shadow. The image registration method is similar to the above image registration method. The ghost image is used as a weight, and the registration results of the short frame and the long exposure frame after multi-frame noise reduction are combined to obtain a long exposure frame after ghosting. . For convenience of description, the long-exposure frame obtained after de-ghosting is referred to as a long frame. According to the result of ghost detection, the ghost frame region of the long frame is used as the fusion reference, and the long frame and the short frame are frequency domain fused, wherein the ghost component short frame pixel fusion weight is greater than the long frame corresponding pixel, and the non-ghost region is long. The frame pixel fusion weight is greater than the short frame corresponding pixel.

As shown in FIG. 5, a schematic diagram of the process of removing ghosts for long exposure frames. The input is a short frame obtained in step 401 and a long exposure frame after registration. The ghost image detected by the ghost is weighted, and the two input images are weighted and fused. The morphological expansion and smoothing operation is to remove the ghost and edge smoothness of the ghost Mask and improve the image fusion effect. The long exposure frame after the ghosting is outputted may be referred to as a long frame in this embodiment.

It should be noted that when the relative motion of the target object and the terminal is relatively large or a macro scene is captured, if the ghost area of the long and short frame image accounts for more than a set threshold (eg, 10%) in the entire image, It indicates that the maximum ghost threshold that can be processed by the algorithm is exceeded. In this case, in order to avoid the misalignment, the long-short frame frequency domain fusion is skipped, and the long-exposure frame is directly output.

In image processing, the frequency domain reflects the intensity of the image in the spatial gray scale, that is, the speed of the image gray, or the gradient of the image. For the image, the edge part of the image is a mutated part, and the change is faster, so the reaction is a high frequency component in the frequency domain; the noise of the image is a high frequency part in most cases; and the low frequency component is a gradual change part of the image. That is to say, the Fourier transform provides a way to freely convert from the airspace to the frequency to observe the image, and the image can be transformed from the grayscale distribution to the frequency distribution to observe the characteristics of the image. The direct relationship between image and frequency: low frequency is mostly image contour, medium frequency is mostly image edge, texture and other details, high frequency is mostly image noise, while Fourier spectrum center bright point represents the gray level mean of the image. According to the direct relationship between the image and the frequency domain, according to the result of the ghost detection, the ghost region of the long frame is used as a fusion reference, and the long frame and the short frame are fused in the frequency domain to achieve the long frame brightness and non-motion. The purpose of the area detail information and the short-frame motion area definition information.

As shown in FIG. 6, a schematic flowchart of frequency domain fusion for long frames and short frames is shown. The input is: short frame after multi-frame noise reduction and long frame with ghost removal.

Step 601: Downsampling the short frame after the multi-frame noise reduction fusion and the long frame removing the ghost, respectively, to reduce the calculation amount.

Step 602: After downsampling the long frame, the upsampling is calculated and the loss error map of the original image is used to restore image detail loss caused by downsampling after image fusion.

Step 603: Perform fast Fourier transform on the two input frames after down-sampling to obtain respective Fourier spectra, and calculate corresponding amplitudes.

In step 604, the two input Fourier spectra are fused with the magnitude of the weight.

In the fusion, the highlight of the long-frame Fourier spectrum is protected, and the value in the 10x10 region centered on the bright spot is assigned to the fused Fourier spectrum, thereby preserving the average luminance information of the long frame. .

Step 605: Perform inverse Fourier transform on the fused spectrum to obtain a fused image.

Step 606: Add the error image calculated by the fused image and the long frame down sampling to recover the loss caused by the down sampling.

The final fusion result is output.

In summary, the embodiment of the present application reduces the exposure time by the preview state for the first time, reduces the exposure time for the second time in the imaging process, and performs multi-frame short frame noise reduction fusion and long-short frame frequency domain fusion processing in the image post-processing stage to improve The ability to reduce motion blur when shooting at the terminal.

As shown in FIG. 7 , the above image processing method is further described in detail below in conjunction with a specific application scenario, and it is assumed that the application scenario is taking a photo through a camera.

Step 701: The terminal receives an instruction to open the camera, starts the camera, and enters a preview state.

Step 702: Perform down sampling on the preview image data.

Step 703: Perform motion detection on the downsampled image data.

The motion detection is performed by analyzing the preview images of two frames before and after, and the detection result is outputted immediately, for example, the motion state and the velocity level can be output, and the greater the motion rate, the larger the rate grade. The rate level is 0, indicating that there is no motion at rest, and the rate of motion indicated by

rate levels

1, 2, and 3 is increasing.

If no motion is detected, step 704 to step 705 are performed; if motion is detected, step 706 is performed.

Step 704: When the camera application end issues a photographing command, the photograph is taken according to the normal exposure parameter of the AE convergence, that is, the sensor sensor is framed according to the normal exposure parameter.

Step 705, outputting and storing an image.

Step 706: Determine whether the motion rate level is greater than the set rate threshold, or determine whether the exposure gain is greater than the set gain threshold, or determine whether the LV value is less than the set brightness threshold; if yes, perform steps 710 to 714, otherwise Steps 707 to 709 are performed.

Step 707: Adjust the exposure parameter on the preview.

The exposure time is reduced by a preset ratio based on the motion speed level, and the exposure gain is increased in proportion to maintain the overall brightness of the image.

Step 708: When the camera application end issues a photographing command, the Sensor deframes according to the adjusted exposure parameter.

Step 709, outputting and storing an image.

Step 710: Adjust the exposure parameter on the preview.

Step 711: When the camera command is issued by the camera application end, firstly, according to the adjusted preview exposure parameter, the exposure time is decreased again and the exposure gain is increased according to the set ratio.

Step 712: The Sensor generates three short exposure frames according to the reduced exposure time and the increased exposure gain, and presses A long exposure frame is generated based on the first adjusted preview exposure parameter. Image post-processing is performed after ISP processing.

In step 713, the short frame multi-frame noise reduction and the long and short frame fusion are performed in the image post-processing, and the details of the short-frame motion area and the details of the long-frame non-motion area are reserved.

Step 714, output and save the final fused image.

In summary, in the image processing method provided by the embodiment of the present application, the exposure time is reduced twice by the imaging stage to obtain a higher shutter speed, and the long exposure frame is generated by using the parameter for reducing the exposure time for the first time, and the second time is adopted. The parameter for reducing the exposure time generates at least two short exposure frames, and the long-short exposure multi-frame is fused in the post-processing stage of the image. In the ghost region, the short-frame pixel fusion weight is greater than the long-frame pixel fusion weight, and the long-frame pixel fusion weight in other regions. It is larger than the short-frame pixel fusion weight, and the ghost region is the motion region, that is, the partial region where the target object and the terminal have relative motion. The non-moving region can be considered as relatively stationary or relatively stationary with the terminal, and the motion region is a jittery flower and a non-motion region. For the sky and other backgrounds. In this way, the brightness information of the long exposure frame can be retained, so that the motion area and the non-motion area of the captured image are more clear, and the problem that the terminal reduces the motion blur due to the limitation of the noise level by reducing the intensity of the exposure time can be effectively avoided. By reducing the exposure time during the preview process, the imaging imaging delay can be effectively shortened, and the exposure time can be reduced again after the shooting command is issued, which can more effectively reduce the motion blur, and the multi-frame short frame is performed in the image post-processing section. The time domain multi-frame noise reduction process can help reduce the noise caused by increasing the exposure gain, thereby having stronger anti-motion blur and better anti-motion blur effect.

Based on the same inventive concept as the image processing method shown in FIG. 2, as shown in FIG. 8, the embodiment of the present application further provides an image processing apparatus 800 for performing the image processing shown in FIG. 2. The image processing apparatus 800 includes:

The adjusting unit 801 is configured to perform a first adjustment on the initial exposure parameter of the camera in the preview state, in the process of previewing the image frame acquired by the camera, when the captured target object and the device have relative motion, the first adjustment includes reducing the initial The exposure time and increase the initial exposure gain.

The adjusting unit 801 is further configured to perform a second adjustment on the first adjusted exposure parameter after receiving the shooting instruction, wherein the second adjusting comprises decreasing the first adjusted exposure time and increasing the first adjusted exposure gain.

The generating unit 802 is configured to generate a first exposure frame according to the first adjusted exposure parameter, and generate at least two second exposure frames according to the second adjusted exposure parameter.

The merging unit 803 is configured to fuse the first exposure frame generated by the generating unit 802 and the at least two second exposure frames to output the fused image.

Optionally, when performing the second adjustment of the first adjusted exposure parameter, the adjusting unit 801 is configured to: when the rate of the relative motion of the target object and the device is greater than the set rate threshold, the first adjusted exposure parameter Performing a second adjustment; or, when the initial exposure gain of the camera is greater than the set gain threshold in the preview state, the second adjusted exposure parameter is second adjusted; or, in the preview state, the brightness value of the camera is less than LV When the brightness threshold is set, the first adjusted exposure parameter is second adjusted.

Optionally, when performing the first adjustment of the initial exposure parameter of the camera in the preview state, the adjusting unit 801 is configured to determine the rate of the relative motion according to a relationship between the preset motion rate and the ratio of decreasing the exposure time. Correspondingly decreasing the ratio of the exposure time, and determining the value of increasing the exposure gain; performing a first adjustment on the initial exposure parameter of the camera in the preview state according to the determined ratio of decreasing the exposure time and increasing the value of the exposure gain.

Optionally, before the merging the first exposure frame and the at least two second exposure frames, the merging unit 803 is further configured to perform time domain multi-frame noise reduction fusion processing on the at least two second exposure frames to obtain a short frame. .

Optionally, the fusion unit 803 is configured to be short when the first exposure frame and the at least two second exposure frames are merged. The frame is a reference frame, and the first exposure frame and the short frame are subjected to image registration and ghost detection, and according to the result of the ghost detection, the first exposure frame after the image registration is de-ghosted to obtain a ghost image. After a long frame; according to the result of ghost detection, the ghost frame area of the long frame is used as a fusion reference, and the long frame and the short frame are frequency domain fused.

Based on the image processing method shown in FIG. 2, as shown in FIG. 9, the embodiment of the present application further provides another image processing apparatus 900, including a camera 901 and a processor 902. Wherein, the camera 901 is configured to acquire an image frame, and the processor 902 is configured to execute a set of codes, and when the code is executed, enable the image processing apparatus to execute the image processing method shown in FIG. 2. The similarities of the methods are not described here. The image processing device 900 can be the terminal 100 shown in FIG. The terminal 100 shown in FIG. 1 can be used to perform the image processing method shown in FIG. 2, the camera 160 performs the functions performed by the camera 901, and the processor 120 executes the functions performed by the processor 902. The camera 160 is used to acquire image frames; the processor 120 is configured to execute the details of the image processing method shown in FIG. 2. The function module adjusting unit 801, the generating unit 802, and the merging unit 803 in FIG. 8 can all be implemented by the processor 902 in the image processing apparatus 900, that is, can also be implemented by the processor 120 in the terminal 100 shown in FIG. .

The embodiment of the present application provides a computer storage medium, which stores a computer program, and the computer program includes an image processing method shown in FIG. 2.

An embodiment of the present application provides a computer program product comprising instructions that, when run on a computer, cause the computer to perform the image processing method illustrated in FIG. 2.

Any of the image processing apparatuses provided in the embodiments of the present application may also be a system chip.

Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment in combination of software and hardware. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

While the preferred embodiment of the present application has been described, it will be apparent that those skilled in the art can make further changes and modifications to the embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and the modifications and

It is apparent that those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, it is intended that the present invention cover the modifications and variations of the embodiments of the present invention.

Claims

An image processing method, comprising:

During previewing the image frame acquired by the camera, when the captured target object and the terminal have relative motion, the first adjustment of the initial exposure parameter of the camera in the preview state is performed, and the first adjustment includes: Initial exposure time and increase initial exposure gain;

After receiving the shooting instruction, the terminal performs a second adjustment on the first adjusted exposure parameter, and generates a first exposure frame according to the first adjusted exposure parameter, and according to the second adjusted The exposure parameter generates at least two second exposure frames, wherein the second adjustment comprises decreasing the first adjusted exposure time and increasing the first adjusted exposure gain;

The terminal fuses the first exposure frame and the at least two second exposure frames to output a fused image.
The method of claim 1, wherein the terminal performs the second adjustment of the first adjusted exposure parameter, including:

When the rate of the relative motion of the target object and the terminal is greater than a set rate threshold, the terminal performs a second adjustment on the first adjusted exposure parameter; or

When the initial exposure gain of the camera is greater than the set gain threshold in the preview state, the terminal performs the second adjustment on the first adjusted exposure parameter; or

When the brightness value LV of the camera is less than the set brightness threshold in the preview state, the terminal performs the second adjustment on the first adjusted exposure parameter.
The method according to claim 1 or 2, wherein the first adjustment of the initial exposure parameter of the camera in a preview state comprises:

Determining, by the terminal, a ratio of decreasing the exposure time corresponding to the rate of the relative motion according to a relationship between a preset motion rate and a ratio of decreasing the exposure time, and determining a value of increasing the exposure gain;

The first adjustment of the initial exposure parameter of the camera in the preview state is performed according to the determined ratio of decreasing the exposure time and increasing the value of the exposure gain.
The method according to claim 1, 2 or 3, wherein the terminal further comprises: before the merging the first exposure frame and the at least two second exposure frames, the terminal further comprises:

The terminal performs time domain multi-frame noise reduction fusion processing on the at least two second exposure frames to obtain a short frame.
The method according to claim 4, wherein the terminal fuses the first exposure frame and the at least two second exposure frames, including:

The terminal uses the short frame as a reference frame, performs image registration and ghost detection on the first exposure frame and the short frame, and performs a first exposure after the image is registered according to the result of the ghost detection. The frame is subjected to ghosting processing to obtain a long frame after ghosting;

According to the result of the ghost detection, the long frame and the short frame are frequency domain fused by using a ghost region of the long frame as a fusion reference.
An image processing apparatus, comprising:

An adjusting unit, configured to perform a first adjustment on an initial exposure parameter of the camera in a preview state when the captured target object and the device have relative motion during previewing an image frame acquired by the camera, where An adjustment includes reducing the initial exposure time and increasing the initial exposure gain;

The adjusting unit is further configured to perform a second adjustment on the first adjusted exposure parameter after receiving the shooting instruction, wherein the second adjusting includes reducing the exposure time and the increasing time after the first adjustment Depicting the first adjusted exposure gain;

a generating unit, configured to generate a first exposure frame according to the first adjusted exposure parameter, and generate at least two second exposure frames according to the second adjusted exposure parameter;

And a merging unit, configured to fuse the first exposure frame generated by the generating unit and the at least two second exposure frames, and output the fused image.
The apparatus according to claim 6, wherein when the first adjusted exposure parameter is second adjusted, the adjusting unit is configured to:

And performing, when the rate of the relative movement of the target object and the device is greater than a set rate threshold, performing the second adjustment on the first adjusted exposure parameter; or

When the initial exposure gain of the camera is greater than the set gain threshold in the preview state, the first adjusted exposure parameter is second adjusted; or

When the brightness value LV of the camera is less than the set brightness threshold in the preview state, the first adjusted exposure parameter is second adjusted.
The apparatus according to claim 6 or 7, wherein, in the first adjustment of the initial exposure parameter of the camera in the preview state, the adjusting unit is configured to:

Determining a ratio of decreasing the exposure time corresponding to the rate of the relative motion according to a relationship between a preset motion rate and a ratio of decreasing the exposure time, and determining a value for increasing the exposure gain;

The first adjustment of the initial exposure parameter of the camera in the preview state is performed according to the determined ratio of decreasing the exposure time and increasing the value of the exposure gain.
The apparatus according to claim 6, 7 or 8, wherein the merging unit is further configured to: before merging the first exposure frame and the at least two second exposure frames:

The at least two second exposure frames are subjected to time domain multi-frame noise reduction fusion processing to obtain a short frame.
The apparatus according to claim 9, wherein when the first exposure frame and the at least two second exposure frames are fused, the merging unit is configured to:

Taking the short frame as a reference frame, performing image registration and ghost detection on the first exposure frame and the short frame, and performing the first exposure frame after the image registration according to the result of the ghost detection. Ghost processing, get a long frame after ghosting;

According to the result of the ghost detection, the long frame and the short frame are frequency domain fused by using a ghost region of the long frame as a fusion reference.
An image processing apparatus includes a camera and a processor, wherein:

The camera is configured to acquire an image frame;

The processor is configured to: during a preview of an image frame acquired by the camera, when the captured target object and the device have relative motion, perform a first adjustment on the initial exposure parameter of the camera in a preview state, and receive After the shooting instruction, performing a second adjustment on the first adjusted exposure parameter, the first adjustment comprising decreasing an initial exposure time and increasing an initial exposure gain, the second adjusting comprising reducing the first adjustment Exposure time and increase the first adjusted exposure gain; generating a first exposure frame according to the first adjusted exposure parameter, and generating at least two second exposure frames according to the second adjusted exposure parameter And merging the generated first exposure frame and the at least two second exposure frames to output the fused image.
The apparatus according to claim 11, wherein when the first adjusted exposure parameter is second adjusted, the processor is configured to:

And performing, when the rate of the relative movement of the target object and the device is greater than a set rate threshold, performing the second adjustment on the first adjusted exposure parameter; or

When the initial exposure gain of the camera is greater than the set gain threshold in the preview state, the first adjusted exposure parameter is second adjusted; or

When the brightness value LV of the camera is less than the set brightness threshold in the preview state, the first adjusted exposure parameter is second adjusted.
The apparatus according to claim 11 or 12, wherein when the first adjustment of the initial exposure parameter of the camera in the preview state is performed, the processor is configured to:

Determining a ratio of decreasing the exposure time corresponding to the rate of the relative motion according to a relationship between a preset motion rate and a ratio of decreasing the exposure time, and determining a value for increasing the exposure gain;

The first adjustment of the initial exposure parameter of the camera in the preview state is performed according to the determined ratio of decreasing the exposure time and increasing the value of the exposure gain.
The apparatus of claim 11, 12 or 13, wherein the processor is further configured to: prior to fusing the first exposure frame and the at least two second exposure frames:

The at least two second exposure frames are subjected to time domain multi-frame noise reduction fusion processing to obtain a short frame.
The apparatus of claim 14, wherein when the first exposure frame and the at least two second exposure frames are fused, the processor is configured to:

Taking the short frame as a reference frame, performing image registration and ghost detection on the first exposure frame and the short frame, and performing the first exposure frame after the image registration according to the result of the ghost detection. Ghost processing, get a long frame after ghosting;

According to the result of the ghost detection, the long frame and the short frame are frequency domain fused by using a ghost region of the long frame as a fusion reference.
A computer storage medium, characterized in that the computer storage medium stores a computer program comprising instructions for performing the method of any one of claims 1 to 5.