WO2021208593A1 - High dynamic range image processing system and method, electronic device, and storage medium - Google Patents

Abstract

A high dynamic range image processing system (100) and method, an electronic device (1000), and a computer-readable storage medium (400). The high dynamic range image processing system (100) comprises an image sensor (10), a high dynamic fusion unit (50), and an image processor (20). A pixel array (11) is exposed at a first exposure time to obtain a first original image, and is exposed at a second exposure time to obtain a second original image. The image processor (20) and the high dynamic fusion unit (50) are used for performing image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain a target image.

Description

High dynamic range image processing system and method, electronic equipment and storage medium

Priority information

This application requests the priority and rights of the patent application with the patent application number 202010304152.6 filed with the State Intellectual Property Office of China on April 17, 2020, and the full text is incorporated herein by reference.

Technical field

This application relates to the field of image processing technology, and in particular to a high dynamic range image processing system and method, electronic equipment, and computer-readable storage media.

Background technique

Ordinary cameras cannot record extremely bright or dark details due to the limitation of dynamic range. Especially when the light in the shooting scene is relatively large, it is prone to overexposure or underexposure. A camera with a high dynamic range (High-Dynamic Range, HDR) function can capture images in a large light ratio, and it can perform better than ordinary cameras in both high light and dark positions.

Summary of the invention

The embodiments of the present application provide a high dynamic range image processing system and method, electronic equipment, and computer-readable storage medium.

The high dynamic range image processing system provided by the embodiments of the present application includes an image sensor, a high dynamic fusion unit, and an image processor. The image sensor includes a pixel array. The pixel array includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. The color photosensitive pixel has a narrower spectral response than the full-color photosensitive pixel. The pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. The pixel array is exposed for a first exposure time to obtain a first original image. The first original image includes first color original image data generated by the single-color photosensitive pixels exposed at the first exposure time and first full-color original image data generated by the full-color photosensitive pixels exposed at the first exposure time. The pixel array is exposed for a second exposure time to obtain a second original image. The second original image includes second color original image data generated by the single-color photosensitive pixel exposed at the second exposure time and second full-color original image data generated by the panchromatic photosensitive pixel exposed at the second exposure time. Wherein, the first exposure time is not equal to the second exposure time. The image processor and the high dynamic fusion unit are used to perform image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain a target image.

The high dynamic range image processing method provided by the embodiment of the present application is used in a high dynamic range image processing system. The high dynamic range image processing system includes an image sensor. The image sensor includes a pixel array, and the pixel array includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. The color photosensitive pixel has a narrower spectral response than the full-color photosensitive pixel. The pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. The high dynamic range image processing method includes: controlling a pixel array to perform at least two exposures, wherein the pixel array is exposed at a first exposure time to obtain a first original image, and the first original image includes the single exposed at the first exposure time. The first color original image data generated by the color photosensitive pixels and the first panchromatic original image data generated by the panchromatic photosensitive pixels exposed at the first exposure time; the pixel array is exposed at the second exposure time to obtain the second original image , The second original image includes second color original image data generated by the single-color photosensitive pixel exposed at the second exposure time and second full-color original image data generated by the panchromatic photosensitive pixel exposed at the second exposure time ; Wherein, the first exposure time is not equal to the second exposure time; and image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing are performed on the first original image and the second original image to obtain the target image.

The electronic device provided by the embodiment of the present application includes a lens, a housing, and a high dynamic range image processing system. The lens and the high dynamic range image processing system are combined with the housing. The lens cooperates with the image sensor of the high dynamic range image processing system for imaging. The high dynamic range image processing system includes an image sensor, a high dynamic fusion unit and an image processor. The image sensor includes an array of pixels. The pixel array includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels. The pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. The pixel array is exposed for the first exposure time to obtain the first original image. The first original image includes first color original image data generated by single-color photosensitive pixels exposed at the first exposure time and first full-color original image data generated by panchromatic photosensitive pixels exposed at the first exposure time. The pixel array is exposed for a second exposure time to obtain a second original image. The second original image includes second color original image data generated by the single-color photosensitive pixels exposed at the second exposure time and second full-color original image data generated by the panchromatic photosensitive pixels exposed at the second exposure time. Wherein, the first exposure time is not equal to the second exposure time. The image processor and the high dynamic fusion unit are used to perform image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain the target image.

In the non-volatile computer-readable storage medium containing a computer program provided by the embodiments of the present application, when the computer program is executed by a processor, the processor executes the high dynamic range image processing method. The high dynamic range image processing method is used in a high dynamic range image processing system. The high dynamic range image processing system includes an image sensor, a color high dynamic fusion unit, a panchromatic high dynamic fusion unit and an image processor. The image sensor includes a pixel array, and the pixel array includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels. The pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. The pixel array is exposed for the first exposure time to obtain the first original image. The first original image includes first color original image data generated by single-color photosensitive pixels exposed at the first exposure time and first full-color original image data generated by panchromatic photosensitive pixels exposed at the first exposure time. The pixel array is exposed for a second exposure time to obtain a second original image. The second original image includes second color original image data generated by the single-color photosensitive pixels exposed at the second exposure time and second full-color original image data generated by the panchromatic photosensitive pixels exposed at the second exposure time. Wherein, the first exposure time is not equal to the second exposure time. The image processor, the color high dynamic fusion unit, and the panchromatic high dynamic fusion unit are used to perform image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain the target image.

Description of the drawings

The above and/or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, in which:

Fig. 1 is a schematic diagram of a high dynamic range image processing system according to an embodiment of the present application;

Fig. 2 is a schematic diagram of a pixel array according to an embodiment of the present application;

3 is a schematic cross-sectional view of a photosensitive pixel according to an embodiment of the present application;

4 is a pixel circuit diagram of a photosensitive pixel according to an embodiment of the present application;

5 to 10 are schematic diagrams of the arrangement of the smallest repeating unit in the pixel array of the embodiment of the present application;

11 to 13 are schematic diagrams of original images output by image sensors in some embodiments of the present application;

FIG. 14 is a schematic diagram of a pixel array according to an embodiment of the present application;

FIG. 15 to FIG. 17 are schematic diagrams of pixel completion processing in an embodiment of the present application;

18 to 20 are schematic diagrams of a high dynamic range image processing system according to an embodiment of the present application;

FIG. 21 is a schematic diagram of black level correction processing according to an embodiment of the present application;

FIG. 22 is a schematic diagram of lens shading correction processing according to an embodiment of the present application;

FIG. 23 and FIG. 24 are schematic diagrams of dead pixel compensation processing in an embodiment of the present application;

FIG. 25 to FIG. 28 are schematic diagrams of demosaicing processing in an embodiment of the present application;

FIG. 29 is a schematic diagram of the mapping relationship between Vout and Vin in the tone mapping process of the embodiment of the present application;

FIG. 30 is a schematic diagram of brightness alignment processing according to an embodiment of the present application;

FIG. 31 is a schematic diagram of pixel addition processing in an embodiment of the present application;

FIG. 32 is a schematic diagram of pixel averaging processing according to an embodiment of the present application;

FIG. 33 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

FIG. 34 is a schematic flowchart of an image acquisition method according to some embodiments of the present application;

FIG. 35 is a schematic diagram of interaction between a non-volatile computer-readable storage medium and a processor in some embodiments of the present application.

Detailed ways

The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the drawings, and the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions throughout. The following embodiments described with reference to the drawings are exemplary, and are only used to explain the present application, and cannot be understood as a limitation to the present application.

The following disclosure provides many different embodiments or examples for realizing the different structures of the embodiments of the present application. In order to simplify the disclosure of the embodiments of the present application, the components and settings of specific examples are described below. Of course, they are only examples, and are not intended to limit the application.

1 and FIG. 2, the high dynamic range image processing system 100 according to the embodiment of the present application includes an image sensor 10, a high dynamic fusion unit 50 and an image processor 20. The image sensor 10 includes a pixel array 11, and the pixel array 11 includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels. The pixel array 11 includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. The pixel array 11 is exposed for a first exposure time to obtain a first original image. The first original image includes first color original image data generated by single-color photosensitive pixels exposed at the first exposure time and first full-color original image data generated by panchromatic photosensitive pixels exposed at the first exposure time. The pixel array 11 is exposed for a second exposure time to obtain a second original image. The second original image includes second color original image data generated by the single-color photosensitive pixels exposed at the second exposure time and second full-color original image data generated by the panchromatic photosensitive pixels exposed at the second exposure time. Wherein, the first exposure time is not equal to the second exposure time. The image processor 20 and the high dynamic fusion unit 50 are used to perform image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain a target image.

The high dynamic range image processing system 100 of the embodiment of the present application controls the pixel array 11 to perform at least two exposures at the first exposure time and the second exposure time respectively, and generates multiple images according to different exposure times and different photosensitive pixels, In order to subsequently perform image preprocessing, high dynamic range processing, image processing and fusion algorithm processing on the multiple images, so as to obtain a target image with high dynamic range. The high dynamic range image processing system 100 of the embodiment of the present application can realize the high dynamic range function without increasing the hardware parameters of the photosensitive pixels of the image sensor 10, so that the bright and dark positions of the target image can be better. Performance, which is conducive to improving imaging performance, while helping to reduce costs.

1 and 2, in some embodiments, the pixel array 11 is exposed at a third exposure time to obtain a third original image, and the third original image includes the third original image exposed at the third exposure time. The third color original image data generated by the single-color photosensitive pixel and the third panchromatic original image data generated by the panchromatic photosensitive pixel exposed at the third exposure time; wherein the third exposure time is not equal to the The first exposure time, the third exposure time is not equal to the second exposure time. The image processor 20 and the high dynamic fusion unit 50 are used to perform image preprocessing, high dynamic range processing, image processing, and processing on the first original image, the second original image, and the third original image. The fusion algorithm processes the target image.

1 and 2, in some embodiments, the image processor 20 includes a color preprocessing module 2023, a full color preprocessing module 2024, a color processing module 2021, a full color processing module 2022, and a fusion module 204, The image preprocessing includes pixel completion processing and demosaicing processing, and the image processing includes first image processing and second image processing. Wherein: the color preprocessing module 2023 is used to perform pixel complement processing on the color original image data to obtain a color original image; the full color preprocessing module 2024 is used to perform demosaicing processing on the full color original image data to obtain a complete Color original image; the color processing module 2021 is used to perform first image processing on the color original image to obtain a color intermediate image; the full color processing module 2022 is used to perform second image processing on the full color original image , To obtain a panchromatic intermediate image; the fusion module 204 is configured to perform a fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image.

In some embodiments, after the fusion module 204 performs fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image: the high dynamic fusion unit 50 is configured to combine at least twice The target image corresponding to the exposure is fused to obtain the highly dynamic target image.

Referring to FIG. 18, in some embodiments, the high-dynamic fusion unit 50 includes a color high-dynamic fusion unit 30 and a panchromatic high-dynamic fusion unit 40. The color preprocessing module 2023 performs pixel processing on the color original image data. Before the completion process to obtain the color original image: the color high-dynamic fusion unit 30 is configured to fuse the color original image data corresponding to at least two exposures to obtain the high-dynamic color original image data. Before the panchromatic preprocessing module 2024 performs demosaic processing on the panchromatic original image data to obtain a panchromatic original image: the panchromatic high dynamic fusion unit 40 is configured to combine the panchromatic original image data corresponding to at least two exposures. The image data is fused to obtain the high dynamic panchromatic original image data.

In some embodiments, the first image processing includes one of black level correction processing, lens shading correction processing, dead pixel compensation processing, demosaicing processing, color correction processing, global tone mapping processing, and color conversion processing. Or more. The second image processing includes: one or more of the black level correction processing, the lens shading correction processing, the dead pixel compensation processing, and the global tone mapping processing.

In some embodiments, the first image processing includes a first image sub-processing and a second image sub-processing, and the color processing module 2021 is used to perform the first image sub-processing on the color original image before performing the first image sub-processing. The second image sub-processing, the first image sub-processing includes: one or more of black level correction processing, lens shading correction processing, and dead pixel compensation processing. The second image sub-processing includes: one or more of demosaicing processing, color correction processing, global tone mapping processing, and color conversion processing.

In some embodiments, the high dynamic fusion unit 50 is configured to perform brightness alignment processing on the target image corresponding to at least two exposures to obtain the brightness-aligned target image, and then merge the brightness-aligned target image. An image and one or more of the target image to obtain the highly dynamic target image.

In some embodiments, the color high dynamic fusion unit 30 is configured to perform brightness alignment processing on the color original image data corresponding to at least two exposures to obtain the brightness-aligned color original image data, and then merge the brightness The color original image data and one or more sheets of the color original image data are aligned to obtain the high dynamic color original image data. The panchromatic high dynamic fusion unit 40 is configured to perform brightness alignment processing on the panchromatic original image data corresponding to at least two exposures, so as to obtain the brightness-aligned panchromatic original image data, and then merge the brightness-aligned panchromatic original image data. Full-color original image data and one or more sheets of the full-color original image data to obtain the high-dynamic full-color original image data.

Please refer to FIG. 1, in some embodiments, the image processor 20 further includes a receiving unit 201 and a memory unit 203. The receiving unit 201 is used to receive the color original image data and the full-color original image data; the memory unit 203 is used to temporarily store the color original image data, the full-color original image data, and the color original image data. One or more of the original image, the full-color original image, the color intermediate image, the full-color intermediate image, and the target image.

In some embodiments, the image processor 20 includes a color preprocessing module 2023 and a panchromatic preprocessing module 2024. The image preprocessing includes pixel addition processing and demosaicing processing. The color preprocessing module 2023 is used for The color original image data is subjected to pixel addition processing to obtain a color original image, and the panchromatic preprocessing module 2024 is configured to perform demosaicing processing on the panchromatic original image data to obtain a panchromatic original image; or, the image preprocessing includes Pixel averaging processing and demosaicing processing, the color preprocessing module 2023 is used to perform pixel averaging processing on color original image data to obtain a color original image, and the panchromatic preprocessing module 2024 is used to perform pixel averaging processing on the color original image. The data is demosaiced to obtain a full-color original image.

In some embodiments, the high dynamic fusion unit 50 is integrated in the image sensor 10.

In some embodiments, the high dynamic fusion unit 50 is integrated in the image processor 20.

Please refer to FIG. 34. This application provides a high dynamic range image processing method. The high dynamic range image processing method of the embodiment of the present application is used in the high dynamic range image processing system 100. The high dynamic range image processing system 100 may include the image sensor 10. The image sensor 10 includes a pixel array 11. The pixel array 11 includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels. The pixel array 11 includes the smallest repeating unit. Each minimum repeating unit contains multiple subunits. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. High dynamic range image processing methods include:

01: Control the exposure of the pixel array 11. Wherein, the pixel array 11 is exposed for the first exposure time to obtain the first original image. The first original image includes first color original image data generated by single-color photosensitive pixels exposed at the first exposure time and first full-color original image data generated by panchromatic photosensitive pixels exposed at the first exposure time. The pixel array 11 is exposed for a second exposure time to obtain a second original image. The second original image includes second color original image data generated by the single-color photosensitive pixels exposed at the second exposure time and second full-color original image data generated by the panchromatic photosensitive pixels exposed at the second exposure time. Wherein, the first exposure time is not equal to the second exposure time. with

02: Perform image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain the target image.

In some embodiments, the pixel array 11 may also be exposed for a third exposure time to obtain a third original image. The third original image includes third color original image data generated by the single-color photosensitive pixels exposed at the third exposure time and third full-color original image data generated by the panchromatic photosensitive pixels exposed at the third exposure time. Wherein, the third exposure time is not equal to the first exposure time, and the third exposure time is not equal to the second exposure time. Performing image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain the target image may include: performing image preprocessing on the first original image, the second original image, and the third original image. Processing, high dynamic range processing, image processing and fusion algorithm processing to obtain the target image.

In some embodiments, the image preprocessing includes pixel completion processing and demosaicing processing, and the image processing includes first image processing and second image processing; image preprocessing and high dynamics are performed on the first original image and the second original image. The range processing, image processing, and fusion algorithm processing to obtain the target image may also include: performing pixel complement processing on the color original image data to obtain the color original image; performing demosaic processing on the panchromatic original image data to obtain the panchromatic original image; Perform the first image processing on the color original image to obtain a color intermediate image; perform the second image processing on the panchromatic original image to obtain the panchromatic intermediate image; perform fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image.

In some embodiments, after the color intermediate image and the panchromatic intermediate image are processed by the fusion algorithm to obtain the target image, image preprocessing, high dynamic range processing, image processing, and fusion are performed on the first original image and the second original image. The algorithm processing to obtain the target image also includes: fusing the target images corresponding to at least two exposures to obtain a highly dynamic target image.

In some embodiments, before performing pixel complement processing on the color original image data to obtain the color original image, image preprocessing, high dynamic range processing, image processing, and fusion algorithms are performed on the first original image and the second original image. The processing to obtain the target image also includes: fusing the color original image data corresponding to at least two exposures to obtain highly dynamic color original image data; before demosaicing the panchromatic original image data to obtain the panchromatic original image, perform the first The original image and the second original image are processed by image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing to obtain the target image. It also includes: fusing the panchromatic primitive image data corresponding to at least two exposures to obtain a highly dynamic panchromatic primitive image data.

In some embodiments, the first image processing includes one or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, demosaicing processing, color correction processing, global tone mapping processing, and color conversion processing. indivual. The second image processing includes one or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, and global tone mapping processing.

In some embodiments, the first image processing includes a first image sub-processing and a second image sub-processing, and the color processing module 2021 is configured to perform the first image sub-processing on the color original image first, and then perform the second image sub-processing, The first image sub-processing includes one or more of black level correction processing, lens shading correction processing, and dead pixel compensation processing. The second image sub-processing includes: one or more of demosaicing processing, color correction processing, global tone mapping processing, and color conversion processing.

In some embodiments, fusing the target images corresponding to at least two exposures to obtain a highly dynamic target image includes: subjecting the target images corresponding to the at least two exposures to brightness alignment processing to obtain a target image with aligned brightness, and then fusing the brightness Align the target image and one or more target images to obtain a highly dynamic target image.

In some embodiments, fusing the color original image data corresponding to at least two exposures to obtain highly dynamic color original image data includes: subjecting the color original image data corresponding to the at least two exposures to brightness alignment processing to obtain brightness alignment. The color original image data is merged with brightness-aligned color original image data and one or more color original image data to obtain highly dynamic color original image data. Fusion of panchromatic original image data corresponding to at least two exposures to obtain highly dynamic panchromatic original image data includes: performing brightness alignment processing on panchromatic original image data corresponding to at least two exposures to obtain a brightness-aligned panchromatic original image Data, and then merge the brightness-aligned panchromatic original image data and one or more panchromatic original image data to obtain highly dynamic panchromatic original image data.

In some embodiments, the high dynamic range image processing method further includes: receiving color original image data and full-color original image data; and temporarily storing color original image data, full-color original image data, color original image, and full-color original image , One or more of the color intermediate image, the full-color intermediate image, and the target image.

In some embodiments, the image preprocessing includes pixel addition processing and demosaicing processing, and image preprocessing, high dynamic range processing, image processing, and fusion algorithms are performed on the first original image, the second original image, and the third original image. The processing to obtain the target image includes: performing pixel addition processing on the color original image data to obtain the color original image; and performing demosaicing processing on the panchromatic original image data to obtain the panchromatic original image; or image preprocessing including pixel averaging processing and Demosaic processing, image preprocessing, high dynamic range processing, image processing and fusion algorithm processing are performed on the first original image, the second original image and the third original image to obtain the target image including: pixel averaging processing on the color original image data , Get the color original image; and perform demosaic processing on the full-color original image data to get the full-color original image.

Please refer to FIG. 33. The present application also provides an electronic device 1000. The electronic device 1000 of the embodiment of the present application includes a lens 300, a housing 200, and the high dynamic range image processing system 100 of any one of the above embodiments. The lens 300 and the high dynamic range image processing system 100 are combined with the housing 200. The lens 300 cooperates with the image sensor 10 of the high dynamic range image processing system 100 for imaging.

Referring to FIG. 35, the present application also provides a non-volatile computer-readable storage medium 400 containing a computer program. When the computer program is executed by the processor 60, the processor 60 is caused to execute the high dynamic range image processing method described in any one of the foregoing embodiments.

FIG. 2 is a schematic diagram of the image sensor 10 in the embodiment of the present application. The image sensor 10 includes a pixel array 11, a vertical driving unit 12, a control unit 13, a column processing unit 14 and a horizontal driving unit 15.

For example, the image sensor 10 may adopt a complementary metal oxide semiconductor (CMOS, Complementary Metal Oxide Semiconductor) photosensitive element or a charge-coupled device (CCD, Charge-coupled Device) photosensitive element.

For example, the pixel array 11 includes a plurality of photosensitive pixels 110 (shown in FIG. 3) arranged two-dimensionally in an array (ie, arranged in a two-dimensional matrix), and each photosensitive pixel 110 includes a photoelectric conversion element 1111 (shown in FIG. 4) . Each photosensitive pixel 110 converts light into electric charge according to the intensity of light incident thereon.

For example, the vertical driving unit 12 includes a shift register and an address decoder. The vertical drive unit 12 includes readout scanning and reset scanning functions. The readout scan refers to sequentially scanning the unit photosensitive pixels 110 line by line, and reading signals from these unit photosensitive pixels 110 line by line. For example, the signal output by each photosensitive pixel 110 in the selected and scanned photosensitive pixel row is transmitted to the column processing unit 14. The reset scan is used to reset the charge, and the photocharge of the photoelectric conversion element is discarded, so that the accumulation of new photocharge can be started.

For example, the signal processing performed by the column processing unit 14 is correlated double sampling (CDS) processing. In the CDS process, the reset level and the signal level output from each photosensitive pixel 110 in the selected photosensitive pixel row are taken out, and the level difference is calculated. Thus, the signals of the photosensitive pixels 110 in a row are obtained. The column processing unit 14 may have an analog-to-digital (A/D) conversion function for converting analog pixel signals into a digital format.

For example, the horizontal driving unit 15 includes a shift register and an address decoder. The horizontal driving unit 15 sequentially scans the pixel array 11 column by column. Through the selection scanning operation performed by the horizontal driving unit 15, each photosensitive pixel column is sequentially processed by the column processing unit 14, and is sequentially output.

For example, the control unit 13 configures timing signals according to the operation mode, and uses various timing signals to control the vertical driving unit 12, the column processing unit 14 and the horizontal driving unit 15 to work together.

FIG. 3 is a schematic diagram of a photosensitive pixel 110 in an embodiment of the present application. The photosensitive pixel 110 includes a pixel circuit 111, a filter 112, and a micro lens 113. Along the light-receiving direction of the photosensitive pixel 110, the microlens 113, the filter 112, and the pixel circuit 111 are arranged in sequence. The microlens 113 is used for condensing light, and the filter 112 is used for passing light of a certain waveband and filtering out the light of other wavebands. The pixel circuit 111 is used to convert the received light into electrical signals, and provide the generated electrical signals to the column processing unit 14 shown in FIG. 2.

FIG. 4 is a schematic diagram of a pixel circuit 111 of a photosensitive pixel 110 in an embodiment of the present application. The pixel circuit 111 in FIG. 4 can be applied to each photosensitive pixel 110 (shown in FIG. 3) in the pixel array 11 shown in FIG. The working principle of the pixel circuit 111 will be described below with reference to FIGS. 2 to 4.

As shown in FIG. 4, the pixel circuit 111 includes a photoelectric conversion element 1111 (for example, a photodiode), an exposure control circuit (for example, a transfer transistor 1112), a reset circuit (for example, a reset transistor 1113), and an amplification circuit (for example, an amplification transistor 1114). ) And a selection circuit (for example, a selection transistor 1115). In the embodiment of the present application, the transfer transistor 1112, the reset transistor 1113, the amplifying transistor 1114, and the selection transistor 1115 are, for example, MOS transistors, but are not limited thereto.

For example, the photoelectric conversion element 1111 includes a photodiode, and the anode of the photodiode is connected to the ground, for example. The photodiode converts the received light into electric charge. The cathode of the photodiode is connected to the floating diffusion unit FD via an exposure control circuit (for example, a transfer transistor 1112). The floating diffusion unit FD is connected to the gate of the amplification transistor 1114 and the source of the reset transistor 1113.

For example, the exposure control circuit is a transfer transistor 1112, and the control terminal TG of the exposure control circuit is the gate of the transfer transistor 1112. When a pulse of an active level (for example, VPIX level) is transmitted to the gate of the transfer transistor 1112 through the exposure control line, the transfer transistor 1112 is turned on. The transfer transistor 1112 transfers the charge photoelectrically converted by the photodiode to the floating diffusion unit FD.

For example, the drain of the reset transistor 1113 is connected to the pixel power supply VPIX. The source of the reset transistor 113 is connected to the floating diffusion unit FD. Before the charge is transferred from the photodiode to the floating diffusion unit FD, a pulse of an effective reset level is transmitted to the gate of the reset transistor 113 via the reset line, and the reset transistor 113 is turned on. The reset transistor 113 resets the floating diffusion unit FD to the pixel power supply VPIX.

For example, the gate of the amplifying transistor 1114 is connected to the floating diffusion unit FD. The drain of the amplifying transistor 1114 is connected to the pixel power supply VPIX. After the floating diffusion unit FD is reset by the reset transistor 1113, the amplifying transistor 1114 outputs the reset level through the output terminal OUT via the selection transistor 1115. After the charge of the photodiode is transferred by the transfer transistor 1112, the amplifying transistor 1114 outputs a signal level through the output terminal OUT via the selection transistor 1115.

For example, the drain of the selection transistor 1115 is connected to the source of the amplification transistor 1114. The source of the selection transistor 1115 is connected to the column processing unit 14 in FIG. 2 through the output terminal OUT. When the pulse of the active level is transmitted to the gate of the selection transistor 1115 through the selection line, the selection transistor 1115 is turned on. The signal output by the amplifying transistor 1114 is transmitted to the column processing unit 14 through the selection transistor 1115.

It should be noted that the pixel structure of the pixel circuit 111 in the embodiment of the present application is not limited to the structure shown in FIG. 4. For example, the pixel circuit 111 may also have a three-transistor pixel structure, in which the functions of the amplifying transistor 1114 and the selecting transistor 1115 are performed by one transistor. For example, the exposure control circuit is not limited to the way of a single transfer transistor 1112, and other electronic devices or structures with the function of controlling the conduction of the control terminal can be used as the exposure control circuit in the embodiment of the present application. The implementation of the transistor 1112 is simple, low in cost, and easy to control.

5 to 10 are schematic diagrams of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the pixel array 11 (shown in FIG. 2) according to some embodiments of the present application. The photosensitive pixels 110 include two types, one is a full-color photosensitive pixel W, and the other is a color photosensitive pixel. 5 to 10 only show the arrangement of a plurality of photosensitive pixels 110 in a minimum repeating unit. The smallest repeating unit shown in FIGS. 5 to 10 is copied multiple times in rows and columns to form the pixel array 11. Each minimum repeating unit is composed of multiple full-color photosensitive pixels W and multiple color photosensitive pixels. Each minimum repeating unit includes multiple subunits. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels W. Among them, in the smallest repeating unit shown in FIGS. 5 to 8, the full-color photosensitive pixel W and the color photosensitive pixel in each sub-unit are alternately arranged. In the smallest repeating unit shown in FIGS. 9 and 10, in each sub-unit, multiple photosensitive pixels 110 in the same row are photosensitive pixels 110 of the same category; or, multiple photosensitive pixels 110 in the same column are photosensitive pixels 110 of the same category 110.

Specifically, for example, FIG. 5 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit of an embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

W represents the full-color photosensitive pixel; A represents the first color photosensitive pixel among the multiple color photosensitive pixels; B represents the second color photosensitive pixel among the multiple color photosensitive pixels; C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.

For example, as shown in FIG. 5, for each sub-unit, full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.

For example, as shown in Figure 5, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, a first type subunit UA and a third type subunit UC are arranged in the first diagonal direction D1 (for example, the direction connecting the upper left corner and the lower right corner in FIG. 5), and two second type subunits UB are arranged In the second diagonal direction D2 (for example, the direction where the upper right corner and the lower left corner are connected in FIG. 5). The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

It should be noted that in other embodiments, the first diagonal direction D1 may also be a direction connecting the upper right corner and the lower left corner, and the second diagonal direction D2 may also be a direction connecting the upper left corner and the lower right corner. In addition, the "direction" here is not a single direction, but can be understood as the concept of a "straight line" indicating the arrangement, and there may be two-way directions at both ends of the straight line. The explanation of the first diagonal direction D1 and the second diagonal direction D2 in FIGS. 6 to 10 is the same as here.

For another example, FIG. 6 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in a minimum repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 36 photosensitive pixels 110 in 6 rows and 6 columns, and the sub-units are 9 photosensitive pixels 110 in 3 rows and 3 columns. The arrangement method is:

W represents the full-color photosensitive pixel; A represents the first color photosensitive pixel among the multiple color photosensitive pixels; B represents the second color photosensitive pixel among the multiple color photosensitive pixels; C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.

For example, as shown in FIG. 6, for each sub-unit, full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.

For example, as shown in Figure 6, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1, and two second type subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

For another example, FIG. 7 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the minimum repeating unit is 8 rows and 8 columns and 64 photosensitive pixels 110, and the sub-units are 4 rows and 4 columns and 16 photosensitive pixels 110. The arrangement method is:

W represents the full-color photosensitive pixel; A represents the first color photosensitive pixel among the multiple color photosensitive pixels; B represents the second color photosensitive pixel among the multiple color photosensitive pixels; C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.

For example, as shown in FIG. 7, for each sub-unit, full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.

For example, as shown in Figure 7, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repetition unit includes four subunits, which are a first type subunit UA, two second type subunits UB, and a third type subunit UC. Among them, one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1, and two second type subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

Specifically, for example, FIG. 8 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

W represents the full-color photosensitive pixel; A represents the first color photosensitive pixel among the multiple color photosensitive pixels; B represents the second color photosensitive pixel among the multiple color photosensitive pixels; C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.

The arrangement of the photosensitive pixels 110 in the smallest repeating unit shown in FIG. 8 is roughly the same as the arrangement of the photosensitive pixels 110 in the smallest repeating unit shown in FIG. The alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the subunit UB is inconsistent with the alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the second type of subunit UB in the lower left corner of FIG. 5, and , The alternating sequence of the full-color photosensitive pixel W and the single-color photosensitive pixel in the third type subunit UC in FIG. 8 is the same as the full-color photosensitive pixel W and the single-color photosensitive pixel W in the third type subunit UC in the lower right corner of FIG. 5 The alternating sequence of photosensitive pixels is also inconsistent. Specifically, in the second type subunit UB in the lower left corner of FIG. 5, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (ie, second-color photosensitive pixels B), and The alternating sequence of the two rows of photosensitive pixels 110 is single-color photosensitive pixels (ie, second-color photosensitive pixels B) and full-color photosensitive pixels W; and in the second-type subunit UB in the lower left corner of FIG. 8, the first row The alternating sequence of photosensitive pixels 110 is single-color photosensitive pixels (ie, second-color photosensitive pixels B), full-color photosensitive pixels W, and the alternating sequence of photosensitive pixels 110 in the second row is full-color photosensitive pixels W, single-color photosensitive pixels (ie The second color photosensitive pixel B). In the third type of subunit UC in the lower right corner of FIG. 5, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, third-color photosensitive pixels C), and the second row The alternating sequence of the photosensitive pixels 110 is a single-color photosensitive pixel (that is, a third-color photosensitive pixel C) and a full-color photosensitive pixel W; and in the third type subunit UC in the lower right corner of FIG. 8, the photosensitive pixels 110 in the first row The alternating sequence of the single-color photosensitive pixels (that is, the third color photosensitive pixel C), the full-color photosensitive pixel W, the alternating sequence of the photosensitive pixels 110 in the second row is the full-color photosensitive pixel W, the single-color photosensitive pixel (that is, the third color Photosensitive pixel C).

As shown in FIG. 8, the alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the first type of subunit UA in FIG. The alternating sequence of pixels is not consistent. Specifically, in the first type of sub-unit CA shown in FIG. 8, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, first-color photosensitive pixels A), and the second row The alternating sequence of the photosensitive pixels 110 is a single-color photosensitive pixel (that is, the first color photosensitive pixel A), a full-color photosensitive pixel W; and in the third type of subunit CC shown in FIG. 8, the photosensitive pixels 110 in the first row The alternating sequence is single-color photosensitive pixels (that is, third-color photosensitive pixels C), full-color photosensitive pixels W, and the alternating sequence of photosensitive pixels 110 in the second row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, third-color photosensitive pixels). Pixel C). That is to say, in the same minimum repeating unit, the alternating sequence of full-color photosensitive pixels W and color photosensitive pixels in different subunits can be the same (as shown in Figure 5) or inconsistent (as shown in Figure 8). Show).

For another example, FIG. 9 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

W represents the full-color photosensitive pixel; A represents the first color photosensitive pixel among the multiple color photosensitive pixels; B represents the second color photosensitive pixel among the multiple color photosensitive pixels; C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.

For example, as shown in FIG. 9, for each sub-unit, multiple photosensitive pixels 110 in the same row are photosensitive pixels 110 of the same category. The photosensitive pixels 110 of the same category include: (1) all full-color photosensitive pixels W; (2) all first-color photosensitive pixels A; (3) all second-color photosensitive pixels B; (4) all The third color photosensitive pixel C.

For example, as shown in Figure 9, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repetition unit includes four subunits, which are a first type subunit UA, two second type subunits UB, and a third type subunit UC. Among them, one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1, and two second type subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

For another example, FIG. 10 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

W represents the full-color photosensitive pixel; A represents the first color photosensitive pixel among the multiple color photosensitive pixels; B represents the second color photosensitive pixel among the multiple color photosensitive pixels; C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.

For example, as shown in FIG. 10, for each sub-unit, a plurality of photosensitive pixels 110 in the same column are photosensitive pixels 110 of the same category. The photosensitive pixels 110 of the same category include: (1) all full-color photosensitive pixels W; (2) all first-color photosensitive pixels A; (3) all second-color photosensitive pixels B; (4) all The third color photosensitive pixel C.

For example, as shown in Figure 10, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1, and two second type subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

For example, in other embodiments, in the same minimum repeating unit, multiple photosensitive pixels 110 in the same row in some sub-units may be photosensitive pixels 110 of the same category, and multiple photosensitive pixels 110 in the same column in the remaining sub-units The pixels 110 are photosensitive pixels 110 of the same type.

For example, in the smallest repeating unit shown in FIGS. 5 to 10, the first color photosensitive pixel A may be a red photosensitive pixel R; the second color photosensitive pixel B may be a green photosensitive pixel G; and the third color photosensitive pixel C may be Blue photosensitive pixel Bu.

For example, in the smallest repeating unit shown in FIGS. 5 to 10, the first color photosensitive pixel A may be a red photosensitive pixel R; the second color photosensitive pixel B may be a yellow photosensitive pixel Y; and the third color photosensitive pixel C may be Blue photosensitive pixel Bu.

For example, in the smallest repeating unit shown in FIGS. 5 to 10, the first color photosensitive pixel A may be a magenta photosensitive pixel M; the second color photosensitive pixel B may be a cyan photosensitive pixel Cy; and the third color photosensitive pixel C may It is the yellow photosensitive pixel Y.

It should be noted that, in some embodiments, the response band of the full-color photosensitive pixel W may be the visible light band (for example, 400 nm-760 nm). For example, the full-color photosensitive pixel W is provided with an infrared filter to filter out infrared light. In other embodiments, the response wavelength bands of the full-color photosensitive pixel W are visible light and near-infrared wavelengths (for example, 400nm-1000nm), and the photoelectric conversion element 1111 (shown in FIG. 4) in the image sensor 10 (shown in FIG. 1) (Shown) to match the response band. For example, the full-color photosensitive pixel W may not be provided with a filter or a filter that can pass light of all wavelength bands. The response band of the full-color photosensitive pixel W is determined by the response band of the photoelectric conversion element 1111, that is, the two match. . The embodiments of the present application include, but are not limited to, the above-mentioned waveband range.

1 to 3, FIG. 5, FIG. 11, and FIG. 12, in some embodiments, the control unit 13 controls the pixel array 11 to expose. Wherein, the pixel array 11 is exposed for the first exposure time to obtain the first original image. The first original image includes first color original image data generated by single-color photosensitive pixels exposed at the first exposure time and first full-color original image data generated by panchromatic photosensitive pixels exposed at the first exposure time. The pixel array 11 is exposed for a second exposure time to obtain a second original image. The second original image includes second color original image data generated by the single-color photosensitive pixels exposed at the second exposure time and second full-color original image data generated by the panchromatic photosensitive pixels exposed at the second exposure time; wherein, the first The exposure time is not equal to the second exposure time.

Specifically, the image processor 20 can control the pixel array 11 to perform two exposures. For example, as shown in FIG. 11, in the first exposure, the pixel array 11 is exposed for the first exposure time L to obtain the first original image. The first original image includes first color original image data generated by the single-color photosensitive pixels exposed at the first exposure time L and first full-color original image data generated by the panchromatic photosensitive pixels exposed at the first exposure time L. In the second exposure, the pixel array 11 is exposed for the second exposure time S to obtain a second original image. The second original image includes second color original image data generated by the single-color photosensitive pixels exposed at the second exposure time S and second full-color original image data generated by the panchromatic photosensitive pixels exposed at the second exposure time S.

In some embodiments, the pixel array 11 may also be exposed for a third exposure time to obtain a third original image. The third original image includes third color original image data generated by the single-color photosensitive pixels exposed at the third exposure time and third full-color original image data generated by the panchromatic photosensitive pixels exposed at the third exposure time. Wherein, the third exposure time is not equal to the first exposure time, and the third exposure time is not equal to the second exposure time. The image processor 20 and the high dynamic fusion unit 50 (which may include a color high dynamic fusion unit 30 and a panchromatic high dynamic fusion unit 40) are used to perform image preprocessing on the first original image, the second original image, and the third original image. High dynamic range processing, image processing and fusion algorithm processing get the target image.

Specifically, referring to FIG. 13, the image processor 20 can control the pixel array 11 to perform three exposures. The first original image, the second original image, and the third original image are obtained respectively. The first original image includes first color original image data generated by a single-color photosensitive pixel exposed at the first exposure time L and first full-color original image data generated by a panchromatic photosensitive pixel exposed at the first exposure time L. The second original image includes second color original image data generated by the single-color photosensitive pixels exposed at the second exposure time M and second full-color original image data generated by the panchromatic photosensitive pixels exposed at the second exposure time M. The third original image includes third color original image data generated by the single-color photosensitive pixels exposed at the third exposure time S and third full-color original image data generated by the panchromatic photosensitive pixels exposed at the third exposure time S.

In other embodiments, the image processor 20 may also control the pixel array 11 to perform more exposures such as four, five, six, ten, or twenty times, so as to obtain more original images. The image processor 20, the color high dynamic fusion unit 30, and the panchromatic high dynamic fusion unit 40 then perform image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on all original images to obtain the target image.

It should be noted that, in some embodiments, the exposure process of the pixel array 11 may be: (1) The pixel array 11 takes at least two exposure times (for example, the first exposure time L and the second exposure time S, or the first exposure time L and the second exposure time S). Exposure time L, second exposure time M, and third exposure time S) sequential exposure (the order of exposure for different exposure times is not limited), and the exposure time of at least two exposures does not overlap on the time axis; (2) The pixel array 11 is exposed with at least two exposure times (for example, the first exposure time L and the second exposure time S, or the first exposure time L, the second exposure time M, and the third exposure time S) (where the exposure of different exposure times The sequence is not limited), and the exposure time of at least two exposures partially overlaps on the time axis; (3) All the exposure time of shorter exposure time is within the exposure time of the longest exposure time; for example, , The exposure time of the second exposure time S is within the exposure time of the first exposure time L; for another example, the second exposure time M and the third exposure time S are both within the exposure time of the first exposure time L. (4) All exposures can start at the same time and end at different times, or all exposures can end at the same time and start at different times. The high dynamic range processing system 100 of the embodiment of the present application can adopt the (3) or (4) exposure mode. Using this exposure mode can shorten the exposure time required by the pixel array 11 in one shot, which is beneficial to improve the image quality. The frame rate is also beneficial to minimize the interval between the exposure times of at least two exposures, so that the exposure times of multiple frames of images are closer, thereby improving the image of a highly dynamic image merged by multiple images with different exposure times quality.

Specifically, the exposure time of at least two exposures has overlapping exposure modes (for example, the above-mentioned (2), (3), and (4) exposure modes), as shown in FIG. 14, which can be determined by the image sensor A buffer processor 16 is provided in 10, and the buffer processor 16 cooperates with the control unit and the pixel array 11 to work. Taking the exposure method (4) as an example, please refer to 12. The image sensor 10 controls the pixel array 11 to perform three exposures, which are respectively a first exposure time of 1s, a second exposure time of 1/8s, and a third exposure time of 1/64s. The control unit of the image sensor 10 controls the pixel array 11 to output exposure image data with an exposure duration of 1/512 every 1/512s, and store the exposure image data in the buffer processor 16. After the buffer processor 16 receives the exposure image data, it stores the received exposure image data in the buffer memory area inside the buffer processor 16, and after a shooting starts, it will accumulate 8 images after receiving 8 exposure data. After the addition processing of the exposure image data, it is transmitted to the image sensor 10 as the third original image, and after the cumulative 64 image exposure data are received, the cumulative 64 exposure image data is added and then processed as the second original image. The image is transmitted to the image sensor 10, after 512 image exposure data are received in total, the accumulated 512 exposure image data are added together, and then transmitted to the image sensor 10 as the first original image, and 512 exposure data are received in total After that, the image sensor 10 controls the exposure of this shooting to end. In the embodiment of the present application, the buffer processor 16 is set to cooperate with the control unit and the pixel array 11, so as to complete at least two exposures in the embodiment of the present application with a simple device and working logic. (2), (3), and (4) exposure methods), which help improve the reliability of the system, and at the same time help shorten the exposure time required by the pixel array 11 in one shot, and improve the image frame At the same time, it is beneficial to reduce the interval between the exposure times of at least two exposures, so that the exposure times of multiple frames of images are closer, thereby improving the image quality of a high dynamic image fused by multiple images with different exposure times.

1, the image processor 20 may include a color preprocessing module 2023, a full color preprocessing module 2024, a color processing module 2021, a full color processing module 2022, and a fusion module 204. Image preprocessing can include pixel completion processing and demosaicing processing. Image processing includes first image processing and second image processing. Among them, the color preprocessing module 2023 can be used to perform pixel complement processing on the color original image data to obtain the color original image. The panchromatic preprocessing module 2024 can be used to demosaicate the panchromatic original image data to obtain the panchromatic original image. The color processing module 2021 may be used to perform first image processing on the color original image to obtain a color intermediate image. The panchromatic processing module 2022 may be used to perform second image processing on the panchromatic original image to obtain a panchromatic intermediate image. The fusion module 204 may be used to perform fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image. In some embodiments, the image processor 20 further includes an image front-end processing unit 202. The color preprocessing module 2023, the full color preprocessing module 2024, the color processing module 2021, and the full color processing module 2022 may be integrated in the image front-end processing unit 202.

The specific operation process of the panchromatic preprocessing module 2024 for demosaicing the panchromatic original image data is similar to the specific operation process of the demosaicing process for the first color original image and the second color original image in the following embodiments of the application. In the following, we will explain in detail in a unified way.

Referring to FIG. 15, FIG. 16, and FIG. 17, the specific operation process of the color preprocessing module 2023 performing pixel complement processing on the color original image data may include the following steps: (1) Decompose the color original image data into the first color original image Data (the original image data generated by the first color photosensitive pixel A described above), the second color original image data (the original image data generated by the second color photosensitive pixel B described above), and the third color Original image data (original image data generated by the third color photosensitive pixel C described above). (2) Perform an averaging operation on the pixel values generated by the multiple first-color photosensitive pixels A of the sub-unit in the original image data of the first color, and after the average value is obtained, the pixel grids in the sub-unit range are merged into a pixel grid. The average value is filled into the pixel grid to obtain the first color intermediate image data. (3) Interpolate the first color intermediate image data by using a bilinear interpolation method to obtain the first color interpolated image data. The specific operation method of bilinear interpolation will be explained in detail below. (4) Fusion of the first color interpolated image data and the first color original image data to obtain the first color original image. (5) After performing the above steps (2), (3) and (4) on the original image data of the first color, the original image data of the second color, and the original image data of the third color, the resulting image with one color channel The original image of the first color, the original image of the second color, and the original image of the third color are synthesized into a color original image with the same resolution of the three color channels and the same resolution of the color original image. The color preprocessing module 2023 can perform the pixel complement processing of the above steps on all color original image data corresponding to at least two exposures, thereby completing the pixel complement processing of all color original image data, and obtain the color corresponding to at least two exposures. The original image. Specifically, referring to FIG. 15, FIG. 16, and FIG. 17, the color preprocessing module 2023 performs pixel complement processing on the first red original image data in the first color original image data as an example for description. As shown in Figure 15, the color preprocessing module 2023 first decomposes the color original image (which can be the first color original image, the second color original image, the third color original image, etc.) data into red original image data and green original image data. And blue raw image data. As shown in FIG. 16, the color preprocessing module 2023 then performs an averaging operation on the pixel values (such as L1 and L2) generated by the multiple red photosensitive pixels R in the subunit of the red original image data to obtain the average value L1' =(L1+L2)/2 After fusing the pixel grid of the sub-unit range into a pixel grid, and filling the average value into the pixel grid, the red intermediate image data is obtained. Then, the color preprocessing module 2023 uses a bilinear interpolation method to interpolate the red intermediate image data to obtain red interpolated image data. Next, the color preprocessing module 2023 fuses the red interpolated image data and the red original image data to obtain a red original image. In the fusion process, first, the color preprocessing module 2023 generates a null image with the same resolution as the red original image data, the pixel color arrangement in the smallest repeating unit and the red interpolated image data, and then the fusion is performed according to the following principles: ( 1) If there are pixel values in the same coordinates of the first red original image data, and the color channels are the same, then directly fill the pixel values in the same coordinates of the first red original image data into the empty image; (2) If in The first red original image data has pixel values in the same coordinates but different color channels, then the pixel values in the corresponding coordinates of the first red interpolated image data are filled into the empty image; (3) If the first red original image is If there is no pixel value in the same coordinate of the data, then the pixel value in the corresponding coordinate of the first red interpolated image data is filled into the empty image. According to the above fusion principle, as shown in Figure 16, a red original image will be obtained. Similarly, as shown in FIG. 17, the color preprocessing module 2023 can obtain a red original image, a green original image, and a blue original image, and combine the resulting red original image, green original image, and blue original image with one color channel. The original image is synthesized into a color original image with 3 color channels. The color preprocessing module 2023 can perform the pixel compensation of the above steps on the first color original image data and the second color original image data (or the first color original image data, the second color original image data, and the third color original image data). Full processing, thereby completing the pixel complement processing of the color original image data to obtain the first color original image and the second color original image (or the first color original image, the second color original image, and the third color original image). The high dynamic range image processing system 100 of the embodiment of the present application performs pixel complement processing on the color information in some pixel grids where the color information is missing and the pixel grids with color information only have color original image data with single color channel information. In the case of high rate, the color information of the complete channel with the complete pixel grid is obtained, and then the color original image is obtained, so that other image processing can be continued on the image subsequently to improve the imaging quality.

Referring to FIG. 1, in some embodiments, after the fusion module 204 performs fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image, the high dynamic fusion unit 50 may expose the target image corresponding to at least two exposures ( It may include the first target image and the second target image) fusion to obtain a highly dynamic target image.

Referring to FIG. 18, in other embodiments, the high-dynamic fusion unit 50 may include a color high-dynamic fusion unit 30 and a full-color high-dynamic fusion unit 40. Before the color preprocessing module 2023 performs pixel complement processing on the color original image data to obtain the color original image, the color high dynamic fusion unit 30 may fuse the color original image data corresponding to at least two exposures to obtain high dynamic color original image data . Before the panchromatic preprocessing module 2024 performs demosaic processing on the panchromatic original image data to obtain the panchromatic original image, the panchromatic high dynamic fusion unit 40 is used to fuse the panchromatic original image data corresponding to at least two exposures to obtain a high dynamic The full-color original image data.

Referring to FIG. 19, in still other embodiments, the high-dynamic fusion unit 50 may include a color high-dynamic fusion unit 30 and a full-color high-dynamic fusion unit 40. Before the color processing module 2021 performs first image processing on the color original image to obtain a color intermediate image, the color high dynamic fusion unit 30 may fuse the color original images corresponding to at least two exposures to obtain a high dynamic color original image. Before the panchromatic processing module 2022 performs the second image processing on the panchromatic original image to obtain a panchromatic intermediate image, the panchromatic high dynamic fusion unit 40 may fuse the panchromatic original images corresponding to at least two exposures to obtain a highly dynamic panchromatic original image. image.

Referring to FIG. 20, in still other embodiments, the high-dynamic fusion unit 50 may include a color high-dynamic fusion unit 30 and a full-color high-dynamic fusion unit 40. Before the fusion module 204 is used to perform fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image, the color high dynamic fusion unit 30 may fuse the color intermediate images corresponding to at least two exposures to obtain a high dynamic color intermediate image, The panchromatic high dynamic fusion unit 40 can merge panchromatic intermediate images corresponding to at least two exposures to obtain a highly dynamic panchromatic intermediate image.

In the color processing module 2021, the first image processing may include one or more of black level correction processing, lens shading correction processing, demosaicing processing, dead pixel compensation processing, color correction processing, global tone mapping processing, and color conversion processing. One; in the panchromatic processing module 2022, the second image processing may include one or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, and global tone mapping processing.

Specifically, the first image processing may include a first image sub-processing and a second image sub-processing. The color processing module 2021 may first perform the first image sub-processing on the color original image, and then perform the second image sub-processing. The first image sub-processing may include one or more of black level correction processing, lens shading correction processing, and dead pixel compensation processing. The second image sub-processing may include one or more of demosaicing processing, color correction processing, global tone mapping processing, and color conversion processing.

Because the information collected by the image sensor undergoes a series of conversions to generate the original image. Taking 8bit data as an example, the effective value of a single pixel is 0-255, but the accuracy of the analog-to-digital conversion chip in the actual image sensor may not be able to convert a small part of the voltage value, which will easily cause the loss of the dark details of the generated image . The process of black level correction processing may be that the color processing module 2021 or the panchromatic processing module 2022 subtracts a fixed value from each pixel value on the basis of the original image data output by the image sensor 10. Each color channel (such as a red channel, a green channel, a blue channel, and a panchromatic channel. In some embodiments, the red channel refers to the red information generated by the red photosensitive pixels in the image output by the image sensor 10, and the green channel refers to Is the green information generated by the green photosensitive pixels in the image output by the image sensor 10. The red channel refers to the blue information generated by the blue photosensitive pixels in the image output by the image sensor 10, and the panchromatic channel refers to the image sensor 10. The fixed value corresponding to the full-color information generated by the full-color photosensitive pixel in the output image may be the same or different. Specifically, referring to FIG. 20, the image sensor 10 controls the pixel array 11 to perform two exposures (which may be two or more times) as an example for description. The image sensor 10 can output the first color original image data and the second Color original image data, first full-color original image data, and second full-color original image data. The image processor 20 receives the first color original image data, the second color original image data, the first full-color original image data, and the second full-color original image data. After the full-color original image data, the color preprocessing module 2023 performs pixel complement processing on the first color original image data and the second color original image data to obtain the first color original image and the second color original image. A color original image and a second color original image are subjected to the black level correction processing in the first image processing; the panchromatic preprocessing module 2024 performs demosaicing processing on the first panchromatic original image data and the second panchromatic original image data. For the first full-color original image and the second full-color original image, the full-color processing module 2022 performs black level correction processing in the second image processing on the first full-color original image and the second full-color original image. Taking the color processing module 2021 performing black level correction processing on the first color original image as an example, the first color original image has a red channel, a green channel, and a blue channel. Please refer to FIG. 21. The color processing module 2021 performs black level correction processing on the first color original image. All pixel values in the first color original image are subtracted from a fixed value of 5, thereby obtaining the first black level correction processing. Color original image. At the same time, the image sensor 10 adds a fixed offset of 5 (or other values) before the input of AD, so that the output pixel value is between 5 (or other values) to 255. With the black level correction processing, it can make While the details of the dark parts of the image obtained by the image sensor 10 and the high dynamic range image processing system 100 of the embodiment of the present application are completely preserved, the pixel value of the image is not increased or decreased, which is beneficial to improving the imaging quality.

Lens shadow is the phenomenon that the lens has a shadow around the lens caused by the uneven optical refraction of the lens, that is, the intensity of the received light in the center and the surrounding area of the image area is inconsistent. The process of lens shading correction processing can be that the color processing module 2021 or panchromatic processing module 2022 can mesh the processed image on the basis of the color original image and the panchromatic original image that have undergone black level correction processing, and then The lens shading correction is performed on the image by the bilinear interpolation method through the compensation coefficients of each grid area adjacent or itself and adjacent circumferences. The following takes the lens shading correction processing on the first color original image as an example for description. As shown in FIG. 22, the color processing module 2021 divides the first color original image (that is, the processed image) into sixteen equally Grid, each of the sixteen grids has a preset compensation coefficient. Then, the color processing module 2021 performs shading correction on the image by a bilinear interpolation method according to the compensation coefficients adjacent to each grid area or itself and its vicinity. R2 is the pixel value in the dashed frame in the first color intermediate image that has undergone lens shading correction processing, and R1 is the pixel value in the dashed frame in the first color original image shown in the figure. R2=R1*k1, k1 is obtained by bilinear interpolation of the compensation coefficients 1.10, 1.04, 1.105, and 1.09 of the grid adjacent to the R1 pixel. Suppose the coordinates of the image are (x, y), x is counted from the first pixel from the left to the right, y is counted from the first pixel on the top, and both x and y are natural numbers, as indicated by the logo on the edge of the image Show. For example, if the coordinates of R1 are (3,3), the coordinates of R1 in each grid compensation coefficient map should be (0.75,0.75). f(x, y) represents the compensation value of the coordinate (x, y) in each grid compensation coefficient graph. Then f(0.75, j0.75) is the compensation coefficient value corresponding to R1 in each grid compensation coefficient graph. The interpolation formula of bilinear interpolation can be f(i+u,j+v)=(1-u)(1-v)f(i,j)+(1-u)vf(i,j+1) +u(1-v)f(i+1,j)+uvf(i+1,j+1), where x=i+u, i is the integer part of x, u is the fractional part of x, j Is the integer part of y, and v is the decimal part of y. Then f(0.75,j0.75)=(0.25)*(0.25)*f(0,0)+0.25*0.75*f(0,1)+0.75*0.25*f(1,0)+0.75* 0.75f(1,1)=0.0625*1.11+0.1875*1.10+0.1875*1.09+0.5625*1.03. The compensation coefficient of each grid has been preset before the color processing module 2021 or the panchromatic processing module 2022 performs lens shading correction processing. The compensation coefficient of each grid can be determined by the following methods: (1) Place the lens 300 in a closed device with constant and uniform light intensity and color temperature, and make the lens 300 face a pure gray target with uniform brightness distribution in the closed device The object is shot to obtain a grayscale image; (2) The grayscale image is gridded (for example, divided into 16 grids) to obtain the grayscale image divided into different grid areas; (3) The different grids of the grayscale image are calculated The compensation coefficient of the grid area. After determining the compensation coefficient of the lens 300, the high dynamic range image processing system 100 of the present application presets the compensation coefficient in the color processing module 2021 or the panchromatic processing module 2022. When the color processing in the high dynamic range image processing system 100 When the module 2021 or the panchromatic processing module 2022 performs lens shading correction processing on the image, the compensation coefficient is obtained, and the color processing module 2021 or the panchromatic processing module 2022 then uses the bilinear interpolation method to perform the correction according to the compensation coefficient of each grid area The image undergoes lens shading correction processing.

The photosensitive pixels on the pixel array of the image sensor may have defects in process, or errors in the process of converting optical signals into electrical signals, resulting in incorrect pixel information on the image and inaccurate pixel values in the image. These are defects The pixels that appear on the output image are image dead pixels. Image dead pixels may exist, so the image needs to be compensated for dead pixels. The dead pixel compensation process may include the following steps: (1) create a 3×3 pixel matrix of the same color photosensitive pixels with the pixel to be detected as the center pixel; (2) use the surrounding pixels of the central pixel As a reference point, it is judged whether the difference between the color value of the central pixel and the surrounding pixels is greater than the first threshold. If yes, the central pixel is a dead pixel; if not, the central pixel is Normal point; (3) Perform bilinear interpolation on the central pixel point judged as a bad point to obtain the corrected pixel value. Please refer to FIG. 23. The following describes the dead pixel compensation processing on the first full-color original image. In the first image in FIG. 23, R1 is the pixel to be detected. A 3×3 pixel matrix of pixels of the same color of the photosensitive pixels, and the second image in FIG. 23 is obtained. And taking the surrounding pixels of the central pixel R1 as a reference point, it is determined whether the difference between the color value of the central pixel R1 and the surrounding pixel is greater than the first threshold Q (Q is preset in the color processing module 2021 ). If it is, the central pixel R1 is a bad pixel, and if not, the central pixel R1 is a normal pixel. If R1 is a dead pixel, then bilinear interpolation is performed on R1 to obtain the corrected pixel value R1' (the case where R1 is a dead pixel is shown in the figure) to obtain the third image in FIG. 23. Please refer to FIG. 24 to describe the dead pixel compensation processing performed by the panchromatic processing module 2022 on the first panchromatic original image that has undergone lens shading correction processing. W1 in the first picture in Fig. 24 is the pixel to be detected. The panchromatic processing module 2022 uses W1 as the center pixel to establish a 3×3 pixel matrix of pixels of the same color as the photosensitive pixel of W1 to obtain the first pixel in Fig. 24 Two pictures. And taking the surrounding pixels of the central pixel W1 as a reference point, it is determined whether the difference between the color value of the central pixel W1 and the surrounding pixels is greater than the first threshold K (K is preset in the panchromatic processing module 2022) . If it is, the central pixel W1 is a bad pixel, and if not, the central pixel W1 is a normal pixel. If W1 is a dead pixel, perform bilinear interpolation on W1 to obtain the corrected pixel value W1' (shown in the figure is the case where W1 is a dead pixel) to obtain the third image in FIG. 24. The color processing module 2021 and the full color processing module 2022 of the embodiment of the present application can perform dead pixel compensation processing on the image, which is beneficial to the high dynamic range image processing system 100 to eliminate the presence of photosensitive pixels in the imaging process of the high dynamic range image processing system 100 Process defects, or image defects caused by errors in the process of converting optical signals into electrical signals, thereby improving the accuracy of the pixel values of the target image formed by the high dynamic range image processing system 100, thereby enabling the implementation of this application The method has a better imaging effect.

Since each pixel grid of the color original image (such as the first color original image and the second color original image) of the embodiment of the present application is a single-color pixel, there is no optical information of other colors, so it is necessary to compare the first color original image Perform demosaicing with the second color original image. In addition, the full-color preprocessing module 2024 can also perform demosaic processing on the full-color original image data to obtain a full-color original image. In the following, the color processing module 2021 performs demosaic processing on the first color original image (for example, including the red channel, the green channel, and the blue channel) as an example. The demosaic processing steps include the following steps: (1) The first color The original image is decomposed into a first red original image, a first green original image, and a first blue original image, as shown in FIG. 25, among the obtained first red original image, first green original image, and first blue original image Some pixel grids have no pixel value. (2) The first red original image, the first green original image, and the first blue original image are respectively interpolated by using a bilinear interpolation method. As shown in FIG. 26, the color processing module 2021 uses a bilinear interpolation method to perform interpolation processing on the first blue original image. The pixel B1 to be interpolated in FIG. 26 performs bilinear interpolation according to the four pixels B2, B3, B4, and B5 around B1 to obtain the interpolated pixel B1' of B1. The pixels to be interpolated in all blank spaces in the first image in FIG. 26 are traversed to use the bilinear interpolation method to complete the pixel values to obtain the interpolated first blue original image. As shown in FIG. 27, the color processing module 2021 uses a bilinear interpolation method to perform interpolation processing on the first green original image. The pixel to be interpolated G1 in FIG. 27 performs bilinear interpolation according to the four pixels G2, G3, G4, and G5 around G1 to obtain the interpolated pixel G1' of G1. All the pixels to be interpolated in the blanks in the first image in FIG. 27 are traversed to use the bilinear interpolation method to complement the pixel values to obtain the interpolated first green original image. Similarly, the color processing module 2021 may use a bilinear interpolation method to perform interpolation processing on the first red original image to obtain the interpolated first red original image. (3) Re-synthesize the interpolated first red original image, the interpolated first green original image, and the interpolated first blue original image into an image with 3 color channels, as shown in FIG. 28. The color processing module 2021 performs demosaic processing on the color image, which is beneficial for the implementation of the present application to complete the color image with the pixel value of a single color channel into a color image with multiple color channels, so that the hardware of the single-color photosensitive pixel On the basis of maintaining the complete presentation of the image color.

The color correction processing can specifically be to use a color correction matrix to correct the color channel values of each pixel of the color original image (which can be the first color original image and the second color original image that have undergone demosaicing processing), thereby realizing Correction of image color. As follows:

Among them, the color correction matrix (CCM) is preset in the color processing module. For example, the color correction matrix can be specifically:

The color processing module traverses all pixels in the image and performs color correction processing through the above color correction matrix to obtain an image that has undergone color correction processing. The color correction processing in the embodiment of the present application is beneficial to eliminate the serious color deviation caused by colored light sources in the image or video frame, and the color distortion of people or objects in the image, so that the high dynamic range image processing system 100 of the embodiment of the present application can Restore the original color of the image and improve the visual effect of the image.

The tone mapping process may include the following steps: (1) Normalize the gray value of the color original image (which may be the first color original image and the second color original image that have undergone color correction processing) into the interval [0,1] , Record the normalized gray value as Vin; (2) Set Vout=Y(Vin), the mapping relationship between Vout and Vin can be as shown in Figure 29; (3) Multiply Vout by 255 (when When the grayscale value of the output image is set to 256 levels, multiply it by 255. In other settings, it can be other values) and then round to the nearest integer to obtain the image after the tone mapping process. For images with high dynamic range, the binary digits of the gray value are often higher than 8 bits (the binary digits of the gray value of ordinary gray-scale images are generally 8 bits), and the gray scale of many displays is only 8 bits Therefore, the color of the high dynamic range image is changed, which is beneficial for the high dynamic range image to have higher compatibility and can be displayed on a conventional monitor. In addition, because the high dynamic range image generally has a very uneven distribution of gray values, only a few pixels are brighter, and most of the pixels are distributed in the interval with a lower gray value. The high dynamic range image processing system of the embodiment of the present application The tone mapping processing of 100 pairs of images is non-linear, but the slope of the mapping relationship in the interval with lower gray value is greater than that in the interval with higher gray value. As shown in Figure 29, it is beneficial to gray The degree of discrimination of pixels with different gray values in the interval with a lower degree value, and most of the pixels are distributed in the interval with a lower gray value, so that the high dynamic range image processing system 100 of the embodiment of the present application has better The imaging effect.

In order for the image to have a wider range of application scenarios or a more efficient transmission format, the high dynamic range image processing system 100 of the embodiment of the present application can perform processing of color original images (which may be the first color original image and the second color original image that have undergone tone mapping processing). Color original images) perform color conversion processing to convert the image from one color space (for example, RGB color space) to another color space (for example, YUV color space) to have a wider range of application scenarios or a more efficient transmission format. In a specific embodiment, the step of color conversion processing may be to convert the R, G, and B channel pixel values of all the pixel values in the image to obtain the Y, U, and V channel pixel values: (1) Y=0.30 R+0.59G+0.11B; (2) U=0.493 (B-Y); (3) V=0.877 (R-Y); thereby converting the image from the RGB color space to the YUV color space. Since the luminance signal Y and the chrominance signal U and V in the YUV color space are separated, and the human eye is more sensitive to luminance than chrominance, the color conversion process to convert the image from the RGB color space to the YUV color space is beneficial to the implementation of this application The subsequent image processing of the high dynamic range image processing system 100 compresses the chrominance information of the image, which can reduce the amount of information of the image while not affecting the viewing effect of the image, thereby improving the transmission efficiency of the image.

In some embodiments, the high dynamic fusion unit 50 may perform brightness alignment processing on the target image (which may include the first target image and the second target image) corresponding to at least two exposures to obtain a brightness-aligned target image, and then merge the target image. Brightly aligned target image and one or more target images to obtain a highly dynamic target image.

In other embodiments, the color high dynamic fusion unit 30 may perform brightness alignment processing on the color original image data corresponding to at least two exposures (for example, the first color original image data and the second color original image data) to obtain the brightness alignment. The original color image data is combined with brightness-aligned color original image data and one or more color original image data to obtain highly dynamic color original image data. The panchromatic high dynamic fusion unit 40 may perform brightness alignment processing on panchromatic original image data corresponding to at least two exposures (for example, the first panchromatic original image data and the second panchromatic original image data) to obtain a brightness-aligned panchromatic The original image data is then fused with brightness-aligned panchromatic primitive image data and one or more panchromatic primitive image data to obtain highly dynamic panchromatic primitive image data.

In still other embodiments, the color high dynamic fusion unit 30 may perform brightness alignment processing on the color original images corresponding to at least two exposures (for example, the first color original image and the second color original image) to obtain brightness-aligned color original images. Image, and then merge the brightness-aligned color original image and one or more color original images to obtain a highly dynamic color original image. The panchromatic high dynamic fusion unit 40 may perform brightness alignment processing on panchromatic original images corresponding to at least two exposures (for example, a first panchromatic original image and a second panchromatic original image) to obtain a panchromatic original image with aligned brightness. Then merge the brightness-aligned panchromatic original image and one or more panchromatic original images to obtain a highly dynamic panchromatic original image.

In still other embodiments, the color high dynamic fusion unit 30 may perform brightness alignment processing on the color intermediate images corresponding to at least two exposures (for example, the first color intermediate image and the second color intermediate image) to obtain a brightness-aligned color intermediate image. Image, and then merge the brightness-aligned color intermediate image and one or more color intermediate images to obtain a highly dynamic color intermediate image. The panchromatic high dynamic fusion unit 40 may perform brightness alignment processing on panchromatic intermediate images corresponding to at least two exposures to obtain panchromatic intermediate images with aligned brightness (for example, a first panchromatic intermediate image and a second panchromatic intermediate image), Then merge the brightness-aligned panchromatic intermediate image and one or more panchromatic intermediate images to obtain a highly dynamic panchromatic intermediate image.

Specifically, the high dynamic range processing performed on the image by the high dynamic fusion unit 50 may include brightness alignment processing. The high dynamic fusion unit 50 (which may include the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40) performs brightness alignment processing on the image and includes the following steps (the number of images can be equal to the exposure of the image sensor 10 to control the pixel array 11 The number of times, the image can be the first color original image data and the second color original image data, the first color original image data, the second color original image data and the third color original image data, the first target image and the second target image , The first color original image and the second color original image, the first color intermediate image and the second color intermediate image, the first panchromatic original image and the second panchromatic original image, the first panchromatic intermediate image and the second panchromatic image Intermediate image, the first color original image, the second color original image and the third color original image, the first panchromatic original image, the second panchromatic original image and the third panchromatic original image, the first color intermediate image, the second The color intermediate image and the third color intermediate image, one of the first full color intermediate image, the second pan The image (corresponding to the long-term L exposure), the second color intermediate image (corresponding to the intermediate time M exposure), and the third color intermediate image (corresponding to the short-term S exposure) are subjected to brightness alignment processing as an example for description): (1) Identify the overexposed image pixels in the first color intermediate image whose pixel value is greater than the first preset threshold; (2) For each overexposed image pixel, expand a predetermined area with the overexposed image pixel as the center; (3) Find intermediate image pixels with pixel values less than the first preset threshold in a predetermined area; (4) Use intermediate image pixels, second color intermediate image, and third color intermediate exposure image to correct the pixel value of overexposed image pixels; ( 5) The first color intermediate image is updated by using the corrected pixel values of the pixels of the overexposed image to obtain the first color intermediate image with aligned brightness. Specifically, please refer to Figure 30, assuming that the pixel value V1 of the image pixel P12 (the image pixel marked with a dashed circle in the first color intermediate image in Figure 30) is greater than the first preset threshold V0, that is, the image pixel P12 is an overexposed image For the pixel P12, the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 extends a predetermined area with the overexposed image pixel P12 as the center, for example, the 3*3 area shown in FIG. 30. Of course, in other embodiments, it may also be a 4*4 area, a 5*5 area, a 10*10 area, etc., which is not limited here. Subsequently, the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 searches for intermediate image pixels whose pixel value is less than the first preset threshold V0 in a predetermined area of 3*3, such as the image pixel P21 in FIG. 30 (FIG. 30). If the pixel value V2 of the image pixel marked with a dotted circle in the first color intermediate image is less than the first preset threshold V0, the image pixel P21 is the intermediate image pixel P21. Subsequently, the color high dynamic fusion unit 30 searches for the image pixels corresponding to the overexposed image pixel P12 and the intermediate image pixel P21 respectively in the second color intermediate image, that is, the image pixel P1'2' (in the second color intermediate image in FIG. 30) The image pixels marked with a dashed circle) and the image pixel P2'1' (the image pixels marked with a dotted circle in the second color intermediate image in Figure 30), where the image pixel P1'2' corresponds to the overexposed image pixel P12 , The image pixel P2'1' corresponds to the intermediate image pixel P21, the pixel value of the image pixel P1'2' is V3, and the pixel value of the image pixel P2'1' is V4. Subsequently, the color high dynamic fusion unit 30 calculates V1' according to V1'/V3=V2/V4. When V1' is less than the first preset threshold V0, the color high dynamic fusion unit 30 uses the value of V1' to replace the value of V1. When V1' is greater than the first preset threshold V0, the color high dynamic fusion unit 30 is in the third In the color intermediate image, look for the image pixels corresponding to the overexposed image pixel P12 and the intermediate image pixel P21 respectively, that is, the image pixels P1"2" and P2"1", the pixel value of the image pixel P1"'2" is V5, and the image pixel The pixel value of P2"1" is V6, and in the same way, V1" is finally calculated according to V1"/V5=V2/V6, and the value of V1" is used to replace the value of V1. From this, the overexposed image can be calculated The actual pixel value of pixel P12. The color high dynamic fusion unit 30 or panchromatic high dynamic fusion unit 40 performs this brightness alignment process on each overexposed image pixel in the first color intermediate image to obtain the brightness alignment Since the pixel value of the overexposed image pixel in the first color intermediate image after brightness alignment has been corrected, the pixel value of each image pixel in the first color intermediate image after brightness alignment is equal More accurate.

In the high dynamic range processing process, after the first color intermediate image (or other images that have undergone the above brightness alignment processing) after the brightness alignment is obtained, the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 can adjust the brightness The aligned images are merged with similar images to obtain a highly dynamic image. Specifically, the following uses the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 to adjust the brightness of the first color intermediate image (obtained by the long-time L exposure) and the second color intermediate image (exposed by the middle time M). Correspondingly obtained) and the third color intermediate image (obtained by the short-time S exposure) are fused to obtain a highly dynamic color intermediate image for description. The color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 first performs motion detection on the first color intermediate image after brightness alignment to identify whether there is a motion blur area in the first color intermediate image after brightness alignment. If there is no motion blur area in the first color intermediate image after brightness alignment, the first color intermediate image, the second color intermediate image, and the third color intermediate image after brightness alignment are directly merged to obtain a highly dynamic color intermediate image. If there is a motion blur area in the first color intermediate image after brightness alignment, remove the motion blur area in the first color intermediate image, and only merge all areas of the second color intermediate image and the third color intermediate image, and after the brightness is aligned The first color intermediate image except for the motion blur area in the first color intermediate image to obtain a highly dynamic color intermediate image. Wherein, the resolution of the high dynamic color intermediate image may be equal to the resolution of the pixel array 11. Specifically, when fusing the first color intermediate image, the second color intermediate image, and the third color intermediate image after brightness alignment, if there is no motion blur area in the first color intermediate image after brightness alignment, then two The fusion of the intermediate image follows the following principles: (1) In the first color intermediate image after brightness alignment, the pixel value of the image pixel in the overexposed area is directly replaced with the pixel value of the image pixel in the second color intermediate image corresponding to the overexposed area. Pixel value; if the pixel value of the image pixel corresponding to the overexposed area in the second color intermediate image is also overexposed, then the pixel value of the image pixel in the overexposed area in the first color intermediate image after brightness alignment is directly replaced with The pixel value of the image pixel in the third color intermediate image corresponding to the overexposed area; (2) In the first color intermediate image after brightness alignment, the pixel value of the image pixel in the underexposed area is: the long exposure pixel value divided by Coefficient K1, coefficient K1 is the average of K2 and K3; K2 is the ratio of long-exposure pixel value to medium-exposure pixel value, K3 is the ratio of long-exposure pixel value and short-exposure pixel value; (3) the first after brightness alignment In the color intermediate image, the pixel value of the image pixel in the area neither under-exposed nor over-exposed is: the long-exposure pixel value divided by the coefficient K1. If there is a motion blur area in the first color intermediate image after brightness alignment, the fusion of the three intermediate images at this time not only follows the above three principles, but also needs to follow the (4) principle: the first color after brightness alignment In the intermediate image, the pixel value of the image pixel in the motion blur area is directly replaced with the pixel value of the image pixel corresponding to the motion blur area in the second color intermediate image and the image pixel corresponding to the motion blur area in the third color intermediate image The average of the pixel values. The high dynamic range image processing system 100 of the embodiment of the present application performs high dynamic range processing on the image through the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40, first performs brightness alignment processing on the image, and then performs brightness alignment on the image after the brightness alignment. It is fused with other images to obtain a highly dynamic image, so that the target image formed by the high dynamic range image processing system 100 has a larger dynamic range, and thus has a better imaging effect.

The fusion module 204 can perform fusion algorithm processing on the color intermediate image and the panchromatic intermediate image. The specific process of the fusion algorithm processing can be as follows. The color intermediate image has the color information of the three color channels of R (i.e. red), G (i.e. green) and B (i.e. blue), and the full-color intermediate image has full-color information as As an example, the panchromatic information can be brightness information, and the specific process of fusion algorithm processing can include: (1) Calculate the auxiliary value Y corresponding to each pixel according to the color intermediate image, where Y=(R*w1+B*w2 +G*w3)/(w1+w2+w3), R is the value of the R channel corresponding to the pixel, G is the value of the G channel corresponding to the pixel, B is the value of the B channel corresponding to the pixel, w1, w2 and w3 are Weight value; (2) Calculate the ratio of each channel value in the color intermediate image to the auxiliary value Y to obtain the reference channel values K1, K2, and K3 corresponding to each pixel, where K1=R/Y, K2=G/Y, K3 =B/Y; (3) Perform color noise reduction processing on the reference channel values K1, K2, and K3; (4) Combine the panchromatic information Y'on the corresponding pixel with the reference channel values K1-K3 after color noise reduction The fusion is performed to generate the fused RGB three-channel values R', G'and B'to obtain the target image; wherein, R'=K1*Y'; G'=K2*Y'; B'=K3*Y'. The fusion module 204 of the embodiment of the present application performs fusion algorithm processing on the color image and the panchromatic image, so that the source of the final target image has both color information and brightness information. Since the human eye is more sensitive to brightness than chroma, it is In terms of eye vision characteristics, the high dynamic range image processing system 100 of the embodiment of the present application has a better imaging effect, and the final target image obtained is closer to human vision.

The high dynamic fusion unit 50 is integrated in the image sensor 10; or the high dynamic fusion unit 50 is integrated in the image processor 20. Specifically, please refer to FIG. 18. In some embodiments, the color high dynamic fusion unit 30 and the panchromatic high dynamic fusion unit 40 may be integrated in the image sensor 10; please refer to FIG. 1, FIG. 19, and FIG. 20, in addition In the embodiment, the color high dynamic fusion unit 30 and the panchromatic high dynamic fusion unit 40 may be integrated in the image processor 20. The color high dynamic fusion unit 30 and the panchromatic high dynamic fusion unit 40 are integrated in the image sensor 10 or the image processor 20, so that the high dynamic range image processing system 100 of the embodiment of the present application does not need to improve the hardware performance of the image sensor 10. Realize high dynamic range processing. At the same time, the color high dynamic range fusion unit 30 and the full-color high dynamic range fusion unit 40 independently encapsulate the high dynamic range processing function, which is beneficial to reduce the design difficulty in the product design process and improve the convenience of design changes. .

In some embodiments, image preprocessing may include pixel addition processing and demosaicing processing. In some embodiments, referring to FIG. 1, the color preprocessing module 2023 can perform pixel addition processing on the color original image data to obtain the color original image, and the panchromatic preprocessing module 2024 can demosaicing the panchromatic original image data. Process to get the full-color original image. In another embodiment, referring to FIG. 1, the color preprocessing module 2023 can perform pixel addition processing on the color original image data to obtain the color original image; the full color preprocessing module 2024 can demosaicing the full color original image data Process to get the full-color original image. Wherein, the demosaic processing is the same as the specific implementation process of the demosaic processing performed by the full-color pre-processing module 2024 on the full-color original image data, and will not be further described here. The high dynamic range image processing system 100 of the embodiment of the present application performs pixel addition processing on the color information in some pixel grids that are missing and the pixel grids with color information only have color original image data with single color channel information. In a less computationally intensive way, the color information of the complete channel is obtained, and then the original color image is obtained, so that other image processing can be continued on the image to improve the imaging quality.

In other embodiments, the image preprocessing may include pixel averaging processing and demosaicing processing. In some embodiments, referring to FIG. 1, the color preprocessing module 2023 can perform pixel averaging processing on the color original image data to obtain the color original image, and the panchromatic preprocessing module 2024 can demosaicing the panchromatic original image data. Process to get the full-color original image. In another embodiment, referring to FIG. 1, the color preprocessing module 2023 can perform pixel averaging processing on the color original image data to obtain the color original image; the full color preprocessing module 2024 can demosaicing the full color original image data Process to get the full-color original image. Among them, the demosaic processing is the same as the specific implementation process of the demosaic processing performed by the full-color preprocessing module 2024 on the full-color original image data, and will not be further described here. The high dynamic range image processing system 100 of the embodiment of the present application performs pixel averaging processing on the color original image data in which the color information in some pixel grids are missing and the pixel grids with color information only have single-color channel information. In a less computationally intensive way, the color information of the complete channel is obtained, and then the original color image is obtained, so that other image processing can be continued on the image to improve the imaging quality.

The following takes the pixel addition processing of the color original image data as an example for description. The specific steps of the pixel addition processing are as follows: (1) Decompose the color original image data into the first color original image data (from the above-mentioned first The original image data generated by the first color photosensitive pixel A), the second color original image data (the original image data generated by the second color photosensitive pixel B described above) and the third color original image data (from the above The third color photosensitive pixel C generates the original image data). (2) The pixel values generated by the multiple first-color photosensitive pixels A of the sub-units in the first-color original image data are added together, and after the sum is obtained, the pixel grids in each sub-unit range are merged into a pixel grid , And fill the average value into the pixel grid to obtain the first color intermediate image data. (3) Interpolate the first color intermediate image data by using a bilinear interpolation method to obtain a color original image with a resolution of one quarter of the color original image data. The specific operation method of bilinear interpolation will be explained in detail below. (4) After the first color original image data, the second color original image data, and the third color original image data are all subjected to the above steps (2) and (3), the obtained first color original image data with one color channel The image, the original image of the second color, and the original image of the third color are synthesized into a color original image with three color channels. The color preprocessing module 2023 can perform the pixel addition processing of the above steps on all color original image data corresponding to at least two exposures, thereby completing the pixel ball addition processing of all color original image data to obtain at least two color original images . Specifically, referring to FIG. 31, the following takes the color preprocessing module 2023 to perform pixel addition processing on the first red original image data in the first color original image data as an example for description. As shown in Figure 31, the color preprocessing module 2023 first decomposes the color original image (which can be the first color original image, the second color original image, or the third color original image, etc.) data into red original image data and green original image data. And blue raw image data. As shown in FIG. 16, the color preprocessing module 2023 then performs an addition operation on the pixel values (such as L1 and L2) generated by the multiple red photosensitive pixels R in the sub-unit of the red original image data to obtain the sum value L1 After'=L1+L2, the pixel grid in the sub-unit range is merged into a pixel grid, and the sum value is filled into the pixel grid to obtain the red intermediate image data. Then, the red preprocessing module uses a bilinear interpolation method to interpolate the red intermediate image data to obtain a red original image with a resolution of a quarter of the red original image data. Similarly, the color preprocessing module 2023 can obtain a red original image, a green original image, and a blue original image, and combine the obtained red original image, green original image, and blue original image with one color channel into three Color original image with three color channels. The color preprocessing module 2023 can perform the pixel addition of the above steps on the first color original image data and the second original image data (or the first color original image data, the second color original image data, and the third color original image data). The processing, thereby completing the pixel addition processing of the color original image data, obtains the first color original image and the second color original image (or the first color original image, the second color original image, and the third color original image).

The following takes the pixel averaging processing on the color original image data as an example. The specific steps of the pixel averaging processing are as follows: (1) Decompose the color original image data into the first color original image data (from the above-mentioned first color image data). The original image data generated by the first color photosensitive pixel A), the second color original image data (the original image data generated by the second color photosensitive pixel B described above) and the third color original image data (from the above The third color photosensitive pixel C generates the original image data). (2) Perform an averaging operation on the pixel values generated by the multiple first-color photosensitive pixels A of the sub-unit in the original image data of the first color, and after the average value is obtained, the pixel grids in the sub-unit range are merged into a pixel grid. The average value is filled into the pixel grid to obtain the first color intermediate image data. (3) Interpolate the first color intermediate image data by using a bilinear interpolation method to obtain a color original image with a resolution of one quarter of the color original image data. The specific operation method of bilinear interpolation will be explained in detail below. (4) After the first color original image data, the second color original image data, and the third color original image data are all subjected to the above steps (2) and (3), the obtained first color original image data with one color channel The image, the original image of the second color, and the original image of the third color are synthesized into a color original image with three color channels. The color preprocessing module 2023 can perform the pixel averaging processing of the above steps on all color original image data corresponding to at least two exposures, so as to complete the pixel averaging processing of all color original image data to obtain at least two color original images. Specifically, referring to FIG. 32, the following takes the color preprocessing module 2023 to perform pixel averaging processing on the first red original image data in the first color original image data as an example for description. As shown in Figure 32, the color preprocessing module 2023 first decomposes the color original image (which can be the first color original image, the second color original image, or the third color original image, etc.) data into red original image data and green original image data. And blue raw image data. As shown in FIG. 16, the color preprocessing module 2023 then performs an averaging operation on the pixel values (such as L1 and L2) generated by the multiple red photosensitive pixels R in the subunit of the red original image data to obtain the average value L1' =(L1+L2)/2 After fusing the pixel grid of the sub-unit range into a pixel grid, and filling the average value into the pixel grid, the red intermediate image data is obtained. Then, the red preprocessing module uses a bilinear interpolation method to interpolate the red intermediate image data to obtain a red original image with a resolution of a quarter of the red original image data. Similarly, the color preprocessing module 2023 can obtain a red original image, a green original image, and a blue original image, and combine the obtained red original image, green original image, and blue original image with one color channel into three Color original image with three color channels. The color preprocessing module 2023 can perform the pixel averaging of the above steps on the first color original image data and the second original image data (or the first color original image data, the second color original image data, and the third color original image data). Processing, thereby completing the pixel averaging processing of the color original image data to obtain the first color original image and the second color original image (or the first color original image, the second color original image, and the third color original image).

The image processor 20 may further include a receiving unit 201 and a memory unit 203. The receiving unit 201 is used to receive color original image data and full-color original image data; the memory unit 203 is used to temporarily store color original image data, full-color original image data, color original images, full-color original images, color intermediate images, and full-color original images. One or more of the intermediate image and the target image. The image processor 20 is provided with a receiving unit 201 and a memory unit 203 to separate the receiving, processing and storage of images, which is conducive to more independent packaging of the modules of the high dynamic range image processing system 100, so that the high dynamic range image processing system 100 has Higher execution efficiency and better anti-interference effect, in addition, are also beneficial to reduce the design difficulty of the redesign process of the high dynamic range image processing system 100, thereby reducing costs.

Please refer to FIG. 33. The present application also provides an electronic device 1000. The electronic device 1000 of the embodiment of the present application includes a lens 300, a housing 200, and the high dynamic range image processing system 100 of any one of the above embodiments. The lens 300 and the high dynamic range image processing system 100 are combined with the housing 200. The lens 300 cooperates with the image sensor 10 of the high dynamic range image processing system 100 for imaging.

The electronic device 1000 may be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, a smart glasses, a smart helmet), a drone, a head-mounted display device, etc., which are not limited here.

The electronic device 1000 of the embodiment of the present application controls the pixel array 11 to perform at least two exposures at the first exposure time and the second exposure time respectively, and generates multiple images according to different exposure times and different photosensitive pixels, so as to follow up. Image preprocessing, high dynamic range processing, image processing and fusion algorithm processing are performed on multiple images to obtain a target image with high dynamic range. The electronic device 1000 of the embodiment of the present application can realize the high dynamic range function without increasing the hardware parameters of the photosensitive pixels of the image sensor 10, so that the bright and dark areas of the target image can have better performance, which is beneficial to While improving imaging performance, it helps reduce costs.

Please refer to FIG. 34. This application provides a high dynamic range image processing method. The high dynamic range image processing method of the embodiment of the present application is used in the high dynamic range image processing system 100. The high dynamic range image processing system 100 may include the image sensor 10. The image sensor 10 includes a pixel array 11. The pixel array 11 includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels. The pixel array 11 includes the smallest repeating unit. Each minimum repeating unit contains multiple subunits. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. High dynamic range image processing methods include:

01: Control the exposure of the pixel array 11. Wherein, the pixel array 11 is exposed for the first exposure time to obtain the first original image. The first original image includes first color original image data generated by single-color photosensitive pixels exposed at the first exposure time and first full-color original image data generated by panchromatic photosensitive pixels exposed at the first exposure time. The pixel array 11 is exposed for a second exposure time to obtain a second original image. The second original image includes second color original image data generated by the single-color photosensitive pixels exposed at the second exposure time and second full-color original image data generated by the panchromatic photosensitive pixels exposed at the second exposure time. Wherein, the first exposure time is not equal to the second exposure time. with

02: Perform image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain the target image.

The high dynamic range image processing method of the embodiment of the present application controls the pixel array 11 to perform at least two exposures at the first exposure time and the second exposure time respectively, and generates multiple images according to different exposure times and different photosensitive pixels, so that Subsequently, image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing are performed on the multiple images to obtain a target image with high dynamic range. The high dynamic range image processing method of the embodiment of the present application can realize the high dynamic range function without increasing the hardware parameters of the photosensitive pixels of the image sensor 10, so that the bright and dark areas of the target image can have better performance , Which is beneficial to improve imaging performance and at the same time help to reduce costs.

In some embodiments, the pixel array 11 may also be exposed for a third exposure time to obtain a third original image. The third original image includes third color original image data generated by the single-color photosensitive pixels exposed at the third exposure time and third full-color original image data generated by the panchromatic photosensitive pixels exposed at the third exposure time. Wherein, the third exposure time is not equal to the first exposure time, and the third exposure time is not equal to the second exposure time. Performing image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain the target image may include:

Perform image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image, the second original image, and the third original image to obtain the target image.

In some embodiments, the image preprocessing includes pixel completion processing and demosaicing processing, and the image processing includes first image processing and second image processing; image preprocessing and high dynamics are performed on the first original image and the second original image. Range processing, image processing and fusion algorithm processing to obtain the target image can also include:

Perform pixel complement processing on the color original image data to obtain the color original image;

De-mosaic processing the panchromatic original image data to obtain the panchromatic original image;

Performing the first image processing on the color original image to obtain a color intermediate image;

Performing second image processing on the full-color original image to obtain a full-color intermediate image;

The fusion algorithm is performed on the color intermediate image and the panchromatic intermediate image to obtain the target image.

In some embodiments, after the color intermediate image and the panchromatic intermediate image are processed by the fusion algorithm to obtain the target image, image preprocessing, high dynamic range processing, image processing, and fusion are performed on the first original image and the second original image. The algorithm processing to obtain the target image also includes:

Fusion of target images corresponding to at least two exposures to obtain a highly dynamic target image.

In some embodiments, before performing pixel complement processing on the color original image data to obtain the color original image, image preprocessing, high dynamic range processing, image processing, and fusion algorithms are performed on the first original image and the second original image The target image obtained by processing also includes:

Fusion of color original image data corresponding to at least two exposures to obtain highly dynamic color original image data;

Before the panchromatic original image data is demosaiced to obtain the panchromatic original image, the first original image and the second original image are subjected to image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing to obtain the target image. :

The full-color original image data corresponding to at least two exposures are fused to obtain high-dynamic full-color original image data.

In some embodiments, the first image processing includes:

One or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, demosaicing processing, color correction processing, global tone mapping processing and color conversion processing;

The second image processing includes:

One or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, and global tone mapping processing.

In some embodiments, the first image processing includes a first image sub-processing and a second image sub-processing, and the color processing module 2021 is configured to perform the first image sub-processing on the color original image first, and then perform the second image sub-processing, The first image sub-processing includes:

One or more of black level correction processing, lens shading correction processing and dead pixel compensation processing;

The second image sub-processing includes:

One or more of demosaicing processing, color correction processing, global tone mapping processing, and color conversion processing.

In some embodiments, fusing the target images corresponding to at least two exposures to obtain a highly dynamic target image includes:

The target image corresponding to at least two exposures is subjected to brightness alignment processing to obtain a brightness-aligned target image, and then the brightness-aligned target image and one or more target images are merged to obtain a highly dynamic target image.

In some embodiments, fusing color raw image data corresponding to at least two exposures to obtain high dynamic color raw image data includes:

Perform brightness alignment processing on the color original image data corresponding to at least two exposures to obtain brightness-aligned color original image data, and then merge the brightness-aligned color original image data and one or more color original image data to obtain a highly dynamic Color original image data;

Fusion of panchromatic raw image data corresponding to at least two exposures to obtain high dynamic panchromatic raw image data includes:

Perform brightness alignment processing on the panchromatic original image data corresponding to at least two exposures to obtain brightness aligned panchromatic original image data, and then merge the brightness aligned panchromatic original image data and one or more panchromatic original image data to Obtain high dynamic panchromatic original image data.

In some embodiments, the high dynamic range image processing method further includes:

Receiving color original image data and panchromatic original image data; and

Temporarily store one or more of color original image data, full color original image data, color original image, full color original image, color intermediate image, full color intermediate image, and target image.

In some embodiments, the image preprocessing includes pixel addition processing and demosaicing processing, and image preprocessing, high dynamic range processing, image processing, and fusion algorithms are performed on the first original image, the second original image, and the third original image. The processed target image includes:

Perform pixel addition processing on the color original image data to obtain a color original image; and

Demosaic processing the panchromatic original image data to obtain a panchromatic original image; or

Image preprocessing includes pixel averaging processing and demosaicing processing. Performing image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image, the second original image, and the third original image to obtain the target image includes:

Perform pixel averaging processing on the color original image data to obtain the color original image; and

Demosaic processing is performed on the panchromatic original image data to obtain a panchromatic original image.

The specific implementation process of the high dynamic range image processing method of any one of the above embodiments is the same as the specific implementation process of the aforementioned high dynamic range image processing system 100 to obtain a target image, and will not be further described here.

Please refer to 35. This application also provides a non-volatile computer-readable storage medium 400 containing a computer program. When the computer program is executed by the processor 60, the processor 60 is caused to execute the high dynamic range image processing method described in any one of the foregoing embodiments.

In summary, the high dynamic range image processing system 100 and method, the electronic device 1000, and the computer-readable storage medium 400 of the embodiments of the present application control the pixel array 11 to at least perform two exposures at the first exposure time and the second exposure time, respectively. And according to different exposure times and different photosensitive pixels, multiple images are generated, so that image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing are performed on these multiple images to obtain a target image with high dynamic range. The high dynamic range image processing system 100 and method, the electronic device 1000, and the computer-readable storage medium 400 of the embodiments of the present application can realize the high dynamic range function without increasing the hardware parameters of the photosensitive pixels of the image sensor 10, so that the target The bright and dark parts of the image can have better performance, which is conducive to improving imaging performance and helping to reduce costs.

In addition, in the related art, the image processor can only process the image formed by the traditional pixel array composed of color photosensitive pixels, and is not applicable to the image produced by the pixel array having both color photosensitive pixels and full-color photosensitive pixels. The high dynamic range image processing system 100 and method, the electronic device 1000, and the computer-readable storage medium 400 of the embodiments of the present application are applicable to images generated by a pixel array 11 having color photosensitive pixels and full-color photosensitive pixels. Under the same light environment and other auxiliary hardware, full-color photosensitive pixels can receive more light than color photosensitive pixels, which can improve the brightness of the final image, and the human eye is more sensitive to brightness than chromaticity, which makes the The high dynamic range image processing system 100 and method, the electronic device 1000, and the computer-readable storage medium 400 of the application embodiment have better imaging effects.

In the related art, methods such as increasing the shutter speed or selecting photosensitive pixels whose photosensitive response curve is in a logarithmic form put forward higher requirements on the hardware parameters of the image sensor of the high-dynamic camera. The high dynamic range image processing system 100 and method, the electronic device 1000, and the computer-readable storage medium 400 of the embodiment of the present application do not need to increase the hardware parameter requirements of the image sensor 10, by providing a high dynamic fusion unit in the image sensor 10 50 and the fusion module 204, in conjunction with the corresponding exposure mode, can realize the high dynamic range processing function, thereby obtaining an image with better imaging effect.

In the description of the embodiments of the present application, it should be noted that, unless otherwise clearly defined and limited, the term “installation” should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral Connection; it can be mechanical connection, it can be electrical connection or it can communicate with each other; it can be directly connected, it can also be indirectly connected through an intermediate medium, it can be the internal communication of two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above-mentioned terms in the embodiments of the present application can be understood according to specific circumstances.

Any process or method description in the flowchart or described in other ways herein can be understood as a module, segment or part of code that includes one or more executable instructions for implementing specific logical functions or steps of the process , And the scope of the preferred embodiments of the present application includes additional implementations, which may not be in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. This should It is understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium, For use by instruction execution systems, devices, or equipment (such as computer-based systems, systems including processing modules, or other systems that can fetch instructions from instruction execution systems, devices, or equipment and execute instructions), or combine these instruction execution systems, devices Or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, device, or device or in combination with these instruction execution systems, devices, or devices. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (control method), portable computer disk cases (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable and editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, because it can be used, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or other suitable media if necessary. The program is processed in a manner to obtain the program electronically, and then stored in the computer memory.

It should be understood that each part of the embodiments of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate array (PGA), field programmable gate array (FPGA), etc.

A person of ordinary skill in the art can understand that all or part of the steps carried in the method of the foregoing embodiments can be implemented by a program instructing relevant hardware to complete. The program can be stored in a computer-readable storage medium. When executed, it includes one of the steps of the method embodiment or a combination thereof.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

The aforementioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

In the description of this specification, the description with reference to the term "certain embodiments" and the like means that a specific feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present application. In this specification, the schematic representation of the above-mentioned terms does not necessarily refer to the same embodiment. Moreover, the described specific features, structures or characteristics can be combined in any one or more embodiments in an appropriate manner.

Although the embodiments of the present application have been shown and described above, it can be understood that the foregoing embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can perform the above-mentioned embodiments within the scope of the present application. For changes, modifications, substitutions and variations, the scope of this application is defined by the claims and their equivalents.

Claims (25)

A high dynamic range image processing system, which is characterized by comprising an image sensor, a high dynamic fusion unit and an image processor; The image sensor includes a pixel array, the pixel array includes a plurality of panchromatic photosensitive pixels and a plurality of color photosensitive pixels, the color photosensitive pixels have a narrower spectral response than the panchromatic photosensitive pixels, and the pixel array includes A minimum repeating unit, each of the minimum repeating units includes a plurality of sub-units, and each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels; The pixel array is exposed at a first exposure time to obtain a first original image, and the first original image includes first color original image data generated by the single-color photosensitive pixel exposed at the first exposure time and The first full-color original image data generated by the full-color photosensitive pixels exposed for the first exposure time; the pixel array is exposed for the second exposure time to obtain a second original image, and the second original image includes the second original image. The second color original image data generated by the single-color photosensitive pixel exposed at the exposure time and the second full-color original image data generated by the panchromatic photosensitive pixel exposed at the second exposure time; wherein, the first The exposure time is not equal to the second exposure time; The image processor and the high dynamic fusion unit are configured to perform image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain a target image. The high dynamic range image processing system according to claim 1, wherein the pixel array is exposed at a third exposure time to obtain a third original image, and the third original image includes a third original image exposed at the third exposure time. The third color original image data generated by the single-color photosensitive pixel and the third panchromatic original image data generated by the panchromatic photosensitive pixel exposed at the third exposure time; wherein the third exposure time is not equal to The first exposure time, the third exposure time is not equal to the second exposure time; The image processor and the high dynamic fusion unit are used to perform image preprocessing, high dynamic range processing, image processing, and fusion algorithms on the first original image, the second original image, and the third original image Process to get the target image. The high dynamic range image processing system according to claim 1, wherein the image processor includes a color preprocessing module, a full color preprocessing module, a color processing module, a full color processing module, and a fusion module, and the image The preprocessing includes pixel completion processing and demosaicing processing, and the image processing includes first image processing and second image processing; wherein: The color preprocessing module is used to perform pixel complement processing on color original image data to obtain a color original image; The full-color preprocessing module is used for demosaicing the full-color original image data to obtain the full-color original image; The color processing module is configured to perform first image processing on the color original image to obtain a color intermediate image; The panchromatic processing module is configured to perform second image processing on the panchromatic original image to obtain a panchromatic intermediate image; The fusion module is used to perform fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image. The high dynamic range image processing system according to claim 3, characterized in that, after the fusion module performs fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image: The high dynamic fusion unit is used for fusing the target images corresponding to at least two exposures to obtain the highly dynamic target image. The high dynamic range image processing system according to claim 3, wherein the high dynamic fusion unit includes a color high dynamic fusion unit and a panchromatic high dynamic fusion unit, and the color original image data is processed by the color preprocessing module Before performing pixel completion processing to get the color original image: The color high-dynamic fusion unit is configured to fuse the color original image data corresponding to at least two exposures to obtain the high-dynamic color original image data; Before the panchromatic preprocessing module performs demosaic processing on the panchromatic original image data to obtain the panchromatic original image: The panchromatic high dynamic fusion unit is used for fusing the panchromatic original image data corresponding to at least two exposures to obtain the high dynamic panchromatic original image data. The high dynamic range image processing system according to claim 3, wherein the first image processing comprises: One or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, demosaicing processing, color correction processing, global tone mapping processing and color conversion processing; The second image processing includes: One or more of the black level correction processing, the lens shading correction processing, the dead pixel compensation processing, and the global tone mapping processing. The high dynamic range image processing system according to claim 6, wherein the first image processing includes a first image sub-processing and a second image sub-processing, and the color processing module is used for processing the color original image Perform the first image sub-processing first, and then perform the second image sub-processing, and the first image sub-processing includes: One or more of black level correction processing, lens shading correction processing and dead pixel compensation processing; The second image sub-processing includes: One or more of demosaicing processing, color correction processing, global tone mapping processing, and color conversion processing. The high dynamic range image processing system according to claim 4, wherein the high dynamic range fusion unit is used for: Perform brightness alignment processing on the target image corresponding to at least two exposures to obtain the brightness-aligned target image, and then merge the brightness-aligned target image and one or more of the target images to obtain the high The dynamic target image. The high dynamic range image processing system according to claim 5, wherein the color high dynamic fusion unit is used for: Perform brightness alignment processing on the color original image data corresponding to at least two exposures to obtain the brightness-aligned color original image data, and then merge the brightness-aligned color original image data and one or more sheets of the color Original image data to obtain the high-dynamic color original image data; The panchromatic high dynamic fusion unit is used for: Perform brightness alignment processing on the panchromatic original image data corresponding to at least two exposures to obtain brightness aligned panchromatic original image data, and then merge the brightness aligned panchromatic original image data and one or more sheets The full-color original image data is used to obtain the high-dynamic full-color original image data. The high dynamic range image processing system according to claim 4 or 5, wherein the image processor further comprises: A receiving unit configured to receive the color original image data and the full-color original image data; and A memory unit, the memory unit is used to temporarily store the color original image data, the full color original image data, the color original image, the full color original image, the color intermediate image, the full color intermediate One or more of the image and the target image. The high dynamic range image processing system according to claim 1, wherein the image processor includes a color preprocessing module and a full color preprocessing module, The image preprocessing includes pixel addition processing and demosaicing processing, the color preprocessing module is used to perform pixel addition processing on color original image data to obtain a color original image, and the panchromatic preprocessing module is used to Demosaicing the original image data to obtain a full-color original image; or The image preprocessing includes pixel averaging processing and demosaicing processing. The color preprocessing module is used to perform pixel averaging processing on color original image data to obtain a color original image. The panchromatic preprocessing module is used to The color original image data is demosaiced to obtain a full-color original image. The high dynamic range image processing system according to claim 1, wherein the high dynamic fusion unit is integrated in the image sensor; or the high dynamic fusion unit is integrated in the image processor. A high dynamic range image processing method for a high dynamic range image processing system, wherein the high dynamic range image processing system includes an image sensor, the image sensor includes a pixel array, and the pixel array includes a plurality of pixels. A color photosensitive pixel and a plurality of color photosensitive pixels, the color photosensitive pixel has a narrower spectral response than the full-color photosensitive pixel, the pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of subunits, Each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels; the high dynamic range image processing method includes: Control the exposure of the pixel array, wherein the pixel array is exposed for a first exposure time to obtain a first original image, and the first original image includes a first image generated by the single-color photosensitive pixel exposed for the first exposure time. One color original image data and first panchromatic original image data generated by the panchromatic photosensitive pixels exposed at the first exposure time; the pixel array is exposed at the second exposure time to obtain a second original image, the first The two original images include second color original image data generated by the single-color photosensitive pixels exposed at the second exposure time and second full-color original image data generated by the panchromatic photosensitive pixels exposed at the second exposure time Image data; wherein the first exposure time is not equal to the second exposure time; and Performing image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain a target image. The high dynamic range image processing method according to claim 13, wherein the pixel array is exposed at a third exposure time to obtain a third original image, and the third original image includes a third original image exposed at the third exposure time. The third color original image data generated by the single-color photosensitive pixel and the third panchromatic original image data generated by the panchromatic photosensitive pixel exposed at the third exposure time; wherein the third exposure time is not equal to The first exposure time and the third exposure time are not equal to the second exposure time; the image preprocessing, high dynamic range processing, and image processing are performed on the first original image and the second original image The target image obtained by processing and fusion algorithm includes: Image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing are performed on the first original image, the second original image, and the third original image to obtain a target image. The high dynamic range image processing method according to claim 13, wherein the image preprocessing includes pixel complement processing and demosaicing processing, and the image processing includes first image processing and second image processing; Performing image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain a target image includes: Perform pixel complement processing on the color original image data to obtain the color original image; De-mosaic processing the panchromatic original image data to obtain the panchromatic original image; Performing first image processing on the color original image to obtain a color intermediate image; Performing second image processing on the full-color original image to obtain a full-color intermediate image; Performing fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image. The high dynamic range image processing method according to claim 15, wherein after the target image is obtained by performing fusion algorithm processing on the color intermediate image and the panchromatic intermediate image, the Performing image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the first original image and the second original image to obtain a target image further includes: Fusion of the target images corresponding to at least two exposures to obtain the target image with high dynamics. The high dynamic range image processing method according to claim 15, characterized in that, before performing pixel complementation processing on color original image data to obtain a color original image, said pairing said first original image and said second original image The original image is processed by image preprocessing, high dynamic range processing, image processing and fusion algorithm processing to obtain the target image also includes: Fusing the color original image data corresponding to at least two exposures to obtain the high dynamic color original image data; Before performing demosaic processing on the panchromatic original image data to obtain a panchromatic original image, the first original image and the second original image are subjected to image preprocessing, high dynamic range processing, image processing, and fusion algorithm The processed target image also includes: The full-color original image data corresponding to at least two exposures are fused to obtain the high-dynamic full-color original image data. The high dynamic range image processing method according to claim 15, wherein the first image processing comprises: One or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, demosaicing processing, color correction processing, global tone mapping processing and color conversion processing; The second image processing includes: One or more of the black level correction processing, the lens shading correction processing, the dead pixel compensation processing, and the global tone mapping processing. The high dynamic range image processing method according to claim 18, wherein the first image processing includes a first image sub-processing and a second image sub-processing, and the color processing module is used for processing the color original image Perform the first image sub-processing first, and then perform the second image sub-processing, and the first image sub-processing includes: One or more of black level correction processing, lens shading correction processing and dead pixel compensation processing; The second image sub-processing includes: One or more of demosaicing processing, color correction processing, global tone mapping processing, and color conversion processing. The high dynamic range image processing method according to claim 16, wherein said fusing the target images corresponding to at least two exposures to obtain the highly dynamic target image comprises: Perform brightness alignment processing on the target image corresponding to at least two exposures to obtain the brightness-aligned target image, and then merge the brightness-aligned target image and one or more of the target images to obtain the high The dynamic target image. The high dynamic range image processing method according to any one of claim 17, wherein the fusing the color original image data corresponding to at least two exposures to obtain the high dynamic color original image data comprises: Perform brightness alignment processing on the color original image data corresponding to at least two exposures to obtain the brightness-aligned color original image data, and then merge the brightness-aligned color original image data and one or more sheets of the color Original image data to obtain the high-dynamic color original image data; The fusing the panchromatic original image data corresponding to at least two exposures to obtain the highly dynamic panchromatic original image data includes: Perform brightness alignment processing on the panchromatic original image data corresponding to at least two exposures to obtain brightness aligned panchromatic original image data, and then merge the brightness aligned panchromatic original image data and one or more sheets The full-color original image data is used to obtain the high-dynamic full-color original image data. The high dynamic range image processing method according to claim 16 or 17, wherein the high dynamic range image processing method further comprises: Receiving the color original image data and the full-color original image data; and Temporarily store one of the color original image data, the full color original image data, the color original image, the full color original image, the color intermediate image, the full color intermediate image, and the target image Or more. The high dynamic range image processing method according to claim 13, wherein the image preprocessing includes pixel addition processing and demosaicing processing, and the first original image, the second original image, and the Performing image preprocessing, high dynamic range processing, image processing, and fusion algorithm processing on the third original image to obtain a target image includes: Perform pixel addition processing on the color original image data to obtain a color original image; and Demosaic processing the panchromatic original image data to obtain a panchromatic original image; or The image preprocessing includes pixel averaging processing and demosaicing processing, and performing image preprocessing, high dynamic range processing, and image processing on the first original image, the second original image, and the third original image The target image obtained by processing and fusion algorithm includes: Perform pixel averaging processing on the color original image data to obtain the color original image; and Demosaic processing is performed on the panchromatic original image data to obtain a panchromatic original image. An electronic device, characterized in that it comprises: Lens Shell; and The high dynamic range image processing system according to any one of claims 1 to 12, wherein the lens, the high dynamic range image processing system are combined with the housing, and the lens is connected to the high dynamic range image processing system. The image sensor cooperates with imaging. A non-volatile computer-readable storage medium containing a computer program, wherein when the computer program is executed by a processor, the processor executes the high dynamic range of any one of claims 13 to 23 Image processing method.

Images