WO2020063507A1 - Image processing method and system, and computer readable storage medium - Google Patents

Abstract

Provided in the present invention are an image processing method and system, and a computer readable storage medium. The image processing method comprises the following steps: S100: an image sensor collects image data; S200: the image sensor sends original data of the image data to an image processing chip, and the original data is processed by means of a digital signal processing module to form optimised image data; S300: the digital signal processing module sends the optimised image data to an image sensing and processing module to perform processing on the optimised image data to form enhanced image data; S400: the image processing chip returns the optimised image data and/or the enhanced image data to a central processing chip, and the central processing chip records the optimised image data and/or the enhanced image data in a storage device; S500: and the image processing chip sends the optimised image data and/or the enhanced image data to a display module to display the optimised image and/or enhanced image, such that smart processing and optimisation of images are independent of the CPU processing of a smart terminal, thereby reducing the CPU load.

Description

Image processing method, system and computer-readable storage medium Technical field

The invention relates to the field of intelligent control, in particular to an image processing method, system and computer-readable storage medium.

technical background

For example, mobile terminal devices such as mobile phones and tablet computers have increasingly entered people's lives. Therefore, such mobile terminals are also integrated with multiple functions such as shooting, camera, positioning, and lighting, so that people can complete multiple requirements after holding a mobile terminal. When using a mobile terminal for shooting and video, the resolution and pixels of the camera module are getting higher and higher, making the imaging form that the mobile terminal can achieve in this function more and more close to professional camera equipment. Specifically, there are two main imaging forms of mobile terminal devices:

1.CPU has built-in image sensor processing module:

As shown in Figure 1, in the central processing unit CPU of the mobile terminal, an image sensor processing module ISP is built in. The camera group will send data to the ISP through the mipi interface. The ISP will process the data to different ones according to different applications. Module. For example, the data path 1 is sent to the data processing unit DPU. After the DPU is processed, it is sent to the display LCD for display of the camera module. For example, the data path 2 is sent to the video processing unit VPU. The VPU encodes and stores the data to In the storage device, it is used for the video recording function of the camera module; for example, it is sent to the image compressor jpeg encoder via data path 3. After the data is compressed into a jpeg file, the data is stored in the storage device for the camera module to take pictures Features;.

2.CPU has image sensor processing module

As shown in Fig. 2, an image sensor processing module is connected outside the CPU and adopts the internal ISP bypass method. The data flow is the same as the first method except the built-in ISP bypass.

In the existing solutions, the internal image sensing and processing functions of the CPU are used. Due to the limitation of a certain internal function of the CPU, the processing of image data is limited, and more professional and detailed adjustments cannot be made to the captured images. The CPU's data processing load makes the mobile terminal run slower.

Therefore, a new image processing method and system are needed, which can separate the image processing flow from the CPU, make full use of external hardware, and improve the imaging quality and display effect of the mobile terminal.

Summary of invention

In order to overcome the above technical defects, an object of the present invention is to provide an image processing method, a system, and a computer-readable storage medium, so that intelligent processing and optimization of an image are independent of CPU processing of an intelligent terminal, thereby reducing CPU load.

The invention discloses an image processing method, which includes the following steps:

S100: The image sensor collects image data of the shooting object;

S200: The image sensor sends the raw data of the image data to an image processing chip, and the digital signal processing module of the image processing chip processes the raw data to form optimized image data;

S300: The digital signal processing module sends the optimized image data to an image sensing processing module of the image processing chip to process the optimized image data to form enhanced image data;

S400: The image processing chip returns the optimized image data and / or enhanced image data to a central processing chip in a smart terminal, and the central processing chip records the optimized image data and / or enhanced image data to A storage device;

S500: The image processing chip sends the optimized image data and / or enhanced image data to a display module, and the display module displays an optimized image corresponding to the optimized image data and / or enhanced image data and / Or enhance the image.

Preferably, the step S200 includes:

S210: The image sensor sends the image data to the image processing chip in a raw data format through a camera interface;

S220: The image sensing module of the image processing chip processes the raw data to form optimized image data.

Preferably, the step S220 includes:

S221: The image sensing module will extract picture frames in the original data;

S222: Synthesize the picture frame or insert a supplementary frame into the picture frame to form multi-frame synthesized image data.

Preferably, the step S220 includes:

S221 ': the image sensing module determines a picture environment of the original data;

S222 ': Render the original data based on the picture environment to form optimized image data.

Preferably, the step S300 includes:

S310: the digital signal processing module forwards the original data to a central processing chip;

S320: The intelligent module of the central processing chip renders and processes the raw data to form optimized image data;

S330: The intelligent module sends the optimized image data to a processing module.

Preferably, the image processing method further includes:

S600: The image sensor sends frame rate data of the image data to an image processing chip;

S700: The image processing chip performs frame interpolation compensation on the frame rate data to form frame interpolation data, and the frame interpolation data matches a refresh rate of a display module of the smart terminal;

S800: compress the frame interpolation data and store the data into a storage device;

S900: The image processing chip sends the frame interpolation data to a display module, and the display module displays the frame interpolation image corresponding to the frame interpolation data.

Preferably, a frame rate of the frame rate data is 60 fps or more.

The invention also discloses an image processing system, which includes a display module, an image sensor, an image processing chip and a central processing chip provided in a terminal;

The image processing chip is connected to the display module, the image sensor and the central processing chip, respectively;

The image sensor collects image data of a shooting object and sends the image data to an image processing chip;

The digital signal processing module of the image processing chip processes the original data of the image data to form optimized image data;

The digital signal processing module sends the optimized image data to an image sensing processing module of the image processing chip to process the optimized image data to form enhanced image data;

The image processing chip returns the optimized image data and / or enhanced image data to a central processing chip in a smart terminal, and the central processing chip records the optimized image data and / or enhanced image data to a storage Equipment, and the image processing chip sends the optimized image data and / or enhanced image data to a display module, and the display module displays an optimized image corresponding to the optimized image data and / or enhanced image data and / Or enhance the image.

Preferably, the digital signal processing module interpolates and supplements the original data to form multi-frame synthetic image data; or

The digital signal processing module renders the original data to form optimized image data.

The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method described above are implemented.

After adopting the above technical solution, compared with the prior art, it has the following beneficial effects:

1. The processing of image data will be divided into CPUs, which will not be processed by the CPU, but will be completed by independent image processing chips, which reduces the CPU load and speeds up processing;

2. Make full use of external hardware to improve imaging quality and display effects of mobile terminals.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic structural diagram of a system for realizing captured image processing in the prior art; FIG.

2 is a schematic structural diagram of a system for realizing captured image processing in the prior art;

3 is a schematic flowchart of an image processing method according to an embodiment of the present invention;

4 is a schematic flowchart of an image processing method according to the first preferred embodiment of the present invention;

5 is a schematic flowchart of an image processing method according to a second preferred embodiment of the present invention;

6 is a schematic flowchart of an image processing method according to a third preferred embodiment of the present invention;

7 is a schematic flowchart of an image processing method according to a preferred embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an image processing system according to a preferred embodiment of the present invention.

Summary of the Invention

The advantages of the present invention will be further described below with reference to the drawings and specific embodiments.

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure, as detailed in the appended claims.

The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

It should be understood that, although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at" or "when" or "in response to a determination"

In the description of the present invention, it should be understood that the terms "vertical", "horizontal", "up", "down", "front", "rear", "left", "right", "vertical", The orientations or positional relationships indicated by "horizontal", "top", "bottom", "inner", "outer" are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than The indicated or implied device or element must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be construed as a limitation on the present invention.

In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connected", and "connected" should be understood in a broad sense. For example, they may be mechanical or electrical connections, or both. The internal connection of these elements can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific situations.

In the following description, the use of suffixes such as “module”, “component”, or “unit” for representing elements is merely for the benefit of the description of the present invention, and it does not have a specific meaning in itself. Therefore, "module" and "component" can be used in combination.

Referring to FIG. 3, it is a schematic flowchart of an image processing method according to a preferred embodiment of the present invention. In this embodiment, the image processing method includes the following steps:

S100: The image sensor collects image data of the shooting object

In order to form an initial image to be processed, an image sensor such as a photographing camera module, a lens module, a prism module, etc. is used to perform image acquisition such as shooting, video recording, recording motion frames, etc. on a photographic object, thereby forming original image data .

It can be understood that, due to the different hardware structures of different image sensors, the image data collected may be unmodified image data without any modification, or it may have been subjected to beauty, cropping, optimization, rendering, etc. Image data for initial processing. Regardless of whether the image data in this step has been processed, it will be further optimized in the subsequent steps.

S200: The image sensor sends the raw data of the image data to the image processing chip, and the digital signal processing module of the image processing chip processes the raw data to form optimized image data.

The image data formed in step S100 is sent by the image sensor to an image processing chip (IPIC) electrically connected to the image sensor. The electrical connection between the image sensor and the image processing chip may be integrated on the same printed circuit board. The two independent image sensing modules and image processing modules may also be two independent devices connected by wires, or two independent devices arranged in the same local area network or wirelessly connected to each other. After receiving the image data, the image processing chip IPIC will perform fine processing such as frame insertion and intelligent optimization (AI processing) to form an optimized image data. Optimized image data Based on the different processing capabilities of the image processing chip IPIC, different levels of image optimization can be performed. In this embodiment, the image processing chip is a device that is separate from any other modules, modules, units, and devices, and runs independently. Optimizing the processing of image data in this way can reduce the load pressure of image processing functions on other devices on the one hand, and on the other hand can be used exclusively for integrated devices that do not castrate some functions on the image processing to increase the efficiency and effect of image processing.

S300: The digital signal processing module sends the optimized image data to the image sensing processing module of the image processing chip to process the optimized image data to form enhanced image data.

After the first round of optimization of the original image is completed, the digital signal processing module sends the optimized image data to the image sensing processing module of the image processing chip, and the image sensing processing module performs optimization on the optimized image data such as denoising and black point correction. Processing to form enhanced image data. As the second round of optimization, after the first round has been determined to be optimized for the type, scene, time, etc. of the subject, it will further enhance the image sense of the image data, thereby calling the full capabilities of the image processing chip to complete the maximization of the image Handle and refine.

S400: The image processing chip returns the optimized image data and / or enhanced image data to a central processing chip in an intelligent terminal, and the central processing chip records the optimized image data and / or enhanced image data to a storage device

The optimized image data and / or enhanced image data formed after the processing is returned from the image processing chip to a central processing chip (CPU or AP side). The central processing chip is set in a smart terminal, and the image processing chip is externally connected to the central processing chip, or it can be set in the same smart terminal, but it is set separately from the central processing chip, and the two are distributed design. The image processing chip and the central processing chip may be designed on the same printed circuit board (but not integrated), or the image processing chip and the central processing chip may be connected through a wire, or may be wirelessly connected through mutually supported wireless protocols. Separate the central processing chip from the image processing chip. The central processing chip can centrally process the instructions of each component in the smart terminal and issue them. For the image processing function, the central processing chip is divided into separate external image processing chips. Complete, while reducing the load of the central processing chip, it can also simplify the standard requirements of the central processing chip. The internal original image processing capability can be compressed or simplified. The optimized processing of externally acquired image data is handed over to a more specialized, The more optimized image processing chip is completed, which can speed up the image processing speed. At the same time, since the image data no longer passes through the central processing chip, the display and loading of the processed image data will be faster.

After the central processing chip receives the optimized image data and / or enhanced image data, it records the optimized image data and / or enhanced image data to a storage device, such as the memory of the smart terminal, and the external storage device (U disk, mobile hard disk) of the smart terminal. , Other devices connected by OTG, etc.). In this step, in order to store the processed optimized image data and / or enhanced image data, the central processing chip performs a record forwarding action, and does not make any changes to the enhanced image data itself, thereby reducing the operating load of the central processing chip.

It can be understood that according to the difference in the model and configuration of the image processing chip, some image processing chips may complete the enhancement of the original data when forming optimized image data. In this embodiment, S300 will no longer be performed. The step of forming enhanced image data (or may be called collectively performed); as in some embodiments, skip step S300 and directly forward the optimized image data to the storage device, and the technical solution of the present invention can also be implemented.

S500: The image processing chip sends the optimized image data and / or enhanced image data to a display module, and the display module displays the optimized image and / or enhanced image corresponding to the optimized image data and / or enhanced image data.

Steps S400 and S500 can be performed synchronously or asynchronously, that is, at any time, before, or after the central processing chip records the optimized image data and / or enhanced image data to the storage device, the image processing chip will optimize the image data and / or enhance The image data is sent to a display module, specifically a display module such as a display screen or a touch screen installed on the smart terminal. The display module receives and displays the optimized image and / or enhanced image corresponding to the optimized image data and / or enhanced image data, that is, Based on the initial image data, it performs intelligent optimization, rendering, color temperature, fill light, AR enhancement, information recognition, beauty, cropping and other image processing to form an enhanced image or scene rendering, object recognition, frame rate Optimized images for compensation, red-eye removal, and more.

The above-mentioned separate image processing chip is used to optimize and enhance the image in two steps, and directly send the processed enhanced image all the way to the display module for display and all the way to the storage device for storage to improve the transmission speed of the processed image, and User-side loading and display speed. In addition, it can improve the effect of the camera module during AI processing, and improve the recording effect of fast motion pictures.

In a preferred embodiment, step S200 includes:

S210: The image sensor sends the image data to the image processing chip in the original data format through the camera interface

The image sensor sends the original data format of the image data to the image processing chip through the camera interface CAMIF. CAMIF is the first part of the video front-end (VFE) hardware. The main task is to synchronize the line and field synchronization signals involved in the process of the sensor sending data. It also has image extraction and image mirroring capabilities. In order to accurately synchronize with the external camera sensor, the CAMIF hardware must provide two programmable interrupt lines, one to control the opening of the shutter and the other to control the flash. CAMIF hardware input devices include PCLK, HSYNC, VSYNC, a pixel enable gate, and a 12-bit data line. The input interface type is controlled by adsp. At power-on, the CAMIF module is closed and will not capture data until the adsp setting is enabled to enable data capture. When the enable data bit is set, CAMIF waits until the beginning of the next frame to start collecting data. Similarly, when adps is set to disable, CAMIF will stop working when the next frame of data is sent to ensure that everything is done. CAMIF hardware provides the function of resampling camera and sensor data. Before CAMIF output data, adsp can independently control whether the complete output image or sub-sampling.

Different from the prior art, the image sensor sends compressed data of the shooting object, that is, data in the YUV format to the CPU for processing. In this embodiment, the image sensor directly sends Raw data, which is a large amount of data and high resolution, that is, The raw data format, such as bayer, is formatted to the image processing chip. In the prior art, since the CPU processes image data, and the CPU does not have processing capability for Raw Data, even if the original data format is sent to the CPU, it cannot be processed. After using the image processing chip of the present invention, the original data format can be directly processed, and the image quality is improved at the data source.

S220: The image sensing module of the image processing chip processes the raw data to form optimized image data.

In different embodiments, the image processing chip has different processing methods for image data or forming optimized image data. Specifically, it will be described in detail through the following descriptions of different embodiments.

Example one

Referring to FIG. 4, in this embodiment, S220 includes:

S221: The image sensing module will extract the picture frames in the original data;

S222: Synthesize a picture frame or insert a supplementary frame into the picture frame to form a plurality of frames of synthesized image data.

The image sensor module will capture a number of frames corresponding to each moment at the same moment or very close moments when shooting the subject. Due to shooting restrictions, the clear pixels of each frame are different . Therefore, in this embodiment, the image sensing module will extract the picture frames in the original data, and send the picture frames to the image processing unit IPU of the image processing chip for screening. The image processing unit is based on the clarity of each picture frame. For multi-frame synthesis, that is, use the clear pixels in other picture frames to replace the blurred pixels in one picture frame, thereby perfecting the defects of the original data by combining or interpolating complementary frames to form multi-frame synthetic image data.

The multi-frame synthetic image data can be returned to the central processing chip, and the central processing chip uses the image compression module to compress the multi-frame synthetic image data and write it to the storage device.

Example two

Referring to FIG. 5, in this embodiment, S220 includes:

S221 ': the image sensing module determines the picture environment of the original data;

S222 ': Render and process the raw data for the picture environment to form optimized image data.

AI intelligent processing of raw data will help users who do not often use the shooting function or do not use the shooting function to take the same professional, correct framing images. Specifically, the image sensor will determine the picture environment of the original data, such as the subject in a certain spot, the brightness of the indoor environment, the type of the subject, the degree of violation between the subject and the surrounding environment, etc., and will be determined through experience and algorithms. . After the judgment is completed, the original data will be rendered, such as automatically adding filters to the original data according to the surrounding environment, and automatically cropping the original data to place the subject in the most suitable position, automatically blur the background, and automatically adjust the picture parameters ( Such as color temperature, brightness, contrast, etc.) to form an optimized original image.

The above AI intelligent processing results will be sent to the image sensing processing module ISP inside the IPIC to continue processing and further enhance.

Example three

Referring to FIG. 6, in another preferred embodiment, step S300 includes:

S310: The digital signal processing module forwards the original data to the central processing chip;

S320: The intelligent module of the central processing chip renders and processes the raw data to form optimized image data;

S330: The intelligent module sends the optimized image data to the processing module.

For smart terminals with poor configuration or no longer suitable for adding image processing chips, at the image sensor, the original data format is compressed to form, for example, RGB, YUV formats, and forwarded directly to the central processing chip, which is then processed by the central processing chip The AI processing is performed as described above, and image compression is performed.

In another preferred embodiment, referring to FIG. 7, in order to cooperate with the formation of high-quality, high-effect image data, a display module with a high refresh rate (the frame rate of the frame rate data of the image display is greater than or equal to 60 fps) will be used. For such display modules, the image processing method further includes:

S600: The image sensor sends the frame rate data of the image data to the image processing chip, such as to the IPIC through the CamIF interface;

S700: The image processing chip performs frame interpolation compensation on the frame rate data to form the frame interpolation data, which matches the refresh rate of the display module of the smart terminal;

S800: Compress and insert frame data and store it in a storage device. The storage device can be the same as or different from the storage device above. Before storage, it can be compressed by the video processing unit VPU of IPIC to speed up the storage speed.

S900: The image processing chip sends the frame interpolation data to the display module, and the display module displays the frame interpolation image corresponding to the frame interpolation data.

Through the above-mentioned high refresh rate image data, even on display modules of different manufacturers and different configurations, images without glare and no dizziness can be displayed, and the user experience and experience will be improved to the best, such as 240 Improved display quality for frames, 120 frames, 60 frames, etc.

In addition, when forming enhanced image data, it can be performed according to the following methods:

Enhance video images

First, the image sensor processing module ISP of the image processing chip IPIC will preprocess the image as described above. After the preprocessing is completed, the preprocessed image data will be sent to the image processing unit IPU of the image processing chip IPIC for images. Enhancement. The enhanced image data formed after the image enhancement processing is completed will be transmitted back to the central processing chip.

After the central processing chip receives the enhanced image data, it performs video encoding. The encoding protocol can be ITU H.261, H.263, M-JPEG of the Moving Picture Experts Group, and MPEG series standards of the International Organization for Standardization Moving Picture Experts Group. Etc. Video data is formed after encoding. Because the video data is based on the enhanced image data formed by the image processing chip, each frame of the formed video data is a picture frame that has been optimized, and the video data after the synthesis of each frame is more colorful on the screen sense.

The final video data is recorded by the central processing chip to a storage device for storage and subsequent recall.

The IPU is connected to the display module in another way. Specifically, the enhanced image is sent to the data processing unit DPU. The DPU adds transparency to the interactive interface and sends it to the display module to load and display.

This embodiment is an application that uses a camera component or a camera module for video shooting based on the technical solution of the present invention. An external storage interface (USB, TF, etc.) is added through an image processing chip, which can be directly stored in a storage device external to the image processing chip when recording. on.

Image enhancement

The image sensor processing module ISP of the image processing chip IPIC will preprocess the image as described above. After the preprocessing is completed, the preprocessed image data will be sent to the image processing unit IPU of the image processing chip IPIC for image enhancement. The enhanced image data formed after the image enhancement processing is completed will be transmitted back to the central processing chip.

The central processing chip will compress the enhanced image data, such as a jpeg encoder jpeg encoder, to form a compressed image. The JPEG encoding process first needs to convert the RGB format in the enhanced image data to YUV format. When recording computer images, the most common is to use RGB (red, green, blue) color components to save color information, such as uncompressed 24 Bit BMP images use RGB space to save the image. A pixel is 24 bits, and a color intensity (0-255) is stored every 8 bits. For example, red is stored as 0xFF0000. YUV is a color coding method adopted by the European television system, and this method is also commonly used in China's radio and television. Wherein "Y" represents the brightness (Luminance or Luma), which is the grayscale value; and "U" and "V" represent the chrominance (Chrominance or Chroma). The use of YUV space in color TVs is precisely to solve the compatibility problem between color TVs and black and white TVs with the luminance signal Y, so that black and white TVs can also receive color TV signals. When converting RGB to YUV, it is usually taken as Y = 0.299R + 0.587G + 0.114B, U = -0.147R-0.289G + 0.436B, V = 0.615R-0.515G-0.100B, R = Y + 1.14 V, G = Y-0.39U-0.58V, B = Y + 2.03U. Then the image is divided into 8 * 8 blocks. After the original image is converted to YUV format, the image is sampled according to a certain sampling format. Common formats are 4: 4: 4, 4: 2: 2, and 4: 2: 0. After the sampling is completed, the image is divided into MCUs according to 8 * 8 (pixel). Subsequently, discrete cosine transform (DCT) is performed. Discrete cosine transform (DCT) is a transform coding method commonly used for digital rate compression. The Fourier transform of any continuous real symmetric function contains only the cosine term, so the cosine transform has the same clear physical meaning as the Fourier transform. DCT first divides the overall image into N * N pixel blocks, and then performs DCT transformation on the N * N pixel blocks one by one. Because the high-frequency components of most images are small, the coefficients corresponding to the high-frequency components of the images are often zero, and the human eye is less sensitive to distortion of high-frequency components, so coarser quantization can be used. Therefore, the bit rate for transmitting the transform coefficients is much smaller than the bit rate used for transmitting image pixels. After arriving at the receiving end, the samples are returned to the samples through the inverse discrete cosine transform. Although there will be some distortion, the human eye is acceptable. The image signal is decomposed into DC components; and various cosine components from low frequency to high frequency; and the DCT coefficient only represents the share of the original image signal by this component; obviously, the restored image information can be expressed as such a matrix form : F (n) = C (n) * E (n), where E (n) is a basis, C (n) is the DCT coefficient, and F (n) is the image signal. If we consider the change in the vertical direction again, then we need a two-dimensional base, that is, the base must not only reflect the change in the frequency in the horizontal direction; but also reflect the change in the frequency of the vertical space; the pixel block corresponding to 8 * 8; space The base is an image composed of 64 pixel values, which is also commonly called a base image. They are called basic images because in the inverse transform of discrete cosine transform, any image block can be represented as a combination of 64 coefficients of different sizes. Since the basic image is equivalent to a single coefficient in the transform domain, any pixel can also be regarded as a combination of 64 basic images with different amplitudes. This has the same physical meaning as any combination of signals that can be broken down into fundamental waves and harmonics of different amplitudes. The quantization process is a process of discretizing the amplitude of a signal. The discrete signal becomes a digital signal after quantization. Because HVS is more sensitive to low-frequency signals, a relatively short quantization step size is used for the low-frequency part of the signal, and a relatively long quantization step size is used for the high-frequency part of the signal. In this way, a relatively clear image and a higher compression ratio can be obtained to a certain extent. Then zigzag scan is performed, and the quantized data is read out according to the zigzag. Finally, run-length coding (RLE) is used to encode the AC coefficient. The so-called run-length coding refers to that a code can simultaneously represent the value of the code and a few zeros in front of it. In this way, the advantages of Z-shaped readout are used. Because Z-shaped readout, there are more chances of continuous zeros. Especially at the end, if it is all zero, after reading the last number, just give "end of block" (EOB) code, you can end the output, thus saving a lot of code rates.

The final compressed image is recorded by the central processing chip into a storage device and stored for subsequent recall.

This embodiment is an application that uses a camera component or a camera module to perform image shooting and photography based on the technical solution of the present invention. An external storage interface (USB, TF, etc.) is added through the image processing chip. Storage device.

Enhance preview image

After the central processing chip is connected to the image processing chip, the central processing chip sends pre-configured interactive interface GUI data of the preview interface for image preview to the image processing chip through the DSI interface. DSI defines a high-speed serial interface between the processor and the display module. It consists of four layers, the PHY layer, Lane management layer, Low level Protocol layer, and Application layer, which correspond to the D-PHY, DSI, and DCS specifications, respectively. The PHY defines the transmission medium, input / output circuits, and clock and signal mechanisms. The Lane Management layer sends and collects data streams to each lane. The Low Level protocol layer defines how to frame and analyze, and detects errors. The Application layer describes The high-level layer encodes and parses the data stream, and the interactive interface data includes transparency information of the interactive interface, that is, Alpha information.

The image processing chip integrates the interactive interface data with the enhanced image data through Alpha Blending, so that the enhanced image is displayed on the interactive interface as a display interface, that is, an integrated image is displayed. The integrated image will be sent to the display module via the DPU for display.

This embodiment is an application that provides a preview function to a user when an image is captured using a camera component or a camera module based on the technical solution of the present invention, and provides a user with a more acceptable and habitual visual perception through cooperation with an interactive interface.

Referring to FIG. 8, the present invention also discloses an image processing system including a display module, an image sensor, an image processing chip, and a central processing chip disposed in a terminal; the image processing chip is separately from the display module, the image sensor, and the central processing chip. Connection; the image sensor collects the image data of the subject and sends it to the image processing chip; the digital signal processing module of the image processing chip processes the original data of the image data to form optimized image data; the digital signal processing module sends the optimized image data to the image The image sensor processing module of the processing chip processes the optimized image data to form enhanced image data; the image processing chip returns the optimized image data and / or enhanced image data to a central processing chip in a smart terminal, which is recorded by the central processing chip The optimized image data and / or enhanced image data is sent to a storage device, and the image processing chip sends the optimized image data and / or enhanced image data to a display module, and the display module displays the optimization corresponding to the optimized image data and / or enhanced image data. Images and / Enhanced image.

Preferably, the digital signal processing module interpolates and supplements the original data to form multi-frame synthetic image data; or the digital signal processing module renders and processes the original data to form optimized image data.

In addition, based on the above image processing method, a computer-readable storage medium may be installed in the smart terminal, and a computer program is stored thereon, and when the computer program is executed by a processor, the steps of the image processing method as described above are implemented.

Smart terminals can be implemented in various forms. For example, the terminals described in the present invention may include smart terminals such as mobile phones, smart phones, notebook computers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), navigation devices, and the like as well as Fixed terminal for digital TV, desktop computer, etc. In the following, it is assumed that the terminal is a smart terminal. However, those skilled in the art will understand that the configuration according to the embodiment of the present invention can be applied to a terminal of a fixed type, in addition to an element particularly used for mobile purposes.

It should be noted that the embodiments of the present invention have better implementation, and do not limit the present invention in any form. Any person skilled in the art may use the disclosed technical content to change or modify the equivalent effective embodiments. However, without departing from the content of the technical solution of the present invention, any modifications or equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims (10)

1. An image processing method includes the following steps:

   S100: The image sensor collects image data of the shooting object;

   S200: The image sensor sends the raw data of the image data to an image processing chip, and the digital signal processing module of the image processing chip processes the raw data to form optimized image data;

   S300: The digital signal processing module sends the optimized image data to an image sensing processing module of the image processing chip to process the optimized image data to form enhanced image data;

   S400: The image processing chip returns the optimized image data and / or enhanced image data to a central processing chip in a smart terminal, and the central processing chip records the optimized image data and / or enhanced image data to A storage device;

   S500: The image processing chip sends the optimized image data and / or enhanced image data to a display module, and the display module displays an optimized image and / or enhanced corresponding to the optimized image data and / or enhanced image data. image.
2. The image processing method according to claim 1, wherein:

   The step S200 includes:

   S210: The image sensor sends the image data to the image processing chip in a raw data format through a camera interface;

   S220: The image sensing module of the image processing chip processes the raw data to form optimized image data.
3. The image processing method according to claim 2, wherein:

   The step S220 includes:

   S221: The image sensing module will extract picture frames in the original data;

   S222: Synthesize the picture frame or insert a supplementary frame into the picture frame to form multi-frame synthesized image data.
4. The image processing method according to claim 2, wherein:

   The step S220 includes:

   S221 ': the image sensing module determines a picture environment of the original data;

   S222 ': Render and process the raw data for the picture environment to form optimized image data.
5. The image processing method according to claim 2, wherein:

   The step S300 includes:

   S310: the digital signal processing module forwards the original data to a central processing chip;

   S320: The intelligent module of the central processing chip renders and processes the raw data to form optimized image data;

   S330: The intelligent module sends the optimized image data to a processing module.
6. The image processing method according to claim 1, wherein:

   The image processing method further includes:

   S600: The image sensor sends frame rate data of the image data to an image processing chip;

   S700: The image processing chip performs frame interpolation compensation on the frame rate data to form frame interpolation data, and the frame interpolation data matches a refresh rate of a display module of the smart terminal;

   S800: compress the frame interpolation data and store the data into a storage device;

   S900: The image processing chip sends the frame interpolation data to a display module, and the display module displays the frame interpolation image corresponding to the frame interpolation data.
7. The image processing method according to claim 6, wherein:

   The frame rate of the frame rate data is 60 fps or more.
8. An image processing system, comprising a display module, an image sensor, an image processing chip, and a central processing chip provided in a terminal;

   The image processing chip is connected to the display module, the image sensor and the central processing chip, respectively;

   The image sensor collects image data of a shooting object and sends the image data to an image processing chip;

   The digital signal processing module of the image processing chip processes the original data of the image data to form optimized image data;

   The digital signal processing module sends the optimized image data to an image sensing processing module of the image processing chip to process the optimized image data to form enhanced image data;

   The image processing chip returns the optimized image data and / or enhanced image data to a central processing chip in a smart terminal, and the central processing chip records the optimized image data and / or enhanced image data to a storage Equipment, and the image processing chip sends the optimized image data and / or enhanced image data to a display module, and the display module displays an optimized image corresponding to the optimized image data and / or enhanced image data and / Or enhance the image.
9. The image processing system according to claim 8, wherein:

   The digital signal processing module interpolates and supplements the original data to form multi-frame synthetic image data; or the digital signal processing module renders and processes the original data to form optimized image data.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1-7 are implemented.

Images