JP2020092302A - Image processing apparatus, image processing apparatus control method, and program - Google Patents

Abstract

To provide an image processing device capable of improving a processing performance in a case where a difference image on the supposition of re-processing after photographing is recorded.SOLUTION: An image processing device comprises: first image processing means 104 that performs irreversible image processing to image data generated by imaging means 102; first compression means 111 that performs compression processing of the image data subjected to the irreversible image processing by the first image processing means 104; second image processing means 109 that performs reversible image processing to the image data; generation means 110 that generates difference data between the image data after the irreversible image processing and the image data after the reversible image processing; second compression means 111 that performs compression processing of the difference data generated by the generation means 110; and recording means 108 that records the image data compressed by the first compression means 111 and the image data compressed by the second compression means 111.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing device, a control method for the image processing device, and a program, and more particularly to an image processing device for image-processing and compressing and recording a captured image.

The function of image-processing an image captured by a digital camera and recording it on a recording medium as a standard compressed file such as JPEG is essential. Furthermore, HEVC compression and recording in HEIF format are becoming widespread. On the other hand, the image processing in the camera is required to have a higher pixel rate and a higher frame rate of the sensor, and thus to have a higher image quality during high-sensitivity shooting. Along with this, the image processing in the camera cannot keep up with the shooting speed during high-speed continuous shooting. Further, with regard to image processing, application of deep learning to image processing takes time, but higher quality image processing has appeared. Therefore, it is considered that the image is once recorded by the conventional image processing at the time of shooting, and the advanced image processing is performed again on the recorded image after the shooting. Further, at the time of shooting, a difference image for restoring the RAW data necessary for reprocessing is also recorded. For example, Patent Document 1 discloses a technique of adding coded data obtained by losslessly coding a difference image, which is a difference from RAW data, to a JPEG file of a main image and recording the same.

JP, 2006-173931, A

However, in the technique disclosed in Patent Document 1, since the JPEG main image is decoded and the inverse conversion process is performed at the time of shooting to generate the difference image, the entire processing time becomes longer than that of the normal shooting process. Memory usage increases.

In view of the above problems, it is an object of the present invention to provide an image processing apparatus that improves processing performance when recording a difference image that is supposed to be reprocessed after shooting.

In order to solve the above-mentioned problems, an image processing apparatus of the present invention includes a first image processing means for performing irreversible image processing on image data generated by an image capturing means, and the first image processing means. And a second image different from the first image processing unit for performing reversible image processing on the image data, the first compression unit for compressing the image data subjected to the irreversible image processing by A processing means; a generation means for generating difference data between the image data after the lossy image processing and the image data after the lossless image processing; and a compression processing for the difference data generated by the generation means It is characterized by comprising a second compression means and a recording means for recording the image data compressed by the first compression means and the image data compressed by the second compression means.

According to the present invention, it is possible to provide an image processing apparatus capable of improving processing performance when recording a difference image assuming reprocessing after shooting.

It is a functional block diagram which concerns on 1st Embodiment. It is a figure which shows the processing flow which concerns on 1st Embodiment. It is a timing chart figure which concerns on 1st this embodiment. It is a figure which shows the 2nd process flow which concerns on 1st Embodiment. FIG. 6 is a second timing chart diagram according to the first embodiment. It is a figure explaining the reversible image processing which concerns on 1st Embodiment, and its inverse conversion. It is a figure explaining the demosaic process which concerns on 1st Embodiment. It is a figure explaining re-bayerization concerning a 1st embodiment. FIG. 6 is a diagram illustrating gamma correction and inverse gamma correction according to the first embodiment. FIG. 3 is a diagram illustrating matrix conversion and inverse conversion according to the first embodiment. It is a figure explaining the distortion correction and inverse correction which concern on 1st Embodiment. It is a figure explaining the difference generation process and addition process which concern on 1st Embodiment. It is a figure explaining the HEIF file structure which concerns on 1st Embodiment. 5 is a flowchart showing an external server process according to the first embodiment. It is a figure explaining the switching of the process which concerns on 1st Embodiment. It is a figure which shows the processing flow which concerns on 2nd Embodiment. It is a timing chart figure which concerns on 2nd Embodiment.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
First, with reference to FIG. 1, an image pickup apparatus that functions as an image processing apparatus that generates an image signal from picked-up images formed by a plurality of image pickup optical systems, performs image processing and compression processing, and performs recording will be described. In the present embodiment, an example of performing image processing and compression processing inside the image pickup apparatus will be described, but the present invention is not limited to these. For example, the imaging device may transfer the generated image data to an external processing device such as an external server or a cloud via communication, and the external processing device may perform image processing and compression processing. FIG. 1 is a block diagram showing the arrangement of the image pickup apparatus according to this embodiment. The image pickup apparatus includes an image forming optical unit 101, an image pickup element 102, an A/D conversion circuit 103, an irreversible image processing unit 104, and a CPU 105. The image forming optical unit 101 includes a lens, a diaphragm, and the like, and performs focus adjustment and exposure adjustment. The image sensor 102 is an image sensor such as a CCD that converts an optical image into an electric signal. The A/D conversion circuit 103 converts the analog image signal from the image sensor 102 into digital image data. The output RAW data is, for example, Bayer array data in which all the color data are not available in the pixel. The irreversible image processing unit 104 performs irreversible image processing such as noise reduction, demosaicing, color conversion, and gamma conversion on the image data output from the A/D conversion circuit 103. The CPU 105 controls various controls via the bus.

Note that the irreversible image processing is image processing for recording the main image of the captured image, and in the case of the solid portion, the edge portion, or the edge portion for each area of the captured image, the feature amount of the image feature such as the direction of the edge. Is extracted, and development processing including noise removal processing and demosaicing processing suitable for this is performed. In addition, depending on the subject, there may be a case where an exposure correction suitable for each image area is performed. Therefore, these image processes are generally irreversible processes according to the characteristics of the captured image, and the number of parameters required for the processes is large.

Further, the imaging device includes a display unit 106, a memory 107, a recording unit 108, a reversible image processing unit 109, an addition/subtraction processing unit 110, a compression/expansion processing unit 111, and a communication unit 112, and the communication unit 112. To the external server 113. The display unit 106 is a display unit configured by a liquid crystal monitor or the like and displaying the image data stored in the memory 107. The display unit 106 also serves as an input device such as a shutter button, a menu button, a direction key, and an enter key for the user to input various instructions and settings to the imaging device. The memory 107 is configured by a DRAM or the like that temporarily stores the image data output from the lossy image processing unit 104 or the lossless image processing unit 109. It is also used as a buffer area for compressed data of the compression/expansion processing unit 111, which will be described later, and a storage area for display data for the display unit 106. The recording unit 108 is composed of an SD card or the like, and records the compressed image data stored in the memory 107 in a file format such as JPEG or HEIF. The reversible image processing unit 109 performs reversible image processing such as demosaic processing, color conversion, and gamma processing on the image data output from the A/D conversion circuit 103. Note that the reversible image processing is image processing for generating image data for RAW restoration, and is characterized in that reversibility is prioritized rather than image quality priority, and the number of parameters is minimized.

The addition/subtraction processing unit 110 generates a difference image between the output image data of the lossless image processing unit 109 and the output image data of the lossy image processing unit 104 during shooting. Further, in the RAW restoration at the time of reprocessing after recording, the main recording image and the difference image data are added to generate image data for RAW restoration. The compression/expansion processing unit 111 performs compression processing of output image data of the irreversible image processing unit 104 or compression processing of output difference image data of the addition/subtraction processing unit 110 at the time of shooting. The compression of the output image data of the irreversible image processing unit 104 may be JPEG compression, HEVC compression, or the like. Reversible lossless compression is preferable for the compression of the output difference image data of the addition/subtraction processing unit 110, but it is also possible to prioritize the compression of the recording file size and make it irreversible. These compressed data are stored in the memory 107. Further, when reprocessing after recording, these compressed image data are expanded. The communication unit 112 transmits/receives data to/from an external communication device such as WiFi. The external server 113 is a server such as a network and cloud server.

Next, the processing flow according to the present embodiment will be described with reference to FIG. FIG. 2A is a diagram illustrating an example of a processing flow at the time of shooting. The RAW data 201 is captured RAW data and is Bayer array RAW data output from the A/D conversion circuit 103. Next, the lossy image processing 202 is lossy image processing performed by the lossy image processing unit 104. The main image YUV data 203 is YUV data for main image recording which is an output of the irreversible image processing 202. Then, the main image YUV data 203 is written in the memory 107. Next, the HEVC compression 204 is HEVC compression processing performed by the compression/expansion processing unit 111, and HEVC compresses the main image YUV data 203. Although the HEVC compression means is taken as an example in the present embodiment, another compression method such as JPEG compression may be used. Then, the compressed data after the compression processing 204 is written in the memory 107.

On the other hand, the reversible image processing 206 is reversible image processing performed by the reversible image processing unit 109. The lossless parameter used in this embodiment is generated based on the parameter used in the lossy image processing 202. The CPU 105 processes the generation of the lossless parameter from the lossy image processing parameter. This processing will be described in detail later. Next, the RAW restoration YUV data 207 is the RAW restoration YUV data output from the lossless image processing 206. Then, the RAW restoration YUV data 207 is written in the memory 107. Next, a difference generation process 208 is a difference generation process performed by the addition/subtraction processing unit 110, and includes the main image YUV data 203 after irreversible image processing on the memory 107 and the RAW restoration YUV data 207 after reversible image processing. Difference data 209 is generated. Then, the difference data 209 is an output of the difference generation processing 208 in the addition/subtraction processing unit 110 and is written in the memory 107. Next, the compression processing 210 is a compression processing of the difference data 209 performed by the compression/expansion processing unit 111. Considering the RAW recoverability, the lossless compression is preferable, but if the data size after compression is prioritized over the RAW recoverability, the lossy compression process may be used.

Then, the HEIF file 205 is formed as the HEIF file 205, and the compressed data after the compression processing 210 of the difference data 209 is recorded in the recording unit 108 as the compressed data after the HEVC compression processing 204 of the main image YUV data 203. The lossless parameter used in the lossless image processing 206 is also added to the HEIF file 205 and recorded.

FIG. 2B is a processing flow for restoring the RAW from the HEIF file 205. The HEIF file 221 is a recording file recorded by the recording unit 108 at the time of shooting, and is the HEIF file 205 of FIG. The HEVC-compressed main image YUV compressed data is read from the HEIF file 221, the HEVC decompression processing unit 222 performs HEVC decompression processing 222, and the decompressed main image YUV data 223 is output. Then, the output decompressed main image YUV data 223 is written in the memory 107. On the other hand, the compressed difference data is read from the HEIF file 221, the compression/expansion processing unit 111 performs the expansion processing 228, and the expansion difference data 229 is output. Then, the output expansion difference data 229 is written in the memory 107.

Next, the decompressed main image YUV data 223 and the decompressed difference data 229 are read from the memory 107, the addition/subtraction processing unit 110 performs addition processing 224, and the RAW restoration YUV data 225 is output. The output RAW restoration YUV data 225 is written in the memory 107. The RAW restoration YUV data 225 is obtained by expanding and restoring the RAW restoration YUV data 207 at the time of shooting.

Further, the reversible parameters recorded in the HEVC compressed data and the differential compressed data at the time of shooting are read from the HEIF file 221. A parameter inverse conversion process 230 is performed on the read lossless parameter to generate an inverse conversion parameter. Next, the RAW restoration YUV data 225 is read from the memory 107, the reversible image processing unit 109 performs the reverse conversion processing 226 of the reversible image processing, and the restored RAW data 227 is generated and written in the memory 107. At this time, as the processing parameter of the inverse conversion processing 226, the inverse conversion parameter generated by the parameter inverse conversion processing 230 is used. The parameter inverse conversion processing 230 is processed by the CPU 105.

FIG. 2C is a processing flow for performing high-load and advanced irreversible image processing, which is different from the irreversible image processing shown in FIG. 2A, on the restored RAW data. Here, the advanced irreversible image processing is, for example, an image processing of extremely high image quality such as applying deep learning to the image processing, but the processing time per image is long and is processed at the time of shooting. That is not preferable in terms of product function. For example, the image processing may take several tens of seconds to several minutes or more per image.

First, the restored RAW data 241 is the restored RAW data 227 restored in FIG. The advanced irreversible image processing A 242 reads the restored RAW data 227 from the memory 107, performs image processing in the irreversible image processing unit 104, and outputs the processed RAW data 243. The output processed RAW data 243 is written in the memory 107.

Next, the advanced irreversible image processing B244 reads the processed RAW data 243 from the memory 107, performs image processing in the irreversible image processing unit 104, and outputs the processed main image YUV data 245. The output processed main image YUV data 245 is written in the memory 107. Next, the processed main image YUV data 245 is subjected to HEVC compression processing 246 in the compression/expansion processing unit 111, and the HEVC compressed data is written in the memory 107.

On the other hand, the selector 248 is a processing flow changeover switch, and switches which of the restored RAW data 241 and the processed RAW data 243 is selected. The selector 248 selects which RAW data to be restored this time. The RAW selected by the selector 248 is subjected to the reversible image processing 249 in the reversible image processing unit 109 and outputs the RAW restoration YUV data 250. The output RAW restoration YUV data 250 is written in the memory 107. The lossless parameter used in this embodiment is generated based on the parameter of the advanced lossy image processing B244. The generation method is the same as the generation of the reversible parameter of the reversible image processing 206 of FIG. The generation of the reversible parameter is processed by the CPU 105.

Next, the processed main image YUV data 245 and the RAW restoration YUV data 250 are read from the memory 107, the addition/subtraction processing unit 110 performs difference generation processing 251, and the generated difference data 252 is written in the memory 107. Next, the difference data 252 is compressed by the compression/expansion processing unit 111 and the compressed data is output. The output compressed data is written in the memory 107. Next, a HEIF file 247 shows a HEIF file, and the compressed data after the HEVC compression processing 246 of the preprocessed main image YUV data 245 and the compressed data after the compression processing 253 of the difference data 252 are formed as a HEIF file 247. Then, it is recorded in the recording unit 108. The lossless parameter used in the lossless image processing 249 is also added to the HEIF file 247 and recorded.

Although the processing flow of FIG. 2A has been described as the processing flow at the time of shooting, the processing flow can be applied to the processing of the restored RAW data 227 restored by the processing flow of FIG. 2B. Further, it can also be applied to the post-processing raw data 243 of the processing flow of FIG.

FIG. 3 is a diagram illustrating a timing chart of the processing flow shown in FIG. The V signal 301 represents the timing at which still image shooting was performed. This embodiment shows that high-speed continuous shooting is performed and continuous shooting is performed at the timing of the V signal 301. As shown in FIG. 3, acquisition of RAW data 201 by imaging, lossy image processing 202, HEVC compression processing 204, lossless image processing 206, difference generation processing 208, difference data compression processing 210, and recording of HEIF file 205 are as follows. The consecutive captured images are sequentially processed in parallel. Further, the irreversible image processing to HEVC compression, or the reversible image processing to the difference generation and the difference data compression are performed in a timing chart such that processing is performed before the processing of one image is completed.

The period T1 is the processing time of the entire series of processing from shooting to recording the HEIF file 205. The entire processing time T1 for one image can be shortened as much as possible by eliminating the waiting for the completion of processing as much as possible and constructing only the forward processing. Further, by doing so, it is possible to suppress the extension of the lifetime of the memory buffer required for each process and to suppress the amount of memory used.

FIG. 4 is a diagram illustrating an example of a processing flow of difference data generation different from that in FIG. The RAW data 401 is captured RAW data and is Bayer array RAW data output from the A/D conversion circuit 103. Alternatively, the restored RAW data 227 restored by the processing flow of FIG. 2B may be used. Further, it may be the post-process raw data 243 of the process flow of FIG.

The image processing 402 is an irreversible image processing performed by the irreversible image processing unit 104. The main image YUV data 403 is the main image recording YUV data output from the irreversible image processing 402. The output main image YUV data 403 is written in the memory 107. Next, the HEVC compression processing 404 is HEVC compression processing performed by the compression/expansion processing unit 111, and HEVC compresses the main image YUV data 403. Although the HEVC compression means is taken as an example in the present embodiment, another compression method such as JPEG compression may be used. Then, the compressed data after the compression processing 404 is written in the memory 107.

On the other hand, the reversible image processing 406 is reversible image processing performed by the reversible image processing unit 109. The lossless parameter used in the present embodiment is generated based on the parameter used in the lossy image processing 402. This generation will be described in detail later. Next, the RAW restoration YUV data 407 is the RAW restoration YUV data output from the lossless image processing 406. The output RAW restoration YUV data 407 is written in the memory 107.

Next, the HEVC decompression process 412 is the HEVC decompression process performed by the compression/decompression processing unit 111. The HEVC compressed data output by the HEVC compression processing 404 on the memory 107 is expanded, and the expanded main image YUV data 411 is written in the memory 107. Next, difference generation processing 408 is difference generation processing performed by the addition/subtraction processing unit 110, and generates difference data 409 of the expanded main image YUV data 411 and the RAW restoration YUV data 407 on the memory 107. The difference data 409 is an output of the difference generation processing 408 in the addition/subtraction processing unit 110 and is written in the memory 107. Next, the compression processing 410 is a compression processing of the difference data 409 performed by the compression/expansion processing unit 111. Considering the RAW recoverability, the lossless compression is preferable, but if the data size after compression is prioritized over the RAW recoverability, the lossy compression process may be used.

Then, the compressed data of the main image YUV data 403 after the HEVC compression process 404 and the compressed data of the difference data 409 after the compression process 410 are formed as a HEIF file 405 and recorded in the recording unit 108. The lossless parameters used in the lossless image processing 406 are also added to the HEIF file 405 and recorded. The HEIF file 405 recorded in FIG. 4 can be restored to the restored RAW data 227 by the processing flow of FIG. Further, the processing flow of FIG. 4 can be applied to the post-processing raw data 243 of the processing flow of FIG.

Next, FIG. 5 is a diagram illustrating a timing chart of the processing flow shown in FIG. The V signal 501 represents the timing when the still image shooting was performed. This embodiment shows that high-speed continuous shooting is performed and continuous shooting is performed at the timing of the V signal 501. As shown in FIG. 5, the acquisition of RAW data 401 by imaging, the irreversible image processing 402, the HEVC compression processing 404, and the reversible image processing 406 sequentially process parallel captured images in parallel.

The HEVC decompression process 412 waits for the HEVC compression process 404 and is performed. Then, the difference generation process 408, the difference data compression process 410, and the recording of the HEIF file 405 in the recording unit 108 are sequentially started after the HEVC decompression process 412 is started. Regarding the lossy image processing to the HEVC compression, or the lossless image processing to the difference generation and the difference data compression, the timing chart is such that the processing is performed before the processing of one image is completed.

The period T2 is the processing time of the entire series of processing from shooting to recording the HEIF file 405. In this embodiment, it can be seen that the period T2 takes longer than the period T1 shown in the timing chart of FIG. That is, in the processing flow shown in FIG. 4, the lifetime of the memory buffer required for each processing becomes longer than that in the processing flow shown in FIG. 2A, and the used memory amount tends to increase.

On the other hand, in the processing flow shown in FIG. 4, the difference data 409 is acquired by performing the difference generation processing 408 between the expanded main image YUV data 411 and the RAW restoration YUV data 407 instead of the main image YUV data 403. Therefore, when the restoration RAW is generated by the processing flow shown in FIG. 2B, the error of the restoration processing is suppressed, and as a result, the quality of the restoration RAW tends to be improved.

Next, FIG. 6 is a diagram for explaining the processing flow of the lossless image processing 206 shown in FIG. 2A and the inverse conversion processing 226 of the lossless image processing shown in FIG. .. First, FIG. 6A is a diagram showing an example of a processing flow of the reversible image processing 206. The Bayer array RAW data 601 may be the captured RAW data 201 of FIG. 2A, or may be the restored RAW data 241 and the post-processing RAW data 243 of FIG. 2C.

Next, the demosaic process 602 interpolates the missing colors at each pixel position from the image data of the Bayer array to generate all RGB colors. The gamma correction processing 603 performs gamma correction according to the gamma characteristic of the main recording image. The matrix conversion processing 604 performs conversion from the RGB color space to the YUV color space. The distortion correction processing 605 corrects the lens distortion aberration. The RAW restoration YUV data 606 is output data of the lossless image processing 206. In the processing flow shown in FIG. 2A, this corresponds to the RAW restoration YUV data 207. The details of each process will be described later.

FIG. 6B is a diagram showing an example of the processing flow of the inverse conversion processing 226 of the reversible image processing. The RAW restoration YUV data 621 is input image data of the inverse conversion 226 of the reversible image processing. It should be noted that in the processing flow shown in FIG. 2B, it is the RAW restoration YUV data 225. The inverse conversion process 622 of distortion correction performs the inverse conversion process of the distortion correction process 605. The matrix inverse conversion process 623 is an inverse conversion process of the matrix conversion process 604, and performs conversion from the YUV color space to the RGB color space. The inverse gamma correction process 624 performs the inverse conversion process of the gamma correction process 603. The re-Bayer conversion 625 generates image data in a Bayer array from image data in which all RGB colors are prepared for each pixel. In the processing flow shown in FIG. 2B, the restored RAW data 626 is the restored RAW data 227 output from the inverse conversion processing 226 of the lossless image processing. The details of each process will be described later.

Next, an example of demosaic processing will be described with reference to FIG. 7. In the image data 701 of the Bayer array, pixel values of the color RG are repeatedly arranged for each pixel in the first row, the third row, and the subsequent odd rows. In the second row name, fourth row, and even-numbered rows thereafter, pixel values of GB color are repeatedly arranged for each pixel.

Interpolation processing 702 indicates R color interpolation processing. R pixels are arranged every other pixel in odd-numbered rows of the Bayer image, and these are output as they are. Since the R pixel value at the G position in the Bayer array is between the left and right or upper and lower R pixels, it is calculated from the average of these two R pixel values. The R pixel value at the position B in the Bayer array is calculated from the average of the four R pixel values at the upper left, upper right, lower left, and lower right. Interpolation processing 703 indicates G color interpolation processing. The G color of the Bayer array is output as it is. The G pixel value at the R or B pixel position in the Bayer array is obtained from the average of the left and right G pixel values. Interpolation processing 704 indicates B color interpolation processing. B-pixels are arranged every other pixel in even rows of the Bayer image, and these are output as they are. Since the B pixel value at the G position in the Bayer array is between the left and right or the upper and lower B pixels, it is calculated from the average of these two B pixel values. The B pixel value at the R position in the Bayer array is calculated from the average of the four B pixel values at the upper left, upper right, lower left, and lower right. Data 705 indicates output image data in which each pixel RGB is complete after interpolation processing (702, 703, 704) for each color.

Next, an example of the re-bayering process will be described with reference to FIG. The data 801 is output image data in which each pixel RGB before re-bayering is complete. The R pixel value 802 indicates the R pixel of the data 801. The thinning-out is performed in the arrangement from the R pixel value 802 to the R pixel value 805. The thinned out areas have zero value. The G pixel value 803 indicates the G pixel of the data 801. The G pixel value 803 to the G pixel value 806 are thinned out. The thinned out areas have zero value. The B pixel value 804 indicates the B pixel of the data 801. The B pixel value 804 to the B pixel value 807 are arranged to perform thinning. The thinned out areas have zero value. The re-bayered data 808 is generated by combining the thinned pixel values (805, 806, 807) of the respective colors.

FIG. 9A is a diagram showing an example of gamma correction processing. In the present embodiment, gamma correction is processed by approximation defined by a polygonal line of three points. The polygonal line is defined by the origin (0, 0) and the output values for the three inputs (BP1, BC1), (BP2, BC2), and (BP3, BC3), and the starting point of the line is defined by the gradient GR0. , GR1, GR2, GR3. Between the origin and the three points, the output for the input is determined by each defined straight line. As a typical example of gamma correction, an upwardly convex curve as shown in FIG. 9A is defined as a polygonal line that approximates the curve.

FIG. 9B is a diagram showing an example of the inverse gamma correction process. This realization method is similar to the gamma correction process in FIG. 9A, and the definitions of three points (BP1, BC1), (BP2, BC2), and (BP3, BC3), and the slopes GR0, GR1, GR2 of the respective straight lines , GR3 is different from that of FIG. 9(A). As a typical example of the inverse gamma correction, a downward convex curve as shown in FIG. 9B is defined as a polygonal line that approximates the curve. In each of the above explanations, the definition processing is performed using a polygonal line of three points. However, the input/output definition polygonal line may be approximated to a target curve by increasing the number of similar definition parameters and increasing the number of points of the polygonal line.

FIG. 10 is a diagram showing an example of matrix conversion. FIG. 10A shows conversion from RGB to YUV, which is processed by 3×3 matrix conversion. The 3×3 matrix coefficients a00, a01, a02, a10, a11, a12, a20, a21, a22 may be defined as coefficients for color space conversion in standards such as video and recording formats. In the case of the present embodiment, it is preferable to conform to the standard of the YUV space that is the target of HEVC compression. Next, FIG. 10B shows an inverse conversion from YUV to RGB, which is processed by 3×3 matrix conversion. The coefficients b00, b01, b02, b10, b11, b12, b20, b21, b22 of the 3×3 matrix may be defined as the coefficients for color space conversion in the standard such as video and recording format. In the case of the present embodiment, it is preferable to conform to the standard of the YUV space that is the target of HEVC compression. This 3×3 matrix corresponds to the inverse matrix of the 3×3 matrix defined in FIG.

FIG. 11 is a diagram showing an example of distortion aberration correction. The image data 1101 is image data having lens distortion. For example, the image data 1101 is corrected like the image data 1102 by the distortion correction process. In the distortion reverse correction process, on the contrary, the distortion aberration is returned to the corrected image data 1102 to obtain the uncorrected image data 1101. As a method of implementing this, there is a method of defining a scaling factor for each distance from the image height center and resampling from a pixel position based on this.

Next, the difference generation processing by the addition/subtraction processing unit 110 will be described with reference to FIG. The difference generation processing by the addition/subtraction processing unit 110 is the difference generation processing 208 of the processing flow shown in FIG. 2A, the difference generation processing 251 of the processing flow shown in FIG. 2C, and the difference generation processing of the processing flow shown in FIG. It corresponds to 408.

The main image YUV data 1201 corresponds to the main image YUV data 203 in the processing flow shown in FIG. Further, the processed main image YUV data 245 in the processing flow shown in FIG. 2C and the expanded main image YUV data 411 in the processing flow shown in FIG. The RAW restoration YUV data 1205 corresponds to the RAW restoration YUV data 207 in the processing flow shown in FIG. Further, it corresponds to the RAW restoration YUV data 250 in the processing flow shown in FIG. 2C and the RAW restoration YUV data 407 in the processing flow shown in FIG. The 422-444 conversion processing 1202 is processing for converting the YUV422 format to the YUV444 format. That is, for UV, UV values at odd-numbered pixel positions are interpolated in the horizontal direction and converted into YUV444 format. For example, the interpolation process is obtained from the average of two adjacent pixel values.

The shifter 1203 is a bit shifter. When the number of bits of the output of the 422-444 conversion processing 1202 is smaller than the number of bits of the RAW restoration YUV data 1205, the bits are left-shifted so that the bit widths match and the lower bits are zero-filled. The 422-444 conversion processing 1202 and the shifter 1203 determine whether the main image YUV data and the RAW restoration YUV data have the same data format as the sampling ratio of the color component with respect to the luminance, the luminance, and the effective bit number of the color component. This is done for the purpose of matching. The difference data 1204 corresponds to the difference data 209 in the processing flow shown in FIG. Further, it corresponds to the difference data 252 in the processing flow shown in FIG. The output of the shifter 1203 is subtracted from the RAW restoration YUV data 1205 to generate and output difference data 1204.

Next, the addition processing by the addition/subtraction processing unit 110 will be described with reference to FIG. The addition processing by the addition/subtraction processing unit 110 corresponds to the addition processing 224 of the processing flow shown in FIG. The main image YUV data 1221 corresponds to the expanded main image YUV data 223 in the processing flow shown in FIG. The difference data 1225 corresponds to the expansion difference data 229 in the processing flow shown in FIG. The 422-444 conversion process 1222 is a process for converting the YUV422 format to the YUV444 format. That is, for UV, UV values at odd-numbered pixel positions are interpolated in the horizontal direction and converted into 444 format. For example, the interpolation process is obtained from the average of two adjacent pixel values.

The shifter 1223 is a bit shifter. When the number of bits of the output of the 422-444 conversion processing 1222 is smaller than the number of bits of the difference data 1225, the left shift is performed so that the bit widths match and the lower bits are zero-filled. Note that the 422-444 conversion processing 1222 and the shifter 1223 use the difference data 1225 as the data format of the main image YUV data when there is a difference in the sampling ratio of the color component with respect to the brightness, the brightness, and the effective bit number of the color component. It is done for the purpose of matching. The RAW restoration YUV data 1224 corresponds to the RAW restoration YUV data 225 in the processing flow shown in FIG. The difference data 1225 and the output of the shifter 1223 are added to generate and output the RAW restoration YUV data 1224.

Next, with reference to FIG. 13, a data configuration of print data to be recorded in the recording unit 108 will be described. The data structure corresponds to the HEIF file 205 of the processing flow shown in FIG. 2A, the HEIF file 221 of the processing flow shown in FIG. 2B, and the HEIF file 247 of the processing flow shown in FIG. 2C. The HEIF file 1301 includes a header 1302, a HEVC main image 1303, differential image compressed data 1304, and a reversible image processing parameter 1305. The header 1302 includes link information to a plurality of container data items that make up the header. The HEVC main image 1303 is HEVC-compressed main image data, and corresponds to output compressed data of the HEVC compression processing 204 shown in FIG. 2A and the HEVC compression processing 246 shown in FIG. 2C. It also includes information such as image data size and data format. The differential image compressed data 1304 corresponds to the output compressed data of the compression processing 210 shown in FIG. 2A and the compression processing 253 shown in FIG. It also includes information such as image data size and data format. The lossless image processing parameter 1305 is a lossless parameter used in the lossless image processing 206 shown in FIG.

Next, image processing via a network will be described with reference to FIG. In the processing flow illustrated in FIG. 2C, it has been described that the advanced irreversible image processing is performed again on the RAW data restored in the imaging device. In this figure, a case is shown in which an advanced irreversible image processing is performed in the external server 113 or the like via the network via the communication unit 112 rather than in the imaging device. Highly lossy image processing is very high quality image processing such as deep learning applied to image processing, but the processing time per image is long and it is necessary to process at the time of shooting. It is not preferable. For example, the image processing may take several tens of seconds to several minutes or more per image.

First, in step S1401, it is determined whether or not there is a HEIF file 205 that is unprocessed recording data in the external server. When there is an unprocessed HEIF file 205 (YES), the process proceeds to step S1402. If there is no unprocessed HEIF file 205 (NO), the process ends. Next, in step S1402, the HEIF file 205 is sent to the external server. That is, the HEIF file 205 is sent to the external server 113 via the network via the communication unit 112. Next, in step S1403, it is determined whether or not the notification that the lossy image processing by the external server is completed is received. It should be noted that this notification is received via the communication unit 112. If the completion notice has not been received (NO), the process waits until the notice is given in step S1403. When the completion notice is sent (YES), the process proceeds to step S1404. Next, in step 1404, the HEIF file in which the image subjected to the irreversible image processing is compressed and stored is received from the external server 113 via the communication unit 112, saved in the recording unit 108, and the process returns to step S1401. The above is repeated until there are no unprocessed HEIF files 205.

Next, switching control between the case of performing the difference generation with the expansion processing shown in the processing flow shown in FIG. 4 and the case of performing the difference generation without the expansion processing shown in the processing flow shown in FIG. 2A will be described. To do. FIG. 15 is a table for managing conditions for switching between expansion and non-expansion. For example, the user displays this table in a menu and selects the switching condition. In the example shown in FIG. 15, “expansion” is set in the low-speed continuous shooting mode, and the difference data 409 is generated from the difference between the RAW restoration YUV data 407 and the expanded main image YUV data 411 in the processing flow shown in FIG. To be done. On the other hand, in the high-speed continuous shooting mode, “no expansion” is performed, and the difference data 209 is generated from the difference between the RAW restoration YUV data 207 and the main image YUV data in the processing flow shown in FIG. As a result, the switching processing is performed under the designated and managed condition of "no expansion" of recording speed priority and "expansion" of RAW restoration quality priority.

Note that in FIG. 15, high-speed continuous shooting/low-speed continuous shooting is the switching condition, and this has been described. However, during other playback editing/shooting, single shooting/continuous shooting, continuous shooting representative image/other image, The same applies when selected. For example, "at the time of shooting" indicates a case of operating in any shooting mode, and "at the time of reproduction and editing" indicates an operation mode other than that. “Continuous shooting” refers to the case where the camera is operating in the continuous shooting mode, and “single shooting” refers to the shooting mode for single shooting. “High-speed continuous shooting” indicates a continuous shooting mode at or above a certain continuous shooting speed, and “low speed continuous shooting” indicates a shooting mode at or below a certain continuous shooting speed including single shooting. “Continuous shooting representative image” indicates a representative recorded image that is automatically selected in continuous shooting, and this image is processed as “with expansion”. On the other hand, “other” is another continuous shot image, and indicates that the process is performed without “expansion”. In addition, in the present embodiment, there are also options of “always with expansion” and “always without expansion”, which can also be selected. In the present embodiment, an example has been described in which the user selects these conditions from the menu. However, without being limited to this, for example, the condition may be determined in advance according to the required processing speed for each function of the image pickup apparatus and the available memory, and “with expansion” or “without expansion” may be automatically selected.

As described above, in the present embodiment, the case where the processing speed and the small amount of used memory are prioritized as much as possible in the generation of the recording file capable of RAW restoration has been described. Also, a case has been described in which a method having a high RAW restoration property based on a difference from a decompressed image is switched depending on conditions. Therefore, according to the present embodiment, it is possible to perform the RAW-restorable recording file generation process in each shooting mode at the optimum processing speed and quality.

(Second embodiment)
The image pickup apparatus according to this embodiment will be described below. The configuration of the image pickup apparatus according to the present embodiment is the same as the configuration described with reference to FIG. 1 of the first embodiment, and the reversible image processing and its inverse conversion processing described with reference to FIGS. 6 to 11 are the same. is there. The same applies to the embodiments described with reference to FIGS. 13, 14, and 15.

Next, the processing flow of this embodiment will be described with reference to FIG. FIG. 16A is a diagram showing an example of a processing flow at the time of shooting. FIG. 16A shows an example of a processing flow of difference data generation different from those in FIGS. First, the RAW data 1601 is the captured RAW data, which is the Bayer array RAW data output from the A/D conversion circuit 103. Further, it may be the restored RAW data 227 restored by the processing flow shown in FIG. Further, it may be the post-processing raw data 243 of the processing flow shown in FIG. Next, the irreversible image processing 1602 is irreversible image processing performed by the irreversible image processing unit 104. The main image YUV data 1603 is YUV data for main image recording which is an output of the irreversible image processing 1602. Then, the main image YUV data 1603 for recording the main image is written in the memory 107. The HEVC compression processing 1604 is HEVC compression processing performed by the compression/expansion processing unit 111, and HEVC compresses the main image YUV data 1603. Although the HEVC compression means is described as an example in the present embodiment, another compression method such as JPEG compression may be used. Then, the compressed data after the HEVC compression processing 1604 is written in the memory 107.

Next, the HEVC decompression process 1612 is a HEVC decompression process performed by the compression/decompression processing unit 111. The HEVC compressed data output from the HEVC compression processing 1604 on the memory 107 is expanded, and the expanded main image YUV data 1611 is written to the memory 107. Next, the reversible image processing inverse conversion processing 1607 is reversible image processing inverse conversion processing performed by the reversible image processing unit 109. The inverse transformation parameter used in this embodiment is generated by converting the lossless parameter based on the parameter used in the lossy image processing 402 into the inverse transformation parameter in the parameter inverse transformation processing 1613. The CPU 105 processes the lossless parameter generation from the lossy image processing parameter and the inverse parameter conversion process 1613.

Then, the inverse conversion raw data 1606 output by the inverse conversion processing 1607 of the lossless image processing is written in the memory 107. The difference generation processing 1608 is a difference generation processing performed by the addition/subtraction processing unit 110, and generates the inverse conversion RAW data 1606 and the difference data 1609 of the RAW data 1601 on the memory 107. The difference data 1609 is an output of the difference generation processing 1608 in the addition/subtraction processing unit 110 and is written in the memory 107. The compression processing 1610 is compression processing of the difference data 1609 performed by the compression/expansion processing unit 111. Considering the RAW recoverability, the lossless compression is preferable, but if the data size after compression is prioritized over the RAW recoverability, the lossy compression process may be used. The HEIF file 1605 indicates a HEIF file. The compressed data after the HEVC compression processing 1604 of the main image YUV data 1603 and the compressed data after the compression processing 1610 of the difference data 1609 are formed as a HEIF file and recorded in the recording unit 108. .. Also, the inverse conversion parameter used in the inverse conversion processing 1607 of the lossless image processing is added to the HEIF file 1605 and recorded. The HEIF file 1605 recorded in FIG. 16A can be applied to the post-process RAW data 243 of the process flow shown in FIG.

Next, FIG. 16(B) is a processing flow for restoring the RAW from the HEIF file 1605 recorded in FIG. 16(A). The HEIF file 1621 is a recording file recorded by the recording unit 108 at the time of shooting, and corresponds to the HEIF file 1605 in FIG. The HEVC-compressed main image YUV compressed data is read from the HEIF file 1621, the compression/expansion processing unit 111 performs HEVC expansion processing 1622, and the expanded main image YUV data 1623 is output. The decompressed main image YUV data 1623 is written in the memory 107.

On the other hand, the compressed difference data is read from the HEIF file 1621, the compression/expansion processing unit 111 performs the expansion processing 1628, and the expansion difference data 1629 is output. The expansion difference data 1629 is written in the memory 107. Next, the decompressed main image YUV data 1623 is subjected to inverse conversion processing 1624 of reversible image processing. The inverse conversion parameter at this time is read from the HEIF file 1621 and used. The output of the inverse conversion processing 1624 of the lossless image processing is the inverse conversion RAW data 1625, which is written in the memory 107. Next, the inversely converted RAW data 1625 and the decompression difference data 1629 are read from the memory 107, the addition/subtraction processing unit 110 performs addition processing 224, and the restored RAW data 1627 is output. The output restored RAW data 1627 is written in the memory 107. The restored RAW data 1627 is obtained by restoring the RAW data 1601 at the time of shooting.

In this way, by performing the inverse conversion processing 1607 of the lossless image processing instead of the inverse conversion processing of the lossy image processing 1602, it is possible to reduce the processing load as much as possible, shorten the processing time, and reduce the memory used. Further, the number of inverse conversion parameters of the inverse conversion processing 1607 of the lossless image processing is the minimum number of parameters as compared with the parameters required for the inverse conversion processing of the lossy image processing 1602. Therefore, it is possible to reduce the processing load, the processing time, and the memory used for the parameter inverse conversion processing 1613. Further, the size of the parameter stored in the HEIF file 1605 can be minimized.

FIG. 17 is a diagram illustrating a timing chart of the processing flow shown in FIG. The V signal 1701 represents the timing when the still image shooting was performed. The present embodiment shows that high-speed continuous shooting is performed and continuous shooting is performed at the timing of the V signal 1701.

As shown in FIG. 17, the acquisition of RAW data 1601 by imaging, the irreversible image processing 1602, and the HEVC compression processing 1604 sequentially process the captured images in parallel. Next, the HEVC decompression 1612 waits for the HEVC compression processing 1604 and performs it. The HEVC decompression 1612, the reverse conversion processing 1607 of the lossless image processing, the difference generation processing 1608, the compression processing 1610, and the recording of the HEIF file 1605 in the recording unit 108 are sequentially started after the HEVC decompression 1612 is started. Regarding the lossy image processing, HEVC compression or HEVC decompression, lossless image reverse conversion processing, difference generation, and difference data compression, the timing chart is such that processing is performed before the processing of one image is completed.

The period T3 is the processing time of the entire series of processing from shooting to recording the HEIF file 1605. It can be seen that T3 takes a longer time than the period T1 shown in the timing chart of FIG. That is, in the processing flow shown in FIG. 16, the lifetime of the memory buffer required for each processing becomes longer than that in the processing flow shown in FIG. 2A, and the used memory amount tends to increase.

On the other hand, in the processing flow shown in FIG. 16, the difference data 1609 is acquired by the difference generation processing 1608, not the main image YUV data 1603, but the difference between the direct RAW data 1601 and the inverse conversion RAW data 1606. Therefore, when the restoration RAW is generated by the processing flow shown in FIG. 16B, the error of the restoration process is suppressed, and as a result, the quality of the restoration RAW tends to be improved.

In the present embodiment, the processing flow targeted for switching between “with expansion” and “without expansion” described with reference to FIG. 15 of the first embodiment is the processing flow of FIG. 16(A) and FIG. 2(A). The processing flow is as follows. The processing flow for switching the HEIF file to be recorded is switched. Further, in the present embodiment, regarding the RAW restoration processing from the HEIF file, the processing flow is shown in FIG. 16(B) in the case of “with decompression”, and is shown in FIG. 2(B) in the case of “without decompression”. The process flow shown is performed.

As described above, in the present embodiment, the case has been described in which the method of switching between the method having a high RAW recoverability based on the difference from the inverse conversion RAW depending on the condition. Therefore, according to the present embodiment, it is possible to perform the RAW-restorable recording file generation process in each shooting mode at the optimum processing speed and quality.

Although the preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof. A program that implements one or more functions of the above-described embodiments is supplied to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus also read and execute the program. It is possible. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

102 image sensor 104 irreversible image processing unit 108 recording unit 109 reversible image processing unit 110 addition/subtraction processing unit 111 compression/expansion processing unit

Claims (19)

First image processing means for performing irreversible image processing on the image data generated by the imaging means,
A first compression unit that compresses the image data that has been subjected to the irreversible image processing by the first image processing unit;
Second image processing means for performing reversible image processing on the image data,
Generating means for generating difference data between the image data after the lossy image processing and the image data after the lossless image processing;
A second compression means for compressing the difference data generated by the generation means,
An image processing apparatus comprising: image data compressed by the first compression means and recording means for recording the image data compressed by the second compression means. The image processing apparatus according to claim 1, wherein the first compression unit performs HEVC compression processing as compression processing. The data format of the image data compressed by the first compression means and the image data compressed by the second compression means to be recorded in the recording means is a HEIF format. 2. Image processing device. 4. The first image processing means, as the irreversible image processing, performs a development process including at least one of a noise removal process and a demosaic process according to a feature amount of an image feature. The image processing device according to claim 1. The second image processing means in the first half performs at least one of demosaic processing, gamma processing, and matrix conversion processing as the reversible image processing. Image processing device. The image processing apparatus according to claim 1, wherein the reversible image processing parameter is determined based on the irreversible image processing parameter. 7. The recording means records the reversible image processing parameter together with the image data compressed by the first compression means and the image data compressed by the second compression means. The image processing device according to. When the data format of the image data after the lossy image processing is different from the data format of the image data after the lossless image processing, the image data after the lossy image processing is changed to the image data after the lossless image processing. The image processing apparatus according to claim 1, wherein the differential data is generated after conversion to the data format of. 9. The image processing apparatus according to claim 8, wherein the data format of the image data is a sampling ratio of a color component with respect to luminance and an effective bit number. Further comprising a decompression unit for decompressing the image data compressed by the first compression unit,
10. The image according to claim 1, wherein the generation unit generates second difference data between the decompressed image data and the reversible image-processed image data. Processing equipment. The image data decompressed by the decompression means is further inversely transformed,
The image processing apparatus according to claim 10, wherein the inverse conversion unit performs the inverse conversion using a parameter obtained by inversely converting the parameter of the reversible image processing. Further comprising a switching means for switching the difference data and the second difference data,
11. The image processing apparatus according to claim 10, wherein the switching unit switches between the difference data and the second difference data according to a shooting mode. 13. The image processing apparatus according to claim 12, wherein the switching unit switches to the difference data in the high speed continuous shooting mode, and switches to the second difference data in the low speed continuous shooting mode. Image data compressed by the first compression means, and decompression means for decompressing the image data compressed by the second compression means,
Image data decompressed by the first compression means, and addition means for adding image data decompressed by the second compression means,
Third image processing means for performing image processing, which is inverse conversion of the reversible image processing by the second image processing means, on the image data generated by the adding means,
The image processing apparatus according to claim 1, further comprising: Fourth image processing means for performing irreversible image processing on the image data generated by the third image processing means, which is different from the irreversible image processing performed by the first image processing means. The image processing apparatus according to claim 14, further comprising: The irreversible image processing performed by the fourth image processing means is a higher-load image processing than the irreversible image processing performed by the first image processing means. 15. The image processing device according to item 15. The imaging means,
An image pickup apparatus comprising: the image processing apparatus according to claim 1. A first image processing step of performing irreversible image processing on the image data generated by the imaging means,
A first compression step of compressing the image data subjected to the irreversible image processing by the first image processing step;
A second image processing step of performing reversible image processing on the image data,
A generation step of generating difference data between the image data after the lossy image processing and the image data after the lossless image processing;
A second compression step of compressing the difference data generated by the generation step;
A control method for an image processing apparatus, comprising: a recording step of recording the image data compressed by the first compression means and the image data compressed by the second compression means. A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.

Images