JP2011029809A - Image processing device, image processing method, and image capturing device - Google Patents

Abstract

In an image processing apparatus that compresses RAW data in a data format in which a plurality of types of color components constituting a color image are repeatedly arranged on a pixel array according to a certain rule, an increase in circuit scale and CPU processing capability The compression efficiency of a plurality of color-specific plane data is further increased without causing an increase.
RAW data reconstructing means 3 receives RAW data 2 that has not been subjected to signal processing after A / D conversion, is decomposed for each color component, and reassembles to obtain a plurality of color-specific plane data. 41 to 44 are generated and further arranged and collected as one file which is a unit of compression processing. One file, which is a unit of compression processing, has a plurality of arrangement areas partitioned for each color component. The RAW data reconstruction unit 3 allocates a plurality of color-specific plane data to a plurality of arrangement areas and rearranges them to generate the reconstructed RAW data 4 and passes it to the compression processing unit 5. The compression processing means 5 compresses the reconstructed RAW data 4.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus and an image processing method for compressing RAW data in a data format in which a plurality of types of color components constituting a color image are repeatedly arranged on a pixel array in accordance with a certain rule. The present invention relates to a technique for improving the compression efficiency of color-specific plane data. The present invention also relates to an image pickup apparatus in which the image processing apparatus as described above is mounted and an image sensor of a type that performs color separation and picks up an image. Examples of the target image processing apparatus and imaging apparatus include a digital still camera, a digital video camera, an independent image scanner, and an image scanner incorporated in a copying machine. “Plain data” refers to data having an array form that is two-dimensionally developed when data is developed on an image memory.

  A CCD-type or MOS-type image sensor for obtaining a color image is a color filter array in which color filters of a plurality of types of color components are repeatedly arranged according to a certain rule corresponding to the two-dimensional array of pixels in the image sensor. Color separation filter). For example, in a Bayer color separation filter in which RGB primary color filters are arranged in a checkered pattern corresponding to the pixels of the image sensor, a filter for each color component of RGB along both the horizontal and vertical directions. Are arranged every other pixel (BGgR). The information obtained from each pixel is only information of one color component. Therefore, in order to enhance the expressive power, each pixel is interpolated using the color information of the surrounding pixels, and information on a plurality of color components is obtained for each pixel. That is, information on all color components is obtained for all pixels. This is called synchronized color interpolation processing. Thus, color image data of each color component having the same number of pixels as that of the image sensor is obtained.

  Furthermore, signal processing such as white balance (WB) adjustment, gamma correction processing, enhancement processing for edge enhancement, etc. is performed and converted into component signals of a luminance signal (Y) and two types of color difference signals (Cr, Cb). Then, the image is compressed by a compression encoding algorithm such as JPEG (Joint Photographic Experts Group) to obtain a small file size and recorded on a recording medium.

  In recent years, as a function of a single-lens digital camera, raw (RAW) digital data (without image processing) immediately after A / D conversion of an imaging analog signal is performed for high-quality development processing and retouch processing performed after shooting. RAW data) is reversibly compressed and recorded on a recording medium. Image reproduction (development) processing is performed by an external device such as a personal computer, thereby realizing high-quality printing and image editing that matches the user's purpose. RAW data without image processing is used because if image data with image processing added is recorded and if it is decoded for development processing or retouch processing and subjected to further image processing, the image quality deteriorates. It is.

  Patent Document 1 discloses a common compression processing step for two modes, a mode for recording after performing the synchronized color interpolation process and a mode for recording in the RAW data state without performing the synchronized color interpolation process. A method of compressing independently is described (see FIGS. 13A and 13B).

  Further, Patent Document 2 describes a method of compressing component data separated for each color component from RAW data for each component data without going through a synchronized color interpolation process (see FIGS. 14 and 15). ).

  In any of these conventional techniques, RAW data is divided into color-specific plane data with high correlation between adjacent pixels, and the plane data is compressed. Therefore, the RAW data is higher than the case where RAW data not separated by color is compressed as it is. It is said that compression efficiency can be obtained.

Japanese Patent No. 3864748 Japanese Patent No. 3956360

  In the conventional technology, a plurality of color plane data obtained by dividing RAW data is individually compressed. That is, the same compression process is repeatedly performed in four times. It takes time to switch the plane data during this repetition. Specifically, switching from the compression process of the first plane data to the compression process of the second plane data, switching from the compression process of the second plane data to the compression process of the third plane data, There are three times of switching, such as switching from the compression processing of the plane data to the compression processing of the fourth plane data. In this way, if a plurality of pieces of plane data for each color are individually compressed sequentially before and after each other, it takes a lot of time in total. Therefore, the compression efficiency is low.

  Patent Document 2 also describes that a plurality of compression processing steps that can be operated in parallel are used (see paragraph [0032]). However, in that case, a CPU (Central Processing Unit) having a very high processing capability is required. In addition, when the hardware configuration is adopted, the circuit scale is significantly increased. If the compression efficiency is low, it is disadvantageous for high-speed processing in the compressed RAW data recording mode in response to the recent increase in pixels of image sensors.

  Note that an imaging apparatus having a continuous shooting function may hinder high-speed continuous shooting in the compressed RAW data recording mode. In addition, the possibility of occurrence of block noise is increased.

  The present invention was created in view of such circumstances, and it is intended to increase the compression efficiency of a plurality of color-specific plane data, and to avoid increasing the circuit scale and CPU processing capacity. The purpose is that. More desirably, the generation of block noise is desired to be suppressed.

  The present invention solves the above problems by taking the following measures.

  An image processing apparatus according to the present invention is a RAW data (image data immediately after A / D conversion, in which signal processing is performed) in which a plurality of types of color components constituting a color image are repeatedly arranged on a pixel array according to a certain rule. RAW data reconstructing means and compression processing means configured as follows.

  The RAW data reconstruction means inputs the RAW data that has not been subjected to signal processing after the analog signal of the image is A / D converted, and processes it as follows. The input RAW data is decomposed for each color component and reassembled to generate a plurality of color-specific plane data. For example, it is assumed that the RAW data is obtained by repeatedly arranging the first to fourth color components on the pixel array according to a certain rule. The RAW data is divided into a first color component, a second color component, a third color component, and a fourth color component, and the first color-specific plane data in which only the first color component is collected, and only the second color component. The second color-specific plane data that collects only the third color component, and the fourth color-specific plane data that collects only the fourth color component. The plane data is data having an array form that is expanded two-dimensionally when expanded on the image memory. In addition, for example, the case where the two color-specific plane data are the same color as in the Bayer array (BGgR) is also included.

  The RAW data reconstruction means arranges the plurality of color-specific plane data and collects them as one file which is a compression processing unit. Here, it is assumed that one file, which is a compression processing unit, has a plurality of arrangement areas divided for each color component. The RAW data reconstruction unit distributes and arranges the plurality of color-specific plane data in a plurality of arrangement areas divided for each color component, and generates reconstructed RAW data. For example, when the above example is drawn, the first color-specific plane data is arranged in the first arrangement area, the second color-specific plane data is arranged in the second arrangement area, and the third arrangement area is in the first arrangement area. 3 color-specific plane data is arranged, and the fourth color-specific plane data is arranged in the fourth arrangement area to generate reconstructed RAW data collected as one file as a compression processing unit. Then, the generated reconstructed RAW data is transferred to the compression processing means.

  The compression processing means inputs the reconstructed RAW data of the compression processing unit generated by the RAW data restructuring means and performs the compression process. The relative positional relationship between the first to fourth arrangement areas is arbitrary.

(1) In summary, the image processing apparatus of the present invention is an apparatus that compresses RAW data in a data format in which a plurality of types of color components constituting a color image are repeatedly arranged on a pixel array according to a certain rule. And
The RAW data is input, the RAW data is separated for each color component and reassembled to generate a plurality of color-specific plane data, and further, the plurality of color-specific data in a plurality of arrangement areas divided for each color component. RAW data restructuring means for generating reconstructed RAW data arranged as a single file as a compression processing unit by arranging plain data;
And compression processing means for inputting and compressing the reconstructed RAW data of the compression processing unit generated by the RAW data restructuring means.

The image processing method of the present invention is a method of compressing RAW data in a data format in which a plurality of types of color components constituting a color image are repeatedly arranged on a pixel array according to a certain rule,
The RAW data is input, the RAW data is decomposed and reassembled for each color component to generate a plurality of color-specific plane data, and the plurality of color-specific data are arranged in a plurality of arrangement areas divided for each color component. A step of generating reconstructed RAW data arranged as a single file as a compression processing unit by arranging plain data;
A step of inputting and compressing the reconstructed RAW data of the compression processing unit generated by the step of generating the reconstructed RAW data.

  According to the image processing apparatus and the image processing method of the present invention configured as described above, the following operational effects are exhibited.

  By providing the RAW data reconstruction means, the compression processing target of the compression processing means is “reconstructed RAW data in which a plurality of color plane data are arranged and collected as one file as a compression processing unit”. The plane data for each color has a high correlation between adjacent pixels, and the compression process provides a higher compression efficiency than the case where raw data with different color components are adjacent is compressed as it is. This is also recognized in the prior art as described above. In the present invention, not only that, but also reconstructed RAW data configured by arranging a plurality of types of color-specific plane data on one file which is a unit of compression processing is targeted for compression processing. Therefore, a plurality of color-specific plane data on the reconstructed RAW data can be compressed at once, and higher compression efficiency can be obtained. Specifically, since the compression processing of the plane data for each color component by color is realized by one compression processing in one file (reconstructed RAW data) that is a unit of compression processing, a plurality of processing is performed as in the case of the prior art. Compared with the case of repeatedly performing compression processing while sequentially switching the color-specific plane data individually, the compression efficiency is greatly improved.

  Further, in order to increase the compression efficiency, it is not necessary to use a plurality of compression processing means operating in parallel as the compression processing means, and a single compression processing means is required, so that the circuit scale does not increase. Alternatively, there is no need to particularly increase the processing capacity of the CPU.

  An image pickup apparatus according to the present invention includes an image pickup unit that converts an optical image input by an image sensor that performs color separation and picks up an image into an analog electric signal, and further converts it into digital RAW data, and the image processing device described above. It is equipped with.

  According to the present invention, the RAW data reconstructing means is provided, and the reconstructed RAW data configured by arranging a plurality of types of color-specific plane data having different color components on one file as a compression processing unit is compressed. By implementing the compression processing of the color-specific plane data for all the color components in one compression process for one file that is the compression processing unit, while switching the plurality of color-specific plane data individually and sequentially Compared with the case of repeatedly performing compression processing, the compression efficiency can be greatly improved.

  In addition, it is not necessary to use multiple compression processing means operating in parallel to improve compression efficiency, and a single compression processing means is required, so there is no need to increase the circuit scale, and the CPU processing capacity is special. There is no need to reinforce.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus that compresses reconstructed RAW data generated from RAW data according to an embodiment of the present invention. FIG. 4 is an exemplary diagram of raw data to be compressed (a), an exemplary diagram (b) in which four color plane data are adjacently arranged in a two-dimensional direction, and four color plane data according to an embodiment of the present invention. Illustrations in which are arranged at appropriate intervals from each other Explanatory drawing which shows the reconstructed RAW data which reconfigure | reconstructed as image data for 1 frame by arrange | positioning four-color plane data of a Bayer arrangement two-dimensionally in connection with embodiment of this invention (the 1) 3 is an explanatory diagram of a block noise occurrence position in image data when decompressing the compressed RAW data in FIG. 3 according to the embodiment of the present invention. Explanatory drawing which shows the reconstructed RAW data which reconfigure | reconstructed as image data for 1 frame by arrange | positioning four color plane data of a Bayer arrangement | sequence two-dimensionally in connection with embodiment of this invention (the 2) Explanatory drawing of the block noise generation position in the image data when decompressing the compressed RAW data in FIG. 5 according to the embodiment of the present invention FIG. 5A shows a block noise corresponding to the embodiment of the present invention, and FIG. 5B shows a block noise corresponding to FIGS. In relation to the embodiment of the present invention, an explanatory diagram (a) representing the block noise occurrence position in FIG. 6 corresponding to FIG. 7 (a) and a block noise occurrence position in FIG. 4 corresponding to FIG. 7 (b). Illustration (b) Explanatory drawing which shows the other form of the four independent arrangement | positioning area | regions in connection with embodiment of this invention corresponding to four plane data according to four colors 1 is a block diagram illustrating a schematic configuration of an imaging apparatus including an image processing apparatus that compresses reconstructed RAW data generated from RAW data according to an embodiment of the present invention. Explanatory drawing which shows the example of the color filter arrangement | sequence of a primary color type in connection with the Example of this invention Description of Effective Use of Resources Using a General Compression Processing Means Comprising a Luminance Signal (Y) and Color Difference Signal (Cr / Cb) Input Terminals as Component Data Input Terminals in the Embodiment of the Present Invention Figure Explanatory drawing which shows the characteristic of the image processing apparatus shown in patent document 1 of a prior art Explanatory drawing which shows the characteristic of the image processing method shown in patent document 2 of a prior art Explanatory drawing which shows the characteristic of the image processing process shown to patent document 2 of a prior art

  The image processing apparatus and image processing method of the present invention having the configuration (1) described above can be further advantageously developed in the following embodiment.

  (2) In the image processing apparatus configured as described in (1) above, the RAW data reconstructing unit includes a compression processing unit block of the compression processing unit when the plurality of color plane data are arranged in the one file. There is an aspect in which the reference position of each color plane data is shifted by a predetermined number of pixels in both the vertical and horizontal directions with respect to the periodic boundary position. As an image processing method corresponding to this image processing apparatus, in the image processing method having the configuration (1), the step of generating the reconstructed RAW data includes arranging the plurality of color-specific plane data in the one file. In doing so, there is a mode in which the reference position of each color plane data is shifted by a predetermined number of pixels in both the vertical and horizontal directions with respect to the periodic boundary position of the compression processing unit block in the compression processing means.

  (3) Also, in the image processing apparatus having the configuration of (1), the RAW data reconstructing unit is configured to use the compression processing unit block in the compression processing unit when arranging the plurality of color-specific plane data in the one file. With respect to the periodic boundary position, the reference position of each color plane data is arranged so as to be shifted by a predetermined pixel in either one of the vertical direction and the horizontal direction. As an image processing method corresponding to this image processing apparatus, in the image processing method having the configuration (1), the step of generating the reconstructed RAW data includes arranging the plurality of color-specific plane data in the one file. In this case, the reference position of each color plane data is shifted by a predetermined number of pixels in either the vertical direction or the horizontal direction with respect to the periodic boundary position of the compression processing unit block in the compression processing means. There is.

  When arranging a plurality of plane data for each color, if the reference position of each plane data for each color is set to the same position with respect to the periodic boundary position of the compression processing unit block in the compression processing means, the compression processing is compressed. If decompression processing is performed for reproduction in the case of irreversible compression with a high rate, the block noise generation position of each color corresponding to the boundary position of the compression processing unit block in the reconstructed RAW data for one frame obtained. Overlapping characteristic image quality degradation may occur, and block noise may increase.

  On the other hand, if each color plane data is shifted relative to the periodic boundary position of the compression processing unit block configured as in (2) and (3) above, the block of each color It is possible to suppress image quality degradation peculiar to where noise generation positions overlap and to improve image quality.

  (4) In the image processing apparatus having the configuration of (1), the RAW data reconstruction unit arranges fixed data in a non-image area between arrangement areas of the plurality of color-specific plane data, and There is an aspect of being configured to generate reconstructed RAW data. As an image processing method corresponding to this image processing apparatus, in the image processing method configured as described in (1) above, the step of generating the reconstructed RAW data is performed between the plurality of color-specific plane data arrangement regions. There is an aspect in which fixed data is arranged in the image area and the reconstructed RAW data is generated. The fixed data is pixel data having no luminance change.

  With this configuration, when arranging a plurality of color-specific plane data, a non-image area composed of fixed data can be formed between the color-specific plane data arrangement areas. If the non-image area does not exist and the color-specific plane data are arranged in contact with each other, when the compression processing is repeated for each compression processing unit block, the same one compression processing unit block is stored. In some cases, the color component of the end part of the plane data for each color in the horizontal direction or the vertical direction of the plane data for each color is mixed with the color component of the head part of the adjacent plane data for each color. When the compression process is repeated for each compression processing unit block due to the presence of the non-image area, the color component of the end portion of a certain color plane data and the color component of the head portion of the adjacent color-specific plane data are mixed. Escape. As a result, in the compression process for the reconstructed RAW data, it is possible to perform the compression process in a state where the plurality of color-specific plane data are not confused with each other. Since the intervening data is fixed data having no luminance change, the adjacent difference is zero, and this also advantageously increases the compression efficiency.

  (5) In the image processing apparatus having the configuration of (1), the compression processing unit includes a plurality of component input terminals, the reconstructed RAW data input from one of the component input terminals, and the other component. There is an aspect in which fixed data input from an input terminal is compressed simultaneously. As an image processing method corresponding to this image processing apparatus, in the image processing method having the configuration (1), the compression processing step inputs the reconstructed RAW data and the fixed data in different systems, and There is an aspect in which the reconstructed RAW data and the fixed data are simultaneously compressed.

  With this configuration, it is possible to further increase the compression efficiency by simultaneously compressing the reconstructed RAW data and the fixed data. In JPEG and the like, a compression processing unit having a luminance signal (Y) and a color difference signal (Cr / Cb) input terminal as component input terminals is used. However, the resources of the compression processing means in a general form can be effectively used.

  (6) In the image processing apparatus having the configuration of (1), the compression processing unit includes a plurality of component input terminals, the reconstructed RAW data input from one of the component input terminals, and the other component. There is an aspect in which another image data input from the input terminal is compressed simultaneously. As an image processing method corresponding to this image processing apparatus, in the image processing method having the configuration of (1), the compression processing step inputs the reconstructed RAW data and another image data in different systems. There is a mode in which the reconstructed RAW data and the other image data are simultaneously compressed.

  With this configuration, the reconstructed RAW data and another image data (for example, small YCrCb data or small RGB data that has been reduced and resized for display) are simultaneously compressed to reduce the size for display. It is possible to further increase the compression efficiency including another image such as an image. Also in this case, the resources of the compression processing means in a general form having component input terminals for luminance signals (Y) and color difference signals (Cr / Cb) such as JPEG can be effectively used.

  (7) In the image processing apparatus having the configuration of (1), the compression processing unit may be configured to irreversibly compress the reconstructed RAW data. As an image processing method corresponding to this image processing apparatus, in the image processing method having the configuration (1), the compression processing step includes irreversible compression of the reconstructed RAW data. JPEG, MPEG (Moving Picture Experts Group), H.M. The lossy compression such as H.264 is a compression process with a higher compression rate than the lossless compression. Therefore, compressed RAW data with a smaller file size can be obtained.

  Note that the image processing apparatus or the image processing method having the configuration (1) may be arbitrarily combined with any of a plurality of items (2) to (7) under a condition that does not contradict each other. Good.

  The image processing method of the present invention may be configured as a single application software, or may be incorporated as part of an application such as image processing software or file management software.

  The image processing program according to the image processing method of the present invention is not limited to being applied to a computer system such as a personal computer, and the operation of a central processing unit (CPU) incorporated in an information device such as a digital camera or a cellular phone. It can also be applied as a program.

  The outline of the present invention has been described above.

  Next, the case of the Bayer arrangement will be described as an example, and the description will be made more easily.

  FIG. 1 is a block diagram illustrating a schematic configuration of an image processing apparatus that generates reconstructed RAW data from RAW data acquired by an imaging unit having a color filter with a Bayer array and compresses the reconstructed RAW data.

  In FIG. 1, reference numeral 1 denotes an image pickup unit (image sensor) having a color filter in which a plurality of types of color components constituting a color image are repeatedly arranged on a pixel array according to a certain rule. Here, the color filter has four types of color components B (blue component), G (first green component), g (second green component), and R (red component) having a Bayer array. Suppose there is. Reference numeral 2 denotes RAW data in a data format based on a Bayer array having four color components B, G, g, and R acquired by the imaging unit 1. 3 inputs RAW data 2, RAW data 2 is decomposed into color components B, G, g, and R and reassembled two-dimensionally to generate four color-specific plane data 41, 42, 43, 44. Further, the four color-specific plane data 41, 42, 43, and 44 are arranged in four independent arrangement areas a1, a2, a3, and a4 divided for each color component to form one file as a compression processing unit. This is a RAW data reconstruction means for generating the combined reconstruction RAW data 4. Reference numeral 5 denotes compression processing means for inputting the reconstructed RAW data 4 of the compression processing unit generated by the RAW data reconstructing means 3 as luminance data and performing irreversible compression processing.

  RAW data 2 output from the imaging unit 1 and input to the RAW data reconstruction unit 3 is image data immediately after A / D conversion of an imaging analog signal in the imaging unit 1, and is a synchronized color interpolation process and a gamma correction process. The image data is not subjected to signal processing such as white balance adjustment. The RAW data 2 is mosaic image data in which only one color information different for each pixel corresponding to the color filter array pattern is held.

  The RAW data reconstruction unit 3 inputs the RAW data 2 that has not been subjected to signal processing, decomposes the input RAW data 2 for each of the color components, reassembles them, and reassembles the four color-specific plane data 41 and 42. , 43, 44 are generated. That is, the RAW data 2 is obtained by repeatedly arranging the first to fourth color components on the pixel array in accordance with a certain rule. The RAW data 2 is composed of the first color component (B) and the second color component (G ), First color-specific plane data 41 that is decomposed into the third color component (g) and the fourth color component (R) and collects only the first color component (B), and the second color component (G) Second color plane data 42 that collects only the third color plane data 43 that collects only the third color component (g), and a fourth color that collects only the fourth color component (R) Another plane data 44 is generated. The four color components in the Bayer array are the first color component (B), the second color component (G), the third color component (g), and the fourth color component (R), and the second color component. Both (G) and the third color component (g) are green, but these two are treated as independent plane data by color.

  The RAW data reconstruction unit 3 further arranges the four color-specific plane data 41, 42, 43, and 44 in a two-dimensional manner and collects them as one file that is a compression processing unit. Here, one file as a compression processing unit has four arrangement areas a1, a2, a3, and a4 divided for each color component. The RAW data reconstruction unit 3 distributes and arranges the four color-specific plane data 41, 42, 43, and 44 in the four arrangement areas a1, a2, a3, and a4 divided for each color component, and reconstructs the data. RAW data 4 is generated. That is, the first color-specific plane data 41 is arranged in the first arrangement area a1 corresponding to the color component B (blue), and the second arrangement area a2 corresponding to the color component G (first green). 2 color-specific plane data 42 is arranged, the third color-specific plane data 43 is arranged in the third arrangement area a3 corresponding to the color component g (second green), and it corresponds to the color component R (red). The fourth color-specific plane data 44 is arranged in the fourth arrangement area a4 to be generated, and the reconstructed RAW data 4 collected as one file as a compression processing unit is generated. Then, the generated reconstructed RAW data 4 is transferred to the compression processing means 5.

  The compression processing means 5 inputs the reconstructed RAW data 4 of the compression processing unit generated by the RAW data reconstruction means 3 and performs compression processing. The compression process may be either lossy compression (lossy encoding) or lossless compression (lossless encoding).

  The relative positional relationship between the first to fourth arrangement regions a1 to a4 is arbitrary. Here, it is arranged two-dimensionally in two horizontal and vertical directions, but as described later, it may be arranged one-dimensionally along the horizontal direction, or one-dimensionally arranged along the vertical direction. It is also possible (see FIG. 9).

  According to the image processing apparatus of the present embodiment configured as described above, the following operational effects are exhibited.

  In the RAW data 2, the second color component (G) is adjacent to the right side of the first color component (B), and the first color component (B) is adjacent to the right side of the second color component (G). There is a fourth color component (R) adjacent to the right side of the third color component (g), and there is a third color component (g) adjacent to the right side of the fourth color component (R). In addition, there is a third color component (g) adjacent below the first color component (B), and there is a first color component (B) adjacent below the third color component (g). The fourth color component (R) is adjacent to the lower side of the second color component (G), and the second color component (G) is adjacent to the lower side of the fourth color component (R). However, the pixel values of these adjacent pixels are not so highly correlated. Therefore, when the RAW data 2 is compressed as it is, the compression efficiency is low (equivalent to the prior art).

  In the embodiment of the present invention, by providing the RAW data reconstructing means 3, the compression processing object of the compression processing means 5 is placed in the “4 color-specific plane data 41, 42, 43, 44 and compressed processing” The reconstructed RAW data 4 "is collected as a single unit file. As described above, the color-specific plane data has a high correlation between adjacent pixels, and the compression process can obtain a higher compression efficiency than the case where the raw data 2 is compressed as it is.

  Further, in the present embodiment, the reconstructed RAW data 4 configured by arranging four color-specific plane data 41, 42, 43, and 44 having different color components on one file as a compression processing unit is compressed. Since it is an object of processing, higher compression efficiency can be obtained. Specifically, since the compression processing of the color-specific plane data 41, 42, 43, 44 of all the color components is realized by one compression processing in one file (reconstructed RAW data 4) that is a compression processing unit. Compared with the case where the four color-specific plane data 41, 42, 43, and 44 are individually switched sequentially and repeatedly compressed as in the case of the prior art, the compression efficiency is greatly improved.

  Further, as the compression processing means 5, a compression control software asset of a conventional method created so as to complete the compression processing for image data for one frame, which is standardly mounted on a digital camera, in one process. It can also be used effectively. Alternatively, existing JPEG hardware processing can also be supported. That is, it is not necessary to use four compression processing means operating in parallel to complete the compression processing of image data for one frame including the four color-specific plane data 41, 42, 43, and 44 in one process. Since only one compression processing means 5 is required, there is no need to increase the circuit scale. Alternatively, there is no need to particularly increase the processing capacity of the CPU.

  Next, generation of block noise will be described.

  FIG. 2A illustrates RAW data 2 to be compressed. FIG. 2B shows the four color-specific plane data 41, 42, 43, 44 arranged adjacent to each other in a two-dimensional direction. FIG. 2 (c) shows four color-specific plane data 41, 42, 43, 44 arranged at appropriate intervals. In the case of the embodiment of the present invention, FIG. 2 (c) corresponds to FIG. 2 (b). In the case of FIG. 2C, fixed data (indicated in gray) whose luminance level is not changed is arranged in the intermediate part and the peripheral part to form a non-image area.

  Let m, n, x, and y be arbitrary (two or more) natural numbers. The size of the RAW data 2 to be compressed is 2 m pixels in the horizontal direction and 2n pixels in the vertical direction.

  The size of the reconstructed RAW data 4 that is one compressed file is larger than that of the RAW data 2. It is 2 (m + x) pixels in the horizontal direction and 2 (n + y) pixels in the vertical direction. The four arrangement regions a1, a2, a3, and a4 divided for each color component have the same size, and are (m + x) pixels in the horizontal direction and (n + y) pixels in the vertical direction. The sizes of the four color-specific plane data 41, 42, 43, and 44 are common and are m pixels in the horizontal direction and n pixels in the vertical direction.

  FIG. 3 shows a reconstructed RAW in which the four color plane data 41, 42, 43, and 44 in the Bayer array are two-dimensionally arranged and reconstructed as image data for one frame by the RAW data reconstructing means 3 in FIG. Data 4 is shown.

  In FIG. 3, 41 is first color-specific plane data obtained by collecting only the first color components (B) arranged at the upper left part of the image, and 42 is the second color component (G) arranged at the upper right part of the image. 2nd color plane data that collects only the color, 43 is the third color plane data that collects only the third color component (g) arranged in the lower left part of the image, 44 is arranged in the lower right part of the image The fourth color-specific plane data is a collection of only the fourth color component (R).

  45 is a periodic boundary position (extending in the vertical direction) in the horizontal direction of the compression processing unit block when the compression processing means 5 performs processing, and 46 is a compression processing unit when the compression processing means 5 performs processing. A periodic boundary position in the vertical direction of the block (extending in the horizontal direction) is shown. In the drawing, reference numeral 47 shown in gray is fixed data for filling between adjacent boundary portions of the four color-specific plane data 41, 42, 43, 44. The fixed data 47 is pixel data that has no luminance signal component, only a luminance signal component, and the luminance signal component is constant. The size of the compression processing unit block is 4 pixels × 4 pixels (equivalent to H.264) here for simplification, but in the case of JPEG, it is 8 pixels × 8 pixels.

  On the reconstructed RAW data 4 that is one compressed file, four color-specific plane data 41, 42, 43, and 44 are two-dimensionally arranged at regular intervals. The reason why the plane data for each color is arranged with a space therebetween is as follows. The compression process is repeatedly performed on a block unit of 4 pixels × 4 pixels or 8 pixels × 8 pixels. If four color plane data are arranged adjacent to each other, some pixel data of the last block of the first color plane data 41 in a certain block row depending on the data size of the color plane data. And some pixel data of the first block of the second color plane data 42 may be included in one compression processing unit block, and the third color plane data 43 and the fourth color The same applies to the other plane data 44. When compression processing is performed in such a state, pixel data having different color components are mixed in the same compression processing unit block, and thus the compressed data is deteriorated. . Further, even in the vertical direction, some pixel data of the last block of the first color-specific plane data 41 and some pixel data of the first block of the third color-specific plane data 43 are 1 in a certain block row. The same block may be included in one compression processing unit block, and the same applies to the second color-specific plane data 42 and the fourth color-specific plane data 44. When compression processing is performed in this state, the same compression is performed. Since pixel data having different color components is mixed in the processing unit block, the data after compression is deteriorated.

  In order to avoid this inconvenience, in order to make sure that only one piece of pixel data of the same color component enters one compression processing unit block, the color-specific plane data are arranged spaced apart from each other and adjacent to each other. An appropriate number of fixed data 47 whose luminance signal component is constant is arranged between the color-specific plane data. By doing so, it is possible to prevent deterioration of data after compression.

  In the compression processing of the compression processing unit block, block noise is likely to occur at the boundary positions 45 and 46 when the compressed RAW data is expanded for photo retouch processing or the like. Block noise is likely to occur in DCT with JPEG. It tends to occur as the compression rate increases. It stands out in areas where there is relatively little change in density value. This is shown in FIG. FIG. 4 is an explanatory diagram of block noise occurrence positions in the image data when the compressed RAW data of FIG. 3 is decompressed.

  In the case of FIG. 3, the relative positional relationship in which the first color-specific plane data 41 is arranged in the first arrangement area a1, and the second color-specific plane data 42 is arranged in the second arrangement area a2. Relative positional relationship, the relative positional relationship in which the third color-specific plane data 43 is arranged in the third arrangement region a3, and the fourth color-specific plane data 44 in the fourth arrangement region a4. The relative positional relations are equivalent to each other. The reference position of each arrangement area is the upper left corner of each arrangement area, and the reference position of each color plane data is the upper left corner of each color plane data. The displacement vector V1 of the reference position of the first color plane data 41 with respect to the reference position of the first arrangement area a1 and the displacement of the reference position of the second color plane data 42 with respect to the reference position of the second arrangement area a2 The vector V2, the displacement vector V3 relative to the reference position of the third arrangement area a3 at the reference position of the third color plane data 43, and the fourth arrangement area a4 of the reference position of the fourth color plane data 44 The displacement vectors V4 with respect to the reference position are equivalent to each other and are exactly overlapped with each other by parallel movement along the horizontal direction or the vertical direction.

  However, it has been found that such an equivalent arrangement form tends to cause block noise. Next, the reason will be described. In order to briefly explain the concept, the size of the basic processing block is 4 pixels × 4 pixels.

  Each of the four color-specific plane data 41, 42, 43, 44 includes therein a horizontal period boundary position 45 extending in the vertical direction and a vertical period boundary position 46 extending in the horizontal direction of the compression processing unit block. Appears repeatedly.

  In FIG. 3, the B (blue) pixel data facing the boundary position 45 from the left side in the first color-specific plane data 41 is the second column, the sixth column, the tenth column, etc., and faces the boundary position 45 from the right side. The B (blue) pixel data is in the third, seventh, eleventh columns, etc., and when these pixel data are expanded, the original 2m × 2n large size is obtained as shown in FIG. Expanded in the image data, the third column, the eleventh column, the nineteenth column, the fifth column, the thirteenth column, the twenty-first column, and the like.

  In FIG. 3, B (blue) pixel data facing the boundary position 46 from the upper side in the first color-specific plane data 41 is the second line, the sixth line, the tenth line, and the like, and the boundary position 46 from the lower side. The pixel data of B (blue) facing the first line is the third line, the seventh line, the eleventh line, etc., which are expanded in FIG. 4 and the third line, the eleventh line, the nineteenth line, etc. , Line 13 and line 21.

  In FIG. 3, the G (first green) pixel data facing the boundary position 45 from the left side in the second color-specific plane data 42 is also in the second column, the sixth column, the tenth column, and the like on the right side. Similarly, the G (first green) pixel data facing the boundary position 45 is also in the third column, the seventh column, the eleventh column, etc., and these are expanded in FIG. , 20th column, 6th column, 14th column, 22nd column and the like.

  In FIG. 3, the pixel data of G (first green) facing the boundary position 46 from the upper side in the first color-specific plane data 41 is also the second row, the sixth row, the tenth row, etc. The G (first green) pixel data facing the boundary position 46 from the lower side is the third row, the seventh row, the eleventh row, and the like as described above, and these are expanded in FIG. Lines 11, 11, 19, etc., lines 5, 13, 21, etc.

  Further, in FIG. 3, the relationship between the g (second green) pixel data and the boundary positions 45 and 46 in the third color-specific plane data 43 is the same. In FIG. The 11th column, the 19th column, etc., the 5th column, the 13th column, the 21st column, etc., and the 4th row, the 12th row, the 20th row, etc., the 6th row, the 14th row, etc. , Line 22 and so on.

  Further, in FIG. 3, the relationship of the R (red) pixel data with respect to the boundary positions 45 and 46 in the fourth color-specific plane data 44 is the same. In FIG. 20th line, 6th line, 14th line, 22nd line, etc. 4th line, 12th line, 20th line, etc., 6th line, 14th line, 22nd line, etc. It has become eyes.

  As described above, the boundary positions at which block noise is likely to occur are equivalent to the four color-specific plane data 41, 42, 43, and 44, and are continuous in the vertical direction and the horizontal direction in the decompressed image data. Become. That is, in the original large-sized image data after the decompression process, pixels corresponding to the boundary positions regularly appear every fourth column and every fourth row. As a result, block noise tends to appear remarkably, and there is a possibility that image quality will be deteriorated (see FIG. 7B).

  The problem of block noise generation described above is that, in order to increase the compression efficiency, a plurality of types of color-specific plane data having different color components are arranged in order on one file as a compression processing unit (displacement with respect to each reference position). This is because the reconstructed RAW data configured by arranging the vectors V1 to V4 (equivalently to each other) is the target of the compression process, and the compression process for the image data for one frame is completed in one process. .

  Therefore, in the present embodiment, the following countermeasure is taken as a more preferable aspect. The data compression here is irreversible compression with a high compression rate.

  As shown in FIG. 5, the relative positional relationship in which the first color-specific plane data 41 is arranged in the first arrangement area a1, and the second color-specific plane data 42 is arranged in the second arrangement area a2. Relative positional relationship, the third color plane data 43 is arranged in the third arrangement area a3, and the fourth color plane data 44 is the fourth arrangement area a4. The relative positional relationship arranged at is not equivalent to each other. That is, the four relative positional relationships are shifted from each other. The shift is one pixel in the right direction and one pixel in the downward direction (when the compression processing unit block is 4 pixels × 4 pixels).

  The displacement vector V1 of the reference position of the first color plane data 41 with respect to the reference position of the first arrangement area a1 and the displacement of the reference position of the second color plane data 42 with respect to the reference position of the second arrangement area a2 The vector V2, the displacement vector V3 relative to the reference position of the third arrangement area a3 at the reference position of the third color plane data 43, and the fourth arrangement area a4 of the reference position of the fourth color plane data 44 The displacement vector V4 with respect to the reference position is not equivalent to each other. V2 = V1 × 2, V3 = V1 × 3, and V4 = V1 × 4.

  The second color-specific plane data 42 is shifted by one pixel in the lower right direction with respect to the first color-specific plane data 41, and the third color-specific plane data 43 is different from the second color-specific plane data 42. The fourth color-specific plane data 44 is shifted by one pixel in the lower right direction with respect to the third color-specific plane data 43.

  FIG. 6 shows the original large size image data after the decompression process corresponding to FIG. The relationship of FIG. 4 to FIG. 3 and the relationship of FIG. 6 to FIG.

  In FIG. 5, the B (blue) pixel data facing the boundary position 45 from the left side in the first color-specific plane data 41 is the third column, the seventh column, the eleventh column, etc., and faces the boundary position 45 from the right side. The pixel data of B (blue) is the fourth column, the eighth column, the twelfth column, etc., and these are expanded in FIG. 6, and the fifth column, the thirteenth column, the twenty-first column, etc. The columns are the 23rd column and the like.

  In FIG. 5, B (blue) pixel data facing the boundary position 46 from the upper side in the first color-specific plane data 41 is the third line, the seventh line, the eleventh line, etc., and the boundary position 46 from the lower side. The B (blue) pixel data facing the first line is the fourth line, the eighth line, the twelfth line, etc., which are expanded in FIG. 6, and the fifth line, the 13th line, the 21st line, etc. 15th and 23rd lines.

  In FIG. 5, the G (first green) pixel data facing the boundary position 45 from the left side in the second color-specific plane data 42 is the second column, the sixth column, the tenth column, etc., and the boundary position from the right side. The pixel data of G (first green) facing 45 is the third column, the seventh column, the eleventh column, etc., which are developed in FIG. 6, and the fourth column, the twelfth column, the twentieth column, etc. , 6th row, 14th row, 22nd row and so on.

  In FIG. 5, the G (first green) pixel data facing the boundary position 46 from the upper side in the second color-specific plane data 42 is the second row, the sixth row, the tenth row, etc., from the lower side. The pixel data of G (first green) facing the boundary position 46 is the third row, the seventh row, the eleventh row, etc., which are expanded in FIG. The fifth line, the 13th line, the 21st line, etc.

  In FIG. 5, the pixel data of g (second green) facing the boundary position 45 from the left side in the third color-specific plane data 43 is the first column, the fifth column, the ninth column, etc., and the boundary position from the right side The pixel data of g (second green) facing 45 is the second column, the sixth column, the tenth column, etc., and these are expanded in FIG. 6, and the first column, the ninth column, the 17th column, etc. The third, eleventh, and nineteenth columns are provided.

  In FIG. 5, the pixel data of g (second green) facing the boundary position 46 from the upper side in the third color-specific plane data 43 is the first row, the fifth row, the ninth row, etc., from the lower side. The pixel data of g (second green) facing the boundary position 46 is the second row, the sixth row, the tenth row, etc., which are expanded in FIG. 6, and the second row, the tenth row, and the 18th row. The fourth line, the twelfth line, the twentieth line, and the like.

  In FIG. 5, the R (red) pixel data facing the boundary position 45 from the left side in the fourth color-specific plane data 44 is the fourth, eighth, and twelfth columns, and faces the boundary position 45 from the right side. The pixel data of R (red) is the 1st column, the 5th column, the 9th column, etc., and these are expanded in FIG. 6, and the 8th column, the 16th column, the 24th column, etc. The columns are the 18th column and the like.

  In FIG. 5, the R (red) pixel data facing the boundary position 46 from the upper side in the fourth color-specific plane data 44 is the fourth line, the eighth line, the twelfth line, etc., and the boundary position 46 from the lower side. The R (red) pixel data facing the first line is the first line, the fifth line, the ninth line, etc., which are expanded in FIG. 6, and the second line, the eighth line, the 16th line, the 24th line, etc. The 10th line, the 18th line, and the like.

  Block noise may occur in the reconstructed RAW data for one frame after decoding that has been subjected to the decompression process due to image quality deterioration in which the image becomes discontinuous at the periodic boundary positions of the compression processing unit blocks.

  In FIG. 6, the pattern of 8 pixels × 8 pixels as described above repeatedly appears along the two-dimensional direction. Comparing FIG. 4 and FIG. 6, in FIG. 4, the pixel group facing the boundary position is aligned in a straight line in both the horizontal direction and the vertical direction, whereas in FIG. 6, it faces the boundary position. The pixel group is in a dispersed arrangement state both in the horizontal direction and in the vertical direction. That is, in the development of FIG. 4, positions where block noise is likely to occur are in a concentrated arrangement state, whereas in the development of FIG. 6, positions where block noise is likely to occur are in a distributed arrangement state.

  In the case of the method of FIGS. 3 and 4 where the block noise generation positions are continuous in the horizontal direction and the vertical direction, when the compressed RAW data is decompressed, the block noise is noticeable as shown in FIG. 7B. On the other hand, in the case of the method of FIGS. 5 and 6 in which the generation positions of block noise are dispersed in the horizontal direction and the vertical direction, when the compressed RAW data is expanded, the block noise is generated as shown in FIG. Not very noticeable.

  The block noise occurrence position in FIG. 4 corresponding to FIG. 7B is applied to FIG. 7B in FIG. 8B, and the block noise occurrence position in FIG. 6 corresponding to FIG. FIG. 8 (a) is applied to FIG. 7 (a). In order to clarify the difference between FIG. 7A and FIG. 7B, FIGS. 8A and 8B are shown as a reference. In FIGS. 8A and 8B, the background is made black in order to make the block noise occurrence position (white portion) clear. In the case of FIG. 8B, the block noise occurrence positions are concentrated in a straight line, and the degree of concentration is extremely high. As a result, as shown in FIG. 7B, image quality degradation that causes discontinuity of the image occurs at periodic boundary positions of the compression processing unit block, and noticeable image quality degradation occurs. On the other hand, in the case of FIG. 8A, the block noise occurrence positions are widely dispersed two-dimensionally, and the degree of concentration is extremely low. As a result, as shown in FIG. 7A, although the image quality is deteriorated, it is not so noticeable.

  In the above case where the size of the compression processing unit block is 4 pixels × 4 pixels, the four color-specific plane data 41, 42, 43, 44 are sequentially shifted by one pixel in the horizontal direction and one pixel in the vertical direction. When the compression processing unit block size is JPEG of 8 pixels × 8 pixels, the four color-specific plane data 41, 42, 43, and 44 may be sequentially shifted by 2 pixels in the horizontal direction and 2 pixels in the vertical direction.

  As shown in FIG. 5, the size of the compression processing unit block is 4 pixels × 4 pixels, and the four color-specific plane data 41, 42, 43, 44 are sequentially shifted by 1 pixel in the horizontal direction and 1 pixel in the vertical direction. Yes. When the size of the compression processing unit block is 8 pixels × 8 pixels JPEG, the boundary positions 45 of the horizontal period extending in the vertical direction appear every 8 columns, and the boundary positions 46 of the vertical period extending in the horizontal direction are 8 It will appear every other line. To shift the position where block noise is likely to occur, the four color-specific plane data 41, 42, 43, and 44 may be sequentially shifted by two pixels in the horizontal direction and two pixels in the vertical direction. By doing this, in the original large size image data (Bayer array) after the decompression process, the positions where the block noises of each color are prevented from overlapping and the positions where the block noises are likely to be generated can be distributed widely. Image quality deterioration can be suppressed.

  Generally, when the size of the compression processing unit block is a pixel × a pixel, the four color-specific plane data may be sequentially shifted in the horizontal direction a / 4 pixel and the vertical direction a / 4 pixel. Here, a is a multiple of 4 and a = 4, 8, 12, 16,.

  Regarding the RAW data reconstruction means 3, the arrangement relationship of the four color-specific plane data 41, 42, 43, 44 in the image data for one frame is a two-dimensional array arranged in both the horizontal direction and the vertical direction as described above. In addition, as shown in FIG. 9, a vertical one-dimensional array arranged in one column in the vertical direction or a horizontal one-dimensional array arranged in one column in the horizontal direction may be used.

  The order in which the four color-specific plane data 41, 42, 43, and 44 are arranged is not particularly limited, and the arrangement information can be arranged in an arbitrary place by using the history of the arrangement information as data additional information after data compression. it can. In the [upper left, upper right, lower left, lower right] array, [B, G, g, R] as shown in FIG. 1, [G, B, R, g] may be used, and [R, G , G, B].

  In the case of a vertical one-dimensional array arranged in a line in the vertical direction, in order to reduce the overlap of block noise occurrence positions, fixed data 47 is placed between the upper and lower adjacent areas of the four color-specific plane data 41, 42, 43, 44. What is necessary is just to insert each color plane data relatively with respect to the periodic boundary position of the compression processing unit block in the vertical direction. Although it is a two-dimensional development, it is better to shift the pixels in the horizontal direction. Even though it is a two-dimensional development, the arrangement of the arrangement regions a1, a2, a3, and a4 is only one-dimensional along the vertical direction.

  In addition, in the case of a horizontal one-dimensional array arranged in a row in the horizontal direction, in order to reduce the overlap of block noise occurrence positions, fixed data between the left and right adjacent four-color plane data 41, 42, 43, 44 47 and the plane data for each color may be shifted relative to the periodic boundary position of the compression processing unit block in the horizontal direction. Although it is a two-dimensional development, it is better to shift the pixels in the vertical direction. Even though it is a two-dimensional development, the arrangement of the arrangement regions a1, a2, a3, and a4 is only one-dimensional along the horizontal direction.

[Example]
Hereinafter, preferred embodiments of the image processing apparatus and the image processing method according to the present invention will be described in detail.

  FIG. 10 is a block diagram showing a schematic configuration of an imaging apparatus equipped with an image processing apparatus that compresses reconstructed RAW data generated from RAW data according to an embodiment of the present invention. This image pickup apparatus 50 is an image pickup apparatus using an image sensor of a type that separates and picks up images, and can record images in the JPEG format and can record RAW data immediately after A / D conversion. It is configured to be.

  In FIG. 10, 50 is an imaging device, 60 is a single-plate imaging unit, and 70 is an image processing device. The imaging device 50 includes an imaging unit 60 and an image processing device 70. The imaging unit 60 includes an optical lens 61, an optical low-pass filter 62, a color filter 63, an imaging element 64, and an analog front end unit 65. The image processing apparatus 70 includes a central processing unit (CPU) 71, a read only memory (ROM) 72, a random access memory (RAM) 73, a preprocessing unit 74, a memory control unit 75, an image memory 76, an image signal processing unit 77, A compression / decompression processing unit (encoder / decoder) 78, a recording media interface unit 79, a display processing unit 80, and a monitor interface unit 81 are included. 91 is an operation panel, and 92 is a recording medium. Hereinafter, the configuration of each unit will be described.

  The image sensor 64 is a CCD type or CMOS type image sensor, and a large number of photodiodes (photosensitive pixels) are two-dimensionally arranged on the light receiving surface thereof. The photodiode photoelectrically converts subject information that has passed through the optical lens 61 and the optical low-pass filter 62. The optical low-pass filter 62 has an action of removing high frequency components higher than the sampling frequency depending on the pixel pitch of the image sensor 64, and prevents aliasing in the final image after image reproduction (signal processing). It is supposed to be. Aliasing means that when a waveform with a frequency component exceeding 1/2 of the sampling frequency is forcibly sampled, a frequency component that should not exist originally appears and is drawn in black letters or lines on a white background. In an image with clear contrast, the characters and lines are crushed and a pattern different from the original image appears. The color filter 63 has a predetermined color arrangement in which any color of R, G, and B exists at a position corresponding to one pixel of the image sensor 64, and the light that enters the photodiode that is the light receiving element. Color selection is performed.

  FIG. 11 shows an example of a primary color filter array. In the Bayer arrangement shown in FIG. 11A, the light receiving elements are arranged in a square matrix at a constant pitch in the row direction and the column direction, respectively. In the honeycomb arrangement shown in FIG. 11B, the centers of the geometric shapes of the light receiving elements (photodiodes) are arranged so as to be shifted by ½ pitch in the row direction and the column direction. On the actual imaging plane of the image sensor 64, the structure of the pixel array is periodically repeated in the horizontal direction and the vertical direction. FIG. 11C shows frequency characteristics in the case of the Bayer array.

  A subject image that passes through the optical lens 61, the optical low-pass filter 62, and the color filter 63 and is imaged on the light receiving surface of the image sensor 64 is converted into signal charges of an amount corresponding to the amount of incident light by each photodiode, and is not shown. Based on the pulse given from the driver circuit, it is sequentially read out as a voltage signal (image signal) corresponding to the signal charge. The image sensor 64 has an electronic shutter function that controls the charge accumulation time (shutter speed) of each photodiode according to the timing of a shutter gate pulse (not shown). The operation (exposure, reading, etc.) of the image sensor 64 is controlled by the CPU 71.

  The analog front end unit 65 performs processing such as analog gain adjustment and CDS (correlated double sampling) on the image signal output from the image sensor 64, and then converts the image signal into a digital signal by the built-in A / D conversion unit. It has a function to convert. The analog front end unit 65 outputs the RAW data immediately after A / D conversion to the preprocessing unit 74 of the image processing apparatus 70.

  The preprocessing unit 74 in the image processing apparatus 70 includes an auto calculation unit that performs calculations necessary for AF and AE control. The preprocessing unit 74 receives the RAW data from the analog front end unit 65, performs the focus evaluation value calculation, the AE calculation, and the like in the auto calculation unit, and transmits the calculation result to the CPU 71. In the mode for recording compressed RAW data, the pre-processing unit 74 performs Bayer array RAW data having four color components B, G, g, and R digitized by the A / D conversion of the analog front end unit 65. Then, the black DC level that is the reference of the data is adjusted.

  The memory control unit 75 in the image processing device 70 includes data and signals between the image memory 76 and the preprocessing unit 74, the image signal processing unit 77, the compression / decompression processing unit 78, the recording media interface unit 79, and the display processing unit 80. Is configured to relay the exchange. The memory control unit 75 incorporates RAW data reconstruction means. The RAW data reconstruction means generates four color-specific plane data 41, 42, 43, and 44, rearranges them so as to be image data for one frame, and writes them in the memory space of the image memory 76. It is configured. The RAW data reconstruction unit in the memory control unit 75 generates four pieces of color-specific plane data 41 and 42 with respect to the periodic boundary position of the compression processing unit block in the compression / decompression processing unit 78 when generating the reconstructed RAW data. , 43, and 44 are configured to be shifted relative to each other.

  A CPU 71 in the image processing device 70 is a control unit that performs overall control of the imaging device 50 in accordance with a predetermined program. Under the cooperation of the ROM 72 and the RAM 73, a preprocessing unit 74, an image signal processing unit 77, and a recording media interface unit. 79 and the operation panel 91 are controlled. The ROM 72 stores programs executed by the CPU 71 and various data necessary for control. The RAM 73 is used as a work area for the CPU 71. The CPU 71 controls the operation of each circuit in the imaging device 50 based on an instruction signal from the operation panel 91. The CPU 71 controls the image pickup unit 60 such as the image pickup device 64 according to various shooting conditions (exposure conditions, presence / absence of strobe light emission, shooting mode, etc.) in accordance with an instruction signal input from the operation panel 91, and automatic exposure (AE). ) Control, automatic focus adjustment (AF) control, auto white balance (AWB) control, lens drive control, image processing control, read / write control of the recording medium 92, and the like.

  The CPU 71 performs automatic focus adjustment (AF) control when detecting half-pressing of the release switch on the operation panel 91, and starts exposure and reading control for capturing a recording image when detecting full-pressing of the release switch. Further, the CPU 71 controls the recording medium interface unit 79 so as to record the captured image data on the recording medium 92 according to the recording mode. Further, the CPU 71 sends a command to a strobe control circuit (not shown) as necessary to control light emission of a flash light emitting tube (light emitting unit) such as a xenon tube.

  The image signal processing unit 77 uses the image memory 76 as a work memory via the memory control unit 75, and performs synchronized color interpolation processing, white balance adjustment, gamma correction, luminance / color difference signal generation, contour enhancement, and electronic zoom function. Are configured as means for performing various processing such as scaling (enlargement / reduction) processing and conversion (resizing) processing of the number of pixels, and configured to process an image signal in accordance with a command from the CPU 71.

  The compression / decompression processing unit 78 reads the image data from the image memory 76 via the memory control unit 75, performs compression processing according to a compression encoding algorithm corresponding to the designated compression format, and sends the compressed RAW data to the memory control unit 75. The image data is stored in the image memory 73. The compression / decompression processing unit 78 performs decompression processing on the compressed RAW data read from the recording medium 92, and stores the RAW data restored by the decompression in the image memory 73 via the memory control unit 75. It is supposed to be. As an algorithm used for compression / decompression processing, there are MPEG and the like in addition to JPEG.

  The recording media interface unit 79 is configured to read image data from the image memory 76 via the memory control unit 75 and transfer it to the recording medium 92 for recording. The image data is read from the recording medium 92 and transferred to the memory control unit 75 for decoding.

  The operation panel 91 is a means for a user to input various instructions to the imaging apparatus 50. For example, a mode selection switch for selecting an operation mode of the imaging apparatus 50, a menu item selection operation (cursor moving operation), and the like. ) And the cross key for inputting instructions such as frame advance / rewind of playback images, execution keys for confirming (registering) selection items and executing operations, and deleting desired objects such as selection items, and canceling instructions Various operation means such as a cancel key, a power switch, a zoom switch, and a release switch.

  As the recording medium 92 for storing image data, various recording media such as a magnetic disk, an optical disk, and a magneto-optical disk can be used in addition to a semiconductor memory represented by a memory card. Further, the recording medium (internal memory) built in the imaging device 50 is not limited to a removable medium.

  Next, the operation of the imaging apparatus 50 of the present embodiment having the above-described configuration will be described.

  The subject image that has passed through the optical lens 61, the optical low-pass filter 62, and the color filter 63 in the imaging unit 60 and is imaged on the light receiving surface of the imaging device 64 is converted into signal charges corresponding to the amount of incident light by each photodiode. Then, it is sequentially read out as a voltage signal (image signal) corresponding to the signal charge based on a pulse given from a driver circuit (not shown), and an image analog signal having a Bayer arrangement of four color components B, G, g, and R is analog. It is sent to the front end portion 65. The image analog signal from the image sensor 64 is digitized by the A / D conversion unit in the analog front end unit 65, and the Bayer array image data is sent to the preprocessing unit 74.

  In the mode for recording the compressed RAW data, the Bayer array image data input to the preprocessing unit 74 is sent to the memory control unit 75 as RAW data after the black DC level as a data reference is adjusted. The memory control unit 75 generates four color-specific plane data 41, 42, 43, and 44 by data rearrangement writing control of the built-in RAW data reconstructing means, and further generates image data for one frame. Arrange and write in the memory space of the image memory 76. This is reconstructed RAW data in units of compression processing. In generating the reconstructed RAW data, the reference positions of the four color-specific plane data 41, 42, 43, and 44 are set relative to the periodic boundary positions of the compression processing unit blocks in the compression / decompression processing unit 78. I try to shift it. This is to prevent the occurrence of block noise occurrence positions of the respective colors from overlapping when the compressed RAW data is read from the recording medium 92 and displayed on the monitor after being decompressed.

  Next, the reconstructed RAW data written in the image memory 76 is input as one component data to the compression / decompression processing unit 78 via the memory control unit 75. At the same time, fixed data that does not change the luminance component is input to the other component data input terminals of the compression / decompression processing unit 78, and compression processing is performed. The processed compressed RAW data is written into the image memory 76 via the memory control unit 75 again. There are four color-specific plane data 41, 42, 43, and 44 in one file that is a unit of compression processing, and they are compressed at once.

  Next, the CPU 71 adds a header file as an image file in the JPEG file format to the data written as one compressed RAW data in the image memory 76, and then format information of the compressed RAW data, the shooting time, and the like. Information useful for searching and recognizing images such as the situation is acquired, added to the image file header, and recorded on the recording medium 92 via the memory control unit 75 and the recording medium interface unit 79.

  When the imaging device 50 confirms playback of an image that has been shot and recorded on the recording medium 92, the compressed file is read from the recording medium 92. That is, when a reproduction instruction is given on the expansion operation panel 91, the CPU 71 controls the compression / expansion processing unit 78 and the recording media interface unit 79. The compressed RAW data read from the recording medium 92 is written into the image memory 73 via the recording medium interface unit 79 and the memory control unit 75. The compressed RAW data written in the image memory 73 is transferred to the compression / decompression processing unit 78 via the memory control unit 75. The decompression processing unit in the compression / decompression processing unit 78 decompresses the compressed RAW data.

  According to this embodiment, the reconstructed RAW data configured by arranging the four color-specific plane data 41, 42, 43, and 44 on one file that is a compression processing unit is the target of the compression processing. The four color-specific plane data 41, 42, 43, 44 can be compressed at once, and the four color-specific plane data 41, 42, 43, 44 are sequentially switched as in the case of the prior art. However, the compression efficiency is greatly improved as compared with the case where the compression process is repeatedly performed.

  This significant improvement in compression efficiency is advantageous when the imaging device 50 has a continuous shooting function, and provides high-speed continuous recording in the compressed RAW data recording mode in response to the recent increase in the number of pixels in an image sensor. It is possible to clear the problem in the prior art that hinders copying (pauses during the continuous shooting operation).

  Further, the compression processing means 5 (compression / decompression processing section 78) does not need to use a plurality of compression processing means operating in parallel, and can be a single compression processing means, so that the circuit scale does not increase, or There is no need to specially increase the processing capacity of the CPU. As shown in FIG. 12 (a), it is possible to use a general compression processing means having an input terminal for luminance signal (Y) and color difference signal (Cr / Cb) as an input terminal for component data. Effective use of resources.

  Supplementary information on effective use of resources. As shown in FIG. 12A, a general JPEG compression processing means 5 (compression processing processor, compression / decompression processing processor) has an input terminal for a luminance signal (Y) and a color difference signal (input) as component data input terminals. Cr, Cb) input terminals. Of these, the reconstructed RAW data generated by the RAW data reconstructing means 4 (memory control unit 75) is input to the input terminal of the luminance signal (Y), and the fixed data having no luminance change is input to the color difference signals (Cr, Cb). If it is configured to input from the input terminal, a general JPEG compression processing means (processor) can be used.

  According to the present embodiment, since the compressed RAW data is recorded on the recording medium 92, there is no influence of the signal processing of the preprocessing unit 74 and the image signal processing unit 77, and the image quality can be kept high. . In addition, since the lossy compression is efficient by JPEG, the compressed RAW data to be generated has a small file size, and the use efficiency of the recording medium 92 is improved. Since the use efficiency of the recording medium 92 is high and the reconstructed RAW data in which the above-described color-specific plane data is arranged in one file as a compression processing unit is the compression processing target, the compression efficiency is greatly improved. When combined with this, high-speed continuous shooting becomes even more advantageous.

  The RAW data reconstructing means in the memory control unit 75 arranges the four color-specific plane data 41, 42, 43, and 44 to generate reconstructed RAW data for each compression processing unit. The reference position of the color-specific plane data is shifted relative to the periodic boundary position of the unit block. Therefore, when the compressed RAW data is read from the recording medium 92, decompressed by the compression / decompression processing unit 78, and displayed on the monitor via the display processing unit 80 and the monitor interface unit 81, the block noise occurrence position of each color is displayed. Overlap can be greatly reduced. Therefore, it is possible to suppress the occurrence of block noise, and to reduce the peculiar image quality degradation in which the occurrence positions of the block noise of each color overlap at the boundary position of the compression processing unit block.

  In the above description, the RAW data rearranged and written as image data for one frame in the image memory 76 is input as one component data to the compression / decompression processing unit 78 via the memory control unit 75, and the other components The input data is compressed as fixed data having no luminance change. However, instead of this, for the purpose of generating a new compressed image file by effectively using one compression process, it is also possible to input other image data to other component inputs and execute the process. is there. As another image data, for example, there is small YCrCb data or small RGB data which is reduced and resized for display.

  For example, as shown in FIG. 12B, the reconstructed RAW data 4 for one frame is inputted to the input terminal of the luminance signal (Y) in the compression processing means 5 and the input terminal of the color difference signal (Cr / Cb) is inputted. The small YCrCb data 6 of the image composed of Y, Cr, and Cb reduced and resized for display, the small RGB data 7 of the image composed of R, G, and B are input. Then, the compression processing means 5 simultaneously compresses the reconstructed RAW data 4, small YCrCb data 6, small RGB data 7, and the like.

  When the captured image is confirmed to be reproduced by the imaging device 50, the compressed file is read from the recording medium 92. At this time, the compression / decompression processing unit 75 displays small YCrCb6 or small RGB7 while performing image signal processing. It is also possible to directly transfer to the processing unit 80 and perform playback preview.

  In the above embodiment, JPEG has been described as an example of the compression encoding algorithm for still images. However, JPEG2000 or JPEG XR capable of supporting the number of bits of input data up to 12 bits exceeding 8 bits may be used. . In these compression encoding algorithms, image quality deterioration due to compression is small compared to JPEG and the like, and image quality deterioration is particularly small at a high compression rate. In addition, lossless compression and lossy compression are possible with the same algorithm. Unlike the JPEG method, the code string deletion process (post quantization) of the encoded data has an advantage that the compression rate can be adjusted without performing recompression.

  If reconstructed RAW data for one frame reconstructed with four color-specific plane data is created and stored in the image memory 76 in advance, the reconstructed RAW data is read from the image memory 76 and compressed. If the reconstructed RAW data has not been created in advance, the reconstructed RAW data for one frame is created in the memory control read process. Either aspect is encompassed by the present invention. In detail, the process of creating the reconstructed RAW data for one frame has been described as the rearrangement process when the RAW data is written to the image memory 76, but the writing process to the image memory 73 is the original Bayer array. The image memory 73 may be read as it is, and may be simultaneously read when it is read out via the memory control unit 75 during the compression process. Any of these modes is included in the present invention.

  As described above, since RAW data is divided into four color plane data and rearranged into one frame of reconstructed RAW data for compression, it is possible to use either lossless compression or lossy compression. Thus, it is possible to efficiently compress and record the RAW data that is not affected by the signal processing inside the imaging apparatus in a single process. In the case of lossless compression, the original RAW data can be completely reproduced by decoding and decompressing the recorded encoded data by the compression / decompression processing unit 78 or an external decoder.

  In the above embodiment, an image sensor having a color separation filter has been described as an example. However, the present invention can also be applied to a case where an image sensor of a type that performs similar color separation by means other than a color filter is used. Is natural.

  Note that the compressed RAW data recorded by the imaging device 50 can be reproduced (developed) by a dedicated image processing device or a personal computer in addition to the imaging device main body. Specifically, using the arrangement information of each color template added to the compressed RAW data, the RAW data rearranged in the plurality of color regions after expansion may be rearranged again into the original RAW data array, When the reproduction (development) process is performed via the memory control unit 75, the original arrangement may be read.

  It is also possible to realize the processing procedure when implementing the present invention and the functions of the means related thereto by a program on a computer such as a personal computer or a microcomputer. Further, part or all of the image processing is not limited to a dedicated hardware (signal processing circuit) mode, and a part of the processing may be realized by a program. A program therefor and various recording (storage) media on which the program is recorded are also included in the present invention. Further, it is natural that the processing method according to such a procedure is also included in the present invention.

  Note that the arrangement structure of the color filter 63 is not limited to the example shown in FIG. 11, and various arrangement structures such as RGB stripes are possible. In this example, the primary color filter is used. However, the present invention is not limited to the primary color filter, and is a complementary color filter composed of yellow (Y), magenta (M), cyan (C), and green (G). Or any combination of primary and complementary colors or white (W) may be used.

  In the image pickup device 64 represented by a CMOS type, a noise processing unit and an A / D conversion unit are mounted in the image pickup device 64 as means for realizing high-speed reading, and output directly as a digital signal from the image pickup device. There is also.

  The present invention relates to an image pickup apparatus equipped with an image sensor of a type that performs color separation such as a digital still camera, a digital video camera, an independent image scanner, an image scanner incorporated in a copying machine, etc. When acquiring RAW data with high data processability in a compressed state, reconstructed RAW data configured by arranging a plurality of types of color-specific plane data on a single file as a compression processing unit is targeted for compression processing. Since the reconstructed RAW data is acquired by one compression process, it is useful as a technique for greatly improving the compression efficiency without causing an increase in circuit scale or CPU processing capacity.

1: Image pickup unit with Bayer color filter (image sensor)
2: RAW data in Bayer array 3: RAW data reconstruction means 4: Reconstructed RAW data for one frame composed of four color-specific plane data 5: Compression processing means 6: Small YCrCb data 7: Small RGB data a1 : First arrangement area corresponding to color component B (blue) a2: second arrangement area corresponding to color component G (first green) a3: third corresponding to color component g (second green) Arrangement area a4: fourth arrangement area corresponding to color component R (red) 41: first color-specific plane data 42: color component G (first) in the first arrangement area corresponding to color component B (blue) 2nd color-specific plane data 43 in the second arrangement area corresponding to the first green) 43: third color-specific plane data 44 in the third arrangement area corresponding to the color component g (second green) 44: color The fourth arrangement region corresponding to the component R (red) Color-specific plane data 45: Periodic boundary position of the compression processing unit block in the horizontal direction 46: Periodic boundary position of the compression processing unit block in the vertical direction 47: Fixed data without luminance change 50: Imaging device 61: Optical lens 62: Optical low-pass filter 63: Color filter 64: Image sensor 65: Analog front end unit 70: Image processing device 71: CPU
72: ROM
73: RAM
74: Pre-processing unit 75: Memory control unit (RAW data reconstruction means)
76: Image memory 77: Image signal processing unit 78: Compression / decompression processing unit (compression processing means)
79: Recording media interface unit 80: Display processing unit 81: Monitor interface unit 91: Operation panel 92: Recording medium

Claims (15)

An image processing apparatus that compresses RAW data in a data format in which a plurality of types of color components constituting a color image are repeatedly arranged on a pixel array according to a certain rule,
The RAW data is input, the RAW data is decomposed and reassembled for each color component to generate a plurality of color-specific plane data, and the plurality of color-specific data are arranged in a plurality of arrangement areas divided for each color component. RAW data restructuring means for generating reconstructed RAW data arranged as a single file as a compression processing unit by arranging plain data;
An image processing apparatus comprising: compression processing means for inputting and compressing the reconstructed RAW data of the compression processing unit generated by the RAW data reconstruction means.   The RAW data reconstructing means, when arranging the plurality of color-specific plane data in the one file, sets a reference for each color-specific plane data with respect to a periodic boundary position of the compression processing unit block in the compression processing means. The image processing apparatus according to claim 1, wherein the image processing apparatus is configured to be shifted in position by a predetermined pixel in both the vertical direction and the horizontal direction.   The RAW data reconstructing means, when arranging the plurality of color-specific plane data in the one file, sets a reference for each color-specific plane data with respect to a periodic boundary position of the compression processing unit block in the compression processing means. The image processing apparatus according to claim 1, wherein the image processing apparatus is configured so that the position is shifted by a predetermined number of pixels in any one of a vertical direction and a horizontal direction.   The RAW data reconstruction means is configured to generate fixed RAW data by arranging fixed data in a non-image area between the arrangement areas of the plurality of color-specific plane data. The image processing apparatus according to 1.   The compression processing means has a plurality of component input terminals and simultaneously compresses the reconstructed RAW data input from one of the component input terminals and the fixed data input from the other component input terminal. The image processing apparatus according to claim 1, configured as described above.   The compression processing unit has a plurality of component input terminals, and simultaneously compresses the reconstructed RAW data input from one of the component input terminals and another image data input from the other component input terminal. The image processing apparatus according to claim 1, configured to process.   The image processing apparatus according to claim 1, wherein the compression processing unit is configured to irreversibly compress the reconstructed RAW data. An image processing method for compressing RAW data in a data format in which a plurality of types of color components constituting a color image are repeatedly arranged on a pixel array according to a certain rule,
The RAW data is input, the RAW data is decomposed and reassembled for each color component to generate a plurality of color-specific plane data, and the plurality of color-specific data are arranged in a plurality of arrangement areas divided for each color component. A step of generating reconstructed RAW data arranged as a single file as a compression processing unit by arranging plain data;
An image processing method comprising: a step of inputting and compressing the reconstructed RAW data of the compression processing unit generated by the step of generating the reconstructed RAW data.   In the step of generating the reconstructed RAW data, when arranging the plurality of color-specific plane data in the one file, each color-specific plane with respect to a periodic boundary position of a compression processing unit block in the compression processing means. The image processing method according to claim 8, wherein the reference position of the data is arranged by being shifted by a predetermined pixel in both the vertical direction and the horizontal direction.   In the step of generating the reconstructed RAW data, when arranging the plurality of color-specific plane data in the one file, each color-specific plane with respect to a periodic boundary position of a compression processing unit block in the compression processing means. The image processing method according to claim 8, wherein the reference position of the data is arranged by being shifted by a predetermined pixel in any one of the vertical direction and the horizontal direction.   9. The step of generating the reconstructed RAW data generates the reconstructed RAW data by arranging fixed data in a non-image area between the arrangement areas of the plurality of color-specific plane data. Image processing method.   The image processing method according to claim 8, wherein the compression processing step inputs the reconstructed RAW data and the fixed data in different systems, and simultaneously compresses the reconstructed RAW data and the fixed data. .   9. The compressing process includes inputting the reconstructed RAW data and different image data in different systems, and simultaneously compressing the reconstructed RAW data and the different image data. Image processing method.   The image processing method according to claim 8, wherein the compression processing step performs irreversible compression on the reconstructed RAW data.   An image pickup unit that converts an optical image input by an image sensor that performs color separation and picks up an image into an analog electrical signal, and further converts it into digital RAW data, and an image according to any one of claims 1 to 7. An imaging apparatus comprising a processing device.

Images