JP2018137642A - Image processing system, image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for realizing color gamut mapping of practical use at high speed, so that the difference of visually prominent degree is reduced at all positions in a picture, while taking account of the input side observation conditions and the output side observation conditions.SOLUTION: A first saliency map mapping the prominent degree of a target pixel in an input image compared with neighborhood pixels is generated based on the input side observation conditions. A second saliency map mapping the prominent degree of the target pixel in the image compared with neighborhood pixels is generated based on the output side observation conditions, for multiple images obtained by performing multiple color gamut mapping. Based on the differential value of the first and second saliency maps, and the multiple images, an image after color gamut mapping is generated.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a color gamut mapping technique.

  2. Description of the Related Art Conventionally, a color management system (color management system) is known as a technique for managing color information between input / output devices such as a digital camera, scanner, monitor, and printer connected to a computer or a computer network. In the color management system, color management such as color matching between input / output devices, out-of-gamut warning, and output preview (soft proof) is performed using a device profile. The device profile stores various device characteristics of the device. Here, various device characteristics include the relationship between device-dependent colors and device-independent colors, device-specific color gamut information, and the color gamut for mapping device-independent colors to device-independent colors within the device-specific color gamut. Mapping etc. are included.

  In order to perform color matching between input / output devices, it is necessary to match the color appearance on the input side device with the color appearance on the output side device. FIG. 1 is a diagram showing the definition of the human visual field in the color appearance model.

  The color appearance model is designed to correctly predict human color appearance when given a color chart with a viewing angle of 2 °. In general, the CIE 1931 colorimetric standard observer is applicable in the range of 1 ° to 4 ° viewing angles, and distinguishes this viewing angle region into stimulation regions 61 within 2 ° and adjacent regions 62 within 4 °. An area outside the adjacent area 62 with a viewing angle of 4 ° to 10 ° is called a background area 63, and an area around the background area 63 is called a surrounding area 64. Furthermore, a visual field area that includes all of the stimulation area 61, the adjacent area 62, the background area 63, and the surrounding area 64 is referred to as an adaptation area. A typical color appearance model is CIECAM02, and this model can set the following observation condition parameters.

L _A : absolute luminance [cd / m ² ] of the adaptation region (usually 20% of the absolute luminance L _w of the white point in the adaptation region)
XYZ: relative XYZ value of the color chart,
X _w Y _w Z _w: relative XYZ values of a white point,
Y _b : relative luminance of the background area,
Ambient conditions: Average Surround, Dim Surround, Dark Surround
The ambient condition is Average if the relative luminance in the surrounding region is 20% or more of the white point in the adaptation region, Dim if it is less than 20%, and Dark if it is almost 0%. Since the color appearance model is derived from the experimental results using a single color chart, it has not been established what observation condition parameters should be applied to an image in which a plurality of colors are mixed. Since neutral gray is 20% of the brightness of the white point, 20% of the absolute brightness L _w of the white point in the adaptation area is generally set as the relative brightness Y _b of the background area.

  In addition, when a color appearance model is applied to an image, one observation condition parameter is generally applied to all pixels. However, if one observation condition parameter is applied to all the pixels, there remains a problem that the visual effect of the single color and the background expressed on the rasterized image cannot be reflected in the color matching result.

  In other words, conventionally, it cannot be said that spatial color matching is performed in consideration of the image resolution and observation distance on the input side and the image resolution and observation distance on the output side. For such a problem, spatial color matching in which the resolution and observation distance of the input image and the resolution and observation distance of the output image are considered has been proposed (Patent Document 1).

  On the other hand, in order to perform common color reproduction using a plurality of devices on the network, a common color gamut of the plurality of devices is generally used. The common color gamut tends to become narrower as the number of devices subject to common color reproduction increases. For this reason, a color gamut mapping that can realize a color reproduction equivalent to that of the original image even in a narrow color gamut is required. In a narrow color gamut, it is not possible to reproduce all the same colors as the original image, so some information must be sacrificed. In the ICC (International Color Consortium) profile, “Perceptual”, “Saturation”, “Relative Colorimetric”, “Absolute Colorimetric” The color gamut mapping based on a plurality of rendering intents such as “)” is stored, and the color gamut mapping selected by the user according to the application can be used. Known gamut mapping methods include a technique for fixing hue and mapping with priority on brightness (sacrificing saturation) and a technique for mapping with priority on saturation (sacrificing brightness). .

  In addition, for narrow color gamuts, it is difficult to reproduce colors that achieve both fidelity and gradation, so there is also a technology that dynamically controls the mapping boundary according to the color distribution of the input image outside the color gamut. Known (Patent Document 2).

  Furthermore, as color gamut mapping considering spatial frequency characteristics, “the square norm of the whole image obtained by applying different filtering processing to the difference image before and after the conversion with the brightness component and the color component” and “the color due to the color difference before and after the conversion” There is also known a technique for optimizing color gamut mapping so as to minimize an evaluation function defined by the sum of “inside / outside determination results” (Patent Document 3).

JP 2010-50744 A JP-A-7-107306 JP-A-10-257295

  However, color gamut mapping based on the prior art often does not consider spatial frequency characteristics (influence of surrounding pixels), and the degree of noticeable pixel noticeability (degree of noticeability compared to the vicinity) is an image before and after color gamut mapping. There was a problem of being different.

  Moreover, even if the spatial frequency characteristics are taken into account, there is a problem that the observation conditions on the input side and the observation conditions on the output side are not considered. There is also a problem that a spatial filter is applied to the difference image twice and many repetitive operations are required to obtain a final color gamut mapping image.

  The present invention has been made in view of such problems, and takes into consideration the observation conditions on the input side and the observation conditions on the output side, and the difference in the degree of visually conspicuousness is reduced at all positions in the image. Thus, a technique for realizing a practical color gamut mapping at high speed is provided.

  One aspect of the present invention is an image processing apparatus that performs color gamut mapping from an input-side color gamut to an output-side color gamut, and is a first that maps the degree to which a pixel of interest in an input image is conspicuous compared to neighboring pixels. For each of a plurality of images obtained by performing each of a plurality of color gamut mappings on the input image, and a first generation unit that generates a saliency map of the input side based on the observation condition on the input side, Second generation means for generating a second saliency map in which the degree of noticeable pixels in the image is conspicuous compared to neighboring pixels is generated based on the output-side observation conditions; and the first saliency And a third generation unit configured to generate an image after color gamut mapping for the input image based on a difference value between the map and the second saliency map and the plurality of images. To do.

  According to the configuration of the present invention, in consideration of the observation conditions on the input side and the observation conditions on the output side, the difference in the degree of visually conspicuousness at all positions in the image is reduced, and it is practical at high speed. Color gamut mapping can be realized.

The figure which shows the definition of the human visual field in a color appearance model. 1 is a block diagram illustrating a hardware configuration example of an image processing apparatus and peripheral devices thereof. The figure which shows the structure of the color matching process in a network environment. The figure which shows an example of a CIE 1931 chromaticity diagram. The block diagram which shows the example of a conceptual diagram of the color matching which considered different observation conditions. The block diagram which shows the function structural example which implement | achieves the forward conversion 504 of spatial CAM. The block diagram which shows the function structural example which implement | achieves spatial GMA508. The block diagram which shows the function structural example which implement | achieves the inverse transformation 512 of spatial CAM. The figure which shows the example of the conceptual diagram of the process of the saliency map production | generation part 720. The figure which shows the difference of hue angle h and Hue quadrature H. The figure of the contrast sensitivity function example of human vision with respect to a lightness component and a color component. The figure of the example which showed the contrast sensitivity function on a two-dimensional frequency plane. The figure of the example which showed the change of the contrast sensitivity function according to adaptation brightness. The flowchart of the process of the color matching which considered different observation conditions. The flowchart which shows the detail of the process in step S205. The flowchart which shows the detail of the process in step S213. The flowchart which shows the detail of a process of step S206 (S209). The figure explaining step S101. The figure explaining step S102.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
First, a hardware configuration example of the image processing apparatus according to the present embodiment and its peripheral devices will be described with reference to the block diagram of FIG. The configuration shown in FIG. 2 is shown as an example of a configuration capable of realizing the color management processing described below, and the hardware configuration example of the image processing apparatus is limited to the configuration shown in FIG. Absent. The image processing device is, for example, a general PC (personal computer) or a tablet terminal device.

  The CPU 101 executes processing using computer programs and data stored in the main memory 102. Thus, the CPU 101 controls the operation of the entire image processing apparatus and executes or controls each process including a color management process described below. The CPU 101 accesses an HDD (hard disk drive) 105 via a system bus 114 such as a PCI (peripheral component interconnect) bus and a serial ATA interface (SATA I / F) 103. The CPU 101 accesses a network 115 such as a local area network (LAN) via the network I / F 104. The CPU 101 communicates with various devices connected to a serial bus interface (I / F) 108 and a serial bus 110 such as USB or IEEE1394. As a result, the CPU 101 acquires, for example, image data subject to color management processing from the image input device 113 including a card reader or outputs the image data to the printer 109 to print an image instructed by the user, for example. . Note that the acquisition source of the image data subject to color management processing is not limited to a specific acquisition source, but may be read from a server on the HDD 105 or the network 115.

  In addition, the CPU 101 displays various processing results on the monitor 107 via the graphic accelerator 106 and inputs user instructions via the keyboard 111 and the mouse 112 connected to the serial bus 110.

  The main memory 102 includes a memory device such as a RAM or a ROM, and provides an area for temporarily storing computer programs and data, a work area used when the CPU 101 executes various processes, and the like. be able to.

  The HDD 105 stores an OS (Operating System) and computer programs and data for causing the CPU 101 to execute each process including a color management process described below. The data stored in the HDD 105 includes data described as known data in the following description, image data to be subjected to color management processing, and the like. Computer programs and data stored in the HDD 105 are appropriately loaded into the RAM in the main memory 102 under the control of the CPU 101 and are processed by the CPU 101.

  The monitor 107 is composed of a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 101 with an image or text. The printer 109 is an example of an image output device capable of printing images and characters. However, instead of the printer 109, for example, an image output device such as a multifunction peripheral may be connected to the serial bus 110.

  The image input device 113 is for inputting image data to the image processing apparatus, and may be the above-described card reader, or may be an imaging apparatus that captures an image, for example.

  The keyboard 111 and the mouse 112 are examples of user interfaces. Note that other user interfaces may be employed in addition to or instead of the keyboard 111 and the mouse 112 as long as the user interface allows the user to input various instructions to the CPU 101.

[Color matching configuration in network environment]
A configuration of color matching processing in a network environment in which a plurality of devices are connected will be described with reference to FIG. Device profiles include conversion-based ICC (International Color Consortium) profiles and colorimetric-value-based WCS (Windows Color System) profiles. Hereinafter, an ICC profile will be described as an example for simplification.

  FIG. 3A shows an example in which a profile optimized individually for each device is applied. Images acquired by input devices such as the digital camera 201, the digital video camera 202, and the scanner 203 are converted into device-independent color spaces such as CIE XYZ and CIE Lab, respectively, by corresponding profiles 204 to 206 for the input device. Next, the image converted into the device-independent color space is mapped into the work color space color gamut using the work color space profile 207 and converted into the work color space. Then, the user edits the image on the work color space or saves the edited image. Further, the image on the work color space is mapped into each monitor color gamut using the profile 207 for the work color space and the profiles 212 to 215 for each monitor device, and is displayed on the monitors 208 to 211 of each user. Similarly, the image on the working color space is mapped into the projector color gamut using the working color space profile 207 and the projector device profile 220, and is output to the projector 216. Similarly, the image on the working color space is mapped into the large format printing device color gamut using the profile 207 for the working color space and the profile 221 for the large format printing device, and is output to the large format printing device 217. Similarly, the image on the working color space is mapped into the electrophotographic device color gamut using the working color space profile 207 and the electrophotographic device profile 222, and is output to the electrophotographic device 218. Similarly, the image on the working color space is mapped into the ink jet device color gamut using the working color space profile 207 and the ink jet device profile 223, and is output to the ink jet device 219.

  Here, color reproduction in each of the above devices (the monitors 208 to 211, the projector 216, the large format printing device 217, the electrophotographic device 218, and the ink jet device 219) is performed on the devices corresponding to the profiles 212 to 215 and 220 to 223. Since it is optimized individually, the color reproduction is optimal for each device. However, since each device has a difference in color gamut, a difference in color gamut mapping, and the like, all color reproduction is different.

  FIG. 3B shows a common display color gamut mapping 224, a common projection color gamut mapping 225, and a common print for the same type of device such as a display device, a projection device, and a printing device before applying individually optimized profiles. This is an example in which the color gamut mapping 226 is applied.

  The method for converting the image acquired by the input device into an image on the work color space is the same as that shown in FIG. An image on the working color space is converted into a device-independent color space by the working color space profile 207, and a common display color gamut mapping created by a known technique as disclosed in JP-A-10-79865 224 is used to map into the common display color gamut of the monitors 208-211. Furthermore, the image mapped into the common display color gamut is mapped using the profiles 212 to 215 for each monitor device and displayed on the monitors 208 to 211 of each user. Similarly, the image on the working color space is mapped using the working color space profile 207, the common projection color gamut mapping 225, and the projector device profile 220, and is output to the projector 216. Similarly, the image on the working color space is mapped using the working color space profile 207, the common printing color gamut mapping 226, and the large printing device profile 221, and is output to the large printing device 217. Similarly, the image on the work color space is mapped using the work color space profile 207, the common print color gamut mapping 226, and the electrophotographic device profile 222, and is output to the electrophotographic device 218. Similarly, the image on the work color space is mapped using the work color space profile 207, the common print color gamut mapping 226, and the ink jet device profile 223, and is output to the ink jet device 219.

  Here, the color reproduction on the monitors 208 to 211 of each user is the same color reproduction because the common display color gamut common to a plurality of devices and the common display color gamut mapping 224 are used. In addition, since the color reproduction in the printing device (large format printing device 217, electrophotographic device 218, ink jet device 219) also uses the common printing color gamut common to a plurality of devices and the common printing color gamut mapping 226, the same color reproduction is achieved. It becomes. However, the color reproduction on the monitors 208 to 211 of each user, the color reproduction on the projector 216, and the color reproduction on the printing device (216 to 219) have different common display color gamut, common projection color gamut, and common print color gamut. Different colors are reproduced among monitors, projectors, and printing devices. Since color reproduction between multiple monitors can be shared, it is suitable for collaborative work. Further, by sharing the color reproduction between the plurality of printing devices, the same content can be simultaneously printed by the plurality of printing devices, and the printing time can be shortened.

  FIG. 3C shows an example in which an all common color gamut mapping 227 common to all devices is applied before applying individually optimized profiles. The method for converting the image acquired by the input device into an image on the work color space is the same as that shown in FIG. The image on the work color space is converted into a device-independent color space by the work color space profile 207, and all the color gamut mappings 227 created by the same technique as in FIG. Mapping is performed within all common color gamuts of the devices (monitors 208 to 211, projector 216, large format printing device 217, electrophotographic device 218, and inkjet device 219). Furthermore, the image mapped into all the common color gamuts is converted using the profiles 212 to 215 and 220 to 223 for each device, and is output to the corresponding device. Here, the color reproduction in each device (the monitors 208 to 211, the projector 216, the large format printing device 217, the electrophotographic device 218, and the ink jet device 219) is an all common color gamut common to all devices and an all common color gamut mapping 227. Because of using the same color reproduction. All devices have the same color reproduction, which is not only suitable for collaborative work, but also saves labor for color calibration.

  Device-independent color space is not limited to CIE XYZ and CIE LAB, but other device-independent colors such as JCh (or JCH) and QMh (or QMH) used in CIE LUV, CIECAM97s and CIECAM02 It may be a color space independent of space / observation conditions. In addition, the plurality of devices used when determining the common display color gamut, common projection color gamut, common print color gamut, and all common color gamuts do not need to be all target devices, and some devices may be used. It does not matter as the target.

  FIG. 4 shows an example of a CIE 1931 chromaticity diagram showing the difference in color gamut of typical standard color spaces employed in various devices. The actual color gamut has a three-dimensional shape including the brightness direction, but the maximum chroma region of each color gamut is shown in this chromaticity diagram.

  For monitors, sRGB color space (RGB system) ideal for Web production / viewing on PC, digital camera image display for general users, Adobe RGB optimal for RGB printing applications and professional digital camera image display There are color spaces (RGB). For projectors, there is a DCI P3 color space (RGB system) that is ideal for digital cinema production. There is also a SWOP color space (CMYK system) for printing.

  As the work color space, the sRGB color space is generally used, but the color space to be used differs depending on the type of business and application. For example, the SWOP color gamut used in the printing industry has a part wider than the sRGB color gamut, so when using the SWOP color space as a working color space for CMYK editing, or the Adobe RGB color space wider than the sRGB space In some cases, RGB editing is used. In the case of digital cinema production, DCI P3 is selected as the work color space. In addition, there are Rec.709 color space (RGB system) for broadcasting, Japan Color color space (CMYK system), JMPA Color color space (CMYK system), Euroscale color space (CMYK system), etc. for printing.

  The advantage of using the working color space is that the color space for image editing can be set independently of the color gamut of the actual input device or display device. Editing the input image on the input device color space without using the work color space has the advantage that conversion errors to the work color space and quantization errors in the color space do not occur, but the content color gamut is input. The problem is that the device color gamut is limited. In addition, when editing the image by converting the input image to the display device color space without using the work color space, there is an advantage that it is not necessary to perform color matching when displaying, and high-speed display is possible. Similarly, there arises a problem that the color gamut of the content is limited to the color gamut of the display device. On the other hand, if the work color space is used, it is possible to edit content images in a color gamut wider than the color gamut of actual input devices and display devices.

  For example, even if you want to create a digital cinema in a work environment where only a monitor equivalent to sRGB or Adobe RGB exists as shown in Fig. 3 (a), if you set the DCI P3 color space as the work color space, Image editing can be performed. As can be seen from FIG. 4, the DCI P3 color space has a wider red and green region than the Adobe RGB color space.

  However, when checking the color of the image being edited by the user, the “color in the color gamut” that can be expressed on the actual display device can be confirmed with the same color as the color in the work color space, but it can be expressed on the actual display device. The “color outside the color gamut” that cannot be confirmed is confirmed by the color after the color gamut mapping. Since the color after the color gamut mapping is different from the color before the color gamut mapping, there arises a problem that it cannot be confirmed with the same color as the color in the work color space. This tendency becomes more conspicuous as the color gamut of the display device becomes narrower. For example, as shown in FIGS. 3B and 3C, the color gamut of the display device is limited to the narrowest color gamut among a plurality of devices. In this case, the common display color gamut is equivalent to sRGB, and the entire common color gamut is equivalent to SWOP. In this way, the common color gamut tends to become narrower as the number of devices subject to common color reproduction increases, so that even in a narrow color gamut, color reproduction equivalent to the original image can be realized. Color gamut mapping is required.

[Color matching considering different viewing conditions]
In color gamut mapping that does not consider the spatial frequency characteristics (influence of surrounding pixels), if the input color is the same, the color is mapped to the same color after color gamut mapping. For this reason, the degree of freedom of color gamut mapping is lowered, and there is a problem that the gradation is lost or the contrast ratio between the target pixel and the peripheral pixels cannot be maintained. Further, when the contrast ratio between the target pixel and the surrounding pixels is lost, there is a problem that the distribution of the degree of attention of the target pixel (the degree of attention compared to the vicinity) is different between the images before and after the color gamut mapping.

  In addition, even if the spatial frequency characteristics are taken into account, the gamut mapping optimization method that minimizes the color difference between the pixels cannot solve the problem that the noticeable pixel is noticeable in the images before and after the gamut mapping, There is a problem that processing time is required because optimization is performed by iterative calculation.

  In order to solve this problem, in this embodiment, a “saliency map” before and after color gamut mapping is used instead of an optimization method that minimizes the color difference of each pixel in consideration of spatial frequency characteristics. The optimization is performed so that the difference value of is minimized. A “saliency map” is a map that expresses the degree of visual conspicuousness (degree of conspicuousness compared to the vicinity) at all positions in the image. By using the saliency map, It is possible to estimate an area in which humans are likely to direct visual attention. In research based on gaze measurement, it is known that there is a high correlation between “region where the gaze is directed” and “region where visual attention is easily directed” on the image. By performing optimization that minimizes the difference between the saliency maps before and after gamut mapping, changes in the distribution of “regions that are easy to draw visual attention” on the image before and after gamut mapping are minimized. Color gamut mapping that matches human perception is possible. Here, the saliency map is a map of the degree that the target pixel is conspicuous compared to neighboring pixels, so the lightness component, color component, and spatial frequency characteristics with respect to the edge direction are also considered at the stage of saliency map generation. The

  Also, in this embodiment, weighting is performed so that the difference value between “a saliency map for an image after multiple gamut mapping” and “a saliency map for an image before gamut mapping” is small rather than an iterative calculation. And merging multiple gamut mapped images by weighted average. This eliminates the need for repetitive calculations, and enables fast and practical color gamut mapping. Here, since images after multiple color gamut mapping are guaranteed to be in the color gamut of the target device, the image to which “gamut mapping considering saliency” after weighted averaging is applied is also the target device In the color gamut.

  In addition, it is necessary to consider an observation condition when obtaining a difference value between “a saliency map for an image after color gamut mapping” and “a saliency map for an image before color gamut mapping”. If the image size (or image resolution) and viewing conditions are the same on the input and output sides, use the same viewing condition parameters and use the “saliency map for images after multiple gamut mapping” and “color” What is necessary is just to produce | generate the saliency map with respect to the image before area | region mapping. However, when the image size (or image resolution) and viewing conditions are different between the input side and the output side, in order to generate a “saliency map for an image after multiple color gamut mapping”, the viewing condition parameter on the output side In order to generate “a saliency map for an image before color gamut mapping”, it is necessary to use observation condition parameters on the input side.

  FIG. 5 shows an example of a conceptual diagram of color matching considering different observation conditions. First, an image on the work color space (or an image on the input side device space) 500 is converted into an image 501 that is device-independent XYZ data on the input side by a work color space profile (or input side profile) 207. Is done.

  Next, the device-independent image 501 on the input side is converted into an image 505 on the JCH space on the input side by a forward conversion 504 of a spatial color appearance model (hereinafter referred to as spatial CAM) (details will be described later). ). Here, J is Lightness, C is Chroma, and H is Hue quadrature. Hues that can be calculated by the forward conversion 504 of the spatial CAM include a hue angle h and a hue quadrature H. FIG. 10 shows the difference between hue angle h and hue quadrature H. R, Y, G, and B are four unique colors of unique red, unique yellow, unique green, and unique blue, and indicate colors that humans perceive as pure colors with no color mixture. A hue angle h (FIG. 10A) is a hue expressed from 0 ° to 360 ° and shows a value close to the hue of the Munsell color system. On the other hand, Hue quadrature H (FIG. 10 (b)) is an NCS table with hues expressed from 0 ° to 400 ° (Red: 0 ° or 400 °, Yellow: 100 °, Green: 200 °, Blue: 300 °). Indicates a value close to the hue of the color system. In general image processing, a hue angle h near the hue of the CIE Lab color system or the Munsell color system is used. In the present embodiment, Hue quadrature H is used in order to use the relationship between the opposite colors in the human visual characteristics of the Redness-Greenness component and the Yellowness-Blueness component. Note that in the spatial CAM forward conversion 504, the following input-side observation condition parameters 502 and 503 can be set.

XwYwZw: Relative XYZ value of the white point on the input side
Surround: Ambient conditions on the input side (Average Surround, Dim Surround, Dark Surround)
Lw: Absolute luminance [cd / m ² ] of the white point in the adaptation area on the input side,
R: Image resolution on the input side [ppi]
D: Observation distance on the input side [m]
Next, the image 505 on the input-side JCH space is color-gamut mapped into the output-side color gamut by a spatial GMA (Gamut Mapping Algorithm) 508 and converted into an image 509 on the output-side JCH space (details). Will be described later). Here, the spatial GMA 508 includes not only the output side GBD (Gamut Boundary Description) 507 that is information indicating the color gamut on the output side, but also the input side GBD (Gamut Boundary Description) 506 that is information indicating the color gamut on the input side. Color gamut mapping can be performed in consideration of the above. Further, as the input side GBD 506, the color gamut of the image 505 on the input side JCH space or the color gamut of the input side device can be used.

  The input side GBD 506 for the color gamut of the image 505 on the input side JCH space can be created by obtaining a three-dimensional convex hull for the JCH values of all the pixels of the image 505 on the input side JCH space. The input side GBD 506 for the color gamut of the input side device can be created, for example, by the following procedure. First, a sampled image is generated by uniformly sampling the device space (RGB, CMYK, etc.) of the input side device. Next, forward conversion of a color appearance model such as CIECAM02 in consideration of the input-side observation conditions is applied to the sampling image in the device space to obtain a sampling image in the JCH space. Next, if a three-dimensional convex hull is obtained for the acquired sampling image of the JCH space, the input side GBD 506 can be created.

  Similarly, the output side GBD 507 for the color gamut of the output side device can also be created by the following procedure. First, a sampling image is generated in which the device space (RGB, CMYK, etc.) of the output device is evenly sampled. Next, forward conversion of a color appearance model such as CIECAM02 in consideration of the output-side observation condition is applied to the sampling image in the device space, and the sampling image in the JCH space is acquired. Next, if a three-dimensional convex hull is obtained for the acquired sampling image of the JCH space, the output side GBD 507 can be created.

  Note that the sampling image is not necessarily composed of colors obtained by dividing the device space evenly, and colors obtained by dividing the device space non-uniformly or colors representing the boundaries of the device space may be used. Further, the method for obtaining the color gamut is not limited to the three-dimensional convex hull, and may be a method that takes into account the concave portion. The color appearance model is not limited to CIECAM02, and may be other color appearance models such as CIECAM97s, Naya model, Hunt model, and RLAB model. In the present embodiment, CIECAM02 will be described as an example for simplification.

  Further, in the spatial GMA 508, optimization is performed so that the difference value of the “saliency map” before and after the color gamut mapping is minimized, so that the observation condition parameter 503 on the input side and the observation condition on the output side Parameter 510 is required.

  The observation condition parameter 503 on the input side in the spatial GMA 508 includes the following information.

Lw: Absolute luminance [cd / m ² ] of the white point in the adaptation area on the input side,
R: Image resolution on the input side [ppi]
D: Observation distance on the input side [m]
Similarly, the observation condition parameter 510 on the output side in the spatial GMA 508 includes the following information.

Lw: absolute luminance [cd / m ² ] of the white point in the adaptation area on the output side,
R: Image resolution on the output side [ppi]
D: Observation distance on the output side [m]
Next, the image 509 in the JCH space on the output side is converted into an image 513 that is device-independent XYZ data on the output side by inverse transformation 512 of the spatial CAM (details will be described later). Here, in the inverse transformation 512 of the spatial CAM, the following viewing-side observation condition parameters 510 and 511 can be set.

XwYwZw: Relative XYZ value of the white point on the output side,
Surround: Output side ambient conditions (Average Surround, Dim Surround, Dark Surround)
Lw: absolute luminance [cd / m ² ] of the white point in the adaptation area on the output side,
R: Image resolution on the output side [ppi]
D: Observation distance on the output side [m]
Finally, the device-independent image 513 on the output side is converted into an image on the large format printing device (or an image on the output side device space) 514 by the profile (or output side profile) 221 for large format printing.

  In FIG. 5, the color gamut mapping to the color gamut of the large format printing device 217 is taken as an example, but the configuration of FIG. 5 can be similarly applied to the case where the color gamut mapping to the color gamut of another device is performed.

[Spatial CAM forward conversion 504]
A functional configuration example of an image processing apparatus that realizes the spatial CAM forward conversion 504 is shown in the block diagram of FIG. Each functional unit shown in FIG. 6 may be implemented by hardware or a computer program. In the following description, the functional unit is described as a main subject of processing. However, when each functional unit shown in FIG. 6 is implemented by a computer program, the CPU 101 executes the computer program corresponding to the functional unit, and the corresponding processing is performed. Will be executed. Further, as described above, the observation condition parameter 503 on the input side includes parameters (resolution R, observation distance D) for determining the region size on the input image based on the viewing angle.

  A low-pass filter (Y LOW-PASS) 607 is a low-pass filter determined from the observation condition parameter 503 on the input side, and performs a low-pass filter process on the image 501 (luminance image data) as XYZ data.

  In the forward conversion 504 of the spatial CAM, luminance image data (input-side low-pass image) output from the low-pass filter 607 and input-side observation condition parameters 502 and 503 are applied to the image 501 to convert the image 501 into image data in the JCH space. The image is converted into an image 505 (JCH data).

  The calculation unit 608 calculates the relative luminance Yb of the input-side background region from the pixel value Y of the pixel in the input-side low-pass image output by the low-pass filter 607 at the same position as the target pixel of the image 501.

The calculation unit 609 calculates the absolute luminance L _A of the input-side adaptation area from the luminance Lw of the white point included in the input-side observation condition parameter 503 and the relative luminance Yb of the input-side background area by the following equation 1. calculate.

The calculation unit 604 uses the absolute luminance L _A obtained by the calculation unit 609 and the adaptation degree factor F corresponding to the ambient condition Surround of the observation condition parameter 502 on the input side shown in the table below, and the following equation 2 To calculate the color adaptation degree factor D.

Calculator 605, the absolute luminance L _A of the input-side adaptation region, by Equation 3 below to calculate the luminance level adaptation factor F _L.

The calculation unit 606 calculates the background induction factor n, the induction factors N _bb and N _cb , and the basic exponent nonlinearity z from the relative luminance Yb of the input-side background region according to the following Equation 4.

  Further, the calculation unit 606 determines the ambient effect c and the color induction Nc as shown in the table below from the ambient condition Surround of the observation condition parameter 502 on the input side.

  The chromatic adaptation conversion unit 601 uses the chromatic adaptation degree factor D for the image 501 to perform forward conversion of CIECAM02 chromatic adaptation conversion corresponding to visual white balance.

Cone response conversion unit 602 uses the luminance level adaptation factor F _L for the converted image by the color adaptation transform unit 601 performs forward conversion of the corresponding CIECAM02 cone response converted into visual receptors.

The perceptual attribute conversion unit 603 uses the background induction factor n, the induction factors N _bb and N _cb , the basic exponent nonlinearity z, the ambient effect c, and the color induction Nc for the image converted by the cone response conversion unit 602. Then, forward conversion of CIECAM02 perceptual attribute conversion corresponding to the visual nerve is performed.

  That is, the target pixel at the pixel position (X, Y, Z) in the image 501 and the white point (Xw, Yw, Zw) of the observation condition parameter 502 on the input side are converted into the chromatic adaptation conversion unit 601 and the cone response conversion unit 602. Are converted into a cone response (Ra ′, Ga ′, Ba ′) of the pixel of interest and a cone response (Raw ′ Gaw ′, Baw ′) of the white point through the forward transformation of the pixel of interest, and the perceptual attribute conversion unit 603 Entered. Then, an image 505 that is JCH data for the pixel of interest is output as a result of forward conversion by the perceptual attribute conversion unit 603.

Note that the intermediate result of the forward conversion calculated from the white point (Xw, Yw, Zw) of the observation condition parameter 502 on the input side is also used by the chromatic adaptation conversion unit 601. In addition, the relative luminance Yb of the input-side background region and the absolute luminance L _A of the input-side adaptation region that are used as intermediate observation condition parameters are not fixed with respect to all the pixels of the image 501, but depend on the values of surrounding pixels. Change.

[Spatial GMA508 considering different observation conditions]
A functional configuration example of an image processing apparatus that realizes the spatial GMA 508 in consideration of different observation conditions is shown in the block diagram of FIG. Each functional unit shown in FIG. 7 may be implemented by hardware or a computer program. In the following description, the functional unit is described as the main subject of processing. However, when each functional unit shown in FIG. 7 is implemented by a computer program, the CPU 101 executes the computer program corresponding to the functional unit to perform the corresponding processing. Will be executed.

  First, an image 505 as JCH data is obtained by a plurality of color gamut mappings 701 to 703 (GMA0, GMA1, and GMA2 respectively) for output side devices, and images 711 to 713 (G0, G1 respectively) that are JCH data after color gamut mapping. , G2). Here, the plurality of color gamut mappings 701 to 703 for the output side device are color gamut mappings on the JCH space in consideration of the input side GBD 506 and the output side GBD 507, for example, based on the rendering intent of the ICC profile. Different color gamut mapping algorithms are used, such as “Perceptual” (GMA0), “Relative Colorimetric” (GMA1), “Saturation” (GMA2). Note that the number of the plurality of color gamut mappings is not limited to three, and two or more color gamut mappings can be applied. As a plurality of color gamut mappings, a plurality of stages of color gamut mapping (hue fixation) from lightness priority to saturation priority may be used.

  Next, the image 505 before color gamut mapping is input to a saliency map generation unit 720 (details will be described later) in consideration of the observation condition parameter 503 on the input side, and converted into a saliency map 730 (S). . Further, the images 711 to 713 after color gamut mapping are input to a saliency map generation unit 720 (details will be described later) in consideration of the viewing condition parameter 510 on the output side, and saliency maps 731 to 733 (S0 and S1 respectively). , S2).

  Next, the saliency map difference calculation unit 799 performs the saliency map 730 (S) for the image 505 before gamut mapping from the saliency maps 731 to 733 (S0, S1, S2) for the images 711 to 713 after gamut mapping. ) Difference images (D0, D1, D2 respectively).

  Next, the optimization unit 740 optimizes the color gamut mapping so that the difference value between the saliency maps before and after the color gamut mapping is minimized. A general formula for obtaining an image 509 (Gx) as JCH data after color gamut mapping considering saliency from a plurality of images 711 to 713 (G0, G1, G2) after color gamut mapping is as follows: Become.

  A (x, y) represents the pixel value at the pixel position (x, y) in the image A. Here, when the pixel values of the difference images (D0, D1, and D2) are zero, an image 509 (Gx) after color gamut mapping considering saliency is as follows.

  Similarly, when the number of the “plurality of color gamut mappings” is two, the image 509 (Gx) after color gamut mapping considering saliency can be obtained by the following equation.

  Here, when the pixel value of the difference image (D0, D1) is zero, the image 509 (Gx) after the color gamut mapping considering the saliency is as follows.

  In the present embodiment, all the images after the plurality of color gamut mapping are used, and the fusion is performed by the weighted average based on the inverse of the absolute value of the difference value of the saliency map. However, multiple color gamut mappings are selected from among the multiple color gamut mappings so that the difference value of the saliency map is smaller than the specified value. You can go.

[Inverse transform of spatial CAM]
FIG. 8 is a block diagram illustrating an example of the functional configuration of an image processing apparatus that implements the spatial CAM inverse transform 512. Each functional unit shown in FIG. 8 may be implemented by hardware or a computer program. In the following description, the functional unit is described as the main subject of processing. However, when each functional unit shown in FIG. 8 is implemented by a computer program, the CPU 101 executes the computer program corresponding to the functional unit to perform the corresponding processing. Will be executed. Note that the viewing condition parameter 510 on the output side includes parameters (resolution R, viewing distance D) for determining the region size on the output image based on the viewing angle.

  The resolution conversion unit 806 converts the image 501 (luminance image data) to the resolution (output side resolution) of the image 513 that is XYZ data based on the resolutions indicated by the input viewing condition parameter 503 and the output viewing condition parameter 510. Convert to luminance image data.

  A low-pass filter (Y LOW-PASS) 607 is a low-pass filter determined from the observation condition parameter 510 on the output side, and performs a low-pass filter process on the luminance image data converted into the output-side resolution by the resolution conversion unit 806. To do.

  The resolution conversion unit 801 generates an image 802 that is JCH data that matches the output-side resolution by enlarging or reducing the image 509 that is JCH data by resolution conversion equivalent to the resolution conversion unit 806. In the spatial CAM inverse transform 512, the image 509 is converted into an image 513 by applying the luminance image data output from the low-pass filter 607 and the output-side observation condition parameters 510 and 511 to the image 509.

  The calculation unit 608 calculates the pixel value of the luminance image data (output-side low-pass image) of the output-side resolution output from the low-pass filter 607 at the same position as the target pixel of the image 802 as JCH data output from the resolution conversion unit 801. The relative luminance Yb of the output side background area is calculated.

The calculation unit 609 calculates the absolute luminance L _A of the output-side adaptation region from the above-described equation (1) from the white point luminance Lw of the output-side observation condition parameter 510 and the relative luminance Yb of the output-side background region.

The calculation unit 606 calculates the background induction factor n, the induction factors N _bb and N _cb , and the basic exponent nonlinearity z from the relative luminance Yb of the output-side background region by the above equation (4). Further, the calculation unit 606 determines the ambient effect c and the color induction Nc from the ambient condition Surround of the output-side observation condition parameter 511 using Table 2 above.

Calculator 605, the absolute luminance L _A of the output side adapting field, calculates a luminance level adaptation factor F _L by the above equation (3).

  The calculation unit 604 calculates the chromatic adaptation degree factor D by the above equation (2) using the adaptation degree factor F corresponding to the ambient condition Surround of the output-side observation condition parameter 511 shown in Table 1 above.

The perceptual attribute conversion unit 803 uses the background induction factor n, the induction factors N _bb and N _cb , the basic exponential nonlinearity z, the ambient effect c, and the color induction Nc for the image 802 to correspond to the visual nerve CIECAM02 Inverse perceptual attribute conversion.

Cone response conversion unit 804 uses the luminance level adaptation factor F _L with respect to the inverse transformed image by perceptual attribute converting unit 803 performs inverse transform of the corresponding CIECAM02 cone response converted into visual receptors.

  The chromatic adaptation conversion unit 805 performs inverse conversion of CIECAM02 chromatic adaptation conversion corresponding to visual white balance using the chromatic adaptation degree factor D for the image inversely converted by the cone response conversion unit 804.

  That is, the image 509 obtained by the spatial CAM forward conversion 504 and the spatial GMA 508 is converted into an output image 802 by the resolution conversion unit 801. Then, the target pixel (J, C, H) of the image 802 passes through the inverse transformation of the perceptual attribute conversion unit 803 and the cone response conversion unit 804, and the cone inverse response (X ′, Y ′, Z ′) and input to the chromatic adaptation conversion unit 805. At this time, the inverse cone response (Xw ′, Yw ′, Zw ′) obtained by the forward conversion of the chromatic adaptation conversion unit 601 from the white point (Xw, Yw, Zw) of the observation condition parameter 511 on the output side is also input. The Then, an image 513 as XYZ data that is output-side device-independent data for the pixel of interest is output as an inverse conversion result of the chromatic adaptation conversion unit 805.

Note that the intermediate result of the forward conversion calculated from the white point (Xw, Yw, Zw) of the observation condition parameter 510 on the output side is also used by the perceptual attribute conversion unit 803. Further, the relative luminance Yb of the output-side background region and the absolute luminance L _A of the output-side adaptation region that are used as intermediate observation condition parameters are not fixed with respect to all the pixels of the output-side image 802, and It changes according to the value.

[Generation of saliency map considering observation conditions]
Hereinafter, generation of the saliency map in consideration of the observation conditions will be described. As examples of generating a saliency map from an input image, Document 1 and Document 2 shown below are known. However, since the conventional saliency map generation is based on the input of an RGB image, it cannot cope with a device-independent image (XYZ data) 501 and an image (JCH data, JCH image) 505 in consideration of observation conditions.

Reference 1… Itti, Koch & Niebur, “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis”, IEEE PAMI, Vol.20, No.11, pp.1254-1259 (1998)
Reference 2: J. Harel, C. Koch, and P. Perona, "Graph-Based Visual Saliency", NIPS, Vol.19, pp.545-552 (2006)
Here, the above-mentioned document 1 relates to a model based on a center-surround structure which is a structure of a receptive field of a retinal ganglion cell, and document 2 is a graph base in which a Markov chain is defined on a visual feature. It concerns the model.

  FIG. 9 shows an example of a conceptual diagram of processing of the saliency map generation unit 720 in consideration of observation conditions. First, the image 505 transformed by the spatial CAM forward transformation 504 is decomposed into lightness components (Lightness) and color components (Redness-Greenness, Yellowness-Blueness) as follows.

  Next, in order to extract visual features, spatial filtering 910 is performed on the image 505 to generate a multi-resolution image 920, and a feature amount map 930 is generated. As spatial characteristics of human vision, it is known that bandpass characteristics are exhibited for lightness components, and lowpass characteristics are exhibited for color components (Redness-Greenness, Yellowness-Blueness).

An example of human visual contrast sensitivity function (Contrast Sensitivity Functions) for the lightness component and the color component is shown in FIG. FIG. 11A shows the contrast sensitivity function for the Lightness component, and shows the characteristics of the bandpass. Here, the horizontal axis represents the spatial frequency [cycles per degree], and the vertical axis represents the contrast sensitivity. At each spatial frequency, areas with higher contrast sensitivity than the contrast sensitivity function indicate areas where the spatial change in brightness cannot be distinguished, and areas with lower contrast sensitivity than the contrast sensitivity can visually distinguish between spatial changes in brightness. Is shown. Similarly, FIG. 11B shows a contrast sensitivity function with respect to the Redness-Greenness component, and FIG. 11C shows a contrast sensitivity function with respect to the Yellowness-Blueness component. The contrast sensitivity function of human vision is known not only to be different for the Lightness component, Redness-Greenness component, and Yellowness-Blueness component, but also to change characteristics with respect to adaptation brightness and observation distance. Yes. For example, Daly's model that expresses the contrast sensitivity for the Lightness component with various parameters, the radial spatial frequency is ρ [cycles per degree], the direction is θ [degrees], the adaptation luminance is l [cd / m ² ], and the image size Where i ² [visual degrees], the observation distance is d [m], and the eccentricity is e.

  Here, P is the absolute peak sensitivity of the contrast sensitivity function. ra, re, and rθ are changes caused by accommodation, eccentricity, and direction, and are expressed by the following equations.

The function S ₁ is an effect of the image size and the adaptation luminance, and is expressed by the following equation.

  An example in which Expression (14) is applied and the contrast sensitivity function on the two-dimensional frequency plane is shown in FIG. FIG. 12B is a diagram obtained by inverting the vertical axis of the two-dimensional frequency plane of FIG. It can be seen that the contrast sensitivity function on the two-dimensional frequency plane shows an anisotropic bandpass characteristic that increases the contrast sensitivity in the horizontal and vertical directions.

  FIG. 13A shows an example in which the expression (14) is applied and changes in the contrast sensitivity function according to the adaptation luminance are shown. It can be seen that the contrast sensitivity increases as the adaptation luminance increases. Similarly, FIG. 13B shows an example in which Expression (14) is applied and changes in the contrast sensitivity function according to the perspective adjustment caused by the observation distance are shown. It can be seen that the contrast sensitivity slightly increases as the observation distance increases.

  Hereinafter, generation of a feature amount map for features of brightness, color, and edge direction will be described. From the spatial characteristics of human vision, a DoG (Difference of Gaussians) filter having a bandpass characteristic is effective for the feature quantity map 931 for lightness, and a Gaussian filter having a lowpass characteristic for the feature quantity map 932 for color. It turns out that it is effective. It can also be seen that the feature map 933 with respect to the edge direction is effective using a Gabor filter and a Difference of Gabors filter in consideration of a center-surround structure. Here, the Gabor filter is defined by the product of a sine wave and a Gaussian function.

  In order to generate the feature amount map 931 for lightness, the following processing is performed. In other words, a Gaussian filter 911 having a different scale is applied to the Lightness component to generate a multiresolution image 921 for the lightness, and a feature map in which a DoG (Difference of Gaussians) filter is applied from the difference between the multiresolution images 921 having different scales. 931 is generated.

  In order to generate the feature map 932 for the color, the following processing is performed. That is, the Gaussian filter 912 having a different scale is applied to the Redness-Greenness component and the Yellowness-Blueness component to generate a multiresolution image 922 for Redness-Greenness and a multiresolution image 922 for Yellowness-Blueness, respectively. Since the feature map 932 for color has low-pass characteristics, the multi-resolution image 922 for Redness-Greenness and the multi-resolution image 922 for Yellowness-Blueness are respectively represented by the feature map 932 for Redness-Greenness and the feature map for Yellowness-Blueness. It is used as it is as 932.

  In order to generate the feature amount map 933 for the edge direction, the following processing is performed. That is, a Gabor filter 913 (θ∈ {0 °, 45 °, 90 °, 135 °}) having a different scale is applied to the Lightness component to generate a multi-resolution image 923 for the edge direction, and a multi-resolution having a different scale. A feature amount map 933 in each direction to which the Difference of Gabors filter is applied is generated from the difference between the images 923.

  Here, in order to generate a feature map based on human visual characteristics, each spatial filter such as DoG filter for Lightness component, Gaussian filter for Redness-Greenness component and Yellowness-Blueness component, Gabor filter for Lightness component, In the frequency domain, each parameter is set so as to have a spatial filter characteristic whose upper limit is a contrast sensitivity function for the Lightness component, the Redness-Greenness component, and the Yellowness-Blueness component. The number of filters for generating each feature map is not limited to one, and may be composed of a plurality of filters. That is, the DoG filter for the Lightness component may use a plurality of DoG filters having different parameters as long as the condition that the contrast sensitivity function for the Lightness component is set as the upper limit is satisfied. Similarly, a plurality of filters having different parameters can be used for the Gaussian filter for the Redness-Greenness component and the Yellowness-Blueness component and the Gabor filter for the Lightness component.

  Next, a feature-specific map 940 is generated from the feature amount map 930 having a different scale. Since the feature map 931 for brightness is composed of a plurality of maps to which DoG filters of different scales are applied, each map is normalized, and an arithmetic average is obtained for each pixel, thereby generating a feature-specific map 941 for brightness. Can do.

  Since the feature amount map 932 (Redness-Greenness and Yellowness-Blueness) for colors is composed of a plurality of maps to which Gaussian filters with different scales are applied, each map is normalized, and an arithmetic average is obtained for each pixel. A feature-specific map 942 for colors can be generated.

  Similarly, the feature amount map 933 (θ∈ {0 °, 45 °, 90 °, 135 °}) with respect to the edge direction is composed of a plurality of maps to which the Difference of Gabors filters having different scales are applied. By normalizing and calculating an arithmetic average for each pixel, a feature-specific map 943 for the edge direction can be generated.

  Next, a saliency map 730 is generated from the feature-specific map 940. Since the feature-specific map 941 for brightness, the feature-specific map 942 for color, and the feature-specific map 943 for edge direction are one map for each feature, a saliency map 730 is generated by calculating the arithmetic average for each pixel. can do. Here, the saliency map 730 is not limited to being generated by the arithmetic average of the feature-specific map, and other methods may be employed. For example, the saliency map 730 may be generated by performing different weighting on the feature-specific map and obtaining a weighted average.

  In this embodiment, an example in which a saliency map is generated from three features of brightness, color, and edge direction has been shown. However, the type of feature is not limited to brightness, color, and edge direction, and the direction of motion , Flicker and the like may be included. Also, regarding the feature map 932 for the color, each feature map was obtained by decomposing the feature into a redness-greenness component and a yellowness-blueness component. However, when a spatial filter is applied, each color component exceeds the achromatic color and is the opposite color. If there is a possibility of becoming a component, the spatial filter may be applied by decomposing the Redness-Greenness component into a Redness component and a Greenness component, and the Yellowness-Blueness component into a Yellowness component and a Blueness component.

[Flow of color matching processing considering different viewing conditions]
A color matching process considering different observation conditions will be described with reference to FIG. 14 showing a flowchart of the process.

  In step S <b> 201, the CPU 101 acquires an image 500 that is a target of color conversion in the main memory 102. In the following, the data acquisition source including the image 500 is not limited to a specific acquisition source, but may be acquired from the HDD 105, acquired from the image input device 113, or externally via the network I / F 104. You may acquire from the apparatus of.

  In step S202, the CPU 101 acquires a profile (source profile) 207 associated with the image 500 and a profile (destination profile) associated with the target device to be output. Here, a device profile, a working color space profile in which a standard color space is used, and the like are designated as the source profile.

  Next, in step S203, the CPU 101 acquires the observation condition parameters 502 and 503 on the input side and the observation condition parameters 510 and 511 on the output side. In step S204, the CPU 101 converts the image 500 into a device-independent image 501 using the profile 207 acquired in step S202.

  In step S205, the CPU 101 converts the device-independent image 501 into the observation condition-independent image 505 (spatial CAM forward conversion 504) in consideration of the input-side observation condition parameters 502 and 503.

  In step S206, the CPU 101 generates a saliency map 730 (S) from the image 505 in consideration of the observation condition parameter 503 on the input side (details will be described later). In step S207, the CPU 101 acquires a plurality of color gamut mappings 701 to 703 for the target device. Here, the plurality of color gamut mappings 701 to 703 for the target device are different color gamut mapping applied to the color gamut of the target device. The number of the plurality of color gamut mappings may be any number as long as it is two or more. For example, “Perceptual” (GMA0), “Relative Colorimetric” (GMA1), “GMA1”, “Relative Colorimetric” stored based on the rendering intent of the ICC profile of the target device A gamut mapping algorithm on the JCH space corresponding to “Saturation” (GMA2) or the like is used.

  In step S208, the CPU 101 selects one of unselected color gamut mappings (GMA0, GMA1, and GMA2) acquired in step S207. The CPU 101 applies the selected color gamut mapping to the image 505 to generate an image after the color gamut mapping.

  In step S209, the CPU 101 generates a saliency map from the image after the color gamut mapping generated in step S208 in consideration of the observation condition parameter 510 on the output side (this is the same processing as step S206, and details will be described later). ).

  In step S210, the CPU 101 calculates a difference image between the saliency map for the image after color gamut mapping and the saliency map 730 for the image 505 before color gamut mapping.

  In step S211, the CPU 101 determines whether all of the plurality of color gamut mappings acquired in step S207 have been selected in step S208. As a result of this determination, if all of the plurality of color gamut mappings acquired in step S207 are selected in step S208, the process proceeds to step S212. If an unselected color gamut mapping remains, the process proceeds to step S212. The process returns to step S208. When the process proceeds to step S212, the saliency maps 731 to 733 (S0, G2) and the images 711 to 713 (G0, G1, G2) after the color gamut mapping and the images 711 to 713 (G0, G1, G2) are processed. S1, S2), difference images (D0, D1, D2) between the saliency maps 731 to 733 for the images 711 to 713 and the saliency map 730 for the image 505 are obtained.

  In step S212, the CPU 101 uses the images 711 to 713 (G0, G1, G2) and the difference images (D0, D1, D2) and the above equations (5) to (7) (or equations (8) to (10). )), The image 509 (Gx) after applying the spatial GMA in consideration of different observation conditions is calculated.

  In step S213, the CPU 101 converts the image 509 after applying the spatial GMA considering different observation conditions into a device-independent image 513 in consideration of the output-side observation condition parameters 510 and 511. In step S214, the CPU 101 applies the profile of the target device, and converts the device-independent image 513 into an image 514 depending on the target device.

[Processing flow of spatial CAM order conversion]
Details of the processing in step S205 will be described with reference to the flowchart of FIG. In step S101, the input image and the distance D of the observer are input from the observation condition parameter 503 on the input side. For example, as shown in FIG. 18, the distance between the monitor screen or printed matter and the observer is 0.4 to 0.7 m, but the user can input an arbitrary distance D. Here, D = 0.5 m is used. Further, as will be described later, the distance between the image and the observer can be set to different values on the input side and the output side.

  In step S102, the CPU 101 calculates an input side region based on the viewing angle. That is, the diameters of the stimulation area 61, the adjacent area 62, and the background area 63 are calculated from the distance D between the input image and the observer. As shown in FIG. 19, it can be considered that the line of sight intersects the surface of the image substantially perpendicularly. Accordingly, the diameters of the stimulation region 61, the adjacent region 62, and the background region 63 are calculated by the following formula. When D = 0.5 [m], Da = 17 [mm], Dp = 35 [mm], and Db = 87 [mm].

  In step S103, the CPU 101 inputs the resolution R [ppi] of the input image from the observation condition parameter 503 on the input side. For example, if the input image is an image displayed on a monitor (hereinafter referred to as a monitor image), R = 72 ppi, and if the input image is a print image, the printer resolution R = 400 ppi. Actually, it depends on the resolution or zoom ratio specified by the application or device driver.

  In step S <b> 104, the CPU 101 calculates the number of pixels on the image corresponding to the diameters of the stimulation area 61, the adjacent area 62, and the background area 63 by the following formula, and inputs the low-pass filter (Y LOW-PASS) 607. Set. That is, the number of pixels related to the passing range of the input-side low-pass filter 607 is calculated.

  In the case of D = 0.5 m and R = 72 ppi, Dap = 48.2 pixels, Dpp = 99.0 pixels, and Dbp = 246.6 pixels. Here, for simplification, the diameter pixel number Dbp of the background region 63 is The function f (x, y) of the low-pass filter 607 on the input side is dynamically determined so as to be equal to 2σ in the following expression. Although 2σ = Dbp, σ varies depending on the distance D and the resolution R, and different values can be set on the input side and the output side.

  In step S105, the CPU 101 inputs observation condition parameters 502 and 503 on the input side. As the observation condition parameter on the input side, an instruction value of a measuring instrument (not shown) connected to the serial bus I / F 108 may be input, or the user may input using the keyboard 111 or the mouse 112.

The input observation condition parameters are the XYZ values (Xw, Yw, Zw) of the white point of the adaptation area, the absolute brightness Lw [cd / m ² ] of the white point of the adaptation area, and the absolute brightness Ls [ cd / m ² ]. The white point of the adaptation area is a monitor white point in the case of a monitor image, and in the case of a printed material, the white point of the printed material is calculated by the white XYZ value of the printed material and (illuminance [lux] / π on the printed material).

  On the other hand, strictly speaking, the absolute luminance Ls of the surrounding region 64 is the absolute luminance of the region outside the viewing angle of 10 degrees with respect to the target pixel, but for the sake of simplicity, in the case of a monitor image, the absolute luminance around the monitor For brightness and printed matter, use absolute brightness around the printed matter. The absolute luminance can be set to a different value depending on the observation conditions on the input side and the output side.

  In step S106, the CPU 101 determines the ambient condition on the input side. The determination of the ambient conditions follows, for example:

if (0.2 ≤ Ls / Lw) Average Surround;
if (0.06 <Ls / Lw <0.2) Dim Surround;
if (Ls / Lw ≤ 0.06) Dark Surround;
If Ls and Lw are different between the input side and the output side, the ambient conditions are determined independently on the input side and the output side. Further, when the ambient condition is known, the ambient condition can be directly designated by manual input using the user's keyboard 111 or mouse 112.

  In step S107, the CPU 101 generates an input-side image 501 from the image 500 by forward conversion of a device model created from the source-side ICC profile or WCS profile. In step S <b> 108, the CPU 101 performs filter processing by the low-pass filter 607 on the Y component (luminance component) of the image 501 to generate an input-side low-pass image.

  In step S109, the CPU 101 sets an unselected pixel in the image 501 as a target pixel. The pixel selection order is not limited to a specific selection order. For example, the pixels may be selected in a raster scan order.

  In step S110, the CPU 101 acquires the luminance value Y of the pixel of interest in the input-side low-pass image, and calculates the relative luminance Yb of the background region 63 with respect to the pixel of interest and the absolute luminance La of the adaptation region using the above equation (1). To do.

  In step S <b> 111, the CPU 101 calculates parameters used for the spatial CAM forward conversion 504 by executing the processes described as being performed by the calculation units 604, 605, and 606. In step S <b> 112, the CPU 101 forward-converts the XYZ value of the target pixel of the image 501 into a JCH value by executing each process described as being performed in the spatial CAM forward conversion 504.

  In step S113, the CPU 101 determines whether or not all the pixels in the image 501 are set as the target pixel. As a result of this determination, when all the pixels in the image 501 are set as the target pixel, the processing according to the flowchart of FIG. 15 is completed. On the other hand, if there remains an unselected pixel that has not been set as the pixel of interest among the pixels in the image 501, the process returns to step S109. When the processing according to the flowchart of FIG. 15 is completed, an image 505 as JCH data is completed. The input side observation distance, the input side resolution, and the input side observation conditions may be acquired from the input side profile.

[Processing flow of spatial CAM inverse transform]
Details of the processing in step S213 will be described with reference to the flowchart of FIG. Note that the flowchart in FIG. 16 illustrates processing for converting JCH data having the same resolution as that of the input side device into output image data having a different resolution.

  First, similarly to the processing in steps S101 to S106, the distance D between the output image and the observer is input (step S114), the output side region is calculated based on the viewing angle (step S115), and the output image resolution R [ppi ] Is input (step S116). Then, an output-side low-pass filter (Y LOW-PASS) 607 is set (step S117), output-side observation condition parameters 510 and 511 are input (step S118), and output-side ambient conditions are determined (step S119). ). It should be noted that the distance D, resolution R, and output side adaptation area parameters can be set to different values on the input side and the output side.

  In step S120, the CPU 101 functions as the resolution conversion unit 806 described above, and converts the resolution of the image 501 to the output-side resolution. The CPU 101 changes the resolution conversion method by comparing the Y value of the target pixel of the image 501 with the Y value of the target pixel of the input low-pass image. For example, when the Y value of the image 501 is equal to the Y value of the input low-pass image, the nearest neighbor method is used, and when these Y values are different, the bicubic method is used to increase the processing speed and the resolution conversion. To improve image quality. The combination of resolution conversion methods is not limited to the nearest neighbor method and the bicubic method, and other resolution conversion methods such as the bilinear method may be combined, and the above Y value comparison is not performed, and a single resolution conversion is performed. You may use the method.

  In step S <b> 121, the CPU 101 performs a filtering process using the low-pass filter 607 on the Y component (luminance component) of the image described as being output by the resolution conversion unit 806 to generate an output-side low-pass image.

  In step S122, the CPU 101 functions as the resolution conversion unit 801, and uses the same resolution conversion method as the resolution conversion method used by the resolution conversion unit 806 to generate an image 802 in which the resolution of the image 509 is converted to the output-side resolution.

  In step S123, the CPU 101 sets an unselected pixel in the image 802 as a target pixel. The pixel selection order is not limited to a specific selection order. For example, the pixels may be selected in a raster scan order.

  In step S124, the CPU 101 functions as the calculation units 608 and 609, obtains the luminance value Y of the target pixel in the output-side low-pass image, and calculates the relative luminance Yb of the background region 63 with respect to the target pixel and the adaptation region using Equation (1). The absolute luminance La is calculated.

  In step S125, the CPU 101 functions as the calculation units 604, 605, and 606, and calculates parameters used for inverse conversion. In step S126, the CPU 101 performs each process described as being performed by the spatial CAM inverse transform 512, thereby inversely transforming the JCH value of the target pixel of the image 802 into an XYZ value.

  In step S127, the CPU 101 determines whether all the pixels in the image 802 are set as the target pixel. As a result of this determination, when all the pixels in the image 802 are set as the target pixel, the processing according to the flowchart of FIG. 16 is completed. On the other hand, if there remains an unselected pixel that has not been set as the pixel of interest among the pixels in the image 802, the process returns to step S123. When the processing according to the flowchart of FIG. 16 is completed, an image 513 as XYZ data is completed.

[Flow of saliency map generation processing]
Details of the processing in step S206 (S209) will be described with reference to the flowchart of FIG. In step S1301, observation conditions (adaptation luminance, white point, background luminance, ambient light level, resolution, observation distance) for the target image (image 500 on the input side, image 514 on the output side) are acquired. Here, when the observation conditions for the target image are stored in advance in the profile, the observation conditions may be acquired from the profile.

  In step S1302, the CPU 101 determines a contrast sensitivity function based on the observation conditions. Here, it is assumed that the contrast sensitivity function changes according to the adaptation brightness and the observation distance.

  In step S1303, the CPU 101 selects an unselected pixel in the image 505 as a selected pixel. Then, the CPU 101 decomposes the JCH value of the selected pixel into a Lightness component, a Redness-Greenness component, and a Yellowness-Blueness component based on the above formulas (11) to (13).

  In step S1304, the CPU 101 determines whether all pixels of the image 505 have been selected as selected pixels. As a result of this determination, when all the pixels of the image 505 are selected as selected pixels, a feature-specific map for brightness, a feature-specific map for color, an edge from the decomposed Lightness component, Redness-Greenness component, Yellowness-Blueness component In order to generate the feature-specific map for the direction, the process proceeds to each of steps S1308, S1312, S1316, and S1322. On the other hand, if there remains a pixel that is not selected as the selected pixel, the process returns to step S1303.

  A feature-specific map for lightness is generated from the Lightness component. In step S1306, the CPU 101 applies a DoG (Difference of Gaussians) filter having a different scale to the Lightness component, and generates a feature amount map having a predetermined number of scales. Here, the feature amount map to which the DoG filter is applied may be generated from a difference between multi-resolution images having different scales using a multi-resolution image of a Gaussian filter. Note that the result D (x, y, c, s) of applying the DoG filter is that the standard deviation of the Gaussian filter is σ, the Gaussian filter is G (x, y, σ), and the image to be applied is I (x, y) L (x, y, σ) for the convolution of the applied image and the Gaussian filter, c for the center side standard deviation in the center-surround structure of the DoG filter, and s for the standard deviation on the surround side in the center-surround structure of the DoG filter Then, it is defined by the following formulas (21) to (23).

  The DoG filters having different scales are composed of one or a plurality of DoG filters whose upper limit does not exceed the contrast sensitivity function for the lightness component in FIG. 11A (or FIGS. 12 and 13) in the spatial frequency domain. . Here, since the contrast sensitivity function is defined as a characteristic for the spatial frequency [cycles per degree], the standard deviation c on the center side in the center-surround structure of the DoG filter, the standard on the surround side in the center-surround structure of the DoG filter The deviation s is determined based on “the distance between the target image (the image 500 on the input side, the image 514 on the output side) and the observer (observation distance)” and “the resolution of the target image”. The contrast sensitivity function for the Lightness component in FIG. 11A is defined by the following equation, for example, where the spatial frequency is f [cycles per degree].

  Here, each parameter is (a, b, c) = (75, 0.2, 0.8). Next, in step S1307, the CPU 101 determines whether or not the process of step S1306 has been executed for a predetermined number of scales. As a result of the determination, when the process of step S1306 is executed for a predetermined number of scales, the process proceeds to step S1308. On the other hand, if the process of step S1306 has not been executed for a predetermined number of scales, the process returns to step S1306.

  In step S1308, the CPU 101 normalizes each map because a feature amount map to which a DoG filter having a different scale is applied is generated. In step S1309, the CPU 101 obtains an arithmetic average for each pixel to generate a feature-specific map for lightness.

  A feature-specific map for the color is generated from the Redness-Greenness component and the Yellowness-Blueness component. In step S1312, the CPU 101 applies a Gaussian filter having a different scale to the Redness-Greenness component, and generates a feature amount map having a predetermined number of scales. Here, the result of applying the Gaussian filter is defined by equation (11). The Gaussian filters having different scales are composed of one or a plurality of Gaussian filters whose upper limit does not exceed the contrast sensitivity function for the Redness-Greenness component in FIG. 11B in the spatial frequency domain. Here, since the contrast sensitivity function is defined as a characteristic with respect to the spatial frequency [cycles per degree], the standard deviation σ of the Gaussian filter is “target image (image 500 on the input side, image 514 on the output side)”. And the distance between the observer (observation distance) and the “resolution of the target image”. The contrast sensitivity characteristic for the Redness-Greenness component in FIG. 11B is defined by the following equation, for example, where the spatial frequency is f [cycles per degree].

  Here, each parameter is (a1, b1, c1, a2, b2, c2) = (109.1413, 0.00038, 3.4244, 93.5971, 0.00367, 2.1677). Next, in step S1313, the CPU 101 determines whether or not the process of step S1312 has been executed for a predetermined number of scales. As a result of this determination, when the process of step S1312 is executed for a predetermined number of scales, the process proceeds to step S1314. On the other hand, if the process of step S1312 has not been executed for a predetermined number of scales, the process returns to step S1312. In step S <b> 1314, the CPU 101 normalizes each map because a feature amount map to which a Gaussian filter with a different scale is applied is generated.

  Similarly, in step S1316, the CPU 101 applies a Gaussian filter having a different scale to the Yellowness-Blueness component, and generates a feature amount map having a predetermined number of scales. Here, the result of applying the Gaussian filter is defined by equation (22). The Gaussian filters having different scales are composed of one or more Gaussian filters whose upper limit does not exceed the contrast sensitivity function for the Yellowness-Blueness component in FIG. 11C in the spatial frequency domain. Here, since the contrast sensitivity function is defined as a characteristic with respect to the spatial frequency [cycles per degree], the standard deviation σ of the Gaussian filter is “target image (image 500 on the input side, image 514 on the output side)”. And the distance between the observer (observation distance) and the “resolution of the target image”. The contrast sensitivity function for the Yellowness-Blueness component in FIG. 11C is defined by Expression (25). Here, each parameter is (a1, b1, c1, a2, b2, c2) = (7.0328, 0.000004, 4.2582, 40.6910, 0.10391, 1.6487).

  In step S1317, the CPU 101 determines whether the process of step S1316 has been executed for a predetermined number of scales. As a result of this determination, if the process of step S1316 has been executed for a predetermined number of scales, the process proceeds to step S1318. On the other hand, if the process of step S1316 has not been executed for the predetermined scale number, the process returns to step S1316. In step S1318, the CPU 101 normalizes each map because a feature amount map to which a Gaussian filter having a different scale is applied is generated.

  In step S1319, the CPU 101 obtains an arithmetic average for each pixel with respect to the normalized feature amount map obtained for the Redness-Greenness component and the normalized feature amount map obtained for the Yellowness-Blueness component. Thus, a feature-specific map for the color is generated.

  A feature-specific map for the edge direction is generated from the Lightness component. In step S1322, the CPU 101 applies a Difference of Gabors filter having a different scale to the Lightness component, and generates a feature amount map having a predetermined number of scales. Here, the feature amount map to which the Difference of Gabors filter is applied may be generated from the difference between the multi-resolution images having different scales using the multi-resolution image of the Gabor filter. As a result of applying the Difference of Gabors filter, O (x, y, λ, θ, ψ, c, s, γ) is the wavelength of the sine function factor λ, the direction of the stripe pattern of the Gabor filter is θ, and the phase offset Ψ, Gaussian filter standard deviation σ, spatial aspect ratio γ, applied image I (x, y), applied image and Gabor filter convolution La (x, y, λ, θ, ψ, σ , γ), the standard deviation c on the center side in the center-surround structure of the Difference-of-Gabors filter, and the standard deviation on the surround side in the center-surround structure of the Difference-of-Gabors filter as s, the following equation ( 26) to (29).

  Here, the direction θ of the stripe pattern of the Gabor filter is set to θ∈ {0 °, 45 °, 90 °, 135 °} for simplification, but is not limited to four and includes an intermediate value. It doesn't matter. The standard deviation σ of the Gaussian filter is based on “the distance between the target image (the image 500 on the input side and the image 514 on the output side) and the observer (observation distance)” and “the resolution of the target image”. Determined.

  In step S1323, the CPU 101 determines whether the process of step S1322 has been executed for a predetermined number of scales. As a result of this determination, when the process of step S1322 is executed for a predetermined number of scales, the process proceeds to step S1324. On the other hand, if the process of step S1322 has not been executed for a predetermined number of scales, the process returns to step S1322. In step S1324, the CPU 101 normalizes each map because a feature amount map to which the Difference of Gabors filter having a different scale is applied is generated. In step S1325, the CPU 101 obtains an arithmetic average for each pixel to generate a feature-specific map for the edge direction.

  In step S1327, the CPU 101 generates a saliency map by arithmetically averaging, for each pixel, the feature-specific map for brightness, the feature-specific map for color, and the feature-specific map for edge direction. Here, the saliency map generation is not limited to the arithmetic average of the feature-specific map, and the weighted average may be obtained by performing different weighting on the feature-specific map.

[Second Embodiment]
In the first embodiment, the image processing apparatus has been described as being configured by one apparatus, but may be configured by a plurality of apparatuses. For example, the color gamut mapping process shown in FIG. 5 may be divided into a plurality of partial processes, and the plurality of partial processes may be executed by a plurality of devices.

  According to the configuration described above, in color gamut mapping considering different viewing conditions (including image resolution and viewing distance), the difference value of the “saliency map” before and after color gamut mapping is minimized. By performing such optimization, color gamut mapping that matches human perception can be performed.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  101: CPU 102: Main memory

Claims (10)

An image processing apparatus for mapping a color gamut on an input side to a color gamut on an output side,
First generation means for generating a first saliency map that maps the degree of noticeable pixels in an input image that stand out from neighboring pixels based on the observation conditions on the input side;
A second saliency map that maps the degree to which the target pixel in the image is conspicuous as compared to neighboring pixels for each of a plurality of images obtained by performing each of a plurality of color gamut mappings on the input image Second generation means for generating the output based on the observation condition on the output side;
Third generation means for generating an image after color gamut mapping for the input image based on a difference value between the first saliency map and the second saliency map and the plurality of images; An image processing apparatus comprising:   The first generation means generates the first saliency map based on a multi-resolution image generated by applying a spatial filter to each component of a color space to which the input image belongs. The image processing apparatus according to claim 1.   The second generating means is configured to generate the second noticeable image based on a multi-resolution image generated by applying a spatial filter to each component of a color space to which the image belongs, for each of the plurality of images. The image processing apparatus according to claim 1, wherein a sex map is generated.   The third generation unit weights and fuses each of the plurality of images based on a difference value between the first saliency map and the second saliency map corresponding to the image. The image processing apparatus according to claim 1, wherein an image is generated as an image after color gamut mapping with respect to the input image.   5. The input side observation condition includes an absolute brightness of a white point in an adaptation area on the input side, an image resolution on the input side, and an observation distance on the input side. 6. Image processing apparatus.   6. The output-side observation condition includes an absolute luminance of a white point in an output-side adaptation region, an output-side image resolution, and an output-side observation distance. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the input image and an image after color gamut mapping for the input image are JCH images.   The image processing apparatus according to claim 1, wherein each of the plurality of color gamut mappings is a different color gamut mapping algorithm. An image processing method performed by an image processing apparatus for mapping a color gamut on an input side to a color gamut on an output side,
The first generation means of the image processing device generates a first saliency map in which the degree of noticeable pixels in the input image is noticeable compared to neighboring pixels based on the observation conditions on the input side. 1 generation process;
For each of a plurality of images obtained by the second generation unit of the image processing device performing each of a plurality of color gamut mappings on the input image, the pixel of interest in the image is compared with a neighboring pixel. A second generation step of generating a second saliency map in which the degree of conspicuous is mapped based on the observation condition on the output side;
A third generation unit of the image processing device, based on a difference value between the first saliency map and the second saliency map and the plurality of images, gamut mapping for the input image An image processing method comprising: a third generation step of generating a subsequent image.   A computer program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 8.