JP2008067093A - Camera system, image processing apparatus, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique for effectively using three-dimensional measurement data of each part of a field for image processing. <P>SOLUTION: A camera system obtains depth data of the field in each part of an image by three-dimensional measurement. The depth data are made into a gradation map so as to generate an image file with a depth map. An image processor acquires the depth map from the image file with the depth map and controls the image processing in each part of the image on the basis of the depth map. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a camera system, an image processing apparatus, and an image processing program.

Patent Document 1 below discloses a technique for acquiring a focus distance (subject distance range, subject distance information) of an imaging lens from an image file and estimating a main subject from the focus distance. For example, when the subject distance range indicates foreground shooting, it is determined that the person is the main subject, and the main subject is detected by performing skin color detection. Further, when the subject distance range indicates distant shooting, it is determined that the scenery is the main subject, and the main subject is detected by blue sky detection or the like.
JP 2005-174247 A

By the way, a three-dimensional measurement apparatus using a known three-dimensional measurement technique (parallax detection, light cutting method, detection of a sharp portion accompanying a focus scan of a lens, etc.) is known. By using this type of three-dimensional measurement apparatus, three-dimensional measurement data can be obtained for each part of the object scene.
An object of the present invention is to provide a technique for effectively using the three-dimensional measurement data of the object scene for image processing.

<< 1 >> The camera system of the present invention includes an imaging unit, a three-dimensional measurement device, a data processing unit, and a file generation unit.
The imaging unit shoots the scene and generates image data.
The three-dimensional measuring device performs three-dimensional measurement on the scene captured by the imaging unit, and generates three-dimensional measurement data of the scene.
The data processing unit generates a depth map corresponding to the image data by arranging the depth data of the object scene obtained from the three-dimensional measurement data in a pixel unit or image area unit of the image data and converting it into a gradation map. To do.
The file generation unit generates an image file with a depth map by recording the depth map attached to the image data.
<< 2 >> The image processing apparatus of the present invention includes a file acquisition unit and an image processing unit.
The file acquisition unit takes in an image file with a depth map generated in the camera system according to claim 1 and obtains image data and a depth map.
The image processing unit controls image processing in each part of the image data based on the depth map.
The image processing here may be either processing for processing an image like image quality adjustment or processing for analyzing an image like light source estimation.
<< 3 >> Preferably, the image processing unit includes an operation unit, a mask generation unit, and a processing calculation unit.
The operation unit receives an input of a position on the image (hereinafter referred to as a control point) according to a user operation.
Based on the depth map, the mask generation unit extracts a selection range that falls within a predetermined depth range including the control point, and generates a processing mask.
The processing operation unit controls the degree of reflection of the image processing of each part of the image according to the processing mask.
<< 4 >> Preferably, the image processing unit includes a flash map generation unit and a processing calculation unit.
The flash map generation unit acquires information on the amount of flash illumination from the image file. The flash map generation unit obtains a light amount distribution of flash illumination in each part of the image based on the depth map and the light emission amount, and sets it as a flash map.
The processing calculation unit controls the brightness adjustment of each part of the image data in accordance with the flash map.
<< 5 >> Preferably, the image processing unit includes a shake detection unit, a filter creation unit, and a processing calculation unit.
The blur detection unit divides the image data into regions based on the depth based on the depth map, and detects image blur for each of the divided image data.
The filter creation unit obtains an inverse filter of a point spread function representing image blur for each depth.
The processing calculation unit applies an inverse filter according to the depth of the image data to reduce image blur of the image data.
<< 6 >> Preferably, the image processing unit includes a light source estimation unit and a processing calculation unit.
The light source estimation unit divides the image data into regions by depth based on the depth map. The light source estimation unit performs light source estimation image processing for each of the divided image data.
The processing calculation unit controls white balance adjustment applied to the image data in accordance with the light source estimation result obtained for each depth.
<< 7 >> The image processing program of the present invention is a program for causing a computer to function as the image processing apparatus described in any one of the above << 1 >> to << 6 >>.

  In the camera system of the present invention, a depth map of the object scene is generated and attached to the image data. Therefore, it is possible to know the depth of the subject in each part of the image by referring to the attached depth map in image data display or post-processing.

  On the other hand, in the image processing apparatus and the program of the present invention, image processing of each part of the image is controlled based on the depth map. As a result, it is possible to perform advanced image quality adjustment and image analysis in consideration of the influence of the depth of each part of the image.

[Configuration of Camera System 101]
FIG. 1 is a block diagram showing a system configuration of the camera system 101.
In FIG. 1, the camera system 101 is provided with a camera body including an imaging unit 11, a data processing unit 12, a file generation unit 13, and the like. Further, a three-dimensional measuring device 14 is attached to an accessory shoe or the like of the camera body.

[Configuration of Image Processing Device 102]
FIG. 2 is a block diagram illustrating a system configuration of the image processing apparatus 102.
In FIG. 2, the image processing apparatus 102 is schematically configured to include a file acquisition unit 21, an image processing unit 22, and a monitor device 23.
The image processing unit 22 includes a processing calculation unit 24, an operation unit 25, a mask generation unit 26, a flash map generation unit 27, a shake detection unit 28, a filter generation unit 29, and a light source estimation unit 30.
The image processing apparatus 102 may be realized as software by an image processing program and a computer.

[Description of Operation of Camera System 101]
FIG. 3 is a flowchart for explaining the operation of the camera system 101.
Hereinafter, this operation will be described along the step numbers shown in FIG.

Step S1: The imaging unit 11 waits for a release operation by the user. When this release operation is accepted, the operation proceeds to step S2.

Step S2: The imaging unit 11 captures an object scene and generates image data A. Image data A is output to the data processing unit 12.

Step S3: The data processing unit 12 instructs the three-dimensional measurement device 14 to perform three-dimensional measurement. The three-dimensional measuring device 14 captures the object scene from a position away from the imaging unit 11 by a predetermined baseline length, and generates image data B for parallax comparison.

Step S <b> 4: The data processing unit 12 obtains a shooting angle of view from the focal length of the imaging unit 11 and transmits it to the three-dimensional measurement device 14. The three-dimensional measuring device 14 adjusts the resolution based on the shooting angle of view so that the size of the number of pixels of the subject in the image data A and B becomes equal.

Step S5: The three-dimensional measuring device 14 performs block matching between the image data A and B processed in step S4, and detects parallax in the baseline length direction.

Step S6: The three-dimensional measurement device 14 obtains depth data in units of blocks based on the parallax in units of blocks. The depth data is preferably a value obtained by compressing the numerical interval as the distance increases, from the viewpoint of measurement accuracy and human distance perception.

Step S7: The three-dimensional measurement device 14 arranges the depth data in units of blocks in pixel units or image area units of the image data A generated in step S2 and generates a gradation map, thereby generating a depth map.

Step S8: The file generation unit 13 generates an image file by combining the image data A generated in step S2 and the depth map.
For example, the depth map may be stored in the α channel of the image data A.
Further, for example, the depth map may be stored in an accompanying information area (such as a manufacturer note) in the image file.
Also, for example, the depth map may be stored in another file associated with the image file (for example, only the extension is changed with the same file name as the image file).
The image file with the depth map created in this way is stored in a recording medium such as a memory card via the file generation unit 13.

[Description of Operation of Image Processing Device 102]
The image file with the depth map created by the camera system 101 is transmitted to the image processing apparatus 102 via a recording medium or a communication medium. The image processing apparatus 102 performs subsequent image processing on the image file while referring to the depth map. By this image processing, advanced image quality adjustment or image analysis in consideration of the depth of each part of the image is realized.
Hereinafter, the operation of the image processing apparatus 102 will be specifically described for each of these image processes.

(1) Processing by Control Point In the image processing apparatus 102, a user performs a GUI (Graphical User Interface) operation to arrange a control point on the image. A desired image quality adjustment can be performed via this control point.
FIG. 4 is a diagram for explaining the flow of image processing using this control point. Hereinafter, this operation will be described along the step numbers shown in FIG.

Step S11: The file acquisition unit 21 of the image processing apparatus 102 takes in an image file with a depth map created by the camera system 101 and stores it in an internal memory area. The file acquisition unit 21 extracts image data and a depth map from the image file with the depth map.

Step S12: The operation unit 25 receives a position designation on the image by a GUI operation from the user, and places a control pointer P at the position (see FIG. 5).
On the control pointer P, the following adjustment slider is displayed.
(a) Front / rear width adjustment slider in the depth range
(b) Hue parameter adjustment slider
(c) Saturation parameter adjustment slider
(d) Brightness parameter adjustment slider
(e) Contrast parameter adjustment slider
(f) Red component parameter adjustment slider
(g) Green component parameter adjustment slider
(h) Blue component parameter adjustment slider
(i) Warm Color Parameter Adjustment Slider A size parameter adjustment slider may be added to limit the selection range (described later) within a predetermined radius from the control pointer P.

Step S13: The mask generation unit 26 refers to the depth map to obtain the depth of the subject located at the position where the control pointer P is disposed.

Step S <b> 14: The mask generation unit 26 determines the depth range by widening the depth of the control pointer P by a predetermined front-rear width. The mask generation unit 26 refers to the depth map, and extracts an image area that falls within the depth range as a selection range.

Step S15: The mask generation unit 26 gives a high opacity to the selection range and creates a processing mask. At this time, the closer to the depth of the control pointer P, the higher opacity may be given.

Step S16: When the operation unit 25 receives an operation of changing the front-rear width adjustment slider (see FIG. 5) of the control pointer P, the operation unit 25 shifts the operation to step S14 and resets the processing mask. When there is no change operation of the front / rear width adjustment slider, the operation unit 25 shifts the operation to step S17.
It should be noted that the selection range may be displayed over the image in the front-rear width adjustment process. Such display of the selection range can inform the user of the state of the selection range.

Step S17: When the operation unit 25 receives an operation for changing the image processing parameter adjustment slider (see FIG. 5) of the control pointer P, the operation unit 25 shifts the operation to step S18. When there is no image processing parameter change operation, the operation unit 25 shifts the operation to step S19.

Step S18: The processing calculation unit 24 reflects the change in the image processing parameter received in step S17 on the image data according to the opacity of the processing mask.

Step S19: When the operation unit 25 is instructed by the user to complete the process of the control pointer P, the operation unit 25 completes the series of processes described above. On the other hand, if there is no instruction to complete the processing, the operation returns to step S16, and the image processing relating to the control pointer P is continued.
By such processing related to the control pointer P, it is possible to extract a subject located in a desired depth range as a selection range and selectively perform image processing on the subject.

(2) Flash Map Processing The image processing apparatus 102 can perform image processing using a flash map on image data shot with flash illumination.
FIG. 6 is a diagram for explaining the flow of image processing using this flash map. Hereinafter, this operation will be described along the step numbers shown in FIG.

Step S31: The same process as step S11 described above.

Step S32: The flash map generation unit 27 acquires information on the light emission amount of the flash illumination (or the irradiation angle distribution of the light emission amount) from the accompanying information of the image file.

Step S33: The flash map generation unit 27 detects the following information based on the depth map.
(a) The distance from the light emitting part of the flashlight to the subject in each part of the image,
(b) Angle from the central axis of flash illumination to the subject in each part of the image
(c) Surface angle of subject in each part of image (inclination angle obtained by connecting depth data in the surface direction)

Step S34: The flash map generation unit 27 estimates the amount of irradiation light reaching the subject of each part of the image from the light emission amount (or the irradiation angle distribution of the light emission amount) and the information (a) and (b).
Further, the flash map generation unit 27 can estimate the amount of reflected light by obtaining what percentage of the amount of irradiation light returns to the camera side from the surface angle of (c).
The flash map generation unit 27 generates the flash map by arranging the irradiation light amount (or reflection light amount) of each part of the image thus obtained in a pixel unit or image region unit of the image data and forming a gradation map. .

Step S35: The flash map generation unit 27 creates a processing mask by giving a higher opacity to a portion where the irradiation light amount (or reflection light amount) of the flash map is lower.

Step S36: The processing calculation unit 24 adjusts the image data brighter as the opacity of the processing mask is higher. Further, the brightness of the image data is lowered as the opacity of the processing mask is lower.
By such processing related to the flash map, it is possible to select a subject that is difficult to reach flash light and to adjust the subject brightly. In addition, it is possible to select a highlight portion where the flash illumination is strongly applied, and to suppress the brightness of the highlight portion.
Therefore, it is possible to moderate the uneven illumination of the flash illumination and obtain image data in a natural illumination state.

(3) Depth-specific blur correction processing The image processing apparatus 102 can perform depth-specific blur correction processing using a depth map.
FIG. 7 is a diagram for explaining the flow of the blur correction process for each depth. Hereinafter, this operation will be described along the step numbers shown in FIG.

Step S51: The same process as step S11 described above.

Step S52: The shake detection unit 28 divides the image data into regions based on the depth based on the depth map.

Step S53: The blur detection unit 28 detects an image blur (blur direction, blur width) for each image data divided into regions.
For example, the absolute value sum of the adjacent pixel differences is calculated in each direction such as vertical and horizontal directions. The direction in which the absolute value sum is smallest may be the blur direction.
Further, for example, the autocorrelation of the image may be obtained while shifting the image in the obtained blur direction, and the blur width may be detected from the extent of the autocorrelation characteristics.

Step S54: Based on the image blur (blur direction, blur width) obtained for each depth, the filter creation unit 29 obtains a point spread function representing the image blur for each depth.

Step S55: The filter creation unit 29 determines an inverse filter of the point spread function for each depth. Note that by preparing a correspondence table of image blur and inverse filter in advance, it is also possible to immediately determine the inverse filter from image blur.

Step S56: The processing calculation unit 24 corrects image blur for each depth by individually applying an inverse filter for each depth to the image data divided into regions for each depth.

Step S57: The processing calculation unit 24 synthesizes image data in which image blur is corrected for each depth.
By such image blur correction for each depth, it is possible to correct subject blur that occurs for each depth. It is also possible to correct camera shake caused by depth due to parallel camera shake.

(4) Light source estimation processing by depth The image processing apparatus 102 can perform light source estimation by depth using a depth map.
FIG. 8 is a diagram for explaining the flow of the light source estimation processing for each depth. Hereinafter, this operation will be described along the step numbers shown in FIG.

Step S71: The same process as step S11 described above.

Step S72: The light source estimation unit 30 divides the image data into regions by depth based on the depth map.

Step S <b> 73: The light source estimation unit 30 performs light source estimation for each image data divided into regions by depth.
For example, a hue distribution may be obtained for a low-saturation location, and light source estimation may be performed from this hue distribution bias. Further, for example, light source estimation may be performed by averaging hues in pixel units and obtaining a color temperature on a black body radiation locus closest to the average hue in the chromaticity space.

Step S74: In the image processing apparatus 102, how much the characteristics of the mixed light source are utilized in white balance is set in advance by custom setting.
Based on the custom setting, the light source estimation unit 30 determines whether to use the characteristics of the mixed light source for image processing.
When taking advantage of the characteristics of the mixed light source, the light source estimation unit 30 shifts the operation to step S76.
On the other hand, when the characteristics of the mixed light source are not utilized, the light source estimation unit 30 shifts the operation to step S75.

Step S75: In this step, the light source estimation unit 30 determines a white balance adjustment value independently for each depth. The white balance adjustment value determined independently in this way suppresses color deviation for each depth. As a result, even under a complicated light source such as a mixed light source, it is possible to accurately reproduce the original color tone of each subject.
After determining such a white balance adjustment value, the light source estimation unit 30 shifts the operation to step S77.

Step S76: In this step, the light source estimation unit 30 collates the light source estimation result for each depth with a predetermined light source priority rule to determine a white balance adjustment value.
For example, when the near view is an indoor light source with a low color temperature and the distant view is a solar light source, white balance adjustment values for the close view and distant view are determined with the solar light source as the center. In this case, white balance adjustment (adjustment that makes use of the light on the spot) that achieves a slight shade of room light is realized.
For example, when the near view is a shade with a high color temperature and the distant view is a solar light source, the white balance adjustment value is determined for each depth. In this case, the distant view can be adjusted to a natural hue while suppressing shaded blue.

Step S77: The processing calculation unit 24 applies the image data divided into regions by depth.
White balance adjustment is performed by applying the white balance adjustment value determined in step S75 or step S76.

Step S78: The processing calculation unit 24 re-synthesizes the image data subjected to the white balance adjustment for each depth, thereby obtaining image data for which the white balance adjustment has been completed.
Such depth-based light source estimation makes it possible to appropriately determine the mixed light source of the object scene. Further, by adjusting the white balance for each depth, it is possible to remove the complicated color deviation caused by the mixed light source and faithfully reproduce the original color tone of the subject. In addition, according to the light source priority rule, it is possible to selectively leave the characteristics of the near view / distant view light source.

[Effects of this embodiment, etc.]
In the camera system 101 described above, the depth data is converted into a gradation map and attached to the image data. As a result, display and image processing that make use of the depth of the subject in each part of the image can be performed later.

  On the other hand, the image processing apparatus 102 described above can perform image processing in consideration of the depth of the subject of each part of the image by using the information of the depth map.

  Further, in the above-described image processing apparatus 102, it is possible to sensuously select subjects located in a predetermined depth range by arranging the control point P on the image and adjusting the front-rear width adjustment slider. Furthermore, by adjusting the adjustment slider for the image processing parameter with respect to the control pointer P, it is possible to perform desired image processing on the subject located in a predetermined depth range. As a result, it is possible to easily and intuitively perform image processing for each depth.

  Further, the image processing apparatus 102 described above can determine how much flash illumination is applied to the subject of each part of the image from the information of the depth map. By adjusting the brightness of each part of the image based on this determination, it is possible to moderately suppress the uneven illumination of the flash illumination that appears in the image data.

  Further, the image processing apparatus 102 described above corrects image blur for each depth. Therefore, it is possible to appropriately correct image blur caused by depth due to subject blur or camera parallel blur.

  Further, the above-described image processing apparatus 102 performs light source estimation for each depth. Therefore, it is possible to distinguish complicated mixed light sources by depth, and more accurate light source estimation is possible.

<< Additional items of embodiment >>
In the above-described embodiment, the three-dimensional measurement device 14 is a device that performs three-dimensional measurement by parallax detection. However, the present invention is not limited to the three-dimensional measurement method.

  In the above-described embodiment, the case where the three-dimensional measuring device 14 is externally attached to the camera body has been described. However, the present invention is not limited to this. A three-dimensional measuring device may be built in the camera body.

  The present invention can be implemented in various other forms without departing from the spirit or main features thereof. For this reason, the above-described embodiments are merely examples in all respects and should not be interpreted in a limited manner. The scope of the present invention is indicated by the scope of claims, and is not restricted by the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

  As described above, the present invention is a technique that can be used for a camera system, an image processing apparatus, an image processing program, and the like.

1 is a block diagram showing a system configuration of a camera system 101. FIG. 2 is a block diagram showing a system configuration of an image processing apparatus 102. FIG. 3 is a flowchart illustrating the operation of the camera system 101. It is a figure explaining the flow of the image processing using a control point. It is a figure explaining the mode of operation of a control point. It is a figure explaining the flow of the image processing using a flash map. It is a figure explaining the flow of the blurring correction process according to depth. It is a figure explaining the flow of the light source estimation process according to depth.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Image pick-up part, 12 ... Data processing part, 13 ... File generation part, 14 ... Three-dimensional measuring device, 21 ... File acquisition part, 22 ... Image processing part, 23 ... Monitor apparatus, 24 ... Processing operation part, 25 ... Operation 26: Mask generation unit, 27: Flash map generation unit, 28 ... Blur detection unit, 29 ... Filter generation unit, 30 ... Light source estimation unit, 101 ... Camera system, 102 ... Image processing device

Claims (7)

An imaging unit that shoots an object scene and generates image data;
A three-dimensional measurement device that performs three-dimensional measurement on the object scene captured by the imaging unit and obtains three-dimensional measurement data of the object field;
The depth data corresponding to the image data is generated by arranging the depth data of the object scene obtained from the three-dimensional measurement data in a pixel unit or an image area unit of the image data and making a gradation map. A data processing unit;
A camera system comprising: a file generation unit that generates an image file with a depth map by recording the depth map attached to the image data. A file acquisition unit that captures an image file with a depth map generated in the camera system according to claim 1 and obtains image data and a depth map;
An image processing apparatus comprising: an image processing unit that controls image processing in each part of the image data based on the depth map. The image processing apparatus according to claim 2,
The image processing unit
An operation unit that accepts input of a position on an image (hereinafter referred to as a control point) according to a user operation,
A mask generation unit that generates a processing mask by extracting a selection range that falls within a predetermined depth range including the control point based on the depth map;
An image processing apparatus comprising: a processing operation unit that controls a degree of reflection of image processing of each part of the image according to the processing mask. The image processing apparatus according to claim 2,
The image processing unit
A flash map generation unit that obtains information on the amount of flash illumination from the image file, and generates a flash map indicating the light intensity distribution of the flash illumination in each part of the image based on the depth map and the amount of light emission;
An image processing apparatus comprising: a processing calculation unit that controls brightness adjustment of each part of the image data based on the flash map. The image processing apparatus according to claim 2,
The image processing unit
Based on the depth map, the image data is divided into regions by depth, and a blur detection unit that detects image blur for each of the divided image data;
A filter creation unit for obtaining an inverse filter of a point spread function representing the image blur for each depth;
An image processing apparatus, comprising: a processing operation unit that reduces the image blur of the image data by applying the inverse filter according to the depth of the image data. The image processing apparatus according to claim 2,
The image processing unit
A light source estimation unit that divides the image data into regions based on the depth based on the depth map, and performs image processing of light source estimation for each of the divided image data;
An image processing apparatus comprising: a processing operation unit that controls white balance adjustment performed on the image data according to the light source estimation result for each depth.   An image processing program for causing a computer to function as the image processing apparatus according to any one of claims 2 to 6.

Images