JP2002133400A - Object extraction image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To regularly stably extract an object at high speed. SOLUTION: This image processor comprises a lighting part 201; an imaging part 202; an optimum lighting parameter calculating means for calculating an optimum lighting intensity; an optimum threshold calculating processing means for calculating an optimum threshold on the basis of the image when lighting the object 11 or a background 12 at a low lighting intensity or extinguishing the lighting part 201 and the image when lighting the object 11 or background 12 at the optimum lighting intensity; a difference and threshold processing part 208 for performing differential processing and threshold processing on the basis of the image when lighting the object 11 and the background 12 at a low lighting intensity or extinguishing the lighting part 201, the image when lighting the object 11 and the background 12 at the optimum lighting intensity, and the optimum threshold; and an object extraction part 209 for extracting the object 11.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a target object extraction image processing apparatus.

[0002]

2. Description of the Related Art Conventionally, a target object extraction image processing apparatus for extracting a target object from an image generally uses a characteristic that an object close to an illumination unit is brighter than an object far from the illumination unit. (Japanese Patent Laid-Open No. 9-259)
No. 278).

FIG. 2 is a diagram showing a positional relationship between a background and a target object in a conventional target object extraction image processing apparatus.

As shown in the figure, a target object 11 is disposed at a position closer to an image pickup unit 13 than a background 12, and the illumination unit 14 is turned on to illuminate the target object 11 and the background 12. By photographing the target object 11 and the background 12 by the imaging unit 13 located in, and comparing the brightness of the target object 11 illuminated by the illumination unit 14 and the brightness of the background 12 illuminated by the illumination unit 14,
The target object 11 is extracted.

For this purpose, for example, an illumination image obtained by turning on the illumination unit 14 and photographing the target object 11 and the background 12, and photographing the target object 11 and background 12 by turning off the illumination unit 14. By performing a difference / threshold process including a difference process and a threshold value process based on the non-illuminated image obtained by the
An area on the image where the brightness has greatly changed between the state where the light is turned off and the state where the light is turned on is extracted as the target object 11.

[0006]

However, in the conventional target object extraction image processing apparatus, the control method of the target object extraction image processing apparatus, the threshold set in the difference / threshold processing, and the difference / threshold processing are performed. Depending on the method of extracting the target object 11 after the above, it is not always possible to always stably and quickly extract the target object 11 in any environment.

[0007] When the target object 11 is photographed by illuminating the target object 11 with a certain illumination intensity, when the target object 11 is far from the imaging unit 13, the target object 11 is dark, and when the target object 11 is close to the imaging unit 13, the target object is dark. The object 11 appears bright. Therefore, when the difference / threshold processing is performed, for example,
When the target object 11 appears dark, it becomes impossible to extract part or all of the target object 11.

Therefore, the difference between the illuminated image and the non-illuminated image
The intensity of the illumination is controlled based on the result obtained by performing the threshold processing.

However, when such control is performed, if the illumination intensity is inappropriate, the illumination image and the non-illumination image are photographed and the difference / threshold processing is performed at least twice before the target object 11 is extracted. This necessitates a longer processing time.

In addition, when the target object 11 is close to the background 12, when the illumination unit 14 is turned on, the background 12 also becomes bright to some extent. Therefore, if an appropriate threshold is not set in the difference / threshold processing, the target object 11 is correctly extracted. Can not do it. For example, if the threshold value is low, the brightly changed background portion is also extracted as the target object 11, while if the threshold value is high, part or all of the target object 11 cannot be extracted.

The present invention solves the problems of the conventional target object extraction image processing apparatus and provides a target object extraction image processing apparatus capable of constantly and stably extracting a target object at high speed. The purpose is to:

[0012]

In order to achieve the object, a target object extraction image processing apparatus according to the present invention comprises: a lighting unit for illuminating a target object and a background; an imaging unit for photographing the target object and the background; An optimal illumination parameter calculation processing means for calculating an optimal illumination intensity when illuminating the background, and an image when illuminating at least one of the target object and the background with an illumination intensity lower than the optimal illumination intensity, Or, based on an image obtained by photographing at least one of the target object and the background by turning off the lighting unit, and an image obtained by illuminating at least one of the target object and the background with the optimal illumination intensity, An optimal threshold value calculation processing means for calculating an image when the target object and the background are illuminated with an illumination intensity lower than the optimal illumination intensity, or the illumination unit is turned off. An image obtained by photographing an object and a background, an image obtained when the target object and the background are illuminated with the optimum illumination intensity, and a difference / threshold processing unit that performs a difference process and a threshold process based on the optimum threshold. A target object extracting unit for extracting a target object based on a result of the processing and the threshold processing.

[0013]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram of a target object extraction image processing apparatus according to a first embodiment of the present invention.

In FIG. 1, reference numeral 10 denotes a target object extraction image processing apparatus, 11 denotes a target object, 12 denotes a background, 201 denotes a lighting unit for illuminating the target object 11 and the background 12, and 202 denotes a position close to the lighting unit 201. An imaging unit 203 for photographing the target object 11 and the background 12; a distance measuring unit 203 for measuring a distance from the imaging unit 202 to the target object 11;
Is an illumination control unit that controls the intensity (brightness) of the illumination of the illumination unit 201, 22 is an imaging control unit that controls the imaging unit 202, and 23 is a distance measurement control unit that controls the distance measurement unit 203. is there. Then, the illumination control unit 21 and the imaging control unit 2
The device control unit 204 is configured by the 2 and the distance measurement control unit 23.

A distance recording unit 205 records the distance from the imaging unit 202 to the target object 11 measured by the distance measuring unit 203, and a 206 records the intensity of the illumination controlled by the illumination control unit 21. An illumination parameter recording unit 207 is an image recording unit that records at least two images captured by the imaging unit 202.

Reference numeral 208 denotes an image recorded in the image recording unit 207, illumination intensity recorded in the illumination parameter recording unit 206, and an optimal threshold read from the optimal threshold database 25 as optimal threshold recording means. A difference / threshold processing unit 209 that performs difference / threshold processing based on the result of the difference / threshold processing by the difference / threshold processing unit 208 and the model of the target object 11 read from the target object model database 210 A target object extracting unit that extracts the target object 11 and outputs an extraction result. The target object model database 210 records model data of the target object 11 to be extracted.

Reference numeral 211 denotes an optimal parameter database. The optimal parameter database 211 stores an optimal illumination intensity as an optimal illumination intensity as a first optimal parameter with respect to a distance from the imaging unit 202 to the target object 11. An optimal illumination parameter database 24 as a recording unit and an optimal threshold database 25 in which an optimal threshold as a second optimal parameter for each illumination parameter used in the difference / threshold processing unit 208 are recorded. The intensity is the illumination control unit 21
Then, the optimum threshold is sent to the difference / threshold processing unit 208.
Reference numeral 212 denotes an optimum parameter calculation unit that calculates an optimum illumination intensity and an optimum threshold, and 213 denotes a warning output unit that outputs a warning when the optimum parameter calculation unit 212 detects an abnormal value.

Next, the operation of the target object extraction image processing apparatus 10 having the above configuration will be described.

The operation of the target object extraction image processing apparatus 10 in the present embodiment has two phases (processing flow): a parameter data setting phase and a target object extraction phase.

The parameter data setting phase is for setting two databases for recording the optimum illumination intensity and the optimum threshold value, and needs to be performed before the target object extraction phase.

The target object extraction phase is for extracting the target object 11 from the image, and is a main phase in the target object extraction image processing apparatus 10. Note that, in the parameter data setting phase, when the optimal parameter database 211 already set for the imaging environment and the target object 11 can be used,
You don't have to.

In the target object extraction phase, first, the distance measuring unit 203 sends the target object 1
The distance to 1 is measured, and the measured distance is recorded in the distance recording unit 205. Next, the illumination control unit 21 reads out the optimal illumination intensity from the optimal illumination parameter database 24 based on the distance recorded in the distance recording unit 205, and turns on the illumination unit 210 at the optimal illumination intensity. Increase the lighting intensity. Further, the illumination control unit 21 records the optimal illumination intensity at that time in the illumination parameter recording unit 206. In addition, it is possible to prevent the target object 11 from being erroneously detected by the optimum illumination intensity, and to stably extract the target object 11.

Next, the illumination unit 201 is turned on with the optimum illumination intensity, and the object unit 11 and the background 1 are illuminated by the imaging unit 202.
2, and the captured illumination images of the target object 11 and the background 12 are recorded in the image recording unit 207. Immediately after capturing the illumination image, the illumination unit 201 is turned off to reduce the intensity of illumination, and the imaging unit 202 captures the target object 11 and the background 12, and the captured target object 1
1 and the non-illuminated image of the background 12 are recorded in the image recording unit 207. Subsequently, the illumination image and the non-illumination image recorded in the image recording unit 207 are sent to the difference / threshold processing unit 208,
A candidate area for extracting the target object 11 (hereinafter referred to as “target object candidate area”) is set. To this end, the difference / threshold processing unit 208 includes an illumination image, a non-illumination image,
Difference / threshold processing is performed based on the illumination intensity and the optimal threshold.

In this embodiment, the illumination unit 201 is turned off when a non-illuminated image is taken to reduce the intensity of illumination.
Without turning off the light, the intensity of illumination can be made lower than when an illumination image is taken. However, it is better that the difference between the illumination intensity when capturing the illumination image and the illumination intensity when capturing the non-illumination image is larger.

Next, the basic principle for extracting the target object 11 will be described.

FIG. 3 is a diagram showing an example of a non-illuminated image according to the first embodiment of the present invention, FIG. 4 is a diagram showing an example of an illuminated image according to the first embodiment of the present invention, and FIG. FIG. 6 is a diagram illustrating an example of a difference image according to the first embodiment of the present invention, FIG. 6 is a diagram illustrating a luminance histogram of the difference image according to the first embodiment of the present invention, and FIG. 7 is a diagram illustrating a first embodiment of the present invention. FIG. 8 is a diagram showing a first example of a target object candidate region in the embodiment, FIG. 8 is a diagram showing a second example of a target object candidate region in the first embodiment of the present invention, and FIG. 9 is a first example of the present invention. FIG. 10 is a diagram illustrating a third example of a target object candidate region according to the embodiment; FIG. 10 is a diagram illustrating a background image in the case where an object having a high reflectance exists in the background according to the first embodiment of the present invention; Is an illumination image when a target object is present in front of the background according to the first embodiment of the present invention. Diagram showing, FIG. 12 is a diagram showing a fourth example of the target object candidate area in the first embodiment of the present invention. In FIG. 6, the luminance difference is plotted on the horizontal axis and the frequency is plotted on the vertical axis.

When the illuminating unit 201 is turned on when the target object 11 (FIG. 1) is located away from the background 12 and close to the imaging unit 202, generally, the object closer to the illuminating unit 201 is better than the background 12. Since it becomes bright, the non-illuminated image is as shown in FIG. 3 and the illuminated image is as shown in FIG.

Then, the difference processing of the difference / threshold processing is performed, and the luminance of the non-illuminated image is subtracted from the luminance of the illuminated image to generate a differential image as shown in FIG. , Usually represented by a luminance histogram as shown in FIG. In FIG. 6, part A mainly represents the area of the background 12, and part B mainly represents the area of the target object 11. In this case, the portion of the background 12 far from the illumination unit 201 has a small change in luminance between when it is turned on and when it is turned off, and the portion of the target object 11 that is close to the illumination unit 201 has a large change in luminance when it is turned on and when it is turned off. Become. Therefore, when threshold processing of the difference / threshold processing is performed on the difference image with a predetermined threshold, images as shown in FIGS. 7 to 9 are obtained, and the areas that have changed to white are the target object candidate areas. Becomes Then, when the threshold processing is performed at an appropriate threshold T1, the target object candidate area is formed as a silhouette of the target object 11, as shown in FIG. 7, so that the target object 11 can be easily extracted. it can. On the other hand, the threshold T2
When the threshold processing is performed in (<T1), a target object candidate area including the background 12 is formed as shown in FIG.
When the threshold processing is performed at the threshold value T3 (> T1), as shown in FIG. 9, a target object candidate region lacking the target object 11 is formed, so that it becomes difficult to extract the target object 11. .

Therefore, in the present embodiment, the threshold T
An optimal threshold value such as 1 is recorded in advance in the optimal threshold database 25 in association with the intensity of each illumination, and when performing threshold processing, the optimal threshold corresponding to the intensity of the illumination at that time is obtained from the optimal threshold database 25. The appropriate threshold value is determined. Therefore, an optimum target object candidate area can be formed in any environment. The optimal threshold database 25 used in the threshold processing is set in the parameter data setting phase.

By performing the above-described processing, the target object candidate area as shown in FIG. 7 may be formed as a silhouette of the target object 11 as it is. If there is an object having a higher reflectance than the target object 11 in the target object 12, the target object candidate area is not necessarily formed as a silhouette of the target object 11. For example, as shown in FIG.
There is a window glass as an object having a high reflectance, and as shown in FIG. 11, when the target object 11 is a person standing in front of the window glass, when the lighting unit 201 is turned on, the background 12 becomes higher than the target object 11. Is brighter. As a result, when the difference / threshold processing is performed on the illumination image and the non-illumination image as described above, a target object candidate region as shown in FIG. 12 is formed, and the window glass of the background 12 is erroneously set as the target object 11. It will be extracted.

Therefore, in the target object extracting unit 209,
The target object 11 is extracted from the target object candidate area.

The method of extracting the target object 11 from the target object candidate area in the target object extraction unit 209 differs depending on the type of the target object 11 to be extracted, but the basic processing flow is the same. That is, the target object extraction unit 2
09 is further necessary for the target object candidate area formed by the difference / threshold processing unit 208 based on the model information of the target object 11 which is assumed in advance and recorded in the target object model database 210. Accordingly, the target object 11 is extracted based on the distance from the imaging unit 202 to the target object 11. For example, when a target object candidate region as shown in FIG. 12 is formed, the target object 11 is extracted by fitting a human-shaped model. In addition, a face model and a body model are prepared in advance as a model of the target object 11, and FIG.
A face model is extracted by fitting a face model from the target object candidate area as shown in, and a body model is fitted from the extracted face size and position to estimate the body size and position. The body can be extracted. The body can be extracted using the target object candidate area shown in FIG. 12 and the illumination image shown in FIG.
1-1149281).

As described above, in the present embodiment, the optimal control of the illumination unit 201 is performed based on the distance from the imaging unit 202 to the target object 11, and the optimal threshold in the difference / threshold processing is further controlled by the illumination intensity. Is determined, and finally the target object 11 is extracted using the model of the target object 11, so that the target object 11 can be extracted stably and at high speed in any environment. it can.

Next, the parameter data setting phase will be described.

The parameter data setting phase is for setting an optimum illumination intensity and an optimum threshold used in the target object extraction phase.

FIG. 13 is a diagram showing a luminance histogram of a difference image when the target object is close to the background according to the first embodiment of the present invention, and FIG. 14 is a diagram showing the illumination intensity in the first embodiment of the present invention. FIG. 15 is a diagram illustrating a luminance histogram of the difference image when the intensity is low, and FIG. 15 is a diagram illustrating a luminance histogram of the difference image when the illumination intensity is high according to the first embodiment of the present invention. 13 to 15, the horizontal axis represents the luminance difference, and the vertical axis represents the frequency. In addition, FIGS.
In 5, the part A mainly represents the area of the background 12 (FIG. 1), and the part B mainly represents the area of the target object 11.

First, an optimal lighting control method and a method of creating the optimal lighting parameter database 24 will be described.

In the target object extraction image processing apparatus 10, when the target object 11 is far from the illumination unit 201 and close to the background 12, the target object 11 and the background
When the image No. 2 is illuminated, in the luminance histogram of the difference image between the illumination image and the non-illumination image, the portion A and the portion B overlap as shown in FIG. In this case, when threshold processing is performed,
Even if the optimum threshold value T1 is set, a part of the target object 11 is missing and a target object candidate area including a part of the background 12 is formed. Further, when the threshold values T2 and T3 are set, a target object candidate area in which the target object 11 is further missing or the background 12 is included is formed. Therefore, the processing load of the target object extracting unit 209 increases, and the processing time increases.

Therefore, an optimal illumination control method for optimally controlling the illumination unit 201 will be described.

In general, for an object whose surface is diffusely reflected,
When the illumination unit 201 is turned on and the object is photographed, the brightness of the object increases in proportion to the intensity of the illumination.
It becomes dark in inverse proportion to the square of the distance from 01. Therefore, except for the case where the reflectance of the background 12 is much higher than the reflectance of the target object 11 (for example, a mirror or the like), the target object 1
1 and the background 12 are photographed with the illumination intensity changed, the luminance histogram of the difference image between the illumination image and the non-illumination image is:
When the illumination intensity is low, the result is as shown in FIG. 14, and when the illumination intensity is high, the result is as shown in FIG.

That is, when the intensity of the illumination is increased, the illumination image becomes brighter as a whole. However, from the characteristic of the diffuse reflection as described above, the illumination unit 201 is more brighter than the portion of the background 12 far from the illumination unit 201. The portion where the target object 11 close to is brighter is larger. Therefore, the brightness histogram of the difference image between the illuminated image and the non-illuminated image is
Since the part A and the part B are separated as shown in FIG. 15, the target object 11 can be easily extracted by the threshold processing. Thus, the higher the intensity of the illumination, the easier it is to extract the target object 11.

However, if the intensity of the illumination becomes excessively high,
Since the image quality of the extracted target object 11 is degraded due to saturation of the luminance, it is preferable to control the illumination intensity of the illumination unit 201 to such an extent that the luminance is not saturated. That is, no matter what distance the target object 11 is from the imaging unit 202, the control of the illumination unit 201 is optimally performed so that the brightness is not saturated and the brightest illumination is performed.

For this purpose, in the present embodiment, the target object 11 can be illuminated with an appropriate illumination intensity according to the distance of the target object 11 from the image pickup unit 202.
The optimal illumination parameter database 24 records the optimal illumination intensity corresponding to each distance.

Next, the optimum illumination parameter database 2
4 will be described.

FIG. 16 is a flowchart showing the operation of the optimum illumination parameter database creation processing in the first embodiment of the present invention, and FIG. 17 is the actual measurement of the distance and the optimum illumination intensity in the first embodiment of the present invention. FIG. 18 is a diagram illustrating an example of a value, and FIG. 18 is a diagram illustrating an example of an optimum illumination parameter database according to the first embodiment of the present invention.

First, a test chart having the same properties as the target object 11 is arranged at a predetermined position in front of the image pickup section 202 (FIG. 1). When the target object 11 and the test chart are photographed, the luminances of the photographed target object 11 and the test chart are substantially equal. In the present embodiment, the target object 11 is configured by the test chart in the process of creating the optimum illumination parameter database 24.

Next, the distance measurement control unit 23 performs a distance recording process, measures the distance from the imaging unit 202 to the test chart by the distance measurement unit 203, and records the measured distance in the distance recording unit 205. . Subsequently, the lighting control unit 21
A lighting control process is performed, and the lighting unit 201 is operated at a predetermined lighting intensity.
The test chart is illuminated, and the intensity of the illumination at that time is recorded as the intensity of the illumination in the illumination parameter recording unit 206, and the imaging control unit 22 performs an image capturing / recording process,
A test chart is photographed by the imaging unit 202 for each illumination intensity, and an image of the photographed test chart is recorded in the image recording unit 207.

The illumination control unit 21 performs an illumination control process for all of the plurality of illumination intensities set in advance, and the imaging control unit 22 performs an image shooting / recording process for each illumination intensity. Is performed, an optimal illumination parameter calculation processing unit (not shown) of the optimal parameter calculation unit 212 performs an optimal illumination parameter determination process, and performs image feature amounts (average luminance, maximum luminance) for all the images recorded in the image recording unit 207. , Etc.), the image feature value takes the most appropriate value, and the test chart selects an image that is illuminated most brightly so that the luminance is not saturated.

The optimal illumination parameter calculation processing means performs an optimal illumination parameter recording process, and stores the intensity of illumination corresponding to the selected image in the illumination parameter recording section 20.
6, the optimal illumination intensity is calculated, the distance at which the image was captured is read from the distance recording unit 205, and the optimal illumination intensity and distance are recorded in the optimal illumination parameter database 24.

These processes are repeated by changing the position where the test chart is provided, and actual measured values of the distance and the optimum illumination intensity at each position as shown in FIG. 17 can be obtained.

Subsequently, the optimum parameter calculating section 212
Performs an interpolation process to interpolate the measured values of the distance and the optimum illumination intensity, and finally, as shown in FIG.
An optimal illumination parameter database 24 in which the optimal illumination intensity is recorded corresponding to the distance is created.

Next, the flowchart will be described. Step S1 A test chart is provided. Step S2 Distance recording processing is performed. Step S3: A lighting control process is performed. Step S4: An image photographing / recording process is performed. Step S5: It is determined whether the illumination control processing and the image capturing / recording processing have been completed for all the illumination intensities. If the illumination control process and the image capturing / recording process have been completed for all the illumination intensities, the process proceeds to step S6,
If not, the process returns to step S3. Step S6: Perform an optimal illumination parameter determination process. Step S7 The optimum illumination parameter recording process is performed, and the process returns to step S1.

Next, a method of creating the optimum threshold value and the optimum threshold value database 25 will be described.

FIG. 19 is a diagram showing a luminance histogram of a difference image of the threshold processing according to the first embodiment of the present invention, and FIG. 20 is a first diagram showing a result of the threshold processing according to the first embodiment of the present invention. FIG. 21 is a second diagram showing the result of threshold processing in the first embodiment of the present invention, and FIG. 22 is a third diagram showing the result of threshold processing in the first embodiment of the present invention. In FIG. 19, the horizontal axis represents the luminance difference, and the vertical axis represents the frequency. In FIG. 19, the part A mainly represents the area of the background 12 (FIG. 1), and the part B mainly represents the area of the target object 11.

First, an optimal threshold for performing the difference / threshold processing will be described. For example, when the luminance histogram of the difference image between the illuminated image and the non-illuminated image shown in FIG. 19 is obtained, the threshold value T1 may be used depending on the target object extraction method.
1, T22 and T33 are optimal. And the threshold value T11
If the target object candidate area shown in FIG. 20 performs the threshold processing at the threshold T22, the target object candidate area shown in FIG. 21 performs the threshold processing at the threshold T33. The indicated target object candidate area is formed.
The target object candidate area shown in FIG. 21 is substantially equal to the target object candidate area when the luminance histogram shown in FIG. 6 is obtained.

In this case, when the threshold processing is performed at the threshold value T22, as shown in FIG. 21, the portion of the background 12 is hardly included, and the portion of the target object 11 is almost included. On the other hand, when the threshold processing is performed with the threshold T11, the portion of the target object 11 does not have a chip as shown in FIG. Further, when the threshold processing is performed at the threshold T33, as shown in FIG. 22, the background 12 is hardly included, but the target object 11 is slightly chipped.

Here, in the extraction processing of the target object extraction unit 209, even if the background 12 is slightly included,
In the case where it is better for the part of the target object 11 to have less chipping, the threshold value T11 is the optimum threshold value. Conversely, if it is better not to include the background 12 even if the target object 11 is slightly missing, the threshold value T33 is the optimal threshold value, and the target object 11 is less missing, and If it is better not to include the portion 12, the threshold value T22 is the optimum threshold value.

As described above, the optimum threshold value is the threshold value T11.
Can extract most of the portion of the target object 11 as shown in FIG. 1, can remove most of the portion of the background 12 as in the threshold T33, and can reduce the portion of the background 12 as in the threshold T22. And some of the target object 11 can be largely extracted.

By the way, the luminance histogram of the difference image as shown in FIG. 19 is obtained by changing the intensity of the illumination in accordance with the position of the target object 11 as shown in FIGS. Changes to Therefore,
It is necessary to determine an optimal threshold based on a difference image between the illumination image and the non-illumination image of the background 12 and the target object 11 when the illumination intensity is changed.

Next, the operation of the optimum threshold database creation processing will be described.

FIG. 23 is a flowchart showing the operation of the first optimum threshold value database creation process in the first embodiment of the present invention, and FIG. 24 is the second optimum threshold value database creation process in the first embodiment of the present invention. FIG. 25 is a flowchart showing the operation of the process, FIG. 25 is a flowchart showing the operation of the third optimum threshold value database creation process in the first embodiment of the present invention, and FIG. 26 is the optimal illumination in the first embodiment of the present invention. And FIG. 27 is a diagram illustrating an example of an optimum threshold database according to the first embodiment of the present invention.

In this case, FIG. 23 uses the threshold T11 shown in FIG.
FIG. 24 shows the processing procedure for creating the optimal threshold database 25 using the threshold T33 as the optimal threshold, and FIG. 25 shows the optimal procedure using the threshold T22 as the optimal threshold. The procedure of processing when creating the database 25 will be described.

First, when creating the optimal threshold database 25 using the threshold T11 as an optimal threshold, the illumination control unit 21 performs a first illumination control process, turns off the illumination unit 201, and the imaging control unit 22 A non-illuminated background image photographing / recording process is performed, and a background 12 having no target object 11 etc.
2, and the captured image of the background 12 is recorded in the image recording unit 207 as a non-illuminated background image.

Next, the illumination control section 21 performs a second illumination control process, and the optimal illumination parameter database 24
Is read out, and the illumination unit 201 is turned on with the optimal illumination intensity, and the imaging unit 202 is turned on.
Performs a lighting background image photographing / recording process, photographs the background 12, and records the photographed image of the background 12 in the image recording unit 207 as a lighting background image. Subsequently, an optimum threshold value calculation processing unit (not shown) of the optimum parameter calculation unit 212
An optimum threshold value calculation process is performed, and a difference process is performed between the non-illuminated background image and the illuminated background image recorded in the image recording unit 207 to create a luminance histogram. Since the luminance histogram generally has a distribution as shown in a portion A in FIG. 19, the threshold value T11 is determined based on a feature in the distribution of the luminance histogram, for example, an average value.

Subsequently, the optimum threshold value calculation processing means performs an optimum threshold value recording process, and records the threshold value T11 as an optimum threshold value in the optimum threshold value database 25 together with the predetermined optimum illumination intensity.

In this way, the above-mentioned processing is performed for all the optimal illumination intensities recorded in the optimal illumination parameter database 24, and an optimal threshold value is calculated for each optimal illumination intensity. 26 is recorded in the optimal threshold database 25, it is possible to obtain actual values of the optimal illumination intensity and the optimal threshold as shown in FIG.

Subsequently, the optimum parameter calculating section 212
Performs an interpolation process to interpolate the actual value of each optimal illumination intensity and the optimal threshold value, and finally, as shown in FIG. 27, the optimal threshold value is set in correspondence with each optimal illumination intensity. The recorded optimum threshold database 25 is created.

Next, the flowchart will be described. Step S11 A first illumination control process is performed. Step S12: A non-illuminated background image photographing / recording process is performed. Step S13 A second illumination control process is performed. Step S14: Illumination background image photographing / recording processing is performed. Step S15: Perform an optimum threshold value calculation process. Step S16: Perform an optimal threshold recording process, and
Return to 11.

Next, when creating the optimal threshold database 25 using the threshold T33 as the optimal threshold, first, the device control unit 204 reads one of the distances recorded in the optimal illumination parameter database 24. Then, the operator arranges the test chart at the position of the distance in front of the imaging unit 202.

Subsequently, the illumination control unit 21 performs a first illumination control process, turns off the illumination unit 201, and turns off the imaging control unit 22.
Performs a non-illuminated image capturing / recording process, captures a test chart by the image capturing unit 202, and records the captured image of the test chart in the image recording unit 207 as a non-illuminated image.

Next, the illumination control unit 21 performs a second illumination control process, and executes an optimal illumination parameter database 24.
Read out the optimal intensity of illumination corresponding to the distance recorded in the, lighting the illumination unit 201 with the optimal intensity of illumination,
The imaging unit 202 performs an illumination image capturing / recording process, captures a test chart, and records the captured image of the test chart in the image recording unit 207 as an illumination image. Subsequently, the optimum threshold value calculation processing unit performs an optimum threshold value calculation process, performs a difference process between the non-illuminated image and the illuminated image recorded in the image recording unit 207, and creates a luminance histogram of only the test chart portion. I do. As described above, since the luminance histogram has a distribution as shown in a part B in FIG. 19, the threshold value T33 is determined based on a characteristic in the distribution of the luminance histogram, for example, an average value.
To determine.

Subsequently, the optimum threshold value calculation processing means performs an optimum threshold value recording process, and records the threshold value T33 as the optimum threshold value in the optimum threshold value database 25 together with the predetermined optimum illumination intensity.

As described above, the test chart is placed at all the distances recorded in the optimal illumination parameter database 24, and the optimal threshold value is calculated for each distance and each optimal illumination intensity, and the optimal illumination intensity is calculated. By recording the optimum threshold value and the optimum threshold value in the optimum threshold value database 25, it is possible to obtain the measured values of the respective optimum illumination intensities and the optimum threshold values similar to those shown in FIG.

Subsequently, the optimum parameter calculating section 212
Performs an interpolation process, interpolates the measured values of the optimal illumination intensity and the optimal threshold, and finally creates an optimal threshold database 25 similar to that shown in FIG.

Next, the flowchart will be described. Step S21 A test chart is provided. Step S22: A first illumination control process is performed. Step S23: A non-illuminated image shooting / recording process is performed. Step S24 A second illumination control process is performed. Step S25: An illumination image photographing / recording process is performed. Step S26: Perform an optimum threshold value calculation process. In step S27, an optimum threshold recording process is performed.
Return to 21.

Then, when creating the optimum threshold database 25 using the threshold T22 as the optimum threshold, first, the lighting control unit 21 performs a first lighting control process, and
Turn off the light. Further, the imaging control unit 22 performs a non-illuminated background image photographing / recording process, and executes a background 12 without the target object 11 or the like.
Is captured by the imaging unit 202, and the captured image of the background 12 is recorded in the image recording unit 207 as a non-illuminated background image.

Next, the illumination control section 21 performs a second illumination control process, and executes an optimal illumination parameter database 24.
Is read out, and the illumination unit 201 is turned on with the optimal illumination intensity, and the imaging unit 202 is turned on.
Performs a lighting background image photographing / recording process, photographs the background 12, and records the photographed image of the background 12 in the image recording unit 207 as a lighting background image.

Subsequently, the device control unit 204 reads the distance corresponding to the optimal illumination intensity recorded in the optimal illumination parameter database 24. Then, the operator arranges the test chart at the position of the distance in front of the imaging unit 202.

Next, the illumination control unit 21 performs a third illumination control process, turns off the illumination unit 201, and sets the imaging control unit 22
Performs a non-illuminated image capturing / recording process, captures a test chart by the image capturing unit 202, and records the captured image of the test chart in the image recording unit 207 as a non-illuminated image.

Subsequently, the illumination control unit 21 performs a fourth illumination control process, turns on the illumination unit 201 with the optimal illumination intensity read out in the second illumination control process, and Then, an illumination image photographing / recording process is performed, a test chart is photographed, and an image of the photographed test chart is recorded in the image recording unit 207 as an illumination image.

The optimum threshold value calculation processing means performs an optimum threshold value calculation process, and outputs the non-illuminated background image recorded in the non-illuminated background image photographing / recording process, the illumination background recorded in the illuminated background image photographing / recording process An optimum threshold value is calculated based on the image, the non-illuminated image recorded in the non-illuminated image shooting / recording process, and the illumination image recorded in the illuminated image shooting / recording process. For example, the distribution of the luminance histogram of the difference image between the non-illuminated background image and the illuminated background image shown by the part A in FIG. 19, and the luminance histogram of the difference image between the non-illuminated image and the illuminated image shown by the part B in FIG. (For example, Mahalanobis distance of two luminance histogram distributions), the threshold T22 is determined.

Subsequently, the optimum threshold value calculation processing means performs an optimum threshold value recording process, and records the threshold value T22 as an optimum threshold value in the optimum threshold value database 25 together with the predetermined optimum illumination intensity.

In this manner, the test chart is placed at all the distances recorded in the optimum illumination parameter database 24, and the optimum threshold value is calculated for each distance and each optimum illumination intensity, and the optimum illumination intensity is calculated. By recording the optimum threshold value and the optimum threshold value in the optimum threshold value database 25, it is possible to obtain the measured values of the respective optimum illumination intensities and the optimum threshold values similar to those shown in FIG.

Subsequently, the optimum parameter calculating section 212
Performs an interpolation process, interpolates the measured values of the optimal illumination intensity and the optimal threshold, and finally creates an optimal threshold database 25 similar to that shown in FIG.

As described above, the optimum threshold value calculation processing means performs processing based on at least one of the non-illuminated background image and the non-illuminated image and the image when at least one of the illuminated background image and the illuminated image is illuminated. Calculate the optimal threshold.

By the way, the optimum threshold calculated in the first to third optimum threshold database creation processing is an abnormal value, and may be inappropriate for extracting the target object 11. For example, when the optimal threshold is too small (very close to 0 in 256 gradations), the shooting environment is extremely bright, and there is no difference in luminance between when the lighting unit 201 is turned on and when it is turned off. Also, the optimal threshold is too large (25
(Very close to 255 in 6 gradations)
The distance from the object to the background 12 is extremely short.

In such a case, since the target object candidate area cannot be formed, it is determined that the optimum threshold is an abnormal value, and the warning output unit 213 outputs a warning that the optimum threshold is inappropriate. .

In the present embodiment, the optimum threshold value is obtained for all the optimum illumination intensities recorded in the optimum illumination parameter database 24. It is also possible to create the optimum threshold database 25 by obtaining an optimum threshold value for the optimum illumination intensity of the above and performing an interpolation process on the obtained optimum threshold value.

As described above, in the present embodiment, each optimum illumination intensity can be obtained in accordance with the distance from the imaging section 202 to the target object 11, so that the illumination intensity is made appropriate. be able to.

Then, an optimal threshold value is calculated and set in correspondence with each optimal illumination intensity.
It is not necessary to repeat the photographing of the illuminated image and the non-illuminated image and the difference / threshold processing until the image is extracted. Therefore, the processing time can be shortened, and the target object 11 can be extracted at high speed.

Further, the optimum threshold value is calculated and set in correspondence with each of the optimum illumination intensities.
1 can always be stably and correctly extracted.

Since the target object 11 is finally extracted by using the model of the target object 11, in any environment, for example, the background 12 has a higher reflectance than the target object 11 in the background 12. Even when there is an object, the target object 11 can be extracted stably and at high speed.

Further, when the optimum threshold value is inappropriate, a warning is output, so that the target object 11 can be correctly extracted.

Next, the flowchart will be described. Step S31 A first illumination control process is performed. Step S32: A non-illuminated background image photographing / recording process is performed. Step S33: A second illumination control process is performed. Step S34: An illumination background image photographing / recording process is performed. Step S35 A test chart is provided. Step S36: A third illumination control process is performed. Step S37: A non-illuminated image photographing / recording process is performed. Step S38: A fourth illumination control process is performed. Step S39: An illumination image photographing / recording process is performed. Step S40: Perform an optimum threshold value calculation process. Step S41: Perform an optimum threshold recording process, and
Return to 31.

Next, a second embodiment of the present invention will be described. In addition, about what has the same structure as 1st Embodiment, the description is abbreviate | omitted by attaching the same code | symbol.

FIG. 28 is a block diagram of a target object extraction image processing apparatus according to the second embodiment of the present invention.

In this case, for example, as in the case of extracting a person in a car, the target object 11 is located at a substantially constant distance from the imaging unit 202. Since the position of the target object 11 is constant, the target object extraction image processing apparatus 100 does not include a distance measurement unit, a distance measurement control unit, and a distance recording unit. Therefore, the cost of the target object extraction image processing device 100 can be reduced.

Further, since the position of the target object 11 is constant, the optimum illumination parameter database 24 does not correspond to a change in distance, but records only one optimal illumination intensity. Then, along with this, one optimal threshold is recorded in the optimal threshold database 25 in correspondence with one optimal illumination intensity.

In the first and second embodiments, for example, in the target object extraction phase, if the photographing environment changes significantly due to sunlight or the like, the illumination image becomes too bright or too dark. Target object 1
1 is too bright or too dark, and the target object 11
The image quality of the image is deteriorated.

Therefore, a third embodiment in which the image quality of the image of the target object 11 can be checked will be described. In addition, about what has the same structure as 1st Embodiment, the description is abbreviate | omitted by attaching the same code | symbol.

FIG. 29 is a block diagram of a target object extraction image processing apparatus according to the third embodiment of the present invention, and FIG. 30 is a diagram showing an example of an optimum illumination parameter database according to the third embodiment of the present invention. 31 is a diagram illustrating an example of an optimal threshold database according to the third embodiment of the present invention.

In the figure, reference numeral 200 denotes a target object extracted image processing apparatus, and 414 denotes an extracted object image quality check unit. The extracted object image quality check unit 414 operates to detect the target object 11 ( The image quality of the extracted image of the target object 11 is checked so that a good image shown in FIG. 1) can be obtained.

In the present embodiment, the extracted object image quality check unit 414 checks the image quality of the image of the target object 11 by brightness. To this end, the extracted object image quality check unit 414 calculates an average luminance in a predetermined area of the image of the target object 11, and determines whether the image quality is appropriate based on whether the average luminance falls within a preset range. Judge whether or not. If the average luminance does not fall within the range and the illumination image is too bright or too dark and the image quality is not appropriate, the extracted object image quality check unit 4
14 sends an instruction to the imaging control unit 42,
Control. The imaging control unit 42 changes the imaging unit parameters (camera gain and the like), captures the non-illuminated image and the illuminated image again, and then performs the processing of the target object extraction phase to extract the target object 11. At this time, the extracted object image quality check unit 414 also sends an instruction to the illumination control unit 21 to control the illumination control unit 21. The illumination control unit 21 changes the intensity of the illumination.

[0105] When the imaging section parameters are changed, the image quality changes, so that the optimum illumination intensity and the optimum threshold value also change. Therefore, the optimal lighting parameter database 24 stores the distance,
The optimal threshold beta base 25 is represented by the imager parameters and the optimal illumination intensity, and the optimal threshold beta base 25 is represented by the optimal illumination intensity, the imager parameters and the optimal threshold, as shown in FIG.

As described above, in the present embodiment, even if the imaging environment changes greatly, the imaging unit parameters are changed, so that the target object 11 can be correctly extracted.

In this embodiment, similarly to the second embodiment, it is possible to omit the distance measuring section, the distance measuring control section and the distance recording section.

In each of the above embodiments, the illumination unit 20
Although visible light is used in No. 1, light in a specific wavelength range such as infrared light other than visible light can also be used. In this case, if the imaging unit 202 can perform imaging only with light in the wavelength range used by the illumination unit 201, the imaging unit 202 is less likely to be affected by environmental light in the imaging environment and can stably extract the target object 11. can do.

In each of the above embodiments, the illumination image is photographed after the non-illumination image is photographed. However, the non-illumination image may be photographed after the illumination image is photographed. In any case, the posture of the target object 11 is determined when the non-illuminated image is captured and when the illuminated image is captured.
If the position or the like is changed, the difference image between the non-illuminated image and the illuminated image cannot be accurately obtained. Therefore, it is better that the interval between the timing of capturing the unilluminated image and the timing of capturing the illuminated image is short .

When there are two or more target objects 11 to be extracted, in the distribution of the luminance histogram, FIG.
9, one region of the background 12 as shown by the portion A is formed, whereas two or more regions of the target object 11 as shown by the portion B in FIG. 19 are formed.
Therefore, when setting the optimum threshold in the parameter data setting phase, the threshold T in the first embodiment is set.
It is preferable to set a threshold value in the region of the background 12 as in the case of setting the threshold value 11, or to set the threshold value in a region of the target object 11 where the luminance difference is the smallest.

The present invention is not limited to the above embodiment, but can be variously modified based on the gist of the present invention, and they are not excluded from the scope of the present invention.

[0112]

As described above in detail, according to the present invention, in a target object extraction image processing apparatus, an illumination unit for illuminating a target object and a background, an imaging unit for photographing the target object and a background, Optimum illumination parameter calculation processing means for calculating an optimal illumination intensity when illuminating the target object and the background, and illuminating at least one of the target object and the background with an illumination intensity lower than the optimal illumination intensity Based on the image at the time, or an image of at least one of the target object and the background captured by turning off the lighting unit, and an image when the target object and the background at least one of the target object and the background are illuminated with the optimal illumination intensity. An optimum threshold value calculating means for calculating an optimum threshold value, and an image or a lighting unit when the target object and the background are illuminated with an illumination intensity lower than the optimal illumination intensity. An image obtained by photographing the target object and the background, an image obtained by illuminating the target object and the background with the optimum illumination intensity, and a difference / threshold processing unit that performs a difference process and a threshold process based on the optimum threshold. A target object extracting unit for extracting a target object based on the result of the difference processing and the threshold processing.

In this case, an image when at least one of the target object and the background is illuminated with an illumination intensity lower than the optimum illumination intensity, or at least one of the target object and the background when the illumination unit is turned off. An optimal threshold is calculated based on an image obtained by photographing the target object and an image obtained by illuminating at least one of the target object and the background with the optimal illumination intensity.

Therefore, the optimum threshold value is calculated corresponding to the optimum illumination intensity, so that it is not necessary to repeat the photographing of the illumination image and the non-illumination image, the difference processing, and the threshold processing until the target object is extracted. The processing time can be shortened. As a result, a target object can be extracted at high speed.

Further, since the optimum threshold value is calculated corresponding to each optimum illumination intensity, the target object can always be stably and correctly extracted.

[Brief description of the drawings]

FIG. 1 is a block diagram of a target object extraction image processing device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a positional relationship between a background and a target object in a conventional target object extraction image processing apparatus.

FIG. 3 is a diagram illustrating an example of a non-illuminated image according to the first embodiment of the present invention.

FIG. 4 is a diagram illustrating an example of an illumination image according to the first embodiment of the present invention.

FIG. 5 is a diagram illustrating an example of a difference image according to the first embodiment of the present invention.

FIG. 6 is a diagram illustrating a luminance histogram of a difference image according to the first embodiment of the present invention.

FIG. 7 is a diagram illustrating a first example of a target object candidate area according to the first embodiment of the present invention.

FIG. 8 is a diagram illustrating a second example of a target object candidate area according to the first embodiment of the present invention.

FIG. 9 is a diagram illustrating a third example of a target object candidate area according to the first embodiment of the present invention.

FIG. 10 is a diagram illustrating a background image when an object having a high reflectance exists in the background according to the first embodiment of the present invention.

FIG. 11 is a diagram showing an illumination image when a target object is present in front of a background according to the first embodiment of the present invention.

FIG. 12 is a diagram illustrating a fourth example of a target object candidate area according to the first embodiment of the present invention.

FIG. 13 is a diagram illustrating a luminance histogram of a difference image when the target object is close to the background according to the first embodiment of the present invention.

FIG. 14 is a diagram illustrating a luminance histogram of a difference image when the intensity of illumination is low in the first embodiment of the present invention.

FIG. 15 is a diagram illustrating a luminance histogram of a difference image when the intensity of illumination is high according to the first embodiment of the present invention.

FIG. 16 is a flowchart illustrating an operation of an optimal illumination parameter database creation process according to the first embodiment of the present invention.

FIG. 17 is a diagram illustrating an example of measured values of a distance and an optimum illumination intensity according to the first embodiment of the present invention.

FIG. 18 is a diagram illustrating an example of an optimal illumination parameter database according to the first embodiment of the present invention.

FIG. 19 is a diagram illustrating a luminance histogram of a difference image of the threshold processing according to the first embodiment of the present invention.

FIG. 20 is a first diagram illustrating a result of the threshold processing according to the first embodiment of the present invention.

FIG. 21 is a second diagram illustrating a result of the threshold processing according to the first embodiment of the present invention.

FIG. 22 is a third diagram illustrating a result of the threshold processing according to the first embodiment of the present invention.

FIG. 23 is a flowchart illustrating an operation of a first optimum threshold value database creation process according to the first embodiment of the present invention.

FIG. 24 is a flowchart showing an operation of a second optimum threshold database creation process according to the first embodiment of the present invention.

FIG. 25 is a flowchart showing an operation of a third optimum threshold value database creation process according to the first embodiment of the present invention.

FIG. 26 is a diagram illustrating an example of actual measurement values of an optimum illumination intensity and an optimum threshold value according to the first embodiment of the present invention.

FIG. 27 is a diagram showing an example of an optimal threshold database according to the first embodiment of the present invention.

FIG. 28 is a block diagram of a target object extraction image processing device according to a second embodiment of the present invention.

FIG. 29 is a block diagram of a target object extraction image processing device according to a third embodiment of the present invention.

FIG. 30 is a diagram illustrating an example of an optimal illumination parameter database according to the third embodiment of the present invention.

FIG. 31 is a diagram illustrating an example of an optimal threshold database according to the third embodiment of the present invention.

[Explanation of symbols]

 10, 100, 200 Target object extraction image processing apparatus 11 Target object 12 Background 24 Optimal illumination parameter database 25 Optimal threshold database 201 Illumination unit 202 Imaging unit 203 Distance measurement unit 205 Distance recording unit 207 Image recording unit 208 Difference / threshold processing unit 209 Target object extraction unit 212 Optimal parameter calculation unit 213 Warning output unit 414 Extracted object image quality check unit

Claims (12)

[Claims] 1. An illumination unit for illuminating a target object and a background, and an imaging unit for photographing the target object and a background.
(C) an optimum illumination parameter calculation processing means for calculating an optimal illumination intensity when the target object and the background are illuminated; and (d) an optimal illumination parameter calculation unit for the target object and the background at an illumination intensity lower than the optimal illumination intensity. An image when illuminating at least one of the images, or an image in which at least one of the target object and the background is captured by turning off the illumination unit, and illuminating at least one of the target object and the background with the optimal illumination intensity (E) an image or a lighting unit when the target object and the background are illuminated with an illumination intensity lower than the optimal illumination intensity based on the image at the time. An image obtained by shooting the target object and the background when the light is turned off, an image obtained when the target object and the background are illuminated with the optimum illumination intensity, and a difference process and a threshold process based on the optimum threshold value And the difference-threshold processing unit that performs, (f)
A target object extraction unit for extracting a target object based on the results of the difference processing and the threshold processing. 2. An illumination unit for illuminating the target object and the background, and an imaging unit for photographing the target object and the background.
(C) an optimal illumination parameter recording unit in which an optimal illumination intensity when illuminating the target object and the background is recorded;
(D) an image when at least one of the target object and the background is illuminated with an illumination intensity lower than the optimal illumination intensity, or at least one of the target object and the background is photographed with the illumination unit turned off. Optimal threshold recording means for recording an optimal threshold calculated based on the image and an image obtained by illuminating at least one of the target object and the background with the optimal illumination intensity; and (e) the optimal illumination When the target object and the background are illuminated with an intensity of illumination lower than the intensity of the image, or when the illumination unit is turned off, the image of the target object and the background is captured, and when the target object and the background are illuminated with the optimal illumination intensity. Image, and a difference / threshold processing unit that performs difference processing and threshold processing based on the optimal threshold,
And (f) a target object extraction unit for extracting a target object based on the results of the difference processing and the threshold processing. 3. The object according to claim 1, comprising: (a) an optimum illumination parameter recording unit for recording the optimal illumination intensity; and (b) an optimal threshold recording unit for recording the optimal threshold. Object extraction image processing device. 4. An apparatus according to claim 1, further comprising: (a) a distance measuring section for measuring a distance from said image pickup section to said target object; and (b) said optimum illumination parameter calculation processing means includes: The target object extraction image processing apparatus according to claim 1 or 3, wherein a strong illumination intensity is calculated for each distance. (5) A distance recording unit for recording a distance from the imaging unit to the target object,
4. The target object extraction image processing apparatus according to claim 2, wherein (b) the optimum illumination parameter recording unit records the optimum illumination intensity when the target object and the background are illuminated for each distance. 5. 6. The image processing apparatus according to claim 1, further comprising an image recording unit that records an image photographed by said imaging unit.
Item extraction image processing apparatus according to Item. 7. The target object extraction apparatus according to claim 1, wherein the target object extraction unit extracts the target object based on a result of the difference processing and the threshold processing and a model of the target object. Image processing device. 8. The target object extraction image processing apparatus according to claim 1, further comprising a warning output unit that outputs a warning when an abnormal value is calculated by the optimum threshold value calculation processing unit. 9. The optimum illumination parameter recording means includes:
4. The target object extraction image processing device according to claim 2, wherein an optimum illumination intensity is recorded in correspondence with a distance from the imaging unit to the target object. 5. 10. The optimum threshold value recording means records an optimum threshold value corresponding to an optimum illumination intensity.
Or the target object extraction image processing apparatus according to 3. 11. The image processing apparatus according to claim 1, further comprising an extracted object image quality check unit that checks whether the image quality of the extracted target object is appropriate, and controls the imaging unit and the illumination unit when the image quality is not appropriate. The target object extraction image processing device according to claim 1. 12. (a) The illumination used in the illumination unit is light in a specific wavelength range such as infrared light, and (b) the imaging unit captures only the light in the wavelength range used in the illumination unit. The target object extraction image processing apparatus according to any one of claims 1 to 3, wherein the processing is performed.