JP2018036884A - Light source estimation apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a light source estimation device that can estimate light source information in a picked-up image obtained with a single camera by analyzing the picked-up image without using special equipment, such as a spherical mirror.SOLUTION: There is provided a light source estimation device 10 comprising: a recognition part 2 that specifies an object picked up in a picked-up image and recognizes a positional attitude of the object with respect to the camera that obtains the picked-up image; a correction part 3 that corrects an area of the object specified in the picked-up image from a state where the picked-up image is seen in the recognized positional attitude to a state where the picked-up image is seen in the predetermined positional attitude in a predetermined reference image when an image of the specified object is picked up with a camera in the predetermined positional attitude in a predetermined light source state, so as to obtain a corrected area; and an estimation part 4 that estimates information on a light source with respect to the object picked up in the picked-up image by comparing the corrected area with a predetermined reference image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a light source estimation device and a program for estimating light source information from a captured image.

  A method of recognizing an imaging target from a captured image and presenting information according to the recognition result is known, such as augmented reality (AR). In particular, for example, when the method is applied to commercial use, and furniture or accessories are presented as virtual objects as presentation information, an image that is inferior to an image obtained when the actual object is present is required. At this time, there is a problem that when the imaging target and the presentation information are greatly different in color and shadow, the user feels uncomfortable.

  As a technique for reducing the sense of incongruity, there is a technique for reflecting the influence of the same virtual light source as the actual light source in the presentation information, but it is necessary to estimate the actual light source in the imaging environment. The following is a conventional technique related to the light source estimation.

  Patent Document 1 discloses a method of estimating light source position information in the surrounding environment based on the principle of triangulation using a plurality of imaging units. Patent Document 2 discloses a technique for capturing ambient light at a plurality of positions assumed in advance and obtaining ambient light information on a photographing object installed at an arbitrary position. Non-Patent Document 1 discloses a method of estimating a light source direction from a high luminance region reflected on a mirror surface by installing a specular sphere on a part of an imaging target.

JP-A-2015-210623 JP 2013-236215 A

Kanbara, Yokoya, Vision-based Augmented Reality, Information Technology Letters 1, 127-128, 2002 considering optical consistency by real-time estimation of light source environment map

  However, the above prior art has the following problems.

  Patent Document 1 requires a plurality of image pickup units, and further requires a plurality of image pickup units to be sufficiently separated in order to increase the estimation accuracy of the light source position.

  Patent Document 2 has a problem in that it cannot be used in a place that has not been photographed in advance because it is necessary to photograph the light source in advance. In addition, there is a problem that it cannot be applied even in a pre-photographed environment if the light source position changes. Furthermore, in order to increase the estimation accuracy of the light source position, there is a problem that a considerable number of images must be taken.

  Non-Patent Document 1 solves a part of the above-mentioned problem because the light source direction is estimated in real time using a single imaging unit. However, since it is necessary to install a special jig called a specular sphere, there is a problem that applicable scenes are limited. In addition, since the reflected light of the specular sphere is too bright, it exceeds the upper limit of the brightness that can be captured regardless of the light source color (out-of-white), and there is a problem that the light source color cannot be estimated.

  In view of the above-described problems of the prior art, the present invention is capable of estimating light source information in a captured image by analyzing a captured image obtained by a single camera without using special equipment such as a spherical mirror. An object is to provide an estimation device and a program.

  In order to achieve the above object, the present invention is a light source estimation device that identifies a target imaged from a captured image and recognizes the position and orientation of the identified target relative to the camera that acquired the captured image. From the state of being recognized in the recognition unit and the recognized position and orientation, it is seen in the predetermined position and orientation in a predetermined reference image when the specified object is imaged in a predetermined light source state with a camera of the predetermined position and orientation. The correction unit that corrects the specified target region in the captured image to obtain a correction region, and compares the correction region with the predetermined reference image, thereby capturing the image in the captured image. It is a 1st characteristic to provide the estimation part which estimates the light source information with respect to the object currently performed.

  Further, the present invention is a light source estimation device, wherein the estimation unit considers a predetermined normal map that reflects a surface shape of the identified target, and the correction region and the predetermined reference image The second feature is to estimate light source information for the object being imaged in the captured image by comparing.

  Furthermore, a third feature of the present invention is a program that causes a computer to function as the light source estimation device.

  According to the first feature, the light source information is obtained by the image analysis processing for the captured image of the single camera without requiring special equipment such as a spherical mirror or the captured images from the plurality of cameras. Can be estimated. According to the second feature, the light source information can be estimated in consideration of the normal map.

It is a functional block diagram of the light source estimation apparatus which concerns on one Embodiment. It is a figure for demonstrating a typical example of the recognition target. It is a figure which shows the typical example for demonstrating the process of each part of a light source estimation apparatus. It is a figure which shows the example of a recognition target. It is a figure which shows the example of a response | compatibility of the feature point coordinate for estimating a position and orientation. It is a flowchart of a process of an estimation part. It is a figure which shows the relationship between the vectors for calculating a light source direction. It is a figure which shows the typical example of the intersection between the straight lines for estimating a light source position in two dimensions. It is a figure which shows the example at the time of giving a normal map etc. with respect to the actual shape of recognition object. It is a figure which shows the example which expands a recognition target to a solid. It is a functional block diagram of the superposition device concerning one embodiment. It is a figure which shows an example of the hardware constitutions of the apparatus which can implement | achieve a light source estimation apparatus or a superimposition apparatus.

  FIG. 1 is a functional block diagram of a light source estimation apparatus according to an embodiment. The light source estimation device 10 includes an imaging unit 1, a recognition unit 2, a correction unit 3, an estimation unit 4, and a storage unit 5. The estimation unit 4 further includes an ambient light estimation unit 41 and a proximity light source estimation unit 42, and a storage unit 5 Further includes a feature information storage unit 51 and an appearance information storage unit 52.

  The light source estimation device 10 recognizes a predetermined recognition target having a normal distribution on the surface (normal distribution at each point on the surface determined according to the surface shape) from the captured image, and then recognizes the recognition in the captured image. Light source information for the object can be output.

  FIG. 2 is a diagram for explaining a typical example of a recognition target that can output light source information by being analyzed by the light source estimation device 10 separately from [1] to [3]. In FIG. 2, as shown in [1], a recognition target OB as an example is formed on a plane PL such as the ground in the real world. Arrows DE, DW, DS, and DN mean the east, west, south, and north directions as directions on the plane PL with respect to the recognition target OB, and will be referred to in the following description. .

  A state in which the recognition target OB is photographed from the vertical direction of the plane PL (indicated by the arrow DV) by the camera C shown in [1] in FIG. 2 is shown in [2]. In the recognition target OB shown in [2], four triangles are formed for each square by drawing diagonal lines on each of the 100 squares arranged as a grid of 10 squares by 10 squares. A total of 400 triangles are drawn, which indicates that the normal direction of the recognition target OB is distributed for each area of the triangles. That is, the surface shape of the recognition target OB is a geometric shape in which the edge line (boundary line between the surfaces) is arranged as shown in [2] when viewed from the front. And, as shown in [3], one of the 100 squares OBP expanded in the form of a perspective view similar to [1] is a flat square pyramid OBP, A triangular side surface SE facing DE, a triangular side surface SW facing west direction DW, a triangular side surface SS facing south direction DS, and a triangular side surface SN facing north direction DN. There is a corresponding normal for each of SW, SS, and SN.

  As described above, the recognition target OB handled by the light source estimation device 10 as a target is generally planar from a macro view as shown in [1] in FIG. 2, but from a micro view as shown in [3]. Forms a solid as a change in shape from the plane, and has the property of defining a normal map that is determined according to the surface shape reflecting the solid shape when viewed from the front as in [2]. Is.

  The details are as described below, but the mechanism that enables the estimation is generally as follows. That is, when a light source (proximity light source described later) is irradiated on a recognition target having a normal map as a reflection of the surface shape as shown in FIG. 2, when viewed from the camera C according to the direction of the light source A distinction occurs between the strongly irradiated surface and the weakly irradiated surface (for example, when viewed from the camera C in the example of FIG. 2 and when the light source is irradiated from the south DS, the south surface SS is strong. The direction of the light source and the like can be estimated using the fact that the surface SN on the north side is only weakly irradiated as compared with the case of irradiation.

  More precisely, the distinction between the intensity of irradiation that enables estimation of the direction of the light source and the like is not only the direction of the surface (normal direction) but also the position of the surface in relation to the position and orientation of the camera and the light source. It also depends on (the position of the normal). That is, the distinction between the intensity of irradiation depends on the normal map itself as illustrated in [2] of FIG. 2 in the relationship between the position and orientation of the camera and the light source.

  The example of FIG. 2 is an example for explaining only the “shape” to be recognized. The actual recognition target has an attribute such as “pattern” in addition to the attribute “shape”, and information based on the attribute is stored in the storage unit 5 described later.

  Hereinafter, each part of FIG. 1 will be described. FIG. 3 is a diagram showing a schematic example for explaining the processing content of each part, and will be referred to as appropriate in the following description.

  The imaging unit 1 captures the recognition target and outputs the captured image to the recognition unit 2 and the correction unit 3. As hardware for realizing the imaging unit 1, a normal digital camera can be used.

  In the example of FIG. 3, the recognition target OB is formed in [1] on the plane PL such as the ground similar to [1] in FIG. 2, and the camera C as hardware that implements the imaging unit 1 is the plane PL. The recognition target OB is imaged in the direction DC slightly westward from the vertical direction DV. As will be described in detail below, according to the light source estimation device 10, the irradiation direction DL of the proximity light source NL that irradiates the recognition target OB and its intensity, and the recognition target OB Thus, it is possible to estimate the intensity of the ambient light source EL that achieves uniform irradiation independent of the direction. For example, although the direction DL is slightly southward from the vertical direction DV of the plane PL, information on such a light source can be estimated.

  In the example of FIG. 3, an example of a captured image P obtained by capturing an image in the westward direction DC shown in [1] is shown in [2]. In the captured image P, a shape distorted from a square shape (a square shape when viewed from the front as shown in [2] in FIG. 2) by capturing the recognition target OB in a direction DC inclined from the front. This is taken as a region R.

  The storage unit 5 stores feature information F (i) of each recognition object i (i = 1, 2,..., N; integer i is an identifier of the recognition object) in the feature information storage unit 51, and stores appearance information. Unit 52 stores appearance information AP (i) of each recognition target i (i = 1, 2,... N), and the stored feature information and appearance information are processed by recognition unit 2 and estimation unit 4, respectively. It is output as information required when performing.

  As is clear from the above, the feature information F (i) and appearance information AP (i) of each recognition object i should be stored in the form of a well-known table or the like in the entire storage unit 5. Please be careful. The distinction between the feature information storage unit 51 and the appearance information storage unit 52 is a distinction for convenience in order to clarify how the stored information is used for each recognition target i.

  As the feature information stored in the feature information storage unit 51 and output to the recognition unit 2, a local image feature amount extracted from the reference image RP (i) of the recognition target i can be used. Specifically, it has a property that is invariant to any one or any combination of rotation and enlargement / reduction or projective change (distortion due to projective transformation) such as a well-known SIFT feature or SURF feature. A local feature amount calculated based on a relative luminance gradient in a local region of an image can be used. Alternatively, a well-known ORB feature amount having the same property may be used. For example, SIFT feature values are disclosed in Non-Patent Document 2 below, SURF feature values are disclosed in Non-Patent Document 3 below, and ORB feature values are disclosed in Non-Patent Document 4 below.

[Non-Patent Document 2] "DGLowe, Distinctive image features from scale-invariant key points, Proc. Of Int. Journal of Computer Vision (IJCV), 60 (2) pp.91-110 (2004)"
[Non-Patent Document 3] “H. Bay, T. Tuytelaars, and LVGool, SURF: Speed Up Robust Features, Proc. Of Int. Conf. Of ECCV, (2006)”
[Non-Patent Document 4] “Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary R. Bradski: ORB: An efficient alternative to SIFT or SURF. ICCV 2011: 2564-2571.”

  Further, in the feature information stored by the feature information storage unit 51, in addition to the local image feature amount of the reference image RP (i) of the recognition target i, information on the coordinates of the feature points extracted from the local image feature amount is also included. Make it correspond.

  Appearance information that is stored in the appearance information storage unit 52 and output to the estimation unit 4 includes a reference image to be recognized that is configured with a texture and the like, light source information when the reference image is captured, and the reference image It is configured as information in which the normal map in the recognition target is associated. That is, for each individual recognition target i (i = 1, 2,..., N), there is information associated with the reference image RP (i), the light source information L (i), and the normal map N (i), respectively. This is the appearance information AP (i) of i to be recognized.

  Here, as described with reference to FIG. 2, the normal map N (i) is given as a map in the normal direction defined for each pixel position (x, y) of the reference image RP (i). be able to. In addition, the normal direction in the normal map N (i) is a coordinate system in which the z axis is taken in a direction perpendicular to the plane including each pixel position (x, y) of the image of the reference image RP (i) ( x, y, z). That is, as described with reference to FIG. 2, each recognition object i such as the recognition object OB has a shape that rides on a plane, and the plane is defined as an (x, y) plane (plane PL in FIG. 2). A normal map N (i) can be given by giving a normal direction in a coordinate system (x, y, z) taking the z axis in the direction. For example, the normal direction of the horizontal plane can be given as (0,0,1).

  Further, in one embodiment of the estimation method described later, the appearance information AP (i) is configured by further associating the reflectance map RF (i) of the recognition target i. Similarly to the normal map N (i), the reflection map RF (i) can be given as a map of reflectance values defined for each pixel position (x, y) of the reference image RP (i). .

  The recognition target for storing the feature information and appearance information in the storage unit 5 can be any object that can define the feature information and appearance information in a planar shape as described in FIG. For example, as shown in [A] of FIG. 4, a predetermined partial region on the surface of the bamboo work can be set as a recognition target. In the bamboo work, the bamboo wing itself as a constituent element has a textured surface shape, and the bamboo chin is further knitted alternately to form a further surface shape, and the bamboo craft according to the surface shape A surface normal map is constructed. In [B] of FIG. 4, the normal map defined in the appearance information is visualized and expressed with respect to the recognition target as the reference image shown in [A], and (x, y, z) A color image in which (R, G, B) is assigned to the value of is shown (in the example shown, the color display is converted to gray scale for convenience).

  In addition to the examples shown in FIGS. 2 and 4, the concrete recognition target is a road surface or a building that has been subjected to processing such as carpets, tatami mats, embossed ones, coins, tiles or other surfaces with irregularities and inclinations. It is possible to set all or a part of a planar region in an arbitrary object such as a wall surface of an object by an administrator's hand, and store the feature information and appearance information in the storage unit 5.

  Returning to the description of each unit of the light source estimation device 10, the recognition unit 2 performs a process of extracting feature information F (P) from the captured image P input from the imaging unit 1 as a first process. The feature information to be extracted is the same type as the feature information stored in the feature information storage unit 51. For example, when the SIFT feature value is stored in the feature information storage unit 51, the recognition unit 2 extracts the SIFT feature value from the captured image.

  The recognizing unit 2 also performs, as a second process, the feature information F (P) of the extracted captured image P, which of the feature information F (i) of each recognition target i stored in the feature information storage unit 51. Is identified as a recognition target i (i = 1, 2,..., N) stored in advance in the captured image P. .

  The first process and the second process described above can be performed by existing methods disclosed in Non-Patent Documents 2 to 4 and others. For example, regarding the second process, it is possible to determine the degree of coincidence based on the distance between vectors using feature information as a vector quantity. In the first process, feature information F (P) can be extracted from the captured image P including target information other than the recognition target and feature information caused by noise. Therefore, in the second process, known robust estimation is performed. You may make it apply and perform collation processing.

  The recognition unit 2 further recognizes the relative position and orientation of the recognition target (recognition target j) specified in the second processing with respect to the camera (imaging unit 1) in the captured image P as the third processing.

  Recognizing the relative position and orientation in the third process is also possible with the existing method used to obtain “geometrical consistency” in Non-Patent Document 1 and others. That is, after recognizing a square marker that is a standard marker in the AR field, the coordinates on the image of the four corners, and the coordinates on the image when shooting from the front with the square marker as a reference arrangement (reference coordinates), By obtaining a plane projection transformation matrix that mutually converts the square markers, the position and orientation of the square marker can be obtained in the form of the plane projection transformation matrix. In the third process, the relative position and orientation between the identified recognition target j and the camera can be estimated in the same manner as the method.

  In the example of FIG. 3, the captured image P as shown in [2] is obtained by performing imaging at the relative position and orientation of the recognition target OB and the camera C as shown in [1]. By the first process and the second process, it is specified that the region R of the captured image P corresponds to the reference image RP (j) of the recognition target j stored in the storage unit 5 shown in [4]. Further, by the third process, the transformation relationship between the feature point coordinates in the feature information of the region R and the feature point coordinates in the feature information of the reference image RP (j) is obtained in the form of a planar projective transformation matrix. The relative position and orientation of the camera C and the recognition target OB (specified as the recognition target j) as shown in 1] are obtained.

  [2] and [4] in FIG. 5 are schematic examples of the correspondence of the feature point coordinates in the feature information in the second process as examples corresponding to [2] and [4] with the same numbers in FIG. It is a figure which shows an example. That is, in the captured image P, feature points p1, p2, p3, and p4 whose local feature amounts can be calculated in the first process are extracted, and in the second process, these are feature objects whose feature information is stored in the storage unit 5. The feature points rp1, rp2, rp3, and rp4 in j are identified as corresponding to the local feature quantities, and the correspondence between the feature points p1 to p4 and the feature points rp1 to rp4 is obtained. In the third process, the relative position and orientation are recognized by obtaining a plane projection transformation matrix between the corresponding feature point coordinates.

  Note that FIG. 5 shows an example of correspondence between four feature points. As is well known, the calculation of a plane projection transformation matrix requires at least four point correspondences, but there are more than four. A planar projective transformation matrix may be obtained from the feature point correspondence. Even when the feature information F (i) of each recognition target i is stored in the feature information storage unit 51, four or more predetermined feature points may be stored.

  The recognizing unit 2 performs the first to third processes described above, and calculates the recognition result indicating that the recognition target j is obtained in the second process and the relative position and orientation obtained in the third process. To output.

  The correction unit 3 recognizes the recognition target in the captured image P output from the imaging unit 1 based on the recognition result indicating that the recognition target j is obtained from the recognition unit 2 as described above and the position and orientation in the captured image P. By applying a planar projection transformation matrix corresponding to the position and orientation to the region R (j) of j, when the reference image RP (j) of the recognition target j stored in the storage unit 5 is imaged A correction region CR (j) corrected to the same appearance is obtained and output to the estimation unit 4.

  In the example of FIG. 3, the correction region CR (j) shown in [3] is obtained by applying the planar projection transformation matrix to the region R (j) of the captured image P shown in [2]. The obtained correction region CR (j) shown in [3] looks the same as the reference image RP (j) of the recognition target j shown in [4] (for example, it is seen from the front with a predetermined distance away). It has become. Therefore, the correction region CR (j) and the reference image RP (j) have the same shape (square in the example of FIG. 3).

  In the above description and the example of FIG. 3, for the sake of convenience, it is assumed that the region R (j) is identified by recognizing the recognition target j in the captured image P, but more precisely as follows. That is, by applying a planar projection transformation matrix corresponding to the position and orientation to the entire captured image P, the reference image RP (j) is converted to a common (x, y) coordinate system with the reference image RP (j). As a region similar to the region occupied by (), a region in the captured image P that has undergone planar projective transformation is identified as the correction region CR (j). (A region R (j) is determined by reversely obtaining the region of the captured image P before applying the planar projection transformation matrix corresponding to the identified correction region CR (j).)

  The estimation unit 4 stores the correction area CR (j) obtained by the correction unit and the appearance information corresponding to the correction region CR (j) obtained as indicated by a bidirectional arrow between [3] and [4] in FIG. The light source information reflected in the recognition target j in the captured image P is estimated by comparing the appearance information AP (j) (including the reference image RP (j)) of the recognition target j read from the unit 52 .

  In the example of [3] (and [2]) in FIG. 3, it is assumed that the upper side of the correction region CR (j) is higher in luminance due to the irradiation light of the proximity light source NL shown in [1]. It is drawn as white. The entire correction region CR (j) is also depicted as having higher brightness and whiteness due to the influence of ambient light from the light source EL shown in [1]. As described above, the correction region CR (j) obtained from the captured image P in each of the above processes reflects that the reference image RP (j) is in a different light source environment from when the image was captured. In addition, each pixel position (x, y) is directly corresponding to the pixel position (x, y) of the reference image RP (j) by correcting in the correction unit 3. Therefore, the estimation unit 4 can estimate the light source information by comparing the correction region CR (j) and the reference image RP (j) that correspond to each other in the pixel unit.

  Note that the corresponding pixels may be compared with all the pixels in the region, or may be compared with some pixels by sampling at a predetermined rate.

  FIG. 6 is a flowchart of light source information estimation by the estimation unit 4 according to an embodiment. In step S1, the ambient light estimation unit 41 calculates the ambient light information as a part of the light source information from the value of each pixel corresponding to the reference image RP (j) of the value of each pixel of the correction region CR (j). Then, the process proceeds to step S2. In step S2, the proximity light source estimation unit 42 estimates the information of the proximity light as a part of the light source information, and the flow ends. In one embodiment, the ambient light information obtained in step S1 can be estimated using the ambient light information obtained in step S1.

  First, as a model used in the estimation process, an existing model used for rendering in the field of computer graphics is used as follows. That is, the luminance distribution (that is, the pixel value distribution) in the captured image (and the reference image) is reflected at each point on the surface of the recognition target, and both the ambient light from the ambient light source that reaches the camera and the near light from the proximity light source The sum of the amount of light. Furthermore, if the existing Lambert model is used as the reflection model of the proximity light source, each position (x, y) of the image (both the captured image and the reference image) (that is, each position of the surface to be recognized) The pixel value I (x, y) in is expressed by the following equation (1).

Here, I _a and I _d are the intensities of the ambient light source and the proximity light source, respectively, and therefore these intensities are values that do not depend on the pixel position (x, y). R _a (x, y) and R _d (x, y) are the ambient light of the recognition target j and the reflectance at the proximity light source, and N (x, y) is the surface position (x, y) of the recognition target j. It is a normal vector, and its value is given to each of the reflectance map R (j) and the normal map N (j) constituting the appearance information AP (j) of the appearance information storage unit 52. L (x, y) is a light source direction vector from the position (x, y) of the recognition target j toward the near light source. “·” Between the normal vector N (x, y) and the light source direction vector L (x, y) is an inner product operation of the two vectors and represents a so-called cosine law in the Lambert model.

Here, when the estimation unit 4 estimates light source information according to the model of the formula (1), I (x, y) is a pixel value in each of the correction region CR (j) and the reference image RP (j). Is known as As described above, the reflectances R _a (x, y) and R _d (x, y) and the normal vector N (x, y), respectively, reflect the reflectance map R (j) constituting the appearance information AP (j). And known as the normal map N (j). Therefore, what is estimated as an unknown quantity is the intensity I _a , I _d of each light source and the light source direction vector L (x, y) of the proximity light source.

Further, in accordance with the model of the above equation (1), the appearance information storage unit 52 stores the reference image RP (j) of each recognition target j with ambient light having a predetermined intensity I _a0 and a proximity light source having zero intensity (that is, a proximity light source). It shall be prepared and saved as a photograph taken under a nonexistent state). That is, it is assumed that the relationship of the following expression (2) is known with respect to the pixel value I ₀ (x, y) at each position of the reference image RP (j) by referring to the appearance information AP (j).
I ₀ (x, y) = I _a0 R _a (x, y)… (2)

  According to the above model, when the estimation unit 4 estimates the ambient light and the proximity light source as unknown information, it is possible to take an approach of using known information in a region in which each influence appears remarkably.

Specifically, first, the ambient light estimation unit 41 has I (x, y) = I _a * R _a regarding each pixel value I (x, y) of the correction region CR (j) in an area where the influence of the proximity light source is small. Since (x, y) can be approximated (can be approximated to the same equation as the above equation (2)), in the region Rn where the influence is small, the correction region CR (j) is compared with the appearance information, and the displacement amount is compared with the region. It is calculated as the average of the amount of displacement of the pixel values.

As a first embodiment, when calculating the amount of displacement as a difference between pixel values, it is assumed that the ambient light at the time of obtaining appearance information I ₀ (x, y) as in equation (2) is I _a0 , The ambient light Ia is estimated by the following equation (3). | Rn | is the number of pixels belonging to the region Rn as a denominator for calculating the average.

That is, as in the above equation (3), the ambient light _Ia in the correction region CR (j) is the pixel corresponding to the reference image RP (j) from the pixel value I (x, y) in the correction region CR (j). The value obtained by subtracting the value I ₀ (x, y) and dividing by the reflectance Ra (x, y) is obtained by calculating the average in the region Rn, and estimating by adding the ambient light I _a0 at the time of appearance information generation be able to.

  Alternatively, when the displacement amount is calculated as a ratio as the second embodiment, the ambient light can be estimated by the following equation (4).

That is, as in the above equation (4), the ambient light _Ia in the correction region CR (j) is the pixel corresponding to the pixel value I (x, y) in the correction region CR (j) and the reference image RP (j). It is possible to estimate by multiplying the average of the ratio with the value I ₀ (x, y) in the region Rn by the environmental light I _a0 (included in the appearance information AP (j) as a constant) at the time of appearance information generation. it can. The ambient light intensity I _a0 is included as a constant in the appearance information AP (j) and stored in the appearance information storage unit 52. Although it is obtained as a geometric mean in Equation (4), it may be obtained as a harmonic mean. In the second embodiment, there is an effect that the reflectance _Ra (x, y) of the recognition target j may be omitted.

  In the first and second embodiments described above, the region Rn that is less affected by the proximity light source reaches the upper limit or lower limit of the luminance value in at least one of the correction region CR (j) or the reference image RP (j). What is necessary is just to set as an area | region away from the area | region which is at least a fixed range. It is preferable that the reference image RP (j) is prepared so as not to generate a region reaching the upper limit or the lower limit, and stored in the appearance information storage unit 52. In addition, in equations (3) and (4), the average value of the region Rn is obtained and added or integrated, but other statistical representative values, for example, the median value may be used.

  Next, the proximity light source estimation unit 42 will be described. Hereinafter, estimation of the light source direction (and the light source position depending on the situation) in the proximity light source estimation unit 42 and estimation of the intensity of the proximity light source based on the estimated light source direction will be described in this order.

As shown in Equation (1), the reflected light of the near light source is the inner product L (x, y) · N (x, y) of the light source direction vector L (x, y), the normal vector, and N (x, y) And the light source intensity I _d and the reflectance R _d (x, y). In the proximity light source estimation unit 42, the direction L (x, y) of the proximity light source is changed to the line-of-sight direction F (x, y) from the camera center toward the pixel (x, y) in the region Re where the influence of the proximity light source is large. From the normal information N (x, y) of the pixel position (x, y), the light source direction L (x, y) based on the pixel position (x, y) is obtained by the following equation (5) be able to.

  FIG. 7 shows the relationship between the direction of the light source direction L (x, y), the line-of-sight direction F (x, y), and the normal direction N (x, y) as a basis for deriving the above equation (5). FIG. In FIG. 7, two line-of-sight directions F (x, y) are drawn with (x, y) as an end point and a start point as vectors from the camera center LC to the point (x, y). A light source direction L (x, y) is drawn as a vector heading from one to the position LL of the near light source, and a normal vector N (x, y) at the point (x, y) is drawn. In the relationship shown in FIG. 7, the ray that is reflected from the proximity light source LL (point light source) and reaches the camera center LC at the point (x, y) is known with the normal vector N (x, y) as a boundary. Since the law of reflection (equal to incident angle and reflection angle) holds, each vector L (x, y), F (x, y), N (x, y) is meaningful only in that direction, By giving a constraint that these absolute values are equal (for example, 1), the above (5) is derived.

For the region Re to which the target pixel position (x, y) to which the expression (5) is applied and which has a large influence of the proximity light source, the region Rl that has reached the upper limit of luminance in the correction region CR (j), or threshold determination It can be determined as a region Rd in which the displacement amount is determined to be large. Regarding the amount of displacement, the pixel value I ₀ (x) of the reference image RP (j) from the pixel value I (x, y) of the correction region CR (j), which is the same type as that obtained in the ambient light estimation unit 41 described above. , y) or a ratio thereof can be obtained at each position (x, y). Hereinafter, a region Re that is greatly influenced by the proximity light source and can be defined as both or one of the region Rl or the region Rd is referred to as a highlight region Re.

  Note that it is assumed that the highlight area Re is determined as a continuous area in which the normal directions are approximately the same. There may be a case where a sudden defect in the region occurs. The highlight area Re may be determined as a result of interpolating the sudden missing area using a known missing interpolation technique.

  Further, in equation (5), the line-of-sight direction F (x, y) is based on the image coordinates (x, y) of the reference image RP (j) when the position and orientation are estimated by the third process of the recognition unit 2. Furthermore, the camera center position LC is known in spatial coordinates (x, y, z) based on the reference image RP (j) obtained by taking the z axis in the direction perpendicular to the image plane. Therefore, it can be determined as the direction from the camera center position LC to the point (x, y, 0) in the equation (5).

  As described above, the proximity light source estimation unit 42 can estimate the direction L (x, y) of the proximity light source by the equation (5). Furthermore, even when there are a plurality of proximity light sources, there are highlight regions Re corresponding to the respective light sources, so that the direction of each light source can be estimated by applying Equation (5) for each region.

  Furthermore, depending on the situation when the captured image P is captured, the same proximity light source may be manifested as a plurality of highlight areas Re. In such a case, as an embodiment that can be applied by automatically determining that it has become apparent, the proximity light source estimation unit 42 simply estimates the light source direction L (x, y) as shown in Equation (5). Instead, the three-dimensional coordinates of the proximity light source may be estimated by obtaining the intersections of the light source directions respectively obtained from the plurality of highlight regions Re.

  In the present embodiment, a plurality of highlight areas Re are obtained by the same proximity light source only when a plurality of highlight areas Re are obtained and it is determined that there are intersections in the light source direction in some or all of them. A determination that Re is present can be obtained, and the three-dimensional coordinates of the same proximity light source can be estimated.

  FIG. 8 is a diagram illustrating a schematic example in which the position of the proximity light source can be estimated by matching a plurality of light source directions. In FIG. 8, for the sake of simplicity of explanation, it is assumed that the example of light travel is limited to a two-dimensional cross section, and the plane PL on which the recognition object j is arranged is a two-dimensional cross section as in FIGS. Surfaces s1 to s13 that are drawn in a straight line and appear on the cross section of the recognition target j and each have a normal direction (surfaces that are straight and have a curvature of zero) are drawn. In FIG. 8, when viewed from the camera center LC with respect to the light source position LL of a single proximity light source, three points of the surfaces s2, s7, and s12 are revealed as highlight areas by the single proximity light source. The case of doing is shown as an example. That is, light rays are reflected at the points pr2, pr7, and pr12 according to the law of reflection with the normal lines n2, n7, and n12 on the surfaces s2, s7, and s12 as boundaries, so that the light rays are directly emitted from the proximity light source LL to the camera center LC. There is a relationship to reach. Conversely, in such a case, it is clear that the position of the proximity light source LL can be estimated by searching for the coincidence point of the light source direction from the surfaces s2, s7, s12, and the proximity light source estimation unit 42 performs the estimation Can do.

  In the example of FIG. 8, as an ideal case, the case where straight lines in the light source direction are completely obtained as one point has been described. However, since there are various errors in reality, it is considered that they do not completely intersect at one point. Therefore, in the proximity light source estimation unit 42, whether or not there is an intersection in the two light source directions may be determined by whether or not the shortest distance of the straight line defined by the light source direction falls within a preset threshold TH. Whether or not an intersection exists in three or more light source directions depends on whether or not the shortest distance between the straight lines is within the threshold TH in the combination of all two light source directions selected from the three or more light source directions. What is necessary is just to judge. When it is determined by the threshold determination that there are intersections in two or more light source directions, the position of the proximity light source is a position where the sum of distances to the respective straight lines defined by the two or more light source directions is minimum, etc. What is necessary is just to determine as a predetermined position.

  Further, when the direction (and position) of the proximity light source is estimated by the proximity light source estimation unit 42, a straight line after reflection at the highlight area Re when viewed from the camera position LC is considered as in the example of FIG. In order to determine the straight line, it may be determined at each point (x, y) in the highlight area Re, or only one that passes through one of the representative points (x, y) may be determined. Good. The representative point that passes can be set as a predetermined point in the highlight area Re, for example, the center of gravity. (Note that even when defining a straight line after reflection at each point (x, y) in each highlight region Re, the straight lines after reflection obtained at points close to each other in the normal direction are approximately the same. It is assumed that the normal directions are almost the same in each highlight region Re, and therefore the same direction L (x, y) in one highlight region Re. (The average value may be determined as the light source direction corresponding to the highlight area Re.)

As described above, the proximity light source estimation unit 42 can estimate the direction of the proximity light source and the position depending on the situation. Further, in one embodiment, the proximity light source estimation unit 42 uses the estimated direction L (x, y), and in the surrounding Rs of the corresponding highlight region Re, the direction L (x , y), the intensity I _d of the proximity light source can be estimated.

In Expression (6), the surrounding Rs may be determined as corresponding to the highlight area Re according to a predetermined rule, such as being within a predetermined distance from at least one point of the highlight area Re. | Rs | is the number of pixels belonging to the peripheral Rs. Further, as the ambient light intensity I _a , the one obtained by the ambient light estimation unit 41 can be used, but one obtained by a separate method may be used. As is clear from equation (6), in equation (6), the intensity I _{d of the} proximity light source is estimated as an average in the region Rs of the model of equation (1). That is, according to the model of equation (1), the influence I _a R _a (x, y) of ambient light is subtracted and excluded from the luminance, and R _d (the coefficient integrated with the intensity I _a in equation (1) Equation (6) estimates the intensity I _a of the ambient light by averaging the values obtained by dividing by x, y) (N (x, y) · L (x, y)) in the region Rs.

As an embodiment different from the above equation (6), the proximity light source estimation unit 42 may estimate the intensity I _{d of the} proximity light source as follows. That is, the intensity I _d of adjacent light sources, the area of the peripheral region Rne with normal direction similar to the normal direction with the pixels in the highlight region Re (i.e. the number of pixels in the region Rne | Rne |) and, the It is obtained as a ratio | Rth | / | Rne | with the area of the region Rth having a displacement amount equal to or larger than a predetermined threshold in the region Rne (that is, the number of pixels of the region Rth | Rth |). The “concept” determined by the ratio is that the intensity I _{d of the} proximity light source is proportional to the size of the region having the displacement amount equal to or greater than the threshold value. Similarity in the normal direction may be obtained by threshold determination for the absolute value of the vector difference.

For the displacement amount at each position (x, y), the difference “{I (x, y) −I ₀ (x, y)} / R _a (x , y) ”or the ratio“ I (x, y) / I ₀ (x, y) ”. In the embodiment, the proximity light source estimation unit 42 can further obtain the color of the proximity light source as an average value of the same displacement amount. That is, the same displacement amount is obtained for each intensity in a color space such as RGB, and the color of the proximity light source can be estimated as the average.

The embodiment is different from the above-described embodiment of Equation (6) in that the reflectance R _d (x, y) of the recognition target j can be implemented without being stored in the appearance information storage unit 52. is there. Further, unlike the embodiment of the equation (6), there is an effect that the intensity of the environmental light separately obtained by the “environment light estimating unit 41” is unnecessary.

In the proximity light source estimation unit 52 as described above, in addition to modeling only the influence of the ambient light and the proximity light source in the equation (1), as is apparent from the explanation example of FIG. The effect of specular reflection is taken into account. As for specular reflection, for example, the intensity I _r (x, y) is _expressed by the following formula (7) using the Phong model, which is an existing specular reflection model used for rendering in the field of computer graphics. Can be represented.
I _r (x, y) = r (θ1 (x, y)) I _d cos ⁿ (θ2 (x, y)) (7)

  Here, θ1 (x, y) is the incident angle (or equivalent reflection angle) from the near light source to the point (x, y), and θ2 (x, y) is reflected from the viewing direction F (x, y). And n is a parameter representing the intensity of specular reflection.

In each embodiment of the proximity light source estimation unit 52 as described above, a highlight that can be simply determined by the upper limit of the brightness without using a model that requires various parameters such as the above formula (7) Estimate the light source direction L (x, y) from the region Re, and estimate the intensity I _d of the proximity light source by applying the model of Equation (1), for example, with the surrounding region Rne as the region where the influence of specular reflection has disappeared doing.

  As described above, according to the present invention, a portable terminal or the like can be used for a captured image obtained from only a single camera without requiring special equipment such as a spherical mirror and without requiring a plurality of cameras. The light source information can be estimated by a method that can be processed by a general computer. Hereinafter, supplementary matters in the present invention will be described.

(1) About the setting of appearance information AP (j) for each recognition target j Of the appearance information AP (j), the normal map N (j), in particular, strictly corresponds to the surface shape of the recognition target j. Instead, it can be set by the administrator or the like as reflecting the change in the appearance of the recognition target j in accordance with the camera position and orientation and the light source state that are expected to vary.

  For example, it is assumed that there is a recognition target having surfaces s20 to s26 as surface shapes as illustrated in [1] of FIG. In FIG. 9, the surfaces s20 to s26 are drawn as a two-dimensional section to be recognized, as in the example of FIG. A strict normal map that completely represents the surface shape must be defined in each of the s20 to s26. However, in the present invention, a normal map N (j) corresponding to the reference image RP (j) may be defined. That is, the portions of the surfaces s22, s23, and s24 in [1] of FIG. 9 constitute a narrow and deep vertical groove, but the normal map is obtained for the surfaces s22 and s24 that face the vertical direction. There is no need to define it, and it is only necessary to define the groove portion as a horizontal plane s30 as shown in [2]. Furthermore, since it is a thin and deep groove, it will appear almost black on the image, so the reflectance of the surface s30 may be set low. In the reference image RP (j), the surface s30 appears as a substantially black region.

(2) About each recognition object j being configured as a substantially planar area Each recognition object j in the present invention has a substantially planar area as its surface shape can be fitted to a plane on a predetermined basis. You can set what you want. For example, each point position (xi, yi, zi) (i = 1, 2, ... N) on the surface is extracted from the surface area to be recognized at a predetermined spatial sampling rate, and the series of points is extracted by the least square method. An object whose error when fitting a plane is within a predetermined threshold can be set as the recognition target j. Further, a recognition target j can be set such that the surface area of the recognition target falls within a predetermined size rectangular parallelepiped (flat tile-shaped rectangular parallelepiped).

  In the present invention, as described above, the object to be recognized is generally planar and has a normal map on the surface as described above. Therefore, various objects existing in everyday space can be set as recognition objects as they are, and augmented reality. For example, a special marker or the like is not necessarily required for realizing the above, and there is an effect that the degree of freedom is high.

(3) About expanding the recognition target to a solid For example, consider that a triangular pyramid having three triangular surface areas SP1, SP2, SP3 protruding from the ground as shown in [1] in FIG. . The shape of each surface SP1, SP2, SP3 does not necessarily have to be a complete plane, and has respective normal maps. In this case, in the embodiment in which the planar recognition target described above is set, each of the regions SP1, SP2, and SP3 can be set as a separate recognition target j. As another embodiment, taking into account the three-dimensional positional relationship that each surface area SP1, SP2, SP3 forms a triangular pyramid as in [1], the entire triangular pyramid is considered as a single recognition target. It can also be set as j.

  In the embodiment in which the solid is set as the recognition target, the appearance information AP1, AP2, AP3 in the embodiment for the plane region is set for each of the surfaces SP1, SP2, SP3 constituting the solid, , The information that each surface SP1, SP2, SP3 is each surface of the common recognition target j, and the three-dimensional positional relationship in each recognition surface j (triangular pyramid) of each surface SP1, SP2, SP3 as three-dimensional information, Add to the appearance information and set. For the processing of each part of the light source estimation device 10, the following additional processing may be performed.

  In the recognition unit 2, each surface is individually recognized by the feature information of each surface SP1, SP2, SP3 constituting the solid, and two or more of the surfaces SP1, SP2, SP3 are detected in the second process. If recognized, the position and orientation of the object to be recognized as a solid based on the one that minimizes the error (error in well-known matrix calculation) when obtaining the planar projective transformation matrix to represent the position and orientation in the third process Is estimated. Then, other recognized surfaces with no error are also referred to by referring to the spatial information (x, y, z) in the estimated position and orientation with reference to the stereoscopic information included in the appearance information. To grasp.

  The correction unit 3 corrects the recognized three-dimensional surfaces SP1, SP2, and SP3 so as to be in front (direction of the reference image on each surface). Therefore, for example, information such as a “development diagram” as shown in [2] of FIG.

  In the estimation unit 4, in consideration of the spatial arrangement relationship of each surface SP1, SP2, SP3 grasped by the recognition unit 2, all the recognized surfaces are integrated, and the above embodiment and Similarly, light source information is estimated. Therefore, for example, the normal direction N (x, y), the light source direction L (x, y), etc. are expanded as N (x, y, z), L (x, y, z) in the grasped spatial arrangement. Will be defined. For example, when the surface SP1 is determined as the minimum error in the recognition unit 2 and becomes a reference for determining the spatial coordinates (x, y, z), the normal direction and the light source direction appearing on the surface SP1 are N (x, y, 0 ), L (x, y, 0), which is on the reference (x, y) plane and whose vertical direction z is 0, other normal direction and light source direction appearing on the surface SP2, for example, In general, it becomes out of the reference (x, y) plane, such as N (x, y, z), L (x, y, z). In addition, the vector values in the normal direction and the light source direction are also corrected to reflect that the surface SP1 is a horizontal plane and the surface SP2 is inclined from the value originally defined as the horizontal plane SP2. It will be used.

  However, in the plane SP2 inclined with respect to the reference plane SP1 as described above, it is necessary to consider the spatial arrangement of the vector direction, but when performing calculations such as equations (2) to (4), The inclined surface SP2 as shown in [2] in FIG. 10 is also considered to be the front in the development view, that is, the point (x, y) on the common two-dimensional map. Just do it.

(4) Separation of position / orientation recognition function and light source information recognition function for recognition target j In the above description, feature information can be extracted from each recognition target j, and the position / orientation with respect to the camera is recognized based on the feature information. The light source information is estimated for the same region from which the feature information is extracted. However, these may be separated after associating as follows.

  That is, in the recognition target j, for example, a square marker (black frame) portion known in the AR field is provided as a region for identifying the recognition target j and recognizing its position and orientation, and a region separate from the region, for example, the black frame Reference image RP (j) in which a normal map N (j) and others are defined for estimating light source information inside a square marker or a planar predetermined region whose position and orientation is determined with reference to the black frame You may make it provide as an area | region corresponding to. These relationships may be stored in the storage unit 5. Accordingly, the local image feature quantity is not necessarily used to estimate the position and orientation. For example, after determining the distorted shape and orientation of the square marker by binarizing the region, the planar projection transformation matrix similar to the above is used. Position / orientation estimation may be performed.

(5) About Color Space The intensity calculation of each embodiment of the estimation unit 4 described above may be performed for each component of the color space when the image is configured in a predetermined color space such as RGB. it can. It can also be implemented for each component or any component in any other predetermined color space. Moreover, it can also convert into a gray scale image and can implement in the said single component.

(6) Superimposition Processing Using the above estimation result by the light source estimation device 10, it is possible to perform superimposition processing by reflecting light source information on a predetermined virtual object or the like in the captured image P. FIG. 11 is a functional block diagram of the superimposing device 20 that realizes the superimposing processing. The superimposing device 6 and the superimposing information storage unit 53 are further added to the same units as those of the light source estimating device 10 described in FIG. 10 is configured. Since the operation of each part other than the additional configuration is the same as described above, description thereof is omitted.

The superimposing unit 6 inputs the light source information estimated by the estimating unit 4, the superimposing information stored in the superimposing information storage unit 53, and the captured image P output from the imaging unit 1, and superimposing the light source information. Information is generated, superimposed on the captured image P, and output as a superimposed image. The superimposition information storage unit 53 stores the superimposition information corresponding to the recognition target j, and outputs the superimposition information corresponding to the recognition target recognized by the recognition unit 2 to the superposition unit 6. The superimposing process in the superimposing unit 6 can be performed by an existing rendering process by using a model similar to the model used when the estimating unit 4 estimates the light source information. When using a Phong model, using near light source intensity I _d determined as the intensity of the light source, as a result of rendering on the basis of the specular reflection coefficient of the model, it is possible to obtain a specular reflection intensity.

(7) Setting of upper limit or lower limit value of luminance In the processing of the estimation unit 4, it is used as a criterion for determining that the upper limit or lower limit of the luminance has been reached (for example, determination of the highlight area Re due to reaching the upper limit). The upper limit or lower limit may be set as a predetermined range of pixel values (for example, when expressing a pixel with 8 bits, upper limit 255 (out-of-white state), lower limit 0 (bright black state)), As additional information for the appearance information AP (j), a value defined for each pixel position (x, y) of the reference image RP (j) (a value defined as a map similar to the normal map N (j)) It may be set as.

(8) Hardware Configuration FIG. 12 is a diagram illustrating an example of a hardware configuration of devices capable of realizing the light source estimation device 10 illustrated in FIG. 1 and the superimposing device 20 illustrated in FIG. The device 30 receives input from a user such as a CPU (Central Processing Unit) 31, a main storage device 32 that provides a work area for the CPU 31, an auxiliary storage device 33 that can be configured with a hard disk, an SSD, and the like, a keyboard, a mouse, a touch panel, and the like. An input interface 34 for receiving, a communication interface 35 for connecting to a network for communication, a display 36 for displaying, a camera 37, and a bus B for connecting them are provided.

  The processing of each unit of the light source estimation device 10 shown in FIG. 1 and the superimposing device 20 shown in FIG. 11 can be realized by the CPU 31 that reads and executes a program for executing the processing. It may be realized in a dedicated circuit not shown in FIG. The imaging unit 1 can be realized by the camera 37 as the hardware.

(9) Modification of Configuration Arbitrary portions of the components of the light source estimation device 10 shown in FIG. 1 and the superimposing device 20 shown in FIG. 11 may be realized as functions realized in an external device. For example, in the configuration of the superimposing device 20, only the imaging unit 1 is mounted on the user terminal, the other functional units are mounted on the server, and the captured image is transmitted from the user terminal to the server. The superimposed image may be returned to the user terminal 1 and the superimposed image may be displayed on the user terminal. Further, the imaging unit 1 in the light source estimation device 10 may be omitted, and the subsequent processing may be realized by receiving the captured image in the recognition unit 2.

  DESCRIPTION OF SYMBOLS 10 ... Light source estimation apparatus, 1 ... Imaging part, 2 ... Recognition part, 3 ... Correction part, 4 ... Estimation part, 41 ... Ambient light estimation part, 42 ... Proximity light source estimation part, 5 ... Storage part, 51 ... Feature information storage Part, 52 ... Appearance information storage part

Claims (12)

A recognition unit that identifies a target imaged from a captured image and recognizes a position and orientation of the identified target with respect to the camera that acquired the captured image;
From the state of being seen in the recognized position and orientation to the state of being seen in the predetermined position and orientation in a predetermined reference image when the identified object is imaged in a predetermined light source state with a camera of the predetermined position and orientation A correction unit that corrects the specified target area in the captured image to obtain a correction area;
A light source estimation device comprising: an estimation unit configured to estimate light source information for the imaged target in the captured image by comparing the correction region with the predetermined reference image.   The estimation unit considers a predetermined normal map in which a surface shape of the identified target is reflected, and compares the correction region with the predetermined reference image, thereby capturing the image in the captured image. The light source estimation apparatus according to claim 1, wherein light source information for a target subject is estimated.   The surface shape of the identified object is a shape that can be fitted to a plane region on a predetermined basis, has a surface shape as a change from the plane region, and the predetermined normal map The light source estimation apparatus according to claim 2, wherein the light source estimation apparatus is a normal map in which a direction is defined as a reference.   The estimation unit compares pixels corresponding to positions in the correction area and the predetermined reference image, and shifts from a value of the first pixel of the predetermined reference image to a value of the second pixel of the correction area. The light source estimation apparatus according to claim 1, wherein the light source information as a displacement from the predetermined light source state is estimated by calculating a quantity.   The estimation unit is configured to capture the captured image based on an average of the displacement amounts obtained by excluding the second pixel having a predetermined upper limit value or lower limit value from the displacement amount calculation target. The light source estimation apparatus according to claim 4, wherein the intensity of the ambient light at is estimated. The estimation unit considers a predetermined normal map in which a surface shape of the identified target is reflected, and compares the correction region with the predetermined reference image, thereby capturing the image in the captured image. The light source information for the target
The estimation unit compares pixels corresponding to positions in the correction area and the predetermined reference image, and shifts from a value of the first pixel of the predetermined reference image to a value of the second pixel of the correction area. The amount of the ambient light and / or the proximity light source in the captured image or the direction of the proximity light source is estimated based on the amount of displacement and the predetermined normal map. The light source estimation apparatus according to any one of 1 to 5.   The estimation unit compares pixels corresponding to positions in the correction area and the predetermined reference image, and shifts from a value of the first pixel of the predetermined reference image to a value of the second pixel of the correction area. The amount is calculated for each component of a predetermined color space in the pixel, and the intensity of the light source in the captured image is estimated for each component of the predetermined color space based on the amount of displacement. A light source estimation device according to claim 1. The estimation unit includes
Among the second pixels in the correction area, the value has reached a predetermined upper limit value, or
The pixels corresponding to the positions in the correction area and the predetermined reference image are compared, and a displacement amount from the value of the first pixel of the predetermined reference image to the value of the second pixel of the correction area is determined. If the amount of displacement exceeds a predetermined threshold,
8. The apparatus according to claim 1, wherein it is estimated that it belongs to a highlight area where the influence of the proximity light source in the captured image is large, and the intensity and / or direction of the proximity light source in the captured image is estimated based on the highlight area. The light source estimation device according to 1. The estimation unit considers a predetermined normal map in which a surface shape of the identified target is reflected, and compares the correction region with the predetermined reference image, thereby capturing the image in the captured image. The light source information for the target
A direction of a proximity light source in the captured image is estimated based on a direction of the predetermined normal map in the highlight area and a line-of-sight direction from the camera position where the captured image is acquired toward the highlight area. Item 9. The light source estimation device according to Item 8.   The estimation unit is a case where a plurality of the highlight areas are estimated, and the straight lines that are directed from the estimated highlight areas toward the estimated proximity light source are mutually connected. The light source estimation apparatus according to claim 9, wherein when there is an object that is determined to intersect, the position that intersects each other is estimated as the position of the proximity light source.   The light source estimation apparatus according to claim 8, wherein the estimation unit estimates an intensity of the proximity light source from a peripheral region of the highlight region.   A program for causing a computer to function as the light source estimation device according to any one of claims 1 to 11.