JP2019191175A - Imaging device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device and method capable of performing imaging with higher accuracy than before even with a non-uniform subject surface with a simpler configuration than before. An imaging device that captures transient subsurface scattered light in an object to be observed and images the internal light propagation, and a predetermined observation surface point Po on an object surface 10S. , A light source 1 for illuminating at least one surface irradiation point Pp separated by a radius, an image sensor 2 for detecting radiance of light rays from an observation surface point, and a light source controlled to change the radius. And receiving the first radiance before changing the radius and the second radiance after changing the radius, the difference between the first radiance and the second radiance And an imaging processor 20 that generates an image capturing the transient radiance by calculating the transient radiance of the subsurface scattered light in. [Selection diagram] Fig. 4A

Description

  The present invention relates to an imaging apparatus and method for imaging light transport inside an object by capturing transient subsurface scattering.

  Recently, many approaches have been proposed to capture transient light transport in real-world scenes. Many of these transient imaging methods directly sample temporal light transport using time of light or ultrafast cameras in the picosecond to femtosecond range for large-scale light transport events, including reflection and interreflection. Allows separation. However, these methods cannot be used to capture light transport on the inner surface where subsurface scattering is much smaller in time scale and consequently dominates optical events that are not publicly accessible from the outside. Can not.

  Several recent studies have been on light transport analysis of subsurface scattering. In Non-Patent Document 1, by directly observing a side cross section of an object surface irradiated with high-frequency illumination as an uppermost interface, imaging of bounces by subsurface scattering, that is, increment by subsurface particles A method for realizing imaging of bounced light has been invented. Here, the parameters of the analytical scattering model determined in advance are recursively estimated from the observation, and the contribution of light bounced n times is calculated.

JP 2018-080393 A

Mukaigawa, Y. et al., "Analysis of light transport in scattering media," Proceeding of IEEE Computer Vision and Pattern Recognition, June 2010; Tanaka, K. et al., "Recovering Inner Slices of Layered Translucent Objects by Multi-frequency Illumination," Proceeding of IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 39, No. 4, April 2017, pp. 746-757 ; Narasimhan, S. et al., "Shedding Light on the Weather," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, June (2003, pp. 665-672; Redo-Sanchez, A. et al., "Terahertz time-gated spectral imaging for content extraction through layered structures," Nature Communications, September 2016; Narasimhan, S. et al., "Acquiring Scattering Properties of Participating Media by Dilution," Proceeding of ACM Transactions on Graphics, Vol. 25, No. 3, July 2006, pp. 1003-1012; Gkioulekas, I. et al., "An evaluation of computational imaging techniques for heterogeneous inverse scattering," Proceeding of European Conference on Computer Vision, Part III, October 2016, pp. 685-701; and Jakob, W. “Mitsuba: Physically based renderer 0.5.0.,” [Internet search], March 26, 2018 search, URL: http://www.Mitsuba-renderer.org/, 2014

  The method of Non-Patent Document 1 requires physical slicing of the surface in order to deeply observe actual light propagation and is limited to specific analytical scattering models. As a result, the method cannot be applied to general surfaces and even eliminates homogeneous surfaces of unknown materials.

  Non-Patent Document 2 proposes a method of restoring the appearance of a surface viewed from above a target object into several layers having different depths. The object surface is captured with high-frequency illumination at various pitches, and its direct component image is then deblurred with known kernels corresponding to each depth. This method has several assumptions that prevent its use for capturing detailed light transport on the inner surface, except for the special ones where texture is noticeable at discrete depths. First, subsurface light transport captured by an external light source cannot be represented by the radiative transfer model they use (see Non-Patent Document 3), but only applies to light sources immersed in scattering media (ie, the surface). And the scattering of incident light cannot be explained. Secondly, unlike what their method assumes, an appearance that depends on the depth of subsurface scattering does not give rise to a unique texture pattern, and its component frequency is so clear that it does not entangle with high frequency illumination. is not. Recently, Non-Patent Document 4 has demonstrated similar discrete layer extraction by subpicosecond imaging using terahertz time-domain spectroscopy, but this method requires special imaging hardware. was there.

  Further, in Patent Document 1, a striped pattern represented by a spatial encoding method or the like is irradiated on a measurement object by an irradiation unit such as a projector, and three-dimensional coordinates are calculated based on a position where reflected light is observed by an imaging unit. A three-dimensional shape measuring apparatus for calculating by a triangulation method is disclosed. In such an apparatus, the measurement accuracy of the three-dimensional coordinates greatly depends on the material of the measurement object. In addition, there is an inconvenience that the configuration is complicated due to the use of a striped pattern.

  An object of the present invention is to provide an imaging apparatus and method that solves the above-described problems and that can perform imaging with higher accuracy than before even with a non-uniform subject surface with a simpler configuration than before. It is in.

An imaging apparatus according to an aspect of the present invention includes:
An imaging device that captures transient subsurface scattered light in an object to be observed and images internal light propagation,
A light source for illuminating at least one surface illumination point that is separated by a predetermined radius from a predetermined observed surface point on the object surface of the object;
An image sensor for detecting the radiance of light from the observation surface point;
Controlling the light source to change the radius, receiving a first radiance before changing the radius and a second radiance after changing the radius, and receiving the first radiance. And an imaging processor for generating an image capturing the transient radiance by calculating a difference between the second radiance and the second radiance as a transient radiance of subsurface scattered light in the object, Prepare.

An imaging method according to another aspect of the present invention includes:
An imaging method that captures transient subsurface scattered light in an object to be observed and images internal light propagation,
Illuminating at least one surface illumination point that is a predetermined radius away from a predetermined observation surface point on the object surface of the object;
Detecting the radiance of light rays from the observation surface point;
Controlling the light source to change the radius, receiving a first radiance before changing the radius and a second radiance after changing the radius, and receiving the first radiance. Generating an image capturing the transient radiance by calculating a difference between the second radiance and the second radiance as a transient radiance of subsurface scattered light in the object. .

  According to the imaging apparatus and method of the present invention, it is possible to provide an imaging apparatus and method having a simpler configuration than before, and to perform imaging with higher accuracy than before even if the object surface is non-uniform. Can do.

It is the photograph containing the light projection photograph (reference | standard) image | photographed with the imaging device of one Embodiment based on this invention, the variable ring light image of a center point, a variable ring light image, and a transient image. It is sectional drawing which shows the isoluminance line of the subsurface scattering of an object. It is sectional drawing which shows that the surface external appearance of subsurface scattering consists of the light ray which advanced through various path lengths naturally restrict | limited by the surface distance between an incident point and an observation point. It is a cross-sectional view showing the surface appearance of scattering under the surface and illuminating the surface with variable ring light, that is, with ring-shaped illumination centered on the observation surface point Po of interest of different radii. Variable ring-shaped illumination that illuminates the surface with circular illumination centered on observation surface points of different radii and limits the light path length observed at each observation surface point Po by taking the difference between them FIG. The upper row is a photograph showing an image rendered by Mitsuba (see Non-Patent Document 7) and a transient image calculated using the variable ring light of the method of this embodiment, and the lower row is a photograph of each path length rendered by Mitsuba. Image—A photograph showing an image recorded by recording the path length of the rendering process. The upper row is a photograph showing an image rendered by Mitsuba (see Non-Patent Document 7) and a transient image calculated using the variable ring light of the method of this embodiment, and the lower row is a photograph of each path length rendered by Mitsuba. Image—A photograph showing an image recorded by recording the path length of the rendering process. 1 is a schematic block diagram of an imaging apparatus 100 according to an embodiment of the present invention. It is a front view of light source 1A comprised by several LED arrange | positioned at two-dimensional shape. 5 is a flowchart of an imaging process using one illumination pixel (illumination that is not ring-shaped), which is executed by the imaging apparatus 100 in FIG. 4. 5 is a flowchart of an imaging process using a plurality of ring-shaped illumination pixels (ring-shaped illumination), which is executed by the imaging apparatus 100 of FIG. 4. FIG. 5 is a flowchart of an imaging process executed by the imaging apparatus 100 of FIG. 4 to form virtually ring-shaped illumination (virtual ring-shaped illumination) using one illumination pixel. It is a flowchart of a subroutine process (step S2B in FIG. 7) of virtual ring-shaped illumination and radiance L (r) detection. 8 is a flowchart of a subroutine process (step S4B in FIG. 7) for detecting virtual ring-shaped illumination and radiance L (r + (p−1) Δr). FIG. 4B is a flowchart of an imaging process executed by the imaging apparatus 100 of FIG. 4A to more effectively form ring-shaped illumination (virtual ring-shaped illumination) more efficiently using one illumination pixel. FIG. 4B is a flowchart of color restoration processing for a multi-color layered object executed by the imaging apparatus 100 of FIG. 4A. Figure 2 shows a top view of two objects, a photograph of some of them, and a photograph containing a transient image due to the spatial propagation of light below the surface. FIG. 5 is a diagram showing a side view and a plan view of two objects having a multi-color layer, a photograph in which a part thereof is floodlighted, and a photograph including a transient image having a chromaticity due to a color change in depth below the surface. is there. It is a figure which shows the top view of five objects, the photograph with a part of illumination, and the photograph containing the transient image by variable ring optical imaging. It is a figure which shows the top view of two objects, those floodlighting photography, and the photograph containing the transient image by variable ring optical imaging. It is a figure which shows the top view of two objects, the photograph with a part of illumination, and the photograph containing the transient image by variable ring optical imaging. It is a figure which shows the top view of four objects, the photograph with a part of illumination, and the photograph containing the transient image by variable ring optical imaging. It is a figure which shows the top view of four objects, the photograph which floodlighted the one part, and the photograph containing the transient image by variable ring optical imaging. It is a figure which shows the top view of four objects, the photograph which floodlighted the one part, and the photograph containing the transient image by variable ring optical imaging. It is a graph which shows distribution of parameter (alpha) (n, r) with respect to the bounce number n. It is a graph which shows distribution of various functional values with respect to the bounce number n. FIG. 7 is a supplementary explanatory diagram of the processing in step 2A of the imaging processing of FIG. 6, and is an observation surface point at the time of ring light irradiation with respect to a plurality of surface irradiation points Pp located at a radius r + (p−1) Δr from the observation surface point Po. It is a top view which shows the radiance L (r + (p-1) (DELTA) r) of Po. FIG. 7 is a supplementary explanatory diagram of the processing in step S4A of the imaging processing of FIG. 6, and the radiance L (of the observation surface point Po at the time of ring light irradiation with respect to a plurality of surface irradiation points Pp located at a radius r + pΔr from the observation surface point Po; It is a top view which shows (r + p (DELTA) r). FIG. 8 is a supplementary explanatory diagram of the imaging process of FIG. 7, and is a plane showing a plurality of observation surface points Po when scanning a plurality of observation surface points Po on the object surface 10 </ b> S by the raster scanning method in steps S <b> 1 to S <b> 9 of FIG. 7. FIG. FIG. 8 is a supplementary explanatory diagram of the imaging process of FIG. 7, in which each predetermined observation surface point Po when each surface irradiation point Pp determined by the first radius (r + (p−1) Δr) is irradiated. It is a top view which shows the procedure which collects 1st radiance and calculates the normalization value. FIG. 8 is a supplementary explanatory diagram of the imaging process of FIG. 7, and shows the second radiance of each predetermined observation surface point Po when each surface irradiation point Pp determined by the second radius (r + pΔr) is irradiated. It is a top view which shows the procedure which collects and calculates the normalization value.

  Embodiments according to the present invention will be described below with reference to the drawings. In addition, the same code | symbol is attached | subjected about the same or similar component.

1. Introduction.
Subsurface scattering is the transport of light inside the surface close to the external interface and has a significant impact on the appearance of real-world surfaces. When incident light penetrates below the surface, particles (eg, pigment) in the medium are reflected a plurality of times. In addition to absorption by the surface medium itself, each of these bounces causes absorption and scattering, resulting in a unique color and direction of light re-emerging from the interface.

  The longer the light travels, the greater the bounce. If the appearance of the subsurface scattering surface can be restored to a series of images, each has a limited range of distances (ie path lengths) or light that has stood a limited number of bounces in proportion to the surface. As a result, the results can teach a lot about the unobservable internal surface structure below the interface.

  FIG. 1 is a diagram illustrating a photograph including a projected photograph (reference), a variable ring light image of a central point, a variable ring light image, and a transient image, which is taken by an imaging apparatus according to an embodiment of the present invention. In the present embodiment, the following imaging method shown in FIG. 1 is proposed. The imaging method allows a limited number of bounces in proportion to the optical path length within the boundary to be restored to a set of images of particles within the surface (ie, transient light transport at the surface). These transient images collectively reveal the internal structure and color of the surface.

  In this specification, “variable ring light” refers to ring-shaped light capable of changing the radius r from the observation surface point Po (see FIG. 2B, FIG. 2C, etc.), which is the center point.

  Imaging path length or bounce reconstructed light has been studied for interreflection mainly as part of a scene-wide global light transport analysis. Unlike surface reflection, subsurface scattering is dominated by statistically distributed particles, rather than much larger three-dimensional scene geometry, as it relates to light transport within the surface. As mentioned above, past prior art methods often rely on the frequency and geometric characteristics unique to interreflection, and past prior art methods cannot be applied to restore subsurface scattering.

  Embodiments according to the present invention propose a first method for light restoration depending on subsurface scattering reconstruction, path length (and proportional bounce), which constitutes the appearance of a real world translucent surface. To do. In order to directly capture the state of very high-speed light propagating in a scene, it is necessary to use a special camera with high temporal resolution, and scattered light is emitted in the surface of the object in femtosecond units. It was difficult to image the state of propagation. The feature of this method is that the propagation process of light is virtually controlled by controlling the propagation distance of light in the steady state, instead of directly resolving the propagation of light. It is noted that the propagation distance of light under the object surface cannot be shorter than the surface distance (shortest path length) between the incident point and the observation point on the object surface of the light. When viewed from above an object with an orthographic illumination method, the object surface uses light called impulse illumination to define the half-space and the shortest path length of light incident on a single surface point, where Thus, the impulse illumination is observed at a distance r (ie, radius r) from the incident point.

  The present inventors make use of the fact that the optical path length of subsurface light transport cannot be shorter than this distance r by observing the same surface point while changing the distance r of the impulse illumination. Instead of using impulse illumination at any surface point, a ring light of radius r around every surface point of surface distance r illuminated by impulse illumination is used. We can obtain variable ring light images, each of which encodes the appearance of each surface point captured with a different radius of ring light.

  We show that taking the difference between two ring light images with slightly different radii limits the optical path length observed at each surface point. In other words, from the steady state appearance captured as a variable ring light image, we calculate a surface appearance with varying degrees of subsurface scattering, ie, a transient image of subsurface light transport. It is also proved empirically based on simulation and analytical approximation that the path length limited transient image can also be interpreted as an approximate and proportionally limited bounce image of the surface.

  The present inventors execute variable ring light imaging by capturing N impulse illumination images, where N is the number of surface points (irradiated pixels in the observation image region). As shown in FIG. 1, a variable ring light image and a transient image are synthetically calculated from their radiance differences. Extensive experimental results are shown when variable ring optical imaging is applied to both synthetic and real-world complex surfaces. The results show that the method proposed by the inventors successfully restores subsurface scattering, and the restored transient image includes its color change and transverse and depth configurations for a general non-uniform surface. Clarify the internal surface structure. We further show that differences in transient images reveal surface color at different depths.

  The method proposed by the present inventors is simple and requires only a normal projector (illumination irradiation apparatus) and an imaging camera. One advantage over prior art methods is that the object is model free and there are no limiting assumptions about subsurface light transport. To the best of our knowledge, the proposed method is the first to enable high-density restoration of continuous variation of internal structure from external observation of steady state appearance using normal imaging components. . Variable ring light imaging deciphers subsurface scattering and consequently provides an unprecedented means for visually probing the interior of a surface, thereby enabling past transient imaging methods, particularly global light for the entire scene. Complement the bounce restoration method of transportation. The method proposed by the present inventors opens a new path to understanding a richer radiometric analysis of real-world scenes and materials.

2. Comparative example.
As described above, in the method of Non-Patent Document 1, it is necessary to physically slice the surface in order to deeply observe actual light propagation, and the method is limited to a specific analytical scattering model. As a result, the method cannot be applied to general surfaces including a homogeneous one of unknown material. On the other hand, the method proposed by the present inventors does not require any modification to the observation surface of the object whose imaging setting is to be observed, and it is model-free for the target object and can be applied to any surface. Assume that it is possible.

  Recently, Non-Patent Document 4 demonstrates similar discrete layer extraction by subpicosecond imaging using terahertz time domain spectroscopy. In contrast, the method proposed by the inventors is model-free, does not assume any frequency characteristics of subsurface light transport, and uses projector imaging setups, especially its depth over the surface range and including its color. The aim is to restore the continuous appearance change in both along the length with a normal camera. The proposed method is the first finding to the inventors' knowledge and provides access to transient light transport of subsurface scattering. The resulting boundary path length image, or proportional approximation n times bounce image, can be used in a wide range of applications. As we demonstrate, the method can reveal complex internal subsurface structures including their colors. The method can also potentially be used, for example, to determine the material composition of a surface by backscattering (see Non-Patent Documents 5 and 6). Therefore, we believe that the proposed method has important implications for understanding the real world surface appearance.

3. Various ring optical imaging.
In this section, we will explain our research results and theoretical basis of our new imaging technique. After clarifying the assumptions of the method, first the main observations underlying the proposed method, i.e. below the surface of light observed at a given distance from the point of incidence using a single impulse illumination. Discuss bounces with less scattering path length. Next, variable ring light is introduced as illumination, and how these differences bounce the optical path length will be described. Therefore, according to the present method, the transient light transport in the surface, that is, the stepwise progression of subsurface scattering is encoded.

3.1. Preparation.
A microscopic scale, ie a surface that is smaller than an image pixel but larger than a wavelength, consists of media and particles. Incident light passes through the outer interface and then travels straight through the medium and scatters until it finally reappears from the interface each time it hits a suspended particle. Each of these bounces redirects the light beam while some of its energy is absorbed. The degree of absorption by both the medium and the particles varies with wavelength, and the light beam accumulates a unique color depending on its movement path. The appearance of surface points consists of rays that have bounced various times below the surface.

  This subsurface scattering causes the light to follow a zigzag path in the surface. As a result, the surface distance of the ray, i.e. the Euclidean distance on the surface between its entry and exit points, is not the same as its optical path length, i.e. the actual geometric distance the ray traveled through the surface. Note that the optical path length is the product of the optical path length and the refractive index of the medium. Incident light may have traveled straight below the interface. Or it may have propagated deeply across the entire surface before leaving the interface again.

  Consider an imaging setup where the orthographic camera and the directional light are collinear with each other and both are perpendicular to the surface of interest. The following theoretical considerations are valid for non-parallel gaze and illumination, and in fact the camera and light source can be tilted with respect to the surface because they only scale the path length range corresponding to each transient image . Nevertheless, to simplify both theoretically and practically, we will assume a vertical imaging setup here.

  Assume that the surface of the target object of interest is flat in order to safely ignore the interreflection at the outer interface of the target object. The surface distance for a non-planar surface corresponds to a ranging distance corresponding to the geometric distance for a flat surface.

3.2. Impulse subsurface scattering.
A directional light source that irradiates a surface point Pp corresponding to a single pixel in an image for a predetermined time is referred to as impulse illumination. In practice, impulse illumination is achieved by adjusting the correspondence between projector pixels and image pixels and the finest resolution. We usually use projector pixels, which are approximately 4 pixels in later actual experiments.

  FIG. 2A shows a subsurface scattering isoluminance line, and FIG. 2B shows a subsurface scattering surface consisting of light rays that have traveled through various path lengths that are naturally limited by the surface distance between the incident and observation points. Appearance is shown. FIG. 2C shows the surface appearance of subsurface scattering, illuminating the surface with variable ring light, i.e. with circular ring-shaped impulse illumination of a given radius centered on an observation surface point Po of a different radius. FIG. 2D is a plan view showing the surface irradiation point Pp and the observation surface point Po having a variable ring shape. In the present embodiment, the surface irradiation point Pp is illuminated using variable ring light with different radii centered on the observation surface point Po on the object surface 10S of the target object 10, and a gradually increasing pair of radii is observed. By calculating the difference in radiance, the path length of the light observed at the observation surface point Po at each radius can be limited. 2A to 2D, Pp represents an impulse surface irradiation point on the object surface 10S of the target object 10, and Po represents an observation surface point on the object surface 10S.

FIG. 2A shows an isoluminance line (also referred to as an isoluminance line) of the luminance distribution in the subsurface scattered radiation inside the object 10. The radiance of the observed surface point Po at a distance r from the impulse surface irradiation point Pp is the sum E (r) of the radiance L (r) of each ray that has traveled through the object surface 10S before appearing again from the interface at the distance r. It is. Individual ray at interface at distance r on object surface 10S
Instead of considering them individually
To distinguish.

(1)

(2)

  Here, Lp (r) is a set of path lengths of light beams observed at a distance r on the object surface 10S, and i represents the i-th light beam.

here,
Is an operator that returns the entry and exit points of a ray,
Is a ray
Is the distance of the moving object surface 10S. Although not explicitly shown, this set includes only rays whose path is contained within the half space defined by the object surface 10S. In other words, as shown in FIG. 2B, the ray set Lp (r) is made up of rays incident on the object surface 10S and exiting from a surface point at a distance r from the incident point. Without loss of generality, assume that the impulse light source is unit radiance and the camera is calibrated for linear radiance capture. Now we are ready to describe the important observations of subsurface scattering, as

(3)

  The proof of this observation is simple. The shortest path between two points on the object surface 10S is a geodesic line between them. This is defined as the surface distance r of our flat surface, and the observed light cannot travel a shorter distance because it reappears from the surface. In order to make this property explicit, equation (2) is rewritten as

(4)

3.3 Scattering under the ring light surface.
Given a bounce with less path length, observation of a surface point using impulse illumination will probably allow us to extract light of a limited path length (ie, transient light). Before discussing exactly how to achieve this transient image calculation in the next section, we first describe the basic problem of using impulse illumination. Using single impulse illumination is inefficient for two reasons. First, the radiance of subsurface scattering decays exponentially with the optical path length, and transient light calculated from impulse subsurface scattering has low fidelity. Second, the path of these transients, especially for non-uniform surfaces, depends on which surface point at distance r is used for impulse illumination.

  We solve these inefficiencies with a special light source observation configuration that we call variable ring light. Variable ring light refers to a set of ring-shaped illumination patterns of different radii. As shown in FIG. 2C, consider performing impulse illumination of all surface irradiation points Pp that are at a distance r from the observation surface point Po of interest. The normalized radiance of the observation surface point Po is expressed by the following equation as a linear combination of all radiance values from each impulse light source.

(5)

Here, K is the number of impulse light emitting points, that is, the projector pixels corresponding to the image pixel at the distance r as shown in the right diagram of FIG. 2C. The notation E is used to avoid confusion by representing both the normalized variable ring light radiance and the denormalized impulse illumination radiance, but this is because r _k = r for all k. The overwriting on the notation becomes intuitive. Ring shaped illumination improves the SNR by the parameter K, which increases with a larger radius r.

  Subsurface scattering captured under ring-shaped illumination centered on the observation surface point Po also effectively averages all possible light paths in the spatial vicinity of the observation point. This results in a more robust observation of the internal structure surrounding the observed surface point, as it does not depend on a single arbitrary selected point of incident impulse illumination. Note that any point other than the center of the ring-shaped illumination (ie, the observation surface point) will have superimposed subsurface impulse scattering at different distances. The inventors consider only the radiance at the center point. In practice, as will be described in detail later in the experimental results section, it is possible to virtually construct ring-shaped illuminations with variable radii around each observation surface point Po, ie with various radii.

3.4. Transient image from ring light image.
Considering that the object surface 10S is radiated using a ring-shaped illumination having a radius (r + Δr) larger than the radius r by a distance Δr, the following equation is obtained.

(6)

  The actual realization of the rays in these two sets of equations (4) and (6) is disjoint because the incident points are slightly different even if they are observed at the same observation surface point Po. Instead, assume that the two sets of equations (4) and (6) overlap approximately for all light with a path length longer than the surface distance (r + Δr).

(7)

  This important assumption is that the radiance of a ray having a path length longer than the surface distance (r + Δr) observed at the same observation surface point Po using impulse illumination at the surface distance r becomes impulse illumination at the surface distance (r + Δr). Means the same as having In other words, the only difference is a ray with a path length shorter than the surface distance (r + Δr) due to less bounce of the two sets. Since the rays accumulate the same spectral radiance regardless of their point of incidence from the observed surface points for the same path length, this property applies exactly for homogeneous surfaces. Strictly speaking, the number of bounces a ray experiences for a given path length varies, but statistically speaking (ie, on average with narrow dispersion), we safely say that this is correct. Assume.

  When equation (7) holds, the difference between the normalized observation radiances of the same observation surface point Po, but illuminated by ring-shaped illumination with slightly different radii is expressed by the following equation.

(8)

  Here, it is represented by the following formula.

(9)

The arithmetic symbol on the left side of equation (9)
Represents a difference set. This is because, as FIG. 1 shows, each ring surface encodes the radiance for a given ring-shaped illumination radius by illuminating each surface illumination point Pp with a gradually increasing radius of variable ring light. It means that it is assembled and formed into a variable ring light image. Then, by calculating their radiance difference between each radius increment, an image of light with a limited path length, ie, a transient image of subsurface light transport, can be restored.

  If the object surface 10S is volumetrically non-uniform, i.e. consists of different material regions distributed both in space and depth, the path length difference between the two includes the integration of subsurface scattering in the additional material regions. Therefore, equation (7) does not necessarily hold. However, the difference in the spectral radiance of light having a path length longer than the surface distance (r + Δr) between the two observations is that the radial increment Δr is different from that of FIG. 1 in either the depth direction or the spatial direction. Can be minimized by keeping it sufficiently smaller than the minimum size of the material region. In this case, the resolving resolution of the delimited path length is smaller than the individual material regions. In other words, the discrete restoration resolution of transient light transport recovered from the difference in the variable ring light image is finer than the material composition of the surface volume. As a result, the transient image will still capture important subsurface scattering events that are fine enough to reveal the unique surface structure. It is also important to note that the radiance of light with longer path lengths decays exponentially, so in any case the brightness of the contradiction is usually small.

  FIG. 3A shows a side view of a composite multi-color laminated surface rendered using a mitsuba (see Non-Patent Document 7) on the left side of the figure, and a bounded image or path length image rendered by the mitsuba on the upper right side of the figure. Show. FIG. 3B shows a side view of a surface containing a spherical object rendered on the left side by the mituba, with a bounded or path length image rendered by the mituba on the top right of the figure. 3A and 3B also show the transient images calculated from the variable ring light image and path length limited rendering for the two composite non-uniform surfaces in the lower right row of each figure. Otherwise, they become exponentially darker for longer path lengths, so that each transient image is independently normalized in its brightness to show its complex structure. In FIGS. 3A and 3B, the variable ring light transient image shows the rendered path except for the inconsistencies expected from the theory and the inconsistencies resulting from rendering noise and aliasing (dark edges around white glitter in the case of red beads). Visually matches the long image.

  Here, these composite surfaces were rendered using a Mitsuba renderer (see Non-Patent Document 7). For variable ring light imaging, we first render an impulse illumination image with an idealized single light source orthogonal to the surface facing each pixel center. By reconstructing these impulse illumination images, we calculate a variable ring light image with various radii in pixel increments, and the difference between adjacent variable ring light images with adjacent radii to calculate a transient image. Take.

  Mitsuba contains only limited bounce rendering, and each bounce only occurs at sampled render nodes, not particles that are statistically distributed on the surface. In order to achieve path length limited rendering in Mitsuba, the bidirectional path trace integrator has been changed to limit the maximum optical path length in rendering. By calculating the difference between the light images having the optical path lengths with these upper limits, the present inventors obtain an image with the optical path length having a predetermined range that can be regarded as ground truth.

  Unfortunately, rendering subsurface scattering, especially for complex and uneven surfaces, and with path length limitations, is almost impractical for them to properly simulate their light transport. This is difficult because it requires a large number of samples. In this case, 262,144 samples were used for path length limited rendering. This takes a day to render but still suffers from excessive noise. For this reason, rendering is noisy and the result of variable ring optical imaging is also noisy.

  However, the results in Figures 3A and 3B show that the restored transient images are in good agreement with the path length limited rendering on both surfaces. For superimposed color surfaces, the integrated light colors in each transient image match, and for surfaces with immersed red beads, light propagation from the beads (red due to specular reflection on the surface). White light from the beads) The white light in the path length limited rendering is not seen for contrast with dark noise because the red light comes out of the bead after bouncing in the bead. Although difficult, they exist.

  There is a small but noticeable contradiction on the surface of the red beads. Here, the first few images of path length limited rendering do not show the full interior red beads, whereas the variable ring light results include the entire red bead light from the first transient image. In the case of a variable ring light image, since the beads are spherical, the red light from the beads does not contribute much to the larger radius (r + Δr), so the first few transient images have red light.

  In contrast, since the light has not reached the red beads, only the red light is observed gradually in the first few path length limited renderings. Apart from this expected difference and the inconsistency that basically arises from rendering noise and aliasing, these combined results show that variable ring optical imaging captures transient subsurface light transport. If the surface where the scatterers are uniformly distributed in the surface volume is homogeneous, the number of particles along the path is directly proportional to the optical path length.

  Even if the object surface 10S is non-uniform in composition, the bounce number increases monotonically with the path length. As a result, it can be safely assumed that the average bounce number experienced by the light beam is proportional to the optical path length. As a result, the difference in radiance between two ring illumination images of different radii limits the number of average bounce lights.

  However, the limits are average, and variable ring light images cannot rule out the progression of subsurface scattering when the variance in the number of bounces of light for a given path length is large. However, on general real world surfaces, this variance is usually small. The supplementary material provides empirical evidence based on a theoretical analysis using Monte Carlo simulation of the radiance ratio of n-bounce light of ring light observation and its analytical approximation. Assume that the reconstructed transient image roughly corresponds to the bounded bounce light.

4). An imaging apparatus according to an embodiment.
FIG. 4A is a schematic block diagram of the imaging apparatus 100 according to the embodiment of the present invention.

Referring to FIG. 4A, a target object 10 to be observed is arranged on a table 11, and the target object 10 has a flat or non-uniform object surface 10S on its upper surface. On the object surface 10S, an observation surface point Po and a surface irradiation point Pp separated from the observation surface point Po by a radius r or (r + pΔr) (p = 1, 2,..., P _max ) are selected.

  A light source 1 such as an irradiation lens is provided to illuminate each surface irradiation point Pp on the object surface 10S via a lens 3 such as a telephoto lens. The image sensor 2 such as a photographing camera is provided to detect a light beam from the object surface 10S via the lens 4 such as a telephoto lens, and an image signal of the incident light beam is input to the image input interface 22 of the imaging apparatus 100. Output.

  The imaging apparatus 100 includes an imaging processor 20, a light source controller 21, an image input interface 22, an image memory 23, and an output interface 24. The imaging processor 20 includes an image input interface 22, an image memory 23, and an output interface 24.

  The light source controller 21 is configured to control the operation of the light source 1 according to the control of the imaging processor 20, and the light source 1 illuminates each surface point on the object surface 10 </ b> S via the lens 3. The image input interface 22 inputs an image signal. The imaging processor 20 performs the processing of FIGS. 5 to 11 to AD convert the image signal from the image sensor 2, AD convert the input image signal, and output the image data to the imaging processor 20. These image data are output to a display 5 such as an LED (Light Emitting Diode) display via an output interface 24 that performs predetermined signal and data conversion by ring-shaped illumination or non-ring light illumination. The display 5 displays a captured transient image of subsurface scattering captured under ring-shaped illumination or non-ring light illumination.

  Instead of the light source 1 of FIG. 4A, the light source 1A of FIG. 4B may be provided. FIG. 4B is a front view of a light source 1A configured by a plurality of LEDs 6 arranged in a two-dimensional shape. As shown in FIG. 4B, the light source 1A is composed of a plurality of LEDs 6 arranged two-dimensionally.

  FIG. 5 is a flowchart of an imaging process using one illumination pixel (illumination not having a ring shape), which is executed by the imaging apparatus 100 in FIG. 4A. For example, it is an embodiment modified from the embodiment using ring-shaped illumination.

  In step S <b> 1 of FIG. 5, first, the imaging processor 20 selects one observation surface point Po defined so that the surface point on the object surface 10 </ b> S corresponds to the pixel of the light source 1. In step S2, the light source controller 21 controls the light source 1 to irradiate the surface irradiation point Pp with a predetermined distance corresponding to the radius r of the ring-shaped illumination from the selected observation position. Here, the surface irradiation point Pp is selected from candidates for the surface irradiation point Pp. Then, the image input interface 22 inputs the radiance L (r) of the image signal from the selected observation surface point Po and outputs it to the imaging processor 20, and the imaging processor 20 outputs the data of the radiance L (r) as an image. The image is stored in the memory 23.

  In step S3, 1 is set to the parameter p. In step S4, the light source controller 21 controls the light source 1 to irradiate the surface irradiation point Pp away from the selected light source by a predetermined distance corresponding to the radius (r + pΔr) of the ring-shaped illumination. At the observation surface point Po, the image input interface 22 inputs the radiance L (r + pΔr) of the image signal from the selected observation surface point Po, and the imaging processor 20 stores the radiance L (r + pΔr) in the image memory 23. To do. In step S5, the imaging processor 20 calculates a radiance difference between the radiance L (r + (p−1) Δr) and the radiance L (r + pΔr) as the radiance Lt at the selected observation point Po of the transient image. To do. Store in the image memory 23.

In step S6, it is determined whether or not the parameter p is equal to the maximum value _pmax . If NO in step S6, the process proceeds to step S7. In step S7, the parameter p is incremented by 1, and the process returns to step 4. On the other hand, if “YES” in the step S6, the process proceeds to a step S8 to determine whether or not all the observation surface points Po on the object surface 10S are selected. If NO in step S8, the process proceeds to step S9, and the imaging processor 20 selects the next one observation surface point Po on the object surface 10S by the raster scanning method, and returns to step S2. On the other hand, if “YES” in the step S8, the process proceeds to a step S10, and the imaging processor 20 forms a plurality of transient images on the object surface 10S by collecting the radiance Lt for each radius at all the observation surface points Po. The transient images formed for a plurality of p _max radii are output to the display 5 via the output interface 24. Then, the imaging process in FIG. 5 ends.

As shown in FIG. 5, the imaging process can provide a transient image formed for multiple p _max radii with non-ring shaped illumination.

FIG. 6 is a flowchart of an imaging process using a plurality of ring-shaped illumination pixels (ring-shaped illumination), which is executed by the imaging apparatus 100 of FIG. 4A. The imaging process of FIG. 6 differs from the imaging process of FIG. 5 in the following points, but the other configurations are the same.
(1) Step S2A is executed instead of step S2.
(2) Step S4A is executed instead of Step S4.
Hereinafter, differences between FIG. 6 and FIG. 5 will be described. In the embodiment, the ring-shaped illumination means a circular ring-shaped illumination.

  In step S <b> 2 </ b> A, the light source controller 21 controls the light source 1 to irradiate the surface illumination point Pp at a distance r corresponding to the radius of the ring illumination from the selected one observation surface point Po. The image input interface 22 inputs the radiance L (r) of the image signal from the selected observation surface point Po, and the imaging processor 20 stores the radiance L (r) in the image memory 23. In step S4A, the light source controller 21 grows the light source 1 and irradiates the ring-shaped illumination on the surface irradiation point Pp at a slightly larger distance (r + pΔr) from the selected one observation surface point Po. Then, the image input interface 22 inputs the radiance L (r + pΔr) of the image signal from the selected observation surface point Po. The imaging processor 20 stores the radiance L (r + pΔr) in the image memory 23.

As shown in FIG. 6, the imaging process can provide a transient image formed for a plurality of p _max radii by ring-shaped illumination.

19A and 19B are diagrams for supplementary explanation of the imaging process of FIG. Here, FIG. 19A shows the radiance L (r + (p−1) of the observation surface point Po at the time of ring light irradiation with respect to the plurality of surface irradiation points Pp located at the radius r + (p−1) Δr from the observation surface point Po. Δr). FIG. 19B shows the radiance L (r + pΔr) of the observation surface point Po at the time of ring light irradiation with respect to a plurality of surface irradiation points Pp located at a radius r + pΔr from the observation surface point Po. As shown in FIGS. 19A and 19B, in the imaging process of FIG. 6, as the radiance Lt at the selected observation point Po of the transient image, the radiance L (r + (p−1) Δr) and the radiance L (r + pΔr). ) And the radiance Lt for each radius at all scanned observation surface points Po are collected, thereby forming a plurality of transient images on the object surface 10S, and a plurality of p _max radii. A transient image formed for can be obtained.

FIG. 7 is a flowchart of an imaging process executed by the imaging apparatus 100 of FIG. 4A to virtually form ring-shaped illumination (virtual ring-shaped illumination) using one illumination pixel. The imaging process of FIG. 7 differs from the imaging process of FIG. 6 in the following points, but the other configurations are the same.
(1) Step S2B is executed instead of step S2A.
(2) Step S4B is executed instead of Step S4B.
Hereinafter, differences between FIG. 7 and FIG. 6 will be described.

  In step S2B, the imaging processor 20 executes a virtual ring-shaped illumination and radiance L (r) detection subroutine process shown in FIG. In step 4B, the imaging processor 20 executes a virtual ring-shaped illumination and radiance L (r + (p−1) Δr) detection subroutine shown in FIG.

  FIG. 8 is a flowchart of the subroutine processing (step S2B in FIG. 7) of virtual ring shape illumination and radiance L (r) detection.

  In step S21 of FIG. 8, the parameter m is set to 1, and in step S22, the imaging processor 20 selects one of the distances equivalent to the radius r of the virtual ring-shaped illumination from the selected observation surface point Po. A surface irradiation point Pp (m) is selected. In step S23, the light source controller 21 controls the light source 1 to irradiate the selected surface irradiation point Pp (m) with impulse illumination, and the image input interface 22 displays the radiance L of the observed image signal. Enter (r, m). Then, the imaging processor 20 stores the radiance L (r, m) in the image memory 23. Next, in step S24, it is determined whether or not all surface irradiation points Pp (m) that form ring-shaped illumination on the object surface 10S have been selected. If NO in step S24, the process returns to step S25, and thereafter The parameter m is incremented by one and the process proceeds to step S22. On the other hand, if YES in step S24, the process proceeds to step S26, and the imaging processor 20 calculates the total value of the total radiance L (r, m) of the ring-shaped illumination as the radiance L (r). Then, the control flow returns to the main routine.

  FIG. 9 is a flowchart of the virtual ring-shaped illumination and the subroutine process (step S4B in FIG. 7) for detecting the radiance L (r + (p−1) Δr).

  After the parameter m is set to 1 in step S31 of FIG. 9, in step S32, the imaging processor 20 determines the radius (r + (p−1) of the virtual ring-shaped illumination from one selected observation surface point Po. ) One surface irradiation point Pp (m) at a distance equivalent to Δr) is selected. In step S33, the light source controller 21 controls the light source 1 to irradiate the selected surface irradiation point Pp (m) with impulse illumination, and the image input interface 22 outputs the radiance L (r + (p−1) Δr. , M). Then, the imaging processor 20 acquires the image signal of the image signal from the observation surface point Po, and stores the radiance L (r + (p−1) Δr, m) in the image memory 23. Next, in step S34, it is determined whether all the surface irradiation points Pp (m) that form a ring shape on the object surface 10S have been selected. If No in step S34, the process proceeds to step S35, the parameter m is incremented by 1, and the process returns to step S32. On the other hand, if “YES” in the step S34, the process proceeds to a step S36, and the imaging processor 20 calculates the total value of all the radiance values L (r + (p−1) Δr, m) as the radiance L (r + (p−1). ) Δr). Then, the control flow returns to the main routine.

  20 to 22 are supplementary explanatory diagrams of the imaging process of FIG. 7, and FIG. 20 illustrates a case where a plurality of observation surface points Po on the object surface 10S are scanned by the raster scanning method in steps S1 to S9 of FIG. It is a top view showing a plurality of observation surface points Po. FIG. 21 is a supplementary explanatory diagram of the imaging process of FIG. 7, and each predetermined observation surface when each surface irradiation point Pp determined by the first radius r + (p−1) Δr is irradiated. It is a top view which shows the procedure which collects the 1st radiance of the point Po, and calculates the normalization value. Further, FIG. 22 is a supplementary explanatory diagram of the imaging process of FIG. 7, and the second of each predetermined observation surface point Po when each surface irradiation point Pp determined by the second radius r + pΔr is irradiated. It is a top view which shows the procedure which collects radiance and calculates the normalization value.

As shown in FIG. 20, each surface irradiation point Pp determined by the first radius r + (p−1) Δr is irradiated as shown in FIG. 21 while scanning the observation surface point Po by the raster scanning method. The first radiance of each predetermined observation surface point Po is collected and its normalized value is calculated. Then, as shown in FIG. 22, each surface irradiation determined by the second radius r + pΔr The second radiance of each predetermined observation surface point Po when the point Pp is irradiated is collected and its normalized value is calculated. Further, as shown in step S10 of FIG. 7, by collecting the radiance Lt for each radius at all observation surface points Po, a plurality of transient images are formed on the object surface 10S, and a plurality of p _max radii are formed. A transient image formed for is obtained.

As described above, as illustrated in FIGS. 7 to 9, the imaging process can provide a transient image formed for a plurality of p _max radii by virtual ring-shaped illumination.

In the imaging processes of FIGS. 7 to 9, radiance data is collected by changing the radius r around the observation surface point Po while scanning the observation surface point Po. However, the present invention is not limited to this, and radiance data may be collected as follows.
(Variation 1) While scanning the surface observation point Pp, radiance data is collected by using ring-shaped illumination having the same radius r so that the radius r from the observation surface point Po is the surface observation point Pp, and then different. Radiance data may be collected using ring-shaped illumination of radius r + Δr.
(Modification 2) While scanning the surface observation point Pp, radiance data is collected using ring-shaped illuminations having different radii r and r + Δr where the radius r from the observation surface point Po becomes the surface observation point Pp. Also good.

  FIG. 10 is a flowchart of an imaging process executed by the imaging apparatus 100 of FIG. 4A to form more efficient ring-shaped illumination (virtual ring-shaped illumination) using one illumination pixel.

  In step S41 of FIG. 10, first, the imaging processor 20 selects one surface irradiation point Pp from the surface points on the object surface 10S, and controls the light source 1 to irradiate impulse illumination in step S42. At the selected surface irradiation point Pp, the image input interface 22 inputs image data of image signals of all images, and the imaging processor 20 stores the image data of image signals of all images in the image data table of the image memory 23. Next, in step S43, it is determined whether or not all surface irradiation points Pp on the object surface 10S have been selected. If NO in step S43, the process proceeds to step S44, and the imaging processor 20 selects the next one surface irradiation point Pp on the object surface 10S by the raster scanning method, and returns to step S42. On the other hand, if “YES” in the step S43, the process proceeds to a step S45, and the imaging processor 20 selects one observation surface point Po from the surface points on the object surface 10S, and sets 1 to the parameter p in a step S46. The process proceeds to step S47.

In step S47, the imaging processor 20 calculates the radiance L (r) and the radiance L (r + pΔr) in the image memory 23 when the input observation surface point Po is virtually captured under the surface irradiation point. Extract from the data table. Then, the difference (virtual ring shape illumination) between the average value of the radiance L (r) and the average value of the radiance L (r + pΔr) is calculated as the radiance Lt. In step S48, it is determined whether or not the parameter p is equal to the maximum value _pmax . If “NO” in the step S48, the process proceeds to a step S49, and then the parameter p is incremented by 1, and the process returns to the step S47.

On the other hand, if “YES” in the step S48, the process proceeds to a step S50 to determine whether or not all the observation surface points Po on the target surface 10S are selected. If NO in step S50, the process proceeds to step S51, and the imaging processor 20 selects one next observation surface point Po on the object surface 10S by the raster scanning method, and returns to step S47. On the other hand, if YES in step S50, the process proceeds to step S52, and the imaging processor 20 forms a plurality of transient images on the object surface 10S by collecting the radiance Lt for each radius at all the observation surface points Po. The transient images formed for a plurality of p _max radii are output to the display 5 via the output interface 24.

As shown in FIG. 10, the imaging process measures the radiance value in advance, reads out the radiance value necessary for forming the transient image, and forms the transient image. Can provide transient images formed for multiple p _max radii with virtual ring-shaped illumination with higher efficiency than

  6-10 above, the imaging device 100 uses real or virtual circular ring-shaped illumination, but instead of full ring-shaped illumination, a portion of the ring-shaped illumination (i.e., arcuate shape). Lighting) may be used. 5 to 10, the imaging apparatus 100 uses radiances L (r) and L (r + pΔr). In this case, each of the radiances L (r) and L (r + pΔr) is a monochromatic radiance. Or RGB color radiance may be sufficient.

  In step S9 of FIGS. 5 to 7 and step S51 of FIG. 10, the observation surface point Po is scanned on the object surface 10S by the raster scanning method, but the observation surface point Po is scanned by the predetermined scanning method to cover the entire area. You may scan.

  FIG. 11 is a flowchart of the color restoration processing of the multi-color layered object executed by the imaging apparatus 100 of FIG. 4A.

In step S61 of FIG. 11, first, the imaging processor 20 executes a plurality of color transient images CTI (p−1) on the object surface 10S by executing one of the imaging processes of FIGS. , P) (p = 1, 2,..., P _max ). In step S62, the imaging processor 20 calculates the average chromaticity (X _ave , Y _ave ) of the plurality of color transient images CTI (p−1, p) (p = 1, 2,..., P _max ). To do. In step S63, the imaging processor 20 sets several color transient images having different colors such that the average chromaticities (X _ave , Y _ave ) of the selected color transient images differ by a predetermined threshold. Select. Next, in step S64, the imaging processor 20 outputs a plurality of color transient images having different colors corresponding to the selected plurality of layers to the display 5 and displays them.

  The color restoration process of FIG. 11 can provide the experimental results shown in FIG. In FIG. 11 described above, the color transient image CTI (p−1, p) can be an image having RGB values, for example.

5. Experimental result.
By examining the reconstructed transient image of the real world object surface in addition to the synthetic surfaces of FIGS. 3A and 3B, we experimentally verified the theory of variable ring light. The experimental results demonstrate that our new imaging method allows the restoration of transient images of subsurface scattering.

5.1. Spatial light transport in transient images.
A variable ring optical imaging is implemented using a USB camera (Point Gray Glasshopper 3) and a DLP projector (TI DLP Light Commander). The USB camera corresponds to the image sensor 2 in FIG. 4A, and the DLP projector corresponds to the light source 1 in FIG. 4A. By arranging the projector and camera sufficiently away from the surface, the line of sight and the irradiation line are approximately aligned and perpendicular to the target surface. In order to achieve orthographic projection and directional light, a telephoto lens is used for both the camera and the projector. Adjust the projector and camera to associate each projector pixel with an image pixel. In our experiments, approximately one projector pixel corresponds to a 22 image pixel area. An impulse illumination image is captured for each projector pixel and a variable ring light image is calculated from these images.

  The light path length in each transient image is determined by the thickness and spacing of the variable ring light radius. The thicker the ring light, the brighter the observed radiance, and therefore more faithful in the restored transient image. However, the thick ring light directly increases the path length limit, ie, the radius r has a wide range in equation (9). For this purpose, the thickness is set to a single projector pixel in order to ensure a sharp boundary of the path length (see, for example, FIG. 2C). The spacing between the different ring-shaped radii r controls the range of optical path lengths we capture in one transient image. This is set to one projector pixel in order to restore the resolution of the maximum possible optical path length. For a non-uniform surface, the condition for radius r for equation (7) that we have described is controlled by ensuring that this single projector pixel is sufficiently smaller than the finest material area. Note that you can. This can be realized by changing the distance from the projector to the object surface 10S.

  FIG. 12 shows a plan view of two objects, a partial illumination photograph thereof, and a photograph showing a transient image due to spatial propagation of light, and FIG. 13 is a side view and a plan view of an object having two multi-color layers. 2 shows a photograph including a transient image having a chromaticity due to a color change at a subsurface depth. That is, FIGS. 12 and 13 encode a bounded integral of subsurface optical interaction, revealing their color that is obscured from the volume composition of the surface and the appearance of its outer interface, A transient image in a pixel is shown. The difference between the transient images, i.e. the difference between the variable ring light images, reveals the surface color at different depths.

  As shown in FIG. 12, for spatially non-uniform subsurface structures, the restored transient image captures interesting spatial light propagation as the corresponding ring light radius increases. For these and all other results, the brightness of each transient image is normalized independently to visualize details independent of the diminishing radiance of light of longer path length. The result is that for red translucent beads inserted in white plastic clay, concentric propagation of white light specularly reflected from the red beads, followed by red light from the beads, and star plastic embedded in a pink polymer Indicates a star-shaped light. As the optical path length increases, it creates a light-propagating shape like a petal and interreflects. These subsurface structures are obscured from the outside, as shown in the leftmost image taken with normal room light.

5.2. Color from transient image.
When the object surface has subsurface structures that vary over its depth, the reconstructed transient image provides an appearance sample of that continuous depth change. Furthermore, since the limited path length light directly encodes the accumulated color along that path, each transient image will reveal an integrated color variation along the depth. This spectral integration of transient light suggests that the true color at each corresponding surface depth can be revealed by taking the light difference while gradually increasing the optical path length. In other words, by taking the difference between the variable ring light images, we can reconstruct the subsurface color at different depths.

  FIG. 13 shows transient images and chromaticity values reconstructed from the differences for surfaces made with various color layer depths. The chromaticity trajectory of the transient image reveals the spectral integration of subsurface scattering. We manually select three transient images at regular intervals, each of which roughly corresponds to the optical path length of each layer depth, and calculate the average chromaticity of each image and their difference. The color was restored at each depth. The restored color displayed in the rightmost column of FIG. 13 matches very well with the basic truth displayed in the leftmost image of FIG. These color changes are not accessible from simple external observations. And even worse, note that the same exterior color may be made up of infinite combinations of colors below the surface.

5.3. Transient image of complex real surface.
FIG. 14 shows that variable ring light imaging reveals a complex intrinsic surface structure in the reconstructed transient image, where deeper and longer traveling light is captured with a larger radius ring light . Wrinkles disappear on the skin, deep red skin appears, arteries become more noticeable, and the marble pendant has a spatial spread of different color areas as the radius increases.

  1 and 14 show a reconstructed transient image of a real surface having a color composition including complex subsurface structures and natural products. The transient images of each example reveal subsurface composition in both its volumetric structure and its color change that would otherwise not be visible from the outside. It should be noted that colorless surface reflections and early scattering, as well as complex volume propagation and richer colors with longer path lengths in larger radius transient images. Supplementary material for more results of transient images is described in the next section.

  Note that HDR (High Dynamic Resolution) capture significantly increases image capture time. This limits the inventors to 64 × 64 temporary images for a reasonable computation time (ie about 1 hour for all image capture and computation). For high-resolution image acquisition of larger surface areas, parallel impulse illumination acquisition of surface points at far distances can be used to speed up imaging, but picking up light accidentally from simultaneously illuminated points Limit the maximum ring light radius to avoid.

  FIG. 15 shows a plan view of two objects, their illumination photographs, and a photograph including a transient image with variable ring optical imaging. In FIG. 15, the width of each transient image is about 800 pixels. That is, FIG. 15 shows an example of a high resolution result of variable ring optical imaging for a large object region. In embodiments, the implementation of the present invention can be speeded up by combining spatial and dynamic range multiplexing with ring optical imaging.

6). Additional variable ring illumination imaging to capture transient subsurface scattering.
6.1. Additional experimental results.
The reconstructed transient image reveals the unique structure of the surface because each of those pixels encodes the integrated interaction of light as it propagates through the volume below the surface. The subsurface volume composition and its color, which cannot be recognized from the outside, are revealed in the transient image. Note that the first transient image encodes a surface reflection that often includes a specular reflection that saturates. The larger the radius of the ring light and the longer the optical path length, the darker the color and the wider the spatial propagation of light. Temporary light entering the image from the border (eg, bright half-ring from the bottom in the last example of pomegranate juice) is due to reflections at the border of the object. The cactus transient image ends with a high-frequency spatial pattern of light coming out of its back. As these results indicate, variable ring optical imaging provides an effective visual tool for probing the volume and radiative structure of real-world surfaces, including those of natural products. (For example, fish, leaves, and fish eggs).

  FIG. 16 is a plan view of two objects, a partial illumination photograph thereof, and a photograph including a transient image by variable ring optical imaging. Each of FIGS. 17A-17C shows a plan view of each of the four objects, a photograph with a portion of it, and a photograph including a transient image from variable ring optical imaging.

6.2. The transient image should be an approximate bounce image.
FIG. 18A shows a representative result of a Monte Carlo simulation of subsurface scattering. The α (n, r) distribution, that is, the ratio of the n-th bounce light to the ring light of radius r has a non-zero value that rises sharply after a certain constant and falls quickly with overlapping at the end at different radii r. . These α (n, r) distributions can be accurately approximated with a unit shape parameter and a Frechet distribution with different scale parameters proportional to the radius r. FIG. 18B shows that a differential approximation of the difference between the two Frechet distributions can safely be assumed that the difference between the two ring light observations of different radii actually encodes the bounce light n times.

  Various values of the optical path length according to the anisotropic scattering parameter g (0.1 to 0.9) of Henney-Greenstein and the average optical path length (average of Gaussian distribution 0.1 to 0.3) A simulation was performed on the combination of. This plot shows two important properties of n times bounce light. First of all, we can confirm that the distribution α (n, r) takes a value only after n bounces proportional to the radius r. This shows empirically that the bounce light n times observed with the ring light of radius r is limited by the parameter n (r). Second, it can be seen that the α (n, r) distribution across parameter n for various radii r drops rapidly and overlaps with the adjacent distribution α (n, r + Δr). This rapid rise before n (r) is exactly what the following characteristics are desired. The number of bounces of light for a given optical path length is concentrated with high density around the average. Less well-defined bounces and longer path lengths or more bounces are almost unlimited.

  Moreover, it can be seen that the shape of the parameter α is unique and particularly similar to a Fréchet distribution having unit shape parameters.

(10)

  Here, β (r) is a scalar different for each observation of the bounce light n times with respect to the ring light having the radius r, and s (r) is a Fréchet distribution f (n; s (r)) depending on the radius r. Scale parameter. For twelve different simulation results, a freche distribution is applied to the distribution α (n, r) for r = 5, 6,..., 25 and n = 1, 2,. Confirm this guess by finding out.

  Consider two n-bounce light ratios observed for ring light at distance r and (r + Δr) on the object surface and representing s (r) = s and s (r + Δr) = s + Δs. Since each can be modeled with a Fréchet distribution, the difference between them is expressed by the following equation.

(11)

Here, for a sufficiently small radius Δr, the inventors assumed the following equation.

  When the derivative of the Fréchet distribution with respect to the shape parameter is approximated by forward differentiation, the following equation is obtained.

(12)

  From the equations (11) and (12), the following equation is obtained.

(13)

FIG. 18B shows a pair of Fréchet distributions with different shape parameters (s and (s + Δs)), their actual difference (f (n; s) −f (n; s + s)), and an approximation using derivatives. Show. Its shape parameter
It can be seen that the approximate expression (formula (13)) using the Frechet distribution (where Δs is about 50% of the shape parameter s) reflects the actual difference shape well. For average reflections greater than the parameter n (r + Δr), the following equation can be safely assumed:

(14)

The difference in the subsurface scattering of the ring light for the two radii is
Since the bounced light is captured, the following equation is obtained.

(15)

here,
Is the optical path length
And the radiance of light that has experienced n times of bounces. Path length
The first notation used for the radiance of rays with
Note that we are reusing. In other words, the transient image calculated from variable ring optical imaging can be thought of as encoding approximately n times bounced light.

7). Conclusion.
In an embodiment according to the present invention, the inventors have introduced variable ring optical imaging for solving subsurface scattered light to various bounded path lengths and proportionally bounce numbers from external observations. The method we have proposed does not assume a restrictive analytical scattering model and can be applied to any real world surface exhibiting subsurface scattering. The theory and its experimental verification show the effectiveness of variable ring optical imaging to reveal the structure under the object surface and its color. We believe that the method proposed by the inventors has great significance in visually exploring the surface and deciphering the transport of light in order to understand a richer scene.

  As described above in detail, according to the imaging apparatus and imaging method of the present invention, it is possible to provide an imaging apparatus and method having a simpler structure than before, and even if the object surface is non-uniform, Imaging can be performed with high accuracy.

DESCRIPTION OF SYMBOLS 1 Light source 1A Light source 2 Image sensor 3, 4 Lens 5 Display 6 LED
DESCRIPTION OF SYMBOLS 10 Target object 10S Object surface 11 Table 20 Imaging processor 21 Light source controller 22 Image input interface 23 Image memory 24 Output interface 100 Imaging apparatus Po Observation surface point Pp Surface irradiation point

Claims (9)

An imaging device that captures transient subsurface scattered light in an object to be observed and images internal light propagation,
A light source for illuminating at least one surface irradiation point separated from a predetermined observation surface point observed on the object surface of the object by a predetermined radius;
An image sensor for detecting the radiance of light from the observation surface point;
Controlling the light source to change the radius, receiving a first radiance before changing the radius and a second radiance after changing the radius, and receiving the first radiance. And an imaging processor for generating an image capturing the transient radiance by calculating a difference between the second radiance and the second radiance as a transient radiance of subsurface scattered light in the object, An imaging apparatus provided. The light source illuminates at a plurality of surface irradiation points incident on a surface point at a distance determined by a radius from the observation surface point, and the plurality of surface irradiation points have a circular ring shape and a circular ring shape. Forming one of the parts and
The imaging processor controls the light source to change the radius, and corresponds to the plurality of surface irradiation points, a plurality of first radiances before changing the radius, and the plurality of surface irradiation points. A plurality of second radiances after changing the radius, and a plurality of corresponding differences between the plurality of first radiances and the plurality of second radiances, The imaging apparatus according to claim 1, wherein the imaging apparatus calculates a plurality of transient radiances of subsurface scattered light in an object. The imaging processor further illuminates from the light source to scan the observation surface point on the object surface and controls the image sensor to produce radiance of each radius at all the observation surface points. The imaging apparatus according to claim 1, wherein a plurality of images capturing the transient radiance of subsurface scattered light in the object is generated by collecting. The imaging processor further comprises an image memory;
The imaging processor illuminates from the light source and scans the image sensor to scan the surface point at a distance determined by a first radius before changing the radius. Collecting a plurality of radiances of the first radius at a point and storing them in the image memory;
The imaging processor illuminates from the light source and controls the image sensor to scan the surface point at a distance determined by a second radius after changing the radius and the observation. Collecting a plurality of radiances of the second radius at a point and storing them in the image memory;
By calculating a difference between an average value of the plurality of first radiances and an average value of the plurality of second radiances as a transient radiance of subsurface scattered light in the object, The imaging apparatus according to claim 3, wherein a plurality of images capturing the transient radiance of the subsurface scattered light is generated. The imaging processor further comprises an image memory;
The imaging processor irradiates illumination at each irradiation point from the light source so as to scan all the observation surface points, and controls the image sensor to collect radiance at all the observation surface points. Stored in the image memory,
At each predetermined observation surface point when each irradiation point at a distance determined by the first radius before changing the radius is irradiated from the stored radiance at all the observation surface points Extracting the plurality of first radiances;
Each predetermined observation surface point when each irradiation point is irradiated at a distance determined by a second radius after changing the radius from radiances at all of the stored observation surface points. Extracting the plurality of second radiances at
At each of the predetermined observation surface points, the difference between the average value of the plurality of first radiances and the average value of the plurality of second radiances is expressed as the transient radiance of subsurface scattered light in the object. The imaging apparatus according to claim 3, wherein a plurality of images capturing transient radiance of subsurface scattered light in the object are generated by calculating as follows. The imaging apparatus according to claim 1, wherein the illumination is impulse illumination. The imaging processor further comprises an image memory;
The light source irradiates a ring-shaped illumination onto a surface illumination point at a plurality of distances determined by a radius from the observation surface point, and the imaging processor controls the light source to change the radius Detecting image data having a plurality of pixels on the object surface of the object and storing the image data in the image memory;
The imaging processor extracts a plurality of first radiances before changing the radius and a plurality of second radiances after changing the radius from the image data stored in the image memory. Calculating a difference between the plurality of first radiances and the plurality of second radiances as a radiance at the observation surface point;
The imaging processor further controls the image sensor to scan the observation surface point and collects a plurality of radiances for each radius of all the observation surface points, thereby subsurface scattered light at the object. The imaging apparatus according to any one of claims 1 to 6, wherein a plurality of images capturing the transient radiance is generated. The imaging device is provided for performing object color restoration of a multi-color layer based on a plurality of differences in a plurality of color radiances,
The imaging processor obtains a color image capturing a plurality of transient radiances of surface scattered light on the object, calculates a plurality of average chromaticities of the color image, and the average chromaticity of the color image is predetermined. The imaging apparatus according to claim 3, wherein a plurality of color images having different colors are selected and output so as to differ depending on a threshold value of the image. An imaging method that captures transient subsurface scattered light in an object to be observed and images internal light propagation,
Illuminating at least one surface illumination point that is a predetermined radius away from a predetermined observation surface point on the object surface of the object;
Detecting the radiance of light rays from the observation surface point;
Controlling the light source to change the radius, receiving a first radiance before changing the radius and a second radiance after changing the radius, and receiving the first radiance. Generating an image capturing the transient radiance by calculating a difference between the second radiance and the second radiance as a transient radiance of subsurface scattered light in the object. Imaging method.