JP7767111B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device that performs three-dimensional modeling using machine learning.

Technology for three-dimensional modeling using images of an object taken from various angles is known. Patent Document 1 discloses a technology that generates, with a small amount of calculation, an image of an object viewed from an angle different from that at which it was photographed. However, because the color of light reflected from an object naturally changes depending on the viewing angle, the technology disclosed in Patent Document 1 can sometimes create an unnatural appearance when the angle at which the image is reconstructed is changed.

Non-Patent Document 1 discloses a technology that reconstructs realistic images that look like real life images by taking into account the direction of light rays in addition to three-dimensional positions in space, sampling points on the light rays, and performing volume rendering.

Japanese Patent Application Laid-Open No. 2018-205863 JP 2008-15754 A

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, In ECCV, 2020. TIANYE LI, MIRA SLAVCHEVA, MICHAEL ZOLLHOEFER, SIMON GREEN, CHRISTOPH LASSNER, CHANGIL KIM, TANNER SCHMIDT, STEVEN LOVEGROVE, MICHAEL GOESELE, ZHAOYANG LV, “Neural 3D Video Synthesis”, arXiv:2103.02597, 2021

However, the technology disclosed in Non-Patent Document 1 requires volume rendering by sampling points on light rays from one end of the target space to the other, which increases the amount of calculations and requires a significant amount of processing time.

The present invention therefore aims to provide an image processing device, image processing method, and program capable of performing volume rendering at high speed.

An image processing device according to one aspect of the present invention comprises: an acquisition means for acquiring a teacher image and a camera position corresponding to the teacher image; a ray calculation means for calculating a ray corresponding to each pixel of the teacher image using the camera position; and a learning parameter calculation means for sampling points on the ray and performing machine learning using the teacher image to calculate learning parameters, wherein the sampling density of the ray within the depth of field range of the teacher image is higher than the sampling density outside the depth of field range.

Other objects and features of the present invention are described in the following embodiments.

The present invention provides an image processing device, image processing method, and program capable of performing volume rendering at high speed.

FIG. 1 is a block diagram of a personal computer according to a first embodiment. 10 is a flowchart of 3D model learning in the first embodiment. FIG. 3 is an explanatory diagram of capturing a teacher image in the first embodiment. 4A and 4B are explanatory diagrams of a teacher image and a focus map according to the first embodiment. 10 is a flowchart of free viewpoint image rendering in the first embodiment. FIG. 2 is an explanatory diagram of a free viewpoint camera according to the first embodiment. 4A and 4B are explanatory diagrams of a teacher image and a focus map according to the first embodiment. FIG. 4 is an explanatory diagram of subject depth calculation in the first embodiment. FIG. 4 is an explanatory diagram of calculation of coordinates of a three-dimensional point in the first embodiment. 5A and 5B are explanatory diagrams of a teacher image and a low-resolution focus map according to the first embodiment. 10A and 10B are explanatory diagrams illustrating the capture of a teacher image and a free viewpoint camera in the second embodiment. 10A and 10B are explanatory diagrams of a peripheral teacher image and a focus map according to the second embodiment. 10A and 10B are explanatory diagrams of a teacher image and a focus map according to the second embodiment. FIG. 10 is a block diagram of a personal computer according to a third embodiment. 10 is a flowchart of dynamic 3D model learning in the third embodiment. FIG. 11 is an explanatory diagram of capturing a teacher image in the third embodiment. FIG. 11 is an explanatory diagram of capturing a teacher image in the third embodiment. 13A and 13B are explanatory diagrams of a teacher image and a focus map according to the third embodiment. 13A and 13B are explanatory diagrams of a teacher image and a focus map according to the third embodiment. 13 is a flowchart of free viewpoint video rendering according to the third embodiment.

Embodiments of the present invention will be described in detail below with reference to the drawings.

(First embodiment)
First, a personal computer (image processing device) according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram of a personal computer (image processing device) 100. Note that this embodiment will be described using a personal computer as an example of an image processing device, but the present invention is not limited to this and can also be applied to image processing devices other than personal computers.

The control unit 101 is, for example, a CPU, which reads out the operation programs for each block of the personal computer 100 from the ROM 102, expands them into the RAM 103, and executes them to control the operation of each block of the personal computer 100. The ROM 102 is a rewritable non-volatile memory such as an SSD, and stores the operation programs for each block of the personal computer 100 as well as parameters required for the operation of each block. The RAM 103 is a rewritable volatile memory such as a DRAM, and is used as a temporary storage area for data output during the operation of each block of the personal computer 100. The data storage unit 104 is a recording medium such as a hard disk that reads and writes image data required for machine learning, metadata for each image, etc.

The shooting camera position and orientation estimation unit 105 estimates the position and orientation of the shooting camera for each image from the image data group using well-known techniques such as SfM (Structure from Motion). In other words, the shooting camera position and orientation estimation unit 105 is an acquisition means for acquiring teacher images and the positions of the cameras corresponding to the teacher images.

The ray calculation unit 106 calculates the rays used in volume rendering using a method such as that disclosed in Non-Patent Document 1. In Non-Patent Document 1, the rays used in volume rendering are defined as r(t) = o + td. Here, o is the camera's principal point in the world coordinate system, d is the ray's direction vector expressed in the world coordinate system, and t is the distance from the camera's principal point to the sampling point on the ray. The ray direction vector d is found by calculating a three-dimensional vector pointing from the camera's principal point to each pixel on the image plane. Furthermore, the camera's principal point o and ray direction vector d are coordinate-transformed from the camera coordinate system to the world coordinate system using the camera position and orientation parameters. In other words, the ray calculation unit 106 is a ray calculation unit that calculates rays corresponding to each pixel of the teacher image using the camera position during learning, and calculates rays corresponding to each pixel of the arbitrary viewpoint camera using the camera position and orientation during inference.

The neural network unit 107 samples points on each ray, calculates the color and density of the corresponding points using a neural network, and performs volume rendering across the target space to determine the color of the pixel corresponding to each ray. For learning and inference, a technique such as that disclosed in Non-Patent Document 1 may be used. During learning, the learning weights are converged using backpropagation, with the L2 loss between the color calculated by volume rendering and the color of the captured image used as a loss function. That is, during learning, the neural network unit 107 is a learning parameter calculation means that performs machine learning by sampling points on ray and using training images to calculate learning parameters. During inference, a free-viewpoint image is rendered by volume-rendering each ray at the free camera position and orientation. That is, during inference, the neural network unit 107 is a rendering means that renders the camera image through machine learning using pre-trained learning parameters sampled by a learning parameter calculation means.

FΘ: (x, d) → (c, σ) ... (1)
In equation (1), FΘ is a neural network consisting of a multi-layer perceptron. It takes as input the three-dimensional coordinate x of the point on the sampled ray and the ray direction vector d. FΘ outputs the RGB color c and density σ for each sampling point. Volume rendering for ray r is expressed as Cvr(r) = ΣTi(1 - exp(-σi δi))ci, i = 1 to N. Here, N is the number of samples, i is the index number for each sampling point, ci and σi are the color and density corresponding to index i, and δi = t(i+1) - t(i), where t(i) is the distance from the camera principal point corresponding to index i to the sampling point on the ray. Also, Ti = exp(-Σσj δj), j = 1 to i-1, which eliminates the influence of the color of objects occluded by occlusion.

The free camera position and orientation acquisition unit 108 is a means for acquiring the camera position and orientation (position of the arbitrary viewpoint camera) for the free viewpoint image. It may acquire camera position and orientation parameters calculated in advance by an external device, or it may acquire the camera position and orientation specified by the user via an operating member such as a joystick.

Next, 3D model learning by the control unit 101 will be described with reference to FIG. 2. FIG. 2 is a flowchart of 3D model learning. First, in step S201, the shooting camera position and orientation estimation unit 105 estimates the camera position and orientation corresponding to each image used for learning. Learning converges to the target learning weights by updating the learning weights through iteration processing. Next, in step S202, the control unit 101 determines whether the iteration processing is complete. If the iteration processing is complete, this flow ends. On the other hand, if the iteration processing is not complete, proceed to step S203.

In step S203, the control unit 101 first randomly selects rays equal to the batch size in each iteration. The batch size may be set to, for example, 4096 rays, as shown in Non-Patent Document 1. In this embodiment, the calculation cost can be efficiently reduced by selecting rays while excluding rays that are not suitable for 3D modeling. This operation will be explained with reference to Figures 3 and 4.

Figure 3 is an explanatory diagram of capturing a teacher image, showing a bird's-eye view to explain the relationship between the subject, the capturing space, and the capturing camera. 301 is the capturing camera (imaging device), 302 is the range of the capturing space to be 3D modeled, 303 is the main subject, and 304 is the background subject. The capturing camera 301 is focused on the main subject 303, and is focused within the angle of view and depth of field indicated by the hatched area 305. Multiple capturing cameras 301 are positioned to capture the subject within the capturing space range 302 from various directions, but for simplicity, only capturing camera 301 is shown in Figure 3.

Figure 4 is an explanatory diagram of a teacher image and a focus map (distance distribution information), showing an image 401 captured by the imaging camera 301 and a focus map 402, which is associated metadata. In Figure 4, the degree of focus on the imaging surface is shown in the form of a grayscale map, with white in the foreground, black in the background, and 50% gray indicating in-focus. As disclosed in Patent Document 2, for example, the focus map 402 can be configured to obtain a defocus map on the imaging surface as a focus map from a phase-difference image obtained from an imaging sensor in which all pixels are phase-difference pixels.

Light rays 306 and 307 in Figure 3 are light rays to be subjected to volume rendering, but since there is no subject within the depth of field on ray 306, it is excluded. Whether or not it is an unnecessary light ray can be determined based on focus map 402 in Figure 4. While pixel 404 corresponding to ray 307 is an in-focus pixel of 50% gray, pixel 403 corresponding to ray 306 is a dark gray outside the depth of field, and is therefore determined to be a light ray to be excluded. In this way, by excluding unnecessary light rays, the volume rendering process can be speeded up. In this embodiment, the acquisition means acquires distance distribution information (focus map) corresponding to the teacher image, and the light ray calculation means determines whether or not the light ray is within the depth of field of the teacher image based on the distance distribution information.

In this embodiment, light rays outside the depth of field are excluded, but this is not limited to this. The light ray calculation unit 106 may select light rays efficiently by varying the density of light rays inside and outside the depth of field (the light ray calculation means may select light rays that are within the depth of field of the teacher image with a focus on that light). For example, it is possible to select all light rays within the depth of field, and to select light rays outside the depth of field that are 10% or less of the total light rays of the target camera.

After rays equal to the batch size are selected in step S203 of Figure 2, the ray calculation unit 106 calculates rays in step S204. As described above, the ray for volume rendering is defined as r(t) = o + td. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray.

The range of distance t sampled for volume rendering is limited to the depth of field indicated by the hatched area 305 in Figure 3. The depth range of hatched area 305 is expressed as Z-Df to Z+Db, using the front depth of field Df, rear depth of field Db, and focused subject distance Z. Furthermore, Df = (r·Av·Z^2)/(f^2+r·Av·Z) and Db = (r·Av·Z^2)/(f^2-r·Av·Z). Here, r is the permissible circle of confusion diameter, Av is the aperture value, and f is the focal length. The permissible circle of confusion diameter r is twice the pixel pitch. In this way, limiting the range of distance t sampled for volume rendering to within the depth of field can speed up the volume rendering process.

In this embodiment, the allowable circle of confusion diameter r is set to twice the pixel pitch, but this is not limited to this and may be set to a coarser or finer value depending on the resolution used when rendering the free viewpoint image. In other words, the allowable circle of confusion diameter for determining the depth of field may be determined based on the resolution used when rendering using learning parameters (rendering resolution).

In this embodiment, the depth of field is defined as Z-Df to Z+Db, but this is not limited to this. Taking into account errors in the focal length f and aperture value Av obtainable from the camera, a more generous range, such as Z-2·Df to Z+2·Db, may be used. In this embodiment, the ray range for volume rendering is limited to the depth of field, but this is not limited to this. Efficient sampling may be achieved by varying the sampling density within and outside the depth of field. In other words, the sampling density of rays within the depth of field of the teacher image may be set higher than the sampling density outside the depth of field. For example, for ray 307 in Figure 3, 32 points are first sampled from the entire range covered by the 3D modeling target range 302, and an additional 128 points are sampled in the range covered by the hatched area 305. In this way, sampling points may be densely arranged only within the depth of field.

After calculating the rays in step S204 of FIG. 2, the neural network unit 107 updates the learning weights in step S205. The control unit 101 determines the learning weights by repeating steps S202 to S205 until the learning weights converge. Note that the completion of the iterations in step S202 can be determined by determining whether 100-300K iterations have been performed, as disclosed in Non-Patent Document 1, for example.

Next, free viewpoint image rendering by the control unit 101 will be described with reference to FIG. 5. FIG. 5 is a flowchart of free viewpoint image rendering. First, in step S501, the free camera position and orientation acquisition unit 108 acquires the camera position and orientation of the free viewpoint to be rendered. Next, in step S502, the control unit 101 determines whether calculation of pixel values (RGB values) by volume rendering has been completed for all pixels of the rendered image. If calculation of pixel values has been completed for all pixels, this flow ends. On the other hand, if calculation of pixel values has not been completed for all pixels, the process proceeds to step S503.

In step S503, the control unit 101 determines whether the corresponding 3D point for each pixel is within the depth of field of the training image, i.e., whether the ray is within the depth of field. If the 3D point is not within the depth of field, the process returns to step S502. On the other hand, if the 3D point is within the depth of field, the process proceeds to step S503, where volume rendering is performed. Note that pixels corresponding to rays outside the depth of field are assigned a fixed pixel value, such as black.

Figure 6 is an explanatory diagram of a free viewpoint camera, showing a bird's-eye view illustrating the relationship between the subject, the shooting space, and each camera. Figure 7 is an explanatory diagram of a teacher image and a focus map, showing an image 701 acquired by the shooting camera 601 and a focus map 702, which is the accompanying metadata. In Figure 6, 603 is the free viewpoint camera that performs rendering, and 301 and 601 are shooting cameras adjacent to the free viewpoint camera 603.

Pixel 404 in Figure 4 and pixel 704 in Figure 7 represent the same three-dimensional point, 607 in Figure 6. Furthermore, ray 605 from free viewpoint camera 603 and ray 307 from shooting camera 301 represent the same three-dimensional point, 607 in Figure 6. If the three-dimensional coordinates of three-dimensional point 607 are known in advance, it can be confirmed that ray 605 corresponds to ray 307, and this can be determined from focus map 402 in Figure 4.

Figure 8 is an explanatory diagram of subject depth calculation, illustrating the procedure for calculating the subject depth Z + ΔZ from the defocus value def for each pixel position indicated by the focus map. 801 is the imaging optical system, 802 is the image plane position, 803 is the focused subject distance position, 804 is the defocused image position, and 805 is the subject distance position. From the lens formula, 1/Z + 1/Z' = 1/f and 1/(Z + ΔZ) + 1/(Z' + def) = 1/f hold, so the subject depth Z + ΔZ can be calculated from these (f: focal length).

If the depth Z + ΔZ of the subject is known, then x/f = X/(Z + ΔZ) and y/f = Y/(Z + ΔZ) hold true from the geometric relationship of the triangle shown in Figure 9, which is an explanatory diagram of calculating the coordinates of three-dimensional points. From these, the X, Y, and Z coordinates (Z + ΔZ) of three-dimensional point 607 can be determined.

From the above, it can be confirmed that ray 605 corresponds to ray 307, and since it is known that ray 307 during training is within the depth of field, ray 605 is subject to volume rendering. On the other hand, pixel 403 in Figure 4 and pixel 703 in Figure 7 represent the same 3D point, 606 in Figure 6, and correspond to ray 306 and ray 604 in Figure 6, respectively. However, since it is known that ray 306 during training is outside the depth of field and the shooting camera 601 did not capture the corresponding 3D point 606, ray 604 is excluded from volume rendering. In this way, by excluding rays corresponding to 3D points that have not been trained from volume rendering, it is possible to speed up the volume rendering process for free viewpoint images.

In step S504, the ray calculation unit 106 calculates the ray for a ray determined to be within the depth of field in step S503 of FIG. 5. As described above, the ray for volume rendering is defined as r(t) = o + td. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. The range of distance t sampled for volume rendering is limited to the depth of field of the adjacent capturing cameras 301 and 601, as indicated by hatched areas 305 and 602 in FIG. 6. In this way, by limiting the range of distance t sampled for volume rendering to the depth of field of the adjacent capturing cameras, the volume rendering process for free viewpoint images can be accelerated.

After the rays are calculated in step S504 of Figure 5, in step S505, the neural network unit 107 performs volume rendering processing and calculates the corresponding pixel values (RGB values).

In this embodiment, the image from the shooting camera (teacher image) and the focus map (distance distribution information) have the same resolution, but this is not limited to this and they may have different resolutions. For example, the resolution of the distance distribution information map may be lower than that of the teacher image. Maps generated based on disparity maps, such as focus maps, are generated by template matching of a predetermined size for stereo correspondence search, so the map size is usually reduced by the size of the template. For example, if the template size is 16 x 16 pixels, the normal map size is 1/16th the size in both height and width. A focus map is also required when rendering a free viewpoint image, but if the focus map stored at this time is a reduced version that is 1/16th the size in both height and width, the data volume required for rendering can be reduced.

Figure 10 is an explanatory diagram of a teacher image and a low-resolution focus map, showing an image 401 acquired by the imaging camera 301 and a focus map 1001, which is the accompanying metadata. The focus map 1001 has a lower resolution by the template size (16 x 16 pixels) for template matching, with one pixel on the focus map corresponding to a 16 x 16 pixel area of the image. Also, as explained with reference to Figure 3, 303 is the main subject and 304 is the background subject.

Pixel 1003 on image 401 corresponds to pixel 1002 on focus map 1001. Pixel 1003 indicates the main subject 303, but the 16 x 16 pixel range of the template includes both the main subject 303 and the ground background. For this reason, the focus map pixel value (defocus value) of pixel 1002 may fall between the defocus values of the main subject 303 and the ground. Therefore, for subject contour regions such as pixel 1003, rays are selected broadly and the sampling range for volume rendering is not restricted. In other words, the ray calculation means identifies the subject contour region based on distance distribution information, makes it easier to select rays for the subject contour region, and sets a wide sampling range.

In this embodiment, distortion due to the lens of the shooting camera is assumed to be very slight, and distortion correction is not performed on the training images, but this is not limited to this, and training images with distortion may also be used. In this case, distortion correction is performed on the training images so that correct light rays can be calculated. Furthermore, distortion correction is also performed on the focus map that is referenced in pairs with the image, so that correct light rays can be selected. In other words, the acquisition means may process the teacher image and distance distribution information based on the distortion components of the camera's optical system.

In this embodiment, the pixel values of the focus map stored in the data storage unit 104 are defocus values, but this is not limited to this and they may also be parallax values or distance values. Furthermore, any format may be used as long as it can be converted into distance values when determining the range of volume rendering. Furthermore, when storing defocus values, focus errors due to lens decentering or tilt of the image sensor may be corrected in advance by the imaging camera, and the defocus values may be recorded as a focus map and stored in the data storage unit 104. In other words, the distance distribution information may include at least one of a map based on the shift amount representing parallax (parallax map), a map based on the defocus amount (defocus map), or a map based on distance (distance map).

In this embodiment, for example, as shown in Figure 3, the range of volume rendering is the hatched area 305 defined by the front depth of field and the rear depth of field, but is not limited to this. For example, it is possible to determine from the focus map whether the subject surface is in front of or behind the subject, and further restrict the range of volume rendering. That is, it is possible to determine whether the subject surface is in front of (in front of) or behind (behind) the subject focus distance based on the distance distribution information (sign of the focus map), and change the sampling density based on the determination result. For example, for subject 303 in Figure 3, the focus map indicates that the subject surface is in front, so the range of volume rendering may be restricted to only the range of the front depth of field.

In addition, in this embodiment, the volume rendering range is determined based on the focus-to-subject distance, but this is not limited to this. For example, a predetermined range may be set as the volume rendering range based on the subject surface position. That is, the distance to the subject surface may be determined based on distance distribution information, and the sampling density may be determined based on the distance to the subject surface. The depth position of the subject surface may be determined using the method described with reference to Figure 8.

In this embodiment, volume rendering is performed using only rays within the depth of field during free viewpoint image rendering (step S503), so pixels corresponding to rays outside the depth of field are assigned a fixed pixel value, such as black, but this is not limited to this. Only pixels outside the depth of field, such as the background, may be rendered using a different method. For example, a spherical environmental texture may be applied to the background, or the background may be rendered from an entirely different background 3D model.

In this embodiment, the focus map can be evaluated across the entire screen, but this is not limited to this. Low-reliability areas where the focus map cannot be calculated can be defined to prevent problems in post-processing. In such cases, all target rays can be selected for low-reliability areas. In other words, the ray calculation means can select all rays for areas (low-reliability areas) that are determined to have low reliability based on distance distribution information, and set a wide sampling range.

According to this embodiment, the rays and sampling range for volume rendering can be appropriately limited, thereby speeding up the volume rendering process.

Second Embodiment
Next, an image processing device according to a second embodiment of the present invention will be described. In this embodiment, the acquisition means further acquires peripheral teacher images of the peripheral space (space including the background subject), and the learning parameter calculation means performs machine learning using the teacher images and the peripheral teacher images to calculate learning parameters. Note that the configuration of the image processing device according to this embodiment is the same as the configuration of the personal computer 100 described in the first embodiment with reference to FIG. 1. In addition, the flowcharts for explaining the 3D model learning operation shown in FIG. 2 and the free viewpoint image rendering operation shown in FIG. 5, which are executed by the control unit 101 in FIG. 1, are also the same as those according to the first embodiment.

Figure 11 is an explanatory diagram of the capture of teacher images and the free viewpoint camera, showing a bird's-eye view illustrating the relationship between the subject, capture space, capture camera, and free viewpoint camera. 1101 and 1105 are the capture cameras, 1103 is the free viewpoint camera, 302 is the range of the capture space for the 3D modeling target, 303 is the main subject, and 304 is the background subject. Capture camera 1101 has a wide-angle focal length and is focused on background subject 304, within the angle of view and depth of field indicated by hatched area 1102. Capture camera 1105 has a standard angle of view focal length and is focused on main subject 303, within the angle of view and depth of field indicated by hatched area 1106.

In this embodiment, in order to photograph the subject within the photographing space range 302 from various directions, multiple photographing cameras are arranged in addition to photographing cameras 1101 and 1105. Specifically, multiple photographing cameras with focal lengths and shooting distances (focus positions) similar to those of photographing camera 1105 are arranged to photograph the main subject 303. Additionally, multiple photographing cameras with focal lengths and shooting distances (focus positions) similar to those of photographing camera 1101 are arranged to photograph subjects other than the main subject 303. For simplicity, only photographing cameras 1101 and 1105 are shown in Figure 11.

Figures 12 and 13 are explanatory diagrams of peripheral teacher images or teacher images and focus maps, showing images 1201 and 1301 captured by the photographing cameras 1101 and 1105, and focus maps 1202 and 1302, which are accompanying metadata. As with the first embodiment, the focus maps are calculated using known technology.

Since the shooting camera 1101 is focused on the background subject 304 at a wide-angle focal length, multiple subjects are captured over a wide area, as shown in image 1201, and the background subject 304 is shown as 50% gray in focus, as shown in focus map 1202. On the other hand, since the shooting camera 1105 is focused on the main subject 303 at a standard-angle focal length, the main subject is mainly captured, as shown in image 1301, and the main subject 303 is shown as 50% gray in focus, as shown in focus map 1302.

With the above configuration, when selecting rays for the batch size (S204) in the 3D model learning of Figure 2, ray 1111 corresponding to 3D point 606 on background subject 304 in standard angle of view camera 1105 of Figure 11 is not selected because it is outside the depth of field. On the other hand, ray 1103 of wide-angle camera 1101 is selected as a target ray because 3D point 606 on background subject 304 is within the depth of field, and not only main subject 303 but also background subject 304 can be included in the 3D model.

Furthermore, during the determination in step S503 in Figure 5, ray 1110 of free viewpoint camera 1108 in Figure 11 is selected as the target ray because it is clear that the corresponding 3D point 607 is captured within the depth of field of ray 1107 of adjacent camera 1105. Furthermore, ray 1109 of free viewpoint camera 1108 in Figure 11 is selected as the target ray because it is clear that the corresponding 3D point 606 is captured within the depth of field of ray 1103 of adjacent camera 1101. In this way, free viewpoint images can be rendered not only for the main subject 303 but also for the background subject 304.

In this embodiment, the imaging cameras 1101 and 1105 are configured to have different focal lengths and shooting distances so that both the main subject and background subject can be volume rendered, but this is not limited to this and only the shooting distances may be different. Increasing the focal length of the imaging camera 1101 for the background subject narrows the angle of view, which may result in an increase in the number of imaging cameras required. Furthermore, shortening the focal length of the imaging camera 1105 for the main subject widens the angle of view, which requires the camera to move closer to the subject, and there is a possibility that the imaging camera 1105 will be reflected in the imaging camera 1101. On the other hand, matching the focal lengths has advantages in terms of preparation for imaging and camera calibration, making it suitable for simple imaging.

In this embodiment, the imaging cameras 1101 and 1105 are configured to have different focal lengths and shooting distances, enabling volume rendering of both the main subject and background subjects. However, this is not limited to this; instead, different aperture values may be used. That is, the imaging camera 1105 for the main subject is set to a bright aperture value near full aperture, while the imaging camera 1101 for the background subject is narrowed down and set to an aperture value so that the entire imaging space range 302 is within the depth of field. This allows the imaging camera 1105 for the main subject to select a slow shutter speed, allowing for sharp images even when the main subject is moving. However, the imaging camera 1101 for the background may be weak against moving subjects or may have increased imaging sensitivity, resulting in increased noise. On the other hand, in cases where multiple background subjects are scattered within the imaging space range 302, this has the advantage of allowing a 3D model to be generated without missing any images. In this way, the peripheral training images need only differ from the training images in at least one of the focus position (shooting distance), focal length, or aperture value.

In this embodiment, a focus map is used to speed up volume rendering even when rendering free-viewpoint images, but this is not limited to this. For example, a focus map may be used to speed up 3D model learning, while free-viewpoint image rendering may be performed by spatial sampling across the entire shooting space range, as in conventional technology. This eliminates the need for a focus map when rendering free-viewpoint images. This means that even renderers using conventional technology can render free-viewpoint images, increasing the versatility of the renderer.

In this embodiment, both the main subject and the background subjects are 3D modeled using the same neural network, but this is not limited to this, and separate neural networks may be configured for the main subject and the background subjects. In other words, the learning parameter calculation means may learn different machine learning models for the teacher image and the peripheral teacher images. This is expected to improve learning efficiency and the reproducibility of the details of the background subject.

According to this embodiment, it is possible to select the optimal light rays for volume rendering for the main subject and background subjects, and to quickly generate 3D models including not only the main subject but also the background subjects, thereby enabling fast rendering of free viewpoint images.

(Third embodiment)
Next, an image processing device according to a third embodiment of the present invention will be described. Fig. 14 is a block diagram of a personal computer (image processing device) 1400. The personal computer 1400 of this embodiment differs from the personal computer 100 having the neural network unit 107 described in the first embodiment in that it has a neural network unit 1401. Note that other components of the personal computer 1400 are similar to those of the personal computer 100, and therefore description thereof will be omitted.

The neural network unit 1401 can handle dynamic scenes (video) where the subject is moving. In such dynamic scenes, points on each ray are sampled, and the color and density of the corresponding points are calculated using a neural network. Volume rendering is then performed across the target space to determine the color of the pixel corresponding to each ray. For learning and inference, the technology disclosed in Non-Patent Document 2, for example, can be used. During learning, the learning weights are converged using backpropagation, with the L2 loss between the color calculated by volume rendering and the color of the captured image used as the loss function. During inference, a free-viewpoint image is rendered by volume rendering each ray at the free camera position and orientation.

FΘ: (x, d, zt) → (c, σ) ... (2)
In equation (2), FΘ is a neural network consisting of a multi-layer perceptron, and inputs the three-dimensional coordinate x of the point on the sampled ray, the ray direction vector d, and the latent code zt in the frame at time t. FΘ outputs the RGB color c and density σ for each sampling point.

The neural network unit 107 of the first embodiment represents only a single 3D model. On the other hand, the neural network unit 1401 of this embodiment can represent scenes in which the 3D shape changes dynamically from frame to frame by changing the latent code zt for each frame.

Next, dynamic 3D model learning by the control unit 101 will be described with reference to Figure 15. Figure 15 is a flowchart of dynamic 3D model learning. First, in step S201, the shooting camera position and orientation estimation unit 105 estimates the camera position and orientation corresponding to each image used for learning. Because the position of the shooting camera is fixed, the camera position and orientation are estimated using the image acquired in the first frame.

Next, in step S1501, the control unit 101 determines whether processing for all frames has been completed. If processing for all frames has been completed, this flow ends. On the other hand, if processing for all frames has not been completed, the process proceeds to step S1502. In step S1502, the control unit 101 generates a latent code corresponding to each frame.

Next, in step S1503, the control unit 101 determines whether the iteration process is complete. Learning for each frame is performed by updating the learning weights through the iteration process, causing the learning weights to converge to the target learning weights. If the iteration process is complete, the process returns to step S1501. On the other hand, if the iteration process is not complete, the process proceeds to step S1504.

In step S1504, the control unit 101 first randomly selects rays equal to the batch size in each iteration. Non-Patent Document 2 discloses a technique for efficiently reducing computational costs by performing ray importance sampling, which selects the next ray for learning based on temporal fluctuations in the input video. In this embodiment, ray importance sampling is performed while also taking into account the depth of field of the input video, thereby achieving further computational efficiency. The operation will be described below with reference to Figures 16 to 19.

Figure 16 is an explanatory diagram of the capture of a teacher image, showing a bird's-eye view illustrating the relationship between the subject, capture space, and capture camera in the first frame of the input video. 1601 is the capture camera, 302 is the range of the capture space to be 3D modeled, 303 is the main subject, and 304 is the background subject. The focus of the capture camera 1601 is on the main subject 303, and the entire capture space range 302 is in focus within the wide-angle angle of view and pan-focus depth of field indicated by the hatched area 1605. Multiple capture cameras 1601 are positioned to capture the subject within the capture space range 302 from various directions, but for simplicity, only the capture camera 1601 is shown in Figure 16.

In subsequent frames, the shooting camera 1601 changes the focal length to a standard angle of view and the aperture to a brighter, wider aperture (F-number), thereby changing the camera parameters so that the main subject 303 can be captured with higher resolution. Figure 17 is an explanatory diagram of capturing a teacher image, showing a bird's-eye view illustrating the relationship between the subject, shooting space, and shooting camera after the camera parameters have been changed. The shooting camera 1601 is focused on the main subject 303, and the shooting space range 302 is partially in focus within the standard angle of view and shallow depth of field indicated by the hatched area 1705.

Figure 18 is an explanatory diagram of the teacher image and focus map, showing an image 1801 acquired by the shooting camera 1601 in the first frame of the input video and a focus map 1802, which is the associated metadata. In Figure 18, the degree of focus on the imaging surface is shown in the form of a grayscale map, with white in the foreground, black in the background, and 50% gray indicating in-focus. Figure 19 is an explanatory diagram of the teacher image and focus map, showing an image 1901 acquired by the shooting camera 1601 after a change in camera parameters, and a focus map 1902, which is the associated metadata.

In the first frame of the input video, rays 1606 and 1607 in Figure 16 are rays to be rendered for volume rendering, and the corresponding 3D points are within the depth of field, so both rays are selected. On the other hand, rays 1706 and 1707 in Figure 17 after the camera parameters have been changed are also rays to be rendered for volume rendering, but since there is no subject within the depth of field on ray 1706, it is excluded. Whether or not a ray is unnecessary can be determined using focus map 1902 in Figure 19. Pixel 1904 corresponding to ray 1707 is an in-focus pixel that is 50% gray, while pixel 1903 corresponding to ray 1706 is a dark gray outside the depth of field, so it is determined to be a ray to be excluded.

After selecting rays for the batch size in step S1504 of Figure 15, the ray calculation unit 106 calculates rays in step S1505. As described above, the ray for volume rendering is defined as r(t) = o + td. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. The range of distance t sampled for volume rendering is limited within the depth of field indicated by the hatched area 1605 in Figure 16 and the hatched area 1705 in Figure 17. In this way, by performing ray importance sampling while taking the depth of field into consideration, it is possible to speed up the volume rendering process.

In this embodiment, even in the case of Figure 16 where the depth of field is deep, the sampling range t is limited according to the depth of field, but this is not limited. In such cases, the effect of limiting the sampling range is low, so a sufficiently wide fixed range may be set to fix the sampling range.

After calculating the rays in step S1505 of Figure 15, the neural network unit 1401 updates the learning weights in step S1506. The learning weights are determined by repeating the operations of steps S1503 to S1506 until the learning weights converge. When processing of all frames of the input video is complete, this flow ends (step S1501).

In this embodiment, ray importance sampling is performed based on the depth of field, but this is not limited to this. Ray importance sampling may also be performed based on both the depth of field and the temporal variation of the input video. In many cases, the input video has a higher resolution than the focus map. Therefore, by observing the temporal variation of the input video, it is possible to remove unnecessary light rays with higher resolution and then limit the sampling range based on the focus map, which is expected to further improve computational efficiency.

Next, free viewpoint image rendering by the control unit 101 will be described with reference to FIG. 20. FIG. 20 is a flowchart of free viewpoint image rendering in this embodiment. First, in step S501, the free camera position and orientation acquisition unit 108 acquires the camera position and orientation of the free viewpoint to be rendered.

Next, in step S2001, the control unit 101 determines whether processing for all frames has been completed. If processing for all frames has been completed, this flow ends. On the other hand, if processing for all frames has not been completed, the process proceeds to step S2002, where pixel values (RGB values) are calculated by volume rendering for all pixels of the rendered image while updating the latent code Zt for each frame. That is, in step S2002, the control unit 101 generates a latent code corresponding to each frame. Next, in step S2003, the control unit 101 determines whether calculation of pixel values (RGB values) by volume rendering has been completed for all pixels of the rendered image. If calculation of pixel values for all pixels has been completed, the process returns to step S2001. On the other hand, if calculation of pixel values for all pixels has not been completed, the process proceeds to step S2004.

In step S2004, the ray calculation unit 106 calculates the ray corresponding to each pixel (S2004). As described above, the ray to be volume rendered is defined as r(t) = o + td. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. After the ray is calculated in step S2004, in step S2005, the neural network unit 107 executes volume rendering processing and calculates the corresponding pixel value (RGB value).

In this embodiment, the focal length and depth of field are changed between the beginning and subsequent portions of the input video, but this is not limited to this. For example, while shooting with a standard angle of view and shallow depth of field, multiple wide-angle, pan-focus shots may be inserted. This makes it possible to accommodate gradual changes in the surrounding environment, such as sunlight, during the course of a long input video.

In this embodiment, the focal length and depth of field are changed between the beginning and subsequent portions of the input video, but this is not limited to this; it is also possible to control only the aperture and change only the depth of field. This makes it possible to shoot with a prime lens that does not have a zoom mechanism. Alternatively, multiple wide-angle prime cameras and multiple standard-angle prime cameras may be arranged. Furthermore, with a zoom lens, the depth of field will deepen even if the focal length is changed from a standard angle of view to a wide angle while keeping the aperture wide open, so if this is sufficient, it is also possible to control only the focal length.

In this embodiment, the focal length and depth of field are changed between the beginning and subsequent parts of the input video, but this is not limited to this; the focus may be gradually changed from the background to the main subject. This allows not only the main subject but also other subjects to be captured in high definition. Changing the focus also changes the image magnification, so by further correcting for this change in image magnification, the 3D model can be trained with higher accuracy.

In this embodiment, the teacher image is a video captured at a predetermined frame rate. The learning parameter calculation means calculates, for each frame of the video, a code feature (latent code) that indicates the characteristics of that frame, and performs machine learning using the code feature and the teacher image for each frame of the video to calculate learning parameters. In this embodiment, the acquisition means acquires regions in the teacher image that change over time (regions where there is a significant change in appearance over time), and the ray calculation means may focus on selecting ray beams in the regions acquired by the acquisition means and output corresponding ray beams. Also in this embodiment, the teacher image is a video captured so as to include frames that differ in at least one of focus position, focal length, or aperture value.

According to this embodiment, volume rendering can be performed efficiently even in dynamic scenes where the subject is moving, allowing 3D models to be generated quickly.

(Other embodiments)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

According to each embodiment, by appropriately limiting the rays and sampling range used for volume rendering, it is possible to measure the detailed shape of an object at high speed and reconstruct a realistic image. Therefore, each embodiment can provide an image processing device, image processing method, and program capable of performing volume rendering at high speed.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the invention.

100 Personal computer (image processing device)
105 Shooting camera position and orientation estimation unit (acquisition means)
106 Light ray calculation unit (light ray calculation means)
107 Neural network unit (learning parameter calculation means, rendering means)
108 Free camera position and orientation acquisition unit (acquisition means)

Claims (20)

an acquisition means for acquiring a teacher image and a camera position corresponding to the teacher image;
a ray calculation means for calculating a ray corresponding to each pixel of the teacher image using the position of the camera;
a learning parameter calculation means for performing machine learning by sampling points on the light ray and using the teacher image, and calculating learning parameters;
An image processing device, characterized in that the sampling density of the light rays within the range of the depth of field of the teacher image is higher than the sampling density outside the range of the depth of field. An acquisition means for acquiring the position of the camera;
a ray calculation means for calculating a ray corresponding to each pixel of the camera using the position of the camera;
a rendering means for rendering an image of the camera by machine learning using learning parameters that have been learned in advance by sampling points on the light ray by a learning parameter calculation means;
An image processing device characterized in that the sampling density of the light rays within the range of the depth of field of the teacher image used in the pre-learning is higher than the sampling density outside the range of the depth of field. the acquiring means further acquires distance distribution information corresponding to the teacher image,
3. The image processing apparatus according to claim 1, wherein the ray calculation means determines whether the ray is within the depth of field of the teacher image based on the distance distribution information. It is determined whether the object surface is closer to or further from the object focus distance based on the distance distribution information,
4. The image processing device according to claim 3, wherein the sampling density is changed based on a determination result as to whether the object surface is closer to or further from the object focus distance. a distance to a surface of the object is determined based on the distance distribution information;
5. The image processing apparatus according to claim 3, wherein the sampling density is determined based on the distance to the surface of the subject. An image processing device according to any one of claims 3 to 5, characterized in that the distance distribution information has a lower resolution than the training image. 7. The image processing device according to claim 3, wherein the ray calculation means identifies a subject contour area based on the distance distribution information, makes it easier to select rays for the subject contour area, and sets a wide sampling range. An image processing device according to any one of claims 3 to 7, characterized in that the ray calculation means selects all rays for areas determined to have low reliability based on the distance distribution information and sets a wide sampling range. An image processing device according to any one of claims 3 to 8, characterized in that the acquisition means processes the teacher image and the distance distribution information based on distortion components of the camera's optical system. An image processing device according to any one of claims 3 to 9, characterized in that the distance distribution information includes at least one of a map based on a shift amount representing parallax, a map based on a defocus amount, or a map based on distance. The acquisition means further acquires a peripheral teacher image of the peripheral space,
The image processing device according to claim 1 , wherein the learning parameter calculation means performs the machine learning by using the teacher image and the peripheral teacher images, and calculates the learning parameters. The image processing device described in claim 11, characterized in that the peripheral teacher image is an image that differs from the teacher image in at least one of focus position, focal length, and aperture value. The image processing device described in claim 11 or 12, characterized in that the learning parameter calculation means learns different machine learning models for the teacher image and the surrounding teacher images. the teacher image is a video captured at a predetermined frame rate,
The learning parameter calculation means
calculating a code feature that indicates a feature of each frame of the video;
The image processing device according to any one of claims 1 to 13, characterized in that machine learning is performed by using the chord features and the teacher image for each frame of the video, and the learning parameters are calculated. the acquiring means acquires a region that changes over time in the teacher image,
15. The image processing apparatus according to claim 14, wherein the light ray calculation means selects light rays intensively from the area acquired by the acquisition means, and outputs corresponding light rays. An image processing device as described in claim 14 or 15, characterized in that the teacher image is a video captured to include frames that differ in at least one of focus position, focal length, or aperture value. An image processing device according to any one of claims 1 to 16, characterized in that the allowable circle of confusion diameter for determining the depth of field is determined based on the resolution during rendering using the learning parameters. acquiring a teacher image and a camera position corresponding to the teacher image;
calculating a ray corresponding to each pixel of the teacher image using the position of the camera;
performing machine learning by sampling points on the ray and using the training image to calculate learning parameters;
An image processing method, characterized in that the sampling density of the light rays within the depth of field range of the teacher image is higher than the sampling density outside the depth of field range. obtaining a camera position;
calculating a ray corresponding to each pixel of the camera using the position of the camera;
and rendering the image of the camera by machine learning using learning parameters pre-trained by sampling points on the light ray by a learning parameter calculation means;
An image processing method characterized in that the sampling density of the light rays within the depth of field range of the teacher image used in the pre-learning is higher than the sampling density outside the depth of field range. A program causing a computer to execute the image processing method described in claim 18 or 19.