JP2022019341A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate three-dimensional shape data of an object at a time different from the shooting time based on a shot image obtained by shooting the object.
SOLUTION: An information processing device has a shape representing a three-dimensional shape of an object at a predetermined shooting time based on a plurality of shot images obtained by shooting an object from different directions by a plurality of shooting devices. Generate data. Further, the information processing apparatus acquires the first posture information representing the posture of the object at the shooting time and the second posture information representing the posture of the object at a specific time different from the shooting time. Then, the information processing apparatus generates shape data representing the three-dimensional shape of the object at a specific time based on the first posture information and the second posture information and the shape data corresponding to the shooting time.
[Selection diagram] Fig. 9

Description

The present invention relates to a technique for generating a three-dimensional model of an object using a plurality of captured images.

There is a technique in which a plurality of photographing devices are installed at different positions to perform synchronous photography from multiple viewpoints, and a virtual viewpoint image representing a scene viewed from an arbitrary viewpoint is generated using the multiple viewpoint images obtained by the photographing. With such technology, for example, it is possible to watch highlight scenes of games such as soccer and basketball, concerts, etc. from various angles, and give the user a high sense of presence as compared with ordinary images. Can be done.

As a method of generating a virtual viewpoint image, three-dimensional shape data of an object in the shooting area is generated using images taken by a plurality of shooting devices, and rendering processing is performed using the three-dimensional shape data to perform a virtual viewpoint. There is a way to generate an image. Further, in Patent Document 1, a preset adjustable three-dimensional object template model is adjusted based on object three-dimensional information obtained from a plurality of camera images, and the adjusted model is subjected to projective transformation. It is described that a virtual viewpoint image is generated in.

Japanese Unexamined Patent Publication No. 2016-126425

Although the photographing device generates a captured image at a predetermined frame rate, it may be required to generate three-dimensional shape data of an object at a time different from the time corresponding to the frame of the captured image. For example, when a virtual viewpoint image is displayed on a device capable of displaying an image at a frame rate higher than the frame rate of the captured image, smooth playback of a moving image is possible by using a virtual viewpoint image having a high frame rate. Further, for example, by slow-playing a virtual viewpoint image having a high frame rate, it is possible to smoothly play a slow moving image. In order to generate a virtual viewpoint image having a frame rate higher than the frame rate of the captured image, it is required to generate three-dimensional shape data at a time different from the time corresponding to the frame of the captured image. However, with the conventional method, it is not possible to acquire the three-dimensional shape data of the object at the time when the shooting is not performed.

The present invention has been made in view of the above problems, and an object of the present invention is to generate three-dimensional shape data of an object at a time different from the shooting time based on a shot image obtained by shooting the object. ..

In order to solve the above problems, the information processing apparatus according to the present invention has, for example, the following configuration. That is, shape data representing the three-dimensional shape of the object at the predetermined shooting time is generated based on a plurality of shot images obtained by shooting the object from different directions by a plurality of shooting devices at a predetermined shooting time. The first generation means, the first posture information representing the posture of the object at the predetermined shooting time, and the second posture information representing the posture of the object at a specific time different from the predetermined shooting time are acquired. Based on the acquisition means, the first posture information and the second posture information acquired by the acquisition means, and the shape data generated by the generation means, the three-dimensional shape of the object at the specific time is obtained. It has a second generation means for generating the shape data to be represented.

According to the present invention, it is possible to generate three-dimensional shape data of an object at a time different from the shooting time based on the shot image obtained by shooting the object.

It is a figure which shows the configuration example of an image generation system. It is a figure which shows the configuration example of an image generation apparatus. It is a figure for demonstrating a three-dimensional model and posture information. It is a figure which shows the time relation of a photographed image, a three-dimensional model, and posture information. It is a flowchart for demonstrating the generation process of the interpolation 3D model by an image generation apparatus. It is a figure for demonstrating the interpolation method of a bone model. It is a flowchart for demonstrating the process of generating the interpolated three-dimensional model using the interpolated posture information. It is a figure for demonstrating the interpolation method of a three-dimensional model. It is a figure which shows the time relationship between a photographed image and an interpolated three-dimensional model. It is a flowchart for demonstrating operation of an image generator. It is a figure which shows the time relationship between a photographed image and an interpolated three-dimensional model. It is a figure for demonstrating the interpolation method of a bone model.

[System configuration]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 shows a configuration example of the image generation system 100. The image generation system 100 generates a virtual viewpoint image representing a view from a virtual viewpoint based on a plurality of images (multiple viewpoint images) taken by a plurality of photographing devices and a virtual viewpoint position and line-of-sight direction. It is a system. The virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, for example, a viewpoint selected by the user from a plurality of candidates. Corresponding images and the like are also included in the virtual viewpoint image. Further, in the present embodiment, the case where the virtual viewpoint is specified by the user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like. The image generation system 100 generates a virtual viewpoint image to be reproduced by updating a virtual viewpoint image of a still image as an image of a frame constituting the moving image at a predetermined frame update interval. In the following description, unless otherwise specified, the word "image" will be described as including the concepts of both moving images and still images.

Further, in the present embodiment, an example in which the image generation system 100 provides virtual viewpoint contents including a virtual viewpoint image and a virtual viewpoint sound will be mainly described. However, the virtual viewpoint content does not have to include audio. Further, the sound included in the virtual viewpoint content may be the sound collected by the microphone closest to the virtual viewpoint. Further, in the present embodiment, for the sake of simplification of the explanation, the description about the sound is partially omitted, but basically both the image and the sound are processed.

The image generation system 100 includes a sensor system 110a to a sensor system 110z, an image generation device 122, a controller 123, a switching hub 121, an end user terminal 126, and a time server 127.

The sensor system 110a includes a microphone 111a, a camera 112a, a pan head 113a, an external sensor 114a, and a camera adapter 120a. The sensor system 110a is not limited to this configuration, and may have at least one camera 112a or microphone 111a. Further, for example, the sensor system 110a may be composed of one camera adapter 120a and a plurality of cameras 112a, or may be composed of one camera 112a and a plurality of camera adapters 120a. That is, the plurality of cameras 112 and the plurality of camera adapters 120 in the image generation system 100 correspond to each other by N to M (N and M are both integers of 1 or more). Further, the sensor system 110a may include a device other than the microphone 111a, the camera 112a, the pan head 113a, and the camera adapter 120a. Further, the camera 112a and the camera adapter 120a may be integrally configured.

The sound collected by the microphone 111a and the image captured by the camera 112a are transmitted to the switching hub 121 via the camera adapter 120a. Although the present embodiment shows an example in which the camera 112a and the camera adapter 120a are separated from each other, they may be integrated in the same housing. In that case, the microphone 111a may be built in the integrated camera 112a or may be connected to the outside of the camera 112a.

In the present embodiment, the sensor system 110b to the sensor system 110z have the same configuration as the sensor system 110a. However, the present invention is not limited to this, and each sensor system 110 may have a different configuration. In the present embodiment, when the 26 sets of systems from the sensor system 110a to the sensor system 110z are not particularly distinguished, they are referred to as the sensor system 110. Similarly, the device in the sensor system 110 is described as a microphone 111, a camera 112, a pan head 113, an external sensor 114, and a camera adapter 120, unless otherwise specified. Although FIG. 1 shows an example of 26 sets of sensor systems, the number of sensor systems 110 included in the image generation system 100 is not limited to this.

Each of the plurality of sensor systems 110 has one camera 112. That is, the image generation system 100 has a camera 112 as a plurality of photographing devices for photographing a subject from a plurality of directions. The shooting area photographed by the plurality of cameras 112 is, for example, a stadium where competitions such as soccer and karate are performed, or a stage where concerts and performances are performed. The plurality of cameras 112 are installed at different positions so as to surround such a shooting area, and shoot in synchronization with each other. The plurality of cameras 112 may not be installed over the entire circumference of the shooting area, and may be installed only in a part of the periphery of the shooting area depending on the limitation of the installation location or the like. Further, the plurality of cameras 112 may include photographing devices having different functions such as a telephoto camera and a wide-angle camera.

The plurality of sensor systems 110 are connected to the switching hub 121, and form a star-shaped network in which data is transmitted and received between the plurality of sensor systems 110 via the switching hub 121. Further, each of the plurality of sensor systems 110 is connected to the image generation device 122 via the switching hub 121, and outputs a plurality of viewpoint images based on the images taken by the plurality of cameras 112 to the image generation device 122.

The time server 127 has a function of distributing the time and synchronization signals, and distributes the time and synchronization signals to a plurality of sensor systems 110 via the switching hub 121. The camera adapter 120 that has received the time and synchronization signal genlocks the camera 112 based on the time and synchronization signal to synchronize the image frame. That is, the time server 127 synchronizes the shooting timings of the plurality of cameras 112. As a result, the image generation system 100 can generate a virtual viewpoint image based on a plurality of captured images taken at the same timing, so that it is possible to suppress deterioration of the quality of the virtual viewpoint image due to a deviation in the shooting timing. In the present embodiment, the time server 127 manages the time synchronization of a plurality of cameras 112, but the present invention is not limited to this, and the camera 112 or the camera adapter 120 may independently perform the processing for time synchronization. good.

The controller 123 has a control station 124 and a virtual camera operation UI 125. The control station 124 is connected to each device constituting the image generation system 100 via a network, and manages the operating state of each device and controls parameter setting. Here, the network may be an IEEE (registered trademark) compliant GbE (Gigabit Ethernet) or 10 GbE, or may be configured by combining an interconnect Infiniband, an industrial Ethernet, or the like. Further, the network is not limited to these, and may be another type of network.

Specifically, the control station 124 executes various settings and controls for the image generation system 100. Further, the control station 124 transmits a three-dimensional model of the stadium or the like to be photographed to the image generation device 122. Further, the control station 124 calibrates the plurality of cameras 112. In camera calibration, a marker is placed on the field to be photographed, images are taken by a plurality of cameras 112, and the position and orientation of each camera 112 in the world coordinate system and the focal length are calculated from the captured images. The calculated position, orientation, and focal length information of the camera 112 is transmitted to the image generator 122. The transmitted 3D model and camera 112 information is used by the image generator 122 to generate a virtual viewpoint image.

The virtual camera operation UI 125 accepts a user operation for designating a virtual viewpoint corresponding to the virtual viewpoint image to be generated, and transmits the viewpoint information corresponding to the user operation to the image generation device 122 that generates the virtual viewpoint image. The viewpoint information used to generate the virtual viewpoint image is information indicating the position and direction (line-of-sight direction) of the virtual viewpoint. Specifically, the viewpoint information has a parameter set including a parameter representing a three-dimensional position of the virtual viewpoint and a parameter representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. In addition, the viewpoint information has a plurality of parameter sets corresponding to a plurality of time points. For example, the viewpoint information has a plurality of parameter sets corresponding to a plurality of frames constituting a moving image of the virtual viewpoint image, and indicates the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points. The content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter representing the size (angle of view) of the field of view of the virtual viewpoint and a parameter representing the time.

The image generation device 122 generates a virtual viewpoint image based on the plurality of viewpoint images acquired from the plurality of sensor systems 110 and the viewpoint information acquired from the virtual camera operation UI 125. The virtual viewpoint image is generated by, for example, the following method. First, a foreground image in which a foreground area corresponding to a predetermined object such as a person or a ball is extracted from a multi-viewpoint image obtained by taking images from different directions by a plurality of image pickup devices and a background area other than the foreground area are obtained. The extracted background image is acquired. In addition, a foreground model representing the three-dimensional shape of a predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and the background model representing the three-dimensional shape of the background such as a stadium is colored. Texture data is generated based on the background image. Then, a virtual viewpoint image is generated by mapping the texture data to the foreground model and the background model and performing rendering according to the virtual viewpoint indicated by the viewpoint information. However, the method of generating the virtual viewpoint image is not limited to this, and various methods such as a method of generating a virtual viewpoint image by projective transformation of the captured image without using a three-dimensional model can be used.

The virtual viewpoint image generated by the image generation device 122 is transmitted to the end user terminal 126 and displayed on the display screen of the end user terminal 126. Note that the end user terminal 126 may output the viewpoint information corresponding to the user operation for designating the virtual viewpoint to the image generation device 122, similarly to the virtual camera operation UI 125. As a result, the user who operates the end user terminal 126 can view images and listen to audio according to the designation of the viewpoint.

The image generator 122 converts the virtual viewpoint image into H. Data may be transmitted to the end user terminal 126 using the MPEG-DASH protocol after being compressed and coded by a standard technique such as 264 or HEVC. Further, the virtual viewpoint image may be transmitted to the end user terminal 126 without compression. For example, when a smartphone or tablet is used as the end user terminal 126, compression coding may be performed, and when the end user terminal 126 is a display capable of displaying an uncompressed image, the uncompressed image may be transmitted. .. That is, the image format can be switched according to the type of the end user terminal 126. Further, the image transmission protocol is not limited to MPEG-DASH, and for example, HLS (HTTP Live Streaming) or other transmission method may be used.

[Hardware configuration]
The hardware configuration of the image generation device 122 as an example of the information processing device included in the image generation system 100 will be described with reference to FIG. 2A. The hardware configuration of the other devices included in the image generation system 100 shown in FIG. 1 may be the same as the configuration of the image generation device 122 described below. The image generation device 122 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication I / F 217, and a bus 218.

The CPU 211 realizes each function of the image generation device 122 shown in FIG. 2B by controlling the entire image generation device 122 by using computer programs and data stored in the ROM 212 and the RAM 213. The image generation device 122 may have one or more dedicated hardware different from the CPU 211, and the dedicated hardware may execute at least a part of the processing by the CPU 211. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 212 stores programs and the like that do not require changes. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, data supplied from the outside via the communication I / F 217, and the like. The auxiliary storage device 214 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display unit 215 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the image generation device 122. The operation unit 216 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 211 in response to an operation by the user. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216. The communication I / F 217 is used for communication with an external device of the image generation device 122. For example, when the image generation device 122 is connected to an external device by wire, a communication cable is connected to the communication I / F 217. When the image generator 122 has a function of wirelessly communicating with an external device, the communication I / F 217 includes an antenna. The bus 218 connects each part of the image generation device 122 to transmit information.

In the present embodiment, it is assumed that the display unit 215 and the operation unit 216 exist inside the image generation device 122, but at least one of the display unit 215 and the operation unit 216 exists as another device outside the image generation device 122. You may be doing it.

[Functional configuration]
FIG. 2B is a diagram showing an example of the functional configuration of the image generation device 122. The data receiving unit 201 receives image data based on shooting by a plurality of cameras 112 via the switching hub 121. The image data received here may be a captured image captured by the camera 112, or may be an image obtained by extracting a region corresponding to a specific object from the captured image. In the present embodiment, the image data acquired by the data receiving unit 201 is assumed to be a captured image of a moving image composed of a plurality of frames. That is, the data receiving unit 201 acquires a plurality of moving images based on shooting by a plurality of shooting devices in a predetermined shooting period.

The model generation unit 202 uses the image data acquired by the data reception unit 201 to generate a three-dimensional model representing the three-dimensional shape of the object in the shooting area at each shooting time corresponding to the frame of the shot image. There are various methods for generating a three-dimensional model, but in this embodiment, a subject area observed from a plurality of cameras 112 among voxels in a three-dimensional space, which is called Visual Hull or visual volume crossing method, is left. Therefore, the method of acquiring a three-dimensional model is used. However, the method of generating a three-dimensional model by the model generation unit 202 is not limited to this. In addition, there are various methods for expressing a three-dimensional model, but in this embodiment, a three-dimensional model expressed by a set of voxels (points) is dealt with. However, the three-dimensional model may be represented by polygons or the like. The details of the three-dimensional model will be described later.

The posture estimation unit 203 uses the image data acquired by the data receiving unit 201 to generate posture information representing the posture of the object in the shooting area at each shooting time corresponding to the frame of the shot image. As a method of generating posture information, in this embodiment, posture estimation using deep learning is used. Further, in the present embodiment, the posture information is assumed to be information representing a bone model representing the skeleton of the target object. However, the content and generation method of the posture information are not limited to these. The details of the posture information will be described later. The posture interpolation unit 204 utilizes the posture information generated by the posture estimation unit 203 at each of the plurality of times, and generates the posture information at a time intermediate between those times by interpolation. The information of the time to be interpolated is instructed by the control unit 208.

The motion vector calculation unit 205 obtains a motion vector showing the difference between the bone model represented by the posture information generated by the posture estimation unit 203 and the bone model represented by the posture information interpolated by the posture interpolation unit 204. get. The model interpolation unit 206 uses the three-dimensional model generated by the model generation unit 202 and the motion vector obtained by the motion vector calculation unit 205 to generate a three-dimensional model at the time of interpolation.

The rendering processing unit 207 performs a process of generating a virtual viewpoint image based on the data of the three-dimensional model and the image data acquired by the data receiving unit 201. The control unit 208 controls the order of each process performed by the image generation device 122.

[3D model and posture information]
The three-dimensional model and the posture information will be described with reference to FIG. FIG. 3 is a schematic diagram showing a person who is an example of an object in the shooting area, a three-dimensional model thereof, and a bone model. The three-dimensional model is a model showing the position and shape of the object in the three-dimensional space, and the bone model is a model showing the posture of the object in the three-dimensional space. Express in. Based on the image data obtained by photographing the object 301 in the photographing area by the plurality of cameras 112, the three-dimensional shape data representing the three-dimensional model 302 and the attitude information representing the bone model 303 are generated.

The three-dimensional model 302 in this embodiment is represented by a point cloud which is a set of voxels. The point cloud is represented by three-dimensional position information (x, y, z) of each voxel in the three-dimensional space and information indicating the size of one voxel. A voxel is a cube, and the size of a voxel is expressed, for example, by the length of one side. Since the three-dimensional shape of the object 301 is represented by the set of voxels, the accuracy of the three-dimensional shape represented by the three-dimensional model 302 becomes higher as the voxels are finer. On the other hand, if the voxels are fine, the number of voxels constituting the three-dimensional model 302 increases, so that the amount of information (data size of the three-dimensional shape data) of the three-dimensional model increases.

As shown in FIG. 3, the bone model 303 represented by the posture information is composed of structural main nodes of the object 301 and lines connecting the nodes. Since the bone model 303 has a smaller amount of information than the three-dimensional model 302, it is possible to express the rough movement and posture state of the object 301 with a data size smaller than that of the three-dimensional shape data.

FIG. 4 will explain the temporal relationship between the photographed image acquired by the camera 112, the three-dimensional model generated by the model generation unit 202, and the attitude information generated by the attitude estimation unit 203. In the present embodiment, it is assumed that the shooting frame rate (frame rate of the shot image) of the camera 112 is 60 fps. That is, one frame of captured image is acquired by the camera 112 every 1/60 second. The three-dimensional model and the attitude information are also generated based on the captured image at the same frame rate of 60 fps as the captured image. When a virtual viewpoint image is generated using such a three-dimensional model having a frame rate of 60 fps, the frame rate of the virtual viewpoint image is also 60 fps.

On the other hand, it may be required to generate a virtual viewpoint image having a frame rate higher than the frame rate of the captured image. Therefore, the image generation system 100 generates a virtual viewpoint image of 120 fps by generating a three-dimensional model at a time different from the time corresponding to the captured image by interpolation. Specifically, the posture interpolation unit 204 generates posture information at a time intermediate between the shooting times corresponding to those frames from the posture information corresponding to each of the two temporally continuous frames by interpolation. Then, the model interpolation unit 206 generates a three-dimensional model corresponding to the attitude information generated by the interpolation at the same time based on the attitude information generated by the attitude interpolation unit 204.

In FIG. 9, the posture information generated by interpolation (hereinafter referred to as interpolated posture information) and the three-dimensional model generated based on the interpolated posture information (hereinafter referred to as interpolated three-dimensional model) are temporally positioned. Is shown. One frame is acquired every 1/60 second of the captured image, and the posture information and the three-dimensional model are obtained every 1/120 second by generating the interpolated posture information and the interpolated three-dimensional model. By using this three-dimensional model, it is possible to generate a virtual viewpoint image of 120 fps, which is twice the frame rate of the captured image.

[Operation flow]
FIG. 10 is a flowchart showing an example of the operation of the image generation device 122. The process shown in FIG. 10 is realized by the CPU 211 of the image generation device 122 expanding the program stored in the ROM 212 into the RAM 213 and executing the program. It should be noted that at least a part of the processing shown in FIG. 10 may be realized by one or a plurality of dedicated hardware different from the CPU 211. The process shown in FIG. 10 is taken by a plurality of cameras 112, and is started at the timing when an instruction for generating a virtual viewpoint image is input to the image generation device 122. However, the start timing of the process shown in FIG. 10 is not limited to this. The process shown in FIG. 10 may be executed during shooting by a plurality of cameras 112, or may be executed after shooting is completed and a shot image is recorded.

In S1001, the data receiving unit 201 acquires a photographed image based on the image taken by the plurality of cameras 112. In S1002, the model generation unit 202 generates three-dimensional shape data representing a three-dimensional model at the same time as the time of the photographed image based on the photographed image. This three-dimensional model will be referred to as a reference three-dimensional model below. In S1003, the posture estimation unit 203 generates posture information at the same time as the time of the photographed image based on the photographed image. This posture information will be referred to as reference posture information below.

In S1004, the attitude interpolation unit 204, the motion vector calculation unit 205, and the model interpolation unit 206 generate three-dimensional shape data representing the interpolation three-dimensional model based on the reference three-dimensional model and the reference attitude information. In S1005, the rendering processing unit 207 renders the virtual viewpoint image of the reference frame using the reference three-dimensional model. The reference frame of the virtual viewpoint image is a frame corresponding to the same time as the frame of the captured image. In S1006, the rendering processing unit 207 renders the virtual viewpoint image of the interpolated frame using the interpolated three-dimensional model. The interpolation frame of the virtual viewpoint image is a frame corresponding to a time different from the frame of the captured image, and is a frame inserted between two consecutive reference frames.

The rendering process in S1004 and S1005 generates a virtual viewpoint image having a frame rate higher than the frame rate of the captured image. In S1007, the rendering processing unit 207 outputs the generated virtual viewpoint image to the end user terminal 126. The output virtual viewpoint image is displayed on the screen of the end user terminal 126. In this way, by generating a virtual viewpoint image with a frame rate higher than the frame rate of the captured image, for example, when displaying the virtual viewpoint image on a device capable of displaying the image at a frame rate higher than the frame rate of the captured image. , Smooth video playback is possible. Further, for example, by slow-playing a virtual viewpoint image having a high frame rate, it is possible to smoothly play a slow moving image.

Next, the details of the process of generating the interpolated three-dimensional model in S1004 will be described with reference to FIG. In S501, the control unit 208 acquires the time information of the interpolation frame to be generated by interpolation. In the present embodiment, since the virtual viewpoint image of 120 fps is generated from the captured image of 60 fps, the time information of the interpolated frame indicates an intermediate time of the time corresponding to each of the plurality of reference frames. The time information of the interpolated frame is acquired based on the user operation. For example, when the user performs an operation of designating "120 fps" or "double speed", the time information of the interpolated frame for generating the virtual viewpoint image of 120 fps is acquired. However, the method of acquiring the time information of the interpolated frame is not limited to this, and the control unit 208 may acquire the time information determined based on the situation of the object in the shooting area, the event to be shot, and the like.

In S502, the posture interpolation unit 204 generates posture information at the time corresponding to the interpolation frame from the reference posture information corresponding to the reference frames before and after the interpolation frame by interpolation. The method of interpolating the posture information performed in S502 will be described with reference to FIG. Here, an example of generating the attitude information of the interpolated frame corresponding to the time in the middle of two consecutive reference frames, frame N and frame N + 1, will be described.

The bone model 600 is a bone model represented by the posture information of the frame N, and represents the posture of the object at the time corresponding to the frame N. Further, the bone model 620 is a bone model represented by the posture information of the frame N + 1, and represents the posture of the object at the time corresponding to the frame N + 1. The bone model 610 is a bone model represented by the posture information of the interpolated frame, and represents the posture of the object at the time corresponding to the interpolated frame.

The attitude interpolation unit 204 calculates the position of the corresponding node 603 in the interpolation frame from the position of the node 601 in the bone model 600 and the position of the corresponding node 602 in the bone model 620 by linear interpolation. In the present embodiment, since the central specific time between the two reference frames is the time of the interpolation frame, the average value of the coordinates of the node 601 and the coordinates of the node 602 is calculated as the position of the node 603 in the interpolation frame. .. In this way, the position of each node in the interpolation frame is calculated, and by connecting the calculated nodes, the posture information representing the bone model 610 of the interpolation frame is generated.

In S503, the motion vector calculation unit 205 and the model interpolation unit 206 generate an interpolation three-dimensional model using the interpolation attitude information generated in S502. The details of the processing in S503 will be described with reference to FIG. 7. In S701, the motion vector calculation unit 205 calculates a motion vector between the bone model represented by the reference posture information and the bone model represented by the interpolated posture information. The reference attitude information used here is preferably reference attitude information at a time close to the time of the interpolation frame in order to improve the interpolation accuracy. For example, when the specific time in the center between the two reference frames is used as the interpolation frame, the reference posture information of any of the reference frames before and after the interpolation frame is used.

In S702, the model interpolation unit 206 divides the bone model of the interpolation frame into regions according to the magnitude of the motion vector. FIG. 8A shows a bone model 610 in the interpolation frame shown in FIG. FIG. 8B shows an enlarged portion 800 of the bone model 610. As shown in FIG. 8B, the movement of the region 811 between the bone model 600 in the reference frame and the bone model 610 in the interpolation frame is represented by the motion vector 801. Similarly, the motion of the region 812 is represented by the motion vector 802, and the motion of the region 811 is represented by the motion vector 801. The motion vector is a vector showing the motion direction and the motion amount per unit time, and is represented by coordinates (vx, vy, vz), for example. In the present embodiment, the bone model is divided into a plurality of regions according to the size of the motion vector, but the present invention is not limited to this, and the bone model is divided into a plurality of regions according to other criteria, and then each region is divided. The motion vector of may be calculated.

In S703, the model interpolation unit 206 generates an interpolation three-dimensional model by changing the position of each voxel constituting the reference three-dimensional model according to the motion vector corresponding to the region to which the voxel belongs. For example, as shown in FIG. 8C, the voxels 821 constituting the reference three-dimensional model of the frame N are moved according to the motion vector 803 corresponding to the region 813 to which the voxels 821 belong, thereby forming the interpolated three-dimensional model. 822 is obtained. Assuming that the coordinates of the voxel 821 in the reference 3D model are (x, y, z), the coordinates (x', y', z') of the corresponding voxels 822 in the interpolated 3D model are as shown in the following equation. Desired.
x'= x + vx x t
y'= y + by x t
z'= z + vz x t
Here, t is the time from the time of the reference frame to the time of the interpolation frame, which is 1/120 second in the present embodiment. In this way, the interpolated three-dimensional model is generated by calculating the position of each voxel constituting the interpolated three-dimensional model.

[Modification example]
In the above-described embodiment, a case of generating a virtual viewpoint image having a frame rate twice the frame rate of the captured image has been described. However, the frame rate of the virtual viewpoint image generated by the image generation system 100 is not limited to this, and the image generation system 100 can generate a virtual viewpoint image at an arbitrary frame rate by the same method as described above. .. In the following, a specific example of generating a virtual viewpoint image having a frame rate three times the frame rate of the captured image will be shown.

FIG. 11 shows the temporal relationship between the captured image, the reference three-dimensional model, the reference attitude information, the interpolated three-dimensional model, and the interpolated attitude information. Frames N, frames N + 1, and frames N + 2 of the captured image are continuous frames, and the interval between the frames is 1/60 second. Then, in order to generate a virtual viewpoint image having a frame rate three times the frame rate of the captured image, two interpolation frames are inserted between two consecutive reference frames, and the interpolation posture information and the interpolation posture information corresponding to each interpolation frame are inserted. An interpolated three-dimensional model is generated. In this modification, since the time interval between a plurality of frames including the interpolation frame is made equal, the time interval between the frames is 1/180 second.

FIG. 12 shows a bone model 600 represented by the posture information of the frame N, a bone model 620 represented by the posture information of the frame N + 1, and a bone model 1210 represented by the posture information of the interpolated frame. This interpolated frame corresponds to the time 1/180 second after the time corresponding to the frame N. The bone model 1210 is generated by interpolation processing using the bone model 600 and the bone model 620. Specifically, the posture interpolation unit 204 calculates the position of the corresponding node 1203 in the interpolation frame from the position of the node 601 in the bone model 600 and the position of the corresponding node 602 in the bone model 620 by linear interpolation. The coordinates (x, y, z) of the node 1203 are obtained by the following equation.
x = x1 + (x2-x1) x t1 / T
y = y1 + (y2-y1) × t1 / T
z = z1 + (z2-z1) × t1 / T
Here, (x1, y1, z1) are the coordinates of the node 601 in the frame N, and (x2, y2, z2) are the coordinates of the node 602 in the frame N + 1. T is the time interval between the frame N and the frame N + 1, and t1 is the time interval between the frame N and the interpolated frame continuous with the frame N.

In this way, the position of each node in the interpolated frame is calculated, and by connecting the calculated nodes, the posture information representing the bone model 1210 of the interpolated frame is generated. A bone model at a time corresponding to another interpolation frame inserted between the frame N and the frame N + 1 (2/180 seconds after the time of the frame N) is also generated by the same method. Then, based on the interpolation posture information representing the bone model of the generated interpolation frame, an interpolation three-dimensional model is generated in the same manner as in the above-described embodiment. This makes it possible to generate a virtual viewpoint image of 180 fps.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a recording medium readable by a computer.

100 Image generation system 112 Camera 122 Image generation device

Claims (14)

First generation that generates shape data representing the three-dimensional shape of the object at the predetermined shooting time based on a plurality of images obtained by shooting the object from different directions by a plurality of shooting devices at a predetermined shooting time. Means and
An acquisition means for acquiring first posture information representing the posture of the object at the predetermined shooting time and second posture information representing the posture of the object at a specific time different from the predetermined shooting time.
Shape data representing the three-dimensional shape of the object at the specific time based on the first posture information and the second posture information acquired by the acquisition means and the shape data generated by the first generation means. An information processing apparatus comprising: a second generation means for generating the data. The first generation means comprises a plurality of moving images included in the plurality of moving images based on a plurality of moving images obtained by shooting the object from different directions by the plurality of photographing devices in a predetermined shooting period. Generates shape data representing the three-dimensional shape of the object at each of multiple times corresponding to the frame.
The predetermined shooting time is a time corresponding to a frame included in the plurality of frames.
The first aspect of the present invention, wherein the specific time is a time included in the predetermined shooting period and not included in the plurality of times corresponding to the plurality of frames. Information processing device. The moving image is a virtual viewpoint image according to a virtual viewpoint position and line-of-sight direction by rendering processing using the shape data generated by the first generation means and the shape data generated by the second generation means. The information processing apparatus according to claim 2, further comprising an image generation means for generating a virtual viewpoint image having a frame rate higher than that of the above. Any of claims 1 to 3, wherein the acquisition means acquires the first posture information based on a plurality of images obtained by photographing with the plurality of photographing devices at the predetermined photographing time. The information processing apparatus according to item 1. The acquisition means
Acquire third posture information representing the posture of the object at the other shooting time based on a plurality of images obtained by shooting with the plurality of shooting devices at another shooting time different from the predetermined shooting time. death,
The information processing apparatus according to claim 4, wherein the second posture information is acquired by interpolation processing using the first posture information and the third posture information. The information processing apparatus according to any one of claims 1 to 5, wherein the first posture information and the second posture information are information expressing a model of the skeleton of the object. The second generation means modifies the three-dimensional shape represented by the shape data generated by the first generation means based on the difference between the posture represented by the first posture information and the posture represented by the second posture information. The information processing apparatus according to any one of claims 1 to 6, wherein the shape data representing the three-dimensional shape of the object at the specific time is generated. The information processing apparatus according to any one of claims 1 to 7, wherein the shape data is data expressing the three-dimensional shape of the object by voxels. The information processing apparatus according to any one of claims 1 to 7, wherein the shape data is data representing a three-dimensional shape of the object by polygons. The information processing apparatus according to any one of claims 1 to 9, wherein the first generation means generates the shape data by using the visual volume crossing method. First generation that generates shape data representing the three-dimensional shape of the object at the predetermined shooting time based on a plurality of images obtained by shooting the object from different directions by a plurality of shooting devices at a predetermined shooting time. Process and
An acquisition step of acquiring first posture information representing the posture of the object at the predetermined shooting time and second posture information representing the posture of the object at a specific time different from the predetermined shooting time.
Shape data representing the three-dimensional shape of the object at the specific time based on the first posture information and the second posture information acquired in the acquisition step and the shape data generated in the first generation step. A second generation step, and an information processing method comprising. In the first generation step, a plurality of moving images included in the plurality of moving images are formed based on a plurality of moving images obtained by shooting the object from different directions by the plurality of photographing devices in a predetermined shooting period. Shape data representing the three-dimensional shape of the object at each of the plurality of times corresponding to the frame of is generated.
The predetermined shooting time is a time corresponding to a frame included in the plurality of frames.
The information processing method according to claim 11, wherein the specific time is a time included in the predetermined shooting period and not included in the plurality of times corresponding to the plurality of frames. The moving image is a virtual viewpoint image according to a virtual viewpoint position and line-of-sight direction by rendering processing using the shape data generated in the first generation step and the shape data generated in the second generation step. The information processing method according to claim 12, further comprising an image generation step of generating a virtual viewpoint image having a frame rate higher than that of the above. A program for making a computer function as each means of the information processing apparatus according to any one of claims 1 to 10.

Images