KR102806920B1 - Creating method of 3d volumetric video and recording medium thereof - Google Patents

Abstract

According to one embodiment of the present invention, a method for generating a 3D volumetric video includes the steps of: extracting 3D skeleton information for an object using a 3D pose estimation method; generating a 3D video model by applying the 3D skeleton information to the object using a rigging method; and generating an animated 3D volumetric video using the 3D video model, wherein the 3D skeleton information is extracted using a 3D point cloud for the object captured using a camera system, and the rigging method includes the steps of estimating a temporal motion between the extracted 3D skeleton information to generate a motion vector, and analyzing the motion vector to generate the 3D video model.

Description

{CREATING METHOD OF 3D VOLUMETRIC VIDEO AND RECORDING MEDIUM THEREOF}

The present invention relates to a method for generating a 3D volumetric video and a recording medium thereof, and more particularly, to a method for generating a 3D volumetric video for generating a 360-degree 3D stereoscopic image of a person using a plurality of cameras and a recording medium thereof.

Recently, the next-generation mixed reality (MR) era is coming. MR technology can provide next-generation media services that maximize human imagination. According to an online market research agency, the mixed reality market size is expected to grow rapidly. Mixed reality (MR) technology can strengthen interaction elements with reality by further expanding augmented reality (AR) technology and overcoming the limitations of virtual reality (VR) technology. In addition, mixed reality technology can be utilized in various fields such as education, entertainment, business consulting, architecture, civil engineering, logistics, energy and environmental management, medicine, and the military.

Mixed reality can be defined as a way to integrate the advantages of augmented reality and virtual reality and further enhance interaction with users. For this, the most essential element is the technology to create dynamic 3D models that have a realistic appearance and allow 360-degree omnidirectional observation of people.

In other words, existing real-world-based AR, VR, MR, and hologram 3D content services have the limitation that they can only be provided at a given point in time. Therefore, in an MR environment that requires interaction and a 360-degree multi-view experience, a system and production technology that can fundamentally provide real-world data as 3D data from all directions are required.

In particular, when processing dynamic 3D models to enable full-scale observation of a person's real-world form, it is important to extract and process accurate skeletal information.

The technical problem to be achieved by the present invention is to provide a method for generating a 3D volumetric video capable of generating a high-precision 3D moving image model and a 3D volumetric video, and a recording medium thereof.

According to one embodiment of the present invention, a method for generating a 3D volumetric video includes the steps of: extracting 3D skeleton information for an object using a 3D pose estimation method; generating a 3D video model by applying the 3D skeleton information to the object using a rigging method; and generating an animated 3D volumetric video using the 3D video model, wherein the 3D skeleton information is extracted using a 3D point cloud for the object captured using a camera system, and the rigging method includes the steps of estimating a temporal motion between the extracted 3D skeleton information to generate a motion vector, and analyzing the motion vector to generate the 3D video model.

The above 3D volumetric video may include multiple motion templates.

The above 3D pose estimation method includes the steps of: generating four virtual projection images of a 3D point cloud for the object; extracting 2D skeleton information of the four virtual projection images using a pose extraction program; connecting 2D joint coordinates of the four 2D skeleton information with virtual projection lines to form a 3D intersection space; and integrating 2D joint coordinates of virtual projection lines passing through the 3D intersection space to generate corrected 3D joint coordinates to extract the 3D skeleton information, wherein in the step of generating the corrected 3D joint coordinates, 2D joint coordinates of error virtual projection lines that deviate from the 3D intersection space may not be included.

The step of generating the above virtual projection image may include the step of determining a front side of the 3D point cloud, the step of setting an axis-aligned bounding box (AABB) surrounding the 3D point cloud based on the front side of the 3D point cloud, and the step of projecting the 3D point cloud onto four virtual projection planes of the axis-aligned bounding box.

The method may further include a step of correcting the extracted 2D skeleton information, wherein the step of correcting the 2D skeleton information may include a step of identifying an error joint coordinate located outside the 3D point cloud, a step of clustering a set of point clouds adjacent to the error joint coordinate, and a step of correcting the error joint coordinate based on the center of each cluster of the set of clustered point clouds to position the joint coordinate inside the 3D point cloud.

In addition, the present invention relates to a computer-readable recording medium having recorded thereon a program for performing the method for generating a 3D volumetric video as described above.

A method for generating a 3D volumetric video and a recording medium thereof according to one embodiment of the present invention can generate a high-precision 3D moving image model by extracting accurate 3D skeletal information for an object using a 3D pose estimation method and estimating temporal movement between the extracted 3D skeletal information using a rigging method.

FIG. 1 is a flowchart of a method for generating 3D volumetric video according to one embodiment of the present invention.
FIG. 2 is a flowchart of a 3D pose estimation method of a method for generating a 3D volumetric video according to one embodiment of the present invention.
FIG. 3 is a drawing illustrating a step of generating a virtual projection image of a 3D pose estimation method of a method for generating a 3D volumetric video according to one embodiment of the present invention.
FIG. 4 is a drawing explaining a step of forming a 3D intersection space with 2D skeleton information extracted from a 3D pose estimation method of a method for generating a 3D volumetric video according to one embodiment of the present invention.
FIG. 5 is a diagram illustrating error joint coordinates located outside a 3D point cloud for an object and correction joint coordinates located inside a 3D point cloud for an object, in a method for generating a 3D volumetric video.
FIG. 6 is a drawing explaining a method for setting an error plane in a 3D pose estimation method of a method for generating a 3D volumetric video according to one embodiment of the present invention.
FIG. 7 is a diagram illustrating a point cloud set before clustering and a point cloud set after clustering in a 3D pose estimation method of a method for generating a 3D volumetric video according to one embodiment of the present invention.
FIG. 8 is a drawing illustrating a step of extracting 3D skeleton information by generating corrected 3D joint coordinates in a 3D pose estimation method of a method for generating a 3D volumetric video according to one embodiment of the present invention.
FIG. 9 is a drawing illustrating a state in which a 3D video model is created by a rigging method of a method for creating a 3D volumetric video according to one embodiment of the present invention.
FIG. 10 is a drawing showing a 3D volumetric video generated using a method for generating a 3D volumetric video according to one embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. The present invention may be implemented in various different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a flowchart of a method for generating a 3D volumetric video according to an embodiment of the present invention, FIG. 2 is a flowchart of a 3D pose estimation method of the method for generating a 3D volumetric video according to an embodiment of the present invention, FIG. 3 is a drawing explaining a step of generating a virtual projection image of a 3D pose estimation method of the method for generating a 3D volumetric video according to an embodiment of the present invention, and FIG. 4 is a drawing explaining a step of forming 2D skeleton information and a 3D intersection space in a 3D pose estimation method of the method for generating a 3D volumetric video according to an embodiment of the present invention.

As illustrated in FIGS. 1 to 4, a method for generating a 3D volumetric video according to one embodiment of the present invention first extracts 3D skeletal information for an object using a 3D pose estimation method (S10).

3D skeletal information can be extracted using a 3D point cloud of an object captured using a camera system.

This is explained in detail below with reference to the drawings.

The 3D pose estimation method generates four virtual projection images (100) for a 3D point cloud (PC) for an object (S100), as illustrated in FIGS. 2 and 3.

A 3D point cloud (PC) of an object is captured using a multi-view RGB-D camera system. A multi-view RGB-D camera system can capture color images and depth images simultaneously from multiple viewpoints.

In order to generate a virtual projection image (100), first, the front of the 3D point cloud (PC) is determined. Since the accuracy of the 2D skeleton information extracted from the virtual projection image (100) projected on the front of the 3D point cloud (PC) is the highest, the front of the 3D point cloud (PC) is determined. To this end, the spatial distribution of the 3D coordinates of the 3D point cloud (PC) is analyzed to find the front of the 3D point cloud (PC), and the 3D point cloud (PC) is rotated so that the front direction of the 3D point cloud (PC) becomes parallel to the Z-axis direction. The front direction of the 3D point cloud (PC) can be found by using principal component analysis that finds principal components of distributed data.

Then, an axis-aligned bounding box (AABB) surrounding the 3D point cloud (PC) is set based on the front of the 3D point cloud (PC). The axis-aligned bounding box (AABB) can determine four virtual projection planes (IP1, IP2, IP3, IP4).

And, by projecting the 3D point cloud (PC) onto four virtual projection planes (IP1, IP2, IP3, IP4) of an axis-aligned bounding box (AABB), four virtual projection images (110, 120, 130, 140) can be generated. That is, the virtual projection image (100) can include a first virtual projection image (110), a second virtual projection image (120), a third virtual projection image (130), and a fourth virtual projection image (140). At this time, the 3D point cloud (PC) can be projected by converting the world coordinate system into coordinates on the virtual projection planes (IP1, IP2, IP3, IP4) using a model view projection matrix.

Next, as illustrated in Fig. 4, 2D skeleton information (200) of four virtual projection images (110, 120, 130, 140) is extracted (S200) using a pose extraction program. The pose extraction program includes, but is not necessarily limited to, the OpenPose library. The OpenPose library is based on a deep learning convolutional neural network (CNN), and is a library that can extract special features of the body, hands, and face of multiple people in real time from a photo. The 2D skeleton information (200) may include first 2D skeleton information (210) extracted from a first virtual projection image (110), second 2D skeleton information (220) extracted from a second virtual projection image (120), third 2D skeleton information (230) extracted from a third virtual projection image (130), and fourth 2D skeleton information (240) extracted from a fourth virtual projection image (140).

Next, the extracted 2D skeleton information (200) is corrected (S300).

FIG. 5 is a diagram illustrating error joint coordinates (OJ) located outside a 3D point cloud (PC) for an object and correction joint coordinates (CJ) located inside a 3D point cloud (PC) for an object.

When extracting 2D skeleton information (200) using a posture extraction program, the posture extraction program including a predetermined algorithm and a deep learning model can extract 2D skeleton information (200) having an error. Due to the 2D skeleton information having an error, 3D skeleton information may also have an error. In this case, as illustrated in FIG. 5, the error joint coordinates (OJ) may be located outside the 3D point cloud (PC) for the object. Therefore, correction is performed to position the error joint coordinates (OJ) inside the 3D point cloud (PC).

To do this, first, the error joint coordinates (OJ) located outside the 3D point cloud (PC) for the object are identified.

Then, the set of point clouds (S1) adjacent to the error joint coordinates (OJ) are clustered.

FIG. 6 is a drawing explaining a method for setting an error plane in a 3D pose estimation method according to one embodiment of the present invention.

As illustrated in Fig. 6, first, an error plane (OP) is set using the direction of the skeleton (B1, B2) connected to the error joint coordinate (OJ). That is, the error plane (OP) is a plane passing through the bisector of the skeleton (B1, B2) connected to the error joint coordinate (OJ). Then, a point cloud set (S1) adjacent to the error plane (OP) by a predetermined distance is obtained, and then clustering is performed to find a point cloud set (S2) valid for the corresponding joint coordinate among the obtained point cloud sets (S1).

FIG. 7 is a diagram illustrating a point cloud set (S1) before clustering and a point cloud set (S2) after clustering in a 3D pose estimation method according to one embodiment of the present invention.

As shown in Fig. 7, after clustering, the point cloud set (S2) can be more densely packed around the error plane (OP). Clustering of the point cloud set (S1) is performed using the DBSCAN algorithm. The DBSCAN algorithm is a density-based algorithm, and has higher accuracy for data sets that follow an unspecified distribution. In addition, since the DBSCAN algorithm can also determine noise, it can remove noise from directly captured 3D data that has a lot of noise.

Then, by correcting the error joint coordinates (OJ) based on the centers of each cluster of the clustered point cloud set (S2), the corrected correction joint coordinates (CJ) are positioned inside the 3D point cloud (PC) for the object, as illustrated in Fig. 4.

At this time, the center of each cluster in the clustered point cloud set (S2) can be calculated using the circle fitting method.

The original fit method can be performed in the following manner.

First, the center coordinates of the circle you want to find for one cluster If n points in a cluster The circle with the smallest error for the fields can be expressed by the mathematical formula 1 below.

[Mathematical formula 1]

Mathematical expression 1 can be expressed as mathematical expression 2 below.

[Mathematical formula 2]

And, among the possible Ws, it makes the smallest error. can be obtained through the mathematical formula 3 below.

[Mathematical Formula 3]

By using this circle fitting method, the center of each cluster of the clustered point cloud set (S2) is calculated and the error joint coordinates (OJ) are corrected, so that the corrected joint coordinates (CJ) can be positioned inside the 3D point cloud (PC) for the object, as shown in Fig. 4.

In this way, by correcting the extracted 2D skeleton information (200) through clustering and correction of error joint coordinates (OJ), more accurate 3D skeleton information can be extracted and an accurate 3D pose can be estimated.

Next, as illustrated in FIGS. 2 and 4, each of the 2D joint coordinates (11, 12, 13, 14) of the four 2D skeleton information (210, 220, 230, 240) is connected to each other by four virtual projection lines (21, 22, 23, 24) to form a 3D intersection space (IS) (S400). That is, four virtual projection images (110, 120, 130, 140) generated on four virtual projection planes (IP1, IP2, IP3, IP4) have four 2D bone information (210, 220, 230, 240), and the four 2D bone information (210, 220, 230, 240) have four 2D joint coordinates (11, 12, 13, 14) that match each other. For convenience of explanation, Fig. 3 illustrates the 2D joint coordinates of the left knee, but is not necessarily limited thereto. By connecting four 2D joint coordinates (11, 12, 13, 14) that match each other with four virtual projection lines (21, 22, 23, 24), a 3D intersection space (IS) can be formed where the four virtual projection lines (21, 22, 23, 24) intersect each other.

FIG. 8 is a drawing illustrating a step of extracting 3D skeleton information by generating corrected 3D joint coordinates in a 3D pose estimation method according to one embodiment of the present invention.

As illustrated in Fig. 8, 2D joint coordinates (11, 12, 13, 14) connected to virtual projection lines (21, 22, 23, 24) passing through a 3D intersection space (IS) are integrated to generate corrected 3D joint coordinates (300) to extract 3D skeleton information (S500).

When generating the corrected 3D joint coordinates (300), the 2D joint coordinates of the error virtual projection line (EL) that goes out of the 3D intersection space (IS) may not be included.

When extracting 2D skeleton information (200) using a posture extraction program, errors may occur, and these errors may cause error virtual projection lines (EL) that deviate from the 3D intersection space (IS). The error virtual projection lines (EL) that deviate from the 3D intersection space (IS) can be confirmed in the front view (FV) and the side view (SV) illustrated in Fig. 8.

And, the corrected 3D joint coordinates (300) can be generated by averaging the four 2D joint coordinates (11, 12, 13, 14) located in the 3D intersection space (IS). That is, the coordinates of (x, z) are determined in the top view (TV), and the y coordinate is determined in the side view (SV). The calculated (x, y, z) coordinates must match the (x, y) coordinates in the front view (FV).

In this way, the 3D pose estimation method of the method for generating a 3D volumetric video according to one embodiment of the present invention extracts 2D skeleton information from a 3D point cloud (PC) for an object, connects and integrates 2D joint coordinates of the 2D skeleton information to generate corrected 3D joint coordinates, and extracts corrected 3D skeleton information, thereby enabling more accurate 3D pose estimation.

Next, a 3D video model (VM) is generated by applying 3D skeleton information (200) to an object (P) using a rigging method (S20). The rigging method is a method of making the object (P) an animable state.

The rigging method first estimates the temporal movement between the extracted 3D skeleton information to generate a motion vector (S21).

Next, the motion vector is analyzed to generate a 3D video model (S22).

FIG. 9 is a drawing illustrating a state in which a 3D video model is created by a rigging method of a method for creating a 3D volumetric video according to one embodiment of the present invention.

As illustrated in Fig. 9, by applying 3D skeleton information (200) to an object (P) using a rigging method, a 3D video model (VM) in which the object can move can be created. Accordingly, animation can be performed thereafter using the 3D video model (VM).

Next, an animated 3D volumetric video (VV) is generated using a 3D video model (S30). A 3D volumetric video refers to a 360-degree 3D stereoscopic image generated by capturing the movement of an object using a multi-view RGB-D camera system that implements high-resolution image quality and applying a rigging method to the movement of the object.

FIG. 10 is a drawing showing a 3D volumetric video generated using a method for generating a 3D volumetric video according to one embodiment of the present invention.

As illustrated in Fig. 10, a 3D volumetric video (VV) can be created by animating 3D video models (VMs) by connecting them. In Fig. 10, a walking motion is animated using a 3D video model, but it is not limited thereto, and other motions such as dancing, clapping, fanning, scratching, and saluting are also possible.

At this time, 3D volumetric video (VV) enables users to easily create 3D volumetric videos by pre-generating multiple motion templates, such as dancing motions, clapping motions, fanning motions, scratching motions, and saluting motions.

In this way, the method for generating a 3D volumetric video and the recording medium thereof according to one embodiment of the present invention can generate a high-precision 3D moving image model by extracting accurate 3D skeletal information for an object using a 3D pose estimation method and estimating temporal movement between the extracted 3D skeletal information using a rigging method.

An embodiment of the present invention may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, that are executed by a computer. The computer-readable medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include all computer storage media. The computer storage media may include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited thereto, and various modifications may be made within the scope of the claims, the detailed description of the invention, and the attached drawings, which also fall within the scope of the present invention.

VM: 3D Video Model VV: 3D Volumetric Video
100: Virtual projection image 200: 2D skeleton information
11, 12, 13, 14: 2D joint coordinates 21, 22, 23, 24: Virtual projection lines
PC: 3D Point Cloud AABB: Axis-aligned bounding box
IPI: Virtual Projection Image IS: 3D Intersecting Space
EL: Error virtual projection line 300: Corrected 3D joint coordinates

Claims (6)

A step of extracting 3D skeleton information for an object using a 3D pose estimation method;
A step of applying the 3D skeleton information to the object using a rigging method to create a 3D video model; and
A step of generating an animated 3D volumetric video using the above 3D video model.
Including,
The above 3D skeleton information is extracted using a 3D point cloud for the object captured using a camera system,
The above rigging method
A step of generating a motion vector by estimating the temporal movement between the extracted 3D skeleton information, and
A step of generating the 3D video model by analyzing the above motion vector.
A method for generating a 3D volumetric video comprising:
The above 3D pose estimation method comprises the steps of generating four virtual projection images of a 3D point cloud for the object;
A step of extracting 2D skeleton information of the four virtual projection images using a posture extraction program;
A step of forming a 3D intersection space by connecting the 2D joint coordinates of the above four 2D skeleton information with virtual projection lines; and
A step of extracting the 3D skeleton information by integrating the 2D joint coordinates of the virtual projection line passing through the 3D intersection space to generate the corrected 3D joint coordinates;
In the step of generating the above-mentioned corrected 3D joint coordinates,
2D joint coordinates of error virtual projection lines that leave the above 3D intersection space are not included,
Further comprising a step of correcting the extracted 2D skeleton information,
The step of correcting the above 2D skeleton information is
A step of checking the error joint coordinates located outside the above 3D point cloud,
A step of clustering a set of point clouds (S1) adjacent to the above error joint coordinates, and
A step of positioning the joint coordinates within the 3D point cloud by correcting the error joint coordinates based on the center of each cluster of the clustered point cloud set (S2),
The step of positioning the joint coordinates inside the above 3D point cloud is
A step of setting an error plane using the direction of the skeleton connected to the above error joint coordinates;
A step of performing clustering to find a valid point cloud set for the corresponding joint coordinates among the point cloud sets obtained after obtaining the point cloud set (S1) adjacent to the error plane by a predetermined distance; and
A method for generating a 3D volumetric video, characterized by comprising a step of positioning the corrected joint coordinates inside the 3D point cloud for the object by correcting the error joint coordinates based on the centers of each cluster of the clustered point cloud set (S2). In paragraph 1,
The above 3D volumetric video is a method for generating a 3D volumetric video including a plurality of motion templates. delete In paragraph 1,
The step of generating the above virtual projection image is
A step of determining the front side of the above 3D point cloud,
A step of setting an axis-aligned bounding box (AABB) surrounding the 3D point cloud based on the front side of the 3D point cloud,
A step of projecting the 3D point cloud onto four virtual projection planes of the above axis-aligned bounding box.
A method for generating a 3D volumetric video, comprising: delete A computer-readable recording medium having recorded thereon a program for performing the method of generating a 3D volumetric video according to claim 1, claim 2, or claim 4.