JP6667785B1 - A program for learning by associating a three-dimensional model with a depth image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program for generating a three-dimensional model of a measurement target user from a point cloud captured by a depth camera. A program for operating a computer mounted on an apparatus for learning by associating a three-dimensional model of teacher data with a point cloud, one three-dimensional model for each three-dimensional model of teacher data. A point cloud creating means for creating a point cloud photographed on software from the above predetermined viewpoint and a three-dimensional model of a plurality of teacher data groups outputs a dimension-compressed component variable of the number m of dimensions and performs statistical learning. A statistical learning engine for constructing a model, and a first correlation learning model for constructing a first correlation learning model of a point cloud of the three-dimensional model and a component variable having a dimension number m for a three-dimensional model of a plurality of teacher data groups. The feature is that a computer functions as a correlation learning engine. [Selection diagram] Fig. 1

Description

  The present invention relates to a technology of a machine learning engine based on a three-dimensional model.

  In recent years, there is a three-dimensional scanner technology capable of detecting human body shape data (for example, see Non-Patent Documents 1 and 2). According to this technique, three-dimensional point cloud data is measured by non-contact optical triangulation with a human body. These point cloud data are ultra-high density of about 1 million points, and realize extremely small errors in human body measurement applications. Such human body shape data is also effective as health management data other than the body shape.

  Conventionally, there is a technique for deforming a muscle model superimposed on a skeletal model in accordance with a measurement result of a subject (for example, see Patent Document 1). According to this technique, a human body model corresponding to the subject himself is created based on the result of the physical measurement of the subject using the body composition meter and the three-dimensional measuring device. The human body model is an anatomical model in which a skeleton, a muscle, and a fat are set, and these are deformed according to a measurement result of a subject. By switching and displaying the model of each of these skeleton, muscle and fat, the subject can visually recognize his or her internal state and the like.

  There is also a technique in which a three-dimensional model in which an object is represented by feature parameters is stored in advance, and an image of a characteristic region detected from a captured image is adapted to the three-dimensional model (for example, see Patent Document 2). According to this technique, a value of a feature parameter of a three-dimensional model representing an object captured in an image of a feature region is calculated. Then, by outputting the value of the characteristic parameter and the image of the region other than the characteristic region, the data amount of the entire three-dimensional image is reduced.

For such a three-dimensional model, a depth camera capable of capturing a 2.5-dimensional image has become popular. The depth camera is a camera having a built-in depth sensor for acquiring depth information. By acquiring depth information (Depth) in addition to a two-dimensional planar image (RGB) acquired by a normal camera, three-dimensional three-dimensional information can be acquired. This information is referred to as a “ depth image ” in which grayscale gradation is set according to distance information (depth) for each pixel. In particular, because only three-dimensional information is viewed from a viewpoint not be obtained, Ru conceptually also referred to as "2.5-dimensional image".

JP 2017-176803 A JP 2009-268088 A

"3D BODY SCANNER SCUVEG4", Space Vision Co., Ltd., [online], [Search December 31, 2018], Internet <URL: http://www.spacevision.tokyo/> “3D Body Station”, 3D Body Lab, Inc., [online], [searched on December 31, 2018], Internet <URL: https://www.3dbodylab.co.jp/3dbodystation/> "Auto Encoder", [online], [Search December 31, 2019], Internet <https://deepage.net/deep_learning/2016/10/09/deeplearning_autoencoder.html> "MAYA Let's adjust the depth of field", [online], [searched on December 31, 2019], Internet <http://thankstotoday.com/modeling-depth1/>

  In the case of Non-Patent Documents 1 and 2 described above, it is necessary to use a three-dimensional scanner that is large in scale and cost in order to generate a three-dimensional model of a human body. In order to increase the accuracy of the three-dimensional model, the number of vertices is set to, for example, 15,000 or more, and each vertex is expressed in three dimensions (x, y, z). Enormous amount of data.

On the other hand, the inventor of the present application can easily associate a three-dimensional model close to the user's body shape only from a depth camera that photographs from a predetermined viewpoint without preparing a three-dimensional scanner for optical triangulation, I thought. For example, they thought that a depth camera mounted on a smartphone or other healthcare device could photograph their own body shape and associate a three-dimensional model from the depth image .

On the other hand, when the depth image has a height h = 960 pixels and a width w = 540 pixels , the depth image has a huge data amount such as w * h = 518,400 dimensions. It is extremely difficult to process such a huge amount of data with an embedded processor.

In addition, the inventor of the present application considered whether it is possible to easily share (transmit and receive) three-dimensional model and depth image data with a small capacity. In particular, as a service provider, even if the body data of the user is acquired, if it is possible to instantaneously transmit and receive this enormous amount of data, various services unique to the user can be provided.

Furthermore, the inventor of the present application has considered that a three-dimensional model and a depth image expressing a user's own body shape should be kept confidential as personal information for the user. In other words, even if such an enormous amount of data can be compressed, it must be implemented so that a third party cannot immediately decode the data.

Furthermore, the invention of the present application has considered that the user's own body shape will vary greatly depending not only on the appearance but also on the user's body composition value. For example, it was considered that even if the users have the same height, abdominal girth, and chest girth, their appearance may differ depending on their body composition values. Then, it was thought that the body composition value acquired by the health care measuring device could be associated with the three-dimensional model or the depth image as the appearance.

Thus, according to the present invention, it is an object to provide a program capable of generating a three-dimensional model of a measurement target user from a depth image captured by the depth camera.

According to the present invention, there is provided a program for causing a computer mounted on an apparatus for learning by associating a three-dimensional model of a human body of a teacher data and a depth image with each other,
Depth image creating means for creating, for each three-dimensional model of the teacher data, a depth image obtained by photographing the three-dimensional model from one or more predetermined viewpoints on software;
A statistic learning engine that outputs a dimensionally compressed m number of component variables from a plurality of three-dimensional models of the teacher data group and constructs a statistic learning model;
A first correlation learning engine for constructing a first correlation learning model of a depth image of the three-dimensional model and a component variable having the number of dimensions m for a plurality of three-dimensional models of the teacher data group;
For a plurality of three-dimensional models of the teacher data group, a second correlation learning model is constructed between a composition value of dimension n including at least height and one or more body composition measurement values and a component variable of dimension m. A computer functioning as a second correlation learning engine .

According to another embodiment of the program of the present invention,
It is also preferred that the statistical learning engine causes the computer to function as being based on Principal Component Analysis or AutoEncoder.

According to another embodiment of the program of the present invention,
First correlation learning engine of state, and are not based on convolutional neural network,
The second correlation learning engine also preferably causes the computer to function as being based on least squares or multilayer perceptron .

  According to another embodiment of the program of the present invention,
  The number of plural bodies of the three-dimensional model of the teacher data group is smaller than the number of vertices of the three-dimensional model.
It is also preferable to make the computer function as described above.

According to another embodiment of the program of the present invention,
A first encoder that encodes a depth variable as target data into a component variable having a dimension number m using a first correlation learning engine;
A second encoder that encodes, using a second correlation learning engine, a composition value of one dimension n as target data into a component variable of dimension m,
It is also preferable that the computer functions as a computer.

According to another embodiment of the program of the present invention,
In order to input a depth image and a composition value associated as target data and re-learn the first correlation learning engine and the second correlation learning engine,
The first encoder encodes the depth image of the target data into a component variable having a first dimension number m,
The second encoder encodes the composition value of the target data into a component variable having a second dimension number m,
Integrating a component variable having a first dimension number m and a component variable having a second dimension number m into a component variable having one dimension number m;
The first correlation learning engine re-learns from the depth image of the target data and the integrated component variables having the number of dimensions m,
It is also preferable that the second correlation learning engine causes the computer to function so as to re-learn from the composition value of the target data and the integrated component variables having the number of dimensions m .

According to another embodiment of the program of the present invention,
The component variable of the first dimension m and the component variable of the second dimension m calculated for each dimension by assigning a smaller weight w as the number of missing composition values of the composition value of the target data increases. It is also preferable to cause the computer to function so as to integrate the weighted average of the two as a component variable having one dimension number m .

According to another embodiment of the program of the present invention,
Only k (<n) composition values are determined for the composition value of one dimension n as the target data, and even if other nk composition values are missing, the component variable of dimension m is determined. A missing value estimating means for estimating the other nk optimized component values using the k component values as constraints for the estimation.
It is also preferable to make the computer function further .

According to another embodiment of the program of the present invention,
It is also preferred that the missing value estimating means causes the computer to function so as to use a Lagrange's method of Lagrange multiplier .

According to another embodiment of the program of the present invention,
A first decoder for decoding the component variable having the number of dimensions m into a three-dimensional model using a statistical learning engine;
It is also preferable to make the computer function further .

According to another embodiment of the program of the present invention,
Grayscale image creation means for creating one or more grayscale images of the decoded three-dimensional model taken from a predetermined viewpoint on software
It is also preferable that the computer functions as a computer.

According to another embodiment of the program of the present invention,
A second decoder that decodes a component variable of dimension number m as target data into a composition value using a second correlation learning engine;
It is also preferable that the computer functions as a computer.

According to another embodiment of the program of the present invention,
A depth image is an image with grayscale gradation depending on the depth
It is also preferable to make the computer function as described above .

According to another embodiment of the program of the present invention,
A shared code output unit that outputs the encoded component variable having the number of dimensions m as a share (Share) code, and embeds the share code in QR (Quick Response, registered trademark), RFID (Radio Frequency IDentifier) or Cookie.
It is also preferable to make the computer function further .

According to the program of the present invention, it is possible to generate a three-dimensional model of a measurement target user from a depth image captured by the depth camera.

It is a functional block diagram of the learning stage in the apparatus of this invention. FIG. 4 is an explanatory diagram illustrating a flow of a process in a learning stage in the device of the present invention. FIG. 4 is an explanatory diagram illustrating a vector space of a three-dimensional model in the statistical learning engine. FIG. 4 is an explanatory diagram illustrating a statistical shape space in a statistical learning engine. This is a simple code representing principal component analysis in a statistical learning engine. 5 is a simple code representing a convolutional neural network in a first correlation learning engine. It is explanatory drawing showing the composition value space derived from the composition value. FIG. 4 is an explanatory diagram illustrating a linear transformation between a statistical shape space and a composition value space. It is a simple code representing a linear transformation between a statistical shape space and a composition value space in a second correlation learning engine. It is a functional block diagram of the encoding side of the operation stage in this invention. It is explanatory drawing showing the composition value space which estimates the missing value in this invention. It is a simple code representing missing value estimation in the present invention. It is a functional block diagram on the decoding side of the operation stage in the present invention. FIG. 3 is an explanatory diagram showing the accuracy of a three-dimensional model decoded by the present invention. It is a functional block diagram showing the re-learning of the operation stage in this invention. FIG. 2 is a configuration diagram in which the encoding side function of the present invention is incorporated in a body composition meter. 1 is a system configuration diagram showing various arrangements of a depth camera using a body composition meter and a terminal. It is an external view showing the attitude | position of a user at the time of mounting a depth camera in the grip part of a body composition meter. FIG. 19 is an explanatory diagram illustrating the accuracy of a three-dimensional model decoded from a depth image captured according to the posture of the user in FIG. 18.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a functional configuration diagram of a learning stage in the apparatus of the present invention.

According to FIG. 1, the device 1 inputs a data group in which a three-dimensional model is associated with a composition value as teacher data.
According to the embodiment of the present invention, the three-dimensional model will be described as a human body.
The three-dimensional model itself may be generated by being artificially dispersed, or may be obtained in advance by an optical scanner from an unspecified number of human bodies.
The composition value may be a body composition measurement value obtained from a person (user) based on the three-dimensional model. Here, the composition value may include a measurement value that can be obtained from the appearance of the three-dimensional model, such as height, abdomen circumference, and chest circumference. However, according to the present invention, in the case of a human body, at least the height is preferably included as a composition value. It is also preferable that the composition value include a body composition meter value that is not linked to the appearance of the three-dimensional model, such as body weight, body fat, visceral fat, and skeletal muscle percentage measured by the body composition meter.
Note that the composition of the present invention is not essentially essential.

  According to the present invention, the device 1 in the learning stage causes each learning engine to construct a learning model in advance. After the learning model is constructed, the learning model can be distributedly incorporated into the encoder side or the decoder side or each device of the system for each learning engine.

According to the present invention, the learning stage device 1 includes the depth image creating unit 10, the statistical learning engine 100, the first correlation learning engine 110, and the second correlation learning engine 120. These functional components can be realized by executing a program that causes a computer mounted on the device to function. In addition, the flow of processing of these functional components can be understood as a learning method of the device.

  FIG. 2 is an explanatory diagram illustrating a flow of a process at a learning stage in the device of the present invention.

[ Depth image creation unit 10]
The depth image creation unit 10 creates, for each three-dimensional model of the teacher data, a depth image obtained by photographing the three-dimensional model from one or more predetermined viewpoints on software.
The depth image is a depth (depth) image and has a gray scale gradation according to the depth.

The depth image creation unit 10 can also shift the position (predetermined viewpoint) of the virtual camera on software. That is, a plurality of depth images captured by a plurality of virtual cameras from different viewpoints can be created from one three-dimensional model. For example, in the case of the video production software MAYA (registered trademark), a gray scale (Z depth) image in the .exr format can be created by adjusting the position of the virtual camera. This grayscale image is one channel of depth only, where each pixel is represented by a 32-bit floating point number between 0.0 and 1.0.
Then, the created depth image is output to the first correlation learning engine 110.

[Statistical learning engine 100]
The statistical learning engine 100 inputs a three-dimensional model of a plurality of teacher data groups, outputs a component variable having the number of dimensions m that has been dimensionally compressed, and constructs a statistical learning model.
The statistical learning engine may be based on Principal Component Analysis or AutoEncoder.

  FIG. 3 is an explanatory diagram illustrating a vector space of a three-dimensional model in the statistical learning engine.

According to FIG. 3, for example, 1,000 human bodies having various body shapes are assumed as teacher data. The three-dimensional model is shape data of a human body or an object, and is represented by the same number of vertices for the same object.
According to FIG. 3A, the three-dimensional model has N = 15,000 vertices per body, and each vertex is expressed in three dimensions (x, y, z). That is, one three-dimensional model is represented by a 3N (= 45,000) -dimensional vector.
According to FIG. 3B, each point of the three-dimensional model is represented by one point in the 3N-dimensional space.

  According to a general machine learning engine, an enormous number of teacher data is required, whereas according to the present invention, the number of plural teacher data groups is smaller than the number of vertices of the three-dimensional model. You may. That is, the number of human bodies 1,000 in the teacher data is smaller than the number of vector dimensions 45,000 of the three-dimensional model. According to the present invention, it is not necessary to prepare the plurality of teacher data in the number of vector dimensions of the three-dimensional model or more, and even in such a case, sufficient accuracy can be maintained.

FIG. 4 is an explanatory diagram illustrating a statistical shape space in the statistical learning engine.
FIG. 5 is a simple code showing the principal component analysis in the statistical learning engine.

The statistical learning engine 100 may specifically be based on Principal Component Analysis.
By “principal component analysis”, a small number (for example, 30) of principal components (component variables) that are uncorrelated and best represent the overall variation are derived from 1000 points in a correlated 3N-dimensional space. The variance of the first principal component is maximized, and the following principal components are selected so as to maximize the variance under the constraint that there is no correlation with the principal components determined so far. By maximizing the variance of the principal components, the principal components have as much explanatory power as possible for changes in the observed values. The principal axis giving the principal component is an orthogonal basis of a group of 1000 points in the 3N-order space. The orthogonality of the principal axes is derived from the fact that the principal axes are eigenvectors of the covariance matrix and the covariance matrix is a real symmetric matrix.

The statistical learning engine 100 constructs a statistical learning model for projecting a 3N-dimensional space onto a statistical shape space (for example, 30 dimensions) having component variables based on principal component analysis as dimensions.
According to the present invention, each three-dimensional model in a 3N (= 45,000) dimensional space is projected onto, for example, a 30-dimensional (component variable) space. The transformation giving the principal component is represented by singular value decomposition of a matrix composed of a set of observation values, and the singular value decomposition of a rectangular matrix X composed of a group of 1000 points in a 3N-dimensional space is represented by the following equation.
X = U * Σ * V ^T
X: Matrix consisting of 1000 points in 3N-dimensional space (1000 rows x 3N columns)
U: n (1000) × n (1000) square matrix (orthogonal matrix of n-dimensional unit vector)
Σ: n (1000) × p (3N) rectangular diagonal matrix (diagonal components are singular values of X)
V: square matrix of p (3N) × p (3N) (orthogonal matrix of p-dimensional unit vector)
Here, the matrix consisting of the first 30 columns of V is referred to as V. Then, the linear transformation using the matrix V gives the main component of X.
V: Matrix representing conversion to 3N-dimensional space-> statistical shape (30-dimensional) space
V ^-1 : Statistical shape (30-dimensional) space-> matrix representing conversion to 3N-dimensional space Note that the superscript -1 of the matrix is not a symbol indicating an inverse matrix, but a conversion of a vector defined by the matrix. , Is used as an abstract symbol meaning the inverse transformation. Here, V ^-1 is equal to the transpose V ^T of V.

As is clear from FIG. 4, it is possible to associate each 3D model between the 3N-dimensional space and the statistical shape space by the linear transformation using the matrix V or V ⁻¹ .
s = x * V
x = s * V ^-1
s: vector of statistical shape space
x: vector in 3N-dimensional space
V: Statistical learning model

The statistical learning engine 100 may be based on an auto encoder (AutoEncoder).
The auto-encoder is a type of neural network and realizes a feature expression with a reduced amount of information (for example, see Non-Patent Document 3). More specifically, the dimensions of the hidden layer are compressed rather than the number of dimensions of the input data. The input data is compressed through a neural network and returned to its original size on output. At this time, the neural network extracts an abstract concept (feature amount) of the input data.
The auto-encoder also derives a 30-dimensional component variable that is uncorrelated with each other and that best represents the overall variation from 1,000 points in the correlated 3N-dimensional space, similar to the principal component analysis.

[First Correlation Learning Engine 110]
The first correlation learning engine 110 constructs a first correlation learning model between a depth image of the three-dimensional model and a component variable having the number of dimensions m for a plurality of three-dimensional models of the teacher data group.

  The first correlation learning engine 110 may be based on a convolutional neural network. This is a type of forward-propagation type deep learning, and can be learned as a regression analysis in order to predict a target variable from an explanatory variable.

  FIG. 6 is a simple code representing a convolutional neural network in the first correlation learning engine.

The convolutional neural network learns in one direction as follows.
Explanatory variables: Depth image (grayscale image) of 3D model
Object variable: component variable of dimension m output from the statistical learning engine 100 By this, in the operation stage, by inputting a depth image as an explanatory variable, the component variable of dimension m as an objective variable is output. Can be.

[Second correlation learning engine 120]
The second correlation learning engine 120 is configured to perform a second correlation between a composition value of the number of dimensions n including at least one or more measuring points and a component variable of the number of dimensions m with respect to a plurality of three-dimensional models of the teacher data group. Build a learning model.

  FIG. 7 is an explanatory diagram illustrating a composition value space derived from the composition values.

The composition value of the dimension number n is associated with each human body of the three-dimensional model in the teacher data.
The composition value is linked to the three-dimensional model, but may include one or more measurement values (for example, height, chest measurement, abdominal measurement, and the like) derivable from the three-dimensional model itself. However, it is preferable that the number of measuring points based on the geometry is one or more. Height alone may be used, height + abdominal circumference may be used, or height + abdominal circumference + chest circumference may be used.
Here, a composition value space in which a plurality of composition values are used as element values can be derived. For example, when ten composition values are given, the composition value space has ten dimensions.

  The second correlation learning engine 120 may be based on least squares or multi-layer perceptron.

FIG. 8 is an explanatory diagram illustrating a linear transformation between the statistical shape space and the composition value space.
FIG. 9 is a simple code showing a linear transformation between the statistical shape space and the composition value space in the second correlation learning engine.

<Least squares method>
The second correlation learning engine 120 may be based on the least squares method.
The "least squares method" is to determine the most probable linear model that minimizes the residual sum of squares when approximating a linear model from multiple multidimensional vectors (data sets). Say.

As is clear from FIG. 8, it is possible to make correspondence between the statistical shape space and the composition value space for each one of the three-dimensional models by the linear transformation using the matrix A or A ⁻¹ .
s = d * A
d = s * A ^-1
A = (D ^T * D) ⁻¹ * D ^T * S (Deriving A that minimizes || D * AS− |)
s: vector of statistical shape space
d: Composition value space vector
S: vector set of statistical shape space
D: Set of vectors in composition value space
A: Correlation learning model

<Multilayer Perceptron>
The second correlation learning engine 120 may be based on a multi-layer perceptron.
"Multilayer perceptron" is a forward-propagation type neural network that uses supervised learning called error backpropagation, which is a generalization of the least squares algorithm in the linear perceptron. is there.

  FIG. 10 is a functional block diagram of the encoding side at the operation stage in the present invention.

According to FIG. 10, the device 1 includes a first encoder 111 using a first correlation learning engine 110, a second encoder 121 using a second correlation learning engine 120, a missing value estimating unit 122, And a code output unit 13. These functional components can be realized by executing a program that causes a computer mounted on the device to function. In addition, the flow of processing of these functional components can be understood as an encoding method of the device.
The first encoder 111, the second encoder 121, and the missing value estimation unit 122 may include only one of them according to the use of the device 1.

[First encoder 111]
The first encoder 111 uses the first correlation learning engine 110 to encode a depth image as target data into a component variable having the number of dimensions m. The encoded component variable having the number of dimensions m has confidentiality that the depth image cannot be recognized. Therefore, it is suitable when the depth image as personal information is confidential information.
The encoded component variable having the number of dimensions m is output to the share code output unit 13.

The first encoder 111 may apply the <Otsu's discriminant analysis function> to the input depth image of the target data.

<Otsu's discriminant analysis function>
S1: Object extraction stage The following three steps are required for object extraction.
(S11) A target depth image including both a human body and a background is input.
(S12) Generates a binary image separated into two classes (object part and non-object part) by binarization processing. For example, when the “Otsu's discriminant analysis method” is applied to the binarization processing, a threshold value at which the value of the separation metrics is maximized is obtained, and the binarization is automatically performed. it can. The degree of separation is determined by the ratio between the between-class variance and the within-class variance.
(S13) The target depth image input in S11 is subjected to mask processing with the binary image calculated in S12 to extract a depth image of only a human body.

S2: Normalization stage The following five steps are required for normalization.
(S21) Three-dimensional coordinates P = (X, Y, Z) are calculated from pixel coordinates p = (x, y, z) on the image plane of the depth image extracted in the object extraction step.
For each point P = (X, Y, Z), X = x * z / f, Y = y * z / f, Z = (Max-Min) * z + Min
<Known camera specifications>
Focal length (physical) F (mm)
Focal length (pixel) f = (fx, fy) = (F * sx, F * sy) (pixel)
Pixel size s = (sx, sy) (pixel / mm)
Depth measurement distance Min, Max (mm)
(S22) A bounding box is specified from the depth image of the object, and its center of gravity is determined.
(S23) The depth image is translated to a predetermined center of gravity based on the center of gravity determined in S22.
(S24) The pixel coordinates on the image plane are calculated for the depth image translated in S23.
For each point p = (x, y, z), x = X * f / Z, y = Y * f / Z, z = (Z-Min) / (Max-Min)
(S25) Since the pixel coordinates calculated in S24 are not necessarily integer values, final pixel coordinates are calculated using interpolation.
(S26) The depth image complemented in S5 is input to the first correlation learning engine 110 as target data.

[Second encoder 121]
The second encoder 121 uses the second correlation learning engine 120 to encode the composition value of one dimension number n as the target data into a component variable of the dimension number m. Also in this case, the encoded component variable having the number of dimensions m has confidentiality in which the composition value cannot be recognized.
The encoded component variable having the number of dimensions m is output to the share code output unit 13.

[Missing value estimation unit 122]
As another optional embodiment, the missing value estimating unit 122 estimates a missing value for the composition value of the input target data.

FIG. 11 is an explanatory diagram illustrating a composition value space for estimating a missing value according to the present invention.
FIG. 12 is a simple code showing missing value estimation in the present invention.

The composition value space represents a composition value having a fixed number of dimensions n (= 10) associated with the three-dimensional model of the teacher data group.
Here, the missing value estimating unit 122 determines only k (<n) composition values for the composition value of one dimension n as the target data, and missing the other nk composition values. May be. That is, according to the present invention, even if a second correlation learning model is constructed from, for example, a ten-dimensional composition value space using the teacher data group, for example, by inputting only k = 3 composition values, the number of dimensions m Can be estimated.

According to FIG. 11, the isosurface of the hyperellipsoid is represented on the composition value (10-dimensional) space. The hyperellipsoid refers to a figure obtained by expanding an ellipse into n (= 10) dimensions. The isosurface refers to a contour map drawn on the dimension (= 10). Here, it is assumed that the variance-covariance matrix C = D ^T * D is known.
Quadratic form representing hyperellipsoid (x: column vector)
f (x) = x ^T * C ^-1 * x
C ^-1 : symmetric matrix
* Actually, the variance-covariance matrix is C = D ^T * D / the number of teacher data groups.
* ^-1 indicates the inverse matrix

  According to the present invention, a component variable having a dimension number m including other optimized nk composition values is calculated using k composition values as constraints. For this, Lagrange's method of Lagrange multiplier is used.

  Lagrange's method of undetermined multipliers is an analysis method that optimizes under constraints, and under the constraint that the values of some functions are fixed for some variables, Consider the problem of finding the extremum of a certain function. For each binding condition, a constant (undetermined multiplier, Lagrange multiplier) is prepared, and the linear combination using these coefficients as a new function (the undetermined multiplier is also a new variable) is considered. Solve as a value problem.

Constraints _{g j (x 1, ···,} x n) = 0 (j = 1, ···, k) under the function _{f (x 1, ···, x} n) is an extreme value About the point
F (x ₁ , ..., x _n , λ ₁ , ..., λ _k )
= F (x ₁ ,..., X _n ) + Σλ _j g _j (x ₁ ,..., X _n )
Satisfies the following equation.
dF / dx _i = 0 (i = 1,..., n)
dF / dλ _j = 0 (j = 1,..., k)

In the case of missing value estimation of a composition value, when k composition values are given, the constraint condition g _j (x) = 0 (j = 1,..., K) is This is a linear equation representing a plane, and is represented by the following equation.
A linear equation representing an affine hyperplane g _j (x) = n _j ^T * (x−p _j ) = 0
n: Normal vector of hyperplane
p: a point on the hyperplane In particular, since each hyperplane is orthogonal to the base (the direction of n coincides with the base direction), it becomes as follows.
g _j (x) = x _i −y _j = 0
y _j : j-th composition value
x _i : corresponding element of x To obtain the minimum value of the function f (x) under the constraint condition, to obtain the body shape closest to the average under the given composition value.

In the case of a 10-dimensional space, it is specifically expressed as follows.
x: 10-dimensional column vector y: 10-dimensional column vector containing k composition values (values of composition values other than k are arbitrary)
λ: k-dimensional column vector having a Lagrange multiplier as an element O: matrix of k rows and 10 columns each row is a one-hot row vector corresponding to a given composition value C: variance-covariance matrix f (x) = 1/2 * x ^T * C ^-1 * x
g (x) = O * (y−x)
F (x) = f (x) + λ ^T * g (x)
^{dF / dx = C -1 * x} -O T * λ = 0 (1)
dF / dλ = O * (y−x) = 0 (2)
(1) ^{than, x = C * O T *} λ (3)
Substitute (3) into (2)
O * y-O * C * O T * λ = 0
λ = (O * C * O T) -1 * O * y
Substitute λ into (3)
^{x = C * O T * (} O * C * O T) -1 * O * y

[Share code output unit 13]
The share code output unit 13 outputs the encoded component variable having the number of dimensions m as a share code.
According to the present invention, when a three-dimensional model is 30-dimensional with component variables (4 bytes), it can be represented by 120 bytes.
At this time, the share code may be embedded in QR (Quick Response, registered trademark), RFID (Radio Frequency IDentifier), or Cookie.

  The QR code is a matrix type two-dimensional code, and can describe a maximum of 2,953 bytes in binary. In a general smartphone, a QR code can be displayed on a display, and the QR code can be read by a camera.

As a tag, RFID refers to a technology for communicating information described in an RF tag by short-range wireless communication using an electromagnetic field or a radio wave. For example, Felica (registered trademark) is used for electronic money and boarding cards.
According to the present invention, for example, a component variable of a three-dimensional model can be instantaneously read by a reader simply by describing it in an RF tag. The reader that reads the component variable from the RF tag can instantaneously display the three-dimensional model corresponding to the component variable on the display.
According to this user interface, a three-dimensional model corresponding to the composition value and a QR code in which component variables of the three-dimensional model are described can be seen at a glance. In particular, a three-dimensional model can be shared simply by reading a QR code with a camera.

  FIG. 13 is a functional configuration diagram on the decoding side at the operation stage in the present invention.

According to FIG. 13, the device 1 on the decoding side includes a share code input unit 14, a first decoder and 113 using the statistical learning engine 100, a second decoder 123 using the second correlation learning engine 120, And a grayscale image creation unit 15. These functional components can be realized by executing a program that causes a computer mounted on the device to function. In addition, the flow of processing of these functional components can be understood as an encoding method of the device.
The statistical learning engine 100 and the first decoder 113, and the second correlation learning engine 120 and the second decoder 123 may be provided with only one of them according to the use of the device 1. Good.

[First decoder 113]
The first decoder 113 uses the statistical learning engine 100 to decode the component variable having the number of dimensions m into a three-dimensional model. Thus, a three-dimensional model can be generated from the share code.

[Grayscale image creation unit 15]
The grayscale image creation unit 15 creates one or more grayscale images of the decoded three-dimensional model photographed from a predetermined viewpoint on software. Thus, a grayscale image of the generated three-dimensional model viewed from an arbitrary viewpoint can be generated from the share code.

[Second decoder 123]
The second decoder 123 uses the second correlation learning engine 120 to decode a component variable having dimensionality m as target data into a composition value. Thus, a composition value can be generated from the share code.

  FIG. 14 is an explanatory diagram showing the accuracy of the three-dimensional model decoded according to the present invention.

According to FIG. 14, the human body of the target data serving as the four models is shown on the left side.
In the center, a depth image obtained by photographing the human body of the target data with a depth camera is shown.
On the right side, a three-dimensional model generated by encoding the depth image , creating a share code, and then decoding the share code is shown.
Here, as a characteristic point, the human body of the target data on the left side and the generated three-dimensional model on the right side have substantially the same body shape.
Note that the encoding and decoding are realized by a first encoder 111 using the first correlation learning engine 110 and a first decoder 113 using the statistical learning engine 100, and include even a composition value. Absent.

  FIG. 15 is a functional block diagram showing the re-learning at the operation stage in the present invention.

According to FIG. 15, a depth image and a composition value associated as target data are input, and the first correlation learning engine and the second correlation learning engine are re-learned.
The integrating unit 16 inputs a component variable of the first dimension number m of the target data encoded by the first encoder 111 and a component variable of the second dimension number m encoded by the second encoder 121. . Then, the integrating unit 16 integrates the component variable having the first dimension number m and the component variable having the second dimension number m into one component variable having the dimension number m.
At this time, similar to the integration function of the share code output unit 13 in FIG. 12, the weight w that becomes smaller as the number of missing composition values of the composition value of the target data increases is calculated and calculated for each dimension. The weighted average of the component variable having one dimension number m and the component variable having the second dimension number m is integrated as a component variable having one dimension number m.
As a result, it is possible to prevent a share code that strongly depends on only one of the depth image captured by the depth camera or the body composition value measured by the body composition meter.
It is necessary that the number of missing composition values is clear. In this case, a flag such as input value 1 / missing value 0 is set in the share code by using 10 bits corresponding to the 10-dimensional composition value so that the missing composition value can be separately detected on the decoding side. It is also preferable to keep it.

The first correlation learning engine 110 re-learns from the depth image of the target data and the integrated component variables having the number of dimensions m.
Similarly, the second correlation learning engine 120 re-learns from the composition value of the target data and the integrated component variables having the number of dimensions m.

  FIG. 16 is a configuration diagram in which the encoding side function of the present invention is incorporated in a body composition meter.

The body composition meter of the present invention is equipped with a depth camera that partially or wholly photographs the user whose composition value is to be measured. In particular, the depth camera is mounted on a grip that is held by a human hand.
The user to be measured stands on the body composition meter and grasps the grip on which the electrodes are mounted with both hands. Then raise your arms horizontally and extend your elbows. As a result, the arm and the back muscles become vertical, and the grip held by both hands is positioned in front of the face. At this time, the depth camera mounted on the grip can capture a partial body shape including the upper body of the measurement target user from the front of the face.

According to FIG. 16, a body composition meter that measures the composition value of a person incorporates the encoding function of FIG.
As a result, the depth image captured by the depth camera can be encoded into a component variable having the number of dimensions m, and the component variable having the number m of dimensions can be associated with the composition value measured by itself.

According to FIG. 16, the share code output unit 13 may include an integrated function.
The integrated function is a shared code that integrates a component variable of the first dimension m output from the first encoder 111 and a component variable of the second dimension m output from the second encoder 121. Output.
As a result, it is possible to prevent a share code that strongly depends on only one of the depth image captured by the depth camera or the body composition value measured by the body composition meter.

  Here, as the number of missing composition values in the composition value of the target data increases, a smaller weight w is assigned, and a component variable of the first dimension number m and a second dimension number m calculated for each dimension are assigned. Are integrated as a component variable having one dimension number m.

  FIG. 17 is a system configuration diagram showing various arrangements of a depth camera using a body composition meter and a terminal.

According to FIG. 16 described above, one body composition meter is equipped with both a body composition value measurement unit and a depth camera.
On the other hand, according to FIG. 17A, the body composition meter that measures the composition value of a person is a terminal (for example, a smartphone) equipped with a depth camera that partially or wholly captures the user whose composition value is to be measured. ).
The body composition monitor receives, from the terminal, the depth image captured by the depth camera, encodes the depth image into a component variable having the number of dimensions m, and converts the component variable having the number of dimensions m and the composition value measured by itself. Can be assigned.

According to FIG. 17 (b), the terminal is capable of communicating with a body composition meter that measures the composition value of a person, and is equipped with a depth camera that partially or wholly photographs the user whose composition value is to be measured. Things.
The terminal receives the composition value measured by the body composition meter, encodes the depth image captured by the depth camera into a component variable having the number of dimensions m, and receives the component variable having the number m of dimensions and the body composition meter. The composition value can be associated with the composition value.

According to FIG. 17 (c), the terminal can communicate with a body composition meter equipped with a depth camera that measures the composition value of a person and partially or wholly photographs the user whose composition value is to be measured. is there.
The terminal receives, from the body composition meter, the measured composition value and the depth image captured by the depth camera, encodes the depth image into a component variable having the number of dimensions m, The composition value can be associated with the composition value.

  FIG. 18 is an external view illustrating a posture of a user when a depth camera is mounted on a grip portion of the body composition meter.

FIG. 18 shows the angle of view of a depth image that can be photographed from the grip portion of the body composition meter. Specifically, the depth camera has an angle of view of 70 degrees and is inclined downward by 20 degrees from the horizontal direction. In this case, the depth image head portion and knee portions, the wrist portion is not crowded reflected, by encoding and decoding the depth image, it is possible to generate a three-dimensional model.

FIG. 19 is an explanatory diagram illustrating the accuracy of the three-dimensional model decoded from the depth image captured according to the posture of the user in FIG. 18.

According to FIG. 19, the human body of the target data serving as the four models is shown on the left side.
In the center, a depth image of the human body of the target data taken by the depth camera from the grip portion of the body composition meter in the same posture as in FIG. 18 is shown.
On the right side, a three-dimensional model generated by encoding the depth image , creating a share code, and then decoding the share code is shown.
Here, as a characteristic point, the human body of the target data on the left side and the generated three-dimensional model on the right side have substantially the same body shape.
As in FIG. 14, encoding and decoding are realized by a first encoder 111 using a first correlation learning engine 110 and a first decoder 113 using a statistical learning engine 100. Only the height is considered as the composition value.

As described above in detail, according to the program of the present invention, it is possible to generate a three-dimensional model of a measurement target user from a depth image captured by the depth camera.

  For the above-described various embodiments of the present invention, various changes, modifications, and omissions of the scope of the technical idea and viewpoint of the present invention can be easily performed by those skilled in the art. The foregoing description is merely an example, and is not intended to be limiting. The invention is limited only as defined by the following claims and equivalents thereof.

Reference Signs List 1 apparatus 10 depth image creation unit 100 statistical learning engine 110 first correlation learning engine 111 first encoder 113 first decoder 120 second correlation learning engine 121 second encoder 122 missing value estimation unit 123 second decoder 13 Share code output unit 14 Share code input unit 15 Gray scale image creation unit 16 Integration unit

Claims (14)

A program that causes a computer mounted on an apparatus for learning by associating a three-dimensional model of a human body of a teacher data with a depth image ,
Depth image creating means for creating, for each three-dimensional model of the teacher data, a depth image obtained by photographing the three-dimensional model from one or more predetermined viewpoints on software;
A statistic learning engine that outputs a dimensionally compressed m number of component variables from a plurality of three-dimensional models of the teacher data group and constructs a statistic learning model;
A first correlation learning engine for constructing a first correlation learning model of a depth image of the three-dimensional model and a component variable having the number of dimensions m for a plurality of three-dimensional models of the teacher data group;
For a plurality of three-dimensional models of the teacher data group, a second correlation learning model is constructed between a composition value of dimension n including at least height and one or more body composition measurement values and a component variable of dimension m. A program causing a computer to function as a second correlation learning engine . The program according to claim 1, wherein the statistical learning engine causes the computer to function as based on Principal Component Analysis or AutoEncoder. First correlation learning engine of state, and are not based on convolutional neural network,
The program according to claim 1 or 2, wherein the second correlation learning engine causes the computer to function as being based on a least squares method or a multilayer perceptron . 4. The computer according to claim 1, wherein the computer is operated such that the number of plural bodies of the three-dimensional model of the teacher data group is smaller than the number of vertices of the three-dimensional model. 5. program. A first encoder that encodes a depth variable as target data into a component variable having a dimension number m using a first correlation learning engine ;
Using a second correlation learning engine to cause a computer to function as a second encoder that encodes a composition value having one dimension number n as a target data into a component variable having a dimension number m. The program according to any one of claims 1 to 4, characterized in that: In order to input a depth image and a composition value associated as target data and re-learn the first correlation learning engine and the second correlation learning engine,
A first encoder that encodes the depth image of the target data into a component variable having a first dimension number m;
A second encoder that encodes the composition value of the target data into a component variable having a second dimension number m;
Integrating a component variable having a first dimension number m and a component variable having a second dimension number m into a component variable having one dimension number m;
A first correlation learning engine that re-learns from the depth image of the target data and the integrated component variables having the number of dimensions m;
The program according to claim 5 , wherein the second correlation learning engine causes a computer to function so as to re-learn from the composition value of the target data and the integrated component variables having the number of dimensions m. The component variable of the first dimension m and the component variable of the second dimension m calculated for each dimension by assigning a smaller weight w as the number of missing composition values of the composition value of the target data increases. 7. The computer-readable storage medium according to claim 6 , wherein the computer is caused to integrate a weighted average of the two as a component variable having one dimension number m. Only k (<n) composition values are determined for the composition value of one dimension n as the target data, and even if other nk composition values are missing, the component variable of dimension m is determined. 6. The computer according to claim 5 , further comprising a computer functioning as a missing value estimating means for estimating other nk optimized composition values by using k composition values as constraints for the estimation. The described program. 9. The program according to claim 8 , wherein the missing value estimating means causes a computer to function so as to use a Lagrange's undetermined multiplier method. The computer according to any one of claims 1 to 9 , wherein the computer further functions as a first decoder that decodes the component variable having the number of dimensions m into a three-dimensional model using the statistical learning engine. program. The program according to claim 10 , wherein the program causes a computer to function as a grayscale image creating unit that creates one or more grayscale images of the decoded three-dimensional model photographed from a predetermined viewpoint on software. The computer according to any one of claims 1 to 11 , wherein the computer is caused to function as a second decoder that decodes a component variable having the number of dimensions m as target data into a composition value using the second correlation learning engine. The program described in. Before Symbol depth image, the program according to any one of claims 1 to 12, characterized in that causes a computer to function as a picture image of the gray-scale gradation, depending on the depth. Outputs the encoded component variables of the number of dimensions m as a share code, and as a share code output means for embedding the share code in QR (Quick Response, registered trademark), RFID (Radio Frequency IDentifier) or Cookie further program according to any one of claims 1 1 to 3, characterized in that causes a computer to function.