KR20190083757A - Calibration method of leap motion-HMD using bundle adjustment algorithm and method thereof - Google Patents

Abstract

The present invention relates to a method and apparatus for calibrating between a lip motion and an HMD using a bundle adjustment algorithm. According to the present invention, there is provided a method of converting an n-dimensional image into a plurality of n-dimensional matrices, the method comprising the steps of: converting n two-dimensional images obtained at different points of time obtained by photographing a front surface of an HMD equipped with lip motion, Acquiring n projection matrices using one 3-D point group and n sparse matrices generated by approximating the 3-D space matrix to 3-D space, Dimensional clusters corresponding to the shape of the infrared ray bulb are detected for each of the n two-dimensional images, and then a plurality of clusters corresponding to the shape of the infrared ray bulb are detected, Determining three-dimensional center points of each bulb cluster based on three-dimensional points corresponding to two-dimensional points located within the global cluster among the belonging points in the three-dimensional point cloud; And estimating and calibrating the origin of the lip motion and the HMD using the three-dimensional center point and the cluster size of each bulb cluster.
According to the present invention, the relative positional error between the HMD and the lip motion can be corrected using the infrared images of the plurality of points of view obtained from the front face of the image-viewing HMD equipped with the lip motion, thereby enhancing the sense of realism and immersion of the user wearing the HMD .

Description

[0001] The present invention relates to a method and apparatus for calibrating a lip motion between a lip motion and an HMD using a bundle adjustment algorithm,

The present invention relates to a method and apparatus for calibration between lip motion and an HMD using a bundle adjustment algorithm and a calibration method between lip motion and an HMD using a bundle adjustment algorithm capable of correcting the coordinate system error between the image- ≪ / RTI >

With the spread of HMD (Head Mount Display) devices, interest in the virtual reality environment is increasing. The HMD (Head Mount Display) is largely classified into an optical see-through system and a video see-through system. Imaginary vision HMD is more immersive for virtual reality than optics perspective, and research using it is actively proceeding.

However, a user who wears a video-based HMD can only view a video of the HMD and can not see the user interface at all, making it difficult to intuitively interact with the user. An intuitive interface method, such as motion recognition, is needed for a user to accurately interact without having to know the location of the interface device.

In the interface device using motion recognition, Leap Motion is a device that receives light in recognition of intuitive hand motions or finger positions of a user. In addition to HMD, lip motion enables intuitive interaction with the user, and is excellent in interoperability with the HMD, and is attached to the HMD.

However, when attaching the lip motion to the front of the HMD, the position where the lip motion is attached differs from device to device, and the coordinate system of the lip motion and the HMD are different from each other. Therefore, when the user uses the HMD with the lip motion, it is difficult to accurately interact with the lip motion while viewing the screen of the HMD.

FIG. 1 is a diagram showing a state in which a lip-motion is mounted on a video-watching HMD. FIG. 1 conceptually shows that it is difficult for the user to accurately interact with the lip motion-attached HMD when the position of the hand detected by the HMD is not matched with the position of the hand sensed by the lip motion.

For accurate user interaction, it is most effective to match the operating position of the lip motion with the operating position of the output screen. Therefore, there is a need for a method of calibrating the positional relationship between the position of the HMD screen and the lip motion.

In the case of the optically-seated HMD, which is one of the HMD types, the actual environment that the user sees is the output screen of the HMD, so that the user can see the position of the finger, so that the user can directly calibrate.

Among them, there is a method of performing calibration between the optical-perspective HMD screen and the lip motion which the user sees in the three-dimensional space. In this method, a user adjusts a virtual reticle with a finger to adjust it with a 3D marker. Since a marker on a three-dimensional space is used, the accuracy of the three-dimensional calibration can be enhanced. However, And the interaction method of adjusting the virtual reticle may feel uncomfortable to the user.

There is also a calibration technique using an eye recognition camera attached to an optically transparent HMD. Since this method determines the position of the user's eye through the camera, the performance of the camera affects the calibration performance and the accuracy of the calibration depends on the characteristics of the user's eyeball or the wear of the HMD. Therefore, You must re-do it.

The above-mentioned optically-sighted HMD and lip-motion calibration techniques are available in AR environment based on actual environment, and correspond to the mechanical characteristics in which the external appearance does not enter the user's eye to be.

However, in the case of attaching lip motion to a video-perspective HMD, the output screen is a virtual reality (VR) environment which is separated from the actual environment due to the nature of the video-based HMD. Thus, the user's eyeball is in close contact with the screen of the video- It is difficult to apply the calibration technique used in the optical perspective HMD.

Therefore, a method for measuring the relative position of the HMD and the lip motion is required for calibration between the video-perspective HMD and the lip motion.

The technology which is the background of the present invention is disclosed in Korean Patent No. 1712350 (published on Mar. 07, 2017).

Between lip motion and HMD using a bundle adjustment algorithm capable of reliably correcting the relative positional difference between the image-wise HMD and lip motion so as to enable accurate inter-device interaction when using the image-through HMD with lip motion And more particularly, to a calibration method and apparatus for calibration.

The present invention relates to a calibration method between a lip motion and an HMD using a calibration apparatus, the method comprising the steps of: capturing n two-dimensional images at different points of time obtained by photographing a front surface of an HMD equipped with a lip motion, And obtaining n projection matrices using the n sparse matrices and a single 3-D point group generated by approximating the n sparse matrices in a three-dimensional space through a bundle adjustment algorithm Dimensional projection of the n-dimensional image; and projecting each of the n 2-dimensional point clouds to the n 2-dimensional images through the n projection matrices, respectively, Dimensional cloud image, the method comprising the steps of: detecting a plurality of light bulb clusters corresponding to a shape of an infrared light bulb in the two-dimensional image; Obtaining three-dimensional center points of each of the bulb clusters based on three-dimensional points corresponding to two-dimensional points located within the bulb cluster among the belonging points in the three-dimensional point cloud group, And estimating and calibrating the origin of the lip motion and the origin of the HMD using the three-dimensional center point and the cluster size of the lip motion and the HMD.

The calibration method between the lip motion and the HMD may further include obtaining n pixel matrices corresponding to the n two-dimensional images, wherein the converting into the n n sparse matrices comprises converting the n pixel matrices Can be converted into the n number of sparse matrices, respectively.

The step of determining the two-dimensional points located within the cluster may include detecting a plurality of circular clusters of a circle through the blob detection technique in the two-dimensional image, Dimensional point group and a distance between the center points of the two-dimensional point cloud and a distance between the center points of the two-dimensional point cloud and a distance between the center points of the two- It can be discriminated by two-dimensional points.

The calibration may be performed by moving the origin coordinates of the lip motion to the origin coordinates of the HMD.

The obtaining of the three-dimensional center point of each of the bulb clusters may include a method of averaging coordinates of three-dimensional points corresponding to two-dimensional points located in the bulb cluster among the belonging points in the three-dimensional point cloud, The center-of-gravity point of the three-

The calibration step may include classifying the first clusters into L first clusters and M remaining second clusters based on the sizes of the respective bulb clusters (L < M) The origin coordinates of the lip motion can be estimated by averaging the coordinates of the 3D center point, and the coordinates of the 3D center point with respect to the second clusters can be averaged to estimate the origin coordinate of the HMD.

According to another aspect of the present invention, there is provided an apparatus for calibrating between a lip motion and an HMD, comprising: an n-dimensional two-dimensional image obtained by photographing a front surface of an HMD equipped with a lip motion, And a n-th spatially ordered matrix. The n-th spatially ordered matrix is generated by approximating the n sparse matrices to a three-dimensional space through a bundle adjustment algorithm, A 2D point estimator for obtaining the projected n two-dimensional point groups by projecting the three-dimensional point cloud onto the n two-dimensional images through the n projection matrices, respectively, A blob detector for detecting a plurality of light bulb clusters corresponding to the shape of the infrared bulb in the two-dimensional image for each of the two-dimensional image, A bladder discriminator for discriminating two-dimensional points located within a bulb cluster among the boundary points, and a bladder classifier for determining three-dimensional points of each bulb cluster based on three-dimensional points corresponding to the two- And a position estimator estimating and calibrating the origin of the lip motion and the origin of the HMD using the three-dimensional center point and the cluster size of each of the bulb clusters, and a calibration device between the lip motion and the HMD .

The calibration device between the lip motion and the HMD further includes a pixel matrix generator for obtaining n pixel matrices corresponding to the n two-dimensional images, wherein the sparse matrix generator is configured to calculate the n pixel matrices And can be converted into the n number of sparse matrices.

According to the present invention, the difference in the origin coordinates between the HMD and the lip motion is obtained by using the two-dimensional infrared image obtained from the front face of the image-viewing HMD so as to enable precise interaction between the devices when the lip- It is possible to further enhance the sense of reality and immersion of the user wearing the video-watching HMD.

FIG. 1 is a diagram showing a state in which a lip-motion is mounted on a video-watching HMD.
2 is a view showing a configuration of an image-viewing HMD.
FIG. 3 is a view showing a concept that lip motion recognizes an operation on the front side of FIG. 2. FIG.
4 is a diagram illustrating a calibration apparatus according to an embodiment of the present invention.
5 is a simplified representation of the operational concept of FIG.
6 is a diagram illustrating a bundle adjustment algorithm used in an embodiment of the present invention.
FIG. 7 is a view showing the lip motion and the three-dimensional position of the infrared ray lamps formed in the HMD.
8 is a diagram showing positional coordinates derived for the lip motion and the HMD.
FIG. 9 is a diagram illustrating an infrared ray image taken from the front side of an HMD equipped with lip motion in the embodiment of the present invention.
10 is a diagram for explaining a calibration method using the apparatus of FIG.
11 shows n infrared images acquired according to an embodiment of the present invention.
12 is a result of generating a sparse matrix after binarizing an infrared image.
13 shows a result of performing a rare bundle adjustment algorithm to output one three-dimensional point cloud.
Fig. 14 is a result of clustering the positions of the infrared ray bulbs from the infrared ray image of Fig.
FIG. 15 is a result of estimating a two-dimensional point cloud corresponding to a global cluster using the images of FIG. 12 and FIG.
FIG. 16 is a result of deriving the three-dimensional center point position of each bulb cluster through the results of FIG.
17 shows various positions of the HMD with lip motion.
FIG. 18 is a representation of the actual physical position difference between the HMD and the lip motion and the positional difference derived by the experiment in the three-dimensional coordinate system.
FIG. 19 shows the result of calculating the error rate with respect to FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention.

The present invention relates to a method and apparatus for calibrating between a lip motion and an HMD using a bundle adjustment algorithm. More particularly, the present invention relates to a method and an apparatus for calibrating the relative positional relationship between devices when a lip motion is mounted on an HMD We propose a technique to calculate and calibrate the positional difference.

2 is a view showing a configuration of an image-viewing HMD. Fig. 2 shows an Oculus Rift, which is a video-watching HMD developed by Ocurus VR. The Oculus Lift has a total of 40 infrared ray bulbs (infrared LEDs) buried invisibly in the front (built in) and operates in a set with an infrared sensor device (not shown) to track its position and attitude.

As described above, in the image-viewing HMD, a plurality of infrared light bulbs are dispersed and arranged in the front part, and the position and the posture of the HMD are detected by detecting the infrared rays emitted from the infrared light bulbs through an infrared sensor device You can output the figure of the direction that you grasped on the screen.

FIG. 3 is a view showing a concept that lip motion recognizes a hand gesture on the front side of FIG. 2. FIG. Leap Motion is a motion recognition sensor that recognizes the user's hand movements. It has three infrared ray bulbs (infrared ray emitters) and two infrared ray sensors (infrared cameras) on the front, and the reflected light of infrared rays emitted by infrared ray bulbs Each infrared sensor receives and recognizes hand movements.

Both the lip motion device and the video-watching HMD device have a common point in that a plurality of infrared ray bulbs are built in the front side, and the position and size of each infrared ray bulb can be acquired through the image of the infrared ray camera.

In the embodiment of the present invention, when the lip motion is attached to the front face of the HMD (ex oculus lift), the difference between the area sensed by the lip motion and the area shown by the HMD device is detected by an external key (infrared sensor) We propose a method to calibrate and calibrate based on two - dimensional infrared images.

When the lip motion is mounted on the front surface of the HMD, a calibration process for correcting the error of the working area between the two devices is required. The embodiment of the present invention obtains infrared images of the front surface of the HMD with lip motion, And then the origin of the lip motion is moved to the origin of the image-viewing HMD to calibrate the HMD and the lip motion.

FIG. 4 shows a calibration apparatus according to an embodiment of the present invention, and FIG. 5 is a simplified view of the operation concept of FIG.

4 and 5, a calibration apparatus 100 according to an exemplary embodiment of the present invention includes a pixel matrix generator 110, a sparse matrix generator 120, a sparse bundle adjuster 130, a 2D point estimator 140, A blob detector 150, a blob detector 160, and a position estimator 170.

5, the calibration apparatus 100 according to the embodiment of the present invention includes n pieces of images obtained by photographing the front portion of the HMD with the lip motion mounted thereon at different points of view through an infrared camera (ex) And operates by receiving a two-dimensional image. A technique for obtaining three-dimensional information of an object in an image by using a plurality of two-dimensional images taken at different viewpoints has been well known in the past.

The pixel matrix generator 110 receives n 2-dimensional images at different points in time from the Kinect and obtains n pixel matrices corresponding to the 2-dimensional images. The pixel matrix generator 110 constructs a pixel matrix through each pixel in the two-dimensional image, thereby generating a pixel matrix corresponding to the two-dimensional image.

Sparse matrix generator 120 converts n pixel matrices back into n sparse matrices, respectively. The reason for converting the pixel matrix to a sparse matrix is to reduce the amount of computation in the rare bundle adjuster 130.

It is inefficient to store and compute all the pixel values of a large amount because the infrared image often has a value of 0 in most cases. In order to solve this problem, a sparse matrix means a matrix that stores non-zero pixel values and their position information in an infrared image, and does not store positions of pixel values of zero. With this method, it is possible to greatly reduce the amount of calculation when processing pixel values in an infrared image.

4, the sparse matrix generator 120 is separate from the rare bundle adjuster 130, but may be included in the rare bundle adjuster 130 as shown in FIG.

The sparse matrix generator 120 receives the pixel value of the photographed image and outputs a sparse matrix M _t to reduce the amount of calculation of the bundling adjustment. If α and β are the width and height of the infrared image, respectively, the sparse matrix M _t has α × β size, and the elements of the sparse matrix are defined as m _{t, x, y} . A sparse matrix is a matrix composed of elements of 0 and 1, and the following equation (1) is used.

Here, i _{t, x, and y} are pixel values of an image acquired at time t, and are achromatic color scale (Gray Scale) and values ranging from 0 to 255. The threshold value [theta] is a constant specified for comparison with each pixel. If i _{t, x, y} is less than or equal to the threshold θ, 0 is assigned to m _{t, x, y} , otherwise 1 is assigned to proceed with the binarization.

The sparse bundle adjuster 130 approximates n sparse matrices to a three-dimensional space using a bundle adjustment algorithm to generate one three-dimensional point cloud P. Thus, the shape of the HMD with lip motion is derived in the form of a 3D point cloud.

Thus, the sparse bundle adjuster 130 receives n sparse matrices and generates a three-dimensional point group P = {(x ₀ , y ₀ , z ₀ ), (x ₁ , y ₁ , z ₁ ) , ... }. Here, each (x, y, z) represents three-dimensional points belonging to the three-dimensional point group P.

The bundle adjustment algorithm is a well-known method. When a feature point is found, a Levenberg-Marquardt optimization technique is used to obtain a three-dimensional point that minimizes the change of all pixels in the image. In this manner, one three-dimensional point cloud corresponding to the object is created using a plurality of two-dimensional images of the object.

In addition, the rare bundle adjuster 130 obtains n projection matrices using a three-dimensional point cloud and n sparse matrices. The projection matrix is a transformation matrix for projecting the three-dimensional point cloud P onto each two-dimensional image. n sparse matrices again You can calculate the projection matrix (transformation matrix) by putting it into the Levenberg-Marquardt algorithm.

6 is a diagram illustrating a bundle adjustment algorithm used in an embodiment of the present invention. Images 1, 2, and 3 are two-dimensional images of HMDs taken at different points in time. Point coordinates on a three-dimensional space can be projected onto each two-dimensional image point through a transformation matrix, but vice versa.

The 2D point estimator 140 obtains n projected two-dimensional point groups by projecting the three-dimensional point cloud onto n two-dimensional images through n projection matrices, respectively. By multiplying the three-dimensional point group P by n projection matrices, it is possible to derive a two-dimensional point group P 'which is a position of an image formed on the two-dimensional camera coordinate system. When the point group P is multiplied by the projection matrix, P is converted into a column vector and multiplied.

In FIG. 4, the 2D point estimator 140 is separate from the rare bundle adjuster 130, but may be included in the rare bundle adjuster 130.

As described above, the 2D point estimator 140 receives the approximate three-dimensional point group P and n projection matrices approximated by the rare bundle adjuster 130 and grasps the positional relationship between the three-dimensional point cloud and each two-dimensional image, And two n-dimensional point groups P 'on which a three-dimensional point cloud is projected are derived.

Here, t the two-dimensional point cloud projected onto a first two-dimensional image (infrared image) is, and defined as _t, P 'P _{t = {(X 't, 0} , y' t, 0), (x 't, 1, y' t, 1), ... }to be. At this time, each (x ', y') is two-dimensional points belonging to the two-dimensional point group P ' _t .

The blob detector 150 clusters the infrared image, and detects a plurality of light bulb clusters corresponding to the shape (circle) of the infrared light bulb in the two-dimensional image for each of the n two-dimensional images.

Accordingly, the infrared light bulbs on the HMD and the lip motion are detected in the form of a circular cluster in the two-dimensional image. Of course, some IR bulbs of the HMD that are obscured by the attachment of lip motion are not detected in the image. This cluster detection operation is performed on each of two n-dimensional images.

The blob detector 150 receives the n 2-dimensional infrared images i _t and applies the blob detection algorithm to obtain the positions of the infrared light bulbs in the 2-dimensional image. When the bubble is detected, the part of the infrared bulb which is the light source appears as a circular bubble. Here, a Hough transform algorithm capable of extracting a circle shape from an image can be used.

The blob detector 150 detects clusters G _{t, p} through blob detection from the two-dimensional image and obtains the center point C _{t, p} and radius d _{t, p} of the cluster. Here, G _{t, p} is defined as a p-th cluster for the infrared image i _t obtained at time t. And a cluster G _t, q th element of _p g _{t, p, q,} g _{t, p,} the center point of _q is defined by a _{c t, p = (c t} , p, x, c t, p, y) do.

After extracting the circular cluster, the center point c _{t, p} is obtained by the following equation (2).

Here, d _{t, p} is the distance between a light bulb cluster G _t, the distance (the maximum distance value) between the center point C _t, it points from _p home away in the _p, bulbs cluster G _{t, p} in the center and the rest of the dots Which means the maximum distance value. That is, the Euclidean distance (E) between the center point c _{t, p} and the remaining points is obtained, and then d _{t, p} is obtained by selecting the largest value among the obtained values. Hereinafter, d _{t, p} is referred to as the radius of the cluster for convenience of explanation.

The blob discriminator 160 discriminates the two-dimensional points located in the global cluster among the belonging points in the two-dimensional point cloud through the output results of the 2D point estimator 140 and the blob detector 150.

For this purpose, the blob discriminator 160 obtains the distance between the center points of the bulb clusters obtained from the blob detector 150 and the belonging points within the two-dimensional point cloud group obtained from the 2D point estimator 140, The two-dimensional points derived from the distance less than the maximum distance value are determined as the two-dimensional points located within the corresponding bulb cluster.

The points in the two-dimensional point group by the 2D point estimator 140 and the points in the cluster by the blob detector 150 are both two-dimensional coordinates. Therefore, if the distance between the belonging point in the two-dimensional point cloud and the center point of the cluster is obtained and compared with the cluster radius, it can be confirmed whether or not the point is located within the cluster (effective point). Of course, such a blob discrimination operation is performed on each of two n-dimensional images.

Specifically, the blob discriminator 160 detects the two-dimensional point cloud P ' _t output from the 2D point estimator 140, the cluster G _{t, p} in the two-dimensional image output from the blob detector 150, the center point c _{t , p} , and distances d _{t and p} , respectively, and discriminates whether or not the points actually present in the global cluster (points located in the global cluster) among the belonging points in the two-dimensional point group P ' _t .

Here, it is possible to construct a separate cluster G ' _{t, p} using points located in the element g _{t, p} in the global cluster among the points in the point group P' _t . Let G ' _{t, p} denote the clusters collected and reconstructed from the two-dimensional point group P' _t belonging to the inside of the cluster G _{t, p} .

이고, 그렇지 않으면

이다.At this time, the Euclidean distance between the two-dimensional positional coordinate p ' _t (x' _t , y ' _t ) of the belonging point in the two-dimensional point cloud group and the center point c _{t, p} of the cluster obtained in the blob detector 150 is obtained, It is possible to confirm whether or not the two-dimensional position coordinate is included in the cluster according to whether or not the cluster is smaller than d _{t, p} . For example, if the obtained value is _{dt, p} or less

, Otherwise

to be.

4, the blob discriminator 160 is separate from the blob detector 150, but may also belong to the blob detector 150 as in FIG.

The position estimator 170 uses the three-dimensional point group P approximated by the rare bundle adjuster 130 and the cluster G ' _{t, p} obtained by the bubble discriminator 160 to calculate the three-dimensional origin coordinate P _L of the lip motion the origin of the lip motion is moved to the origin of the HMD by using the three-dimensional origin coordinate P _O = (x _o , y _o , z _o ) of the HMD ( _l , y _l , z _l ) And calibrate it.

Specifically, the position estimator 170 extracts only the three-dimensional points corresponding to the two-dimensional points located in the global cluster among the belonging points in the three-dimensional point cloud P, and calculates the three- The center point is acquired first. Then, the origin of the lip motion and the origin of the HMD are estimated and calibrated using the three-dimensional center point and the cluster size of each bulb cluster.

Here, the two-dimensional points located in the bulb cluster mean the points of the cluster G ' _{t, p} output from the blob discriminator 160. That is, the position estimator 170 extracts three-dimensional points corresponding to the two-dimensional points determined to be located in the global cluster, and then extracts the three-dimensional center point . At this time, the three-dimensional center point of the corresponding bulb cluster is calculated using the coordinate averages of the three-dimensional points corresponding to the two-dimensional points located in the corresponding bulb cluster.

The relationship between the three-dimensional point group and the corresponding two-dimensional point can be confirmed through the projection matrix. If the three-dimensional point group is included in the three-dimensional point group, the three-dimensional point corresponding to the two- .

In other words, by deriving three-dimensional points corresponding to two-dimensional points in a cluster from a three-dimensional point group, the coordinates of three-dimensional points in the cluster can be obtained, and if all the coordinates of the three- The three-dimensional center point can be known.

Thereafter, the position estimator 170 calculates L (three in the case of lip motion) first clusters (light motion bulb clusters) based on the sizes of the respective bulb clusters and the remaining M And classified into second clusters (global cluster of HMD) (L < M). Of course, the cluster size comparison can use a two-dimensional or three-dimensional size.

In addition, the position estimator 170 estimates the origin coordinates of the lip motion by averaging the coordinates of the three-dimensional center point with respect to the first clusters, then averages the coordinates of the three-dimensional center point with respect to the second clusters, .

FIG. 7 is a view showing the lip motion and the three-dimensional position of the infrared ray lamps formed on the HMD, and FIG. 8 is a diagram showing the position coordinates derived from the lip motion and the HMD.

The position estimator 170 first obtains the three-dimensional center point of each bulb cluster (oculus bulb, lip motion bulb) shown in FIG. 7, and then divides the lip motion and the cluster of the HMD using the size of each bulb cluster 8, the origin position P _L of the lip motion and the origin position P _O of the HMD are obtained.

FIG. 9 is a diagram illustrating an infrared ray image taken from the front side of an HMD equipped with lip motion in the embodiment of the present invention. As shown, three positions of the intermediate point of the lip motion higher than the infrared ray bulb of the HMD and larger than the circumference of the infrared ray image are the position of the infrared ray bulb of the lip motion.

That is, when the three-dimensional center points of the three large clusters are averaged, the origin position P _L of the lip motion is derived, and when the three-dimensional center points of the remaining clusters are averaged, the origin position P _O of the HMD is derived. When the origin position of the lip motion is moved to the origin position of the coordinate system of the HLD, the position coordinates of the changed lip motion become P ' _L (x _o -x _l , y _o -y _l , z _o -z _l ).

The position estimator 170 corrects the origin coordinates of the lip motion by adjusting the origin coordinates of the HMD using the difference between the lip motion and the HMD origin coordinate. Accordingly, the origin of the coordinate system of the lip motion and the origin of the coordinate system of the Oculus screen become equal to each other.

The following describes the calibration method based on the above description. 10 is a diagram for explaining a calibration method using the apparatus of FIG.

Referring to FIG. 10, the sparse matrix generator 120 receives n two-dimensional images at different points in time taken by an infrared camera and converts the front face of the HMD with lip motion into n sparse matrices, respectively (S1001). Here, the sparse matrix generator 120 receives n pixel matrices corresponding to n two-dimensional images from the pixel matrix generator 110, and then converts each of the n pixel matrices into n sparse matrices.

Next, the sparse bundle adjuster 130 generates one three-dimensional point group by approximating the n sparse matrices to the three-dimensional space through the bundle adjustment algorithm (S1002), and generates n points by using the three-dimensional point cloud and n sparse matrices And obtains a projection matrix respectively (S1003).

Then, the 2D point estimator 140 again uses n projection matrices to project a three-dimensional point cloud onto n two-dimensional images, respectively, and obtain n two-dimensional point clouds projected onto n two-dimensional images ( S1004).

Next, the blob detector 150 detects a plurality of light bulb clusters corresponding to the shape of the infrared light bulb in the two-dimensional image, for each of the n two-dimensional images acquired by the infrared camera (S1005). Then, the blob discriminator 160 discriminates the two-dimensional points located within the global cluster among the belonging points in the two-dimensional point cloud group (S1006).

Thereafter, the position estimator 170 acquires the three-dimensional center point of each bulb cluster based on the three-dimensional points corresponding to the two-dimensional points located in the cluster among the belonging points in the three-dimensional point cloud group (S1007) And the cluster size, the coordinates of the origin of the lip motion and the origin of the HMD are estimated and calibrated (S1008).

The following describes performance experiments and analysis results of the calibration technique of the present invention.

The purpose of the experiment is to judge whether the technique of the present invention, which can be utilized as a preprocessing technology for contents using HMD with lip motion in the future. for teeth. After the calibration program corresponding to the present invention is manufactured, errors in the results of the technique of the present invention are calculated using the known physical distance difference between the HMD and the lip motion as an experimental control, and the performance thereof is verified. The development environment of the experimental computer is Visual Studio 2013 based on Windows 8.1, and the specification uses Intel i7-6700 @ 3.40GHz CPU, NVidia's Geforce GTX 1060 GPU, and DDR5 16GB 2133MHz.

In the experiment, Kinect sensor, infrared sensor, is used to detect lip motion and infrared image of HMD. That is, 40 infrared ray bulbs of the HMD and 3 infrared ray bulbs attached to the HMD are photographed with the infrared sensor, Kinect. The output value of the program is a three-dimensional relative position coordinate between the lip motion estimated according to the infrared image and the HMD, and is generated in a coordinate system parallel to the x, y and z axes of the coordinate system of the estimated three-dimensional point cloud group. Relative position coordinates are expressed in cm.

In order to determine the accuracy of the three-dimensional relative position coordinates of the HMD and the lip motion, calibration was performed by varying the positions of the lip motion on the actual HMD, and using this, it is checked whether the proposed calibration technique has a certain accuracy Respectively.

The program is executed in the following manner.

First, n-dimensional infrared images of two-dimensional HMDs with lip motion are input. 11 shows n infrared images acquired according to an embodiment of the present invention. As shown in Fig. 11, the number of infrared images input to the program must be at least three or more.

12 shows the result of generating a sparse matrix after binarizing an infrared image. Each input infrared image is converted into a sparse matrix through a sparse matrix generator. Converting all n infrared images to a sparse matrix can reduce the amount of computation.

13 is a result of performing a rare bundle adjustment algorithm to output one three-dimensional point cloud. The sparse bundle adjuster computes a 3D point cloud by approximating n sparse matrices to a three-dimensional space and then uses a single 3D point cloud and n transformation matrices to generate 3D point clouds for each infrared image Respectively. When a projection matrix is multiplied by a three-dimensional point group, two-dimensional point coordinates projected on the two-dimensional image are generated.

The 2D point estimator receives the infrared image of FIG. 13 and projects it onto n two-dimensional images. The 2D point estimator uses a projection matrix (transformation matrix) T and a three-dimensional point cloud to determine which two-dimensional position each point in the three-dimensional point cloud is projected when the three-dimensional point cloud is projected onto the two-dimensional color image.

Fig. 14 is a result of clustering the positions of the infrared ray bulbs from the infrared ray image of Fig. This is representative of the results for one of several infrared images.

The blob detector receives the image of FIG. 11 and outputs it as shown in FIG. That is, the position of the infrared bulb is clustered by using the two-dimensional infrared image.

FIG. 15 is a result of estimating a two-dimensional point cloud corresponding to a global cluster using the images of FIGS. 12 and 14. The blob discriminator receives the images of FIGS. 12 and 14 and outputs the images as shown in FIG. The blob discriminator distinguishes the two-dimensional position coordinates in the global cluster using the two-dimensional position coordinates derived from the 2D point estimator and the global cluster derived from the blob detector.

FIG. 16 is a result of deriving the three-dimensional center point position of each bulb cluster through the results of FIG. The position estimator derives the three-dimensional center point position of each cluster using the three-dimensional space coordinates corresponding to the clusters derived from the bubble discriminator. 16, the position of the center point of each light bulb cluster included in the lip motion and the HMD can be confirmed. When the 3D center point is derived for all the clusters, the position estimator divides the lip motion and HMD clusters by using them and calculates the origin coordinates of the lip motion and the HMD, respectively.

Finally, n infrared images are input to the program and output to the origin position of one lip motion and the origin position of one oculus (HMD).

17 shows various positions of the HMD with lip motion. In order to verify the method of the present invention, the position of the HMD to which the lip motion is attached is divided into 11 positions, and 10 calibration steps are performed for each position. The result of comparison between the actual position difference and known position difference between HMD and lip motion derived from calibration is as follows.

Table 1 shows the position of the HMD and lip motion derived from the calibration and the difference between the HMD and lip motion through actual physical measurements.

Experimental group Actual measured value (cm) Experimental value (10 times average, cm) x y z x y z ① 0.5 6.9 5.6 0.44 6.75 5.51 ② -5.8 6.9 5.6 -5.11 6.75 5.55 ③ -7 4.6 5.5 -7.49 4.82 5.33 ④ -5 3.2 5.5 -5.43 3.70 4.9 ⑤ 0.6 3 5.9 -0.48 2.82 6.16 ⑥ 5 1.9 5.7 4.72 3.23 5.27 ⑦ 5.4 4.9 6.1 5.31 4.7 5.91 ⑧ 5.3 6.8 6 6 7.16 6.36 ⑨ -0.2 10.1 5.2 -0.09 8.21 6.07 ⑩ -10.3 4.2 4.9 -7.93 4.72 5.22 ⑪ 10 4.4 6 9.38 4.77 5.6

FIG. 18 is a representation of the actual physical position difference between the HMD and the lip motion and the positional difference derived by the experiment in the three-dimensional coordinate system. The y-plane is the surface of the HMD, and the z-axis is the z-axis distance from the lip motion to the lens on which the HMD screen is output. The average error distances of the x, y, and z coordinates for all lip motion locations are 0.59 cm, 0.42 cm, and 0.34 cm, respectively.

FIG. 19 shows the result of calculating the error rate with respect to FIG. The error rate is calculated as a percentage of the probability that the theoretical value is divided by the theoretical value (actually measured value) minus the experimental value. FIG. 19 is a graph showing the error rate using the difference between the actual measurement value and the experimentally derived value.

The error distances of the x, y, and z values in the experimental groups 9 to 11 were relatively large because the positions of the lip motion on the HMDs in the experimental groups 9 to 11 were different from those in the other experimental groups 1 to 8 .

The error rate for each experiment is shown in Table 2 below. Each error rate represents the second decimal place. The mean error rates in Table 2 are 24.88%, 7.61%, and 0.02% for the x, y, and z coordinates, respectively.

Lip motion position Average error rate (%) x y z ① 11.2 2.12 1.55 ② 11.88 2.14 0.89 ③ -7.04 -4.83 3.02 ④ -8.52 -15.72 10.91 ⑤ 6.21 6.07 -4.32 ⑥ 5.54 -8.5 7.63 ⑦ 1.72 4.06 3.18 ⑧ -5.64 -5.24 -5.92 ⑨ 55.5 18.7 -16.71 ⑩ 23.01 -12.3333 -6.59 ⑪ 179.83 -70.16 6.63 Average 24.88 -7.61 0.02 ① ~ ⑧ Average 6.42 6.09 4.68

In Table 2, the error values appear high in the x and y axes, and relatively low in the z values. This is because the relative position of the lip motion used in the experimental group is often located on the surface of the HMD. Approximately 10.84% is obtained when the average error rate is obtained for all x, y, and z axes. Most of this error occurred when lip motion was not attached to the surface of the HMD. This is because infrared rays emitted from an infrared light bulb indicate different directions when detecting an infrared image.

The mean error rates except for ⑨ ~ ⑪ which did not have lip motion on the surface of HMD were 6.42%, 6.09% and 4.68% in x, y and z coordinates, respectively. If the actual value of the distance difference between HMD and lip motion is maximum 10.3cm, the maximum error is reduced to about 0.066cm.

Therefore, it can be seen that more precise calibration is possible when the lip motion is attached parallel to the front of the HMD. From the above results, it can be seen that the technique of the present invention can be applied to a user content as a preprocessor in the future.

As described above, the present invention proposes a lip motion calibration method of an HMD for solving the problem of different operation regions between devices when lip motion is attached to the HMD. For this purpose, images of the infrared light bulb embedded in HMD and lip motion are acquired and the 3D point cloud coordinates are estimated using the bundle adjustment algorithm. Infrared bulb images were clustered using blob detection algorithm to locate the infrared bulb. Finally, we used the estimated 3D point cloud and clusters to calculate the difference between the HMD and lip motion origin positions. As a result, we can derive the experimental results showing the maximum error distance of 0.066cm for the position estimation of lip motion attached parallel to the front of the HMD.

Experimental results show that the error rate of the calibration is increased when the lip motion is attached to the surface of the HMD without flatness. This is because infrared rays emitted from an infrared light bulb indicate different directions when detecting an infrared image. The present invention can be applied to contents such as a virtual reality using a video-watching HMD and contributes to the construction of a user's convenient interaction environment in the field of virtual reality and augmented reality.

As a result, according to the method and apparatus for calibrating the lip motion between the lip motion and the HMD using the bundle adjustment algorithm of the present invention, The two-dimensional infrared image of the viewpoint can be used to estimate and calibrate the difference between the origin coordinates of the HMD and the lip motion, thereby further enhancing the sense of realism and immersion of the user wearing the HMD.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

100: calibration device 110: pixel matrix generator
120: sparse matrix generator 130: sparse bundle adjuster
140: 2D point estimator 150: blob detector
160: Blubber discriminator 170: Position estimator

Claims (12)

A method of calibrating between a lip motion and an HMD using a calibration device,
Converting each of the n 2-dimensional images acquired at different points of time obtained by capturing the front face of the HMD having the lip motion and the infrared ray lamps into an n number of sparse matrices;
Acquiring n projection matrices by using a single 3-D point group generated by approximating the n sparse matrices in a three-dimensional space through a bundle adjustment algorithm and the n sparse matrices;
Projecting the three-dimensional point cloud onto each of the n two-dimensional images through the n projection matrices to obtain n projected two-dimensional point clouds;
For each of the n two-dimensional images, a plurality of bulb clusters corresponding to the shape of the infrared bulb in the two-dimensional image are detected, and then two-dimensional points located within the bulb cluster among the belonging points in the two- step;
Acquiring a three-dimensional center point of each bulb cluster based on three-dimensional points corresponding to two-dimensional points located within the bulb cluster among the belonging points in the three-dimensional point cloud; And
And estimating and calibrating the origin of the lip motion and the origin of the HMD using the three-dimensional center point and the cluster size of each of the bulb clusters. The method according to claim 1,
Further comprising obtaining n pixel matrices corresponding to the n two-dimensional images,
Wherein transforming into the n &lt; RTI ID = 0.0 &gt; sparse &
And converting the n pixel matrices to the n sparse matrices, respectively. The method according to claim 1,
Wherein determining two-dimensional points located within the bulb cluster comprises:
A plurality of circular clusters of a circular shape are detected through the blob detection technique in the two-dimensional image, and a maximum distance value between the center point of the clusters and the remaining points in the clusters cluster is obtained,
A calibration method between a lip motion and an HMD for determining two-dimensional points that are derived from distances equal to or smaller than the maximum distance value after determining a distance between the belonging points in the two-dimensional point cloud and the center point, . The method according to claim 1,
Wherein the calibrating comprises:
And calibrating movement of the origin coordinates of the lip motion to the origin coordinates of the HMD to calibrate the lip motion. The method according to claim 1,
Wherein acquiring the three-dimensional center point of each bulb cluster comprises:
Wherein the three-dimensional center point of each bulb cluster is calculated by averaging coordinates of three-dimensional points corresponding to two-dimensional points located in the bulb cluster among the three-dimensional point belonging points in the three-dimensional point group. The method according to claim 1,
Wherein the calibrating comprises:
(L < M), the coordinates of the three-dimensional center of gravity of the first clusters are averaged to obtain the first clusters of the first clusters, And estimating the origin coordinates of the HMD by estimating the origin coordinates of the lip motion and averaging the coordinates of the three-dimensional center point with respect to the second clusters. An apparatus for calibrating between lip motion and an HMD,
A sparse matrix generator for converting each of the n 2-dimensional images acquired at different points of time obtained by capturing the front surface of the HMD equipped with the lip motion with the infrared camera into n sparse matrices;
A rare bundle adjuster that obtains n projection matrices using one of a three-dimensional point cloud generated by approximating the n sparse matrices to a three-dimensional space through a bundle adjustment algorithm and the n sparse matrices;
A 2D point estimator for projecting the projected n two-dimensional point clouds by projecting the three-dimensional point cloud onto the n two-dimensional images through the n projection matrices, respectively;
A blob detector for detecting, for each of the n two-dimensional images, a plurality of light bulb clusters corresponding to the shape of the infrared bulb in the two-dimensional image;
A blob discriminator for discriminating two-dimensional points located within the global cluster among the belonging points in the two-dimensional point cloud; And
Acquiring a three-dimensional center point of each bulb cluster based on three-dimensional points corresponding to two-dimensional points located within the bulb cluster among the belonging points in the three-dimensional point cloud group, and using the three-dimensional center point and the cluster size of each bulb cluster And a position estimator for estimating and calibrating the origin of the lip motion and the origin of the HMD. The method of claim 7,
Further comprising a pixel matrix generator for obtaining n pixel matrices corresponding to the n two-dimensional images,
Wherein the sparse matrix generator comprises:
And converting the n pixel matrices to the n sparse matrices, respectively. The method of claim 7,
Wherein the blob detector comprises:
A plurality of circular clusters of a circular shape are detected through the blob detection technique in the two-dimensional image, and a maximum distance value between the center point of the clusters and the remaining points in the clusters cluster is obtained,
Wherein the blob discriminator comprises:
A calibration device between the lip motion and the HMD for determining two-dimensional points derived from the distance less than the maximum distance value as the two-dimensional points located within the global cluster after obtaining the distance between the belonging points in the two- . The method of claim 7,
The position estimating unit may calculate,
And a calibration unit for calibrating the lip motion by moving the origin coordinates of the lip motion to the origin coordinates of the HMD. The method of claim 7,
Wherein the position estimator comprises:
Wherein the three-dimensional center point of each of the bulb clusters is calculated by averaging coordinates of three-dimensional points corresponding to two-dimensional points located in the bulb cluster among the belonging points in the three-dimensional point cloud group. The method of claim 7,
Wherein the position estimator comprises:
(L < M) is divided into L first clusters based on the sizes of the respective bulb clusters and M remaining second clusters smaller than the first cluster,
Estimating the origin coordinates of the lip motion by averaging the coordinates of the three-dimensional center point with respect to the first clusters, and averaging the coordinates of the three-dimensional center point with respect to the second clusters to estimate the origin coordinates of the HMD, And an HMD.

Images