WO2018207388A1 - Program, device and method relating to motion capture - Google Patents

Abstract

[Problem] To provide a motion-capture-related program, device and method capable of creating a three-dimensional video on the basis of a two-dimensional video in which the motion of an object, such as a human, has been captured. [Solution] The present invention includes: a frame image acquisition unit 31 that acquires a prescribed frame image constituting a two-dimensional video; a two-dimensional part identification unit 32 that identifies a part within the frame image; a two-dimensional joint position identification unit 33 that identifies the position of a joint within the frame image; a scale determination unit 34 that determines the scale of a three-dimensional object model; a reference part matching unit 36 that matches the position and direction of a reference part of the three-dimensional object model; a joint position matching unit 37 that matches the joint position in the three-dimensional object model; and a part matching unit 38 that matches, in accordance with the degree of freedom of the joint, the position and direction of a part other than the reference part in the three-dimensional object model.

Description

Program, apparatus and method for motion capture

The present invention relates to a motion capture technique, and more particularly to a motion capture program, apparatus, and method for creating a 3D moving image by converting a 2D moving image related to the operation of an object such as a person into 3D data.

Motion capture technology is to record the movement of an object as three-dimensional data, and is widely used in the sports field, medical field, entertainment field, and the like. For example, in the sports field, movements of athletes are converted into three-dimensional data, and the movements are analyzed in detail, and used for improving technology and preventing injuries. In the medical field, in physical rehabilitation, the movement before and after rehabilitation is converted into three-dimensional data, which is used for selecting an effective rehabilitation method and confirming the progress to rehabilitation. Furthermore, in the entertainment field, the action performed by an actual human being in a movie or a game is converted into three-dimensional data, and a character created by computer graphics is applied to the three-dimensional data, so that the character operates smoothly like a human. It is used to make it.

As for such motion capture technology, for example, in Japanese Patent Application Laid-Open No. 2017-53739, the object is placed at a different angle from the object on which the marker is mounted, and the movement of the marker is measured by the principle of triangulation. There has been proposed a motion capture system including a plurality of cameras that shoot in series and an image processing device that obtains three-dimensional time-series position information of markers from shooting information of the plurality of cameras (Patent Document 1). In addition to this, there is a method of attaching various sensors such as a gyro sensor and an accelerometer to an object and converting the operation into data from numerical values measured by the various sensors.

JP 2017-53739 A

However, in the conventional motion capture technology and method such as the invention described in Patent Document 1, markers and various sensors must be attached to the object. For this reason, it is practically impossible to attach to athletes during competitions in competitions for delicate movements in the world of a few seconds of commas. In addition, in the method of using a plurality of cameras as in the invention described in Patent Document 1, the range that can be photographed is limited, and therefore enormous costs are required for use in outdoor competitions such as ski competitions. Become. Therefore, in the conventional motion capture technology, the movement of the object is limited, and the movement under the condition that the movement of the marker can be easily obtained can be converted into the three-dimensional data, and the movement during the competition is converted into the three-dimensional data. I can't.

By the way, in the sports field, a two-dimensional video is shot with a video camera or the like, and technical analysis and technical guidance are performed while checking the video. However, a single video camera can only obtain information taken from one direction, and even if it is the same posture, it may appear as a different posture depending on the direction in which it is photographed. For example, FIG. 15 shows the same posture of the ski player during the right turn as seen from the front, left front and right front, but as shown in FIGS. 15 (a) and 15 (b), The posture is originally a letter C. However, as shown in FIG. 15C, when viewed from the right front, the posture of the crus seems to be parallel to the left and right. As described above, the posture and motion during the competition may be erroneously grasped only by the two-dimensional moving image taken from one direction. Therefore, if a three-dimensional video of a sports player who is actually competing can be acquired, it is possible to analyze and examine the viewing direction as appropriate, and the utility value is extremely high.

The present invention has been made to solve the above-described problems, and can create a three-dimensional moving image related to the object based on a two-dimensional moving image obtained by photographing an action of an object such as a person. It is an object to provide a motion capture program, a motion capture device, and a motion capture method.

The motion capture device and the motion capture program according to the present invention provide a two-dimensional moving image obtained by photographing the motion of the object in order to solve the problem of creating a three-dimensional moving image related to the object from the two-dimensional moving image obtained by photographing the motion of the object. A motion capture device and a motion capture program for creating a three-dimensional moving image relating to the object using a three-dimensional object model comprising a plurality of parts constituting the object and joints connecting the parts, A frame image acquisition unit for acquiring a predetermined frame image from among a plurality of frame images constituting a three-dimensional moving image, and each part constituting the three-dimensional object model for a two-dimensional object in the frame image A two-dimensional part specifying unit for specifying a corresponding part, a two-dimensional joint position specifying unit for specifying a joint position of the two-dimensional object, and a scale of the three-dimensional object model according to the size of the two-dimensional object. A scale determining unit to be determined and a predetermined part of the two-dimensional object specified by the two-dimensional part specifying unit are overlapped with a reference part of the three-dimensional object model to match the position and direction of the reference part. A joint position for matching the position of the joint in the three-dimensional object model after aligning the reference part with the joint of the reference part matching unit and the joint of the two-dimensional object specified by the two-dimensional joint position specifying unit A matching unit and the reference in the two-dimensional object Parts that align the position and direction of parts other than the reference part in the three-dimensional object model after matching the reference part with each part other than the part corresponding to the part according to the degree of freedom of the joint And a matching portion.

Further, as one aspect of the present invention, in order to solve the problem of shortening the time for matching a three-dimensional object model to a two-dimensional object in a frame image and improving the accuracy of matching, the basics related to the operation of the object A model acquisition unit that acquires a 3D object model to be matched with the 2D object in the frame image from a 3D object model storage unit in which a plurality of 3D object models set as postures are stored; Also good.

Furthermore, as one aspect of the present invention, in order to solve the problem of shortening the time for matching the three-dimensional object model to the two-dimensional object in the frame image and improving the accuracy of the matching, the object is a person or an animal. In this case, in the reference part matching unit, the pelvis of the three-dimensional object model may be set as the reference part.

Further, as one aspect of the present invention, in order to solve the problem of shortening the time for matching a three-dimensional object model to a two-dimensional object in a frame image and improving the accuracy of matching, the object is a person or an animal. In this case, the head of the three-dimensional object model may be set as the reference part in the reference part matching unit.

Furthermore, as one aspect of the present invention, in order to solve the problem of shortening the time for matching the three-dimensional object model to the two-dimensional object in the frame image and improving the accuracy of the matching, the object is equipped with a tool. In this case, in the reference part matching unit, the tool of the three-dimensional object model may be set as a reference part.

Further, as one aspect of the present invention, in order to solve the problem of shortening the time for matching the three-dimensional object model with the two-dimensional object in the frame image and improving the accuracy of the matching, the object is equipped with a tool. In this case, the scale determining unit may determine the scale according to the size of the tool.

In order to solve the problem of creating a three-dimensional moving image related to the object from a two-dimensional moving image obtained by photographing the motion of the object, the motion capture method according to the present invention is based on the two-dimensional moving image obtained by photographing the motion of the object. A motion capture method for creating a three-dimensional video concerning the object using a three-dimensional object model comprising a plurality of parts constituting an object and joints connecting the parts, and a plurality of frames constituting the two-dimensional video A frame image acquisition step of acquiring a predetermined frame image of the image, and a two-dimensional part specifying step of specifying a part corresponding to each part constituting the three-dimensional object model for the two-dimensional object in the frame image And the two-dimensional object Two-dimensional joint position specifying step for specifying the joint position of the project, a scale determining step for determining the scale of the three-dimensional object model in accordance with the size of the two-dimensional object, and the two-dimensional part specifying step. Further, the predetermined part of the two-dimensional object is specified by the reference part aligning step in which the reference part of the three-dimensional object model is overlapped and the position and direction of the reference part are aligned and specified by the two-dimensional joint position specifying step. A joint position matching step for matching the position of the joint in the three-dimensional object model after aligning the reference part with the joint of the two-dimensional object, and a part corresponding to the reference part in the two-dimensional object Less than For each part of, and a part matching step of matching in accordance with the position and orientation of the part other than the reference part in the three-dimensional object model after matching the reference part to the flexibility of the joint.

According to the present invention, it is possible to create a three-dimensional moving image related to the object based on the two-dimensional moving image obtained by photographing the motion of the object.

It is a block diagram which shows one Embodiment of the motion capture apparatus which concerns on this invention. It is an image which shows the two-dimensional object in one frame image which comprises a two-dimensional moving image in this embodiment. It is a computer graphic figure which shows an example of the name of the three-dimensional object model in this embodiment, and each part which comprises this. It is a schematic diagram which shows the statistical value of the Western body data used in order to set the length ratio of each part of a ski player model in this embodiment. It is a computer graphic figure which shows the name of each joint of the three-dimensional object model in this embodiment. It is a mimetic diagram showing a basic posture in one turn of ski competition. It is a figure which shows an example of the various three-dimensional object models set as a basic posture in the left turn in this embodiment. It is a figure which shows an example of the various three-dimensional object model set as a basic posture in the right turn in this embodiment. It is a flowchart which shows the process process of the motion capture method of this embodiment. It is a perspective view which shows the three-dimensional object model of the upright posture in this embodiment, and its reference | standard part. It is an image which shows the state which overlap | superposed the reference | standard part (pelvis) of the three-dimensional object model in this embodiment on the two-dimensional object in a frame image. It is an image which shows the state which piled up the thigh of the three-dimensional object model in this embodiment on the two-dimensional object in a frame image. It is an image which shows the state which all the parts of the three-dimensional object model in this embodiment overlapped with the two-dimensional object in a frame image, and the attitude | position of the three-dimensional object model. It is an image which shows the example of the three-dimensional moving image which arranged the state which looked at the matched three-dimensional object model in this embodiment from the back in time series. It is a schematic diagram which shows the state which looked at the same attitude | position in the right turn of a ski player from (a) front, (b) left front, and (c) right front.

Hereinafter, an embodiment of a motion capture program, a motion capture device, and a motion capture method according to the present invention will be described with reference to the drawings.

The motion capture device 1 of the present embodiment is a device for creating a three-dimensional moving image related to the object based on a two-dimensional moving image obtained by photographing the motion of the object. As shown in FIG. 1, the motion capture program 1a is mainly used. And storage means 2 for storing various data, and arithmetic processing means 3 for acquiring various data from the storage means 2 and executing arithmetic processing. Details of each component will be described below.

The storage means 2 is composed of a ROM, a RAM, a hard disk, a flash memory, etc., and stores various data and functions as a working area when the arithmetic processing means 3 executes arithmetic operations. As shown in FIG. 1, the storage means 2 in this embodiment mainly includes a program storage unit 21 that stores a motion capture program 1a, and a two-dimensional moving image storage unit 22 that stores a two-dimensional moving image obtained by photographing the motion of an object. And a three-dimensional object model storage unit 23 for storing a plurality of three-dimensional object models set as basic postures related to the motion of the object, and a three-dimensional object model matched with a two-dimensional moving image are created in time series. And a three-dimensional moving image storage unit 24 that stores the three-dimensional moving image.

The motion capture program 1a of this embodiment is installed in the program storage unit 21. The arithmetic processing unit 3 executes the motion capture program 1a and functions as each component of the arithmetic processing unit 3 described later, thereby causing the computer to function as the motion capture device 1.

Note that the usage form of the motion capture program 1a is not limited to the above configuration. For example, the motion capture program 1a may be stored in a computer-readable non-transitory recording medium such as a CD-ROM or a USB memory, and read directly from the recording medium and executed. Moreover, you may utilize from an external server etc. by a cloud computing system, ASP (application | service | provider) system, etc.

The two-dimensional moving image storage unit 22 stores a two-dimensional moving image obtained by photographing the motion of the object. In this embodiment, a plurality of frame images constituting the two-dimensional moving image are stored. Specifically, for example, a frame image of 30 frames per second is stored for a two-dimensional video image of a ski competition. The frame image includes a two-dimensional object obtained by photographing the object. Here, the two-dimensional object is an object for creating a three-dimensional moving image in the frame image. Examples of objects include various objects such as a person, an animal, a robot, a character, and a tool. In the present embodiment, as shown in FIG. 2, the ski player in the frame image and the part of the ski that the ski player wears correspond to the two-dimensional object in the frame image.

The three-dimensional object model storage unit 23 stores a three-dimensional object model for matching with a two-dimensional object in the frame image. In this embodiment, a plurality of sets set as basic postures related to the motion of the object are stored. A three-dimensional object model is stored.

Here, the 3D object model will be described. The three-dimensional object model is a three-dimensional model for matching with a two-dimensional object in the frame image. This three-dimensional object model is created by CAD software, rigid body modeling software used for computer graphics, or the like. The 3D object model of the present embodiment is attached to the ski player model and the ski player as shown in FIG. 3 in order to match the ski player and the ski in the frame image of the 2D video of the ski competition. And a ski model.

The ski player model has a head, torso, pelvis, left upper arm, left forearm, left hand, right upper arm, right forearm, right hand, left thigh, left lower leg, left foot, right according to the main composition of the person. It consists of a total of 15 parts including the thigh, right lower leg, and right foot.

In this embodiment, in order to match the two-dimensional video of a plurality of ski players, the length ratio of each part in the ski player model is as shown in FIG. , “Body Segment Parameters, Art Part II”, Artificial Limbs 16, (1972), pp. 1-19.). In addition, when the object to match is specified, the length of each part and its ratio may be set based on the data of the object, for example, in the case of a ski player, the actual body shape of the player.

Also, in the 3D object model, each part is connected by a joint, and each joint has a degree of freedom indicating the direction of bending. For example, in a ski player model, as shown in FIG. 5, a neck joint that connects the head and the torso, a shoulder joint that connects the torso and the left and right upper arms, an elbow joint that connects the upper arm and the forearm, A wrist joint that connects the hand part, a hip joint that connects the torso and the left and right thighs, a knee joint that connects the thigh and the lower leg, and an ankle joint that connects the lower leg and the foot part.

Also, the degree of freedom of each joint is set based on the degree of freedom of human joints. Specifically, as shown in Table 1, a degree of freedom 3 is set for the neck joint, and the joint can be moved in the front-rear direction, the left-right direction, and the rotation direction. Similarly, the degree of freedom 3 is set so that the shoulder joint and the hip joint can be moved in the front-rear direction, the left-right direction, and the rotation direction. The wrist joint is set to have a degree of freedom of 3 by combining the degree of freedom of the wrist joint of 2 and the degree of freedom of pronation / extraction of the forearm. Furthermore, since the elbow joint and the knee joint are only bent and stretched, a degree of freedom of 1 is set. On the other hand, since the human ankle joint can be moved in the front-rear direction and the left-right direction, the degree of freedom is 2, but ski players are restricted from moving in the left-right direction because the ankle joint is covered with ski boots. Limited to direction bending. Therefore, the degree of freedom of the ankle joint of the ski player model in this embodiment is set to 1 degree of freedom. As described above, the degree of freedom of the joint can be freely set in accordance with a tool worn by the object, restrictions on the movement of the object, and the like.

[Table 1]

Further, as shown in FIG. 3, the ski model is composed of eight parts divided in the length direction and one part related to the binding plate, and a total of 18 parts from the left and right pairs. It consists of individual parts. The length of the ski model is set based on the FIS (International Ski Federation) alpine ski equipment standard in order to match the two-dimensional moving images of a plurality of ski players. Note that the length of the equipment worn by the object, such as a ski model, is not limited to that set based on the standard, but the target data, for example, the actual ski equipment worn by the ski player. You may set based on a length dimension.

In addition, each part of the ski model is connected by a joint, and a degree of freedom 1 is set for the direction of warping so that the front side or rear end side of the ski can bend up and down. Yes.

Furthermore, since the ski part of the ski model and the binding plate, and the binding plate and the foot part of the ski player model are fixed in the two-dimensional model, the degree of freedom of these joints in the three-dimensional object model is free. The degree is set to zero.

Further, in the three-dimensional object model storage unit 23 in the present embodiment, a plurality of three-dimensional object models set as basic postures related to the movement of the object are stored. In the case of ski competition, in one turn, as shown in FIG. 6, a neutral posture at the time of switching the turn, an edging start posture for starting an operation called edging that raises the edge of the ski on the slope, There are a turn-maximum posture in which the angle at which the edge is raised and the angle at which the body is inclined in the turn direction is the largest, and an etching end posture in which the edging is finished. Therefore, in the present embodiment, various three-dimensional object models are set as shown in FIGS. 7 and 8 with these turn postures as basic postures. Note that the basic postures of the plurality of three-dimensional object models shown in FIGS. 7 and 8 are examples, and a plurality of basic postures that are intermediate postures are set between the basic postures.

The 3D moving image storage unit 24 stores a 3D moving image created as data capable of displaying a 3D object model matched for each frame image in the same time series order as the frame image. The three-dimensional moving image storage unit 24 in the present embodiment stores a predetermined joint angle in the three-dimensional object model in the same time series as the order of the frame images together with the three-dimensional moving image.

Next, the arithmetic processing means 3 will be described. The arithmetic processing means 3 is composed of a CPU (Central Processing Unit) and the like, and by executing a motion capture program 1a installed in the storage means 2, as shown in FIG. Frame image acquisition unit 31, 2D part specification unit 32, 2D joint position specification unit 33, model acquisition unit 34, scale determination unit 35, reference part alignment unit 36, joint position alignment unit 37, part alignment unit 38, and It is made to function as the three-dimensional moving image creation unit 39.

The frame image acquisition unit 31 acquires a predetermined frame image from among a plurality of frame images constituting a two-dimensional moving image. In this embodiment, the frame image acquisition unit 31 stores two frame images stored in the two-dimensional moving image storage unit 22. Acquired according to the time series of dimensional animation.

The two-dimensional part specifying unit 32 specifies a part corresponding to each part constituting the three-dimensional object model for a two-dimensional object in the frame image. In this embodiment, which part in the frame image is the head, torso, pelvis, left upper arm, left forearm, left hand, right upper arm, right forearm, right hand, left thigh, left lower leg, left foot, right thigh, The right lower leg, the right leg, and the ski part are identified and identified. The two-dimensional part specifying unit 32 in the present embodiment specifies each part using an image recognition technique using deep learning, an image recognition technique used for face authentication, and the like, and a part (range) related to each specified part. The position coordinates in the frame image showing are obtained. Thereby, the data regarding the position and magnitude | size in the frame image of each part can be obtained.

The two-dimensional joint position specifying unit 33 specifies the position of the joint of the two-dimensional object in the frame image. The two-dimensional joint position specifying unit 33 in the present embodiment, like the two-dimensional part specifying unit 32, uses the image recognition technology by deep learning, the image recognition technology used for face authentication, or the like to calculate the position coordinates of the joint of the two-dimensional object. To get. The method for specifying the joint position is not limited to the image recognition technique, and for example, the boundary of each part specified by the two-dimensional part specifying unit 32 may be specified as the joint position.

The model acquisition unit 34 acquires a 3D object model to be matched with the 2D object in the frame image from the 3D object model storage unit 23. In addition, the model acquisition unit 34 according to the present embodiment has a plurality of three-dimensional object models set as basic postures related to the movement of the object, as shown in FIGS. 7 and 8, in order to shorten the matching time and improve the matching accuracy. Thus, a three-dimensional object model suitable for matching with a two-dimensional object in a frame image is discriminated using, for example, and acquired. At this time, AI (artificial intelligence) technology based on an evolutionary algorithm (Evolutionary Algorithm: EA) that acquires an optimal solution that cannot be obtained analytically as a converged solution by trial and error is used. The evolutionary algorithm is one of algorithms that can automatically match each part in the same process as a human manual matching without any special theoretical background. Genetic Algorithm (GA), Genetic Programming (GP), Evolution Strategy (ES), Evolutionary Programming (EP), etc. are available.

A method for determining whether or not the two-dimensional object is most suitable for matching is not particularly limited. For example, the two-dimensional object portion and other portions are binarized from the frame image, and the two-dimensional object is obtained. The area value calculated from the part and the area value projected from the predetermined direction of the three-dimensional object model may be compared to determine that the one having a small difference value is suitable. Or you may discriminate | determine from the difference value of the distance of the predetermined parts in a two-dimensional object, and the distance of the corresponding parts of a three-dimensional object model. Alternatively, when the obtained three-dimensional object model is matched with the two-dimensional object of the frame image, if the movement amount or movement angle of a predetermined part in the three-dimensional object model exceeds a predetermined value, a different three-dimensional The object model may be reacquired. Further, the three-dimensional object model used for the previous matching may be used as it is. In addition, when the processing time related to the matching when the three-dimensional object model is used becomes longer than a predetermined time, the three-dimensional object model related to the next basic posture is acquired according to a predetermined order. You may do it.

The scale determining unit 35 determines the scale of the 3D object model according to the overall size of the 2D object or the size of an arbitrarily selected part. In this embodiment, since the object is a ski player related to a person, the size of the pelvis is calculated from the position coordinates related to the pelvis part specified by the two-dimensional part specifying unit 32, and the pelvis of the three-dimensional object model is The scale of the three-dimensional object model is determined according to the size.

It should be noted that the reference parts for determining the scale of the three-dimensional object model are not limited to a part of a person such as a pelvis or an animal, and may be appropriately selected according to the characteristics of the object. For example, when the object is wearing a tool, the scale may be determined according to the size of the tool. In the present embodiment, as described above, the ski player wears the ski on the foot, and the size of the ski has a substantially constant size based on the FIS standard. By using the size as a reference, it is expected to improve the accuracy of the scale. In addition, in order to match the entire size of the two-dimensional object, the two-dimensional object portion and other portions are binarized from the frame image, and the binarized three-dimensional object is adjusted to the entire size. The scale of the dimensional object model may be determined.

The reference part matching unit 36 superimposes the reference part of the three-dimensional object model on the predetermined part of the two-dimensional object specified by the two-dimensional part specifying unit 32 by using AI technology or the like based on an evolutionary algorithm. This aligns the position and direction of the reference part. In this embodiment, a pelvis of a three-dimensional object model is adopted as a reference part. The pelvis is the central part of human or animal movement, and is located at the center of the body, easy to identify in image recognition technology, and has less twisting and tilting than other parts. Therefore, it is possible to reduce the matching time and improve the matching accuracy.

Note that the reference part is not limited to the pelvis, and may be appropriately selected according to the feature of the part or the feature of the operation of the object. For example, since the head is a spherical part that is characteristic in each part of a person, it is easier to identify by the image recognition technique than other parts. Also, since the head has eyes, nose, mouth and the like, the direction can be easily adjusted. Therefore, shortening the alignment time and improving the alignment accuracy can be expected even if the head is used as the reference part. Further, when the object is equipped with a tool, the tool may be used as a reference part. This is because the tool can be easily distinguished from a person or the like by color or shape, and can be easily specified by an image recognition technique. Further, a long tool such as a ski board or a golf club or a tool having a striking surface such as a tennis racket can easily determine the inclination angle and the direction of the inclined surface. Therefore, shortening the alignment time and improving the alignment accuracy can be expected even when the tool is used as a reference part.

The joint position matching unit 37 matches the position of the joint in the three-dimensional object model after matching the reference part with each joint of the two-dimensional object specified by the two-dimensional joint position specifying unit 33. is there. The joint position matching unit 37 according to the present embodiment is configured to match each joint position constituting the three-dimensional object model with respect to the joint position coordinates of the two-dimensional object.

The part aligning unit 38 uses the AI technique based on an evolutionary algorithm or the like to set the position and direction of parts other than the reference part in the 3D object model after aligning the reference part to the 2D object. Matching according to the degree of freedom. In the present embodiment, the part aligning unit 38 is a reference part aligning unit 36 or another part connected to a part whose position and direction are aligned by the part aligning unit 38 according to the degree of freedom in the joint that connects them. The position and direction of the parts are aligned by bending. In other words, since the part alignment unit 38 aligns the position and direction of each part under the condition of the degree of freedom of the joint, it is possible to prevent the 3D object model from taking a posture that cannot be a real object.

The 3D video creation unit 39 creates a 3D video as data that can be continuously displayed in the same time series as the order of the frame images of the 3D object model matched for each frame image. The 3D video creation unit 39 in this embodiment creates a video of the 3D object model based on the viewpoint by arbitrarily setting the viewpoint for the 3D object model, and stores it in the 3D video storage unit 24. It has become. Further, the three-dimensional moving image creation unit 39 acquires the predetermined joint angles in the three-dimensional object model in the same time series as the order of the frame images, and stores them in the three-dimensional moving image storage unit 24.

Next, the operation of each component in the motion capture program 1a and the motion capture device 1 according to this embodiment will be described together with a motion capture method.

In the motion capture method of this embodiment, as shown in FIG. 9, a predetermined frame image is acquired from a plurality of frame images constituting a two-dimensional moving image (frame image acquisition step S1). In the present embodiment, the frame image acquisition unit 31 acquires a plurality of frame images stored in the two-dimensional moving image storage unit 22 in chronological order at appropriate intervals.

Then, a part corresponding to each part constituting the three-dimensional object model is specified for the two-dimensional object in the acquired frame image (two-dimensional part specifying step S2). In the present embodiment, the two-dimensional part specifying unit 32 determines which part (range) of the frame image corresponds to which part of each part constituting the three-dimensional object model by image recognition technology such as deep learning. Identify and get the position coordinates of that part.

Then, the position of the joint of the two-dimensional object is specified for the two-dimensional object in the acquired frame image (two-dimensional joint position specifying step S3). In the present embodiment, the two-dimensional joint position specifying unit 33 specifies which part of the frame image corresponds to the position of the joint by an image recognition technique using deep learning or the like, and acquires the position coordinates of the position.

Next, the 3D object model is matched with the acquired frame image. Specifically, a three-dimensional object model to be matched with the two-dimensional object in the frame image is acquired (model acquisition step S4). In the present embodiment, the model acquisition unit 34 is stored in the three-dimensional object model storage unit 23 based on a difference value between the binarized two-dimensional object and the projected area of the three-dimensional object model from a predetermined direction. A three-dimensional object model set as a basic posture most suitable for matching with a two-dimensional object in a frame image is acquired from the plurality of three-dimensional object models. In this way, by using a three-dimensional object model that is close to the posture of the two-dimensional object in the frame image, it is possible to shorten the time for matching and improve the matching accuracy.

In the following, in order to make it easy to understand the change of each part of the 3D object model before and after the alignment, the 3D object model most suitable for alignment with the 2D object is upright as shown in FIG. A case where the posture is a three-dimensional object model will be described as an example.

First, the scale of the 3D object model is determined in accordance with the size of the 2D object (scale determination step S5). In the present embodiment, the scale determining unit 35 calculates the size of the pelvis on the frame screen from the position coordinates of a predetermined portion in the frame image identified as the pelvis by the two-dimensional part identifying unit 32, and calculates the three-dimensional object model. The scale of the pelvis is determined by matching the size of the pelvis with the calculated size of the pelvis.

Next, the reference part of the three-dimensional object model is overlapped with the two-dimensional object to align the position and direction of the reference part (reference part alignment step S6). In this embodiment, as shown in FIG. 10, the pelvis of the three-dimensional object model is used as a reference part. Therefore, the reference part matching unit 36 uses the AI technique based on the evolutionary algorithm to superimpose the pelvis of the three-dimensional object model on the part specified by the two-dimensional part specifying unit 32 as the pelvis as shown in FIG. To align the position and direction. The pelvis is easy to identify in image recognition technology and the like, and is less twisted and tilted than other parts, so that the time required for alignment can be shortened and the accuracy of alignment can be improved.

When the alignment process of the reference parts is completed, the positions of the joints in the 3D object model after the alignment of the reference parts are then aligned with each identified joint of the 2D object (joint position alignment step S7). ). In the present embodiment, the joint position matching unit 37 matches the position of the knee joint of the thigh connected to the reference part, for example, in accordance with the position coordinate of the knee joint specified by the two-dimensional joint position specifying unit 33.

Then, for each identified part of the two-dimensional object, the position and direction of parts other than the reference part in the three-dimensional object model after matching the reference part are matched according to the degree of freedom of the joint (parts). Matching step S8). In the present embodiment, the part aligning unit 38 is connected to the pelvis whose position and direction are aligned by the reference part aligning unit 36, as shown in FIG. 12, using AI technology or the like based on an evolutionary algorithm. Are bent under conditions of 3 degrees of freedom at the hip joint connecting them to align the position and orientation of the thighs. Here, since the thigh is bent under the condition of the degree of freedom of the hip joint, it does not assume a posture that cannot be taken by actual joint movement.

Then, the parts matching unit 38 determines whether or not all parts matching processing has been completed (parts matching end determination step S9). Here, when the alignment process for all parts is not completed (S9: NO), the process returns to the joint position alignment step S6, and the processes are repeated until the alignment process for all parts is completed. When the alignment process for all parts is completed (S9: YES), the alignment process for one frame image is completed. Thereby, as shown in FIG. 13, the attitude | position of the three-dimensional object model with respect to the two-dimensional object of the acquired frame image is determined.

Next, the frame image acquisition unit 31 determines whether or not the alignment process for all the frame images has been completed (all frame alignment end determination step S10). Here, when the matching process of all the frame images is not completed (S10: NO), the process returns to the frame image acquisition step S1, and the process is repeated until the matching of all the frame images is completed. If all the frame images have been matched (S10: YES), the matching process for the two-dimensional moving image is finished.

When the matching processing of all the frame images with respect to the 2D video is completed, the 3D video creation unit 39 creates a 3D video that can be displayed by arranging the 3D object models matched for each frame image in time series. And it memorize | stores in the three-dimensional moving image memory | storage part 24 (three-dimensional moving image creation step S11). For example, if the viewpoint of the three-dimensional video is set as seen from the back of the ski player, as shown in FIG. 14, a series of actions of making a left turn from the neutral posture and then making a right turn from the neutral posture is followed from behind. Can be viewed as a video. Therefore, if you can create a 3D video viewed from the back of a player who performs exemplary actions, it will be possible to check the quality of one's posture while referring to the turn posture, which can be one of useful training .

Finally, the three-dimensional video creation unit 39 acquires the predetermined joint angles in the three-dimensional object model in the same time series as the order of the frame images, and stores them in the three-dimensional video storage unit 24 (joint angle acquisition step S12). ). For example, time-series data regarding the hip joint and knee joint angles of the three-dimensional object model is acquired. This makes it possible to calculate not only the overall posture of the player, but also the amount of change in angle, the amount of work, etc., required for bending and stretching the lower limbs composed of the thighs and lower legs. Therefore, if time series data about the angle of a predetermined joint of a player who performs an exemplary motion can be acquired, it becomes possible to analyze the skill required for bending and stretching the joint, and strength training planned based on the analysis By carrying out endurance training, etc., it is possible to efficiently improve the skills of the players.

According to the present embodiment as described above, the following effects can be obtained.
1. By superimposing a three-dimensional object model on each frame image constituting a two-dimensional moving image in which the motion of the object is photographed, the posture of the object can be specified, and based on this, a three-dimensional moving image related to the object can be specified. Can be created.
2. When aligning a 3D object model to a 2D object in a frame image, each joint is set to an appropriate degree of freedom depending on the object, so there is actually a 3D object model for that object. It is possible to prevent a posture that cannot be obtained.
3. By providing various three-dimensional object models set as basic postures related to the motion of the object, it is possible to shorten the time for matching the three-dimensional object model with the two-dimensional object in the frame image.
4. By appropriately setting the reference parts used when aligning the three-dimensional object model according to the characteristics of the parts and the characteristics of the operation of the object, the alignment time can be shortened and the alignment accuracy can be improved.

Note that the motion capture program, the motion capture device, and the motion capture method according to the present invention are not limited to the above-described embodiments, and can be changed as appropriate. For example, in the present embodiment, a ski competition in the sports field is targeted, but the object of the present invention is not limited to the sports field, and may be targeted in the medical field, the entertainment field, and the like.

DESCRIPTION OF SYMBOLS 1 Motion capture apparatus 1a Motion capture program 2 Memory | storage means 3 Operation processing means 21 Program memory | storage part 22 2D moving image memory | storage part 23 3D object model memory | storage part 24 3D moving image memory | storage part 31 Frame image acquisition part 32 2D part specification part 33 2D joint position specifying unit 34 Model acquisition unit 35 Scale determining unit 36 Reference parts alignment unit 37 Joint position alignment unit 38 Parts alignment unit 39 3D animation generation unit

Claims (8)

A motion capture program for creating a three-dimensional moving image related to an object using a three-dimensional object model composed of a plurality of parts constituting the object and a joint connecting the parts based on a two-dimensional moving image obtained by photographing the movement of the object. There,
A frame image acquisition unit that acquires a predetermined frame image among a plurality of frame images constituting the two-dimensional moving image;
A two-dimensional part specifying unit for specifying a part corresponding to each part constituting the three-dimensional object model for the two-dimensional object in the frame image;
A two-dimensional joint position specifying unit for specifying the position of the joint of the two-dimensional object;
A scale determining unit that determines the scale of the three-dimensional object model according to the size of the two-dimensional object;
A reference part matching unit that superimposes a reference part of the three-dimensional object model on the predetermined part of the two-dimensional object specified by the two-dimensional part specifying unit and aligns the position and direction of the reference part;
A joint position matching unit for matching the position of the joint in the three-dimensional object model after matching the reference part with the joint of the two-dimensional object specified by the two-dimensional joint position specifying unit;
For each part other than the part corresponding to the reference part in the two-dimensional object, the position and direction of the part other than the reference part in the three-dimensional object model after aligning the reference part is the degree of freedom of the joint. The motion capture program for causing a computer to function as a part aligning unit that performs matching according to the above. Model acquisition for acquiring a three-dimensional object model to be matched with the two-dimensional object in the frame image from a three-dimensional object model storage unit in which a plurality of three-dimensional object models set as basic postures related to the motion of the object are stored The motion capture program according to claim 1, which causes a computer to function as a unit. 3. The motion capture program according to claim 1, wherein, when the object is a person or an animal, the pelvis of the three-dimensional object model is set as the reference part in the reference part matching unit. 3. The motion capture program according to claim 1, wherein when the object is a person or an animal, the reference part matching unit sets a head of the three-dimensional object model as the reference part. 3. The motion capture program according to claim 1, wherein when the object is equipped with a tool, the reference part matching unit sets the tool of the three-dimensional object model as a reference part. The motion capture program according to any one of claims 1 to 5, wherein when the object is equipped with a tool, the scale determining unit determines the scale according to the size of the tool. A motion capture method for creating a three-dimensional video concerning the object using a three-dimensional object model comprising a plurality of parts constituting the object and a joint connecting the parts based on a two-dimensional video obtained by photographing the motion of the object Because
A frame image acquisition step of acquiring a predetermined frame image among a plurality of frame images constituting the two-dimensional moving image;
A two-dimensional part specifying step for specifying a part corresponding to each part constituting the three-dimensional object model for the two-dimensional object in the frame image;
A two-dimensional joint position specifying step of specifying a joint position of the two-dimensional object;
A scale determining step for determining a scale of the three-dimensional object model in accordance with the size of the two-dimensional object;
A reference part alignment step of aligning a reference part of the three-dimensional object model with a predetermined part of the two-dimensional object specified by the two-dimensional part specification step and aligning the position and direction of the reference part;
A joint position matching step of matching the position of the joint in the three-dimensional object model after matching the reference part with the joint of the two-dimensional object specified by the two-dimensional joint position specifying step;
For each part other than the part corresponding to the reference part in the two-dimensional object, the position and direction of the part other than the reference part in the three-dimensional object model after aligning the reference part is the degree of freedom of the joint. And a part aligning step for aligning according to A motion capture device that creates a three-dimensional moving image related to an object using a three-dimensional object model composed of a plurality of parts constituting the object and a joint connecting the parts based on a two-dimensional moving image obtained by photographing the movement of the object. There,
A frame image acquisition unit that acquires a predetermined frame image among a plurality of frame images constituting the two-dimensional moving image;
A two-dimensional part specifying unit for specifying a part corresponding to each part constituting the three-dimensional object model for the two-dimensional object in the frame image;
A two-dimensional joint position specifying unit for specifying the position of the joint of the two-dimensional object;
A scale determining unit that determines the scale of the three-dimensional object model according to the size of the two-dimensional object;
A reference part matching unit that superimposes a reference part of the three-dimensional object model on the predetermined part of the two-dimensional object specified by the two-dimensional part specifying unit and aligns the position and direction of the reference part;
A joint position matching unit for matching the position of the joint in the three-dimensional object model after matching the reference part with the joint of the two-dimensional object specified by the two-dimensional joint position specifying unit;
For each part other than the part corresponding to the reference part in the two-dimensional object, the position and direction of the part other than the reference part in the three-dimensional object model after aligning the reference part is the degree of freedom of the joint. The motion capture device comprising: a part aligning unit that aligns according to

Images