KR20200092893A - Augmented reality video production system and method using 3d scan data - Google Patents

Abstract

According to an embodiment of the present invention, loading 3D scan data including bone data for a plurality of joints as scan data generated by 3D (dimension) scanning a character having a plurality of joints; Assigning a plurality of movements to the character in a virtual space using the present data, and disposing each movement of the character in a unit of a clip in a first layer; Applying motion blending to the plurality of motions so that images of adjacent clips are continuously connected; Disposing a background image of the virtual space on a second layer; Setting a virtual camera for photographing the virtual space, and applying a view point of the camera to the character and the background image; And generating a temporal change in the virtual space as a video according to the viewpoint of the camera.

Description

Augmented Reality Video Production System and Method Using 3D Scan Data {AUGMENTED REALITY VIDEO PRODUCTION SYSTEM AND METHOD USING 3D SCAN DATA}

This embodiment relates to a video production technology.

Recently, as a video sharing site or social network service (SNS) has become popular, a lot of novel and unique videos have been made. These videos are mainly produced on the basis of direct directing and shooting by the user, or are produced by editing content created by others. However, such a video production method based on production/editing based on directing/shooting is a method that requires a lot of time or financial investment by a user and is difficult to use by the general public. For example, in order to produce a video based on directing/shooting, a financial investment in the camera is required, and a shooting activity (temporal investment) involving a lot of trial and error in a specific place (directing space) is required. In addition, the video production method based on editing also requires expensive financial tools because it requires expensive editing tools. In particular, this method is more difficult to use by the general public in that it requires a high level of skill to handle the editing tools. . In order to induce more active participation of the non-professional people in the recent video sharing site or social network service, it is required to develop a new technology that solves the aforementioned problems of the conventional video production method.

On the other hand, the conventional video production method is a method of producing a video using other people as the main character, not the user (producer). For example, in the case of producing a video based on directing/shooting, the user in charge of directing/shooting produces a video by inviting others to be in charge of the central character and filming the acting or activities of others. The central character becomes another person. The method of setting the other person as the central character has a problem in that it is difficult to produce a video if the user (producer) and the other person's breathing do not match well, but the desire to produce the video to a large number of users who want to appear as the central character There are aspects that are insufficient to meet. When the user himself becomes the central character of the video and the central character is easily controlled by the user's manipulation, many users who want to express themselves may be involved in video production.

The user himself can be digitalized by photographing the digital imaging device. For example, the user himself may be inserted as a character in a still picture or video through shooting by a digital camera. However, since the character generated by the digital camera only contains planar information about the user, it is not easy to deform-for example, rotate, etc., and it is not easy to manipulate the motion. It is difficult to utilize in the method of making. On the other hand, a 3D (dimension) scanning device may be much more useful for producing a video having a user as a central character, in terms of generating 3D scan data including stereoscopic information about a user. However, recent technologies utilizing 3D scanning devices have limited scope of use, such as simply producing 3D figures by combining 3D scan data with 3D printers, and satisfying the user's desire for the above-described video production method. Has not reached.

Against this background, the purpose of this embodiment is to provide a technique for a user to easily produce a video using 3D scanned data.

In order to achieve the above object, an embodiment is a scan data generated by 3D (dimension) scanning of a character having a plurality of joints, and includes bone data for the plurality of joints Loading 3D scan data; Assigning a plurality of movements to the character in a virtual space using the present data, and disposing each movement of the character in a unit of a clip in a first layer; Applying motion blending to the plurality of motions so that images of adjacent clips are continuously connected; Disposing a background image of the virtual space on a second layer; Setting a virtual camera for photographing the virtual space, and applying a view point of the camera to the character and the background image; And generating a temporal change in the virtual space as a video according to the viewpoint of the camera.

In the video production method, in the step of arranging each motion of the character in the first layer, the clip is set so that one motion for the character is completed for each clip, but the motion of the motion is determined according to the length of the clip. Slow or fast motion can be achieved by adjusting the speed.

The video production method may further include the step of arranging the motions of the effective objects for the virtual space on the third layer as a peripheral area of the character.

In the video production method, in the step of applying motion blending to the plurality of motions, in two adjacent clips, the motion of each clip can be controlled such that the finish posture of the previous clip of the character and the start posture of the next clip match. have.

In the video production method, in the step of applying the viewpoint of the camera, when the camera is set from the first time to the first position and from the second time to the second position, the first time to the second time Meanwhile, the position of the camera may be gradually changed from the first position to the second position.

In the video production method, in the step of generating the video, the video may be generated by further combining sound.

The video production method comprises: after the step of generating the video, synthesizing the character into a real-time image being shot by a real camera; Causing the character to continuously take the plurality of actions in the real-time image; The method may further include generating an augmented reality video of the augmented reality image in which the character continuously takes the plurality of motions in the real time image.

The video production method may further include, before the step of generating the augmented reality video, synthesizing a specific image in a background portion having a chroma key color among the background of the real-time video.

The video production method comprises: after the step of generating the video, synthesizing the character and one or more separate characters separate from the character into a real-time video being taken by a real camera; Causing the character and the one or more separate characters to continuously take the plurality of operations in the real-time image; The method may further include generating an augmented reality video for the augmented reality image in which the character and the one or more separate characters successively perform the plurality of operations in the real-time image.

Another embodiment includes a 3D scanner that scans a character having a plurality of joints in 3D to generate scan row data; A first server that processes the scan row data and generates 3D scan data including bone data for the plurality of joints; And downloading the 3D scan data from the server, and using the original data of the 3D scan data to give a plurality of actions to the character in a virtual space, and according to the viewpoint of the camera shooting the virtual space. Provided is a video production system including a terminal device that generates a temporal change of a virtual space as a video.

In the video production system, a plurality of layers are displayed on the screen of the terminal device to be lined up in the time axis direction, and each action of the character is arranged in units of clips in the first layer, and the virtual in the second layer. The background image of the space is arranged, and motions of effective objects for the virtual space may be arranged on the third layer.

In the video production system, the movement of the effective objects in the third layer is displayed such that a playback point and a playback section are set in the form of a bar, and the terminal device is displayed on an adjustment indicator displayed at both ends of the bar. It is possible to set a reproduction section for the movement of the effective objects by recognizing the user's operation on the.

In the video production system, the virtual space is displayed as a floor surface and a structure on the floor surface, and the terminal device can change an image of the floor surface or the structure according to a user setting.

Another embodiment includes a terminal device for generating scan row data by scanning a 3D (dimension) character having a plurality of joints; And a first server that processes the scan row data and generates 3D scan data including bone data for the plurality of joints, wherein the terminal device downloads the 3D scan data from the first server. Then, using the original data of the 3D scan data, a plurality of motions are given to the character in a virtual space, and the temporal change of the virtual space is converted into a video according to a viewpoint of a camera shooting the virtual space. It provides a video production system to create.

As described above, according to the present embodiment, a user can easily produce a video using 3D scanned data. Users can create videos that make themselves the main character and share it with others through communities such as online. Through this, video sharing in the online community can be further activated, and a user's desire to express themselves can be easily satisfied. In addition, as the utilization of 3D scanning data increases, the demand for 3D scanning devices increases and can contribute to the development of related industries.

1 is a block diagram of a video production system according to an embodiment.
2 is a block diagram of a terminal device according to an embodiment.
3 is a diagram illustrating a configuration of a 3D character according to an embodiment.
4 is a flowchart of a video production method according to an embodiment.
5 is a diagram illustrating receiving 3D scan data in a terminal device according to an embodiment.
6 is a diagram illustrating a control screen for each layer in a terminal device according to an embodiment.
7 is a view showing the control window of FIG. 6 in more detail.
8 is a view showing a control screen for a character effect in a terminal device according to an embodiment.
9 is a view showing a control screen for recording in a terminal device according to an embodiment.
FIG. 10 is a diagram illustrating a screen in which a plurality of 3D characters are combined with real-time images in a terminal device according to an embodiment.
11 is a block diagram of a video production system according to another embodiment.
12 is a configuration diagram of a terminal device according to another embodiment.
13 is an exemplary view of scan data generated by a terminal device according to another embodiment.
14 is an exemplary view of a configuration for editing scan data in a terminal device according to another embodiment.
15 and 16 are exemplary views of a configuration for generating a 3D character using a 2D image in a terminal device according to another embodiment.

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. It should be noted that in adding reference numerals to the components of each drawing, the same components have the same reference numerals as possible, even if they are displayed on different drawings. In addition, in describing the present invention, when it is determined that detailed descriptions of related well-known configurations or functions may obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only for distinguishing the component from other components, and the nature, order, or order of the component is not limited by the term. When a component is described as being "connected", "coupled" or "connected" to another component, that component may be directly connected to or connected to the other component, but another component between each component It will be understood that elements may be "connected", "coupled" or "connected".

1 is a block diagram of a video production system according to an embodiment.

Referring to FIG. 1, the video production system 100 may include a terminal device 110, at least one server 120, 140, a 3D scanner 130, and the like.

The terminal device 110 is a device that provides a user interface, receives user manipulation for video production, and displays information required for video production and a result of video production to a user.

The terminal device 110 may include a display panel including a touch panel as a user interface. The terminal device 110 may recognize a user manipulation by receiving a signal for the touch panel, and display information required for video production and a result of video production through a display panel.

The terminal device 110 may further perform other functions in addition to video production. For example, the terminal device 110 may perform wireless communication. The terminal device 110 is a type of telephone called a mobile device, and may include a touch panel, a display panel, and a data processor. The terminal device 110 may perform functions necessary for the video production system 100 using such a touch panel, display panel, and data processor used for wireless communication.

The 3D scanner 130 may 3D scan the character 10 and generate scan data. Here, the character 10 is an object including a plurality of joints, and may be a person or an animal.

The 3D scanner 130 includes a 3D depth camera, a distance measuring device, and the like, and can generate a three-dimensional shape of the appearance of the character 10 as scan data in the form of digital data.

The 3D scanner 130 and the terminal device 110 may be physically arranged in different spaces. For example, the terminal device 110 may be arranged in a space where the user resides as a small device, and the 3D scanner 130 may be arranged in a specific space as a large device, for example, a 3D scanning specialty store. .

The 3D scanner 130 and the terminal device 110 which are spaced apart from each other may be interlocked with each other through the server 120. For example, the 3D scanner 130 may transmit the generated scan data to the first server 120 via the network. Then, the terminal device 110 may download scan data through the first server 120 or download 3D scan data in which the scan data is reprocessed. At this time, an identification number may be assigned to the scan data and/or the 3D scan data. The 3D scanner 130 transmits the scan data to which the identification number is assigned to the first server 120, and the terminal device 110 is applicable. The scan data or 3D scan data corresponding to the identification number can be downloaded from the first server 120. The identification number may be referred to as a scan code, and detailed examples of the scan code will be described later.

The 3D scanner 130 and the terminal device 110 may be directly linked through a network. The 3D scanner 130 may directly transmit the scan data or 3D scan data obtained by reprocessing the scan data to the terminal device 110. In addition, the terminal device 110 may produce a video using the scan data or 3D scan data.

The 3D scanner 130 and the terminal device 110 may be devices implemented with the same hardware. For example, the terminal device 110 may include a 3D depth camera, and the terminal device 110 may perform the same function as the 3D scanner 130 using the 3D depth camera. In this embodiment, the terminal device 110 may directly generate a scan data by 3D scanning the character 10.

The first server 120 may convert the scan data into 3D scan data. The 3D scan data may include bone data for a plurality of joints included in the character 10. The first server 120 analyzes the features of the character 10 identified through the scan data-for example, an inflection point, an end point, an intersection, etc.-to determine the position of the joint, information about the joint, joints and joints This data can be generated, including information on the skeleton (skeleton) between, and information on the skin covering the skeleton. In addition, 3D scan data may be generated by combining the present data with the scan data including the live image of the character 10.

The first server 120 may generate the present data using an analysis algorithm programmed therein, or may receive the present data generated by the control of an administrator who has access to the scan data.

In one embodiment, the present data is described as being generated by the first server 120, but depending on the embodiment, the present data may be generated by the 3D scanner 130 or the terminal device 110.

The terminal device 110 may upload the video produced using the 3D scan data to the second server 140. The second server 140 may be a server that shares videos or supports social network services. The user using the terminal device 110 can share the video produced directly through the second server 140 with other people.

2 is a block diagram of a terminal device according to an embodiment.

Referring to FIG. 2, the terminal device 110 includes a storage unit 210, a motion control unit 220, a background setting unit 230, a camera control unit 240, a sound control unit 250, a video production and playback unit 260 ), an interface unit 270, a terminal communication unit 280, a central control unit 290, and the like.

The storage unit 210 may store various data necessary for video production. For example, the storage 210 may store 3D scanned data. The 3D scan data may include three-dimensional photorealistic images of the appearance of the character, and this data corresponding to the 3D modeling data of the character. The storage unit 210 may store various setting data. The terminal device 110 may recognize a user manipulation and select one setting from various settings, where the setting data corresponding to the setting may be stored in advance in the storage 210. Hereinafter, it can be understood that components that are stored in advance or described as being preset are stored in advance in the storage unit 210.

The motion control unit 220 may provide a plurality of motions to the 3D character in the virtual space using the original data of the 3D scan data. Here, the 3D character means a character in a virtual space, not a character in a real space. The 3D character described as being manipulated and controlled below may be understood as a character in a virtual space.

The motion control unit 220 may use the data to give a plurality of motions to the 3D character in the virtual space. The operation of the 3D character in the virtual space may mean that the coordinates corresponding to the 3D character change temporally on the spatial coordinates assigned to the virtual space. In a digital aspect, a combination of coordinates of a 3D character corresponding to time can be understood as a character's motion. Here, the coordinates may be understood as one method for specifying the position in space, and the motion of the 3D character may be implemented in another way to specify the position in space.

On the other hand, the 3D character includes a plurality of joints, and the motion of the 3D character may be implemented as such a plurality of joints move over time in a virtual space. The movement here can be understood as a concept involving rotation as a movement in space. The motion control unit 220 applies information about a plurality of joints obtained through the present data, information about a skeleton between the joints, and information about a flesh covering the skeleton to an algorithm incorporating a dynamic concept in 3D You can implement the character's action.

The motion control unit 220 stores the digital data in the virtual space representing the motion of the 3D character-for example, coordinate data according to the passage of time-in the storage unit 210, and the video production and playback unit 260 described later. ) Can generate a video using this digital data.

The motion control unit 220 may give a plurality of motions to a 3D character in a virtual space, where each motion may be formed in units of clips. A clip is a small unit of a video, and a clip corresponding to each operation may have the same time or different times. The speed of the motion of the 3D character can be controlled. At this time, the user can control the speed of the motion of the 3D character by adjusting the time length of the clip corresponding to each motion. For example, if the time length of the clip is increased, it means that the time required for one operation to be completed is increased, and accordingly, the motion of the 3D character can be slow-motion. Conversely, if the time length of the clip is short, it means that the time required to complete one motion is shortened, and accordingly, the motion of the 3D character can be fast-motiond.

The motion control unit 220 may arrange the motion of the 3D character in the virtual space on the first layer. Hereinafter, a video production process will be described using the concept of a layer. Here, the layer is a virtual layer, and it can be understood that components arranged on different layers can overlap each other at the same time without colliding with each other.

The motion control unit 220 may apply motion blending to a plurality of motions so that images of adjacent clips are continuously connected. Blending is a technique that allows the composition of different characteristics to be continuously connected at a time boundary, and in one embodiment, a blending technique can be applied to the motion of a 3D character and the setting of a camera. The blending applied to the motion of the 3D character may be referred to as motion blending, and the blending applied to the setting of the camera may be referred to as camera blending.

Motion blending can be implemented by the pose of the 3D character. For example, the motion control unit 220 may control the motion of each clip such that the finishing posture of the previous clip of the 3D character and the starting posture of the next clip coincide in two adjacent clips. Under this control, the motions of 3D characters can be naturally connected in two adjacent clips.

The background setting unit 230 may place a background image for a virtual space on the second layer. The virtual space can be represented by floors and structures on the floor. The bottom surface is a portion where the bottom of the 3D character comes into contact while the 3D character is not floating in the air, and the 3D character can move along the bottom surface while not floating in the air.

The background setting unit 230 may change the image of the floor surface or the structure according to the user setting. In this way, if the virtual space is not divided into a floor surface and a space above it, the bottom surface of the image according to the user setting may be inconsistent with the floor surface of the virtual space. In this case, the 3D character is not the floor surface. An unnatural situation that stands on location can be created.

A plurality of images stored by the user may be stored in the storage unit 210. The background setting unit 230 may set a floor surface or a space on the top of one of these multiple images according to a user setting.

The background setting unit 230 may arrange a background effect in a virtual space. The background effect can be arranged on the third layer. The background effect can be generated by arranging the movement of the effective objects in the virtual space into the surrounding area of the 3D character. For example, a background effect may be realized by arranging movements of effective objects corresponding to snowfall or rainfall, eg, eyes or rain, in a peripheral area of a 3D character.

The background setting unit 230 may adjust the playback time for the background effect by repeatedly playing digital data defined in a short time period. In the user interface, which will be described later, the movement of the effective objects in the third layer may be displayed so that the playback time and playback section are set in the form of a bar, and the background setting unit 230 is an adjustment indicator displayed at both ends of the bar. By recognizing the user's operation for, it is possible to set the playback section for the movement of effective objects.

Each motion of the 3D character is arranged in units of clips in the first layer, a background image for a virtual space is arranged in the second layer, and motions of effective objects for the virtual space can be arranged in the third layer. have. Here, each layer may be displayed on the screen of the terminal device to be lined up in the time axis direction. And, the user can control the playing time of each arrangement element by controlling the implementation length of the arrangement elements of each layer-the motion of the character, the background image, and the movement of the effective objects-in the time axis direction.

The camera controller 240 may set a virtual camera for photographing a virtual space, and generate an image by applying a camera's view point to 3D characters and background images. Here, the camera is a virtual camera, and performs a function of viewing a virtual space in terms of display, and in terms of data processing, converts three-dimensional (3D) spatial data into two-dimensional image data. You can provide the criteria. For example, the camera control unit 240 may set the camera on the first side with respect to the bottom surface, and convert the virtual space into two-dimensional image data as the viewpoint of the first side. As another example, the camera control unit 240 may set the camera on the second side rotated 90 degrees clockwise from the first side, and convert the virtual space into two-dimensional image data as the viewpoint of the second side. The moving image may be generated by arranging two-dimensional image data on a time axis. At this time, the two-dimensional image data may be determined by a camera setting.

The video production and playback unit 260 may generate a temporal change of the virtual space according to the viewpoint of the camera as a video. For example, the camera's viewpoint is applied to the virtual space according to the camera control unit 240, and the 2D image data of the virtual space taken by the camera as the viewpoint is captured by the camera. Data- can be generated, and the video production and playback unit 260 can produce a video by continuously arranging and integrating two-dimensional image data according to this temporal flow.

The camera control unit 240 may apply a camera blending technique in setting the camera. The camera control unit 240 may set the camera from the first time to the first position, and set the camera from the second time to the second position. At this time, the camera control unit 240 may set the camera so that the position of the camera is gradually changed from the first position to the second position for the first time to the second time so that the viewpoint of the camera is continuously changed.

The sound control unit 250 may control sound data. The sound data can be extracted from the sound file by user setting, or can be generated by recording. When the user wants to add dubbing according to the motion of the 3D character, the sound control unit 250 may convert the user's voice into sound data.

The video production and playback unit 260 may generate video by combining sound data with image data according to the viewpoint of the aforementioned camera.

In addition, the video production and playback unit 260 may play the previously generated video through the display unit (not shown) of the terminal device 110.

In addition, the video production and playback unit 260 may synthesize a 3D character into a real-time image being photographed by a real camera (not shown) provided in the terminal device 110. In addition, the video production and playback unit 260 may apply a character motion of a previously generated video to a 3D character synthesized in real-time video.

In other words, the video production and playback unit 260 recognizes the real bottom surface from the real-time image being shot by a real camera (not shown), and places the 3D character on the real bottom image through augmented reality technique It is possible to generate an augmented reality video synthesized motion image of

Here, when a plurality of 3D scan data corresponding to a plurality of 3D characters are stored in the storage unit 210, the video production and playback unit 260 may transmit a number of 3D characters to the actual bottom surface of the real-time image according to the user's setting. Can be placed.

For example, when the 3D scan data corresponding to the first character, the second character, and the third character is stored in the storage unit 210, the video production and playback unit 260 is configured according to the user's setting as shown in FIG. The first character 1010, the second character 1020, and the third character 1030 may be arranged on the real bottom surface 1040 of the real-time image. Here, the actual bottom surface 1040 may be a top surface such as a ground surface, a road surface, a desk, a table, or the like.

In addition, the video production and playback unit 260 may apply different motions to a plurality of 3D characters according to a user's setting, or apply the same motion (for example, military service motion) to a plurality of 3D characters.

In one embodiment, the storage 210 may store a plurality of objects for augmented reality that can be arranged in a real-time image. Here, the object for augmented reality is a structure (for example, a performance stage, a running track, a grass field) that can be disposed on a real floor surface, a device that can be applied to a structure (for example, stage lighting, stadium lighting, billboards) ) Etc.

When a plurality of augmented reality objects are stored in the storage 210 as described above, the video production and playback unit 260 may appropriately place one or more augmented reality objects and 3D characters in the real-time image according to the user's setting. Can.

For example, when the object for augmented reality is a performance stage, the video production and reproducing unit 260 may place a performance stage on a real bottom surface of a real-time image, and place a 3D character on the upper surface of the performance stage.

Meanwhile, the video production and playback unit 260 may synthesize a user-selected image from a plurality of images stored in the storage unit 210 into a real-time image through a chroma key technique.

Specifically, the video production and playback unit 260 may synthesize an image selected by the user on a background portion having a chroma key color among backgrounds of a real-time image.

The interface unit 270 recognizes user manipulation and provides information to the user. The interface unit 270 may include a touch panel and recognize a user manipulation through the touch panel. It can be understood that the above-described user setting or user setting described later is determined according to the user operation. The interface unit 270 includes a display panel, and may display information generated in a video production process or a result of video production on the screen while using the display panel as a screen.

The terminal communication unit 280 is a part that can transmit and receive data to and from an external device, and can receive 3D scan data from the first server and transmit identification information to the first server. In addition, the terminal communication unit 280 may download a video of another person uploaded to the online community and display it on the screen, and upload the video produced by the video production and playback unit 260 to the online community.

The central control unit 290 is a component that processes the overall control of the terminal device 110 and controls organic connections of each other component, and can implement functions that are not implemented in each component.

3 is a diagram illustrating a configuration of a virtual 3D character according to an embodiment.

Referring to FIG. 3, a 3D character 20 in a virtual space may be composed of a plurality of bones (BN). And, each bone (BN) may include at least one joint (JT), skeleton (SK), flesh (WT), and the like. Each bone (BN) may be matched to a part of the body, for example, arms, legs, hands, feet, body, head, and the like, and this matching information may be included in this data. The characteristics of the joint JT may be limited according to matching information. For example, the range of rotation of the joint JT may be limited according to a portion corresponding to the body. Skeletons (SK) positioned at the end may be connected to one joint (JT), and skeletons (SK) positioned between the two skeletons (SK) may be connected to two joints (JT) at both ends. The weight WT may have a weight value, and the movement speed, acceleration, and the like of the bone BN may be determined according to the weight value.

Some of the descriptions are explained as concepts in the real space, but these concepts can be applied to virtual spaces as well, and can be understood as being processed as digital values in the virtual spaces in the video production system. The dynamic processing of virtual data in the virtual space may be implemented through a number of pre-developed skeleton algorithms, or may be implemented by a commercially available or freely distributed application programming interface (API).

4 is a flowchart of a video production method according to an embodiment.

Referring to FIG. 4, the device may load 3D scan data (S400)-the device described with reference to FIG. 4 may be the aforementioned terminal device or another device. 3D scan data is scan data generated by 3D scanning a character having a plurality of joints, and may include main data for a plurality of joints.

The device may use the present data to give a plurality of motions to the 3D character in a virtual space, and arrange each motion of the 3D character in the first layer in units of clips (S402). Here, the 3D character is a virtual character that imaged 3D scan data, and the motion of the 3D character is a 3D character by moving a body such as an anterior gait, a posterior gait, a left gait, a right gait, a gumbo, a spin, or an arm turn. Forming a movement, it can be performed through control of a joint of a 3D character located in a virtual space.

In addition, the gesture that can be given to the 3D character may include an emotion expression gesture. Emotional expression may be implemented through fine position control of detailed components of the face, or may be implemented through position control of body components such as arms, bodies, legs, and fingers.

The apparatus may store a control sequence for each operation in advance, and, when one operation is selected by user setting, may operate the 3D character according to a control sequence corresponding to the operation. The control sequence may include time-dependent position control values for each joint of the 3D character.

The apparatus may apply motion blending at the boundary of adjacent motions (S404). Motion blending controls different motions to be continuously performed, and the device may perform motion blending such that the last posture of the previous motion and the first posture of the next motion are the same.

The device may place a background image of the virtual space on the second layer (S406). The background image may include a background screen and a background effect.

The background image may be 3D data. When the background image is composed of 3D data, the background image may also be changed according to the viewpoint of the camera shooting the virtual space.

The device may set a virtual camera for photographing a virtual space, and determine an image at each time by applying a camera's viewpoint to 3D characters and background images (S408). The device may generate 2D image data for each time unit-for example, 10ms, 100ms, 1s, etc., and a view point of the 2D image data may be determined according to the direction of the camera.

The setting of the camera may be, for example, a setting for a position and a direction, or a setting for a camera effect. The camera effect may be, for example, a technique such as zoom-in, zoom-out, camera rotation, or the like.

The device can apply camera blending between different camera settings. For example, when the camera is set from the first time to the first position and from the second time to the second position, the device may change the position of the camera from the first position to the second position during the first time to the second time. Camera settings can be controlled to change gradually.

The device may generate a video by combining 2D image data generated for each time unit (S410). Such a video may be the same as capturing a temporal change in the virtual space according to the viewpoint of the camera. The device may generate a video by further combining sound. The sound may be a pre-generated music file or a voice file recorded by a user.

The device may share the video by uploading the generated video to an online community, a video sharing site, a social network service, etc. (S412).

On the other hand, the terminal device may be implemented in the form of an app (APP) on a personal portable wireless communication device such as a smartphone, and an example of the interface and control of the terminal device in such an implementation method will be further described.

5 is a diagram illustrating receiving 3D scan data in a terminal device according to an embodiment.

Referring to FIG. 5, the user may generate scan data SD for himself through the 3D scanner 130. At this time, the 3D scanner 130 may be located at a spatially spaced distance from the terminal device 110, and the user may use the scan code (SC) so that the 3D scanner 130 can use the data scanned by the terminal device 110 later. ) To match the scan data SD.

The 3D scanner 130 may transmit the scan data SD matching the scan code SC to the first server 120. The scan data SD is data including a three-dimensional (3D) photorealistic image of the user's appearance, and may not include information about the user's joint.

The first server 120 may convert the scan data SD into 3D scan data 3DSD or recognize the 3D scan data 3DSD corresponding to the scan data SD by recognizing another device or an administrator's operation.

Then, the terminal device 110 may transmit the scan code SC to the first server 120, receive the corresponding 3D scan data 3DSD, and store or load it in the storage unit. The terminal device 110 may display a window for inputting a scan code on the screen, and transmit the scan code input to the window to the first server 120.

6 is a diagram illustrating a control screen for each layer in a terminal device according to an embodiment, and FIG. 7 is a diagram more specifically showing the control window of FIG. 6.

6 and 7, the terminal device may display a control screen of each layer for video production.

The control screen may include a preview window 610 and a control window 620.

The preview window 610 may display an image of the video at the current playback time or a video at the entire duration of the playback time. In the control window 620, there is a time axis window 670, and a playback time indicator 622 indicating the current playback time may be displayed. The preview window 610 indicates the playback time indicator 622 on the still screen. The image of the video at the current playback time can be displayed, and the video at the entire playback time section can be displayed on the video screen.

The control window 620 may include a time axis window 670 and a plurality of layer windows 630, 640, 650, and 660.

The time axis window 670 may display the current playing time indicated by the playing time indicator 622, and may display the entire playing time of the video. If necessary, a storage button or an aspect ratio control button may be further disposed in the time axis window 670.

In the operation layer window 630, operation blocks 632 that give a plurality of operations to the character may be disposed. Each motion block 632 may form one clip.

The length of the motion blocks 632 in the time axis direction may determine the reproduction time of each clip. The user can adjust the length of each operation block 632 by manipulating the left and right ends 734 in the time axis direction of each operation block 632. Each motion block 632 may be arranged to have a preset playback time according to the motion of the 3D character when initially arranged. And, when the user adjusts the length of the motion block 632, slow motion or fast motion of each motion may be implemented.

Camera setting blocks 642 may be disposed in the camera layer window 640. One camera setting may be applied to each of the camera setting blocks 642, and the time of such camera setting may be determined by the time axis length of each camera setting block 642. In a time zone in which the camera setting block 642 is not disposed, the default setting for the camera may be applied.

On the other hand, the playback time of the video may be determined by the sum of the lengths of the motion blocks 632 arranged in the motion layer window 630, wherein the camera setting block 642 portion 643 exceeding the determined play time is It is disabled and may not be included in the video.

A background image block 652 may be disposed in the background layer window 650. The background layer window 650 may be divided into a background effect layer window 754 and a background screen layer window 756. Further, a background effect block 755 may be disposed in the background effect layer window 754, and a background screen block 757 may be disposed in the background layer layer 756. Depending on the length of the time axis direction of the background effect block 755 and the wallpaper block 757, the background effect and the playback time of the background screen may be determined. If the background effect is implemented in a form of repeatedly playing a certain image, the background effect block Even if the length of 755 is changed, the playback speed of the background effect does not change, and only the playback time can be changed.

The reproduction time may be determined by a start time and an end time of the motion blocks 632, before the end time of the motion blocks 632 in the time axis direction is the active part 722, and thereafter indicated by the inactive part 724. The components of the camera layer window 640 and the background layer window 650 corresponding to the inactive portion 724 may be ignored in video production.

8 is a view showing a control screen for a character effect in a terminal device according to an embodiment.

The terminal device can give an effect to the 3D character. For example, the terminal device can control the effects of fire, lightning, and wind to occur in a specific part of the body of the 3D character.

The terminal device may display a location on which the effect can be applied along with the 3D character on the screen. In addition, the terminal device can recognize the user's operation and display the effect of the 3D character at the user-selected position, and after the time when the effect of the 3D character is granted on the time axis, the terminal is a 3D character with an effect applied to the operation of the 3D character. The action can be displayed.

9 is a view showing a control screen for recording in a terminal device according to an embodiment.

The terminal device can record a user's voice and combine it with a video. As shown in FIG. 9, a recording button is displayed on the control screen, and a length of recording may be displayed on a time axis. Meanwhile, although not shown in the drawings, a recording control window may be displayed on the screen together with a preview window of a video. In this embodiment, the user can record the voice in the form of dubbing while watching a video.

In the above, an embodiment in which the 3D scanner 130 scans the appearance of the character 10 and generates scan data therefor has been described.

Hereinafter, an embodiment in which the scan data of the character 10 is generated by the terminal device 110 or a separate portable device other than the 3D scanner 130 will be described.

11 is a block diagram of a video production system according to another embodiment.

Referring to FIG. 11, the video production system 1100 may include a terminal device 1110 and at least one server 1120 or 140. Here, since the second server 140 is the same as the configuration of one embodiment, description of the second server 140 will be omitted.

The terminal device 1110 is a device that provides a user interface, and receives user manipulation for video production, and can display information required for video production and a result of video production to a user.

The terminal device 1110 may include a display panel including a touch panel as a user interface. The terminal device 1110 may recognize a user operation by receiving a signal for the touch panel, and display information required for video production and a result of video production through a display panel.

The terminal device 1110 may perform other functions in addition to video production. For example, the terminal device 1110 may perform wireless communication. The terminal device 1110 is a type of telephone called a mobile device, and may include a touch panel, a display panel, and a data processor. The terminal device 1110 may perform functions necessary for the video production system 1100 using such a touch panel, display panel, and data processor used for wireless communication.

Specifically, the terminal device 1110 may perform a 3D scan of the character 10 performed by the 3D scanner 130 in one embodiment.

To this end, the terminal device 1110 may include a 3D depth camera.

In other words, the terminal device 1110 may 3D scan the character 10 using a 3D depth camera, and generate scan data for the character 10. Here, the scan data may include polygon data for 3D scan and 2D texture data.

The terminal device 1110 generating the scan data may transmit the scan data to the first server 1120.

Here, the terminal device 1110 may transmit the scan data to the first server 1120 after editing.

In other words, when the scan data is the scan data for the whole body of the character 10 as shown in FIG. 13, the terminal device 1110 transmits the whole body scan data, which is the unedited scan data as shown in FIG. 13 according to the user's setting, or , As shown in 14A of FIG. 14, the head scan data, which is the scan data obtained by removing the body of the character 10 from the whole body scan data, may be generated and transmitted to the first server 120.

In addition, the terminal device 1110 may generate and transmit body scan data, which is the scan data obtained by removing the head of the character 10 from the whole body scan data, as shown in 14B of FIG. 14 and transmit it to the first server 1120.

On the other hand, in another embodiment, without performing the 3D scan of the character 10 in the terminal device 1110, the character in a separate portable device (for example, a tablet PC equipped with a 3D depth camera, digital camera, laptop, etc.) Scan data may be generated by performing the 3D scan of (10).

In this case, the terminal device 1110 may receive the scan data from the portable device through short-range wireless communication such as Bluetooth or cable communication, and perform the editing of the scan data as described above, and then transmit it to the first server 1120. .

After the terminal device 1110 transmits the scan data to the first server 1120, the terminal device 1110 may download 3D scan data from the first server 1120. Here, the 3D scan data may include bone data for a plurality of joints included in the character 10.

In other words, when the scan data transmitted from the terminal device 1110 to the first server 1120 is whole body scan data, the 3D scan data may include the present data about the whole body joint of the character 10.

When the scan data transmitted from the terminal device 1110 to the first server 1120 is head scan data, the 3D scan data may be head head 3D scan data including this data on the neck joint of the character 10.

When the scan data transmitted from the terminal device 1110 to the first server 1120 is body scan data, the 3D scan data may be body 3D scan data including the present data on the body joint. Here, the body joint may mean a joint excluding a wood joint among the whole body joints of the character 10.

The terminal device 1110 that downloads the 3D scan data can produce a video using the 3D scan data and upload the video to the second server 140 as in one embodiment.

The second server 140 may be a server that shares videos or supports social network services. A user using the terminal device 1110 can share the video produced directly through the second server 140 with other people.

In another embodiment, the terminal device 1110 may store a plurality of head 3D illustration data, a plurality of body 3D illustration data, and a plurality of whole body 3D illustration data. Here, the head 3D illustration data, the body 3D illustration data, and the whole body 3D illustration data may be data obtained by combining data viewed with a 3D illustration image.

When the 3D scan data is two-phase 3D scan data, the terminal device 1110 may generate a virtual 3D character by combining any one of a plurality of body 3D illustration data with the two-phase 3D scan data according to the user's setting.

When the 3D scan data is the body 3D scan data, the terminal device 1110 may generate a virtual 3D character by combining any one of a plurality of head 3D illustration data with the body 3D scan data according to a user setting.

In addition, the terminal device 1110 extracts the head 2D texture of the character 10 from the scan data of the character 10, or after extracting the head 2D image from the photo data of the character 10, the whole body 3D as shown in FIG. It is also possible to generate a virtual 3D character as shown in 16A in FIG. 16 by combining a two-phase 2D texture or a two-phase 2D image on the 2D image composite surface 1512 of the illustration 1510. Here, the terminal device 1110 may further store various 3D items that can decorate the 2D image composite surface 1512 as shown in 16B of FIG. 16.

The terminal device 1110 may produce a video using a virtual 3D character, and upload the video to the second server 140.

The first server 1120 may convert the scan data into 3D scan data. Here, the first server 1120 may express the texture and contour of the 3D scan data as 2D texture data included in the scan data.

When the scan data is whole body scan data, the first server 120 may generate 3D scan data (full body 3D scan data) including the original data of the whole body joint of the character 10.

When the scan data is head scan data, the first server 1120 may generate 3D scan data (head 3D scan data) including the present data on the neck joint of the character 10.

When the scan data is body scan data, the first server 1120 may generate 3D scan data (body 3D scan data) including this data for the body joint of the character 10.

The configuration for generating the main data and the configuration for generating the 3D scan data in the first server 1120 is the same as in one embodiment, so a description of this is omitted in other embodiments.

12 is a configuration diagram of a terminal device according to another embodiment.

Referring to FIG. 12, the terminal device 1110 includes a storage unit 1210, a scan data editing unit 1220, a motion control unit 220, a background setting unit 230, a camera control unit 240, and a sound control unit 250, It may include a video production and playback unit 260, the interface unit 270, the terminal communication unit 280, the central control unit 290, and the like. Here, the operation control unit 220, the background setting unit 230, the camera control unit 240, the sound control unit 250, the video production and playback unit 260, the interface unit 270, the terminal communication unit 280, the central control unit 290 is the same as the configuration of one embodiment, so a description thereof will be omitted.

The storage unit 1210 may store various data necessary for video production. For example, the storage unit 1210 may store scan data and 3D scan data. Here, the storage unit 1210 may receive and store 3D scan data from the first server 1120. The 3D scan data may include three-dimensional photorealistic images of the appearance of the character, and this data corresponding to the 3D modeling data of the character.

In addition, the storage unit 1210 may further store photo data of the character 10.

The storage unit 1210 may store various setting data. The terminal device 1110 may recognize a user operation and select one setting from various settings, where the setting data corresponding to the setting may be stored in advance in the storage unit 1210. Hereinafter, it can be understood that components previously stored or described as being preset are stored in advance in the storage unit 1210.

Meanwhile, the storage unit 1210 may store a plurality of head 3D illustration data, a plurality of body 3D illustration data, and a plurality of whole body 3D illustration data.

The scan data editing unit 1220 may edit scan data generated through an actual camera (not shown) provided in the terminal device 1110 or a separate portable device.

In other words, when the scan data is the scan data for the whole body of the character 10 as shown in FIG. 13, the scan data editing unit 1220 may display the character 10 in the whole body scan data as shown in 14A in FIG. It is possible to generate head scan data, which is scan data with the body removed.

In addition, the scan data editing unit 1220 may generate body scan data, which is scan data in which the head of the character 10 is removed from the whole body scan data as illustrated in 14B of FIG. 14 according to a user's setting.

After editing the scan data as described above, the scan data editing unit 1220 may transmit the scan data to the first server 1120 through the terminal communication unit 280.

Meanwhile, the scan data editing unit 1220 may extract a 2D texture of the head of the character 10 from the scan data of the character 10 according to the user's setting.

Further, the scan data editing unit 1220 may extract a 2D image of the head of the character 10 from the photo data of the character 10 according to the user's setting.

The scan data editing unit 1220 may transmit unedited scan data to the first server 1120 according to a user's setting.

When the 3D scan data stored in the storage unit 1210 is a head-to-head 3D scan data, the scan data editing unit 1220 can perform any one of a plurality of body 3D illustration data stored in the storage unit 1210 according to user settings. It can be combined with data to create virtual 3D characters.

When the 3D scan data stored in the storage unit 1210 is body 3D scan data, the scan data editing unit 1220 scans any one of a plurality of head 3D illustration data stored in the storage unit 1210 according to user settings. It can also be combined with data to create virtual 3D characters.

After generating the 3D character as described above, the terminal device 1110 may produce a video for the 3D character through the same configuration as in one embodiment.

On the other hand, when the 2D texture or 2D image of the head image is extracted from the scan data editing unit 1220, the scan data editing unit 1220 may use any one of a plurality of whole body 3D illustration data stored in the storage unit 1210 according to a user setting. You can also create virtual 3D characters by combining 2D textures or 2D images.

Additional examples that have not been described previously will be described further.

The image production method according to an embodiment may be used as a tool for generating revenue in combination with a business model.

The app constituting the terminal device may be distributed free of charge, but some of the items added as an option to the app may be distributed in a paid form. In technical terms, the terminal device may recognize a user operation, recognize a user selection for a paid item, and transmit the user selected item and user information to the first server. Then, the first server checks the item cost payment related information in the user account through the user information, and when the item cost is paid or processed as being paid, the terminal device transmits related data so that the corresponding item can be activated in the terminal device. Can be sent to The paid item may, for example, be for the motion of the character, and may be for the setting of the camera. Presets in which a plurality of actions of a character are continuously combined may be used as a paid item. In this case, the user can easily apply the continuous motion included in the preset to his character.

The 3D character may be generated by 3D scan data, but in some embodiments, a virtually generated 3D illustration character may be further used. These 3D illustration characters can be distributed to each terminal device in the form of paid items.

In addition, the user can create a 3D illustration character using a computer on which the 3D production tool is installed, and may be stored and used in the terminal device through cable communication or short-range wireless communication between the computer and the terminal device.

The entire 3D illustration character can be distributed as one, or it can be distributed as a head, body, or leg. When the illustration characters are divided and distributed, one 3D illustration character may be completed in combination according to the user's selection.

The terminal device may further arrange a filter layer according to a user's operation, and the terminal device may apply a filter effect to an image of a video according to characteristics of a filter constituting the filter layer.

The terminal device may further arrange a text layer according to user manipulation, and the terminal device may further combine texts constituting the text layer into a video.

The present embodiment provides a system capable of producing a video that is a subject in a simple manner by using a smart phone and a personal computer (PC) device that everyone has own 3D scan data produced through 3D scanning. As a result, it helps you to create your own fun and cool video by using various characters as if you were actually acting in a virtual space, and adding various effects.

A large amount of content is being produced and shared so that video content is becoming a means of communication online. In addition, there are companies that generate indirect sales, such as advertising revenue, by regularly creating and sharing videos that appear on their own through personal broadcasting channels to secure the number of subscribers on the channel. However, as video production requires a lot of time and technology, even if there are users who have a desire to create, it is not easily accessible, and as many videos are produced, it cannot solve the exhaustion of the material.

This embodiment will be a tool for creating new community contents that can be produced and shared through a simple and easy manipulation of a planned video along with the provision of a material called a unique character that has made its appearance in a way to solve this.

In addition, the business model of operators with 3D scanners could be a solution to present a new additional business model to a business type that is only selling figures produced by 3D printers using 3D printers to extract 3D modeling data extracted using expensive cameras.

The terms "comprise", "compose" or "have" as described above mean that the component can be inherent, unless otherwise stated, and do not exclude other components. It should be interpreted that it may further include other components. All terms, including technical or scientific terms, have the same meaning as generally understood by a person skilled in the art to which the present invention belongs, unless otherwise defined. Commonly used terms, such as predefined terms, should be interpreted as being consistent with the meaning in the context of the related art, and are not to be interpreted as ideal or excessively formal meanings unless explicitly defined in the present invention.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the claims below, and all technical spirits within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

Claims (14)

Loading 3D scan data including bone data for the plurality of joints as scan data generated by 3D (dimension) scanning a character having a plurality of joints;
Assigning a plurality of movements to the character in a virtual space using the present data, and arranging each movement of the character in units of clips in a first layer;
Applying motion blending to the plurality of motions so that images of adjacent clips are continuously connected;
Disposing a background image of the virtual space on a second layer;
Setting a virtual camera for photographing the virtual space, and applying a view point of the camera to the character and the background image; And
Generating a temporal change of the virtual space as a video according to the viewpoint of the camera
Video production method comprising a. According to claim 1,
In the step of placing each movement of the character in the first layer,
The clip is set so that one operation for the character is completed for each clip, but a slow or fast motion is realized by adjusting the speed of the operation according to the length of the clip. According to claim 1,
And arranging the movement of the effective objects for the virtual space in the third layer as a peripheral area of the character. According to claim 1,
In the step of applying the motion blending for the plurality of actions,
A method of making a video in which two adjacent clips control the motion of each clip so that the ending posture of the previous clip and the starting posture of the next clip match. According to claim 1,
In the step of applying the viewpoint of the camera,
When the camera is set from the first time to the first position and from the second time to the second position, the position of the camera from the first time to the second time is from the first position to the second position Slowly changing video production method. According to claim 1,
In the step of generating the video,
A video production method for generating the video by further combining sound. According to claim 1,
After the step of generating the video,
Synthesizing the character into a real-time image being shot by a real camera;
Causing the character to continuously take the plurality of actions in the real-time image;
And generating an augmented reality video for the augmented reality video in which the character continuously takes the plurality of motions from the real-time video. The method of claim 7,
Before the step of generating the augmented reality video,
The method of video production further comprising the step of synthesizing a specific image on a background portion having a chroma key color among the background of the real-time image. According to claim 1,
After the step of generating the video,
Synthesizing the character and one or more separate characters separate from the character into a real-time image captured by a real camera;
Causing the character and the one or more separate characters to continuously take the plurality of operations in the real-time image;
And generating an augmented reality video for the augmented reality video in which the character and the one or more separate characters continuously take the plurality of motions from the real-time video. A 3D scanner that scans a character having a plurality of joints and generates scan row data by 3D (dimension) scanning;
A first server that processes the scan row data and generates 3D scan data including bone data for the plurality of joints; And
Downloading the 3D scan data from the server, using the original data of the 3D scan data to give a plurality of actions to the character in a virtual space, the virtual according to the viewpoint of the camera shooting the virtual space Terminal device that generates a temporal change of space in video
Video production system comprising a. The method of claim 10,
A plurality of layers are displayed on the screen of the terminal device to line up in the time axis direction,
Each movement of the character is arranged in units of clips in the first layer, a background image for the virtual space is arranged in the second layer, and effective objects for the virtual space are arranged in the third layer. Video production system where movements are arranged. The method of claim 10,
The movement of the effective objects in the third layer is displayed such that a playback time point and a playback section are set in the form of a bar,
The terminal device is a video production system that recognizes the user's operation of the adjustment indicator displayed at both ends of the bar and sets a playback section for the movement of the effective objects. The method of claim 10,
The virtual space is represented by a floor surface and structures on the floor surface,
The terminal device is a video production system that changes the image of the floor or the structure according to the user setting. A terminal device for generating scan row data by scanning 3D (dimension) characters having a plurality of joints; And
A first server that processes the scan row data and generates 3D scan data including bone data for the plurality of joints
Including, but, the terminal device downloads the 3D scan data from the first server, and using the original data of the 3D scan data to give a plurality of operations to the character in a virtual space, the virtual space A video production system that generates a temporal change of the virtual space as a video according to the viewpoint of the camera shooting the video.

Images