JP4899733B2 - Image generation method and image generation system - Google Patents

Description

  The present invention relates to an image generation method and generation system in which CG (computer graphics) images are combined, and more particularly to an image generation method and generation system suitable for movie production.

In the production of a movie that makes full use of VFX (visual effects) video, the work of developing an image of the final video using the actual background by pre-visualization by CG is performed before the final video is shot. The work using pre-visualization by CG is equivalent to the work of imaging the final video using a conventional storyboard.
In such pre-visualization by CG, in order to develop an actual actor's movement and background image, the CG corresponding to the actual actor's movement and background is adjusted by trial and error.

  When the image of the final video is developed by pre-visualization by CG, actual shooting is performed. In the actual shooting, a real object (background, etc.) was shot based on the image of the final video developed by the pre-visualization by CG, and the final video was obtained by synthesizing a CG character or the like with the shot video.

Conventional CG pre-visualization is only a means of trial and error when developing an image of the final video before the actual performance, and is a process independent of the stage of the actual photographing.
Therefore, when trying to obtain the final video according to the final video image developed by the pre-visualization by CG, the person (cameraman) takes a picture taking the camera work of the shooting camera along the final video image. There is a need to do. As a result, the photographing work is costly and time consuming.

  SUMMARY An advantage of some aspects of the invention is that it provides a new image generation method and image generation system for improving the cost and time efficiency of photographing.

  The present invention relating to an image generation method is a method for generating an image obtained by combining a video captured by a camera and a CG image, wherein the camera has a resolution higher than the first resolution video and the first resolution for the same subject. A high-resolution second resolution video can be output, and a simple composite image obtained by synthesizing a first resolution video captured by the camera and a CG image generated based on simple CG model data is displayed on a display device in real time. In addition, a shooting step for storing camera work information at the time of shooting by the camera in a storage unit, and the same shooting target as the first resolution image shot in the shooting step using the camera work information in the shooting step High-quality composite image generation step of combining the second resolution video and the CG image generated based on the high-quality CG model data , Characterized by having a.

  Further, the image generation method includes a simple composite image editing step for editing the simple composite image, and the high-quality composite image is edited using editing information of the simple composite image in the simple composite image editing step. Is preferred.

  The present invention relating to an image generation system is a system for generating an image obtained by synthesizing a video captured by a camera and a CG image, and a second resolution having a higher resolution than the first resolution video and the first resolution for the same subject. A simple composite image obtained by combining a camera capable of outputting both resolution images and a first resolution image captured by the camera and a CG image generated based on simple CG model data based on camera work information of the camera. A simple composite image generation unit that generates the image, a display device that displays the simple composite image in real time, a camera work information storage unit that stores the camera work information, and the camera used to generate the simple composite image Based on the work information, a high-quality composite image is generated by combining the second resolution video and a CG image generated based on the high-quality CG model data. Characterized in that it comprises a high-quality composite image generation unit that, the.

  Preferably, the system further includes a simple composite image editing unit for editing the simple composite image, and edits the high-quality composite image using the editing information of the simple composite image.

According to the present invention, since a simple composite image obtained by synthesizing a CG image generated based on the simple CG model data is displayed in real time when shooting is performed, the camera is checked while confirming the simple composite image as pre-visualization. Work and trial and error can be performed. Moreover, since the simple composite image is a composite image of the first resolution video with a relatively low resolution and the CG image generated based on the simple CG model data, the composite image generation process can be performed at high speed, and in real time. Suitable for display.
Furthermore, in the present invention, since the second resolution video having a high resolution and the CG image generated based on the high-quality CG model data are synthesized based on the stored camera work information at the time of shooting, separately, A high-quality final image can be obtained without performing actual shooting.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Entire system configuration]
FIG. 1 shows an overall view of the image generation system 1.
The system 1 includes a camera 10, a processing device 20 for image generation, a composite image display device 30, a high-quality composite image recording device 40, and the like.

[Camera configuration]
The camera 10 shoots an object to be photographed and outputs an image, a lens device 12 attached to the camera body 11 to adjust the zoom, and a pan / tilt for panning and tilting the camera body 11. And a tilt device 13.

  The camera body 11 is capable of outputting a plurality of resolution images of the same subject. Specifically, the camera body has a standard resolution video (SD (Standard Definition) video) such as NTSC video as the first resolution video and a high definition video (HD (High Definition) video such as a high definition video as the second resolution video). ) Can be output.

  The lens device 11 can output zoom information indicating a zoom amount and focus information related to focus. The pan / tilt device 13 can output the pan amount and tilt amount operated by the cameraman as pan / tilt information (camera position and orientation information). Here, zoom information, focus information, and pan / tilt information (camera position and orientation information) are collectively referred to as camera work information (imaging information). With this camera work information, it is possible to grasp the position and orientation (angle) of the camera with respect to the shooting target, the size of the shooting target, and the like.

  In addition to the pan / tilt device 13, a device such as a crane or a dolly may be used as a device for determining the position and orientation of the camera body 11. In this case, the camera position / orientation information (camera work information) may be acquired from the movement of the crane or the like.

  The SD video, HD video, and camera work information output from the camera 10 are provided to the processing device 20. In addition, the camera 10 may be one unit as illustrated, or a plurality of units.

[Processing equipment]
The processing device 20 is configured by a computer having a storage device such as a hard disk. The installed computer program is executed by the processing device 20, whereby each function described later in the processing device 20 is realized.

  FIG. 2 shows a functional block diagram of the processing device 20. The processing device 20 can centrally manage the simple composite image and the high-quality composite image, and can generate two composite images as a single photographing result. For this reason, the processing device 20 is input with SD video and HD video having different resolutions, and simple CG model data and high-quality CG as CG model data for generating a CG image to be synthesized with both videos. It has model data with different quality (data amount, precision).

  That is, the processing device 20 generates an SD video (first resolution video) with a small amount of data (low precision) and a CG image generated based on simple CG model data with a small amount of data (low precision). By combining, a simple composite image can be generated. Further, the processing device 20 synthesizes a high-quality HD image (second resolution video) with a large amount of data and a CG image generated based on the high-quality CG model data with a large amount of data and a high-quality image. A quality composite image can be generated.

Details of the functions of the processing device 20 will be described with reference to FIG.
[About CG model data]
The processing device 20 has a CG model data management unit 21 that stores CG model data used for generating CG images to be combined with video captured by a camera and performs management for combining. The CG model data management unit 21 stores CG model data information such as a CG character (CG model data having a motion such as a person / animal) or a CG object (CG model data serving as a background such as a building). Model data creation / editing / processing, layout adjustment of a CG image generated by CG model data, and the like can be performed.

  FIG. 3 shows the configuration of CG model data information managed by the CG model data management unit 21. As the CG model data information, for example, both high-quality CG model data and simple CG model data are provided for a scene of a certain CG character, and both CG model data are managed in association with each other by an ID. As a result, high-quality CG model data corresponding to the simple CG model data can be easily discriminated using the ID as a key. Note that a method for managing both CG model data in association with each other is not limited to ID, and various methods for associating both CG model data can be employed.

Here, the high-quality CG model data is precise and high-quality CG model data used for the final video to be a movie, and the amount of data is very large. On the other hand, the simple CG model data is the same character or object as the high-quality CG model data, but is not made as precisely as the high-quality CG model data, and thus is simple and has a relatively small amount of data.
The “CG model data” here includes position information and other information necessary for rendering to be combined with video, and can also be referred to as virtual space information.

[Shooting using real-time simple composite image generation function (shooting step)]
As one of the functions, the processing device 20 includes a real-time simple composite image generation unit (real-time rendering unit) 22 that generates a simple composite image in real time at the time of shooting.
The real-time simple composite image generation unit 22 calculates and generates a simple composite image at high speed using SD video and simple CG model data with a small amount of data, and displays the simple composite image on the display device (monitor) 30 in real time. .
The real-time simple composite image generation unit 22 emphasizes real-time processing rather than image quality, and generates a composite image by a high-speed composite image generation processing algorithm even though the image quality is inferior to that of a high-quality composite image generation unit 25 described later. Process.

The real-time simple composite image generation unit 22 receives CG model data to be combined from the simple CG model data management unit 21a of the CG model data management unit 21 and generates a CG image in order to generate a simple composite image. And the SD video output from the camera 10.
In addition, the real-time simple composite image generation unit 22 receives real-time alignment information of the SD video and the CG image when combining, and combines the CG image at a predetermined position of the SD image based on the alignment information.

Real-time alignment information is generated by the real-time (simple) alignment processing unit 22a. The real-time (simple) alignment processing unit 22a performs simple alignment processing that prioritizes processing speed over the precision of the position where the CG image is synthesized in order to generate alignment information in real time. .
The real-time (simple) alignment processing unit 22a calculates the positional relationship between the object to be photographed and the camera from the camera position and orientation information, zoom information, and SD video output from the camera 10, and the CG image is the SD video. It calculates which position it should be located, and outputs the calculation result as real-time (simple) alignment information.

  When shooting is performed using the real-time simple composite image generation function, it is possible to synthesize a CG image in real time while shooting a real landscape and / or actor with the camera 10. A film production staff such as a movie director can determine the camera work and composition while confirming the image of the final video by the simple composite image obtained by combining the CG images.

  In addition, the conventional CG pre-visualization is full CG model data created entirely by CG including not only a person (character) but also a background, etc. According to the system 1 of this embodiment, the final video This image can be refined using a composite image by CG-synthesizing a person or a creature with a video shot in the background of a real scene, such as in a sound stage, open set, outdoor location, etc., where tools are already built. Therefore, trial and error by camera work can be easily performed while confirming the composite image.

[Shooting information (camera work information) and video management]
The processing device 20 includes a shooting information and video management unit 23 for managing shooting information including camera work information and other information together with SD video and HD video.
In addition, the processing device 20 includes a simple video storage 23 a that stores SD video output from the camera 10, a high-definition video storage unit 23 b that stores HD video output from the camera 10, and shooting information output from the camera 10. An imaging information storage unit 23c for storing (camera work information) is included.

  The shooting information and video management unit 23 performs processing for storing the shooting information input to the processing device 20 in the shooting information storage unit 23c in association with the corresponding SD video and HD video. Here, the shooting information (camera work information) is both SD video shooting information (camera work information) and HD video shooting information (camera work information), and is common to both videos. In addition, the shooting information stored in the shooting information storage unit 23c includes real-time (simple) alignment information generated by the real-time (simple) alignment processing unit 22a.

  As illustrated in FIG. 4, the shooting information storage unit 23 c includes, together with shooting information, information specifying an SD video corresponding to the shooting information (SD video specifying information; first video specifying information) and information specifying an HD video. (HD video specific information; second video specific information) is stored. Therefore, referring to the shooting information storage unit 23c, an SD video or HD video corresponding to a certain shooting information can be known, and shooting information corresponding to a certain SD video or HD video can be found.

[Composite image editing function (editing step)]
The processing device 20 includes an editing unit 24 for performing video editing work such as determination of a cut ratio for a simple composite image. The processing device 20 also displays a simple composite image generation unit 24a that generates a simple composite image for editing and the like, and displays the generated simple composite image on a display device (not shown) such as the processing device 20. A simple composite image display unit 24b is provided.
In addition, the processing device 20 includes an editing information management unit 24c that manages editing information and stores editing results such as cuts and the like edited by the function of the editing unit 20 as editing information.

In the simple composite image generation unit 24a, the SD image stored in the simple video storage unit 23a and the simple CG model data from the simple CG model data management unit are used as shooting information stored in the shooting information unit 23c (particularly in real time). (Simple registration information generated by the registration processing unit 22a). Similar to the real-time simple composite image generation unit 22, the simple composite image generation unit 24a places importance on real-time processing rather than image quality, and even if the image quality is inferior to the high-quality composite image generation unit 25 described later. A composite image generation process is performed by a high-speed composite image generation process algorithm.
In addition, if there is already edited editing information, the simplified composite image generation unit 24a generates a simple composite image corresponding to the editing information (cut cut, etc.).

  The editor can edit the simple composite image easily and at high speed by the editing function of the editing unit 24. That is, since the simple composite image (SD image and simple CG model data) has a small amount of data, it can be edited quickly. Therefore, editing can also be performed during shooting at a shooting site or the like.

[Generation of high-quality composite image (high-quality composite image generation step)]
As one of the functions, the processing device 20 includes a high-quality composite image generation unit 25 that generates a high-quality composite image that becomes a final video after shooting. The high-quality composite image generation unit 25 calculates and generates a high-quality composite image using the HD video and high-quality CG model data with a large amount of data, and outputs the high-quality composite image to the recording device 40.
Unlike the simple composite image generation units 22 and 24a, the high-quality composite image generation unit 25 places importance on image quality rather than real-time processing, and uses the composite image generation processing algorithm of the simple composite image generation 22 and 24a. However, the composite image generation processing is performed by a composite image generation processing algorithm that can generate a high-quality composite image even if time is required.

The high-quality image generation unit 25 receives high-quality CG model data to be combined from the high-quality CG model data management unit 21b of the CG model data management unit 21 to generate a high-quality composite image, and generates a CG image. At the same time, it is synthesized with the HD video recorded in the high quality video storage unit 23b.
In addition, the high-quality composite image generation unit 25 receives high-precision alignment information for the alignment of the HD video and the CG image at the time of synthesis, and at the position of the HD image based on the high-precision alignment information, A CG image is synthesized.

  The high-precision alignment information is generated by the high-precision alignment processing unit 25a. The high-accuracy alignment processing unit 25a generates alignment information by giving priority to the accuracy of the position where the CG image is synthesized rather than the processing speed in order to ensure the quality in the final video. Process.

The high-precision alignment processing unit 25a calculates the positional relationship between the shooting target and the camera from the shooting information (camera work information) and the HD video stored in the shooting information storage unit 23c, and the CG image is an HD video. The position to be located is strictly calculated, and the calculation result is output as high-quality alignment information.
The high-precision alignment processing unit 25a receives editing information for the simple composite image from the editing information management unit 24c, and automatically performs editing similar to the editing performed on the simple composite image according to the editing information. The edited high-quality composite image is output.

  As described above, the high-definition image generation processing unit 25a performs processing using the HD video obtained together with the SD video at the time of shooting through repeated trial and error by the real-time pre-visualization using the real-time simple composite image generation function. . That is, a high-quality composite image can be generated based on the HD video and the shooting information (camera work information) at the time of shooting the HD video.

Therefore, if the system 1 of this embodiment is used, pre-visualization based on mixed reality using the simple CG model data and the SD video, that is, confirming the composite image in real time, Trial and error is possible.
Moreover, after trial and error, it is possible to perform an editing operation while confirming a simple composite image using the recorded SD video, camera work information, and simple CG model data.
Furthermore, based on the editing result obtained by the editing operation, the HD video recorded simultaneously with the SD video and precise high-quality CG model data are used for offline rendering of a high-quality composite image with higher accuracy and image quality. Can be generated.
As a result of the above, when this system 1 is used, it is possible to easily perform trial and error of camera work and editing by real-time pre-visualization using mixed reality while confirming a composite image to which special effects are applied. The final video (completed video) can be generated by off-line rendering without performing another shooting for production.

  In addition, this invention is not limited to the said embodiment, A various deformation | transformation is possible.

1 is an overall configuration diagram of a system. It is a functional block diagram of a processing apparatus. It is a data structure figure of CG model data information. It is a figure which shows an imaging | photography information storage part, a simple image storage part, and a high quality image storage part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image generation system 10 Camera 20 Processing apparatus 30 Display apparatus 22 Real-time simple composite image generation part (simple composite image generation part)
23c Shooting information storage unit (camera work information storage unit)
25 High Quality Composite Image Generation Unit 24 Editing Unit

Claims (2)

A method of obtaining a high-quality composite image edited by shooting with a camera while performing pre-visualization,
The camera is capable of outputting both a first resolution video and a second resolution video having a higher resolution than the first resolution for the same subject,
A simple synthesized image obtained by synthesizing a first resolution video photographed by the camera and a CG image generated based on simple CG model data by a computer is captured by the camera as an image for pre-visualization. A shooting step for displaying on a display device in real time and storing camera work information at the time of shooting by the camera in a storage means;
A simple composite image editing step of editing the simple composite image ;
A computer uses the camera work information in the shooting step to provide a second resolution video for the same shooting target as the first resolution video shot in the shooting step, and a higher quality than the simple CG model data. The CG image generated based on the CG model data is synthesized to generate a high-quality composite image having a higher quality than the simple composite image, and the edit result for the simple composite image in the simple composite image editing step is displayed. A high-quality composite image generation step for performing the same editing as that performed on the simple composite image with respect to the high-quality composite image using the editing information shown, and outputting the edited high-quality composite image; ,
An image generation method characterized by comprising: A system for obtaining a high-quality composite image edited by shooting with a camera while performing pre-visualization,
A camera capable of outputting both a first resolution video and a second resolution video having a higher resolution than the first resolution for the same subject;
Based on the camera work information of the camera, a simple composite image obtained by combining a first resolution video captured by the camera and a CG image generated based on simple CG model data is generated as an image for pre-visualization. A composite image generation unit;
A display device that displays the simple composite image in real time during shooting by the camera as an image for pre-visualization;
A camera work information storage unit for storing the camera work information;
A simple composite image editing unit for editing the simple composite image ;
Based on the camera work information used to generate the simple composite image, the second resolution video and the CG image generated based on the high-quality CG model data higher in quality than the simple CG model data are combined. The high-quality composite image having a higher quality than the simple composite image is generated, and editing information indicating an editing result of the simple composite image by the simple composite image editing unit is used for the simple composite image. An image generation system comprising: a high-quality composite image generation unit configured to perform editing similar to the above-described editing on the high-quality composite image and output the edited high-quality composite image.

Images