JP2002271691A - Image processing method, image processing device, storage medium, and program - Google Patents

Abstract

(57) [Summary] [Problem] To match the spatial coordinates of a real-life space and a virtual space,
Provided are an image processing method, an image processing device, a storage medium, and a program that eliminate a positional shift when a real shot video and a CG (computer graphics) video are combined. SOLUTION: Before being synthesized with a CG image by an image synthesizing unit 107, the distortion of the real image is corrected by a correcting unit 105 so that the spatial coordinates of the real image space and the virtual space are matched, and the real image and the CG image are corrected. Eliminates misalignment when combining with

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to real space video and CG
The present invention relates to an image processing method, an image processing apparatus, a storage medium, and a program for creating a mixed reality by synthesizing a virtual space image by (computer graphics).

[0002]

2. Description of the Related Art Conventionally, a mixed reality apparatus (Oshima, Sato, Yamamoto, Tamura: "AR2 Hockey: Realization of a Cooperative Mixed Reality System") which creates a mixed reality by combining a real space image and a virtual space image by CG. ”, Transactions of the Virtual Reality Society of Japan, Vo1.3, No.2, pp.55-6
0, 1998, etc.) are known, but in these conventional mixed reality apparatuses, mixed reality images are not created in consideration of distortion due to the imaging system.

[0003]

Therefore, in the conventional mixed reality apparatus (image processing apparatus), there is a potential problem that the spatial coordinates of the real space and the virtual space do not match due to the distortion of the real image. Existing. As a result, there has been a problem that a displacement occurs when the real space image and the virtual space image based on the CG are combined, and the mixed reality space becomes unnatural.

[0004] The present invention has been made in view of the above-mentioned problems of the related art, and has as its object to provide an image processing method, an image processing apparatus, a storage medium, and a program capable of obtaining a natural mixed reality. And

[0005]

According to an aspect of the present invention, there is provided an image processing method, comprising: an imaging step of imaging a real image at a viewpoint position of a subject in accordance with a gaze direction; An image acquisition step of converting a real video image captured by the step into a format that can be synthesized by a computer, a gaze direction detection step of detecting a gaze direction of a subject, and a CG (CG) from a viewpoint direction detected by the gaze direction detection step. Computer graphics)
A CG image generating step of generating an image, an image synthesizing step of synthesizing the CG image generated by the CG image generating step and a real shot image obtained by the image obtaining step, and an image synthesized by the image synthesizing step Displaying the CG to the subject.
The image processing apparatus further includes a correction step of correcting the actual photographed image before being combined with the image.

In order to achieve the above object, an image processing method according to a second aspect of the present invention includes an image capturing step of capturing a real image in accordance with a gaze direction at a viewpoint position of a subject, and an image captured by the image capturing step. An image acquisition step of converting the photographed image into a format that can be synthesized by a computer, a gaze direction detection step of detecting a gaze direction of the subject, and CG (computer graphics) from the viewpoint detected by the gaze direction detection step. A CG image generating step of generating an image, an image synthesizing step of synthesizing the CG image generated by the CG image generating step and a real shot image obtained by the image obtaining step, and an image synthesized by the image synthesizing step Displaying the image to the subject. Step by characterized by having a step of correcting the CG image before being combined with the live-action video.

In order to achieve the above object, an image processing apparatus according to a third aspect of the present invention comprises: an image pickup means for picking up a photographed image at a viewpoint position of a subject in accordance with a line of sight; Image acquisition means for converting the actual photographed image into a format that can be synthesized by a computer, gaze direction detection means for detecting the gaze direction of the subject, and CG (computer graphics) from the viewpoint detected by the gaze direction detection means CG image generating means for generating an image, image synthesizing means for synthesizing a CG image generated by the CG image generating means and a real image obtained by the image obtaining means, and an image synthesized by the image synthesizing means And a display means for displaying the image to the subject. Characterized in that it has a correction means for correcting a copy image.

In order to achieve the above object, an image processing apparatus according to a fourth aspect of the present invention is the image processing apparatus according to the third aspect, wherein the correction means is attachable to the imaging means. It is a lens device.

In order to achieve the above object, an image processing apparatus according to a fifth aspect of the present invention is the image processing apparatus according to the third aspect, wherein the correction means comprises the image pickup means and the image acquisition means. It is characterized by being installed between the means.

In order to achieve the above object, an image processing apparatus according to a sixth aspect of the present invention is the image processing apparatus according to the third aspect, wherein the correction means comprises the image acquisition means and the image acquisition means. It is characterized by being installed between the synthesizing means.

According to a seventh aspect of the present invention, there is provided an image processing apparatus according to the third aspect, wherein the imaging means is a video camera. It is characterized by being.

According to another aspect of the present invention, there is provided an image processing apparatus comprising:
In the image processing device according to any one of the above, the video acquisition unit is a capture board.

In order to achieve the above object, an image processing apparatus according to a ninth aspect of the present invention is the image processing apparatus according to the third aspect, wherein the line-of-sight direction detecting means is a position sensor. It is characterized by.

In order to achieve the above object, an image processing apparatus according to a tenth aspect of the present invention is the image processing apparatus according to the third aspect, wherein the CG generating means is a computer. And

In order to achieve the above object, an image processing apparatus according to claim 11 of the present invention is the image processing apparatus according to claim 3 or 6, wherein the image synthesizing means is a computer. It is characterized by.

In order to achieve the above object, an image processing apparatus according to a twelfth aspect of the present invention is the image processing apparatus according to the third aspect, wherein the display means is a graphics board. Features.

In order to achieve the above object, an image processing apparatus according to a thirteenth aspect of the present invention is the image processing apparatus according to the third aspect, wherein the display means is an HMD (Head Mount Display). There is a feature.

In order to achieve the above object, an image processing apparatus according to claim 14 of the present invention comprises: an image pickup means for picking up a photographed image at a viewpoint position of a subject according to a gaze direction;
A video acquisition unit that converts a real video image captured by the imaging unit into a format that can be synthesized by a computer, a gaze direction detection unit that detects a gaze direction of a subject, and a viewpoint direction detected by the gaze direction detection unit. CG (Computer Graphics) image generating means for generating a CG image, image synthesizing means for synthesizing the CG image generated by the CG image generating means and the real image obtained by the image obtaining means, and the image synthesizing An image processing apparatus having display means for displaying a video synthesized by the means to a subject, comprising a correction means for correcting the CG video before the video synthesis means synthesizes the CG video with the real shot video. .

In order to achieve the above object, an image processing apparatus according to a fifteenth aspect of the present invention is the image processing apparatus according to the fourteenth aspect, wherein the correction means is attachable to the imaging means. It is a lens device.

In order to achieve the above object, an image processing apparatus according to a sixteenth aspect of the present invention is the image processing apparatus according to the fourteenth aspect, wherein the correction means comprises the CG.
It is characterized in that it is installed between an image generating means and the image synthesizing means.

In order to achieve the above object, a mixed reality apparatus according to claim 17 of the present invention is an image processing apparatus according to claim 14 or 15, wherein the imaging means is a video camera. It is characterized by the following.

In order to achieve the above object, an image processing apparatus according to claim 18 of the present invention is characterized in that, in the image processing apparatus according to claim 14, the image acquisition means is a capture board. Features.

In order to achieve the above object, an image processing apparatus according to a nineteenth aspect of the present invention is the image processing apparatus according to the fourteenth aspect, wherein the line-of-sight direction detecting means comprises:
It is a position sensor.

In order to achieve the above object, an image processing apparatus according to claim 20 of the present invention is the image processing apparatus according to claim 14 or 16, wherein the CG generating means is a computer. It is characterized by.

In order to achieve the above object, an image processing apparatus according to claim 21 of the present invention is the image processing apparatus according to claim 14 or 16, wherein the video synthesizing means is a computer. It is characterized by.

In order to achieve the above object, an image processing apparatus according to claim 22 of the present invention is the image processing apparatus according to claim 14 or 16, wherein the video synthesizing means is a computer. It is characterized by.

According to another aspect of the present invention, there is provided an image processing apparatus according to the present invention, wherein the display means is a graphics board. Features.

In order to achieve the above object, an image processing apparatus according to a twenty-fourth aspect of the present invention is the image processing apparatus according to the fourteenth aspect, wherein the display means comprises an HMD.
(Head-mounted display).

In order to achieve the above object, a storage medium according to the twenty-fifth aspect of the present invention uses a physical space image and CG
A storage medium storing a computer-readable program for controlling an image processing apparatus that creates a mixed reality by synthesizing with a virtual space image by (computer graphics), and is a real image in accordance with a gaze direction at a viewpoint position of a subject. An image capturing step of capturing an image of the subject, a video obtaining step of converting a real image captured by the image capturing step into a format that can be synthesized by a computer, a visual line direction detecting step of detecting a visual line direction of the subject, and the visual line direction detecting step. A CG image generating step of generating a CG (computer graphics) image from the detected viewpoint direction; and an image for combining the CG image generated by the CG image generating step with the real image obtained by the image obtaining step. A synthesizing step and the video synthesizing step A display step of displaying the made video in a subject, characterized by comprising a correction step of correcting the photographed image before being combined with the video said CG images by synthetic steps.

[0030] In order to achieve the above object, a storage medium according to claim 26 of the present invention stores a real space image and a CG image.
A storage medium storing a computer-readable program for controlling an image processing apparatus that creates a mixed reality by synthesizing with a virtual space image by (computer graphics), and is a real image in accordance with a gaze direction at a viewpoint position of a subject. An image capturing step of capturing an image of the subject, a video obtaining step of converting a real image captured by the image capturing step into a format that can be synthesized by a computer, a visual line direction detecting step of detecting a visual line direction of the subject, and the visual line direction detecting step. A CG image generating step for generating a CG (computer graphics) image from the detected viewpoint direction; and an image for combining the CG image generated by the CG image generating step with the real image obtained by the image obtaining step. A synthesizing step and the video synthesizing step A display step of displaying the made video in a subject, characterized by comprising a correction step of correcting the CG image before being combined with the video the Stock Video synthetically step.

In order to achieve the above object, a program according to a twenty-seventh aspect of the present invention includes a program for real-time image
A computer-executable program for controlling an image processing apparatus that creates a mixed reality by synthesizing a virtual space image by G (computer graphics),
An imaging step of capturing a real image at a viewpoint position of a subject in accordance with a gaze direction; a video acquisition step of converting the real image captured by the imaging step into a format that can be synthesized by a computer; and a gaze direction of detecting a gaze direction of the subject A detecting step; a CG image generating step of generating a CG (computer graphics) image from the viewpoint direction detected by the line-of-sight direction detecting step;
A video synthesizing step of synthesizing a CG video generated by the G video generating step and a real shot video obtained by the video obtaining step; a display step of displaying a video synthesized by the video synthesizing step to a subject; And a correcting step of correcting the real image before being combined with the CG image by the combining step.

Further, in order to achieve the above object, a program according to claim 28 of the present invention provides a program for real-time image
A computer-executable program for controlling an image processing apparatus that creates a mixed reality by synthesizing a virtual space image by G (computer graphics),
An imaging step of capturing a real image at a viewpoint position of a subject in accordance with a gaze direction; a video acquisition step of converting the real image captured by the imaging step into a format that can be synthesized by a computer; and a gaze direction of detecting a gaze direction of the subject A detecting step; a CG image generating step of generating a CG (computer graphics) image from the viewpoint direction detected by the line-of-sight direction detecting step;
A video synthesizing step of synthesizing a CG video generated by the G video generating step and a real shot video obtained by the video obtaining step; a display step of displaying a video synthesized by the video synthesizing step to a subject; And a correction step of correcting the CG image before being combined with the actual photographed image by the combining step.

[0033]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The gist of the present invention is to make the spatial coordinates of a real image space and a virtual space coincide with each other and to eliminate a positional shift at the time of synthesizing a real image and a CG (computer graphics) image, thereby achieving a natural image. That is, mixed reality can be obtained.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First Embodiment) First, the first embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS.

FIG. 1 is a block diagram showing the concept of a mixed reality apparatus (image processing apparatus) according to the present embodiment. In FIG. Imaging means for capturing an image of a real space, 1
Reference numeral 03 denotes a gaze direction detecting unit that detects the gaze direction of the subject. Reference numeral 104 denotes a video acquisition unit that converts a video signal from the imaging unit 102 into RGB format or YUV format digital data. Reference numeral 105 denotes a correction unit that corrects image distortion included in the digital video data acquired by the video acquisition unit 104. Reference numeral 106 denotes a CG image generation unit, which is CG (computer graphics) for the subject's gaze direction obtained from the gaze direction detection unit 103
This is to generate an image. 107 is a video synthesizing means.
This is to combine the video corrected by the correction unit 105 and the CG video obtained from the CG video generation unit 106.
The correction unit 105 is provided between the video acquisition unit 104 and the video synthesis unit 107 in the present embodiment.
An output unit 108 outputs an output signal of the video synthesizing unit 107 to the display unit 101.

In the present embodiment, the correcting means 105 is provided between the video obtaining means 104 and the video synthesizing means 107, but may be provided between the image capturing means 102 and the video obtaining means 104. .

FIG. 2 is a diagram showing a specific implementation example of the mixed reality apparatus (image processing apparatus) according to the present embodiment. In FIG.
(Head-mounted display), which is mounted on the subject's head. Reference numeral 202 denotes a small-sized video camera as the imaging means 104, which is fixedly attached to the HMD 201 and always captures an image of the subject's line of sight. Reference numeral 203 denotes a magnetic direction sensor (position sensor) serving as the line-of-sight direction detecting unit 103, which is fixedly attached to the HMD 201. The magnetic direction sensor 203 may be a transceiver type or a single type.

An HMD 201, a video camera 202, and a magnetic direction sensor 203 are connected to the computer 204. The computer 204 includes a CPU (Central Processing Unit) 205, a memory 206, a HDD (Hard Disk Drive: storage device) 207, an I / O (Input / Output Device) 208, a graphics board 209, and a video capture board 210. .

The image acquisition means 104 is implemented as a video capture board 210, and the output means 108 is implemented as a graphics board 209.

The HDD 207 stores a correction program 211 as the correction means 105, a CG video synthesis program 212 as the CG video generation means 106, and a video synthesis program 213 as the video synthesis means 107. Then, these programs 211 to 213
Is executed by the computer 204 as needed.

Next, the mixed reality apparatus according to the present embodiment
The operation of the (image processing apparatus) will be described.

The video signal of the subject's line of sight obtained by the video camera 202 is converted into video digital data via the video capture board 210 and taken into the computer 204. The video digital data captured by the computer 204 is input to the correction program 211. The correction program 211 corrects the distortion of the digital video data in the real space so as to match the virtual space generated by the CG video generation program 212.

The magnetic direction sensor 203 detects the gaze direction of the subject, and transmits a detection signal to the computer 204. The CG image synthesizing program 212 running on the computer 204 receives a detection signal from the magnetic direction sensor 203 and generates a virtual space image in the line of sight of the subject.

The real space video data from the correction program 211 and the virtual space video data from the CG video generation program 212 are synthesized by the video synthesis program 213. Specifically, the image in the virtual space without the background is superimposed on the image in the real space. As a result, an image in the direction of the line of sight of the mixed reality space where an object in the virtual space (for example, a character such as a robot) exists in the real space is generated. The generated mixed reality space image is converted into a signal that can be displayed on the HMD 201 by the graphics board 209 and displayed on the HMD 201.

In this way, it is possible to generate an image of a mixed reality space in which an object in the virtual space exists in the real space in front of the subject.

Next, the correction means 105 (correction program 2)
Details of 11) will be described with reference to FIG.

FIGS. 3A and 3B are diagrams for explaining an image correction processing operation by the correction means 105 (correction program 211). FIG. 3A shows an input image, and FIG. Images distorted by the acquisition means 104),
(C) shows an image corrected by the correction unit 105 (correction program 211).

The input image shown in FIG.
Dots, b = 640 dots, a: b = 3: 4. The image shown in FIG. 3B has a ′ = 480 dots,
b '= 600 dots, a': b '= 4: 5. Further, the image shown in FIG. 3C has a ″ = 480 dots,
b ″ = 640 dots, a ″: b ″ = 2: 3.

In the correction program 211, correction information for correcting image distortion caused by the video camera 202 and the capture board 210 is registered. Then, the correction program 211 processes the video digital data based on the correction information.

For example, when the image distortion caused by the video camera 202 and the capture board 210 is an aspect ratio distortion as shown in FIG.
Cancels the image distortion by performing the reverse conversion of the conversion that causes the distortion, and the CG image generation program 212
Creates a real-space image as shown in FIG. 3C that matches the coordinate system of the virtual space generated by. This is an inverse transformation method, but in the case of aspect ratio distortion like this example,
There is no problem if known linear interpolation or the like is used.

The state of the image distortion is determined by the video camera 2
It is more convenient for the correction program 211 to have a function of registering and adapting a plurality of pieces of correction information, because the correction program 211 changes according to the change of the capture board 210 or the capture board 210.

As described in detail above, according to the mixed reality apparatus (image processing apparatus) according to the present embodiment, the correction means 10
5 (correction program 211), the actual image is corrected before being synthesized with the CG image by the image synthesizing means (image synthesizing program 213). ) Can be obtained in real space without distortion. Then, it is possible to generate an image of a mixed reality space in which a positional shift of a virtual space object does not occur from a real space image having no image distortion.

The correction program 211 as the correction means 105 is used in a game or the like utilizing mixed reality.
It can be used not only for the purpose of removing video data distortion but also for producing effects in games. For example, when the game scene is changed or when an enemy attacks, the real space can be displayed in a twisted state.

(Second Embodiment) In the first embodiment, the correcting means 105 is provided between the video acquiring means 104 and the video synthesizing means 107.
Does not apply to the video data obtained from the video acquisition unit 104 (capture board 210).
02 and the image acquisition unit 104.

When there is no image distortion caused by the image pickup means 102, the correction means may be constituted in the form of a lens portion connected to the lens of the image pickup means 102 so as to be superposed.

In these cases, the correction means 105 is configured as a device instead of a correction program.

(Third Embodiment) Next, the third embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS. 4 and 5. FIG.

The mixed reality apparatus according to the present embodiment
Since the configuration of the mounting state of the (image processing apparatus) is the same as that of FIG. 2 in the first embodiment, the description will be made with reference to FIG.

FIG. 4 is a block diagram showing the concept of a mixed reality apparatus (image processing apparatus) according to the present embodiment. In FIG. 4, the same parts as those in FIG. 1 in the first embodiment are described. Are denoted by the same reference numerals.

FIG. 4 differs from FIG. 1 in the installation position of the correction means 105. That is, in the present embodiment, the correction means 105 is provided between the CG video generation means 106 and the video synthesis means 107. Thus, the CG image generated by the CG image generation unit 106 can be corrected by the correction unit 105 (correction program 211) before being synthesized with the real image by the image synthesis unit 107.

Next, the correction means 105 (correction program 2)
Details of 11) will be described with reference to FIG.

FIGS. 5A and 5B are diagrams for explaining the image correction processing operation by the correction means 105 (correction program 211). FIG. 5A shows an input image, and FIG. Images distorted by the acquisition means 104),
(C) shows the CG generated by the CG image generating means 106.
The video, (D) shows correction means 105 (correction program 21)
Each of the CG images corrected by 1) is shown.

The input image shown in FIG.
Dots, b = 640 dots, a: b = 3: 4. The image shown in FIG. 3B has a ′ = 480 dots,
b '= 600 dots, a': b '= 4: 5. The image shown in FIG. 3C has A = 480 dots and B =
It is 600 dots. Further, the image shown in FIG.
A '= 480 dots and B' = 600 dots.

In the correction program 211, correction information for correcting the image distortion caused by the video camera 202 and the capture board 210 is registered. Then, the correction program 211 processes the video digital data based on the correction information.

For example, when the image distortion caused by the video camera 202 and the capture board 210 is an aspect ratio distortion as shown in FIG.
Adapts this distortion to the virtual space image and cancels the shift due to the image distortion, thereby correcting the CG image as shown in FIG. 3 (D), and the corrected CG image is shown in FIG. 3 (B). The image shown is synthesized with the image synthesizing means 107 (image synthesizing program 212).

As described in detail above, according to the mixed reality apparatus (image processing apparatus) according to the present embodiment, the correcting means 10
5 (correction program 211), the CG image is corrected before being synthesized with the real image by the image synthesis means (image synthesis program 213), so that the imaging system (imaging means 102, image acquisition means 104) ) Can be obtained in real space without distortion. Then, it is possible to generate an image of a mixed reality space in which a positional shift of a virtual space object does not occur from a real space image having no image distortion.

The correction program 211 as the correction means 105 is used in a game or the like utilizing mixed reality.
It can be used not only for the purpose of removing video data distortion but also for producing effects in games. For example, when the game scene is changed or when an enemy attacks, the real space can be displayed in a twisted state.

(Other Embodiments) Even if the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), a device including one device (for example, , Copiers, facsimile machines, etc.).

Another object of the present invention is to provide a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or an apparatus, and to provide a computer (or CPU or MP) of the system or apparatus.
U) can also be achieved by reading and executing the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy (registered trademark) disk, a hard disk, a magneto-optical disk, a CD-RO
M, CD-R, CD-RW, DVD-ROM, DVD-
A RAM, a DVD-RW, a DVD + RW, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

The functions of the above-described embodiments are implemented when the computer executes the readout program codes, and the OS (Operating System) running on the computer is executed based on the instructions of the program codes. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instruction of the program code, It goes without saying that the CPU or the like provided in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0075]

As described above, according to the present invention,
A natural mixed reality can be obtained by matching the spatial coordinates of the real space and the virtual space and eliminating a positional shift at the time of synthesizing the real image and the CG (computer graphics) image. Special effects can be obtained in games using

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a concept of a mixed reality apparatus (image processing apparatus) according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a mounted state of the mixed reality apparatus (image processing apparatus) according to the first embodiment of the present invention;

FIG. 3 is a diagram for describing a correction processing operation by a correction unit of the mixed reality apparatus (image processing apparatus) according to the first embodiment of the present invention.

FIG. 4 is a block diagram illustrating a concept of a mixed reality apparatus (image processing apparatus) according to a third embodiment of the present invention.

FIG. 5 is a diagram for explaining a correction processing operation by a correction unit of a mixed reality apparatus (image processing apparatus) according to a third embodiment of the present invention.

[Explanation of symbols]

Reference Signs List 101 display means 102 imaging means 103 gaze direction detection means 104 video acquisition means 105 correction means 106 CG video generation means 107 video synthesis means 108 output means 201 head mounted display (HMD) 202 video camera 203 magnetic direction sensor 204 computer 205 CPU (central Arithmetic processing device) 206 memory 207 HDD (hard disk drive: storage device) 208 i / O (input / output device) 209 graphics board 210 video capture board 211 correction program 212 CG video generation program 213 video synthesis program

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/222 H04N 13/00 5C061 13/00 G09G 5/36 520M 5C082 F term (reference) 2C001 BC08 BC10 CA09 CC01 5B050 AA09 BA07 BA08 BA11 BA18 EA07 EA19 EA27 FA02 5B057 CA13 CB13 CC03 CE08 DA07 DA16 DC08 5C022 AB00 AB51 AC69 CA02 5C023 AA06 AA11 AA37 AA38 BA11 CA01 DA08 5C061 AA20 AB02 AB08 CA05 AB12CAB CA8

Claims (28)

[Claims] An image capturing step of capturing a real image at a viewpoint position of a subject in accordance with a gaze direction; an image obtaining step of converting the real image captured by the image capturing step into a form that can be synthesized by a computer; A CG image generation step of generating a CG (computer graphics) image from the viewpoint direction detected by the gaze direction detection step; and a CG image generated by the CG image generation step. An image processing method, comprising: a video synthesizing step of synthesizing a real shot video acquired by the video acquiring step; and a display step of displaying a video synthesized by the video synthesizing step to a subject. Compensation to correct the live-action video before it is combined with video An image processing method comprising a correct step. 2. An image capturing step of capturing a real image at a viewpoint position of a subject in accordance with a line-of-sight direction, an image acquiring step of converting the real image captured by the image capturing step into a format that can be synthesized by a computer, and a line-of-sight direction of the subject. A CG image generation step of generating a CG (computer graphics) image from the viewpoint direction detected by the sight line direction detection step, and a CG image generated by the CG image generation step. An image processing method comprising: a video synthesizing step of synthesizing a real shot video acquired by the video obtaining step; and a display step of displaying a video synthesized by the video synthesizing step to a subject. Compensation for correcting the CG image before it is combined with the image An image processing method comprising a correct step. 3. An image pickup means for picking up a real image at a viewpoint position of a subject in accordance with a line of sight, an image acquisition means for converting the real image picked up by the image pickup means into a format that can be synthesized by a computer, and a line of sight of the subject. Line-of-sight detecting means for detecting CG (computer graphics) images from the viewpoint direction detected by the line-of-sight direction detecting means
G image generating means, image synthesizing means for synthesizing the CG image generated by the CG image generating means and the real image obtained by the image obtaining means, and displaying the image synthesized by the image synthesizing means to the subject An image processing apparatus comprising: a correction unit configured to correct the photographed image before being combined with the CG image by the image combining unit. 4. The image processing apparatus according to claim 3, wherein the correction unit is a lens device that can be attached to the imaging unit. 5. The image processing apparatus according to claim 3, wherein the correction unit is provided between the imaging unit and the video acquisition unit. 6. The image processing apparatus according to claim 3, wherein the correction unit is provided between the video acquisition unit and the video synthesis unit. 7. The image processing apparatus according to claim 3, wherein said imaging means is a video camera. 8. The image processing apparatus according to claim 3, wherein the image acquisition unit is a capture board. 9. The image processing apparatus according to claim 3, wherein said line-of-sight direction detecting means is a position sensor. 10. The image processing apparatus according to claim 3, wherein the CG generation unit is a computer. 11. The image processing apparatus according to claim 3, wherein the image synthesizing unit is a computer. 12. The image processing apparatus according to claim 3, wherein said display means is a graphics board. 13. The image processing apparatus according to claim 3, wherein the display unit is a head mounted display (HMD). 14. An image pickup means for picking up a real image at a viewpoint position of a subject in accordance with a gaze direction, an image acquisition means for converting the real image picked up by the image means into a format that can be synthesized by a computer, and a gaze direction of the subject. Line-of-sight detecting means for detecting CG (computer graphics) video from a viewpoint direction detected by the line-of-sight direction detecting means
G image generating means, image synthesizing means for synthesizing the CG image generated by the CG image generating means and the real image obtained by the image obtaining means, and displaying the image synthesized by the image synthesizing means to the subject An image processing apparatus comprising: a correction unit configured to correct the CG image before being combined with the photographed image by the image combining unit. 15. The apparatus according to claim 1, wherein the correction unit is a lens device attachable to the imaging unit.
5. The image processing device according to 4. 16. The image processing apparatus according to claim 14, wherein the correction unit is provided between the CG video generation unit and the video synthesis unit. 17. The image processing apparatus according to claim 14, wherein the imaging unit is a video camera. 18. The image processing apparatus according to claim 14, wherein the image acquisition unit is a capture board. 19. The image processing apparatus according to claim 14, wherein said line-of-sight direction detecting means is a position sensor. 20. The image processing apparatus according to claim 14, wherein the CG generation unit is a computer. 21. An image processing apparatus according to claim 14, wherein said video synthesizing means is a computer. 22. An image processing apparatus according to claim 14, wherein said video synthesizing means is a computer. 23. The image processing apparatus according to claim 14, wherein said display means is a graphics board. 24. The apparatus according to claim 14, wherein said display means is an HMD (Head Mount Display).
An image processing apparatus according to claim 1. 25. A storage medium storing a computer readable program for controlling an image processing apparatus for creating a mixed reality by combining a real space image and a virtual space image by CG (computer graphics), An imaging step of capturing an actual video at the viewpoint position of the subject in accordance with the gaze direction; an image acquisition step of converting the real video captured by the imaging step into a format that can be synthesized by a computer; and a gaze direction of detecting the gaze direction of the subject A detecting step, a CG image generating step of generating a CG (computer graphics) image from the viewpoint direction detected by the line-of-sight direction detecting step, and a CG image generated by the CG image generating step and the image obtaining step. A video synthesizing system that synthesizes the acquired live-action video And a display step of displaying to the subject the video synthesized by the video synthesis step; and a correction step of correcting the real shot video before being synthesized with the CG video by the video synthesis step. Characteristic storage medium. 26. A storage medium storing a computer readable program for controlling an image processing apparatus for creating a mixed reality by combining a real space image and a virtual space image by CG (computer graphics), An imaging step of capturing an actual video at the viewpoint position of the subject in accordance with the gaze direction; an image acquisition step of converting the real video captured by the imaging step into a format that can be synthesized by a computer; and a gaze direction of detecting the gaze direction of the subject A detecting step, a CG image generating step of generating a CG (computer graphics) image from the viewpoint direction detected by the line-of-sight direction detecting step, and a CG image generated by the CG image generating step and the image obtaining step. A video synthesizing system that synthesizes the acquired live-action video And a display step of displaying the image synthesized by the image synthesizing step to a subject; and a correcting step of correcting the CG image before being synthesized with the real image by the image synthesizing step. Characteristic storage medium. 27. A computer-executable program for controlling an image processing apparatus for creating a mixed reality by synthesizing a real space image and a virtual space image by CG (computer graphics), the program being executed at a viewpoint position of a subject. An image capturing step of capturing a real image in accordance with a line-of-sight direction, an image obtaining step of converting the real image captured in the image capturing step into a format that can be synthesized by a computer, a line-of-sight direction detecting step of detecting a line-of-sight direction of the subject, A CG image generation step for generating a CG (computer graphics) image from the viewpoint direction detected by the gaze direction detection step; a CG image generated by the CG image generation step; and a real shot image obtained by the image acquisition step Synthesizing the image and the image; A program code for causing a computer to execute a display step of displaying an image synthesized by the step to a subject, and a correction step of correcting the real image before being synthesized with the CG image by the image synthesis step. A program characterized by: 28. A computer-executable program for controlling an image processing apparatus that creates a mixed reality by synthesizing a real space image and a virtual space image by CG (computer graphics). An image capturing step of capturing a real image in accordance with a line-of-sight direction, an image obtaining step of converting the real image captured in the image capturing step into a format that can be synthesized by a computer, a line-of-sight direction detecting step of detecting a line-of-sight direction of the subject, A CG image generation step for generating a CG (computer graphics) image from the viewpoint direction detected by the gaze direction detection step; a CG image generated by the CG image generation step; and a real shot image obtained by the image acquisition step Synthesizing the image and the image; A display step of displaying to a subject an image synthesized by the step, and a correction step of correcting the CG image before the image is synthesized by the image synthesis step with the real image. Features program.