JP2009015648A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To more easily perform a process of superimposing a virtual space image only on other regions except a subject region in the real space image when the virtual space image is superimposed on the real space image. To provide.
Color information corresponding to imaging position and orientation information closest to imaging position and orientation information of an imaging apparatus 303 is acquired. And the area | region of the hand in a captured image is specified using this acquired color information. Then, the virtual space image is superimposed on the captured image so that the virtual space image is not superimposed on the specified region.
[Selection] Figure 1

Description

  The present invention relates to a technique for providing mixed reality.

  2. Description of the Related Art In recent years, devices (mixed reality devices) that apply a mixed reality (MR) technology that naturally couples the real world and the virtual world naturally are not proposed. These mixed reality devices present mixed reality to the user of the device as follows. That is, a virtual world image drawn by computer graphics (CG) is synthesized with a real world image taken by an imaging device such as a camera. Then, the synthesized image by the synthesis is displayed on a display device such as a head-mounted display (HMD).

  In order to enhance the mixed reality, these mixed reality devices acquire the viewpoint position and orientation of the user of this device in real time, follow the change of the real world image, generate the virtual world image, It is necessary to display in real time on a display device such as an HMD for the user. In the mixed reality device, the viewpoint position / orientation of the user measured by the sensor device is set as a virtual viewpoint position / orientation in the virtual world, and based on this setting, an image of the virtual world is drawn by CG, and the real world Composite with the image.

  In addition, since the HMD presents mixed reality, the visual field of the user of the mixed reality device includes display by the HMD, and the display includes a region for drawing CG.

  With these technologies, the user of the mixed reality apparatus can observe an image as if a virtual object exists in the real world. However, when a real world image and a virtual world image are synthesized, one's hand may be hidden by the virtual world image, which impairs the reality of mixed reality.

  A method for improving the reality of such mixed reality with a small processing load is disclosed in Patent Document 1. In the technology disclosed in Patent Document 1, the virtual world image is not superimposed on the user's hand among the images in the real space, and a more realistic mixed reality experience is experienced (hand). Overlay function).

  However, in the conventional technology, since the image of the user's hand is determined using a single parameter such as specific color information, the user's viewpoint position and posture change, and the space in which the user is located In many cases, sufficient effects could not be obtained with respect to changes in lighting conditions. Specifically, when there is an image close to the color of the hand in an image other than the user's hand, the image in the virtual world is not superimposed on the portion of the image. The problem of appearing unnatural occurred frequently. Conversely, depending on the position and orientation of the user's hand, the color of the virtual world is superimposed on the user's hand because the user's hand cannot be recognized due to a different color due to changes in lighting or the like. The problem that the user's hand disappeared frequently occurred.

There is also a technique for simulating real-world lighting conditions and approximating the virtual-world CG lighting conditions to the simulation results. However, the processing time until the simulation result is calculated is very long, and it is difficult to use the mixed reality presentation device for the purpose of experiencing mixed reality in real time. Even if the hardware capability is improved in the future, the processing load of the mixed reality presentation device itself is pressed, which is not practical.
JP 2003-296759 A

  As described above, the prior art has the following problems when, for example, a composite image in which CG is superimposed only on a region excluding a subject region in a real-world image is generated.

  First, using a single piece of parameter information regardless of the position and orientation of the imaging device, the subject region is appropriately extracted from the real-world image at the position and orientation of the imaging device where the illumination conditions change. Will be difficult. As a result, there arises a problem that the subject area is hidden by the CG, or the CG is not superimposed on an area other than the subject area. This makes it difficult to output an accurate composite image.

  Also, it is difficult to generate a plurality of parameter information that is practical. Furthermore, it is difficult to display the composite image in which the generated CG image is superimposed by properly using the generated parameter information and accurately removing the subject area.

  In addition, processing that simulates lighting conditions has a high processing load, and it has been difficult to provide a composite image in which a CG image is appropriately superimposed in real time to the user of the mixed reality presentation apparatus.

  The present invention has been made in view of the above problems, and when a virtual space image is superimposed on a real space image, the virtual space image is only applied to other regions other than the subject region in the real space image. An object is to provide a technique for performing the process of superimposing more simply.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, for each of a plurality of images obtained by performing the imaging of the subject a plurality of times with different imaging positions and orientations, the color information in the region of the subject in the image, the imaging location and orientation information of the images, and Storage holding means for storing and holding a set,
First acquisition means for acquiring imaging position and orientation information of an imaging device that images a real space including the subject;
Second acquisition means for acquiring a captured image captured by the imaging device;
Third acquisition means for acquiring from the storage holding means a set including imaging position and orientation information closest to the imaging position and orientation information acquired by the first acquisition means;
Specifying means for specifying the area of the subject in the captured image acquired by the second acquisition means using the color information in the set acquired by the third acquisition means;
Generating means for generating a virtual space image based on the imaging position and orientation information acquired by the first acquisition means;
Superimposing means for superimposing the virtual space image on the captured image acquired by the second acquisition means so that the virtual space image is not superimposed on the area specified by the specifying means;
Output means for outputting a composite image obtained by the superimposing means.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, for each of a plurality of images obtained by performing the imaging of the subject a plurality of times with different imaging positions and orientations, the color information in the region of the subject in the image, the imaging location and orientation information of the images, and , And an image processing method performed by an image processing apparatus stored and held in a memory,
A first acquisition step of acquiring imaging position and orientation information of an imaging apparatus that images the real space including the subject;
A second acquisition step of acquiring a captured image captured by the imaging device;
A third acquisition step of acquiring from the memory a set including imaging position and orientation information closest to the imaging position and orientation information acquired in the first acquisition step;
Using the color information in the set acquired in the third acquisition step, a specifying step for specifying the region of the subject in the captured image acquired in the second acquisition step;
A generation step of generating a virtual space image based on the imaging position and orientation information acquired in the first acquisition step;
A superimposing step of superimposing the virtual space image on the captured image acquired in the second acquisition step so that the virtual space image is not superimposed on the region specified in the specifying step;
And an output step of outputting a composite image obtained in the superimposing step.

  According to the configuration of the present invention, when the virtual space image is superimposed on the real space image, the process of superimposing the virtual space image only on the other region excluding the subject region in the real space image can be simplified. Can be done.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a hardware configuration of a system according to this embodiment for providing a mixed reality to a user. As shown in FIG. 1, the system according to the present embodiment includes a computer 100, a display unit 200, an operation input unit 400, an HMD 300, a sensor 600, and a sensor control unit 500.

  First, the computer 100 will be described.

  The CPU 101 controls the entire computer 100 using programs and data stored in the RAM 102, and executes or controls each process described later as what the computer 100 performs.

  The RAM 102 has an area for temporarily storing programs and data loaded from the disk device 105 and data received via the input device 106 and the image capturing device 107. Furthermore, the RAM 102 also has a work area used when the CPU 101 executes each process. That is, the RAM 102 can provide various areas as appropriate.

  The image generation device 103 performs a series of processes for generating a virtual space image, a series of processes for superimposing the virtual space image on the real space image, and the generated superimposed image (composite image). Processing for outputting to the HMD 300 and the display unit 200 is performed. The image generation device 103 is configured by, for example, a graphics card.

  The disk device 105 is a large-capacity information storage device represented by a hard disk drive device. The disk device 105 stores an OS (Operating System) and programs and data for causing the CPU 101 to execute or control each process described below as performed by the computer 100. Such data includes data of each virtual object constituting the virtual space.

  The input device 106 is for connecting an operation input unit 400 and a sensor control unit 500 described later to the computer 100. Data sent from the operation input unit 400 or the sensor control unit 500 is stored in the disk device 105 or the RAM 102 via the input device 106.

  The image capturing device 107 receives a real space image (captured image) sent from the imaging device 303 included in the HMD 300 described later, and sends it to the RAM 102 and the disk device 105. The image capturing device 107 is constituted by, for example, a video capture card.

  A bus 104 connects the above-described units.

  Next, the operation input unit 400 will be described.

  The operation input unit 400 is configured by a keyboard, a mouse, and the like, and can input various instructions to the computer 100 when operated by an operator of the computer 100. Various instructions input using the operation input unit 400 are input to the input device 106 as data, and the input device 106 notifies the CPU 101 of this data.

  Next, the display unit 200 will be described.

  The display unit 200 includes a CRT, a liquid crystal screen, and the like, and displays various information to the operator of the computer 100.

  Next, the HMD 300 will be described.

  The HMD 300 includes a display unit 301, a measurement target 302, and an imaging device 303. The HMD 300 is an example of a so-called head-mounted display device that is worn on the head of a user who experiences mixed reality. In this embodiment, one HMD 300 is connected to the computer 100. However, when a plurality of HMDs 300 are connected to the computer 100, each process described later may be performed for each HMD 300. .

  The display unit 301 is for displaying an image sent from the computer 100 (a composite image generated by the image generation apparatus 103), and is attached to the HMD 300 so that the HMD 300 is positioned in front of the user wearing the head. It has been. Therefore, such a composite image is displayed in front of the user wearing the HMD 300 on the head.

  The measurement target 302 is attached to the HMD 300 in order to measure its position and orientation using the sensor 600 and the sensor control unit 500. Details of the measurement target 302 will be described later.

  The imaging device 303 is a camera that can capture a moving image or a still image in the real space. The captured image of each frame (real space image) or the still image (real space image) is an image capturing device included in the computer 100. It is sent to 107.

  Next, the sensor 600 and the sensor control unit 500 will be described.

  The measurement target 302, the sensor 600, and the sensor control unit 500 may be any sensor system, but the purpose is to acquire the position and orientation of the HMD 300. Therefore, any sensor system may be used as long as the object can be achieved, and the object may be achieved by a method other than the sensor system. In this embodiment, a magnetic sensor system is used as the sensor system. As a magnetic sensor system, a magnetic sensor Fastrak manufactured by Polhmus can be used.

  In this case, the sensor 600 is a magnetic field transmission source, and the measurement object 302 detects a change in magnetism according to its position and orientation in the magnetic field. The detection result is sent to the sensor control unit 500. The sensor control unit 500 obtains position and orientation information of the measurement target 302 in the sensor coordinate system based on the detection result. The sensor coordinate system is a coordinate system in which the position of the sensor 600 is the origin, and the three axes orthogonal to each other at the origin are the x axis, the y axis, and the z axis. Then, the sensor control unit 500 sends the obtained position and orientation information to the input device 106 included in the computer 100.

  Moreover, as a sensor system other than the magnetic sensor system, an optical system can be used. In this case, the measurement object 302 of the optical sensor OPTOTRAK 3020 of Northern Digital is constituted by a light emitting device and emits light according to control by the sensor control unit 500. The sensor 600 is constituted by a light receiving device, and observes light emitted from the measurement object 302. The sensor control unit 500 obtains position and orientation information of the measurement target 302 based on the observation result.

  As another example of the optical sensor system, in the case of the optical sensor HiBall manufactured by 3rdTech, the measurement object 302 is constituted by a light receiving device, and the sensor 600 is constituted by a light emitting device. The sensor 600 emits light according to control by the sensor control unit 500. The measurement target 302 observes light emitted by the sensor 600. The sensor control unit 500 obtains position and orientation information of the measurement target 302 based on the observation result.

  The position and orientation information of the measurement target 302 can be obtained using any of the various sensor systems described above. Since any sensor system has a well-known mechanism for obtaining position and orientation information, a description thereof will be omitted.

  Note that since the position and orientation information of the imaging device 303 is finally obtained instead of the position and orientation information of the measurement target 302, the position and orientation relationship (bias) between the measurement target 302 and the imaging device 303 needs to be obtained in advance. . Such bias data is registered in the disk device 105 in advance. When the position / orientation information of the measurement target 302 is obtained, the position / orientation information of the imaging device 303 can be obtained by applying a bias to the position / orientation information.

  On the other hand, there is also a method for obtaining position and orientation information of the imaging device 303 using image processing without using a sensor system, as will be described below.

  A marker whose placement position is known is placed in the real space. When the imaging device 303 captures a real space image including the marker, the marker is extracted from the real space image, and the position and orientation information of the imaging device 303 is obtained from the extracted coordinates and the known arrangement position. . Note that natural features in the real space may be used instead of the markers whose arrangement positions are known.

  In addition, a marker is attached to the imaging device 303, and the marker is imaged by an imaging device arranged in the real space. Then, a marker may be extracted from the captured image, and the position / orientation information of the imaging device 303 may be obtained based on the extracted coordinates.

  Further, a user wearing the HMD 300 on the head may be imaged by the imaging device 303, the user's natural features may be extracted from the captured image, and the position and orientation information of the imaging device 303 may be obtained based on the extracted coordinates. .

  As described above, the position and orientation information of the imaging device 303 can be obtained using image processing. Of course, the sensor system and the image processing may be used in appropriate combination.

  In the present embodiment, when a virtual space image is superimposed on a real space image using the system having the configuration shown in FIG. 1, a virtual space image is used for an area of a hand reflected as a subject in the real space image. Avoid overlapping. That is, the hand overlay function is performed. At this time, it is necessary to extract a hand region from the real space image. In this embodiment, the hand information is used by using color information determined based on the position and orientation information of the imaging device 303 when the real space image is captured. Extract the region. This is because the visible color differs depending on the position and orientation of the imaging device 303 when imaging the same hand. As a result, it is possible to generate an appropriate composite image as compared with the case where single parameter information is used as in the prior art. In particular, when the hand overlay function is used, a composite image can be displayed more accurately and a highly effective hand overlay function can be provided.

  First, before actually providing the user with mixed reality, when a hand is imaged from various positions and orientations, it is necessary to perform processing for registering the color information in the disk device 105. FIG. 2 is a diagram illustrating how a hand is imaged from various positions and orientations.

  As shown in FIG. 2, a light source 2001 such as a fluorescent lamp or natural light exists in the real space. Although FIG. 2 shows only the light source 2001 as the light source, there are many cases in practice. Therefore, if the imaging position / posture for imaging the hand is changed, the color of the observed hand is likely to change. Therefore, in the present embodiment, as shown in FIG. 2, the user 2000 wearing the HMD 300 on the head moves to various positions and postures and images his / her hand with the imaging device 303. Of course, the colors of the hands in the images taken at the respective positions and postures are often slightly different. Then, the color information of the hand region extracted from each image and the imaging position and orientation information of each image are registered as a set in the disk device 105.

  FIG. 3 shows a disk device in which, when a plurality of images are obtained by imaging a hand from various positions and orientations, hand color information and imaging position and orientation information are set from each image. 10 is a flowchart of processing to be registered in 105. Note that a program and data for causing the CPU 101 to execute or control the process according to the flowchart shown in FIG. Such programs and data are appropriately loaded into the RAM 102 under the control of the CPU 101. Then, the CPU 101 executes or controls the process using the loaded program and data, thereby realizing the process according to the flowchart shown in FIG.

  First, in step S301, the entire computer 100 is initialized. In the initialization process, various programs and data used in each process described below are loaded from the disk device 105 to the RAM 102. Of course, initialization is also performed for devices other than the computer 100.

  Next, when the user wearing the HMD 300 on the head inputs an imaging instruction using the operation input unit 400 and the image capturing apparatus 107 detects the imaging instruction, in step S302, the image capturing apparatus 107 Perform the following process. That is, the real space image for one frame sent from the imaging device 303 is transferred (acquired) to the RAM 102.

  Further, when the input device 106 detects such an imaging instruction, the input device 106 transfers (acquires) the position and orientation information sent from the sensor control unit 500 to the RAM 102 in step S303. After the position / orientation information is acquired in the RAM 102, the position / orientation information of the imaging device 303 can be obtained by applying the bias to the acquired position / orientation information.

  Here, the imaging instruction is not limited to being input using the operation input unit 400, and various input methods can be considered. For example, when a user performs a predetermined hand gesture, an imaging instruction may be automatically input if such a hand gesture is detected.

  Next, in step S304, the hand region shown in the real space image transferred to the RAM 102 in step S302 is extracted. Then, the value of each pixel constituting the extracted hand region is described as, for example, a distribution on the CrCb plane, and is obtained as color information specifying the coordinate range of each axis of the color space. There are various methods for extracting the hand region. For example, the contour of the hand in the real space image may be obtained using edge extraction processing or the like, or the real space image is displayed on the display unit 200 and the operator of the computer 100 uses the operation input unit 400 to display the hand. An area may be indicated. Alternatively, the virtual space image may be displayed in a superimposed manner on the hand, and the color information may be selected so that the virtual space image is accurately superimposed on an area other than the real image of the hand. As an example of selection, for example, there is a method as disclosed in JP-A-2005-107967.

  Next, in step S305, the position / orientation information of the imaging device 303 acquired in step S303 and the color information acquired in step S304 are registered as a set and stored in the disk device 105 (stored).

  In step S306, the operation input unit 400 is used to check whether an instruction to end the process is input or whether any end condition is satisfied. As a result of the check, when an instruction to end the process is input using the operation input unit 400, or when any end condition is satisfied, the process ends. On the other hand, if the end instruction of this process is not input using the operation input unit 400 and any end condition is not satisfied, the process returns to step S302 to perform the next set registration process.

  Thus, for each of a plurality of images obtained by performing the hand imaging a plurality of times with different imaging positions and orientations, the color information in the hand region in the images, the imaging position and orientation information of the images, and , Can be registered in the disk device 105 as a set.

  Next, a composite image obtained by superimposing the virtual space image on the real space image is displayed to the user wearing the HMD 300 on the head, which is performed after the processing according to the flowchart of FIG. Processing for providing will be described.

  FIG. 4 is a diagram for sending a composite image in which a virtual space image is superimposed on a real space image to the display unit 301 of the HMD 300 so that the virtual space image is not superimposed on the hand region in the real space image. It is a flowchart of a process. Note that programs and data for causing the CPU 101 to execute or control the processing according to the flowchart shown in FIG. 4 are stored in the disk device 105. Such programs and data are appropriately loaded into the RAM 102 under the control of the CPU 101. Then, the CPU 101 executes or controls the process using the loaded program and data, thereby realizing the process according to the flowchart shown in FIG.

  In step S401, the entire computer 100 is initialized. In the initialization process, various programs and data used in each process described below are loaded from the disk device 105 to the RAM 102. Of course, initialization is also performed for devices other than the computer 100.

  Next, in step S <b> 402, the image capturing device 107 transfers (second acquisition) the real space image for one frame sent from the imaging device 303 to the RAM 102. Then, the CPU 101 performs an appropriate correction process on the transferred real space image as necessary, and then transfers the real space image to a frame buffer on the graphics memory included in the image generation device 103. Note that the real space image sent from the imaging device 303 may be directly transferred to the frame buffer without going through the RAM 102.

  In step S <b> 403, the input device 106 transfers (first acquisition) the position and orientation information sent from the sensor control unit 500 to the RAM 102. After the position / orientation information is acquired in the RAM 102, the position / orientation information of the imaging device 303 is obtained by applying the bias to the acquired position / orientation information. As described above, since there are various types of processing for acquiring the position and orientation information of the imaging device 303, there are some processes that do not need to be biased.

  Next, in step S404, it is checked whether or not the set group described as being registered in the disk device 105 according to the flowchart shown in FIG. 3 is actually registered in the disk device 105.

  That is, it is preferable to register the set group in the disk device 105 by performing the process according to the flowchart shown in FIG. 3 before the process according to the flowchart shown in FIG. However, in the present embodiment, a case where such a set group is not registered in the disk device 105 is dealt with.

  If it is registered as a result of the check in step S404, the process proceeds to step S409, and if not registered, the process proceeds to step S405.

  In step S405, color information registered in the disk device 105 is selected as default color information.

  On the other hand, in step S409, the set including the position and orientation information closest to the position and orientation information of the imaging device 303 acquired in step S403 is identified (third acquisition) from the set group. Here, the specific process will be described. Strictly speaking, it is preferable that a set including the position and orientation information exactly the same as the position and orientation information of the imaging device 303 acquired in step S403 can be specified, but in practice it is almost impossible. Therefore, for example, paying attention only to the position information, out of the position information in the position and orientation information in each set of the set group, the position information in the position and orientation information of the imaging device 303 acquired in step S403. You may identify the set containing the positional information with the smallest difference. Alternatively, focusing only on the posture information, the difference between the posture information in the position and posture information of the imaging device 303 acquired in step S403 out of the posture information in the position and posture information in each set of the set group is the difference. A set including the smallest posture information may be specified. Of course, you may pay attention to both position information and posture information.

  Then, the color information in the specified set is selected.

  Here, another form is demonstrated as a process in step S409. When creating the set group, if the space in which the user wearing the HMD 300 is movable is divided into unit volumes and color information is collected for each divided space, the processing in step S409 is performed. Can be done as follows. That is, if it is possible to specify which divided space the position information acquired in step S403 belongs to, it is only necessary to select the color information collected in the specified divided space.

  If the user who wears the HMD 300 on the head changes the posture of the imaging device 303 every 30 degrees and collects color information when creating the set group, step S409 is performed. The processing in can be performed as follows. For example, when the posture information acquired at step S403 is 10 degrees, the color information collected at the posture 0 degrees is selected, and when the posture information acquired at step S403 is 40 degrees, the posture 30 The color information collected at a time may be selected.

  In step S406, the image generation apparatus 103 detects the hand region in the real space image (captured image) acquired in step S402 using the color information selected in step S405 or step S409. . Such detection processing is disclosed in, for example, Japanese Patent Laid-Open No. 2003-296759, and is a well-known technique.

  In step S407, the image generation apparatus 103 first sets a viewpoint having the position and orientation indicated by the position and orientation information of the imaging apparatus 303 acquired in step S403 in the virtual space, and generates a virtual space image that can be seen from the viewpoint. To do. Since processing for generating a virtual space image viewed from a viewpoint having a predetermined position and orientation is well known, description thereof will be omitted.

  Then, the generated virtual space image is superimposed on the real space image (captured image) acquired in step S402. In this superposition, the virtual space image is superimposed on the area in the real space image detected in step S406. The virtual space image is not superimposed on the. In other words, the process of superimposing the virtual space image on the real space image is to draw the virtual space image on the real space image transferred to the frame buffer in step S402. As a result, a composite image is generated on the frame buffer.

  In step S407, the composite image is sent to the display unit 301 included in the HMD 300.

  By the processing described above, it is possible to detect the hand region in the real space image in consideration of the hand color that is visible according to the position and orientation of the imaging device 303, so that the hand region can be detected more accurately. Can do. Therefore, when the virtual space image is superimposed on the real space image, the process of not superimposing the virtual space image on the hand region can be performed more accurately.

  Next, in step S408, it is checked whether an end instruction for this process is input using the operation input unit 400 or whether any end condition is satisfied. As a result of the check, when an instruction to end the process is input using the operation input unit 400, or when any end condition is satisfied, the process ends. On the other hand, if the end instruction of this process is not input using the operation input unit 400 and any end condition is not satisfied, the process returns to step S402 to process the next frame.

  As described above, according to the present embodiment, the hand region can be accurately detected even when the illumination condition for the hand changes due to the change in the position and orientation of the imaging apparatus that observes the hand. In addition, as in the prior art, it is possible to accurately detect a hand region without performing a heavy load on the computer, such as simulating lighting conditions for the hand.

[Second Embodiment]
In the first embodiment, when the set group is registered in the disk device 105, the user wearing the HMD 300 on the head moves in the space. However, such work is not limited to being performed by a single user, and a plurality of users may use images taken by other users or images taken by a so-called objective camera.

  Further, in the first embodiment, the processing according to the flowchart shown in FIG. 4 is performed after the processing according to the flowchart shown in FIG. 3, but in parallel with the processing according to the flowchart shown in FIG. Then, processing according to the flowchart shown in FIG. 3 may be performed.

  Of course, when implementing the modification of 1st Embodiment demonstrated in this embodiment, what is necessary is just to change suitably the content demonstrated in 1st Embodiment.

  Furthermore, in the first embodiment, when selecting one of the set groups registered in the disk device 105, the imaging position / orientation information is used. However, when imaging position and orientation information indicating the same imaging position and orientation is registered in the plurality of disk devices 105, one set may be selected based on the color distribution of the real space image during operation. good.

  In the first and second embodiments, a hand is used as an example of the subject. However, even if another hand is used as the subject, the description in the first and second embodiments is not essentially changed.

[Other Embodiments]
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the hardware constitutions of the system which concerns on the 1st Embodiment of this invention for providing a user with mixed reality. It is a figure which shows a mode that a hand is imaged from various position and orientation. When a plurality of images are obtained by imaging the hand from various positions and orientations, the hand color information and the imaging position and orientation information are registered from each image and registered in the disk device 105. It is a flowchart of a process. In the flowchart of the process for sending a composite image in which the virtual space image is superimposed on the real space image to the display unit 301 of the HMD 300 so that the virtual space image is not superimposed on the hand region in the real space image. is there.

Claims (6)

For each of a plurality of images obtained by performing the imaging of the subject a plurality of times with different imaging positions and orientations, color information in the region of the subject in the image and imaging location and orientation information of the images are obtained. Memory holding means for storing and holding as a set;
First acquisition means for acquiring imaging position and orientation information of an imaging device that images a real space including the subject;
Second acquisition means for acquiring a captured image captured by the imaging device;
Third acquisition means for acquiring from the storage holding means a set including imaging position and orientation information closest to the imaging position and orientation information acquired by the first acquisition means;
Specifying means for specifying the area of the subject in the captured image acquired by the second acquisition means using the color information in the set acquired by the third acquisition means;
Generating means for generating a virtual space image based on the imaging position and orientation information acquired by the first acquisition means;
Superimposing means for superimposing the virtual space image on the captured image acquired by the second acquisition means so that the virtual space image is not superimposed on the area specified by the specifying means;
An image processing apparatus comprising: output means for outputting a composite image obtained by the superimposing means.   The third acquisition unit is the imaging position / orientation information acquired by the first acquisition unit among the position information of the position / orientation information in each set of the set group stored and held by the storage / holding unit. The image processing apparatus according to claim 1, wherein a set including position information having the smallest difference from the position information is identified and acquired.   The third acquisition unit is the imaging position / orientation information acquired by the first acquisition unit among the posture information of the position / orientation information in each set of the set group stored and held by the storage / holding unit. The image processing apparatus according to claim 1, wherein a set including posture information having the smallest difference from the posture information is identified and acquired. For each of a plurality of images obtained by performing the imaging of the subject a plurality of times with different imaging positions and orientations, color information in the region of the subject in the image and imaging location and orientation information of the images are obtained. An image processing method performed by an image processing apparatus stored in a memory as a set,
A first acquisition step of acquiring imaging position and orientation information of an imaging apparatus that images the real space including the subject;
A second acquisition step of acquiring a captured image captured by the imaging device;
A third acquisition step of acquiring from the memory a set including imaging position and orientation information closest to the imaging position and orientation information acquired in the first acquisition step;
Using the color information in the set acquired in the third acquisition step, a specifying step for specifying the region of the subject in the captured image acquired in the second acquisition step;
A generation step of generating a virtual space image based on the imaging position and orientation information acquired in the first acquisition step;
A superimposing step of superimposing the virtual space image on the captured image acquired in the second acquisition step so that the virtual space image is not superimposed on the region specified in the specifying step;
And an output step of outputting a composite image obtained in the superimposing step.   A program for causing a computer to execute the image processing method according to claim 4.   A computer-readable storage medium storing the program according to claim 5.

Images