JP2003141569A - Information processing method and video composition apparatus - Google Patents

Abstract

[57] An object of the present invention is to make it possible to easily create a three-dimensional model of a real object. An information processing method for setting, using an adjustment screen, a three-dimensional model of an object included in an image showing the real world, which is used to synthesize a computer image showing the virtual world with an image showing the real world. An image showing a real world including the object obtained from a plurality of viewpoints is acquired, and based on spatial position information of the image showing the plurality of real worlds and position information of a three-dimensional model corresponding to the object , A three-dimensional model corresponding to the object is synthesized with each of the plurality of images showing the real world, displayed on the adjustment screen, and based on the plurality of synthesized images, a user manually instructs the object in the synthesized image. The position and shape of the three-dimensional model corresponding to are adjusted.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION Setting a three-dimensional model of an object included in an image showing the real world, which is used for synthesizing a computer image showing the virtual world with an image showing the real world, using an adjustment screen. Regarding

[0002]

2. Description of the Related Art A technique for sharing a mixed reality space with a plurality of people is disclosed in Japanese Patent Application Laid-Open No. 2000-350865 "Mixed Reality Game Device, Image Processing Method and Program Storage Medium". In the game device disclosed herein, the data has the same geometric shape as the object existing in the real world, and the three-dimensional model of the real world for functioning as a mask is input and held. The three-dimensional model of the real world is converted into depth data of the real world from the viewpoint of the user. This depth data of the real world is used for the processing for determining the portion that should be shielded by the real object when rendering the virtual object as a three-dimensional CG and for not rendering that portion of the virtual object. By this processing, the virtual object is drawn as a three-dimensional CG so that the three-dimensional front-rear relationship with the real object is correctly represented.

[0003]

However, in the case of applying the technique disclosed above, both a real object and a virtual object need three-dimensional geometric shapes, and these must be input to the device. However, inputting an accurate geometric shape requires a great deal of labor and cost, and improvement has been required.

It is desirable that the three-dimensional model of the real object has the same geometric shape and size as the object existing in the real world, and is appropriately arranged with respect to the corresponding position and orientation in the real world. This geometry, size,
If the position and orientation are different from the actual ones, when the three-dimensional model of the real object is converted into the depth data of the real world from a certain viewpoint position, it will not match the image of the real world from that viewpoint. As a result, the concealment relationship between the real object and the virtual object is not properly expressed. For example, in the vicinity of the boundary between a real object and a virtual object, a part of the virtual object that should originally be hidden may be seen by being drawn on the real world, or conversely, a part of the real object that should not be hidden is a virtual object. The phenomenon that it is hidden by occurs. Further, in the case where a plurality of people share a mixed reality space, depth data that matches the image of the real world is necessary not only from one viewpoint but also from all viewpoints. However, when creating this depth data from a three-dimensional model of a physical object, in addition to creating an accurate three-dimensional model, the position and orientation in the real world for arranging the three-dimensional model of the physical object must be accurate. It is difficult to measure and requires a lot of cost.

The present invention has been made in view of the above points, and it is an object of the present invention to easily create a three-dimensional model of a physical object.

[0006]

The video synthesizing apparatus according to the present embodiment for achieving the above object has the following configuration.

According to the first aspect of the present invention, a three-dimensional model of an object included in an image showing the real world, which is used for synthesizing an image showing the real world with a computer image showing the virtual world, is displayed on an adjustment screen. An information processing method set by using, acquiring an image showing a real world including the object, obtained from a plurality of viewpoints, corresponding to spatial position information of the image showing the plurality of real world, and the object. Based on the position information of the three-dimensional model, a three-dimensional model corresponding to the object is combined with each of the plurality of images showing the real world, displayed on the adjustment screen, and based on the plurality of combined images, the user It is characterized in that the position and the shape of the three-dimensional model corresponding to the object in the composite image are adjusted by manual instruction.

According to the invention of claim 8 of the present application, a three-dimensional model of an object included in an image showing the real world, which is used for synthesizing an image showing the real world with a computer image showing the virtual world, is displayed on an adjustment screen. An information processing method that is set using, acquiring an image showing a real world including the object, displaying the image on the adjustment screen, from a plurality of stereo models displayed on the adjustment screen, based on a user instruction An arbitrary stereo model is selected, the selected stereo model is combined with the image showing the real world, displayed on the adjustment screen, and the stereo combined with the image showing the real world according to a user instruction. An information processing method characterized by adjusting the position and shape of a model.

According to a tenth aspect of the present invention, an image pickup section for photographing the real world, an instruction section for supporting the image pickup section, a posture measuring section for measuring the posture of the image pickup section, and a computer image are generated. A computer that generates a computer image using the data held in the data unit based on the data unit that holds the data of 3D model data of the real world and the posture information measured by the posture measuring unit. An image generation unit, an image synthesis unit that synthesizes the captured image captured by the imaging unit and the computer image generated by the computer image generation unit based on the three-dimensional model data of the real world; The present invention is characterized by further comprising a three-dimensional model creation unit for creating and adjusting a dimensional model using a photographed image photographed by the photographing unit.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of this embodiment will be described below with reference to the accompanying drawings.

<First Embodiment> The image synthesizing apparatus of the present embodiment is used as a telescope in an observatory or the like, or as an information display terminal in a museum or the like. FIG. 1 is a block diagram showing the schematic arrangement of the video composition apparatus according to the present embodiment.

Reference numeral 1 denotes an image pickup section, which is composed of, for example, a camera. The image capturing unit 1 captures an image of the real world such as an outdoor landscape or an indoor exhibit, and outputs the captured image as a video signal to the captured image capturing unit 2. Reference numeral 2 is a captured image capturing unit, and the image capturing unit 1
The video signal output from the above is converted into a format suitable for the video synthesizing unit 3 and sent to the video synthesizing unit 3. An image synthesizing unit 3 synthesizes the photographic image from the photographic image capturing unit 2 and the computer image from the computer image generating unit 7 and sends it to the display unit 9.

A pan head controller 4 is installed on the floor,
Supports the imaging unit 1. The camera platform control unit 4 has a movable part, and allows a user to manually rotate the imaging unit 1 freely in the left-right direction and the up-down direction. The platform controller 4 may be fixed to the floor or the like, but may be mounted on a rail or a carriage so as to be movable. Further, the platform control unit 4 is composed of a drive unit (not shown) and a control device, and it is possible to automatically rotate freely in the left and right direction and the up and down direction by a control signal from the outside. The drive part comprises, for example, a motor or a hydraulic or pneumatic jack. A posture measuring unit 5 measures the relative postures of the platform control unit 4 and the image pickup unit 1. The posture information of the image pickup unit 1, that is, the rotation information of the image pickup unit 1 in the vertical direction and the horizontal direction is obtained by a computer image generation unit. It is sent to the computer image generation unit 7 in response to a request from the computer 7 or without a request from the computer image generation unit 7. The attitude measuring unit 5 is composed of, for example, a mechanical encoder. The posture measuring unit 5 may have any configuration as long as it can measure the relative postures of the support unit and the imaging unit, and it is also possible to use, for example, a gyro or an optical sensor.

Reference numeral 6 denotes a lens state control unit, which receives lens state information such as a zoom value and a focus value of a lens used in the image pickup unit 1 in response to a request from the computer image generation unit 7 or from the computer image generation unit 7. It is sent to the computer image generation unit 7 without request. Here, the zoom value / focus value is a value output by an encoder or the like in the lens state control unit 6, and, for example, when the lens state control unit 6 includes an 8-bit encoder,
The output value is in the range of 1 to 256. The lens state control unit 6 may include, for example, a microcomputer, calculate the focal length of the lens using the value output by the encoder, and output the calculated value as the lens state information. The lens state control unit 6 operates the state of the lens by an operation unit (not shown) or a control signal. The lens state control unit 6 includes, for example, a mechanical encoder or a stepping motor.

Reference numeral 7 denotes a computer image generation unit, which takes out the posture information of the image pickup unit 1 from the posture measurement unit 5 and picks up the image pickup unit 1.
Is estimated, the lens state information is extracted from the lens state control unit 6, and the angle of view of the image pickup unit 1 is estimated. If the direction in which the image pickup unit 1 is facing and the angle of view can be estimated, the visual field of the image pickup unit 1 can be obtained. Therefore, the computer image generation unit 7 extracts the data of the portion within the visual field of the image pickup unit 1 from the data unit 8 and Generate a computer image that overlaps the field of view. The generated computer image is sent to the image synthesizing unit 3.

Reference numeral 8 denotes a data section, which is composed of a hard disk, for example, and has data to be transferred to the computer image generation section 7. The data stored in the data unit 8 may be three-dimensional CG data of a virtual object, depth data of the real world, three-dimensional model of the real world, text information, panoramic video, and the like. The data unit 8 sends appropriate data to the computer image generation unit 7 in response to a request from the computer image generation unit 7. For example, the computer image generation unit 7 combines the field of view of the image pickup unit 1 with 3
When requesting the three-dimensional CG data, the data unit 8 extracts the three-dimensional CG data included in the visual field of the imaging unit 1 from the stored three-dimensional CG data and sends it out. The data section 8 is not limited to a hard disk, and may be any medium capable of holding data, such as a tape or a memory.

When a three-dimensional CG (computer graphics) is stored in the data section 8, the stored three-dimensional CG may be, for example, a virtual character.
Here, the three-dimensional CG data stored in the data unit 8 includes not only geometric information such as position, size, shape, and posture, but also
Data such as texture data, constraint relationships, movements, and identification information for identifying each CG data are also included. For example, by storing the shape and movement information of the virtual character in the data unit 8, it becomes possible to make the virtual character appear in the real world.

In the data section 8, the depth data of the real world which represents the depth information of the real world is inputted in advance and used. By using and using the depth data of the real world in advance, it is possible to synthesize a video that accurately reflects the interrelationship between the real world video and the computer video, including the front-back and contact relationships. For example, when it is desired to move a three-dimensional CG character between a building and a certain building, the whole body of the three-dimensional CG character will be displayed in front of the building if the depth data of the real world is not included. Figure 7
The front and rear relationship between the building and the character is not correctly reflected as in. On the other hand, if you have depth data in the real world,
By comparing the depth up to the character and the depth up to the building, and not generating the computer image of the part where the depth up to the character is larger than the depth of the building, the front-rear relationship between the building and the character is correctly reflected as shown in FIG. A synthetic image can be generated.

To give depth information to the real world, that is, to obtain depth data of the real world can be realized by using a three-dimensional model of the real world. When the depth information of the real world is given as a three-dimensional model, it is possible to express a contact relationship such as placing a three-dimensional CG character on a building as shown in FIG. 9, for example.
The three-dimensional model in the real world includes shape data, position and orientation data, identification information for identifying each model, and the like.

Reference numeral 9 denotes a display unit, which displays the composite video signal sent from the video composition unit 3. At this time, if the display unit 9 moves integrally with the image pickup unit 1, it is intuitive and easy for the user to directly look into a telescope or the like, but this is not always necessary. Alternatively, the display unit 9 may be fixed while the display unit 1 is movable. Also,
A plurality of display units 9 may be provided, in which case a large number of people can view the composite video.

FIG. 2 is a block diagram showing the configuration in the case where the image synthesizing devices are mutually connected via a communication network.

Reference numeral 10 denotes a video device, which is a device including all the functions 1 to 9 in FIG.

Reference numeral 11 is a communication network, and a plurality of video devices 1
0s are connected to each other, and data such as images, videos, and numerical values are transferred between the video devices 10 as needed. The communication network 11 is a LAN (Local Area Network).
A communication network for a relatively narrow range such as k: a local area communication network) may be independently constructed, or a part of the Internet or a WAN (Wide Area Network) may be configured. It may be in any form. FIG. 2 shows a configuration diagram when a plurality of video devices 10 are interconnected by a communication network 11. The video device 10 and the communication network 11 in FIG. 2 are the video device 1 in FIG.
0, it has the same function as the communication network 11.

Reference numeral 12 is a video device management unit. The image device management unit 12 also acquires the lens state control data and the attitude data of each image device 10 acquired from the lens state control unit 6 and the attitude measurement unit 5 in each image device 10 and held by the data management unit 13. In addition, the state of the entire system is managed, and a signal for controlling the lens state control unit 6 and the platform control unit 4 is sent to each image device 10 as needed. For example, the video device management unit 12 detects an object observed by the video device 10 from the position, orientation, zoom value, etc. of the video device 10, and the video device management unit 12 is the same for other video devices 10. To observe the object of
It is possible to perform control such that a control command is sent to the lens state control unit 6 and the platform control unit 4.

A data management unit 13 is composed of a storage device such as a hard disk. The original data of the data held by the data unit 8 in the video device 10 is stored, and the data held by the data unit 8 is the data unit 13
It is included in the data held by. As the data, like the data unit 8, three-dimensional CG data, depth data of the real world, three-dimensional model of the real world, text information, panoramic video and the like can be mentioned.

For example, when new data is added to the system, it is first added to the data management unit 13. The data management unit 13 sends the added data to each video device 10, and the video device 10 copies and holds the received data in the data unit 8. By doing so, it becomes possible to have common data in all video devices 10. Further, even if the video device 10 does not use the data unit 8 and always receives data from the data management unit 13, common data can be used, but in the present embodiment, the video device 10 does not change the data. Only when there is, the data of the data management unit 13 is referred to, and the data unit 8 is cached. Further, the data management unit 13 also holds the position data of the image pickup unit 1 in each video device 10, the posture data obtained from the posture control unit 5, and the lens state data obtained from the lens state control unit 6, and if necessary. And is used to send a control signal from the video device management unit 12 to each video device 10.

In the present embodiment, the data is added to the data management unit 13. However, a certain specific video device 10 is set and the data is added to the data unit 8, and the data management unit 13 performs the specific addition. It is also possible to refer to the additional data of the data section 8 of the video device 10 and copy it, and then reflect it on the data section 8 of another video device 10.

Reference numeral 14 denotes a server, which is a video device management unit 12
And the function of the data management unit 13. The server 14 is connected to the communication network 11 and can communicate with any video control device 10. Not only can the server 14 be used alone, but a plurality of servers 14 can be installed to distribute control targets and data. Further, the server 14 can also have its function built into a specific video device 10.

A control terminal 15 is connected to the communication network 11. It is possible to monitor the internal state of each video device 10 and the server 14 and to perform an operation. The control terminal 15 may be used not only by itself but also by installing a plurality thereof. Moreover, it is also possible to use it as a form built in a certain specific video device 10 or server 14.

Each image device 10 is arranged so that the same object such as a building, a mountain, or an exhibit in a museum can be viewed from different viewpoints. As long as the same object can be observed, its placement is not restricted. For example, a plurality of image devices 10 may be installed on the same floor in the building where the object can be observed, or one image device 10 may be installed in a different building. In the case of a large object such as a mountain, it may be possible to install it on an observatory or another mountain that is far away.

The control of this embodiment having the above configuration will be described below. FIG. 3 is a flowchart illustrating a procedure of processing in the video composition device.

In step S0, data preparation is performed. In step S0, data to be stored in the data management unit 13 or the data unit 8 is prepared. Examples of the data to be prepared include text information, panoramic images, three-dimensional CG data, depth data of the real world, and three-dimensional model of the real world. This data needs to be associated with the position in the real world in advance. For example, in the case of a three-dimensional CG, the position in the real world where it should be arranged must be specified. In the case of a panoramic image, it is necessary to specify which part of the panoramic image corresponds to which part of the real world.

When the panoramic video is stored in the data management unit 13 or the data unit 8, the panoramic video to be stored may be, for example, a panoramic video taken at different time zones or places. By storing panoramic images captured in different time zones in advance, for example, even when the observatory is cloudy, the view from the observatory during fine weather can be reproduced. In addition, by storing panoramic images captured at different locations, panoramic images captured at completely different locations can be reproduced as if they were there.

The panoramic video stored in the data management unit 13 or the data unit 8 is captured by the image capturing unit 1, for example. In this case, as shown in FIG. 18, the image captured by the image capturing unit 1 is captured by the captured image capturing unit 2, and the captured image is sent to the data unit 8. Although the panoramic video can be captured by the image capturing unit 1 and stored in the data unit 8 as described above, the present invention is not limited to this as long as the panoramic video can be stored in the data management unit 13 and the data unit 8. Any method is acceptable.

When the panoramic video is stored in the data management unit 13 or the data unit 8, the panoramic video may include a blank portion. As a method of expressing the blank in the panoramic video, for example, there is a method in which each pixel of the panoramic video is provided with an α value representing the transparency, and the α value of the blank pixel is set to zero. In the captured panoramic video, the α value of the part that does not require the panoramic video synthesis is set to 0.
If so, when the image synthesizing unit 3 synthesizes the captured image and the panoramic image, the captured image is synthesized in the part where the α value is 0, and the panoramic image is synthesized in the part where the α value is not 0. By doing so, it becomes possible to display the panoramic video and the captured video in one composite video. At this time, if all the α values of each pixel of the panoramic video are set to 1, a composite video including only the panoramic video without the captured video is composited at the time of video composition.

A method of giving an α value to each pixel of this panoramic image will be described with reference to FIG. 11 when an image synthesizing apparatus according to this embodiment is used in an observation room. In step S0, it is assumed that a panoramic image I is captured by photographing the surrounding scene from the observatory, and the panoramic image I includes an indoor image R and an outdoor image P. By setting the α value of the indoor image R to 0 and the α value of the outdoor image P to 1, when the image combining unit 3 performs image combination, the captured image is combined for the indoor part and the outdoor part for the outdoor part. It becomes possible to synthesize the outdoor image P. This makes it possible to see a real-world shot image when looking inside a room, and to see a panoramic image shot in advance when looking outside the room.

Here, the method of using the α value is described as a method of expressing the blank portion in the panoramic image, but this method may be any method as long as it can express the blank portion in the panoramic image. .

The panoramic video data stored here may be, for example, three-dimensional CG rendered in advance as a panoramic video. Generally, when the three-dimensional CG is generated in real time, the precision of the three-dimensional CG is limited due to the limitation of the drawing ability. It is possible to further improve the quality of the image of the three-dimensional CG. At this time, an α value representing transparency is given to each pixel of the rendered panoramic image, and C is displayed in the panoramic image.
If the α value of the portion where G is not drawn is set to 0, when the image synthesis unit 3 synthesizes the captured image and the computer image, the captured image is synthesized by synthesizing the captured image in the portion where the α value is 0. 3D C rendered in the video in advance
It becomes possible to synthesize G.

FIG. 4 is a flow chart for explaining the processing procedure for inputting the real world three-dimensional model, which is carried out in step S0 of preparing data in the processing procedure of FIG. Step S0 of FIG. 3 corresponds to steps S101 to S105 of FIG.
including.

In step S101, a plurality of video devices 1
Point 0 towards the object being modeled (eg, building, mountain, etc.). Then, the target object (object) is photographed from a plurality of viewpoints using the plurality of video devices 10, and a plurality of images of the real world including the target object are acquired. This work may be manual or automatic.

In step S102, the plurality of video devices 1
The position and orientation information is acquired from 0 and transmitted to the data management unit 13 in association with the captured image obtained in step S101. It is important to manage the captured image and the position / orientation information of the video device, which indicates the spatial position information of the captured image, in association with each other correctly.

Means for obtaining the position include, for example, GPS (Global Positioning S).
(system: global radio positioning system) or a method of estimating the position where the video device 10 is installed on the basis of a map or a drawing of a building.
As for the posture, the result measured by the posture measuring unit 5 is used. These pieces of position and orientation information are sent to the data management unit 13 in the server 14.

In step S103, the real world three-dimensional model of the target object or the depth data of the real world is input. The three-dimensional model and the depth data are necessary for synthesizing the real world image and the computer image in a state in which the mutual relationship between them is correctly reflected.

Here, a method for generating a three-dimensional model of the real world from map data will be described with reference to FIGS. 16 and 17.

Assuming that there is a map as shown in FIG. 16, buildings E, F, G, and H are drawn in the map, and the height information of the buildings E, F, G, and H is obtained by some method. It shall be obtained. Here, with the lower left corner of the map as the origin, 1 m
If the XYZ coordinate system as shown in FIG. 17 is defined with 1 as a unit, the building H can be expressed as a rectangular parallelepiped having a height of 10, a width of 15 and a depth of 15, and the other buildings E, F and G are also the same. Can be expressed in

As described above, a three-dimensional model of the real world can be generated from the map data. Figure 1 here
Although a coordinate system such as 7 is defined, it goes without saying that a real world three-dimensional model can be generated from map data regardless of the coordinate system. Further, the method of providing the three-dimensional shape model data in the real world is not limited to the above method, and any method may be used as long as the three-dimensional shape model of the real world can be constructed.

The depth information in the real world does not necessarily have to be a three-dimensional model, but may be a two-dimensional model as shown in FIG. Further, it is not always necessary to have a three-dimensional model of the real world that is unified in all video devices 10, and it is possible to have and use the depth data of the real world obtained by a range scanner or the like in each video device 10. Good. When one video device 10 has a plurality of image capturing units 1 and captured image capturing units 2, the images obtained from the captured image capturing units 2 are used in the real world by triangulation. Depth information may be obtained and used. The method of giving depth information to the real world is not limited to the above method, and for example, a method using a distance image may be used, and any method may be used as long as the depth to the real world is known.

In step S104, step S103
With respect to the three-dimensional model of the real world obtained by, the work of adjusting the size, the position to be arranged, the orientation, etc.

Specifically, with respect to the real-world image acquired by the captured image capturing section 2 of the image device 10, the depth data of the real world from the viewpoint of the certain image device 10 is a three-dimensional model of the real world. Convert to and synthesize the visualized image. Then, an image in which a three-dimensional model is combined with the combined captured image of the real world is displayed. Specifically, a three-dimensional model of the real world is generated as a CG from the viewpoint of the video device 10, and is combined with the captured image.

The three-dimensional model of the real world or the depth data of the real world created in step S103 does not always accurately represent the shape of the real object. An error may be included in the measurement of a real object, and when the object to be modeled is a building, it may be possible to approximate it with a simple solid such as a rectangular parallelepiped. In addition, even if the shape of a real object is accurately obtained, it is possible that the position and orientation information of the real world in which it is placed is not accurate. Due to these factors, the image obtained by converting the three-dimensional model of the real world into the depth data of the real world from the viewpoint of the image device 10 and the object in the image actually observed from the same viewpoint are not always strictly speaking. When they do not match and a three-dimensional CG of a virtual object is synthesized, a shift may occur between the real-world image and the computer image.

In step S104, the operator manually operates the size and position of the three-dimensional model of the real world using an adjustment screen described later so that the image of the real world and the CG of the three-dimensional model of the real world match. , Adjust parameters such as orientation.

The data management unit 13 manages the three-dimensional model of the real world using the same parameters in the entire system. Therefore, if this adjustment work is performed only from the viewpoint of the single video device 10, the deviation may be enlarged from other viewpoints. In particular, regarding the change in the position in the depth direction, it is difficult to grasp the amount thereof, and therefore the tendency becomes remarkable.

Therefore, in the present embodiment, the operator provides an adjustment screen where the operator can simultaneously check the composite video images from a plurality of viewpoints and adjust the three-dimensional model so as to reduce the deviation at all viewpoints.

When the three-dimensional model in the real world is not accurate, there may not be a parameter that allows accurate matching in all viewpoints. In such a case, however, there is an overall deviation depending on the purpose. The adjustment should be made so that the number becomes smaller, or the particular viewpoint should be emphasized.

A specific example of the adjusting method will be described below.

(Three-dimensional model adjusting method 1) The operator uses the GUI (Graphic) as shown in FIG.
The adjustment work is performed using an al User Interface (graphical user interface). In the GUI of FIG. 5, an image obtained by combining a real-world image 50 observed from a plurality of image devices 10 with a CG 51 of a three-dimensional model of the real world drawn from the viewpoint is displayed at the same time. Further, it has a parameter display unit 52 for displaying the parameters of the three-dimensional model in the real world in detail.

The operator moves the cursor 54 on an arbitrary composite image using an input device such as a mouse to adjust the size, position and orientation of the real world three-dimensional model.

The data of the three-dimensional model of the real world whose parameters have been adjusted is immediately sent from the data management unit 13 to each image device 10, and each image device 10 again uses the changed parameters to recreate the three-dimensional model of the real world. The model is converted to depth data in the real world, and an image combined with the real world image is generated. The plurality of composite videos generated by each video device 10 are displayed on the GUI. By using the GUI in this way, it is possible to perform the adjustment work of the three-dimensional model in the real world while immediately confirming the result of parameter change.

According to this adjusting method, the adjustment result of the three-dimensional model can be confirmed on the images from a plurality of viewpoints, and the appropriate three-dimensional model can be easily adjusted.

(Three-dimensional model adjusting method 2) In the adjusting method 2, the map data 53 is added to the adjusting screen (FIG. 5) used in the adjusting method 1.

Then, it has a function of creating a three-dimensional model of the real world from the map data. The operator is called a map 2
The two-dimensional information of the target object is specified in the dimension by operating the cursor 54.

A predetermined height information is added to the two-dimensional information of the target object designated on the map data 53 to create a three-dimensional model, which is displayed on the composite image. Then, the operator uses the cursor to adjust the parameters such as the height information on the composite video in the same manner as the adjusting method 1.

According to the adjusting method 2, the two-dimensional information of the target object can be easily created by using the map data.
Therefore, the position and direction parameters can be easily adjusted.

The two-dimensional information of the target object using the map data is created by analyzing the image in the designated area when the operator roughly indicates the area including the target object as shown in FIG. However, the contour of the target object may be extracted and the two-dimensional information may be automatically created. By using image analysis, two-dimensional information of a target object having a complicated shape can be easily created with high accuracy.

Further, as shown in FIG. 21, the viewpoint position (viewpoint and direction) of the composite image displayed on the adjustment screen.
May be displayed on the map data by using the arrow 55.

By displaying the viewpoint position on the map data with the arrow, the operator can visually confirm the viewpoint position of the composite image.

Further, when the photographed images from the video devices of which the number cannot be displayed on the adjustment screen are acquired in step S101, the position and orientation information acquired in step S102 is displayed on the map data, Then, a composite image from the viewpoint corresponding to the position and orientation information selected by the operator may be displayed on the adjustment screen.

By doing so, the operator can adjust the three-dimensional model using an arbitrary captured image from the captured image acquired in step S101. Further, since selectable viewpoints are displayed on the map data, it is possible to easily select a desired viewpoint (captured image).

(Three-dimensional adjustment method 3) Adjustment method 1 or 2
Adjustment method 3, which is a modified example of, will be described. In the adjusting method 3, as shown in FIG. 22, a plurality of shape models are prepared in advance on the pallet 55. Then, the shape model selected by the operator using the cursor 54 is arranged at an arbitrary position on the map. Then, the arrangement result is reflected in the composite video. The operator adjusts the placed shape model using the composite video and map data,
Adjust the parameters of the 3D model.

According to the adjusting method 3, it is possible to deal with a shape (cone, sphere, etc.) that cannot be created only by adding height information to the two-dimensional information created by the map data.

(3D Model Adjusting Method 4) The adjustment of the 3D model is not limited to manual adjustment by the operator. For example, the operator uses a GUI as shown in FIG. 19 to identify characteristic points such as vertices of a three-dimensional model in the real world,
The corresponding point at which the point is observed in the image of the real world viewed from the viewpoint of the image device 10 is input. A plurality of feature points are input for one video device 10, and this is performed for the plurality of video devices 10. The data of the input feature points and corresponding points is sent to the server 14, and the server 14 uses the real world 3
Automatically change the size, position, and orientation of the dimensional model.

The input of feature points and corresponding points is not limited to manual input by the operator. Corresponding points may be automatically extracted by applying image processing such as edge extraction to the acquired real-world image, or the feature may be automatically detected by using the geometric features of the three-dimensional model of the real world. The points may be associated with the corresponding points.

The size and position of the three-dimensional model in the real world,
As a method of automatically changing the orientation, for example, an evaluation using the distance between the coordinates of a feature point in a three-dimensional model of the real world projected on the image of the real world and the coordinates of the corresponding points as a scale. A method of defining a function and obtaining a parameter that minimizes this function can be considered. If this evaluation function takes the size, position, orientation, etc. of the three-dimensional model as parameters, in all viewpoints, the image of the three-dimensional model of the real world and the image of the real world that best match the image of the real world will be obtained. The size, position, orientation, etc. of the three-dimensional model can be obtained.

Here, an example of the evaluation function is shown. It is assumed that there are M video devices 10 and N characteristic points are set and input in a three-dimensional model of the real world. At this time, the two-dimensional coordinate of the pixel of the corresponding point on the image of the real world is P, and the three-dimensional coordinate of the three-dimensional model of the feature point of the real world is X. The position, orientation, and size of the three-dimensional model in the real world are L,
A and S are matrices representing rotation, translation, and scaling of the three-dimensional model, which are represented by functions of L, A, and S. Let this be R (L, A, S). Draw a three-dimensional model of the real world at the current position and orientation of a certain video device 10,
Let M be a matrix projected onto a two-dimensional image. At this time, the evaluation function F (L, A, S) is based on the Euclidean distance between the corresponding point and the point where the characteristic point of the three-dimensional model of the real world is projected on the image of the real world.

[Outer 1]

It can be expressed as

When determining the evaluation function, the weighting coefficient is used according to the viewpoint and role of the video device, and the three-dimensional model of the real world and the video of the real world are made to coincide with each other with respect to a certain viewpoint. You may do it. When there is a video device 10 for playing a game and another video device 10 for watching the game, for example, for the video device 10 for playing the game, a large weighting coefficient is set and a video device 10 for watching the game is taken. With respect to, if the weighting coefficient is set to be small, the three-dimensional model of the real world is more accurately matched to the real world image viewed from the video device 10 playing the game.

The operations from step S101 to step S104 are repeated for all target objects that model the real world. However, these operations are performed by the video device 10.
It is sufficient to read the saved result from the next time unless the position of the video device 10 and the modeled real world itself change by performing the operation once at the time of installation and saving the result.

After preparing the data, the system is started in step S1. In step S2, an image is acquired from the image capturing unit 1, and the acquired captured image is sent to the captured image capturing unit 2 as an NTSC image signal, for example. At this time, the captured image sent from the image capturing unit 1 to the captured image capturing unit 2 is not limited to the NTSC image signal, but may be anything that can express the image.

The picked-up image picked up by the image pickup unit 1 is converted into an appropriate format by the picked-up image taking unit 2 and sent to the image combining unit 3. In the captured image capturing unit 2, for example, the image capturing unit 1
The NTSC video signal sent from the device is converted into digital data and sent to the video synthesizing unit 3, but the video sent by the image pickup unit 1 is not limited to the NTSC video signal, and any image can be displayed. The captured image sent out by the image capturing unit 2 to the image synthesizing unit 3 is not limited to digital data and may be any image as long as the image can be expressed and processed by the image synthesizing unit 3.

At step S3, the posture measuring unit 5 detects the posture of the image pickup unit 1, and the detected posture information is sent to the computer image generating unit 7. In step S4, the lens state control unit 6 detects lens state information such as a zoom value and a focus value of the lens used in the image pickup unit 1, and the detected lens state information is sent to the computer image generation unit 7.

In step S5, the computer image generation unit 7 uses the posture information sent from the posture measurement unit 5 and the lens state information sent from the lens state control unit 6 to take an image from the image pickup unit 1.
The field of view is estimated, and the data in the range included in the field of view of the imaging unit 1 is acquired from the data unit 8.

There are various methods for obtaining the visual field of the image pickup unit 1 from the posture information sent from the posture measuring unit 5 and the lens state information sent from the lens state control unit 6, but here, there are various methods. One of them is shown. In general, the field of view can be obtained by obtaining the position of the lens center of the lens used in the image pickup unit 1 and the view angle of the image pickup unit 1. Therefore, in the following, first, from the posture information sent from the posture measurement unit 5, A method for obtaining the position of the lens center of the lens used in the image pickup unit 1 will be described, and then a method for obtaining the viewing angle of the image pickup unit 1 from the lens state information sent from the lens state control unit 6 will be described.

A method of obtaining the position of the lens center of the lens used in the image pickup unit 1 from the posture information sent from the posture measuring unit 5 will be described with reference to FIGS. 12 and 13. FIG. 12 is an external view of the video compositing apparatus of this embodiment. The image pickup unit 1 and the display unit 9 are integrally supported by the camera platform control unit 4 and can rotate about the rotation center 0 in the vertical direction and the horizontal direction. Since the platform controller 4 is fixed to the floor,
The positional relationship between the floor and the center of rotation 0 is unchanged. Therefore, if the relative positional relationship between the rotation center 0 and the lens center L0 is known, the position of the lens center L0 with respect to the real world can be uniquely determined.

Here, an XYZ coordinate system with the center of rotation 0 as the origin is defined as shown in the figure, and the left and right rotation angles φ and the upper and lower rotation angles θ are defined as shown in FIG. When the distance from the lens center L0 of the lens used in the imaging unit 1 to the rotation center 0 is represented as d, the coordinates of the lens center L0 are represented as x = −dsin θ sin φ y = d sin θ cos φ z = d cos θ. Here, the XYZ coordinate system is defined as shown in FIG. 13. However, no matter how the coordinate system is defined or the vertical and horizontal rotation angles are defined, the horizontal rotation angle and the vertical rotation angle are defined. It goes without saying that the position of the lens center of the lens used in the imaging unit 1 can be obtained from the rotation angle of.

A method of obtaining the viewing angle of the image pickup section 1 from the lens state information sent from the lens state control section 6 will be described with reference to FIGS. 14 and 15. First, the focal length of the lens used in the image pickup unit 1 is calculated from the zoom value sent from the lens state control unit 6. Here, it is assumed that the zoom value sent from the lens state control unit 6 is a value output from the 8-bit encoder provided in the image pickup unit 1, and the value and the focal length of the lens are as shown in the table of FIG. It is assumed that Then, by using the table in FIG. 14 and interpolating the values not in the table,
The focal length of the lens used in the image pickup unit 1 can be obtained using the zoom value sent from the lens state control unit 6. However, when the lens state control unit 6 can directly output the focal length, it is not necessary to calculate the focal length using the table shown in FIG.

When the focal length of the lens used in the image pickup unit 1 obtained as described above is f and the lateral width of the image pickup surface provided in the image pickup unit 1 is x, the horizontal viewing angle of the image pickup unit 1 from FIG. 15 is obtained. ψ can be calculated by the following formula.

Ψ = 2 arctan (x / 2f) Similarly, if the height of the image pickup surface provided in the image pickup unit 1 is y,
From FIG. 15, the vertical viewing angle μ of the imaging unit 1 can be obtained by the following equation.

Μ = 2 arctan (y / 2f) As described above, the lens center of the lens used in the image pickup unit 1 is calculated from the posture information sent from the posture measurement unit 5 and the lens state information sent from the lens state control unit 6. And the viewing angle of the image pickup unit 1 are obtained, and thus the view of the image pickup unit 1 can be obtained. At this time, the method as described above is used as a method for obtaining the visual field of the image pickup unit 1, but any method may be used as long as the visual field of the image pickup unit 1 can be obtained without being limited to the above method.

In step S5, the posture data obtained by the posture estimation unit 5 and the lens state data obtained by the lens state control unit 6 are sent to the server 14 through the communication network 11. At the same time, the server 14 receives the state of the entire system, the control signal sent to the camera platform control unit 4, the lens control signal sent to the lens state control unit 6, and the like.
The platform control unit 4 and the lens state control unit 6 control the equipment in charge according to the received signal.

Further, in step S5, the computer image generation unit 7 estimates the visual field of the image pickup unit 1 from the posture information sent from the posture measurement unit 5 and the lens state information sent from the lens state control unit 6, and the data unit 8 The data of the range included in the visual field of the imaging unit 1 is acquired from. At the same time, this data is also sent to the server 14 through the communication network.

In step S6, the computer image generation unit 7 uses the data acquired from the data unit 8 to generate a computer image. The generated image is sent to the image synthesizing unit 3.

At step S6, if panoramic images are used in the data section 8 in multiple stages with different resolutions, the panoramic image resolution can be determined by using the zoom information of the lens state information of the lens used in the image pickup section 1. It is also possible to switch and display. That is, when a zoom-out is performed, a coarse panorama image is displayed, and as the zoom-in is performed, a finer resolution image is displayed, so that a clear panoramic image can always be presented without stress.

In step S7, the captured image transmitted from the captured image capturing section 2 and the computer image transmitted from the computer image generation section 7 are synthesized in the image synthesis section 3. The combined composite image is sent to the display unit 9.
In step S8, the display unit 9 displays the video information sent from the video composition unit 3. After that, in step S9, it is checked whether or not to terminate the system. If the system is to be terminated, the system is terminated in step S10,
When not ending, it returns to step S2 and repeats the above-mentioned processing.

As described above, according to the first embodiment,
For example, it is possible to reproduce the view from a observatory on a clear day even when it is cloudy. A panoramic image taken in fine weather at the observatory is stored in advance in the data management unit 13 or the data unit 8, the field of view of the imaging unit 1 is determined using the posture information and the lens state information, and the field of view is cut out from the panoramic image. By displaying on the display unit 9, the above items are achieved.

As described above, according to the image synthesizing apparatus of this embodiment, a virtual character can be seen, for example, by superimposing it on a building of an actual landscape. The above items are achieved by storing in advance the three-dimensional model of the real world, the depth data of the real world, and the three-dimensional CG data of the virtual character in the data management unit 13 or the data unit 8. That is, the position of a virtual three-dimensional CG character is set in advance between the buildings in the real world, and the virtual character is set at an appropriate position using the position / orientation information and the lens state information, that is, the portion between the buildings. Image may be generated and combined with the image acquired by the captured image capturing unit 2.

Further, for example, a plurality of video devices 10 are provided.
It is possible to see virtual characters from different viewpoints by overlaying them on the same building in the actual landscape.
Even if the 3D model of the real world input to the device is not accurate, by adjusting the parameters such as position, size, and orientation, the front-back relation between the real building and the virtual character can be represented more accurately. It is possible. This adjustment is performed by simultaneously observing from a plurality of viewpoints an image obtained by visualizing and synthesizing the depth data of the real world viewed from the viewpoint with the image captured from the captured image capturing unit 2 of the image device 10. The adjustment can be performed accurately and efficiently. It is also possible to automatically perform these adjustment operations by inputting the coordinates of the three-dimensional model and the corresponding points in the image.

Further, according to the present embodiment, for example, a person manually operates one specific video device 10, and the other video devices 10 are observed by the manually operated video device 10. By tracking the same object as the object being taught, it can be useful for teaching and the like. In addition, by transmitting an image synthesized on another image device 10 to another image device 10, an object or a part thereof (for example, the back side of a building) that is a blind spot from a certain viewpoint and cannot be seen is separated. It is also possible to observe from the angle.

The image synthesizing device of this embodiment has another image pickup unit in addition to the image pickup unit 1, and each image pickup unit has a photographed image capturing unit 2, a computer image generating unit 7, an image synthesizing unit 3, By having the display unit 9, it becomes possible to use the image synthesizing apparatus of this embodiment as binoculars.
Needless to say, the processing procedure of the video synthesizing device at this time is no different from that in the case where the image pickup unit 1 has a single configuration. At this time, when operating the state of the lens by an operation unit (not shown) or a control signal, it is necessary to interlock the state of the lens used in each imaging unit.

<Second Embodiment> In the second embodiment,
A composite video storage unit 20 (not shown) and a storage video output unit 21 (not shown) are provided to store a composite video image observed by the user of the present embodiment and output it to a medium other than the display unit 9.

The composite video storage unit 20 is composed of a hard disk, for example, and receives the composite video signal from the video composition unit 3 and records it. It is conceivable to use, for example, a video recorder or a hard disk as the composite video storage unit 20, and any medium can be used as long as it can store video. Information other than the composite video is also stored in the composite video storage unit 20, for example, text information and image information such as the date and place of the composite are recorded.

The process of storing information such as a composite video in the composite video storage unit 20 is included in step S8. The video synthesizing unit 3 sends the video information to the display unit 9 and also sends the video information to the synthetic video storage unit 20. Text information, image information, etc. are sent from the data management unit 13 or the data unit 8 to the composite video storage unit 20.

Further, the composite video storage unit 20 is the video device 10
It is possible to provide not only for the server 14 but also for the server 14. The composite video output from the video composition unit 3 is the communication network 1
1 to the composite video storage unit 20. In this case, since the composite video from all the video devices 10 is recorded in the composite video storage unit 20, it is possible to simultaneously refer to the composite video from the plurality of video devices 10.

The storage image output unit 21 is composed of, for example, a printer, and outputs the information held in the composite image storage unit 20 to a medium other than the display unit 9. This process is included in step S10. When the user of this video synthesizer finishes using it in step S9, the result output from the stored video output unit 21 in step S8 is passed to the user. If a user brings back a still image of a composite image when he / she uses the image composition apparatus of this embodiment, or if a video recorder is used for the stored image output unit 21, the user himself or another user It is also possible to bring back a moving image of a composite image when using the composition device.

As described above, according to the second embodiment, it becomes possible for the user to bring back a printout of an image that he / she has experienced and observed, or a moving image of a composite image.

As described above, according to each of the above embodiments,
At the observatory, etc., the user can change the viewpoint freely through the image synthesizing device according to the present embodiment, and the user overlays the C on the real world.
You can see G or see the panoramic video stored in advance instead of the real world. Also,
The user can freely move the CG character appearing in the real world, and can bring back a still image or a moving image of the composite image as a souvenir. Furthermore, it is possible to efficiently perform the adjustment work for expressing the context between the real world and the virtual world.

The video synthesizing apparatus of this embodiment is naturally not limited to use in an observatory or a museum, but is used in all situations where the video synthesizing apparatus of this embodiment can be effectively used. It is possible.

The object of the present embodiment is to supply a storage medium (or recording medium) recording a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, and to supply a computer (or computer) of the system or apparatus. It is also achieved by the CPU or MPU) reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the storage medium storing the program code constitutes the present embodiment. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the operating system (0S) or the like operating on the computer is executed based on the instruction of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, based on the instruction of the program code, It goes without saying that a case where the CPU included in the function expansion card or the function expansion unit performs a part or all of the actual processing and the processing realizes the functions of the above-described embodiments is also included.

[0109]

According to the present invention, it is possible to easily create a three-dimensional model of a physical object.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of a video composition device.

FIG. 2 is a block diagram showing a configuration in the case where video synthesizing apparatuses are mutually connected via a communication network.

FIG. 3 is a flowchart illustrating a processing procedure of a video composition device.

FIG. 4 is a flowchart illustrating details of data preparation in the processing procedure of the video composition device.

[Fig. 5] GU performing adjustment work for a three-dimensional model in the real world
It is a figure which shows the example of I.

FIG. 6 is a diagram showing an example of a GUI for creating a three-dimensional model of the real world from map data.

FIG. 7 is a diagram showing a case where the front-rear relationship between a real world building and a virtual character is not expressed.

FIG. 8 is a diagram showing a case where the front-rear relationship between a real world building and a virtual character is expressed.

FIG. 9 is a diagram when a contact relationship between a real world building and a virtual character is expressed.

FIG. 10 is a diagram when a real world model is expressed as a two-dimensional model.

FIG. 11 is a diagram illustrating a case where an α value is given to a part of a panoramic video.

FIG. 12 is a diagram illustrating a method of obtaining the position of the lens center of the lens used in the imaging unit from the posture information sent from the posture control unit.

FIG. 13 is a diagram illustrating a method of obtaining the position of the lens center of the lens used in the imaging unit from the posture information sent from the posture control unit.

FIG. 14 is a diagram illustrating a method of obtaining a focal length of a lens used in an imaging unit from lens state information sent from a lens state control unit.

FIG. 15 is a diagram illustrating a method of obtaining a viewing angle of a lens from a focal length of the lens used in the imaging unit.

FIG. 16 is a diagram illustrating a method of generating a three-dimensional model of the real world from map data.

FIG. 17 is a diagram illustrating a method of generating a three-dimensional model of the real world from map data.

FIG. 18 is a diagram illustrating a case where a panoramic video is captured by the image capturing unit and stored in the data unit.

FIG. 19 is a diagram illustrating a work of associating points of a three-dimensional model in the real world with points in an image of the real world.

FIG. 20: G performing adjustment work of a three-dimensional model in the real world
It is a figure which shows the example of UI.

FIG. 21: G performing adjustment work of a three-dimensional model in the real world
It is a figure which shows the example of UI.

FIG. 22: G performing adjustment work of a three-dimensional model in the real world
It is a figure which shows the example of UI.

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 5/272 H04N 5/272 (72) Inventor Mahoro Anabuki 3-30-2 Shimomaruko, Ota-ku, Tokyo Kyano (72) Inventor Eita Ono 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Co., Ltd. (72) Inventor Masahiro Suzuki 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Co., Ltd. (72) Inventor Akihiro Katayama 3-30-2 Shimomaruko, Ota-ku, Tokyo F-Term (in reference) 2C001 BA03 BC00 BC07 BC08 DA06 5B050 AA09 BA06 BA08 BA09 BA11 CA07 CA08 DA07 EA07 EA19 EA24 EA27 FA02 FA09 5B057 AA20 BA02 CA01 CA08 CA12 CA13 CA16 CA17 CB01 CB08 CB13 CB16 CE08 DA07 DA16 DB03 DB09 DC02 DC36 5C023 AA16 AA38 BA02 BA11 CA03 DA08 EA03

Claims (16)

[Claims] 1. An information processing method for setting, using an adjustment screen, a three-dimensional model of an object included in an image showing the real world, which is used for synthesizing a computer image showing the virtual world with an image showing the real world. That is, an image showing the real world including the object obtained from a plurality of viewpoints is acquired, and the spatial position information of the image showing the plurality of real world and the position information of the three-dimensional model corresponding to the object are acquired. On the basis of the plurality of images showing the real world, the three-dimensional model corresponding to the object is synthesized and displayed on the adjustment screen. An information processing method characterized by adjusting the position and shape of a three-dimensional model corresponding to an object. 2. The map data corresponding to the image showing the real world is displayed on the adjustment screen, and the three-dimensional model corresponding to the object is synthesized and displayed on the map data. The information processing method according to claim 1. 3. The information processing method according to claim 2, wherein the position and shape of the three-dimensional model can be adjusted on the map data. 4. The information indicating the plurality of viewpoints is combined on the map data.
Information processing method described. 5. The information processing method according to claim 1, further comprising synthesizing information indicating the plurality of viewpoints and directions of the viewpoints on the map data. 6. The information processing method according to claim 1, further comprising creating a three-dimensional model of the object based on the map data. 7. A program for realizing the information processing method according to claim 1. 8. An information processing method for setting, using an adjustment screen, a three-dimensional model of an object included in an image showing the real world, which is used for synthesizing a computer image showing the virtual world with an image showing the real world. And acquiring an image showing the real world including the object,
Displayed on the adjustment screen, from the plurality of three-dimensional models displayed on the adjustment screen,
An arbitrary three-dimensional model is selected based on a user instruction, the selected three-dimensional model is combined with an image showing the real world, displayed on the adjustment screen, and an image showing the real world is displayed according to the user instruction. An information processing method characterized by adjusting the position and shape of a synthesized stereo model. 9. A program for realizing the information processing method according to claim 8. 10. An image pickup section for photographing the real world, an instruction section for supporting the image pickup section, a posture measuring section for measuring the posture of the image pickup section, data for generating a computer image, and three of the real world. A data unit that holds dimensional model data; a computer image generation unit that generates a computer image using the data held in the data unit based on the posture information measured by the posture measurement unit; An image synthesizing unit for synthesizing a photographed image photographed by a computer unit and a computer image generated by the computer image generating unit based on the three-dimensional model data of the real world; and a three-dimensional model of the real world by the photographing unit. An image synthesizing apparatus having a three-dimensional model creating unit that creates and adjusts using a captured image that has been captured. 11. The image composition according to claim 10, wherein the three-dimensional model of the real world includes depth data obtained by three-dimensional survey using photographed images photographed by a plurality of photographing units. apparatus. 12. The adjustment of the three-dimensional model of the real world is automatically performed by inputting feature points of the three-dimensional model of the real world and corresponding points on an image of the real world. Item 10. The video synthesizing apparatus according to Item 10. 13. The image synthesizing apparatus according to claim 12, wherein the input of the corresponding points is automatically performed by performing image processing on a real world image. 14. The video according to claim 12, wherein the input feature points and the corresponding points are automatically associated with each other based on the geometric shape of the three-dimensional model of the real world. Synthesizer. 15. The adjustment of the three-dimensional model of the real world uses the attributes such as size, position and orientation of the three-dimensional model of the real world as parameters and projects the corresponding points and the feature points on an image. 13. The image synthesizing apparatus according to claim 12, which is realized by obtaining a parameter that minimizes an evaluation function indicating a distance to a point. 16. The image synthesizing apparatus according to claim 15, wherein the evaluation function is capable of changing a weighting coefficient depending on a role of each viewpoint.