JP2003141574A - Image display method and system - Google Patents

Abstract

(57) [Summary] [Problem] To display a three-dimensional stereoscopic image independent of a viewpoint. An object of the present invention is to observe a reference object that provides a coordinate system serving as a reference when displaying a three-dimensional stereoscopic image to be displayed and a reference object placed in a field of view, and determine the position and orientation of the object in real time. And a three-dimensional information input device for measuring by means of. From the measured position and orientation of the reference object, a conversion matrix between the coordinates fixed to the reference object and the coordinates fixed to the three-dimensional information input device is calculated in real time, and based on that, the coordinate conversion of the three-dimensional stereoscopic image to be displayed is performed. I do. Calculate a two-dimensional image that can be seen from the position of the left and right eyes of the coordinate-transformed three-dimensional image, and display a slightly different composite image for both eyes of the user on a display device having a see-through function,
Show the calculated 3D image to the user. The image which is actually seen by the user is displayed as it is in the portion where the composite image is not displayed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention measures the relative position and orientation of a reference object with respect to a display device using a three-dimensional information input device, and a three-dimensional stereoscopic image based on a coordinate system fixed with respect to the reference object. The present invention relates to an image display method and system for calculating and displaying on a display device.

[0002]

2. Description of the Related Art Conventionally, a commonly used device for displaying a three-dimensional image displays an image for left and right eyes on a fixed screen of a theater and sees it with glasses which is decomposed by polarization or color. Method was the main. This method requires a large screen to actually display the screen, and is not suitable for carrying. Also, the audience cannot see the object from any direction.

Some computer displays have realized the same function, but when changing the direction of an object, it is necessary to use a pad or keyboard with a dial, or a device such as a data glove for measuring the movement of a hand. there were. Pads and keyboards require special manipulations and cannot be manipulated like real objects. The data glove is an input device that is often used in virtual reality, and it is in the shape of a glove and measures the position and posture of the hand and the bending angle of the finger in real time. Although the data glove enables a fairly realistic operation, it is necessary to be extraordinary because it cannot measure the physical force only by measuring the position. In addition, as a matter of course, the target object is limited to a small object on the computer display, and it is not possible to look around a large three-dimensional image.

In virtual reality, a position measuring device using a magnetic field or the like is used to measure the position of a display device worn on the user's head or a data glove for operation, and the position is measured in a virtual three-dimensional manner. The main method was to project in space (Fig. 11). In this figure, the user has a display (HMD
= Head Mount Display) is attached and the data glove is put on the hand. The positions and orientations of the HMD and the data glove based on the room coordinates are measured and used to display the composite image. This also requires measurement of the position of the face and hands in the entire operation space, and therefore requires a large-scale device for the entire room, and there is no small device. Also, in this method, since only the position of the hand is measured during handling, it is necessary for the user to be different from the actual operation feeling.

[0005]

SUMMARY OF THE INVENTION The present invention solves the above problems and eliminates the need for a huge three-dimensional measuring device that has been required in the past, and also eliminates the need for measuring the hand itself. An object of the present invention is to provide an image display method and system for displaying a three-dimensional stereoscopic image that does not depend on the display on a display device. It is another object of the present invention to provide an image display method and system that is realistic in operation and does not require user's familiarity. Another object of the present invention is to enable walking in a three-dimensional image of the size of a room or a building.

[0006]

FIG. 3 is a diagram illustrating the basic configuration of the system of the present invention. The system of the present invention includes a reference object, a three-dimensional information input device, a coordinate conversion device, a display image synthesizing device, and a display device. Further, a display data recording device and a communication device can be included if necessary. The reference object is an object that provides a coordinate system that serves as a reference when displaying a stereoscopic image to be displayed. Anything can be used as long as it can uniquely calculate the coordinates when it is measured by the three-dimensional information input device. Specifically, there may be a rectangular frame that can be held by hand. When the displayed three-dimensional image is large, the reference object may be a plurality of patterns that can be distinguished from each other.

The display device is a device for showing a combined stereoscopic image to both eyes of the user, and is a device in which two small displays are connected. This display presents a slightly different composite image to both eyes, showing the user a calculated 3D image. At the same time, a see-through function is required to display the image actually seen by the user as it is in the part where the composite image is not displayed. In this actual portion, the display element itself may be transparent and transparent, or a front image taken by a camera may be displayed.

The three-dimensional information input device is a device for observing a reference object placed in the field of view and measuring its position and orientation in real time. At present, there is a stereo vision system using two cameras that can be used for such a purpose. This three-dimensional information input device needs to be fixed to the display device (stereovision is a method of calculating the three-dimensional coordinates of the contour of a symmetrical object from the shift of the images of two or more TV cameras). .

The coordinate conversion device is fixed to the coordinate fixed to the reference object and the coordinate fixed to the three-dimensional information input device based on the measured position and orientation of the reference object (this is the coordinate relative to the three-dimensional information input device and the display device). The transformation matrix with the coordinates is calculated in real time, and the coordinate transformation of the three-dimensional image to be displayed is performed based on it. The display image synthesis device is a calculation device that calculates a two-dimensional image that can be seen from the left and right eye positions of the coordinate-converted three-dimensional image.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, a method and system for performing three-dimensional display for realizing the mixed reality of the present invention will be described based on examples. FIG. 1 shows an example of embodying the system of the present invention illustrated in FIG. In this example, the device is composed of a stereoscopic image display device 1 in the form of glasses, two cameras 2, and a computing device 3. Further, the object 4 which provides the reference coordinates is required as a necessary accessory. In this figure, a rectangular plate with a size like a PDA is used.

The display device 1 can superimpose and display an actual landscape actually seen by the wearer. It is possible to display a stereoscopic composite image by superimposing different images with both eyes. Stereoscopic vision is stereoscopic vision that humans routinely perform with both eyes. A common way to display stereoscopic images with glasses is to superimpose the images of both eyes on a large screen or display them with a temporal shift, and then separate them with a color filter of the glasses, a liquid crystal shutter, a polarizing filter, etc. Is the way to do it. The device used here does not use such a huge device but directly synthesizes an image on the surface of the spectacles.

The camera 2 is a small video camera. These can input moving images electronically, and most of them are currently using CCD image elements. The photographing interval is usually 1/30 second / frame in the case of NTSC. However, for the purpose of tracking an object, it is desirable that the movement amount for each frame is as small as possible because recognition errors are reduced. Recently, commercial cameras capable of high-speed shooting have begun to appear. "Tracking an object" means inputting the position and orientation of an object moving in a three-dimensional object space in real time, and stereo image processing can be performed every time.
Normally, the amount of movement for each screen is small, so rather than searching for the object (reference object in this case) from the entire image, searching near the previous position on the screen reduces the amount of calculation and speeds up the calculation. Will be possible.

The present invention performs stereo image processing to reconstruct the shape of a three-dimensional object from images taken by two or more cameras. Basically, the principle of triangulation is used to calculate the three-dimensional coordinates of the same feature point captured on the left and right cameras from the arrangement of the cameras. However, in reality, (1) when a general object is targeted, the feature points do not appear clearly on the image. (2) If there is a figure with a shape similar to the feature point, it will correspond incorrectly, and the three-dimensional shape will be calculated accurately. There are difficult implementation points such as not being possible.
When the target object has a known shape, not only point correspondence but also global information such as boundary lines and their combinations can be used, so that corresponding point search can be performed more smoothly and accurately. become. In the case of the present invention, it is possible to further reduce the probability of mishandling by giving the reference object a shape suitable for stereo image processing, or by adding a mechanical device (such as attaching an LED) that emphasizes the characteristics. is there.

The arithmetic unit 3 is a computer that performs image processing, coordinate conversion, display and the like. Specifically, it is a computer. If specialized in coordinate conversion, it can be downsized and speeded up by using dedicated hardware. The reference object 4 is an object that serves as an index that gives reference coordinates. The form may be any object as long as the position and orientation can be accurately calculated in three-dimensional measurement. The size may be small, but if the size is too small, the accuracy of the posture may be deteriorated due to the error during measurement, and the reference coordinates may become unstable. Also,
If it is too large, it will not fit in the field of view of the camera for three-dimensional measurement, and there is also a problem in this case. Moreover, since three-dimensional measurement is performed by image processing, it is desirable to use a color or shape that is easily distinguishable from the surroundings.

Next, the operation of the example system illustrated in FIG. 1 will be described. (1) As shown in FIG. 12, an image of the reference object 4 is input using two cameras 2, three-dimensionally measured by the arithmetic unit 3, and the relative reference object 4 viewed from the coordinate system C2 of the cameras.
Measure the position and posture of. The shape, color, etc. of the object 4 are stored in advance as data so that no mistake is made even if there is another object in the visual field.

(2) As shown in FIG. 13, a transformation matrix R42 from the coordinate system C4 fixed on the object 4 to the camera coordinate system C2 is calculated from the information on the position and orientation of the object 4. (3) Further, as shown in FIG. 13, a transformation matrix R21 from the camera coordinate system C2 to the coordinate system C1 of the display device is also calculated in advance. Since the camera and display are fixed, this will not change once calibration is done. (4) The data of the stereoscopic image to be combined is defined and prepared on the coordinate system C4. The conversion matrix R42 calculated from this in (2)
To the coordinate system C2, and further to the coordinate system C1 by the conversion matrix R21 in (3).

(5) An image viewed from both eyes of the user is created from the thus converted three-dimensional data, and the image is displayed on each display device of the display device 1. Through the above processing, the user can see a three-dimensional stereoscopic image that follows the movement and rotation of the object 4. The three-dimensional image does not have to be a still image, and may be a moving image if the capability of the arithmetic device allows.

FIG. 4 is a diagram illustrating a second system of the present invention different from that of FIG. In FIG. 3, the three-dimensional data used to display the image is recorded inside the arithmetic device, but in FIG. 4, it is stored outside and acquired by communication. FIG. 2 shows an example of embodying the system illustrated in FIG. As shown in FIG. 2, buttons can be added to the reference object and the display object can be operated by communicating between the reference object and the system. The operation that can be performed here is, for example, that when a three-dimensional image of a moving image is displayed, the image reproduction speed such as reproduction, stop, fast forward, and rewind is performed like a video. Further, operations such as enlarging, reducing, and changing colors can be considered even for a static three-dimensional image. As described above, by using the button, the operation other than the change of the position and orientation of the object, which is possible by the reference object, can be performed on the three-dimensional image. As a result, it is possible to use it as an operation for playing or stopping a moving image or as a game interface.

FIG. 5 is a diagram showing a first application example in which the present invention is applied to a display device for displaying a three-dimensional image, and freely moving the three-dimensional image for viewing. The reference object is a frame of a size that can be held by a hand, and even if there are a plurality of users, the same reference coordinates are defined for the same reference object, so that discussion with a three-dimensional image is possible.

FIG. 6 is a diagram showing a second application example in which the present invention is applied to a virtual large screen display that moves in association with a mobile phone. The reference object is a mobile phone or an antenna protruding from the mobile phone.

FIG. 7 is a diagram showing a third application example in which the present invention is applied to a virtual large screen display on a desk. The reference object is the frame attached to the keyboard.

FIG. 8 is a diagram showing a fourth application example to which the present invention is applied. In this example, the entire desktop becomes the computer's virtual display. The reference object is a pattern placed on a desk.

FIG. 9 is a diagram showing a fifth application example to which the present invention is applied. It is possible to display moving image advertisements on clothes and automobiles. The reference object is a fixed pattern drawn on clothes or a car. In this case, the data of the three-dimensional image to be displayed is on the person wearing the clothes or the automobile side on which the reference object is drawn, and it is sent to the nearby wearer by communication.

FIG. 10 is a diagram showing a sixth application example to which the present invention is applied. Enables three-dimensional decorations such as clothes and costumes on the stage. The reference object is a pattern fixed to the body. In this figure, the actor has a large number of reference objects attached to his body, which can be distinguished from each other.By superimposing the composite image displayed for each coordinate of the four limbs, the panda can walk to the observer. It looks like you are.

[0025]

According to the present invention, the position and orientation of the reference object are measured by the visual three-dimensional measuring device attached to the face, and the relative positional relationship between the display system and the object system is based on the coordinates of the face. By performing the real-time measurement by using the above-mentioned technique, the huge three-dimensional measuring device required in the above-mentioned conventional technique can be eliminated. Further, in the operation of holding the displayed three-dimensional object by hand and looking at the back side, the reference object actually moved by hand is measured in the present invention, so that the measurement of the hand itself is unnecessary, and the operation is also unnecessary. There is no need for the user to get used to it because of the reality. Also, as long as one of the reference objects is in the field of view of the three-dimensional measuring device, coordinate conversion is possible, so
It is also possible to walk in a three-dimensional image of the size of a room or building.

[Brief description of drawings]

FIG. 1 shows an example of embodying the system of the present invention illustrated in FIG.

FIG. 2 shows an example of embodying the system illustrated in FIG.

FIG. 3 is a diagram illustrating a basic configuration of the system of the present invention.

FIG. 4 is a diagram illustrating a second system of the present invention different from that of FIG. 3;

FIG. 5 is a diagram showing a first application example to which the present invention is applied.

FIG. 6 is a diagram showing a second application example to which the present invention is applied.

FIG. 7 is a diagram showing a third application example to which the present invention is applied.

FIG. 8 is a diagram showing a fourth application example to which the present invention is applied.

FIG. 9 is a diagram showing a fifth application example to which the present invention is applied.

FIG. 10 is a diagram showing a sixth application example to which the present invention is applied.

FIG. 11 is a diagram for explaining a method of projecting in a virtual three-dimensional space based on the conventional technique.

FIG. 12 is a diagram for explaining a method of inputting an image of a reference object using two cameras and measuring the position and orientation thereof.

FIG. 13 is a diagram for explaining calculation of a transformation matrix R42 from the coordinate system C4 fixed on the object to the camera coordinate system C2, and a transformation matrix R21 from the camera coordinate system C2 to the coordinate system C1 of the display device. is there.

[Explanation of symbols]

1 Eyeglass-shaped stereo image display device 2 camera 3 arithmetic unit 4 Target

Continued front page    F-term (reference) 5B050 AA09 BA09 BA13 EA07 EA19                       EA27 FA02 FA06 FA09                 5B057 AA20 CA13 CB12 CD14 CE08                       CH01 DA16                 5C061 AA01 AA20 AB04 AB08 AB18                       AB20 AB24                 5C082 AA27 BA12 BA46 BB25 BB26                       CA55 CB03 DA22 DA42 DA73                       DA86 MM09 MM10

Claims (2)

[Claims] 1. In an image display method for displaying a three-dimensional stereoscopic image on a display device, a reference object that provides a coordinate system that serves as a reference when displaying a three-dimensional stereoscopic image to be displayed, and a reference object placed in the visual field. It is equipped with a 3D information input device that observes the reference object and measures its position and orientation in real time.From the measured position and orientation of the reference object, the coordinates fixed to the reference object and the 3D information input device are fixed. Calculate the transformation matrix with the coordinates in real time, and based on that, perform the coordinate transformation of the 3D stereoscopic image to be displayed, and calculate the 2D image that can be seen from the left and right eye positions of the coordinate transformed 3D image. , A part that displays a slightly different composite image for both eyes of the user on a display device having a see-through function and shows the user a calculated three-dimensional image while not displaying the composite image Image display method comprising the image that appears in reality to the user is displayed as it is, since it is. 2. In an image display system for displaying a three-dimensional stereoscopic image on a display device, a reference object that provides a coordinate system that serves as a reference when displaying a three-dimensional stereoscopic image to be displayed, and a reference object placed in the visual field. A three-dimensional information input device that observes a reference object and measures its position and orientation in real time, and the coordinates fixed to the reference object and the coordinates fixed to the three-dimensional information input device based on the measured position and orientation of the reference object. Calculates the transformation matrix of in real time, and based on it, calculates the coordinate transformation device that transforms the coordinates of the three-dimensional stereoscopic image to be displayed, and the two-dimensional image that can be seen from the left and right eye positions of the coordinate-transformed three-dimensional image. Display image synthesizer and display a slightly different composite image for both eyes of the user,
An image display system comprising a display device having a see-through function for displaying the calculated three-dimensional image to the user and at the same time displaying the image actually viewed by the user in a portion where the composite image is not displayed.