WO2006121365A1 - Formation three dimensional images for a video conference - Google Patents

Abstract

The inventive method for forming a three dimensional image for a video conference consists in recording the images of an object (1) by using at least two WEB-cameras (2, 3) and in subsequently reconstructing a three-dimensional image (3D) displayed on a screen in a microprocessor, thereby obtaining the spatial spectra of the image pairs which are recorded by the first and second WEB-cameras and reconstructing the three-dimensional image by resetting a topogram. The relative position of WEB-cameras and the object is determined according to the modifications of spatial spectra of the image pairs generated by the shifts and turns of the object in the field of view of the both WEB-cameras. Said invention makes it possible to automatically determine the relative position of the WEB cameras (2, 3) which are randomly positioned prior to a video conference.

Description

 The formation of three-dimensional images for video conferencing

Technical field

The invention relates to the field of information technology, to the formation of a three-dimensional image of an object, in particular, the face of a human user of a personal computer during communication sessions on the Internet in real time.

State of the art

The known method, WO 2004/082285 of 09/23/2004, for providing a three-dimensional (3D) image (stereo image) to a user during a video conference, i.e. in real time. The disadvantage of this method is the mandatory fixed position of the user relative to the camera and the observation screen and the inability to display the actual three-dimensional image on the screen.

The known method, US patent application N ° 2005/0046698 of 03.03.2005, providing the formation of a three-dimensional (3D) image (stereo image) of the user, providing for the transmission of information on the Internet in real time. The method provides for the transmission of information about the "scene", which is observed by many pairs of cameras, and provides for the free placement of the object within the scene. The specified method does not allow to form a three-dimensional image on the screen.

The known method, US patent N ° 6028672 of 02.22.2000, forming a three-dimensional (3D) image, including obtaining images of an object using at least two cameras and subsequent reconstruction of a three-dimensional image. This way provides three-dimensional image formation with high speed and can be used for video conferencing. The disadvantage of this method, which is based on the triangulation method, is the mandatory fixation of the position of the cameras and the object relative to them.

The known method, European application JNk 1355274 from 10.22.2003, forming a three-dimensional (3D) image comprising obtaining images of an object using at least two video cameras and the subsequent reconstruction of a three-dimensional image. The disadvantage of this method, which is based on the triangulation method, is the mandatory fixation of the position of the cameras. Although the position of the object relative to the cameras is not fixed, but the large amount of calculations does not allow to reconstruct a three-dimensional image in real time for video conferencing.

Thus, it follows from the prior art that when using images of an object for video conferencing, either they do not create a three-dimensional image on the screen, or severe restrictions are placed on the position of the object in front of the cameras. In addition, all known methods suggest a fixed - a priori used in the processor relative positioning of cameras.

A device for the formation of three-dimensional (3D) image for video conferencing, US patent N ° 6028672 from 02.22.2000, which includes two cameras and a processor connected to them. A disadvantage of the known device is mandatory fixed - a priori used in the processor relative position of the cameras, which complicates their location when organizing a video conference in various conditions and under various circumstances. Besides, a fixed location of the object - a participant in a video conference is required and its illumination is required.

Disclosure of invention

The invention is aimed at providing a comfortable videoconference over the Internet and allows you to arbitrarily install video cameras before starting a video conference, and the user can move freely during a video conference, while remaining in the field of view of video cameras.

The technical result achieved in the method of forming a three-dimensional image for video conferencing is to ensure the formation of a real-time three-dimensional image of a user during video conferencing during his movements (rotations) in the field of view of the cameras and in determining the relative position of the cameras arbitrarily set before the start of the video conference.

The technical result achieved in both versions of the device for forming a three-dimensional (3D) image for video conferencing is to automatically determine the relative position of the cameras after their arbitrary installation before starting the video conference and to ensure the formation of a three-dimensional image of the user (object) in real time when it is arbitrarily moved in the field of view of the cameras .

The specified technical result is achieved using the method of forming a three-dimensional image for video conferencing, including registering images of an object using at least two video cameras and subsequent reconstruction in the processor of a three-dimensional image displayed on the screen. Wherein spatial spectra of pairs of images recorded by the first and second video cameras are obtained, and a three-dimensional image is reconstructed from them, and the relative position of the cameras and the object is determined by changes in the spatial spectra of image pairs as a result of displacements and rotations of the object in the field of view of both cameras.

Reconstruct a three-dimensional (3D) image by constructing a tomogram from the spatial spectra of a number of pairs of images obtained by the first and second video cameras. The use of frequency space is the basis of the algorithm for determining the relative position of cameras and an object and reconstruction of three-dimensional images in real time.

You can first determine the relative position of the cameras and the object using the spatial spectra of a number of pairs of images, and then reconstruct its three-dimensional image.

Reconstruction of a three-dimensional image of an object can be synchronized with a frame scan of video cameras. In this case, it is possible to synchronize the frame scans of the first and second video cameras.

An image of an object can be recorded by two or more pairs of cameras, then the spatial spectra of each pair of images obtained by the corresponding pair of cameras are obtained, and a single three-dimensional image of the object is reconstructed from them, and the relative position of all cameras and the object is determined by the changes in the spatial spectra of the pairs of images as a result of displacements and turns the object in the field of view of both cameras.

The specified technical result is achieved when using the device for forming a three-dimensional image for video conferencing, which implements the above method and includes at least two video cameras and a processor connected to them, as well as a multiplexer, a spectral conversion unit (coprocessor) and a memory unit through which the cameras are connected to a processor, providing: determining the relative position of the cameras and the object from changes in the spatial spectra of the image pairs as a result of the displacements and rotations of the object in the field of view of both cameras and constructing a tomogram of the object using the stored a memory unit for information on pairs of spectral (spatial - frequency) image images recorded by the first and second video cameras.

The block of spectral conversion is designed to obtain spatial spectra of images and can be made in the form of a block of fast Fourier transform.

The spectral transform block can also be made in the form of a wavelet transform block.

The device may additionally include one or more spectral transform blocks, while all spectral transform blocks that are coprocessors form a cluster of coprocessors.

The device may further include a unit for converting an analog signal from video cameras to digital before they enter the multiplexer.

The device may additionally include a display connected to the processor, indicating the location of the object in the field of view of the cameras and showing its three-dimensional image.

The specified technical result is achieved when using the device for forming a three-dimensional image for video conferencing, which implements the above method and includes at least two cameras and a processor connected to them, as well as a multiplexer, and a memory unit through which the cameras are connected to the processor, providing: spatial - frequency conversion of images entering the processor from the multiplexer; determining the relative position of the cameras and the object from the changes in the spatial spectra of the image pairs as a result of the displacements and rotations of the object in the field of view of both cameras and constructing a tomogram of the object using information stored on the pairs of spectral (spatial - frequency) image images recorded by the first and second cameras .

Brief Description of the Drawings

Figure l shows a diagram of a device that implements a method of forming a three-dimensional (3D) image for video conferencing.

The best embodiment of the invention

A device that implements a method for forming three-dimensional images of object 1, see FIG. 1, includes video cameras (WEB cameras) 2 and 3, a multiplexer with an analog-to-digital converter 4, a fast Fourier transform unit (coprocessor) 5, a memory unit 6, a processor 7 and a display 8. The fast Fourier transform unit can be supplemented by several connected to it parallel to the fast Fourier transform blocks (coprocessors), forming, forming together with the coprocessor 5 cluster 9.

The images of the object registered by two WEB cameras after the multiplexer 4 are sequentially fed into the fast Fourier transform unit 5, where they are converted into spatial spectra of images and sequentially fed into the memory unit 6, where they are stored in pairs and used by the processor 7 to determine the relative position of the WEB cameras 2 and 3 and tomogram recovery. The operation of the processor 7 to restore the tomogram is synchronized with the cycle of registering a pair of flat projections, i.e. with a frame scan of WEB cameras 2 and 3.

Before starting a video conference, the user (object) places the WEB cameras 2 and 3 in a position convenient for themselves. At the same time, the user exercises control over his presence in the field of view of WEB cameras 2 and 3 through display 8. Through display 8, recommendations can be issued on the sequence of rotations and displacements of user 1 in the field of view of WEB cameras 2 and 3. At the same time, one can limit oneself to natural movements of the user in preparation for a video conference. The processor 7 from the images of user 1, obtained in different angles, determines the Fourier space of the user's images, which serves as the basis for the restoration of the tomogram. The more accurately the Fourier space of the images is determined, the better the three-dimensional image of the user is reconstructed. The operation of restoring a tomogram is carried out by iterations using the method of restoring digital Fourier holograms generated for each pair of flat projections by performing the calculation of the Fourier transform of each projection. Thus, the process of "training" of a specialized program is carried out, and at the same time, an iterative process of restoring a three-dimensional image from pairs of flat projections is carried out.

The condition for completing the training of the program is the consent of the user to complete the learning process, expressed by the corresponding command entered by the user, if the latter is satisfied with the received three-dimensional image, which he can see on display 8. When performing each step of the device’s operation (the step of registering a pair of flat images with WEB cameras), the solution of the tomogram recovery problem using Fourier is refined — images recorded at a frame rate of each of the WEB cameras. Through the use of well-known methods for solving the Radon problem, see G.G. Levin, G.N. Vishnyakov, Optical tomography. - M .: Radio and communications, 1989, pp. 10-12, on the basis of a discrete Fourier sequence of projection images, the current tomogram is formed and the iterative process of refining the three-dimensional image of the object is interrupted by the user upon reaching a satisfactory result, in his opinion.

Since the registration time of 2 images taken from 2 Web cameras is incomparably short compared to the time of significant (for image quality deterioration) user movement at the expected speed in the fields of view of Web cameras, the registration processes of Web cameras can be considered parallel . When using a multiprocessor PC, the parallelism of image processing can be considered absolute. Thus, simultaneous registration of 2 flat projections is achieved, by which the tomogram is restored.

When the user moves arbitrarily in the fields of view of the Web cameras, the position of the image changes, generally speaking, randomly. However, shifts in the plane of the perpendicular optical axis of the WEB camera objective lead to the appearance of phase factors in the Fourier transforms, namely, exp [- 2πj (υ _xΔ x + υ _yΔ y)], where - υ _xΔ , υ _yΔ are the corresponding spatial frequencies, _Δ x, _Δ y, the spatial shift of the user-associated XOY system relative to the fixed coordinate system. As you know, Fourier - the hologram is invariant to the position of the object in the registration plane. A digital Fourier hologram, as well as a physical one, is a Fourier - the image of a holographic object. Therefore, the transmission of spatial frequencies in digital Fourier transforms is invariant to the position of the object (user). This fact is the reason to abandon the hard positioning of WEB cameras. The changes relate to the scale determined by the linear increase along the optical axes of each Web camera, however, the composition of the spatial spectrum, i.e. the relative position of spatial frequencies remains unchanged.

Since image processing can be carried out in real time, the training program can correct the 3-dimensional image of the object (user) in the operating mode. In other words, the training-recovery cycle can continue many times. The speed of formation and transmission of dynamic ZD images synthesized by the proposed method significantly increases with an increase in the number of Web cameras, while the number of these cameras must be even.

Claims

Claim 1. A method of forming a three-dimensional image for video conferencing, comprising registering images of an object using at least two video cameras and subsequent reconstruction in the processor of a three-dimensional image displayed on the screen, characterized in that spatial spectra of pairs of images recorded by the first and second video cameras are obtained, and a three-dimensional image is reconstructed from them, while the relative position of the cameras and the object is determined by the changes in the spatial spectra of the image pairs as a result ate the displacement and rotation of the object in the field of view of both cameras. 2. The method according to p. 1, characterized in that first determine the relative position of the cameras and the object using the spatial spectra of a number of pairs of images, and then reconstruct its three-dimensional image. 3. The method according to p. 2, characterized in that the reconstruction of the three-dimensional image of the object is synchronized with the frame scan of the cameras. 4. The method according to p. 3, characterized in that they synchronize the frame scan of the first and second video cameras. 5. The method according to p. 1, characterized in that the image of the object is recorded by two or more pairs of cameras and receive spatial spectra of each pair of images obtained by the corresponding pair of cameras, and from them reconstruct a single three-dimensional image of the object, and the relative position of all cameras and the object is determined by changes in the spatial spectra of image pairs as a result of displacements and rotations of an object in the field of view of both cameras. 6. A device for generating a three-dimensional image for video conferencing, comprising at least two video cameras and a processor connected to them, characterized in that it further includes a multiplexer, a spectral conversion unit and a memory unit through which the cameras are connected to a processor, providing: determining the relative position video cameras and an object according to changes in spatial - frequency images of image pairs as a result of displacements and rotations of an object in the field of view of both cameras and tomograms s object using the stored information in the memory unit of the vapor space - frequency images of images for the first and second video cameras. 7. The device according to p. 6, characterized in that the spectral transform block is made in the form of a fast Fourier transform block. 8. The device according to p. 6, characterized in that the spectral transform block is made in the form of a wavelet transform block. 9. The device according to p. 6, characterized in that it further includes a unit for converting the analog signal from the cameras to digital before they enter the multiplexer. 10. A device for generating a three-dimensional image for video conferencing, comprising at least two video cameras and a processor connected to them, characterized in that it further includes a multiplexer, and a memory unit through which the video cameras are connected to a processor, providing: fast Fourier transform of the images coming in to the processor from the multiplexer; determining the relative position of the cameras and the object according to the Fourier changes - images of pairs of images as a result of displacements and rotations of the object in the field of view of both cameras and building a tomogram of the object using information about the Fourier pairs stored in the memory unit - image images recorded by the first and second cameras.