GB2337390A - Deriving a 3D representation from two or more 2D images - Google Patents

Abstract

Apparatus for deriving a reconstruction (30) of an object (3) from first and second images comprises a camera (1) movable freely between two viewpoints (1, 1'). The camera orientations are arranged to be parallel at the instant of capture of the images, eg by superimposing the camera image plane at the second viewpoint on the camera image plane at the first viewpoint and tilting the camera about the X, Y and Z axes until the lines (L1 to L3) connecting correlated points of the first image and the new image seen by the camera converge at a common "vanishing point" (VP). The line (ST) joining the camera positions is derived and a partial 3D reconstruction is derived by aligned virtual projectors (PR1, PR2/PR2') located an arbitrary distance apart (in dependence on a parameter entered by the user) on this line. The process is repeated from other pairs of viewpoints and the resulting partial 3D reconstructions scaled and combined on screen.

Description

2337390 1 Method and Apparatus for 3D Representation The present invention

relates to a method and apparatus for deriving a threedimensional representation from two or more two-dimensional images.

As is well known, in general such a 3D representation can only be generated if the features (eg points) of the images are correlated. Respective features of two overlapping images are correlated if they are derived from (ie conjugate with) the io same feature (eg a point) of the object. If the positions and orientations of the camera at which the overlapping images are acquired are known, then the 31) coordinates of the object in the region of overlap can be determined, assuming that the camera geometry (in particular its focal length) is known.

Hu et al "Matching Point Features with ordered Geometric, Rigidity and Disparity Constraints" MEE Transactions on Pattern Analysis and Machine Intelligence Vol 16 No 10, 1994 ppl041-1049 (and references cited therein) discloses suitable algorithms for correlating features of overlapping images. Such algorithms can be used in the present invention and will not be discussed further. However it should be noted that in some circumstances (eg when the object is relatively simple or one of the known algorithms has failed to correlate sufficient features to enable an accurate reconstruction of part of an object to be derived) the features of the overlapping images can be correlated by eye and the correlation recorded on screen when using an embodiment of the method and apparatus of the present invention.

Assuming that the correlated points of the images have been derived, in principle the information about the position and orientation of the camera required to reconstruct the object in 31) can be obtained either by direct measurement or by various sophisticated mathematical techniques involving processing all the correlated pairs of features. (Stereoscopic camera arrangements in which the cameras are fixed are ignored for the purposes of the present discussion).

EP-A-782,100 discloses a method and apparatus in the first category, namely a photographic 3D acquisition arrangement in which a camera displacement signal and a lens position signal are used to enable a 3D representation of an object to be built up from 2D images acquired by a digital camera. Both the position and the orientation of the camera are monitored. The hardware required to achieve this is expected to be expensive.

A number of research papers have addressed the more difficult question of deriving 2 the correct orientation from the correlated features of the images, as follows:

i) E H Thompson "A rational Algebraic Formulation of the problem of Relative Orientation" Photogrammetric Record Vol 3 No 14 (1959) pp 152-159 sets out a mathematical procedure for aligning two images of the same scene to the correct orientation for reconstructing the scene in 3D, involving an iterative solution of five simultaneous equations derived from five pairs of correlated points.

ii) Richard Hartley et al "Stereo from uncalibrated cameras" Proc. lEEE Conf Computer Vision and Pattern Recognition (1992) pp 761-763 teaches that two 3x4 camera matrices define the camera orientations and locations as well as the internal camera parameters such as focal length. If the cameras are calibrated (ie the internal parameters are known) then the camera matrices can be found from the matched points and hence the true 3D locations of all the points can be found. If the cameras are not calibrated then known "ground control- points must be used to derive the camera matrices and hence the 31) locations of the matched points.

iii) Richard Hartley "Estimation of Relative Camera Positions for Uncalibrated Cameras" Computer Vision-ECCY92, LNCS-Series, Vol 588, 1992 pp 579-587 is a development of the above Hartley paper and shows that the focal lengths as well as the positions and orientations of the cameras can be found from the matched points if all the other internal camera parameters are known.

iv) Hartley et al "Computing Matched-epipolar projections" Computer Vision and Pattern Recognition 1993, pp 549-555 is a development of the above which introduces the epipolar transformation matrix to transform the images to images that would be acquired by cameras placed side-by-side with their optical axes parallel.

The remaining points can then be correlated more easily. Also the cameras do not need to be calibrated.

A disadvantage of the above mathematical methods is the intensive computation required to process all the correlated points. Much of this computation involves the determination of the camera positions and odentations from multiple points and is effectively wasted because the positions and odentations are grossly overdetermined by the large number of processed points.

It has been shown that in general all the information on the viewpoints (ie position and orientation) of the camera(s) can be found from eight pairs of correlated points. However there are special cases in which this is not possible. One extreme example is a situation in which all eight points on the object corresponding to the pairs of 3 corelated points in the images are colinear.

Importantly, another situation in which the camera positions cannot be determined is when the camera orientations (in the reference frame of the object) are identical. As will be apparent from Figure 3 (discusssed below) the camera separation in such a situation is quite indeterminate, no matter how many pairs of points are correlated. In such a situation the size of the object cannot be determined from the two images, even if the focal length and other camera parameters are known, and the 3D reconstruction will need to be multiplied by a scaling factor.

Bearing in mind that the resolution of many digital cameras is severely limited (eg to 640 x 480 pixels) it will be apparent that if the camera orientations are only slightly different then there will be considerable uncertainty in the camera positions and hence in the 3D reconstruction of the object. On the other hand if the camera orientations are deliberately chosen to be very different (ie converging sharply on the object) then points on the object in overhanging regions which are in the field of view when one image is acquired will not be in the field of view when the other image is acquired, preventing 3D reconstruction of regions of overhang.

An object of the present invention is to overcome or alleviate at least some disadvantages of the known methods and apparatus, particularly when the resolution of the images from which the 3D reconstruction is generated is limited Accordingly, in one aspect the invention provides a method of processing overlapping 21) images of an object acquired from different spaced apart viewpoints relative to the object, the separation between the viewpoints not being precisely known, the 21) images being associated with pairs of correlated features, each such pair of correlated features comprising a feature of one 2D image and a feature of another 2D image derived from a common feature of the object, there being a mutual offset between the features of each pair which is determined by the relative rotation (if any) and translation between the viewpoints, the method comprising the step of digitally processing the 2D images to form a 31) reconstruction in which said common feature of the object is located in a simulated 3D space in dependence upon both said mutual offset and a scaling variable which is preferably entered manually by the user.

For example if the actual camera separation is a and the partial 3D reconstruction is generated by virtual projectors with the same optical parameters as the camera(s) and having a separation of a' in simulated 31) space then the scaling factor could be 0 ala' to magnify the partial 3D reconstruction by a factor of ala' and thereby generate a partial 31) reconstruction having the same size as the object.. Such a partial 3D 4 reconstruction will be able to be fitted to other life-size partial 3D reconstructions generated similarly from other pairs of images. In these embodiments, any value of scaling factor which will enable the partial 31) reconstructions to be fitted together will be satisfactory, and can for example be applied by the user during a process of fitting together the partial 3D reconstructions on-screen.

In these embodiments the camera orientation (in the reference frame of the object) at each of the two viewpoints is preferably the same or nearly the same (eg 10 degrees) and the or each partial 31) reconstruction is preferably generated by virtual projectors having the same orientation as the camera(s) and having optical centres 10 on the line joining the optical centres of the camera(s).

In other embodiments the variable will correct for one or more distortions of the partial 3D reconstruction either laterally or in the depth direction (curvature of field) which, as shown below in connection with Figures 8 to 11 can arise from incorrect positioning of the virtual projectors eg a misalignment relative to the camera viewpoints.

In related embodiments the partial 3D reconstruction will be distorted by delibarately misaligning, one or both the virtual projectors relative to the camera 20 viewpoints.

Such a feature is useful in the fields of design, graphics and animation.

Preferably a view of the reconstruction in the simulated 3D space is displayed on a screen and the variable is varied by the user in response to the view displayed on screen.

Preferably the method further comprises the step of acquiring the overlapping 2D images from a camera which is moved relative to the object between the different viewpoints, the net movement of the camera between the viewpoints not being fully constrained.

For example the camera could be mounted on a fixed slide so as to move transversely to its optical axis (so that its orientation and movement along two axes is constrained but its movement along the third axis is not) or it could be mounted on a tripod (so that its movement in the vertical direction and rotation about a horizontal axis are constrained).

However in a preferred embodiment the camera is hand-held.

Optionally, the camera orientation can be measured with an inertial sensor eg a vibratory gyroscope and appropriate filtering and integrating circuitry as disclosed in our UK patent GB 2,292,605B which is hereby incorporated by reference. A Kalman filter is presently preferred for filtering the inertial sensor output signals.

Preferably the orientation of the camera is varied after acquiring a first image from one viewpoint and before acquiring a second image from the other viewpoint so as to maintain its orientation relative to the reference frame of the object when acquiring the second image.

One purely optical way of ensuring that the orientation of the camera is unchanged relative to the orientation at thefirst viewpoint is to vary the orientation until the projections in the image plane of the camera of the correlated points of the first image and the corresponding points of the second image (which is preferably instantaneously displayed on screen) converge at a common point. In the special case in which the distance to the object from the image plane along the camera's optical axis is unchanged between the two viewpoints (in which case the above common point would be at infinity) then the orientation at the second viewpoint can be adjusted until the above projections are parallel.

In another aspect the invention provides a method of determining the motion of a camera relative to an object in the field of view of the camera comprising the steps of projecting the paths of features of the image of the object to a common vanishing point and determining the orientation of the optical center of the camera lens in relation to this vanishing point.

In another aspect the invention provides apparatus for determining the motion of a camera relative to an object in the field of view of the camera comprising means for projecting the paths of features of the image of the object to a common vanishing point and means for determining the orientation of the optical center of the camera lens in relation to this vanishing point.

Some difference in orientation can be tolerated and the resulting distortion in the 31) reconstruction subsequently corrected, as will be shown below.

The smaller the difference in orientation, the smaller the distortion in the 3D reconstruction. Ideally the orientation of the camera is maintained unchanged between the two viewpoints.

Preferably the angle subtended by a pair of correlated features at the corresponding 6 feature of the object is 90 degrees 30 degrees (more preferably 10 degrees).

Ideally the subtended angle is exactly 90 degrees. This feature enables any distortion resulting from a slight change in orientation to be corrected more accurately.

Following the partial reconstruction by the above method, complementary 31) reconstructions of different parts of the object obtained similarly from further sets of overlapping images can be fitted together.

To the extent that the reconstruction is distorted then simple compensations for 10 distortion parallel to the image plane and for curvature of field can be applied in order to enable the 3D reconstruction to be fitted.

It is not necessary (and indeed in many cases it will be computationally inefficient) to correlate all possible features prior to determining the viewpoints of the camera(s) relative to the object. It will usually be simpler to derive the remaining correlations between features of the respective images once the viewpoints have been determined from a small number eg eight pairs of correlated features, by searching for further correlated features along epipolar lines determined from the viewpoint determination.

Once the correlations and positions and orientations of the cameras are known, the 3D configuration of the object is obtainable by projecting each image from (eg a virtual) projector having the same focal length and viewpoint (position and orientation) as the camera which acquired that image. The principal rays from corresponding features of the respective images will intersect in (virtual) 3D space at the location of the object feature.

Accordingly, in another aspect the invention provides a method of generating a 3D reconstruction of an object comprising the steps of projecting images of the object acquired by mutually aligned cameras into simulated 31) space from aligned virtual projectors, the separation of the virtual projectors being variable by the user.

In another aspect the invention provides apparatus for generating a 3D reconstruction of an object comprising two mutually aligned virtual projector means arranged to project images of the object acquired by mutually aligned cameras into simulated 31) space, the separation of the virtual projectors being variable by the user.

Preferably the difference in alignment of the virtual projectors (ie the angle between 7 them) is less than 45 degrees, more preferably less 30 degrees, and is most preferably less than 20 degrees, eg less than 10 degrees. Desirably this angle is less than 5 degrees, eg less than one degree. This feature enables overhanging features of the object to be captured from both camera viewpoints and also facilitates the determination of the line connecting the optical centres of the camera(s) at the different viewpoints as well as the correlation of features between the images.

In yet another aspect the invention provides a method of deriving a 3D reconstruction of at least part of an object comprising the steps of acquiring overlapping 21) images of the object from two spaced apart viewpoints, correlating at least three pairs of features of the images in a region of overlap, digitally pr3cessing the images by projecting the images into virtual 3D space and varying the relative positions andlor orientations of the projections to ensure that the projections of the respective features of at least three of said correlated pairs intersect, and reconstructing a further portion of the object in the region of overlap by projecting further features of the 2D images from the resulting varied relative positions and orientations of projection.

Preferably the origin of each projection is located on a line in simulated 3D space connecting the corresponding optical centres of the camera at the two viewpoints. As explained below with reference to Figure 3, this will result in a scaled partial 31) reconstruction of the object.

In one embodiment a distortion parameter is entered by the user and applied to the 3D reconstruction. For example, after projecting the overlapping 2D images from their nominal viewpoints such that the projections intersect to form an initial 31) reconstruction in simulated 3D space, the initial 3D reconstruction can be rotated whilst constraining the features of the initial 3D reconstruction which are generated from the intersecting projections of correlated features of the projected images to lie on the projections of those features from one of the 2D images, thereby forming a further 31) reconstruction. As is illustrated in Figure 9 (discussed below) this can be used to generate a reconstruction which is parallel to, and therefore a scaled replica of, the actual object surface. The above- mentioned rotation and constraint correct for the lateral distortion caused by the lack of parallelism of the initial 3D reconstruction and the 31) reconstruction generated from correctly aligned virtual projectors.

However as noted above, in many applications it will be desirable to distort the 31) reconstruction relative to the original object in order to achieve a desired artistic effect.

8 It should be noted that the aspect of the invention concerning the derivation of a 3D reconstruction is applicable to previously acquired images in conjunction with their associated pairs of correlated points, however acquired.

Preferably the viewpoints are calculated from at least two (desirably at least three) pairs of correlated features and the 3D reconstruction of the object is generated in dependence upon said calculation of the viewpoints, the calculation of the viewpoints being performed on fewer than all derivable pairs of correlated features.

This preferred feature of the invention is illustrated in Figure 4 (discussed below) which shows the derivation of the line connecting the viewpoints from the pencil of projections from three pairs of correlated features. Since the third projection from the third pair of correlated features P3 and PY intersects the point VP already defined by the other two projections' intersection (and thus merely onfirms the unchanged orientation between the viewpoints) it is not strictly necessary to find the straight line joining the viewpoints. Hence this line can be found from just two pairs of correlated features if there is no change in camera orientation. However greater accuracy will be obtained if more than two pairs of correlated features are processed.

Preferably said calculation is performed on fewer than one thousand pairs of correlated features, more preferably fewer than one hundred pairs of correlated features, desirably fewer than fifty pairs of correlated features eg eight or fewer pairs.. For example the calculation can be performed on four, three or two pairs of correlated points.

Particularly when the viewpoints are parallel or nearly parallel (eg within 10 degrees of each other) the above preferred features result in a greater or lesser degree of economy of processing.

In another aspect the invention provides image -processing apparatus for deriving a 3D representation of at least part of an object from overlapping 2D images thereof acquired from different viewpoints and correlated features of the respective 2D images in a region of overlap which are derived from a feature of the object common to the respective 2D images, the apparatus comprising reconstruction means arranged to form a 31) reconstruction in which said common feature of the c object is located in a simulated 3D space in dependence upon both the mutual offset 9 between said correlated features and avariable which in use is preferably entered by a user.

Further preferred features of the invention are defined in the dependent claims.

Preferred embodiments of the invention are described below by way of example only with reference to Figures 1 to 15 of the accompanying drawings, wherein:

Figure 1 is a diagrammatic view of one apparatus in accordance with the invention; Figure 2 is a flow diagram of one method in accordance with the invention; Figure 3 is a ray diagram showing the relationship between the object, camera is viewpoints, projector viewpoints and partial 3D reconstruction in one embodiment of the invention; Figure 4 is a ray diagram in 3D showing a derivation in accordance with one aspect of the invention of the direction of movement of the camera from the acquired images in the method of Figure 2 and apparatus of Figure 1; Figure 5 is a ray diagram in 3D showing a derivation of the direction of movement of the camera from the acquired images in the method of Figure 2 and apparatus of Figure 1 in the special case in which the camera does not move relative to the object in the Z direction; Figure 6 is a diagram showing the movement of the image of the object in the image plane 1 of Figure 5; Figure 7 is a flow diagram summarising the image processing steps utilised in t he method of Figure 2 and the apparatus of Figure 1; Figure 8 is a 2D ray diagram illustrating the curvature of field resulting from misalignment of one virtual projector relative to the other in the apparatus of figure 1 and method of Figure 2; Figure 9 is a 2D ray diagram illustrating correction of distortion of the partial 3D reconstruction in an embodiment of the invention; Figure 10A is a 2D ray diagram illustrating the curvature of field resulting from a misalignment of a virtual projector by 5 degrees;

Figure 10B is a ray diagram illustrating the curvature of field resulting from a misalignment of a virtual projector by 10 degrees;

Figure 10C is a ray diagram illustrating the curvature of field resulting from a 10 misalignment of a virtual projector by 15 degrees;

Figure 11 is a plot of curvature of field: misalignment in the arrangements of Figures 10A to 1OC;

Figure 10 is a diagrammatic perspective view illustrating the tracking of relative movement between an object and the camera by tracking 3 principal ray lines from a region of the object substantially parallel to the image plane of the camera; Figure 11 is a diagrammatic elevation showing the movement in 2D of the object as seen by the camera of Figure 10; Figure 12 is a view displayed by the screen of the apparatus of Figure 1 during the fitting together of two reconstructions of respective portions of the object; Figure 13 is a view subsequently displayed by the screen of the apparatus of Figure 1 during the fitting together of the 3D reconstruction of Figure 12 with a similarly obtained but relatively elongated partial 3D reconstruction of the object; Figure 14 is a view subsequently displayed by the screen of the apparatus of Figure 1 illustrating the alternative compression in the vertical direction and compression in the horizontal direction of the resulting complete 3D reconstruction in order to achieve a desired relationship to the actual object, and Figure 15 is a diagrammatic representation of a screen shot generated by a program for manipulting the partial 3D reconstructions.

Referring to Figure 1, the apparatus comprises a personal computer 4 (eg a Pentium@ PC) having conventional CPU, ROM, RAM and a hard drive and a connection at an input port to a digital camera 1 as well as a video output port connected to a screen 5 and conventional input ports connected to a keyboard and a mouse 6 or other pointing device. The hard drive is loaded with conventional operating system such as Windows@95 and software:

a) to display images acquired by the camera 1; b) to correlate points in overlapping regions of images input from the camera 1; c) to derive the line of movement of the camera from the acquired images; d) to project images acquired by the camera into a simulated 3D space from virtual projectors located on the line of movement at a separation selected (with the keyboard or pointing device) by the user (thereby creating a partial 3D reconstruction); e) to scale the partial 31) reconstruction along one or more axes and combine such partial 3D reconstructions as illustrated in Figures 12 and 13, and f) to apply or compensate for lateral distortion and curvature of field as illustrated in Figure 9 and Figures 10A to 10C.

The software to carry out function a) can be any suitable graphics program and the software to carry out function b) can be based on the algorithms disclosed in Hu et al "Matching Point Features with ordered Geometric, Rigidity and Disparity Constraints" IEEE Transactions on Pattern Analysis and Machine Intelligence Vol 16 No 10, 1994 ppl041-1049 (and references cited therein). One suitable algorithm is the Gruens algorithm.

Referring to Figure 1, the camera 1, which may be hand-held and carrying a 3-axis vibratory gyroscope G with associated filtering circuitry (a Kalman filter is presently preferred) and integrating circuitry to generate and display on screen 3- axis orientation signals or (as shown at I') may be mounted on a tripod T or other support, for example, defines a first set of axes x, y, z in its initial viewpoint (ie 12 position and orientation) and is used to acquire an image of the object 3 and to display the image on screen 5. The origin of the x y z coordinate system is taken to be the optical centre of the camera lens. The camera is then moved an arbitrary distance to a new position l' and a second image of object 3 is acquired with the orientation of the camera (relative to the x y z coordinate system) maintained unchanged. This unchanged orientation can be achieved with the aid of a suitable support if necessary. A method of checking for changes in orientation, based on the convergence of the projections from corresponding points of the two images in the image plane I of the camera, will subsequently be described with reference to Figure 4. Only a few (eg 3, 4, 5 or up to eg 10) pairs of correlated points need to be found for this purpose and can be derived visually by the user from the images displayed on screen or by the software in computer 4. As will become apparent from Figure 4, the camera movement (ie the line in the x y z coordinate system joining the optical centre of the camera lens in its two positions) can also be found at his stage. In some cases however in which a highly accurate reconstruction is not required, the direction of camera movement can be estimated by the user.

The remaining corresponding points in the two images are then correlated by the computer 4 (preferably taking advantage of the information on camera movement obtained in the previous step, eg by searching along the epipolar line in one image corresponding to a point of interest in the other image) and a partial 3D reconstruction of the object 3 in simulated 31) space is generated from the correlated points as will be described below with reference to Figure 3. Since the distance between the positions 1 and 1' is not known, a parameter representing this distance is entered by the user, either form the keyboard or from the pointing device 6 for example. The computer 4 is programmed to display the resulting 3D reconstruction on screen and the user can vary the parameter interactively in order to arrive at a partial 3D reconstruction in the x y z coordinate system which bears a desired relationship to the actual object 3. Typically this will be stretched or compressed in the direction of movement between positions 1 and V, relative to the actual object.

Depending on the accuracy with which the camera orientation is maintained in the two positions (eg with the aid of orientation signals fromgyroscope G) and the accuracy of the estimated or optically derived camera movement there may be distortions in the partial 3D reconstruction which can be corrected at this stage. The correction of such distortion (and indeed the deliberate addition of such distortion 13 when this is desired) will be described subsequently with reference to Figures 9 and 10A to 10C.

A further partial 31) reconstruction is then generated by moving the camera 1 to a new viewpoint I A such that the object lies in a region of overlap ie such that at least one point P in the camera's field of view at viewpoints 1 and 1' remains in the camera's field of view at viewpoint 1A. This movement can be represented by a rotation 0 about the y axis (resulting in new axes xi, yi and zi) followed by a rotation 0 about the xi axis (resulting in new axes x', y' and z') followed by a rotation K about the z' axis, followed by translations AX, AY and AZ along the resulting axes. Therotation K about the z' axis will in many cases be zero, as shown in Figure 1.

After acquiring an image at the viewpoint 1 A the camera is moved to a new position IA' and a further image is acquired and displayed (the orientation of the camera being adjusted to remain the same as at viewpoint 1A). A further partial 31) reconstruction of object 3 is then performed by the computer 4 in a manner analogous to that described above in connection with viewpoints 1 and F.

If it is assumed that the origin of the x', Y', z' coordinate system is shifted to the optical centre of the camera at viewpoint 1A to give a new coordinate system X, Y, Z then the relationship between any point XYZ in the new XW coordinate system and the same point xyz in the xyz coordinate system is given by:

X COSKCOSO sinKCOSO - COSKsinOsino sinKSino + COSKS cos' in' -sinKCOSO COSKCOSO + siniesinOsino COSKsin - sinKSine)coso -sinO -COSOsino cosecoso X + AX Y + AY z A + 2 The terms in the 30 matrix can be shown to be the cosines of the angles between the axes of the XW and xyz coordinate systems (see eg Boas "Mathematical Methods in the Physical Sciences" Pub John Wiley & Sons, 2nd Edn pp437 and 438).

Accordingly the partial 3D reconstructions can be transformed to a common coordinate system and elong ated/c orn pressed along all three axes to minimise discrepancies between them in their region of overlap and to build up a more complete reconstruction of the object 3.

14 In the present embodiment this process is carried out under the control of a user by displaying the partial 3D reconstructions on screen and varying their relative elongation/compression along three axes, as will be described in ore detail with reference to Figure 13.

In another embodiment the partial 3D reconstructions are combined without user intervention using the Iterative Closest Point algorithm. This algorithm is publicly available and therefore it is not necessary to describe it in detail. Briefly however, it registers two surface edges by searching for a group of (say) the ten closest pairs of points on the respective edges. The surfaces are then repositioned to minimise the aggregate distance between these pairs of points and then a new group of closest pairs of points is found. The repositioning step is then repeated. Other methods of is correlating 3D surface regions are disclosed in our GB 2,292,605B (supra). In accordance with the present invention, a scaling factor or other variable is generated either by the user or iteratively under software control in order to adjust the relative sizes of the partial 31) reconstructions to ensure they can be fitted together into a self-consistent overall surface description of the object.

Returning to the description of the embodiment of Figures 1 and 2, further partial 3D reconstructions can be derived similarly from the other sides of the object 3 and combined with each other and/or with the existing combination of partial 31) reconstructions until a complete 3D representation of the object is achieved.

The method is summarised in Figure 2.

Overlapping images are captured (step S10), pairs of points (or larger features eg lines or areas) are correlated (step S20) and at least the approximate camera movement between the viewpoints is determined (step S30). In the preferred embodiment this is determined with a high degree of accuracy by processing the two images. Partial 31) reconstructions of the object surface are then generated in a simulated 31) space using the computer 4 to process the correlated pairs of points and camera movement (step S40) and these are combined, preferably interactively on screen by the user to give a consistent but possibly distorted (eg compressed or elongated) 3D representation (step S50). Optionally, this is distorted or undistorted by applying appropriate compression or elongation (step S60).

is In an alternative embodiment, steps S30 and S40 can be combined in a matrix processing method, eg using the commercially available INTERSECT program produced by 3D Construction Inc.

Step S40 will now be described in more detail with reference to Figure 3.

Figure 3 shows the object 3 in the field of view of the camera 1 at positions 1 and 1'. 10 Principal ray lines from points Pl, P2 and P3 on the surface of the object pass through optical centres 01 and 02 of the camera in the respective positions 1 and l' and are each imaged on the image plane. The pair of points thus formed on the c image plane from each of points Pl, P2 and P3 (and many other points, not shown) are corresponding points and can be correlated by known algorithms.

is Accordingly it will be appreciated that if virtual projectors PRI and PR2 with the same orientation, focal length and other optical characteristics as the camera at its respective viewpoints were placed at positions 1 and 1' then they would project ray lines into virtual 3D space from the correlated pairs of points which would intersect at the true locations of the corresponding points P I, P2, P3... and all other points on the object surface corresponding to a correlated pair of image points. By a virtual projector is meant any operator on the image which behaves as an optical projector of the image. A suitable software routine for processing the image in the required manner could be written without difficulty be persons skilled in the art.

Although the projectors are shown in Figure 3 with the same orientation in the reference frame of the object (corresponding to the common camera orientation) this is merely a preferred feature which enables the direction of the line ST to be determined more easily by the image processing method disclosed in Figure 4. In principle the above analysis is also applicable to virtual projectors of different orientations, corresponding to respective, different camera orientations. As noted above, the camera orientations can be determined by an inertial sensor and integrating circuitry similar to that disclosed in our above-mentioned UK patent GB 2,292,605B.

If the line ST joining the optical centres of the cameralprojector lenses in positions 1 and 1' is extended and projector PR2 is moved along this line with its orientation 16 unchanged (eg to position PR2' as shown) and the same ray lines are projected from it then these will still intersect the ray lines from projector PR1 but at different positions in simulated 3D space. For example the intersections at Pl, P2 and P3 will be replaced by intersections at Q1, Q2 and Q3 respectively. This is because any ray line from 03 parallel to a ray line from 02 will lie in the same plane as a ray line from 01 intersecting the ray line from 02 and therefore intersect that ray line from 01. For example line 03Q1 lies in the same plane as triangle 01P102 and will therefore intersect line 01P1, in this case at Q1.

Hence a scaled representation 30 can be generated by any pair of virtual projectors on line ST, if line ST is parallel to the line joining the optical centres of the camera lens at the two positions at which the projected images are acquired and the projectors have the same orientation(s) as the camera. This last condition can be satisfied even if the correct orientation is initially unknown, namely by adjusting the orientation about any axis perpendicular to line ST until the respective ray lines from any pair of correlated points intersect. This procedure can be carried out either manually be the user with the aid of a suitable display of the images and ray lines on screen or automatically in software.

Accordingly a 31) representation of the object can be generated once the direction of line ST is known.

Figure 4 illustrates one derivation of line ST (step S40 in Figure 2). For ease of understanding, the camera (having a lens with optical centre 0) is considered to be stationary and the object 3 is considered to move to position 3' along line M. Points Pl, P2 and P3 are imaged as points pl, p2 and p3 when the object is in position 3 and as points p!', p2' and pY when the object is in position Y. If and only if the orientation of the camera relative to the object frame is unchanged between the two positions, then the projections of the lines L I, L2 and L3 joining Pn and pn' (n = 1 to 3 in this illustration but in general will be much larger) will meet at a common point VP, which is somewhat akin to the vanishing point in perspective drawing.

It will be seen that the line joining VP to the optical centre 0 is the line ST which is the locus of the optical centre of the camera (or desired locus of the virtual projector) relative to the object.

17 in general, if the camera is hand-held, the lines Ll, L2, L3.... Ln connecting the correlated points will not meet at a common point, owing to a change in orientation of the camera between the two positions. However the orientation of the camera can be varied about the X, Y and Z axes by the user at the second position whilst displaying the lines Ll, L2, L3... Ln on screen and the second image captured only when these lines converge on or near a common point, as determined either visually or by a software routine. Indeed the image can be captured, under control of the software routine, only when the necessary convergence is achieved.

If gyroscope G is used then its 3-axis orientation signals can be used to maintain the orientation of the camera between the two viewpoints.

It should be noted that if the camera is considered to be moving and the object to be fixed, then the movement of the camera from its original position is derived by superimposing the image plane and associated image of the camera in its new position on the image plane (and image) of the camera in its first position, projecting the lines L I, L2, L3.... Ln in the image plane of the camera in its first position and connecting the resulting point of intersection VP to the optical centre of the camera in its first position. The resulting line ST is the movement of the object in the coordinate frame of the camera at its first position.

in a variant of the above method, a moving image or a rapid succession of still images are acquired by the camera as it moves and the (assumed) rectilinear movement of the camera between each still image or between sequential frames of the moving image is derived by projecting the lines Ll, L2, L3.... Ln in the image plane of the camera to derive the point VP for each incremental movement of the camera. The resulting line segment ST for each incremental movement of the camera will change direction as the direction of movement of the camera moves and the segments can be integrated to determine the overall movement (including any change in orientation) of the camera.

It should be noted that when there is little or no movement of the camera relative to 35 the object in the depth (Z) direction then lines Ll, L2, L3... Ln will be parallel or nearly so and VP will be at infinity or too far away to be determined with accuracy.

This special case is illustrated in Figures 5 and 6. Points A, B and C at corners of 18 object 3 in its initial position are imaged by lens L as triangle abc in the image plane 1 of the camera. When the object moves to a new position 3' these corners A', B' and C' are imaged as triangle a'bV which will be similar (in the narrow geometrical sense) to triangle abc but may for example be rotated. For reasons whch are explained below, it will be assumed initiallly that points A, B and C lie in a plane which is substantially parallel to the image plane I.

It should be noted that the images abc and a'b'c' would also be obtained from a smaller object of the same shape correspondingly nearer the camera, as illustrated by face A1B1C1.

Referring now to Figure 6 which shows the object 3/Y (not its image) as seen by the camera, various possible faces ABC, A1B1C1 and A2B2C2 are shown. There will be a continuous range of possible sizes for face ABC; for the sake of clarity only the above three are shown. However the possible faces all have a common centroid P.

When the object moves to a new position 3' illustrated by face A'B'C' the centroid will move to a new position Q and the line PQ, which will be parallel to the image plane I (Figure 5) will represent the direction of movement irrespective of the true size and distance of the object. In fact this will remain true even if there is some rotation about the Z axis (such that the lines AX, BB' and CC' are not parallel).

However if the camera is rotated at the second position about the Z axis to ensure that the above lines AK, BW and CC' are in fact parallel, then the above analysis holds true even if the points A, B and C do not lie in a plane parallel to the image plane I; for example if the image abc is derived from points A1BC in Figure 5. The corresponding centroid (not shown) of A1BC in Figure 6 is displaced from centroid P and the corresponding centroid of Al'B'C' after movement of the object to position 3' is similarly displaced from centroid Q. However the line joining these new centroids will be parallel to line PQ and will therefore correctly indicate the direction of movement of the object relative to the camera.

Even if there is some movement of the camera in the Z direction such that lines AX, BB' and CC' are not parallel but converge to a distant point, line PQ will still correctly represent the direction of movement of the object 3.

19 Thus the above method overlaps to some extent with the method of Figure 4.

The overall method of determining the direction of movement of the camera is illustrated in Figure 7. A first image is captured (step S 11) and the camera is moved to a new position with the object still in its field of view and the first image is displayed on screen 5 (Figure 1), superimposed on the instantaneous image seen by the camera at the new position (step S12). Corresponding points are correlated, either by eye (ie by the user) or by a suitable software routine (step S 13). Only a small number of points need to be correlated at this stage, eg 100 or fewer, eg 10, depending on the processing speed and the accuracy required.

is If it appears to the user that there has been movement in the Z direction the method branches to step S 14 at which the orientation of the camera is adjusted about the X, Y and Z axes until the correlated points converge to a common "vanishing point" VP (Figure 4). At this stage the second image is captured and stored in memory.

In step S15 a line is projected from the point VP through the optical centre of the camera lens to find the locus ST (Figure 4) and with the aid of this information, further correlation of the images is performed and a partial 3D reconstruction is generated by the moethod illustrated in Figure 3 (step S 17).

Preferably, step S 15 also involves or is preceded by the selection of a camera model (eg from a list displayed on screen) by the user. The computer 4 is programmed to store a list of camera models M1, M2.... Mn in association with the parameters of each model needed for the calculation of the required projection and other processing.The following parameters may be stored:

Camera model Mn:

pixel size in x direction pixel size in y direction focal length x dimension of film plane (in pixels) y dimesnion of film plane (in pixels).

If it appears to the user (eg as a result of a failure to find a reasonably close "vanishing point" VP) that there has not been significant movement in the Z direction then the movement of the object is assumed to be parallel to the image plane and the line PQ is found from the images of a group of eg three or more points (not necessarily corners) preferably lying in or close to a plane parallel to the image plane (step S16) before proceeding to step S17.

If only because most digital cameras currently on the market have a limited resolution (eg 1024 x 768 pixels, 640 x 480 pixels or even less) there will almost inevitably be some error in the determination of the camera movement and hence in 10 the relative placement and orientation of the virtual projectors PRI and PR2 in Figure 3. The effect of this error is illustrated in Figure 8.

Figure 8 is a ray diagram orthogonal to the image plane showing the imaging of a 0 line of points P 1, P2 and P3 on the surface of the object 3 by a camera at positions 1 and 1'. For the sake of clarity the lens L is shown only at the first position 1. It will be noted that the orientation of the camera is the same at the two positions.

Virtual projectors PRI and PR2 would correctly reconstruct the object as reconstruction 30 if located at the above positions with the above orientation. If projector PR2 were moved to position PR2' along the direction of movement ST (Figure 4) of the cameras then the resulting reconstruction 30' would still show the points (P1', P2' and PY) on a straight line with no curvature of field,with correct scaling in the lateral direction (ie the length ratio P1P2/P2P3 = Pl'P27P2'PY).

If the projector is successively rotated to a new positions PR2" and PR2.. . then successively greater curvature is introduced into the reconstruction 3C or 30, as shown. The relationship between the above curvature of field of the reconstruction

30 and the angular error in the orientation of the second virtual projector relative to the first is considered below in relation to Figures 10A to 10C.

However, referring to Figure 9, in which the respective ray lines from the correlated points of the images projected by projectors PR1 amd PR2 are substantially orthogonal, it can be seen that an error in the relative orientation of the projectors, resulting in the projection of the second image from projector PR2', results in a negligible curvature of field: line 30' defined by the intersection of the ray lines from PR2' with the corresponding ray lines from projector PRI is substantially

21 straight, like line 30 which is defined by the corresponding intersections of the ray lines from correctly co-oriented projectors PRI and PR2. Accordingly it is a preferred feature of the invention that the camera positions are so chosen that the angle subtended by the ray lines from a pair of correlated points at their intersection at their corresponding point at the object surface is substantially 90 degrees, eg 90 degrees 30 degrees, preferably 90 degrees 20 degrees, most preferably 90 degrees 10 degrees. This minimises the correction required due to curvature of field.

There remains the lateral distortion of the reconstruction, ie the discrepancy between the ratios P1P2/P2P3 and PPP2'/P2TY (Figure 8). If however the partial 3D reconstruction 30' is rotated to position 30C so as to lie parallel to the image plane of projector PRI, as shown in Figure 9, and all its points are shifted so as to to be intercepted by the ray lines from projector PRI, then the triangles defined by these shifted points and the ray lines from PRI will be geometrically similar to the triangles defined by the corresponding points on line 30 and these ray lines, and hence the distribution of points along line 30C will be a scaled replica of the corresponding distribution along line 30. It will be understood that this is so 20 irrespective of the angle subtended by the ray lines at their intersection at their corresponding point at the object surface.

Accordingly, distortion parallel to the image plane can be corrected for (or deliberately applied) by rotating the partial 31) reconstruction about an axis perpendicular to the image plane of a projector used to generate that partial reconstruction whilst constraining the points in the partial 31) reconstruction to lie on their ray lines from that projector.

Figures 10A to 10C illustrate the curvature of field resulting from angular misalignment of one projector PR2' from its correct orientation PR2. In each case the correct reconstruction 30 defined by PRI and PR2 is planar and the centre of curvature CN of the actual reconstruction 30' is shown (and was derived geometrically). In Figures 10A to 10C, the misalignment is generated by rotation of the projector PR2 about the optical centre of its lens by 5 degrees, 10 degrees and 15 degrees respectively.

The radius of curvature R of reconstruction 30' is inversely proportional to the 22 misalignment as shown in Figure 11. It should be noted that in Figures 10A to 10C, the angle subtended by the ray lines at their intersection at their corresponding point at (the reconstruction 30 of) the object surface is very much less than the optimum angle of 90 degrees and hence the degree of curvature is very much greater than would normally be obtained in practice.

Hence the method of the invention allows a considerable latitude in the orientation of the projectors which implies that a considerable uncertainty in the relative orientation of the camera at the two viewpoints is permissible.

The corrections noted above can be applied at any stage during the generation or fitting together of the partial 31) reconstructions.

It should be noted however that only a limited misalignment of the virtual projectors PRI and PR2 (FIgure 3) is permissible. Referring to Figure 3, all the principal ray lines projected from the correlated points will intersect only if the projectors have the correct orientation. The latitude in orientation arises only from the finite resolution of the camera(s) which implies that the ray lines from the projectors have a finite thickness or do not intersect exactly. Thus a point on the reconstruction 30 can be found by determining the midpoint of the shortest line joining the ray lines from the respective projectors, provided that the length of this shortest line does not exceed a predetrmined limit corresponding to the resolution of the camera(s).

Figures 12 to 14 illustrate the fitting together and manipulation of the partial 31) reconstructions on screen.

In Figure 12 a partial 3D reconstruction R1 and a partial 31) reconstruction R2 are fitted together at their corresponding points F and F', A and A' and result in a composite (but still incomplete) 3D reconstruction R3 as shown in Figure 13 having an array comprising points E, B and D from R1 and points C' and G' in defined positions which can be fitted to a further partial 3D reconstruction R4 as shown in Figure 13.

Referring to Figure 13, it is assumed that faces ABC, WCB, DGEF and ABDEF of R4 are formed but that the region bounded by points A, C, G and F is not. The user rotates R3 on screen using eg mouse 6 (Figure 1) to give a rotated partial 31) 23 representation RY which is aligned with R4 and which he/she then sees is horizontally compressed relative to R4. Accordingly R3 can be expanded horizontally or R4 can be compressed horizontally to enable them to be fitted together to form a self consistent, complete 3D representation of the object R5.

As shown in Figure 14, this can be compressed or expanded along any desired axis or otherwise distorted under the control of the user to produce a final representation R6 or R6' having any desired relationship with the actual object 3.

Figure 15 shows a screen shot of the user interface generated by a program for manipulating the 3D reconstructions 30. Buttons BN are provided which can be clicked on by the mouse under the control of the user and,when thus selected, enable the user to drag portions of the displayed representation so as to zoom, rotate and panthe view of the object, as well as tocome closer to and move away from the object ("nearer" and "further" buttons respectively). As described this far, the interface is similar to the publicly available interface of the COSMO PLAYER web browser plug-in produced by CosmoSoftware Inc, of California USA. However, in accordance with a feature of the present invention, "wheels- W1 and W2 are provided which are rtoatable by the mouse and enable the user to adjust the separation between the virtual projectors and to vary the distance to the object respectively. The latter control is effectively a perspective control. Optionally, further buttons or other means (not shown) may be provided to enable distortions (such as thos described above) to be applied in a graphical fashion, or to enable other distortions such as shear dostortion to be applied.

Preferred features of any aspect of the invention can be combined with any other aspect of the invention.

Claims (53)

1. Claims

   24 1. A method of processing overlapping 2D images of an object acquired from different spaced apart viewpoints relative to the object, the separation between the viewpoints not being precisely known, the 2D images having pairs of correlated features, each such pair of correlated features comprising a feature of one 2D image and a feature of another 2D image derived from a common feature of the object, there being a mutual offset between the features of each pair which is determined by the relative rotation (if any) and translation between the viewpoints, the method comprising the step ofdigitally processing the 2D images to form a 3D reconstruction in which said common feature of the object is located in a simulated 3D space in dependence upon both said mutual offset and ascaling variable.

   is
2. 2. A method as claimed in claim 1 wherein a view of the reconstruction in the simulated 3D space is displayed on a screen and the scaling variable is entered by the user.
3. 3. A method as claimed in claim 1 or claim 2 further comprising the step of acquiring the overlapping 2D images from a camera which is moved relative to the object between the different viewpoints, the net movement of the camera between the viewpoints not being fully constrained.
4. 4. A method as claimed in claim 3 wherein the respective orientations of the camera at the different viewpoints relative to a reference frame which is fixed with respect to the object differ by less than 45 degrees.
5. 5. A method as claimed in claim 4 wherein the difference between said respective odentations of the camera is less than 30 degrees.
6. 6. A method as claimed in claim 5 wherein the difference between said respective orientations of the camera is less than 10 degrees.
7. 7. A method as claimed in any preceding claim wherein the 3D reconstruction is generated by projecting the correlated features from positions on a straight line in simulated 3D space which corresponds to the straight line joining the positions of the camera at which said 21) images are acquired.
8. 8. A method as claimed in claim 7 wherein said straight line in simulated 3D space is determined from a pencil of projections from correlated pairs of points of aligned 2D images.
9. 9. A method as claimed in claim 7 wherein said straight line in simulated 3D space is determined from parallel projections from correlated pairs of points of aligned 2D images.
10. 10. A method as claimed in any preceding claim wherein the camera orientation is determined from an output signal of an inertial sensor.
11. 11. A method as claimed in any of claims 3 to 10 wherein the 2D images are 15 acquired by a hand-held camera.
12. 12. A method as claimed in any preceding claim wherein said scaling variable is varied by the user to enable the 3D reconstruction to be fitted to another 3D reconstruction similarly derived from 2D images acquired at another pair of viewpoints.
13. 13. A method as claimed in any preceding claim wherein said features are points.
14. 14. A method as claimed in any of claims 1 to 12 wherein said features are lines.
15. 15. A method as claimed in any of claims 1 to 12 wherein said features are areas.
16. 16. A method as claimed in any of claims 3 to 15 wherein the viewpoints are calculated from one such mutual offset between a first pair of correlated features at and a further such mutual offset between a second pair of correlated features, the calculation being performed on fewer than allderivable pairs of correlated features, further pairs of correlated features are derived from a search along epipolar lines derived from the viewpoint determination and the 3D reconstruction of the object is generated from the viewpoints and further pairs of correlated features.
17. 17. A method as claimed in claim 16 wherein said calculation is performed on fewer than one thousand pairs of correlated features.

    26
18. 18. A method as claimed in claim 17 wherein said calculation is performed on fewer than one hundred pairs of correlated features.
19. 19. A method as claimed in claim 18 wherein said calculation is performed on fewer than fifty pairs of correlated features.
20. 20. A method as claimed in claim 19 wherein said calculation is performed on four eight or fewer pairs of correlated points.
21. 21. A method as claimed in claim 20 wherein said calculation is performed on two or three or four pairs of correlated points.
22. 22. A method as claimed in any preceding claim comprising the step of repeating the method of claim 1 by electronically processing further overlapping 2D images of the object acquired from different further viewpoints to form a further 3D reconstruction, the first-mentioned 31) reconstruction and the further 3D reconstruction being combined by manipulations in a simulated 3D space involving one or more of rotation and translation, any remaining discrepancies between the 31) reconstructions being reduced or eliminated by scaling one 3D reconstruction relative to the other along at least one axis.
23. 23. A method as claimed in claim 22 wherein at least two of the 3D reconstructions are simultaneously displayed on screen and their manipulations are performed in response to commands entered by a user.
24. 24. A method as claimed in claim 23 wherein the manipulations of the 3D reconstructions are performed under the control of a computer pointing device operated by the user.
25. 25. A method as claimed in any of claims 22 to 24 wherein a distortion parameter is entered by the user and applied to said first-mentioned andlor said further 3D reconstruction.
26. 26. A method as claimed in claim 25 wherein overlapping 2D images are projected from their nominal viewpoints such that the projections intersect to form an initial 31) reconstruction 27 in simulated 31) space, and the initial 3D reconstruction is rotated whilst constraining the features of the initial 3D reconstruction which are generated from the intersecting projections of correlated features of the projected images to lie on the projections of those features from one of the 2D images, thereby forming a further 31) reconstruction.
27. 27. A method as claimed in any of claims 22 to 26 wherein a further parameter indicative of curvature of field is entered by the user and used to adjust the curvature of field of said first-mentioned andlor said further 31) reconstruction.
28. 28. A method as claimed in any preceding claim wherein the angle subtended by a pair of correlated features at the corresponding feature of the object is 90 degrees degrees.
29. 29. A method as claimed in claim 28 wherein the angle subtended by a pair of correlated features at the corresponding feature of the object is 90 degrees 10 degrees.
30. 30. Image processing apparatus for deriving a,3D representation of at least part of an object from overlapping 21) images thereof acquired from different viewpoints and correlated features of the respective 21) images in a region of overlap which are derived from a feature of the object common to the respective 2D images, the 25 apparatus comprising reconstruction means arranged to form a 31) reconstruction in which said common feature of the object is located in a simulated 3D space in dependence upon both the mutual offset between said correlated features and a scalingvariable.
31. 31. Apparatus as claimed in claim 30 comprising display means arranged to display a view of the reconstruction in simulated 3D space, the size of the displayed reconstruction being dependent upon the value of th scaling variable.
32. 32. Apparatus as claimed in claim 30 or claim 31 wherein said features are points.
33. 33. Apparatus as claimed in claim 30 or 31 wherein said features are lines.
34. 34. Apparatus as claimed in claim 30 or claim 31 wherein said features are areas.

    28
35. 35. Apparatus as claimed in any of claims 30 to 34 which is arranged to calculate the viewpoints from the mutual offset between at least two but not all derivable pairs of correlated features in an overlapping region of the 2D images,to derive further pairs of correlated features from a search along epipolar lines derived from the viewpoint determination and to generate the 3D reconstruction of the object from the further pairs of correlated features and the viewpoints.
36. 36. Apparatus as claimed in claim 35 which is arranged to perform said calculation on fewer than one thousand pairs of correlated features.
37. 37. Apparatus as claimed in claim 36 which is arranged to perform said calculation on fewer than one hundred pairs of correlated features.
38. 38. Apparatus as claimed in claim 37 which is arranged to perform said calculation on fewer than fifty pairs of correlated features.
39. 39. Apparatus as claimed in claim 38 which is arranged to perform said calculation on eight or fewer pairs of correlated points.
40. 40. Apparatus as claimed in claim 39 which is arranged to perform said calculation on two or three or four pairs of correlated points.
41. 41. Apparatus as claimed in any of claims 30 to 40 which is arranged to derive a further 3D reconstruction from further overlapping 2D images acquired from different viewpoints and correlated features of the respective further 2D images in a region of overlap which are derived from a feature of the object common to the respective further 21) images, the combining means being arranged to combine the first-mentioned 3D reconstruction and the further 3D reconstruction by manipulations in a simulated 3D space involving one or more of rotation and translation, the apparatus further comprising scaling means arranged to reduce or eliminate any remaining discrepancies between the 3D reconstructions by scaling one 31) reconstruction relative to the other along at least one axis.
42. 42. Apparatus as claimed in claim 41 which is arranged to display both 3D reconstructions simultaneously and to manipulate them in simulated 31) space in 29 response to commands entered by a user.
43. 43. Apparatus as claimed in claim 42 which is arranged to perform the manipulations of the 3D reconstructions under the control of a computer pointing device.

    is
44. 44. Apparatus as claimed in any of claims 41 to 43 which is arranged to correct distortion of said first-mentioned andlor said further 3D reconstruction resulting from an incorrect or incomplete calculation of a said viewpoint.
45. 45. Apparatus as claimed in any of claims 41 to 44 which is arranged to correct the curvature of field of said first-mentioned and/or said further 3D reconstruction resulting from an incorrect or incomplete calculation of a said viewpoint.
46. 46. Image processing apparatus as claimed in any of claims 30 to 45, further comprising a camera whose position andlor orientation are not fully constrained with respect to the frame of the object, the camera being arranged to acquire said 2D images.
47. 47. Apparatus according to claim 46 comprisfing inertial sensor means arranged to determine the orientation of the camera relative to the object at the time of acquisition of said 2D images.
48. 48. Apparatus for deriving a 3D representation of at least part of an object, substantially as described hereinabove with reference to Figures 1 to 4 and 12 to 16 15 of the accompanying drawings, optionally in conjunction with Figures 5, 6 and 7 or any of Figures 8 to 11.
49. 49. A method of deriving a 3D representation of at least part of an object, substantially as described hereinabove with reference to Figures 1 to 4 and 12 to 16 15 of the accompanying drawings, optionally in conjunction with Figures 5, 6 and 7 or any of Figures 8 to 11.
50. 50. A method of determining the motion of a camera relative to an object in the field of view of the camera comprising the steps of projecting the paths of features of the image of the object to a common vanishing point and determining the orientation of the optical center of the camera lens in relation to this vanishing point.
51. 51. Apparatus for determining the motion of a camera relative to an object in the field of view of the camera comprising means for projecting the paths of features of the image of the object to a common vanishing point and means for determining the orientation of the optical center of the camera lens in relation to this vanishing point.
52. 52. A method of generating a 3D reconstruction of an object comprising the steps of projecting images of the object acquired by mutually aligned cameras into simulated 3D space from aligned virtual projectors, the separation of the virtual projectors being variable by the user.
53. 53. Apparatus for generating a 3D reconstruction of an object comprising two aligned virtual projector means arranged to project images of the object acquired by mutually aligned cameras into simulated 3D space, the separation of the virtual projectors being variable by the user.