WO2011089538A1

WO2011089538A1 - A stereo-calibration-less multiple-camera human-tracking system for human-computer 3d interaction

Info

Publication number: WO2011089538A1
Application number: PCT/IB2011/050138
Authority: WO
Inventors: Naveen Chawla
Original assignee: Individual
Current assignee: Individual
Priority date: 2010-01-25
Filing date: 2011-01-12
Publication date: 2011-07-28
Anticipated expiration: 2012-07-25

Abstract

A human-computer 3D interaction system that does not require a stereo calibration step by the user or installer of the system is proposed.

Description

A STEREO-CALIBRATION-LESS MULTIPLE-CAMERA HUMAN-TRACKING SYSTEM FOR HUMAN-COMPUTER 3D INTERACTION

Description

One of the problems of multiple-camera 3D tracking is stereo calibration. What is proposed is a system that does not require a stereo calibration step by the user or installer of the system.

In 3D human-computer interaction (e.g. with virtual object 18 in Fig.1), it is important to establish a 3D coordinate space relative to the screen (17 in Fig. 1). Normally one would perform a stereo calibration step to establish the position and orientation of the cameras in order to enable 3D tracking of objects located within viewing range of two or more of the cameras.

However for home use this step is cumbersome, and requires re-doing every time any camera is moved even slightly, for example by accident when someone brushes past.

What is therefore proposed first is the use of right-angled brackets such as 1 and 2 in Fig.1, which would be mounted on corners of the user's screen by the user, in line with the edges of the screen, via a self-attaching method, such as magnets, a magnet and a ferrous metal (one of which is attached to the screen by adhesive), adhesive or suction pumps. The size of the user's screen would then either then be input into the system by the user, or calculated automatically by the system by using at least one ultrasonic receiver such as 3 in Fig. 1 and transmitter such as 4 in Fig.1 mounted on different brackets, and using a time-of-flight calculation to calculate the distance between the two, and hence deduce the size of the screen, or using at least one camera such as 5 in Fig. 2, to pick up two or more objects or features in known locations on the other bracket such as

LEDs

6 and 7 in Fig. 2, and using their pixel locations on the camera (or cameras) to calculate, given the known information, the actual distance to those objects from that camera.

The cameras do not need be exposed as shown by 8 in Fig. 1. That is an under-the-hood drawing. The screen-corner-brackets could typically provide casing for the cameras as denoted by 16 in Fig. 1 for discreetness and extra protection and security. The casing in front of the cameras would need to be transparent to whatever wavelengths of electromagnetic radiation the cameras need to detect, or could be transparent to those wavelengths exclusively.

Using screen-corner-mounted brackets is significantly better than using, for example, a single bar at the top of the screen because the user does not have to perform any kind of measurements to place them in a precise way. Also, the units allow for more compact packing of the system than a long bar would, allowing for more convenient transit, storage and stocking of the system.

The cameras used for 3D tracking, denoted by 8 in Fig. 1, would be pre-mounted on the brackets securely and in a precise way, so that their position and orientation in relation to their bracket is already known. This would then be used in conjunction with the calculation of the screen size as described above to deduce the position and orientation of the cameras in relation to the screen. The position and orientation of the cameras relative to the screen is sufficient information to deduce the 3D location, relative to the screen, of any tracked object in view of two or more of the cameras.

Typically, in order for realistic 3D interaction, all versions of the system would need to track, or estimate using tracking, either the 3D locations of the user's left and right eyes or a single compromise between the two, in order to place at least one virtual camera position (two if stereoscopic presentation is being used) in the virtual 3D world to match that location, and set its viewing frustum to the asymmetric pyramid shape that the user's viewing position (or each eye position respectively) would make with the corners of the screen if you were to draw a straight line from that position to each of the four corners of the screen at any given time. Example virtual camera frustums for a stereoscopic system with respect to the left and right eyes is denoted by 9 and 10 in Fig. 1. Markers mounted on the user's stereoscopic glasses such as LEDs 11 and 12 in Fig. 1 provide one way of estimating those positions. Another way would be tracking the user's head and estimating the left and right eye positions that way. This requires a little more computation than tracking LEDs but would be far more desirable.

The system would also typically need to track another part of the user, or a device that the user is in control of, to enable interaction with the virtual world.

An example of such a user-controlled device is denoted by the proposed

handsets

13 and 14 in Fig. 1. Markers such as LEDs 15 in Fig. 1 mounted on each device can provide a simpler way of tracking the device's position and orientation than tracking the device or user by itself.

The system is significantly beneficial in comparison to systems that require stereo calibration because it would be incredibly easy and quick for the user to set up.

A good way of increasing the range of view of the cameras is to use fisheye lenses on them.

Claims

A system which comprises one or more fixtures, each of which has two straight edges at right angles to one another, and which, by its shape, leaves empty the internal region of the conceptual right angle considered from the imaginary point of intersection of the two straight edges and along the two straight edges up to infinity, where each fixture has at least one camera attached to it in a known position and orientation in relation to the fixture, and where each camera is able to send its data to a computer system
A system as claimed in Claim 1 in which the fixture is backed by a magnet
A system as claimed in Claim 1 in which the fixture is backed by a ferrous metal
A system as claimed in Claim 2 which is accompanied by a piece backed by adhesive, which forms an empty inner right angle shape, that includes a ferrous metal
A system as claimed in Claim 2, Claim 3 or Claim 4 which is accompanied by a piece backed by adhesive, which forms an empty inner right angle shape, that includes a magnet
A system as claimed in Claim 1 in which the fixture is backed by adhesive
A system as claimed in Claim 1 in which the fixture is backed by suction pumps
A system as claimed in Claim 1 which is accompanied by suction pumps
A system as claimed in any of Claims 1 to 8 in which the cameras use fisheye lenses
A system as claimed in any of Claims 1 to 9 which automatically calculates the size of a screen on which the fixtures are mounted by using at least one camera and at least two distinguishable feature points within its view, both mounted in known locations on another of the fixtures, and the pixel locations of the features as detected by each camera
A system as claimed in any of Claims 1 to 9 which automatically calculates the size of a screen on which the fixtures are mounted by using at least one ultrasonic transmitter and one ultrasonic sensor on different fixtures within the system, and a the time of flight calculation of at least one ultrasonic signal from the transmitter to the sensor
A system as claimed in any of Claims 1 to 11 which calculates an estimate of a user’s viewing location in 3D relative to the screen using images from the cameras and the hitherto known locations and orientations of the cameras relative to the screen
A system as claimed in Claim 12 which sets the location of a virtual camera relative to a virtual 3D environment to match the estimated viewing location of the user, and which sets the virtual viewing frustum of the virtual camera in the virtual 3D environment to match the shape conceivable if straight lines are drawn between the estimated viewing location of the user and the corners of the screen at a given moment
A system as claimed in any of Claims 1 to 13 which additionally estimates the 3D locations of one or more other parts of the user
A system as claimed in any of Claims 1 to 14 which additionally estimates the 3D locations of one or more devices held in the user’s hands
A system as claimed in any of Claims 1 to 15 which additionally estimates the 3D locations of one or more devices mounted on the user’s body
A system which includes in a single package, an indication of how and all of the components required to assemble the system as claimed in any of the preceding claims
A system as herein described and illustrated by the accompanying drawings