WO2009016624A2

WO2009016624A2 - System and method employing thermal imaging for object detection

Info

Publication number: WO2009016624A2
Application number: PCT/IL2008/001042
Authority: WO
Inventors: Ariel Almos; Guy Kotlizky
Original assignee: Eyeclick Ltd.
Priority date: 2007-07-30
Filing date: 2008-07-29
Publication date: 2009-02-05
Also published as: WO2009016624A3

Abstract

A system for identifying and/or tracking a subject in an environment is provided. The system includes a thermal imaging device for capturing a thermal image of the environment and a computing platform executing a software application being configured for analyzing at least a portion of the thermal image to thereby automatically identify and/or track the subject within the environment.

Description

SYSTEM AND METHOD EMPLOYING THERMAL IMAGING FOR OBJECT

DETECTION

FIELD AND BACKGROUND OF THE INVENTION

The present invention relates to a system and method which utilize thermal imaging for silhouetting subjects in an environment and for translating presence and motion of the subject into input commands for interactive systems.

Image processing is used in many areas of analysis, and is applicable to numerous fields including robotics, control engineering and safety systems for monitoring and inspection, medicine, education, commerce and entertainment. It is now postulated that emergence of computer vision on the PC in conjunction with novel projected display formats will change the way people interact with electronic devices.

Detecting the position and movement of an object such as a human is often referred to as "motion capture." With motion capture techniques, mathematical descriptions of an objects movements are input to a computer or other processing system. For example, natural body movements can be captured and tracked in order to study athletic movement, capture data for later playback or simulation, to enhance analysis for medical purposes, etc. Although motion capture provides benefits and advantages, simple visible-light image capture is not accurate enough to provide well-defined and precise motion capture and as such presently employed motion capture techniques utilize high-visibility tags, radio-frequency or other types of emitters, multiple sensors and detectors or employ blue-screens, extensive post-processing, etc. Some motion capture applications allow a tracked user to interact with images that are created and displayed by a computer system. For example, an actor may stand in front of a large video screen projection of several objects. The actor can move, or otherwise generate, modify, and manipulate, the objects by using body movements. Different effects based on an actor's movements can be computed by the processing system and displayed on the display screen. For example, the computer system can track the path of the actor in front of the display screen and render an approximation, or artistic interpretation, of the path onto the display screen. The images with which the actor interacts can be displayed on the floor, wall or other surface; suspended three- dimensionally in space, displayed on one or more monitors, projection screens or other devices. Any type of display device or technology can be used to present images with which a user can interact or control.

Although several such interactive systems have been described in the art (see, for example, U.S. Pat. Application Nos. 08/829107; 09/909857; 09/816158; 10/207677; and U.S. Pat. Nos. 5,534,917; 6,431,711; 6,554,431 and 6,766,036), such systems are incapable of accurately translating presence or motion of an untagged object into input data. This limitation of the above referenced prior art systems arises from their inability to efficiently separate an object from its background; this is especially true in cases where the background includes a displayed image.

In order to traverse this limitation, Reactrix Inc. has devised an interactive system which relies upon infra-red grid tracking of individuals (U.S. Pat. Application No. 10/737730). Detection of objects using such a system depends on differentiating between surface contours present in foreground and background image information and as such can be limited when one wishes to detect body portions or non-human objects. In addition, the fact that such a system relies upon a projected infrared grid for surface contour detection substantially complicates deployment and use thereof.

Thus, the prior art fails to provide an object tracking system which can be used to efficiently and accurately track untagged objects within an environment without the need for specialized equipment.

While reducing the present invention to practice, the present inventors have uncovered that thermal imaging can be utilized for silhouetting subjects in an environment. As described hereinbelow systems employing thermal imaging are highly suitable for use with interactive applications which translate presence and/or motion of subjects in an environment into input commands.

SUMMARY OF THE INVENTION

According to one aspect of the present invention there is provided an interactive system for translating presence and/or motion of a subject in an environment into input commands for altering a function of a device, the system comprising: (a) a thermal imaging device for capturing a thermal image of the environment; (b) a device having a controllable function; and (c) a computing platform executing at least one software application being configured for: (i) analyzing at least a portion of the thermal image to thereby automatically detect the presence and/or motion of the subject within the environment; and (ii) translating the presence and/or motion of the subject within the environment into input commands for controlling a function of the device. According to further features in preferred embodiments of the invention described below, the software application employs a silhouetting algorithm.

According to still further features in the described preferred embodiments the device is an image display device.

According to still further features in the described preferred embodiments an image displayed by the image display device is modifiable according to the input commands.

According to still further features in the described preferred embodiments the image is a video image and whereas a content of the video changes according to the input commands. According to still further features in the described preferred embodiments the system further comprises an image capture device capable of capturing a visible-light image of the environment.

According to still further features in the described preferred embodiments the image display device is an image projector. According to still further features in the described preferred embodiments the image display device is a display.

According to another aspect of the present invention there is provided a method of translating presence or motion of a subject in an environment into commands for controlling a function of a device, the method comprising: (a) acquiring a thermal image of the environment; (b) computationally analyzing at least a portion of the thermal image to thereby automatically identify and/or track the subject within the environment;

(c) computationally translating presence and/or motion of the subject into input data; and (d) using the input data as input commands for controlling the function of the device. According to still further features in the described preferred embodiments step

(b) is effected using a silhouetting algorithm. According to still further features in the described preferred embodiments the device is an image display device and whereas the input commands are utilized to control an image displayed by the image display device.

According to still further features in the described preferred embodiments the image display device is an image projector.

According to still further features in the described preferred embodiments the image display device is a display.

The present invention successfully addresses the shortcomings of the presently known configurations by providing a system and method for efficiently extracting a silhouette from a background image.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, suitable methods and materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and not intended to be limiting.

Implementation of the method and system of the present invention involves performing or completing selected tasks or steps manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of preferred embodiments of the method and system of the present invention, several selected steps could be implemented by hardware or by software on any operating system of any firmware or a combination thereof. For example, as hardware, selected steps of the invention could be implemented as a chip or a circuit. As software, selected steps of the invention could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In any case, selected steps of the method and system of the invention could be described as being performed by a data processor, such as a computing platform for executing a plurality of instructions. BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present invention only, and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice.

In the drawings:

FIG. 1 illustrates an interactive system in accordance with the present invention

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is of a system and method which can be used to detect objects present in an environment. Specifically, the present invention can be used to detect presence and motion of an object in an environment by utilizing thermal imaging and silhouetting algorithms. The principles and operation of the present invention may be better understood with reference to the drawings and accompanying descriptions.

Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.

Detecting the position and movement of an object such as a human in an environment such as an indoor or an outdoor space is typically effected by various silhouetting techniques. Such techniques are typically utilized to determine presence and motion of an individual within the environment for the purpose of tracking and studying athletic movement, for simulation, to enhance analysis for medical purposes, for physical therapy and rehabilitation, security and defense applications, Virtual reality applications, computer games, motion analysis for animation production, robot control through body gestures, etc. Several silhouetting algorithms are known in the art, examples include Plankers and P. Fua. "Model-Based Silhouette Extraction for Accurate People Tracking" In European Conference on Computer Vision, Copenhagen, Denmark, May 2002; and Wren et al. "Real-time tracking of the human body" /EEE Transactions on Pattern Analysis and Machine Intelligence, 1997. Such silhouetting algorithms typically process visible light image information captured from an environment to separate foreground and background information and generate a detectable silhouette.

While searching for ways to improve the efficacy of object silhouetting in an environment, the present inventors postulated that use of thermal imaging can greatly enhance silhouette generation especially in cases where the detected object is an individual.

Thermal imaging devices are well known in the prior art. In principle, infrared radiation emitted by a warm object is directed onto a photoconductive detector (see, for example, Thermal Imaging Systems by J. M. Lloyd, Plenum Press, 1975) and the thermal image is reconstructed from the electrical response. Thermal imaging is utilized in a variety of applications ranging from surveillance [Handheld thermal imaging for law enforcement applications. Terence L. Haran and Melinda K. Higgins and Michael L. Thomas. Proceedings of SPIΕ ~ Volume 5403; Sensors, and Command, Control, Communications, and Intelligence (C3I) Technologies for Homeland Security and Homeland Defense III, Edward M. Carapezza, Editor, September 2004, pp. 831-84] to medical applications (see, for example, www.cti- net.com). Thermal imaging has also found use in Fire Service, HAZMAT, EMS, Law Enforcement and Homeland Security.

Studies have shown that thermal imaging can be utilized to efficiently differentiate between an individual and an environment, including environments where light imaging cannot be utilized for such purposes (see Tracking and Imaging Humans on Heterogeneous Infrared Sensor Array for Law Enforcement Applications Feller et al. Duke University). As such, thermal imaging is highly suitable for subject identification and tracking and can be combined (as suggested herein) with various silhouetting algorithm to enable automatic subject identification and tracking in, for example, interactive media systems. Although infrared lighting has been used for subject silhouetting, see "Virtual

PAT, a virtual personal aerobics trainer" authored by James W. David and Aaron F. Bobick of the MIT media labs, the prior art fails to describe interactive systems which translate subject presence and/or movement as determined by silhouetting of thermal images into input commands for altering a function of a device, such as for example, an image display device.

Thus, according to one aspect of the present invention there is provided a system for identifying and/or tracking a subject in an environment. The system includes a thermal imaging device for capturing a thermal image of the environment and a computing platform which executes a software application designed and configured for analyzing at least a portion of the thermal image to thereby automatically identify and/or track the subject within the environment.

The phrase "environment" refers to either an outdoor or an indoor three dimensional space of any size. The environment can be a room, a hall, a stadium and the like. Based on its ability to automatically identify and track individuals present within an environment, the system of the present invention can be utilized for a variety of purposes.

A presently preferred embodiment of the system of the present invention is utilized for providing interactive media, such as interactive advertising. It should be noted that this embodiment is just one example of one use of such tracking. Other embodiments of the present system which are suitable for use in security applications, the medical field and the like can easily be realized by an ordinary skilled artisan privileged to the teachings provided herein.

Referring now to the drawings, Figure 1 illustrates an interactive system (which is referred to herein as system 10) for translating presence and/or motion of a subject in an environment into commands for controlling a device having a controllable function.

The phrase "device having a controllable function" refers to any mechanical or electronic device which has a function which can be controlled. Examples include image or light projectors or displays, speakers, water fountains, electric motors, mechanical devices and the like.

System 10 includes a thermal imaging device 12 for acquiring a thermal image of the environment. Examples of thermal imaging devices that can be utilized in system 10 include the Omega camera from Indigo Systems. Thermal imaging device 12 is preferably a camera fitted with a lens capable of a field of view (FOV) having a depth ranging from less than a foot or several feet to several tens of feet.

System 10 also includes a controllable device 16, which in this case is an image displaying device. The image displaying device can be an image projector (as shown in Figure 1) or a display (e.g. DLP, LCD, OLED, plasma and the like). Examples of other controllable devices which can be utilized by the present invention are provided above. Figure 1 illustrates a configuration (layout) of system 10 which includes a free standing display surface, other configurations can include layouts in which the image display device projects an image on a floor surface or a window. System 10 further includes a computing platform 14 which can be a personal computer running Windows XP™, Linux™ or Mac OSX™, or a workstation such as a

Silicon Graphics™ work station or a Sun Microsystems™ workstation or any other computational device which can perform the functions described herein. Computing platform 14 receives thermal image information from thermal imaging device 12 and executes a software application for analyzing at least a portion of the thermal image to automatically identify and/or track a subject 18 within the environment. Such a software application can include a silhouetting algorithm, such as that described in, for example, Migdal and Grimson "Background Subtraction Using Markov Thresholds"

Computer Science and Artificial Intelligence Laboratory, MIT; or any other type of algorithm which can utilized to process a thermal image and identify a subject based on color or intensity differences between the subject and the background.

It will be appreciated that a thermal image is typically presented in color image in order to amplify contrast for a human viewer. Since most silhouetting algorithms can be employed with grayscale images, the thermal image is preferably captured or processed as a grayscale image. Transition from color to grey scale can be effected by setting the hue color channel (in HSB color model) to grayscale while maximizing the saturation and brightness values. Subject identification can be effected by system 10 by utilizing a background subtraction algorithm which can be improved using more complicated algorithms such as that described by Migdal and Grimson (Ibid). The simplest form of background subtraction includes subtraction of a static (or slowly changing) reference image (referred to herein as the "background image") which represents the static objects (wall, floor, field etc) present in the environment from a captured frame which includes the static image information and subjects (humans, animals, dynamic objects) present in the environment. The algorithm is utilized to process each image received from the camera (referred to herein as the "camera image") and compare the captured camera image to the background image. The output of the algorithm is a black and white image where black indicates the stored background image and white indicates a subject of interest in the environment, the composite image is referred to herein as the "silhouette image". For each pixel in the camera image the algorithm takes the pixel in the same location in the background image and subtracts between the two. It then finds the absolute value of the subtraction and this absolute value is checked against a set threshold, if the threshold is met then the pixel in the same location in the silhouette image is marked white, otherwise it is marked black. The following pseudo code defines such an algorithm: For each pixel in the camera image

If the absolute value of (camera image [current pixel] - background image[current pixel]) > threshold then

Silhouette image [current pixel] = white Otherwise

Silhouette image [current pixel] = black

The resulting silhouette image can be used directly as a height map image or processed further by algorithms such as blob finding, motion analysis (optical flow) or any other algorithm as needed by the application exactly as if is was a silhouette from a visible light camera.

Computing platform 14 is also capable of translating presence and/or motion of the subject or a portion thereof (e.g. hand, finger etc.) into input data. Examples of algorithms which are suitable for performing such translation include, but are not limited to an optical flow algorithm such as that described in Lucas et al. "An iterative image registration technique with an application to stereo vision" Proceedings of Imaging understanding workshop, pp 121-130, 1981; or Berthold et al. "Determining

Optical Flow" Artificial Intelligence, 17, pp. 185-203, 1981; a Blob Analysis algorithm such as that described by Senior et al. "Appearance Models for Occlusion Handling," in proceedings of Second International workshop on Performance Evaluation of Tracking and Surveillance systems in conjunction with CVPR¹Ol, December 2001; a Tracking algorithms such as that described by Comaniciu et al. "Real-time tracking of non-rigid objects using mean shift," IEEE International Conference on Pattern Recognition, vol. 2, pp. 142-149, 13-15 June, 2000 and a Trajectory Analysis algorithm such as that described by Johnson and Hogg, "Learning the distribution of object trajectories for event recognition," Image and Vision Computing, 14:609-615, 1996.

Input data can be stored by computing platform 14 and or translated into input commands for controlling controllable device 16.

In the case where controllable device 16 is an image display device, the input commands generated by computing platform 14 can be used to alter an image (static or dynamic) displayed by the image display device. The image can be altered in as far as image characteristics such as color, size, canvas shape or orientation, or it can be altered in as far as content, e.g. displaying a modified or different image in response to an input command.

For example, an image display device such as a projector can be utilized to provide a video game (e.g., a role playing game); in such cases, movement of subject 18 (or a portion thereof, e.g. hand) in an environment can be translated into commands for controlling a game object (e.g., a game character).

System 10 can also be utilized in interactive advertising, for example, system 10 can project a static banner with the logo of mineral water brand on the floor of a mall. As subjects walk over the projected banner, thermal imaging device 12 captures images of the projected banner (background image) and subjects and relays the captured information to computing platform 14. Computing platform 14 silhouettes and identifies the subjects walking over the banner and translates the movement or presence of the subject(s) into commands for modifying the projected background image in real time to display a water ripple video effect around each subject present over the banner.

It will be appreciated that in cases where system 10 is utilized for interactive media, the placement of the subject within the environment (with respect to the FOV of thermal imaging device 12 and controllable device 16) can be important, and thus in such cases an area of activity (where presence or motion is translated into input commands) is preferably defined.

To facilitate the above, system 10 of the present invention can employ a visible- light capturing device such as a CCD camera. The FOV of such a camera can be co- aligned with that of thermal imaging device 12, such that information relating to the area of activity within the environment can be captured and utilized to align thermal imaging device 12. This is particularly true in cases where controllable device 16 is an image projector, since in such cases, thermal imaging device 12 will not be able to efficiently identify the displayed image and thus the area of activity within the environment .

Although the use of a CDD camera is preferred, it will be appreciated that determination of the area of activity can also be effected by thermal imaging device 12 by, using, for example, thermal markings for defining the area of activity, or by marking an image display area with such markings.

System 10 represents an Example of an onsite installation. It will be appreciated that a networked system including a plurality of system 10 installations is also envisaged by the present invention.

Such a networked configuration can include a central server which can carry out part or all of the functions of computing platform 14. The central server can be networked (via LAN, WAN, WiFi, WiMax or a cellular network) to each specific site installation which includes a thermal imaging device 12, a local computing platform 14 and a controllable device 16.

Thus, the use of thermal imaging provides numerous benefits in subject detection and tracking rendering the present system highly suitable for the purposes described herein.

It will be appreciated that although the present invention was described in context of identification and tracking of living objects (e.g. humans), it can be used to identify and track non-living objects present in an environment as long as they are thermally distinguishable from the environment. For example, many electronic devices (e.g. laptops) are hotter than the environment and thus are identifiable via thermal imaging. Alternatively, infra red tags can be added to objects to make them thermally identifiable (see for example, Sakata et al. "Active IR-Tag User Location System for MR Collaborative Environment" at www.crestserver.naist.jp/crest/workshop/2004/pdf/crest-sakata.pdf).

It will be appreciated that thermal imaging provides additional benefits especially in cases where subject gender or age determination or discrimination is desired. Due to their unique thermal signature, men and women can be easily differentiated using thermal imaging and well known image processing algorithms (Nishino et al. "Gender determining method using thermography" International Conference on Image Processing, 2004; Oct. 2004 Volume: 5, Pages 2961- 2964). In addition, differences in size and proportions also enable differentiation between children and adults.

In cases where system 10 is utilized for providing interactive media, such as interactive advertising, determination of subject age or gender can be utilized to alter the content of a displayed advertisement according to the age or gender of the subject detected. For example, advertisement of a product, such as, a car, can be presented in one of several ways depending on the gender of the subject. Men can be presented with a video illustrating performance while women can be presented with a video illustrating safety and comfort options.

It is expected that during the life of this patent many relevant silhouetting algorithms will be developed and the scope of the term silhouetting is intended to include all such new technologies a priori.

It is appreciated that certain features of the invention, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination.

Although the invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims. All publications, patents and patent applications mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual publication, patent or patent application was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention.

Claims

WHAT IS CLAIMED IS:

1. An interactive system for translating presence and/or motion of a subject in an environment into input commands for altering a function of a device, the system comprising:

(a) a thermal imaging device for capturing a thermal image of the environment;

(b) a device having a controllable function; and

(c) a computing platform executing at least one software application being configured for:

(i) analyzing at least a portion of said thermal image to thereby automatically detect the presence and/or motion of the subject or a portion thereof within the environment; and (ii) translating the presence and/or motion of the subject within the environment into input commands for controlling a function of said device.

2. The system of claim 1, wherein said software application employs a silhouetting algorithm.

3. The system of claim 1, wherein said device is an image display device.

4. The system of claim 3, wherein an image displayed by said image display device is modifiable according to said input commands.

5. The system of claim 4, wherein said image is a video image and whereas a content of said video changes according to said input commands.

6. The system of claim 1, further comprising an image capture device capable of capturing a visible-light image of the environment.

7. The system of claim 3, wherein said image display device is an image projector.

8. The system of claim 5, wherein said image display device is a display.

9. A method of translating presence or motion of a subject in an environment into commands for controlling a function of a device, the method comprising:

(a) acquiring a thermal image of the environment;

(b) computationally analyzing at least a portion of said thermal image to thereby automatically identify and/or track the subject or a portion thereof within the environment;

(c) computationally translating presence and/or motion of the subject into input data; and

(d) using said input data as input commands for controlling the function of the device.

10. The method of claim 9, wherein step (b) is effected using a silhouetting algorithm.

11. The method of claim 10, wherein the device is an image display device and whereas said input commands are utilized to control an image displayed by said image display device.

12. The method of claim 11, wherein said image display device is an image projector.

13. The method of claim 11 , wherein said image display device is a display.