WO2002017235A2

WO2002017235A2 - Method and apparatus for tracking an object of interest in a digital image

Info

Publication number: WO2002017235A2
Application number: PCT/EP2001/009448
Authority: WO
Inventors: Daphna Weinshall; Damian M. Lyons
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2000-08-25
Filing date: 2001-08-16
Publication date: 2002-02-28
Also published as: CN1404682A; KR20020068330A; JP2004507189A; EP1314137A2; WO2002017235A3

Abstract

A method and apparatus are disclosed for tracking an object of interest in a digital image having at least one high resolution portion. In a 'shaped mirror embodiment,' a shaped rotating mirror magnifies at least one portion of the field of view so that a stationary camera obtains a higher resolution image corresponding to the magnified region. In a 'two camera embodiment,' two stationary cameras with varying resolution are focused on a flat rotating mirror. A first stationary camera having lower resolution produces the overall panoramic image, while the second stationary camera having higher resolution produces the higher resolution image with a smaller field of view. In the shaped mirror embodiment, a rotating mirror contains one magnifying region that magnifies a portion of the field of view, and additional regions that reduce the remaining portions of the field of view. In the illustrative embodiment, the rotating mirror is curved to produce regions of variable magnifying resolution. In the illustrative two-camera embodiment, a 360-degree panoramic image is obtained using a first stationary camera and a flat rotating mirror. A second camera having a higher resolution or higher zoom than the first camera is also utilized. A semi-reflective mirror transmits a predefined percentage of the reflected light to the first camera and reflects a predefined percentage of the reflected light to the second camera.

Description

Method and apparatus for tracking an object of interest in a digital image

Field of the Invention

The present invention relates to methods and apparatus for producing panoramic images, and more particularly, to a method and apparatus for tracking an object of interest in such panoramic images and zooming into desired portions of such panoramic images.

Background of the Invention

A number of techniques have been proposed or suggested for obtaining panoramic images from digital cameras. Generally, panoramic images have a field of view between 180 and 360 degrees. Most techniques for generating panoramic images utilize a number of overlapping images that are aligned using well-known image processing techniques to create a single, integrated image. For a general discussion of such image processing techniques, see, for example, P. Anandan and P.J. Burt, "Image Stabilization by Registration to a Reference Mosaic," ARPA94(I:425-434), BibRef 9400, or P. Anandan et al., "Real-time Scene Stabilization and Mosaic Construction," ARPA94(I:457-465), BibRef 9400 And: A4, Al, A3, A5 and A4: WACV94(54-62), each incorporated by reference herein.

A number of techniques have been proposed or suggested for capturing the plurality of overlapping images that are used to generate the panoramic image. For example, a number of systems employ multiple cameras to record the multiple images. The multiple images are then processed to create the desired panoramic image. Since the relationship between the various cameras is typically fixed, the relationship between the images is also known. The problem with a multiple camera arrangement, of course, is the additional expense relative to a single camera implementation. Thus, a number of systems obtain the plurality of images using a single camera. For example, International Patent Application Number WO 99/62252 discloses a video camera that rotates at a constant speed to capture video images of a panoramic scene of interest. The digital video file can then be processed to create the desired panoramic image. In another variation, Japanese Patent Application Number 11004373 obtains a 360-degree panoramic image using a fixed camera and a rotating mirror. The mirror is mounted above the camera with a predefined tilt, such that the field of view of the mirror captures the desired panoramic scene as the mirror is rotated about an optical axis of the camera. The individual overlapping images are then processed to create the desired panoramic image.

A number of techniques have been proposed or suggested for tracking objects in digital images, including panoramic images. In such tracking systems, direct tracking is typically performed using a low-resolution image due to computational issues, and the portion of the image of interest can be magnified using a higher resolution image. The tracking task maintains the target in the field of view of the camera so the operator can monitor the object of interest. In this manner, the operator can switch between a high resolution with a smaller field of view and a low resolution with a larger field of view, as desired. The higher resolution image is typically obtained using a moving part, such as an additional mirror or camera, whose motion is controlled during a zoom operation, to magnify the region of interest. For example, Mitsubishi Electric America provides an omnidirectional vision system utilizing two mirrors. See, for example, http://www.mitsubisM.com/mea/tecomn.html. In an implementation that utilizes an additional minor to magnify the area of interest, complex optical arrangements are typically required. Likewise, in an implementation that utilizes an additional camera to magnify the area of interest, the increased expense over a single camera implementation is prohibitive.

A need therefore exists for a method and apparatus that provides tracking in a panoramic image without controllable moving hardware for magnification. A further need exists for a method and apparatus that provides tracking in a panoramic image with simultaneous magnification of multiple portions of the field of view, without the mechanically controlled motion of additional moving parts.

Summary of the Invention Generally a method and apparatus are disclosed for tracking an object of interest in a digital image having at least one high resolution portion. In one embodiment, referred to herein as the "shaped mirror embodiment," a shaped rotating mirror magnifies at least one portion of the field of view so that a stationary camera obtains a higher resolution image corresponding to the magnified region. In a second embodiment, refeπed to herein as the "two camera embodiment," two stationary cameras with varying resolution are focused on a flat rotating mirror. A first stationary camera having lower resolution produces the overall panoramic image, while the second stationary camera having higher resolution produces the higher resolution image with a smaller field of view. In the shaped minor embodiment, a rotating minor contains one magnifying region that magnifies a portion of the field of view, and one or more additional region(s) that reduce the remaining portions of the field of view. In the illustrative embodiment, the rotating minor is curved to produce regions of variable magnifying resolution. The magnifying region that magnifies a portion of the field of view has a generally concave shape while reduction regions that reduce the remaining portions of the field of view have a convex shape.

In the illustrative two-camera embodiment, a 360-degree panoramic image can be obtained using a first stationary camera and a flat rotating minor. The flat rotating minor rotates about an optical axis of the camera. The minor is mounted above the camera with a predefined tilt, such that the field of view of the minor captures the desired panoramic scene as the minor rotates. A second camera having a higher resolution or higher zoom than the first camera is also utilized. A second, semi-reflective minor is positioned along the optical axis. The semi-reflective minor transmits a predefined percentage of the reflected light to the first camera and reflects a predefined percentage of the reflected light to the second camera. A more complete understanding of the present invention, as well as further features and advantages of the present invention, will be obtained by reference to the following detailed description and drawings.

Brief Description of the Drawings

FIG. 1 illustrates a panoramic image tracking system in accordance with one embodiment of the present invention;

FIGS. 2 A and 2B illustrate a front elevation view and a top elevation view, respectively, of one embodiment of the rotating minor of FIG. 1;

FIG. 3 illustrates a panoramic image tracking system in accordance with a two camera embodiment of the present invention; and FIG. 4 is a flow chart describing an exemplary panorama tracking process incorporating features of the present invention. Detailed Description

FIG. 1 illustrates a panoramic image tracking system 100 in accordance with one embodiment of the present invention. The present invention obtains a 360-degree panoramic image using a stationary camera 120 and a rotating mirror 200, discussed further below in conjunction with FIG. 2. In the illustrative embodiment described herein, the stationary camera 120 is suspended in a room, defined by a ceiling 110 and two walls 115 a, 115b. As shown in FIG. 1, the rotating minor 200 rotates about an optical axis 130 of the camera 120. The minor 200 is rotated using a passive, non-controllable motor that rotates about 360 degrees at a fixed speed. The minor 200 is mounted above the camera 120 with a predefined tilt, such that the field of view 140 of the minor 200 captures the desired panoramic scene as the minor 200 is rotated about the camera 120. In this manner, the panoramic image tracking system 100 processes a sequence of images that can be processed using well-known image processing techniques to create the desired panoramic image.

The present invention provides a panoramic image having at least one high- resolution portion. According to one feature of the present invention, tracking is performed in a panoramic image without controllable moving hardware. In one embodiment, refened to herein as the "shaped minor embodiment," a shaped rotating minor 200 magnifies at least one portion of the field of view so that the stationary camera 120 obtains a higher resolution image conesponding to the magnified region. In another embodiment, referred to herein as the "two camera embodiment," two stationary cameras 120 with varying resolution are focused on a flat rotating minor 200. A first stationary camera 120-1 having lower resolution produces the overall panoramic image, while the second stationary camera 120-2 having higher resolution produces the higher resolution image with a smaller field of view. .

Shaped Minor Embodiment FIG. 2A is a front elevation view of an illustrative rotating minor 200 in accordance with the present invention. As shown in FIG. 2A, the illustrative rotating minor 200 contains one magnifying region 220 that magnifies a portion of the field of view, and two regions 210a, 210b that reduce the remaining portions of the field of view. Thus, the illustrative rotating minor 200 shown in FIG. 2A rotates along the horizontal axis, and the magnifying region 220 produces a vertical high-resolution band in each image.

FIG. 2B is a top elevation view of the illustrative rotating minor 200 of FIG. 2A. As shown in FIG. 2B, the rotating mirror 200 is curved to produce regions of variable magnifying resolution. The magnifying region 220 that magnifies a portion of the field of view has a generally concave shape while reduction regions 210a, 210b that reduce the remaining portions of the field of view, have a convex shape. In other words, the magnifying region 220 is a high-resolution area that is obtained by magnifying the horizontal dimension and projecting a portion 240 of the field of view to an image strip wider than the field of view portion 240. Likewise, the reduction regions 210a, 210b are low resolution areas that are obtained by shrinking the horizontal dimension and projecting the corresponding portions 230a, 230b of the field of view to an image strip nanower than the field of view portions 230a, 230b. Generally, the horizontal resolution in the image of portions 230a, 230b will be f/a pixels per panoramic degree, whereas the horizontal resolution in the image of portion 240 will be fb pixels per panoramic degree, where f is a constant and both a and b are greater than one.

Thus, as the rotating minor 200 rotates, the stationary camera 120 captures the plurality of images that are used to generate the desired panoramic image. Each image captures the entire field of view 140 (shown in FIGS. 1 and 2B), comprised of a high- resolution portion 240 and two low resolution portions 230a, 230b. According to another feature of the present invention, the high resolution portion 240 is sub-sampled to achieve the same resolution as the two low resolution portions 230a, 230b. Specifically, the high resolution portion 240 is sub-sampled by a factor of 1/ab.

In this manner, the desired panoramic image is obtained as a low resolution image. Thus, moving objects appear and may be tracked in the single low-resolution image. For a detailed discussion of the tracking of objects of interest in a low-resolution image, see, for example, D. Gutchess et al., "Automatic Surveillance Using Omni-Directional and Active Cameras" Proc. Asian Conference on Computer Vision 2000 (2000), or C. Wren et al., "Pfinder: Real-Time Tracking of the Human Body", IEEE. Trans. PAMI, 19(7) (1997), each incorporated by reference herein. The rotation speed of the minor 200 and the horizontal width of the vertical band 240 are coordinated for a given implementation, such that a high resolution image is obtained for each portion of the overall field of view 140 for each rotation cycle of the minor 200. In this manner, the panoramic image is updated with each cycle of the minor rotation. A low resolution version of the entire panorama is maintained, as well as each of the adjacent high resolution vertical bands 240 conesponding to full panorama.

As discussed further below in conjunction with FIG. 4, given a request to magnify a portion of the field of view, the conesponding high resolution vertical strip 240 is retrieved, and super-imposed on the low resolution image of the entire panorama. It is noted that the present invention allows any resolution between the low-resolution image and the high-resolution image by sub-sampling the high-resolution image to the desired resolution. The field of view, however, for anything above the low resolution image is the smaller field of view of the high resolution image.

Two-Camera Embodiment FIG. 3 illustrates a panoramic image tracking system 300 in accordance with the two-camera embodiment of the present invention. The two-camera embodiment obtains a 360-degree panoramic image using a first stationary camera 320-1 and a flat rotating minor 325. The first stationary camera 320-1 is suspended in a room, defined by a ceiling 310 and two walls 315 a, 315b. As shown in FIG. 3, the flat rotating minor 325 rotates about an optical axis 330 of the camera 320-1. The minor 325 is rotated using a passive, non- controllable motor that rotates about 360 degrees at a fixed speed. The minor 325 is mounted above the camera 320-1 with a predefined tilt, such that the field of view 340 of the minor 325 captures the desired panoramic scene as the minor 325 is rotated about the camera 320- 1. In this manner, the panoramic image tracking system 300 processes a sequence of images that can be processed using well-known image processing techniques to create the desired panoramic image.

In the two-camera embodiment, a second camera 320-2 having a higher resolution or higher zoom than the first camera 320-1 is also utilized. As shown in FIG. 3, a second, semi-reflective minor 345, such as a half-silvered minor, is positioned along the optical axis 330. The semi-reflective mirror 345 transmits a predefined percentage of the reflected light from minor 325 to camera 320-1 and reflects a predefined percentage of the reflected light from minor 325 to camera 320-2. The first camera 320-1 captures a series of overlapping low-resolution images that can be integrated to obtain the panoramic image. In addition, the second camera 320-2 captures a series of higher resolution images that can be utilized to zoom into a desired area of interest.. In the illustrative embodiment, the higher resolution images captured by the second camera 320-2 are vertical bands, in a similar manner to the shaped minor embodiment discussed above.

Tracking Process The present invention can track an object of interest in the single low- resolution image of the panorama. FIG. 4 is a flow chart describing an exemplary panorama tracking process 400 incorporating features of the present invention. As shown in FIG. 4, the panorama tracking process 400 continuously captures a panoramic image with low resolution and strips of high resolution using the shaped minor or two camera embodiments discussed above during step 410. Once a request is received to magnify a portion of the field of view (for example, from an automated tracking module following a moving object), as detected during step 415, the panorama tracking process 400 retrieves the corresponding high-resolution vertical strip 240 during step 420. The retrieved high-resolution vertical strip 240 is then super-imposed during step 430 on the low-resolution image of the entire panorama. It is again noted that the present invention allows any resolution between the low-resolution image and the high-resolution image by sub-sampling the high-resolution image to the desired resolution. Program control then returns to step 410 to continue updating the panoramic image in real-time, until a new magnification region is selected. It is to be understood that the embodiments and variations shown and described herein are merely illustrative of the principles of this invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

CLAIMS:

1. A digital image system (100), comprising: a shaped rotating minor (200) having a field of view (140), wherein said field of view (140) includes a region of higher resolution (220); and a camera (120) focused on said rotating minor (200) and capturing a plurality of images of said field of view ( 140) .

2. The digital image system (100) of claim 1 , wherein said shaped rotating minor (200) includes a generally concave region for obtaining said region of higher resolution (220) of said field of view (140).

3. The digital image system (100) of claim 1, wherein said shaped rotating minor (200) includes at least one generally convex region (210) for obtaining a region of lower resolution of said field of view (140).

4. The digital image system (100) of claim 1 , wherein said shaped rotating minor

(200) rotates at a fixed speed.

5. The digital image system (100) of claim 1, wherein said shaped rotating minor (200) is rotated using a passive, non-controllable motor that rotates said minor about a fixed axis.

6. The digital image system (100) of claim 1, wherein said field of view (140) captures a panoramic image.

7. A digital image system (100), comprising: a rotating mirror (325) having a field of view (340); a first stationary camera (320-1) focused on said rotating minor (325) and capturing a plurality of images of said field of view (340); a second minor (345) in an optical path before said first stationary camera (320-1); and a second stationary camera (320-2) focused on said second minor (345) and capturing a plurality of images of said field of view (340), wherein said second camera (320- 2) has a higher resolution than said first camera (320-1).

8. The digital image system (100) of claim 7, further comprising a semi- reflective minor positioned in an optical path of said first and second stationary cameras (320-1,2) such that said semi-reflective minor transmits a predefined percentage of light reflected light from said rotating minor (325) to said first camera (320-1) and reflects a predefined percentage of said light reflected from said rotating minor (325) to said second camera (320-2).

9. The digital image system (100) of claim 7, wherein said field of view (340) captures a panoramic image.

10. The digital image system (100) of claim 7, wherein said rotating minor (325) rotates at a fixed speed.

11. The digital image system (100) of claim 7, wherein said field of view (140) captures a panoramic image.

12. The digital image system (100) of claim 7, wherein said second minor (345) is a semi-reflective minor.

13. The digital image system (100) of claim 7, wherein said second minor (345) is a half-silvered minor.

14. A method for obtaining a digital image, comprising the steps of: rotating a shaped minor (200) having a field of view (140), wherein said field of view (140) includes a region of higher resolution (220); and focusing a camera (120) on said rotating minor (200); and capturing a plurality of images of said field of view (140).

15. The method of claim 14, wherein said shaped minor (200) includes a generally concave region for obtaining said region of higher resolution (220) of said field of view (140).

16. The method of claim 14, wherein said shaped minor (200) includes at least one generally convex region (210) for obtaining a region of lower resolution of said field of view (140).

17. The method of claim 14, wherein said rotating step rotates said shaped minor (200) at a fixed speed.

18. The method of claim 14, wherein said rotating step rotates said shaped minor (200) using a passive, non-controllable motor that rotates said minor about a fixed axis.

19. The method of claim 14, wherein said field of view (140) captures a panoramic image.

20. The method of claim 19, further comprising the step of integrating said plurality of images to obtain said panoramic image.

21. A method for obtaining a digital image, comprising the steps of: rotating a minor (325) having a field of view (340); focusing a first stationary camera (320-1) on said rotating minor (325); capturing a plurality of images of said field of view (340) using said first stationary camera (320-1); positioning a second minor (345) in an optical path before said first stationary camera (320-1); and focusing a second stationary camera (320-2) on said second minor (345) and capturing a plurality of images of said field of view (340), wherein said second camera (320- 2) has a higher resolution than said first camera (320-1).

22. The method of claim 21, further comprising the step of positioning a semi- reflective minor (345) in an optical path of said first and second stationary cameras (320-1,2) such that said semi-reflective minor transmits a predefined percentage of light reflected light from said rotating minor (325) to said first camera (320-1) and reflects a predefined percentage of said light reflected from said rotating minor (325) to said second camera (320-2).

23. The method of claim 21 , wherein said plurality of images can be integrated to obtain a panoramic image.

24. The method of claim 21, wherein said rotating step rotates said minor (325) at a fixed speed.

25. The method of claim 21, wherein said second minor (345) is a semi-reflective minor.

26. The method of claim 21 , wherein said second minor (345) is a half-silvered minor.

27. An article of manufacture for obtaining a digital image, comprising: a computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising: a step to rotate a shaped minor (200) having a field of view (140), wherein said field of view (140) includes a region of higher resolution (220); and a step to focus a camera on said rotating minor (200); and a step to capture a plurality of images of said field of view (140).

28. An article of manufacture for obtaining a digital image, comprising: a computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising: a step to rotate a mirror (325) having a field of view (340); a step to focus a first stationary camera (320-1) on said rotating minor (325); a step to capture a plurality of images of said field of view (340) using said first stationary camera (320-1); a step to position a second minor (345) in an optical path before said first stationary camera

(320-1); and a step to focus a second stationary camera (320-2) on said second minor (345) and capture a plurality of images of said field of view (340), wherein said second camera (320-2) has a higher resolution than said first camera (320-1).