HK1106372B - Method, system and apparatus for forming a panoramic image - Google Patents

Description

Method, system and apparatus for forming panoramic image

Technical Field

The present invention relates to a digital still camera and a method for forming a panoramic image, and particularly, but not exclusively, to a digital still camera in which a panoramic image is automatically formed, and a method for automatically forming a panoramic image using such a camera.

Background

A common method of taking panoramic images with a digital still camera is for the user to take multiple overlapping images with the digital still camera, transfer them to a computer, process them on the computer, and stitch them together with a stitcher such as Apple's "QuickTime VR".

In the capture phase, the user typically needs to fix exposure, white balance, and focus to ensure color consistency. This can be simplified by having a "panoramic mode" which is commonly available in many digital still cameras. The user decides the orientation of the camera, whether to pan or tilt, and whether to pan the camera lens left or right. The relevant instructions are input to the camera. Next, the user operates the camera to take a plurality of overlapping snapshots (snapshots) with appropriate overlap. It is the responsibility of the user to ensure that there is sufficient overlap. This process requires the user to pause between each shot to perform a framing. Some cameras provide visual assistance for framing the next panoramic image by displaying a portion of the previous snapshot. While it removes most of the guesswork, the user will still need to perform a manual framing for each shot stop, which is cumbersome compared to taking a single snapshot.

Next, in the stitching stage, the user typically transfers all the images to the computer and manually selects a panoramic sequence from a mix of panoramic and static snapshots. The panoramic image sequence is brought into a stitching application to stitch one at a time. The user may also need to provide orientation and pan direction information to the stapler. The result of the stitching process is a panoramic image. While most stitchers are fully automated, the process of sorting and searching snapshots for panoramic image sequences for stitching adds unnecessary work. This is shown in fig. 3 and 4.

A "movie" mode using more advanced digital cameras has been proposed to capture many images and use these images to form a panoramic image. By using the "movie" mode, a large number of images are taken (typically 30 per second) and therefore there should always be sufficient overlap. Most users take several seconds to take a large panorama, so such cameras may capture more than 200 images for a panoramic image. This requires very powerful, fast processing and huge memory. Thus, only very powerful digital cameras can be used, which are very expensive. Furthermore, the image resolution in movie mode is typically much smaller than in still image mode.

Disclosure of Invention

According to a preferred form, there is provided a digital still camera for taking still images and having a panoramic mode for creating panoramic images, the camera comprising: a photographic imaging system for capturing a plurality of still photographic images used to form a panoramic image; a transfer module for receiving a plurality of still photographic images to be automatically stitched together to automatically form a panoramic image; a processor for processing a plurality of still photographic images to automatically form a panoramic image; and a memory for storing at least one of the panoramic image and the plurality of still photographic images.

Preferably, the processor comprises a stitching engine for automatically forming the panoramic image; and the transfer module includes a processor.

According to another form, there is provided a system for forming a panoramic image, the system comprising:

(a) a digital still camera for taking still images and having a panoramic mode for creating a panoramic image, the camera including a photographic imaging system for capturing a plurality of still photographic images used to form the panoramic image;

(b) a transfer module for receiving a plurality of still photographic images to be automatically stitched together to form a panoramic image, the transfer module comprising a processor;

(c) the processor for automatically processing a plurality of still photographic images to automatically form a panoramic image, the processor comprising a stitching engine for automatically forming the panoramic image; and

(d) a memory for storing at least one of a panoramic image and a plurality of still photographic images.

For both aspects, data of the plurality of still photographic images not used in the panoramic image may be deleted after the panoramic image is formed and/or stored. The first of the plurality of still photographic images may be used to set up color and light related processing for all subsequent images of the plurality of still photographic images. Each of the plurality of still photographic images may have an overlap region with a previous image of the plurality of still photographic images. The overlap region may be determined by at least one of the group consisting of a user, a panning speed, a digital camera processing capability, a digital camera processing speed, a digital photographic shutter speed. The overlap area may be in the range of 5% to 50%; and each of the plurality of subsequent images may be marked with a unique serial number.

In another aspect, there is provided a method for generating a panoramic image with a digital still camera including a photographic imaging system for capturing a plurality of still images to be used to form the panoramic image, the method comprising:

(a) the digital still camera captures a plurality of still photographic images while the digital still camera is set in a panoramic mode, a shutter switch is operated, and the digital camera is panned;

(b) saving each of the plurality of still photographic images in a memory;

(c) automatically processing the plurality of still photographic images in the transfer module to automatically form a panoramic image;

(d) at least one of the panoramic image and the plurality of still photographic images is stored in a memory.

After the panoramic image is formed and/or stored, the still photographic image may be deleted from memory. The transfer module may be in the digital camera or in a separate computer to which the plurality of still photographic images are downloaded. The download may be direct or may be through the use of a removable memory device.

The first of the plurality of still photographic images may be used to set up color and light related processing for all subsequent images of the plurality of still photographic images.

Each of the plurality of still photographic images may have an overlap region with a previous image of the plurality of still images; and the overlap region may be determined by at least one of the group consisting of a user, a panning speed, a digital camera processing capability, a digital camera processing speed, a digital photographic shutter speed. The overlap area may be in the range of 5% to 50%; and each of the plurality of subsequent images may be marked with a unique serial number.

The automatic processing may include determining the camera orientation and panning direction by: determining a plurality of suitable features in two adjacent images of the plurality of still photographic images, the adjacent images having overlapping regions; determining a degree of movement of the plurality of suitable features from a first one of the adjacent images to a second one of the adjacent images; and adding the movement along the X-axis and the movement along the Y-axis to two single values, respectively. If the absolute sum of the movements in the X-axis direction is less than the absolute sum of the movements in the Y-axis direction, the digital still camera is in the landscape orientation; and if the absolute sum of the movements in the X-axis direction is not less than the absolute sum of the movements in the Y-axis direction, the digital still camera is in the portrait orientation. For landscape orientation, if the sum in the X-axis direction is less than zero, the panning direction is to the left; and for portrait orientation, if the sum in the Y-axis direction is less than zero, the panning direction is to the left.

In a last preferred aspect, there is provided a method for determining an orientation and a panning direction of a digital still camera capturing a plurality of still photographic images to be used for forming a panoramic image, the method comprising:

(a) selecting a plurality of suitable features in two adjacent images of the plurality of still photographic images, the adjacent images having overlapping regions;

(b) determining a degree of movement of the plurality of suitable features from a first one of the adjacent images to a second one of the adjacent images; and

(c) the movement along the X-axis and the movement along the Y-axis are added to two single values, respectively.

If the absolute sum of the movements in the X-axis direction is greater than the absolute sum of the movements in the Y-axis direction, the digital still camera is in the landscape orientation; and if the absolute sum of the movements in the X-axis direction is not greater than the absolute sum of the movements in the Y-axis direction, the digital still camera is in the portrait orientation. For landscape orientation, if the sum in the X-axis direction is less than zero, the panning direction is to the left; and for portrait orientation, if the sum in the Y-axis direction is less than zero, the panning direction is to the left.

The present invention also provides a computer usable medium comprising computer program code configured to cause a processor to execute one or more functions for performing one or more of the methods described above.

Drawings

For an easy understanding and realization of the invention, a preferred embodiment thereof will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:

FIG. 1 is a perspective view of one form of a digital camera;

FIG. 2 is a block diagram of components of the camera of FIG. 1;

FIG. 3 is a flow diagram of a known prior art process for capturing images prior to creating a panoramic image;

FIG. 4 is a flow diagram of a known prior art process for creating panoramic images on a computer;

FIG. 5 is a flow chart of a preferred embodiment of the present invention during a capture phase; and

fig. 6 is a flow chart of the embodiment of fig. 5 during a download phase.

Detailed Description

Referring to fig. 1 and 2, a digital still camera 10 is shown. Although a simple form of a digital still camera is shown, the present invention is also applicable to all forms of digital still cameras, including a single-lens reflex camera, and a digital moving picture camera in a still camera mode.

The camera 10 has an imaging system, generally indicated at 12, that includes a lens 14, a viewfinder 16, a shutter 18, a built-in flash 20, a shutter switch 22, and other controls 24. An image capture device 25 (e.g., a charge coupled device) forms part of the imaging system 12. Within the camera 10 there is a processor 26 for processing image data received in a known manner, a memory 28 for storing each image as image data, and a controller 30 for controlling the data sent to be displayed on a display 32. The imaging system 12 is capable of taking and capturing photographic images of everyday scenes. The imaging system 12 may have a fixed or variable focal length, size scaling, and other functions provided in a digital still camera.

Starting with the capture phase shown in fig. 5, the user first selects the panoramic mode (50). In this mode, all color and ray related processing is fixed in the first snapshot, as in the conventional approach, to ensure color consistency (53). Further, while in the panoramic mode, subsequent snapshots are marked as panoramic sequences. The tag also includes a unique number, such as a serial number, that uniquely identifies the sequence and/or the individual snapshot (53).

The user then decides the camera orientation and panning direction (51) and releases the shutter switch (52). The user then pans (55) the camera to span the target scene in a single scan. The camera automatically takes snapshots repeatedly at predetermined time intervals (e.g., every 0.2, 0.5, 0.8, 1.0, 1.5, or 2.0 seconds) and/or with a defined overlap (56). The speed of taking the snapshot is determined by the camera and is such that only the required number of images are taken. This is to prevent that too many images are captured which would require very high processing power and very large memory, for example like the known camera using "film mode". Preferably, only predetermined time intervals are used. The rate at which snapshots are taken may be determined by one or more of pan speed, amount of overlap of images, photographic processing power, camera processing speed, and camera shutter speed. It may be preset. At the end of the scene, the user again releases the shutter release (58) to indicate the end of the sequence. The camera takes the final snapshot and the sequence ends (59). The camera increments the sequence number in preparation for the next sequence.

In this simplified process, the user does not need to worry about overlapping areas. The user captures the entire scene in a single continuous scanning motion and does not need to stop to take each snapshot.

The camera may automatically overlap the snapshots by a predetermined amount, for example in the range of 5% to 50%, such as (but not limited to) 5%, 10%, 15%, 20%, or 50%, to ensure a complete capture of the snapshots and avoid missing any part of the panorama. The amount of overlap may be predetermined, determined by the user, or determined by one or more of the following: lens pan speed, camera processing power, camera processing speed, and camera shutter speed.

Instead of a separate stitching application, the stitching engine 34 may reside in the transfer module 36 of the camera 10 or in the transfer module 13 of the computer 11. The transfer module 36 is responsible for transferring the data of the snapshot from the camera 10 to the computer 11 and/or a separate memory module (not shown). The transmission to the computer 11 may be through the use of a cable 15 in a known manner, or may use a memory module, such as a flash memory card or thumb disk. Preferably, it is able to detect image sequences of panoramic images from other still images by examining the image markers and passing the image sequences to a stitcher to form a panoramic image.

If in the computer 11, the transfer module 13 will have a processor 17 that includes a stitching engine 19. The memory 21 may be part of the processor 17 or may be separate.

As shown in fig. 6, during image transfer from the camera 10, the transfer module 36 or the transfer module 13 checks for image markers (60) and automatically transfers or copies each panoramic sequence in the correct order to the stitching engine 34 or 19 (61). If the camera does not provide orientation and panning direction (62), the stitching engine 34 or 19 derives it from the image sequence by tracking selected image features (63) (64).

The panning direction and camera orientation estimation (64) assumes that the panning direction and camera orientation are fixed for a given sequence.

Given two images i1, i2, where i1 and i2 overlap:

1. select the appropriate features in i1 and track them to i 2. Feature selection and tracking may be performed using known methods, such as those described in the article "Good Features to Track" by Jianbo Shi and Carlo Tamasi, IEEE Computer Vision and Pattern recognition, 1994, the contents of which are incorporated herein by reference. It is then calculated how much of the selected feature in i1 has moved into i 2. These movements are added to two single values, SumDeltaX and SumDeltaY, where SumDeltaX is the sum of the movements on the X-axis and SumDeltaY is the sum of the movements on the Y-axis.

2. For the results in step 1 above to be reliable, the number of tracked features should be greater than 5. In the case where the number of tracked features is less than 5, the tracking accuracy can be increased by making the following assumption and performing a specific region search instead of a general search over the entire image.

Assume that 1: pan and pan lens to the right: good features in the right half of i1 are selected and tracked in the left half of i 2.

Assume 2: pan and shake the lens left: good features in the left half of i1 are selected and tracked in the right half of i 2.

Assume that 3: rotate 90 ° to the right (vertical) and pan the lens to the right: good features in the upper half of i1 were selected and then tracked in the lower half of i 2.

Assume 4: rotate 90 ° to the right (vertical) and pan the lens to the left: good features in the lower half of i1 are selected and then tracked in the upper half of i 2.

Until the results of step 1 are acceptable, each hypothesis is tested until the number of features tracked is greater than 5. If the number of tracked features is less than or equal to 5 for all hypotheses, then i1 and i2 may not overlap at all, in which case the next pair, i.e., i2 and image #3, is evaluated.

3. If Absolute (SumDeltaX) is greater than Absolute (SumDeltaY), then the camera orientation is landscape. If Absolute (SumDeltaX) is not greater than Absolute (SumDeltaY), then the camera orientation is portrait.

For landscape orientation, the panning direction is to the left if SumDeltaX <0, and to the right otherwise.

For portrait orientation, the panning direction is to the left if SumDeltaY <0, and to the right otherwise.

The stitching engine 34 or 19 processes the panoramic image sequence into a single panoramic image in a usual and known manner (65). The transfer module then delivers a single panoramic image (66), rather than a sequence of images. In this way, the user only sees the final stitched panoramic image. The camera 10 or computer 11 does not reveal the panoramic image sequence to the user, and the individual snapshots, or any data sets of the individual snapshots remaining after the stitching operation, may be deleted from memory 26 or 21 during or after formation of the panoramic image (68). The individual snapshots, or any data sets of the individual snapshots remaining after the stitching operation, may be deleted after completing the stitching of the snapshot into the panoramic image. For the first snapshot, it will be deleted along with the second snapshot when the second snapshot is stitched to it.

The panoramic image is saved in the memory 28 or 21 (67). The process then ends (69). Thus, the individual snapshots are either stored in memory and transferred to the panoramic image; or stored in memory and copied to the panoramic image. The memory buffer may be used to store the intermediate warped image during the stitching process. The camera orientation and panning direction are referenced to determine whether the camera is in portrait or landscape mode, and whether the camera is panning left or right.

The panoramic image may be displayed on a display of the camera 10 or the computer 11. Displayed may be a finished panoramic image; or the first and last image that have been or will be used to create the panoramic image.

By being in the transfer module 13 or the transfer module 36, the step of stitching to form the panoramic image occurs when the sequence of images is transferred. In this way, all stitching of the individual images takes place automatically within the transfer module 13 or 36. Stitching occurs automatically on the image sequence when the stitching engine 19 is in the transfer module 13 of the computer 11, so that the panoramic image is downloaded to the computer 11. The panoramic image will not be stored in the camera 10 but will be stored in the computer 11 and the sequence of images will be stored in the camera 10.

The present invention also provides a computer usable medium comprising computer program code configured to cause a processor to execute one or more functions for performing one or more of the methods described above.

Whilst there has been described in the foregoing description preferred embodiments of the present invention, it will be understood by those skilled in the technology concerned that many variations or modifications in details of design or construction may be made without departing from the present invention.

Claims (25)

1. A digital still camera for taking still images and having a panoramic mode for creating panoramic images, the camera comprising:

(i) a photographic imaging system for capturing a plurality of still photographic images used to form the panoramic image;

(ii) a transfer module to receive the plurality of still photographic images to be automatically stitched together to automatically form the panoramic image;

(iii) a processor for processing the plurality of still photographic images to automatically form the panoramic image; and

(iv) a memory for storing at least one of the panoramic image and the plurality of still photographic images,

wherein data of the plurality of still photographic images not used to form the panoramic image is deleted after the panoramic image is stored in the camera; and is

Wherein the processing of the plurality of still photographic images comprises determining a camera orientation and a panning direction by:

(a) selecting a plurality of suitable features in two adjacent images of the plurality of still photographic images, the adjacent images having overlapping regions;

(b) determining a degree of movement of the plurality of suitable features from a first one of the adjacent images to a second one of the adjacent images; and

(c) the movement along the X-axis and the movement along the Y-axis are respectively added to two single values,

wherein the digital still camera is in a landscape orientation if the absolute sum of the movements in the X-axis direction is greater than the absolute sum of the movements in the Y-axis direction; and if the absolute sum of the movements in the X-axis direction is not greater than the absolute sum of the movements in the Y-axis direction, the digital still camera is in a portrait orientation, and wherein

For the landscape orientation, if the sum in the X-axis direction is less than zero, the panning direction is to the left, and

for the portrait orientation, the panning direction is to the left if the sum in the Y-axis direction is less than zero.

2. The digital camera of claim 1, wherein the processor includes a stitching engine for automatically forming the panoramic image.

3. The digital camera of claim 1, wherein the transfer module comprises the processor.

4. The digital camera of claim 1, wherein the transfer module comprises the processor, the processor comprising a stitching engine for automatically forming the panoramic image.

5. The digital camera as in claim 1, wherein a first one of the plurality of still photographic images is used to set color and light related processing for all subsequent images of the plurality of images.

6. The digital camera as in claim 5, wherein each of the plurality of still photographic images has an overlap region with a previous image of the plurality of still images.

7. The digital camera of claim 6, wherein the overlap region is determined by at least one of the group consisting of a user, a pan speed, a digital camera processing capability, a digital camera processing speed, a digital photographic shutter speed.

8. The digital camera as in claim 5, wherein each of the plurality of subsequent images is marked with a unique serial number.

9. A system for forming a panoramic image, the system comprising:

(i) a digital still camera for taking still images and having a panoramic mode for creating a panoramic image, the camera including a photographic imaging system for capturing a plurality of still photographic images for forming the panoramic image;

(ii) a transfer module for receiving the plurality of still photographic images to be automatically stitched together to form the panoramic image, the transfer module comprising a processor;

(iii) the processor for automatically processing the plurality of still photographic images to automatically form the panoramic image, the processor comprising a stitching engine for automatically forming the panoramic image; and

(iv) a memory for storing at least one of the panoramic image and the plurality of still photographic images,

wherein data of the plurality of still photographic images not used to form the panoramic image is deleted after the panoramic image is stored; and is

Wherein the processing of the plurality of still photographic images comprises determining a camera orientation and a panning direction by:

(a) selecting a plurality of suitable features in two adjacent images of the plurality of still photographic images, the adjacent images having overlapping regions;

(b) determining a degree of movement of the plurality of suitable features from a first one of the adjacent images to a second one of the adjacent images; and

(c) the movement along the X-axis and the movement along the Y-axis are respectively added to two single values,

wherein the digital still camera is in a landscape orientation if the absolute sum of the movements in the X-axis direction is greater than the absolute sum of the movements in the Y-axis direction; and if the absolute sum of the movements in the X-axis direction is not greater than the absolute sum of the movements in the Y-axis direction, the digital still camera is in a portrait orientation, and wherein

For the landscape orientation, if the sum in the X-axis direction is less than zero, the panning direction is to the left, and

for the portrait orientation, the panning direction is to the left if the sum in the Y-axis direction is less than zero.

10. The system of claim 9, wherein the transfer module is in a position selected from the group consisting of: the digital still camera, and a computer to which the plurality of still photographic images are transferred from the digital still camera.

11. The system of claim 9, wherein a first one of the plurality of still photographic images is used to set color and light related processing for all subsequent images of the plurality of images.

12. The system of claim 9, wherein each of the plurality of still photographic images has an overlap region with a previous image of the plurality of still images.

13. The system of claim 12, wherein the overlap region is determined by at least one of the group consisting of a user, a pan speed, a digital camera processing capability, a digital camera processing speed, a digital photographic shutter speed.

14. The system of claim 11, wherein each of the plurality of subsequent images is marked with a unique serial number.

15. A method for generating a panoramic image with a digital still camera including a photographic imaging system for capturing a plurality of still images to be used to form the panoramic image, the method comprising:

(i) capturing the plurality of still photographic images by the digital camera while the digital still camera is set in a panoramic mode, a shutter switch is operated, and the digital camera is panned;

(ii) saving each of the plurality of still photographic images in a memory;

(iii) automatically processing the plurality of still photographic images in a transfer module to automatically form the panoramic image;

(iv) storing at least one of the panoramic image and the plurality of still photographic images in the memory,

wherein the plurality of still photographic images that have been used to form the panoramic image are deleted from the memory after the panoramic image is formed and saved; and is

Wherein the processing of the plurality of still photographic images comprises determining a camera orientation and a panning direction by:

(a) selecting a plurality of suitable features in two adjacent images of the plurality of still photographic images, the adjacent images having overlapping regions;

(b) determining a degree of movement of the plurality of suitable features from a first one of the adjacent images to a second one of the adjacent images; and

(c) the movement along the X-axis and the movement along the Y-axis are respectively added to two single values,

wherein the digital still camera is in a landscape orientation if the absolute sum of the movements in the X-axis direction is greater than the absolute sum of the movements in the Y-axis direction; and if the absolute sum of the movements in the X-axis direction is not greater than the absolute sum of the movements in the Y-axis direction, the digital still camera is in a portrait orientation, and wherein

For the landscape orientation, if the sum in the X-axis direction is less than zero, the panning direction is to the left; and is

For the portrait orientation, the panning direction is to the left if the sum in the Y-axis direction is less than zero.

16. The method of claim 15, wherein the processing step (iii) occurs in a transfer module of the digital still camera.

17. The method of claim 15, wherein the processing (iii) occurs in a transfer module of a computer to which the plurality of images are transferred from the digital still camera.

18. The method of claim 15, wherein a first one of the plurality of still images is used to set color and light related processing for all subsequent images of the plurality of images.

19. The method of claim 15, wherein each of the plurality of still images has an overlapping region with a previous image of the plurality of still images.

20. The method of claim 19, wherein the overlap region is determined by at least one of the group consisting of user, pan speed, digital camera processing power, digital camera processing speed, digital photographic shutter speed.

21. The method of claim 16, wherein the transfer module comprises a stitching engine for stitching the plurality of still images into the panoramic image.

22. The method of claim 17, wherein the transfer module comprises a stitching engine for stitching the plurality of still images into the panoramic image.

23. The method of claim 18, wherein each of the plurality of subsequent images is marked with a unique serial number.

24. A method for determining an orientation and a panning direction of a digital still camera that captures a plurality of still photographic images to be used to form a panoramic image, the method comprising:

(a) selecting a plurality of suitable features in two adjacent images of the plurality of still photographic images, the adjacent images having overlapping regions;

(b) determining a degree of movement of the plurality of suitable features from a first one of the adjacent images to a second one of the adjacent images; and

(c) the movement along the X-axis and the movement along the Y-axis are respectively added to two single values,

wherein the digital still camera is in a landscape orientation if the absolute sum of the movements in the X-axis direction is greater than the absolute sum of the movements in the Y-axis direction; and if the absolute sum of the movements in the X-axis direction is not greater than the absolute sum of the movements in the Y-axis direction, the digital still camera is in a portrait orientation, and wherein

For the landscape orientation, if the sum in the X-axis direction is less than zero, the panning direction is to the left; and is

For the portrait orientation, the panning direction is to the left if the sum in the Y-axis direction is less than zero.

25. The method of claim 24, wherein the method is performed in a stitching engine in a delivery module at a location selected from the group consisting of: the digital camera, and a computer to which the plurality of still photographic images are transferred from the digital still camera.