JP3241243B2 - Panorama image composition system and panorama image composition method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a panoramic image synthesizing system and a panoramic image synthesizing method for obtaining a single image by synthesizing a plurality of images in which some of the images overlap.

[0002]

2. Description of the Related Art Due to a demand for taking a wide image into one image, a method called panoramic image synthesis for synthesizing a plurality of images in which some of the images overlap has been generally used. Are known.

[0003] In electronic cameras, low resolution (small number of images) is specified as a disadvantage compared to silver halide cameras and scanners. For images taken with this electronic camera, panoramic image synthesis is important not only for taking a wide image, but also as a means for taking a high-resolution image. More specifically, one paper document, magazine, or the like is divided into a plurality of pieces to be photographed, and high-resolution data equivalent to that of a scanner is obtained. Especially effective.

FIGS. 16A, 16B and 16C show an example of conventional panoramic image synthesis. In FIGS. 7A and 7B, an overlapping portion is detected from the two images 201 and 202, and the images are synthesized so that they overlap each other to obtain a panoramic image 203.

[0005]

However, the above-mentioned panoramic image processing has the following problems.

If the image obtained by combining a plurality of images is not rectangular as in the case of the panoramic image 203 in FIG. 16B, the resulting image is made rectangular and has a data format that can be handled by a computer. As shown in FIG. 16C, a rectangle 204 containing the panoramic image 203 is set, and an area (dummy area) 205 in the rectangle 204 where no image information exists is filled with an appropriate color or pattern. Was. For this reason, the image formed by the panoramic image synthesis is not a good image including the dummy area.

The present invention has been made in consideration of the above-described conventional problems, and has as its object to provide a panoramic image synthesizing system and a panoramic image synthesizing method capable of obtaining a good image without a dummy area.

[0008]

In order to achieve the above object, a panoramic image synthesizing system according to claim 1 synthesizes a plurality of images partially overlapping as regions to form one panoramic image. In the panoramic image synthesizing system to be created, the image data included in the rectangular area from within the image after synthesizing the plurality of images is converted into a plurality of rectangular patterns.
And select any of them to select
The selection result is the panoramic image .

A panoramic image synthesizing system according to claim 2 is
In the panoramic image synthesizing system according to claim 1, the rectangular area is determined in advance by a shape of an image obtained by synthesizing the plurality of images.

In a third aspect of the present invention, a panoramic image synthesizing system comprises:
3. The panoramic image synthesizing system according to claim 1 , wherein the plurality of rectangular patterns are extracted.
Includes rectangles with different rotation angles.
is there.

[0011] panoramic image synthesis method according to claim 4, in the panoramic image synthesis method partially to create a panoramic image by synthesizing the plurality of images are overlapped as a region, combining the plurality of images Rectangle from within later image
Image data included in the area
And select any of them to select
Selection result is the panoramic image.
You.

[0012]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of a panoramic image synthesizing system according to an embodiment of the present invention, and FIG. 2 is a block diagram of a personal computer system which is a platform on which the panoramic image synthesizing system of the present invention is implemented. It is an external view which shows a structure. This embodiment shows a case where a plurality of images taken by an electronic camera are combined on a personal computer to create one panoramic image.

First, in the personal computer system shown in FIG.
Is a display for displaying data, 3 is a mouse, which is a typical pointing device, 4 is a mouse button, 5
Is a keyboard. Further, an electronic camera 7 is connected to the computer system main body 1 by a general-purpose interface 6. The general-purpose interface 6 is a general-purpose interface capable of high-speed image transfer, such as a bidirectional parallel interface or a SCSI interface.

Next, the configuration of the panoramic image synthesizing system of the present embodiment will be described with reference to FIG.

In FIG. 1, reference numeral 11 denotes hardware, 12
Is an operating system (OS) that runs on the hardware 11, and 13 is application software that runs on the OS 12. Note that hardware 1
Of the blocks constituting the OS 1 and the OS 12, blocks which are naturally included as constituent elements but are not directly required for describing the embodiment of the present invention are not shown. Examples of such a block (not shown) include a CPU and a memory in hardware, and a memory management system in OS.

Reference numeral 14 denotes a hard disk for physically storing files and data, and reference numeral 15 denotes a file system constituting an OS, which has a function of enabling application software to input and output files without being aware of hardware. is there. Reference numeral 16 denotes a disk IO interface for the file system 15 to read and write the hard disk 14. Reference numeral 17 denotes a drawing management system included in the OS, which has a function of performing drawing without being conscious of application software.

Reference numeral 18 denotes a video interface for the drawing management system 17 to draw on the display 2. Reference numeral 19 denotes an input device management system constituting the OS, which has a function of receiving a user's input without being conscious of application software. Reference numeral 20 denotes a keyboard interface for the input device management system 19 to receive input from the keyboard 5, and reference numeral 21 denotes a mouse interface for enabling the input device management system 19 to receive input from the mouse 3.

Further, the electronic camera 7 is connected to a bidirectional interface or a SCSI interface 22, and can exchange image data and the like through the input device management system 19. Reference numeral 23 denotes an image data management system, and reference numeral 24 denotes a data management unit for managing image data by a file name, attribute information, or a keyword input by a user. Reference numeral 25 denotes a data display unit for searching and displaying managed image data by using attribute information or a keyword input by a user.

The panorama image creation system 26 includes a panorama synthesis means 27 for obtaining an overlap position between the images and synthesizing the images, and an image extraction for extracting the synthesized image, which is a feature of the present invention, in an appropriate rectangle. Means 28.

As described above, the present system combines a plurality of images taken by an electronic camera to create one panoramic image. This system performs panoramic image synthesis when copying (transferring) image data from the electronic camera 7 to a personal computer.

In the electronic camera 7, a photographing date and time, a photographing mode, and the like are recorded as attribute information together with the photographed image. When shooting an image for a panoramic image, the shooting mode of the electronic camera 7 is set to “panoramic image shooting mode”.

Images captured during the panoramic image capturing mode include, in the attribute information, a “panoramic image capturing mode” as the capturing mode and a series of images captured to form one panoramic image. Are set to the same panorama ID.

FIG. 3 is a diagram showing an operation for copying image data in the camera to the computer.

The camera 7 is connected to a computer via the general-purpose interface 6 and an image data management system 23
Start The image data management system 23 displays data in the camera in a window 31 named as a camera catalog. Reference numeral 32 denotes a reduced image (thumbnail image) of the image data, and reference numeral 33 denotes a photographing date and time as attribute information of the image. Numeral 34 indicates a part of the user's image database existing in the hard disk of the personal computer, which is read as a user catalog in this system.

The user selects an image from the camera catalog 31 (35 is a frame indicating that the image is selected), and performs a Drag & Drop operation with the mouse 3 on the user catalog 34 to copy. At this time, whether the data is a copy (data remains in the camera) or a movement (data in the camera is erased) can be switched to either according to a user's specification.

During the copying operation, if the image has been shot in the panoramic shooting mode, a panoramic image synthesizing process is performed. The above processing will be described with reference to the flowchart of FIG.

In FIG. 4, first, in step S1, image data corresponding to the thumbnail dropped in the user catalog 34 and attribute information associated therewith are acquired. In step S2, the photographing mode in the attribute information is checked to check whether the image is photographed in the panoramic photographing mode. If the image is not a panoramic image, it is registered in the user catalog 34 as normal image data in step S6. If it is a panoramic image, image data including the same panoramic ID in the attribute information is transferred from the camera in step S3. In step S4, using the obtained images,
The panorama synthesis processing described later is performed. Then, step S
In step 5, the final panoramic image is obtained by converting the panoramic image obtained by the image extraction processing into appropriate rectangular image data.

FIG. 5 is a flowchart of the panorama synthesizing process in step S4.

First, in step S11, the images to be synthesized are examined to detect the overlap position (corresponding point) between the images. Next, in step S12, the aforementioned step S
From the corresponding points detected in step 11, parameters for deformation of the image when used for synthesis are calculated. On the basis of this, in step S13, a plurality of images are synthesized to obtain one panoramic image.

FIG. 6 is a flowchart of the algorithm of the corresponding point extraction processing, and FIG. 7 is a diagram showing a template image and a matching range in the corresponding point extraction processing. In addition,
Here, two examples of left and right images 41 and 42 are shown.
When the number of images is larger than two, the combination of the two images may be repeated several times, so that the processing is basically the same.

In the present system, as a rule for taking a panoramic image, a minimum of 10% and a maximum of 50% are overlapped, and a deviation in a direction perpendicular to the minimum is 5% or less. As a result, the range 43 in which the template is set is 90% vertically,
Set to a range of 10% horizontally. Also, the search range is
The range of the possibility that the corresponding point exists.
It is set in a range 44 of 00% and 50% in width.

From the area 43 in the image, a point whose edge is stronger than a predetermined value is searched, and a rectangle of n pixels vertically and horizontally is cut out as a template image as a center of the point. The difference between the template image 45 and the search range 46 is calculated for each pixel. The position where the sum becomes minimum is obtained by shifting the search range by one pixel. If the minimum value of the results of searching all over the search range is equal to or less than a predetermined value,
The points (x, y) and (x ', y') are held as a pair of corresponding points.

The above is an outline of the processing, which will be described in detail again with reference to the flowchart of FIG.

First, in step S21, an edge-extracted image is created, and in step S22, a point whose edge is stronger than a predetermined value is searched from a range 43 in which a template is set in the edge-extracted image. If there is the point, an image is cut out from the point in a rectangle of ± n pixels vertically and horizontally to form a template image 45 (step S23), and a search range 46 in the right image 42 is set from the position of the point (step S24). ).

Then, the image 46 in the search range and the template image 45 are superimposed, and the absolute value of the difference between the pixel values is obtained for each pixel, and the sum is obtained (step S2).
5). It is checked whether the sum of the differences is the minimum value up to that time (step S26). If so, in step S27, the coordinates of the point in the search range and the minimum value are held. The above is repeated for the entire search range, and the point that matches the most (has the smallest difference) is found.

In step S28, it is checked whether the entire search range has been performed. In step S29, whether the minimum value obtained as a result is a sufficiently small value (certain corresponding point) is compared with a predetermined value L. To judge. If the value is smaller than the predetermined value L, the coordinates (x, y) of the point obtained by cutting the template image into the corresponding point list, the coordinates (x ′, y ′) of the point at which the minimum value is obtained, and the minimum value Is registered (step S30).

The above process is performed for the entire template setting range (step S31). When the process is completed, the average value is obtained from all the minimum values in the list of corresponding points, and the average value is stored as the matching level value (step S32). This completes the corresponding point extraction processing.

Next, a description will be given of a composite parameter calculation process in step S12 performed after the corresponding point extraction process in step S11.

When two images are used (in the case of combining two or more images, the combination of two images is repeated, so it may be considered as two images first), and the deviation is caused by translation and rotation in the x and y directions. And the difference between the magnifications. Therefore, the corresponding points (x, y) and (x ', y') can be expressed as follows.

[0041]

(Equation 1) Here, θ indicates a rotation angle, Δx and Δy indicate translation, and m indicates a magnification. Therefore, this coordinate transformation can be represented by obtaining the parameters A, B, C, and D. In the corresponding point extraction processing, a plurality of pairs of corresponding points (x, y) and (x ′, y ′) were obtained. The parameters A, B, C, and D are obtained using the least square method.

[0042]

(Equation 2) Under the condition

[0043]

(Equation 3) Parameters A, B, C, and D that satisfy the following are obtained. here,

[0044]

(Equation 4) Then, the parameters A, B, C, and D can be expressed as follows.

[0045]

(Equation 5) The parameters A1, B, C, and D are calculated by obtaining the parameters p1 to p8 and substituting them into the above equation.

Next, the image synthesizing process in step S13 will be described. Since the parameters A, B, C, and D were determined,

[0047]

(Equation 6) To obtain a composite image. FIG. 8 illustrates this image synthesis processing. Images are left and right images 51 and 52
In the case of, a size twice as large as the left image 51 is secured as the composite image 53. First, the left image 51 is copied as it is (51 '). Next, for the remaining area (x, y) 54 of the composite image, a corresponding area (x ′, y ′) 55 is obtained from the above equation. Then, the pixel at (x ′, y ′) of the right image 52 is copied to (x, y) (5
2 '). This is performed for all the remaining areas of the composite image.

FIG. 9 is a flowchart showing the image synthesizing process.

In step S41, an area twice as large as the first image (the left image in FIG. 8) is secured as a composite image area.
In step S42, the first image is simply copied to the composite image area. In step S43, for the remaining area (x, y) of the composite image, a corresponding area (x ′, y ′) is obtained from the above equation. In step S44,
It is checked whether (x ′, y ′) is movable in the second image (the right image in FIG. 8), and if there is, the pixel of (x ′, y ′) is changed to (x, y) in step S45. ).

The above processing is repeated for all the remaining areas of the composite image, and the composite processing ends. Thus, a panoramic image can be obtained.

Next, the image extraction processing of step S5 shown in FIG. 4 will be described. This image extraction processing is performed by the image extraction means 2
8 is performed. In the present system, a plurality of extraction methods are shown to the user, and an extraction image is determined by selecting an arbitrary method from among them. FIGS. 10A to 10D
FIG. 4 is a diagram showing a method of extracting a rectangular area from a panoramic image formed by combining two images.

When two images are photographed by an electronic camera or the like to create a panoramic image, when a tripod or the like is not used, the images tend to shift or tilt. FIG.
The panoramic images shown in (a) to (d) are two images 6
1 and 62 are combined at an angle θ and shifted as shown in FIG. The image extracting means 28 of the present embodiment is provided with several extracting methods, and it is possible to automatically extract a rectangle according to each of the original images of the panoramic image or to extract the image by taking the middle of the inclination. And

First, an extraction method 1 shown in FIG. 9A is based on a horizontal image, and in this example, a rectangle 63 is extracted. The extraction method 2 shown in FIG.
This is based on the other image tilted for synthesis, and in this example, a rectangle 64 is extracted. An extraction method 3 shown in FIG. 9C extracts a rectangle inclined by half the inclination between two images. In this example, a rectangle 65 inclined by θ / 2 is extracted. Note that the extracted images 64 and 65 of the extraction method 2 and the extraction method 3 are rotated by −θ and −θ / 2, respectively, and become image data without inclination. An extraction method 4 shown in FIG. 3D extracts a rectangle including both images, and in this example, the rectangle 66 corresponds to this.

By the way, various patterns are conceivable as to how two images overlap. Various methods for extracting a rectangular area in a panoramic image are conceivable. In the present system, a rectangular area extraction method is determined in advance for all overlapping methods. FIG. 11 is a diagram showing an example of how two images overlap and a rectangle to be extracted.
The shaded portion in each panoramic image 71 is an extraction area 72. In this system, these data are held as an extraction pattern table.
FIG. 12 shows a part of the extraction pattern table.

A row 81 in FIG. 12 corresponds to the case of the panoramic image shown in FIG. This will be described with reference to FIG.

FIG. 13 is an explanatory diagram of the extraction pattern table. The position of the rectangle is represented by the upper left coordinate and the rotation angle θ of each rectangle. As shown in FIG. 13, a condition 91 is obtained as a condition for making the two rectangles overlap like a panoramic image 71. The coordinates of each vertex of the extraction area 72 are obtained as shown by 92. Extraction pattern table 80
Stores these data. further,
If each side of the region after extraction is not horizontal to the x-axis or y-axis, it is necessary to rotate the extracted image, and thus includes an item describing the angle at that time.

The values obtained in this way are stored for every overlapping manner. An extraction pattern table exists for each extraction method.

Next, the process of extracting an image using the above-described extraction pattern table will be described with reference to the flowchart of FIG.

First, in step S51, a pattern corresponding to a panoramic image to be processed is checked by checking the item of "condition" in the table corresponding to the extraction method in the above-mentioned extraction pattern table.
Next, in step S52, the data of the corresponding extraction area is extracted from the table, and the image data of the corresponding portion is extracted from the panoramic image. In the subsequent step S53, the "rotation" item of the table is checked, and the extracted image data is rotated as necessary.

As described above, it is possible to obtain extracted image data when various overlapping methods and various extracting methods are used.

Next, FIG. 15 shows an operation screen for selecting a rectangle to be extracted from a panoramic image.

Reference numeral 100 in FIG. 15 denotes a screen displayed when panoramic image data is dragged and dropped from the camera catalog 31 in the user catalog 34 in FIG. In FIG. 15, reference numeral 101 denotes a panorama image display area where a panorama-combined image 102 is displayed.
Abstract image samples 103 to 106 are panoramic images 10
2, the results obtained when the rectangular areas are extracted by the various extraction methods described above are displayed as reduced images, and correspond to extraction methods 1 to 4, respectively.

When the user clicks an arbitrary one of the displayed extracted image samples with the mouse 3, a rectangular area corresponding to the extraction method is displayed on the panoramic image 102 as an extracted rectangle 107. In addition, the image is displayed surrounded by a frame like the selected extracted image sample 108. Until a desired extraction method is found, a rectangular area to be extracted by repeatedly specifying an extracted image sample can be confirmed in the panoramic image display area 101. If a desired extraction method is found, by specifying an OK button 109, an image of the specified method is registered in the system. Note that by previously specifying the extraction method in the system, the step of selecting the extracted image described above can be omitted.

In the above description, a case has been described in which two images for creating a panoramic image are connected in the horizontal direction and are synthesized. However, the present invention is not limited to this. The present invention can also be applied to panoramic images that can be used in various ways and panoramic images that lead to various methods. Further, a method of extracting a rectangular image from a panoramic image is not limited to the method described in the above embodiment, and various methods can be applied.

[0065]

As described above in detail, according to the panoramic image synthesizing system according to the first aspect, a panoramic image is created by synthesizing a plurality of images partially overlapping as regions. In the image synthesizing system, the image data included in the rectangular area is extracted from the synthesized image of the plurality of images by a plurality of rectangular patterns,
Select any one of them and copy the selection result to the
Since it is a Norama image, the image included in the rectangular area
Select the data extraction method according to the content of the image
It is possible to easily and accurately obtain a good image having no dummy area.

According to the panoramic image synthesizing system of the second aspect, in the panoramic image synthesizing system of the first aspect, the rectangular area is determined in advance according to the shape of the image after the plurality of images are synthesized. Therefore, it is possible to easily and accurately obtain a good image having no dummy area.

A panoramic image synthesizing system according to claim 3 is
3. The panoramic image synthesizing system according to claim 1 , wherein the plurality of rectangular patterns are extracted.
As we included those with different angles of rotation of the rectangle,
Even if the subject of the created panorama image is tilted,
It is possible to select an image that does not matter the degree and inclination
Become.

[0068] panoramic image synthesis method according to claim 4, in the panoramic image synthesis method partially to create a panoramic image by synthesizing the plurality of images are overlapped as a region, combining the plurality of images Rectangle from within later image
Image data included in the area
And select any of them to select
The selection result is the panorama image, so the rectangle
The extraction method of the image data included in the area
It can be selected according to the content, and a good image without a dummy area can be obtained easily and accurately .

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a panoramic image synthesis system according to an embodiment of the present invention.

FIG. 2 is an external view showing the configuration of a personal computer system which is a platform on which the panoramic image synthesizing system of the present invention is implemented.

FIG. 3 is a diagram illustrating an operation of copying image data in a camera into a computer.

FIG. 4 is a flowchart illustrating processing performed when image data in a camera is copied into a computer.

FIG. 5 is a flowchart of a panorama synthesis process.

FIG. 6 is a flowchart of an algorithm of a corresponding point extraction process.

FIG. 7 is a diagram showing a template image and a matching range in a corresponding point extraction process.

FIG. 8 is a diagram showing an image synthesis process.

FIG. 9 is a flowchart illustrating an image combining process.

FIG. 10 is a diagram illustrating a method of extracting a rectangular area from a panoramic image.

FIG. 11 is a diagram illustrating an example of how two images overlap and a rectangle to be extracted;

FIG. 12 is a diagram illustrating an example of an extraction pattern table.

FIG. 13 is an explanatory diagram of an extraction pattern table.

FIG. 14 is a diagram illustrating an image extraction process.

FIG. 15 is a diagram illustrating an operation of selecting a rectangular area to be extracted from a panoramic image.

FIG. 16 is a diagram illustrating an example of conventional panoramic image synthesis.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Computer system main body 2 Display 3 Mouse 4 Mouse button 5 Keyboard 6 General-purpose interface 7 Electronic camera 11 Hardware 12 OS 13 Application software 14 Hard disk 15 File system 16 Disk IO interface 17 Drawing management system 18 Video interface 19 Input device management system 20 Keyboard Interface 21 Mouse Interface 22 Bidirectional Interface / SCSI Interface 23 Image Data Management System 24 Data Management Means 25 Data Display Means 26 Panorama Image Creation System 27 Panorama Synthesis Means 28 Image Extraction Means

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Kotaro Yano 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc. (72) Inventor Tatsuji Katayama 3- 30-2 Shimomaruko, Ota-ku, Tokyo (56) References JP-A-7-212642 (JP, A) JP-A-5-260430 (JP, A) JP-A-5-260264 (JP, A) JP-A-4-32375 (JP) (A) JP-A-2-178646 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 3/00 300-400 G06T 1/00 280-320 H04N 1/387 H04N 5 / 262 H04N 7/18 A61B 3/14 A61B 6/00 350-360 Patent file (PATOLIS) JICST file (JOIS)

Claims (4)

(57) [Claims] 1. A panoramic image synthesizing system for creating a single panoramic image by synthesizing a plurality of images, some of which overlap as an area, in a panoramic image from the synthesized image of the plurality of images Extract image data included in the area using multiple rectangular patterns
And select any one of them, and
Panoramic image synthesis system is characterized in that a serial panoramic image. 2. The panoramic image synthesizing system according to claim 1, wherein the rectangular area is determined in advance according to a shape of an image obtained by synthesizing the plurality of images. 3. The method according to claim 2, wherein the plurality of rectangular patterns are rectangular patterns to be extracted.
A contractor characterized by including shapes with different rotation angles
The panoramic image synthesizing system according to claim 1 or claim 2.
M 4. A panoramic image synthesizing method for synthesizing a plurality of images partially overlapping as an area to create a single panoramic image, wherein a rectangular image is formed from within the synthesized image of the plurality of images. In the area
Extract contained image data by multiple rectangular patterns
And select any of them, and then select the result
A panoramic image synthesizing method characterized by using a norama image .