JPH04329484A - Image synthesizer - Google Patents

Abstract

PURPOSE:To accurately synthesize plural images, to obtain flexibility for setting the reference position of the image and to obtain the continuous image from the limited input images. CONSTITUTION:A picture element position correspondence detecting means 3 is provided to detect the correspondent picture element positions between the plural images, an image reference position setting means 4 is provided to set the reference coordinate position of the image, transformation parameter calculating means 5 and 6 are provided to calculate a transformation parameter expressing image transformation by using the image corresponding positions and the reference position, image transforming means 7 and 8 are provided to execute coordinate transformation by using the transformation parameter for the plural inputted images, plural image segmenting means 9 and 10 are provided to segment plural coordinate-transformed images according to the picture element corresponding position information, and an image synthesizing means 11 is provided to synthesize and output the plural segmented images.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus that integrates and displays a plurality of images.

0002

[Prior Art] Conventionally, technologies in this field include:
For example, there is one shown in FIG. FIG. 7 shows a configuration diagram of this conventional image synthesis apparatus. In Figure 7, 3
71 is a pixel position correspondence detecting means that detects the corresponding pixel position between both input images, and 71 is an affine unit that coordinates transforms one of the images based on the result of the correspondence of the images detected by the pixel position correspondence detecting means 3. It is a means of conversion. Reference numeral 72 denotes an image synthesizing means for synthesizing both the input image and the affine-transformed image. Reference numeral 73 denotes an image cutting means for cutting out a part of the combined image. The operation of the conventional image synthesizing apparatus configured as described above will be described below.

In FIG. 7, pixel position correspondence detection means 3 detects a portion where a common object exists from a plurality of input images by a method such as pattern matching, and detects the parallel movement of input image 1 and input image 2. Find correspondence relationships such as quantities or rotation angles around an axis perpendicular to the screen. Using this correspondence, the affine transformation means 71 converts the input image 1 into
By applying coordinate transformation to the input image 1, the input image 1 is regarded as an extension of the input image 2, and the affine transformed image and the input image 2 are combined by the image synthesis means 72. It is intended to synthesize the images and output the entire synthesized image or a portion thereof through the image cutting means 73.

0004

[Problems to be Solved by the Invention] However, in the configuration shown in FIG. 7, since the system uses one of the plurality of images as a reference and performs affine transformation on the other images, it is difficult to use a wide-angle lens, for example. In the case of images taken of objects with three-dimensional depth, there is a problem that the images after transformation do not necessarily overlap well with similar transformations such as parallel translation or rotation around an axis perpendicular to the screen. Was.

[0005] Furthermore, when composing images, one of the plurality of images is used as a reference and another image is transformed and the compositing is performed, making it impossible to obtain an image photographed at an intermediate viewing angle. For example, if there are two input images (left image and right image) taken with the camera swung left and right,
If you can only combine images that have been converted from the right image using the left image as a reference, or conversely, combine images that have been converted from the left image using the right image as a reference, and the camera is facing forward. There was a problem in that it was not possible to obtain a composite image equivalent to

[0006]Furthermore, when a plurality of images of an object are taken so that parts of the object overlap, the image reference position has not been set according to the relative positional relationship of the images.
It was not possible to perform image synthesis that suppressed geometric deformation from the input original image.

[0007] Furthermore, when compositing images, images are not cut out and synthesized according to the correspondence of pixel positions of multiple images, so if an object moves between multiple images, the cropping The problem was that the parts were discontinuous.

[0008]Also, even if you try to obtain a moving image effect by continuously changing the area from which a part of the image is cut out, the entire screen will be synthesized based on a specific image among multiple images. , the equivalent camera angle always became a constant image, and even if the cropped area was changed continuously, the user could only get the feeling that the image was simply displaying part of a picture drawn on a wall.

The present invention solves the various problems of the conventional image synthesis apparatus described above, and is capable of accurately synthesizing images even when images of objects with three-dimensional depth are photographed using a wide-angle lens. It is an object of the present invention to provide an image compositing device that can effectively combine a plurality of images by realizing flexible setting of reference positions of images.

0010

[Means for Solving the Problems] In order to achieve the above object, the present invention inputs a plurality of images in which a part of the image includes a common object with a part of other images, and pixel position correspondence detection means for detecting corresponding pixel positions; image reference position setting means for setting the reference coordinate position of the image; and an output of the image position correspondence determining means and an output of the image reference position setting means. a transformation parameter calculation means for calculating a transformation parameter representing transformation of a two-dimensional position of an image; an image transformation means for coordinate-transforming a plurality of input images using the transformation parameters; The apparatus includes a plurality of image cutting means for cutting out images based on the output of the pixel position correspondence detection means, and an image synthesizing means for synthesizing and outputting the plurality of cut out images.

[0011]

[Operation] According to the present invention, with the above-described configuration, corresponding pixel positions of a plurality of images are detected, and the input images are transformed and synthesized again using a point set at an arbitrary position in the images as a reference. Therefore, in the case of an image taken using a wide-angle lens of an object with three-dimensional depth, for example, when the angle between two straight lines changes as the camera angle changes, It is now possible to transform images according to the camera angle, and it is possible to synthesize images that could not be processed simply by similar transformations such as parallel translation or rotation.

Furthermore, instead of fixing the image reference at the center of the image, the image conversion parameters are calculated by moving it to an arbitrary position and using this as the reference, and using these parameters, the coordinates of the input image are transformed and the images are synthesized. Therefore, it is possible to virtually synthesize images taken at camera angles that are not actually taken.

[0013] Furthermore, when a plurality of images are taken such that a part of the object overlaps, the relative positional relationship of the images is detected, and if there is an image corresponding to the center of the plurality of images, By using this as the image reference position, it is possible to perform image synthesis in which geometric deformation from the input original image is suppressed as much as possible. Also, if multiple images are determined to be multiple images taken with a certain area as the center based on their relative positions, even if the images were taken at an angle such that the central area is in the center of the image. Even when no image exists, by setting the virtual center position as the image reference position, it is possible to perform image synthesis in which geometric deformation from the input original image is suppressed as much as possible.

[0014] Furthermore, when compositing images, images may be cut out and composited according to the correspondence of pixel positions of the plurality of images, so that if an object moves between the plurality of images, This also suppresses the occurrence of discontinuities in the portions where cutting and combining are performed.

[0015] Furthermore, when trying to obtain a moving image effect by continuously cutting out a part of an image synthesized by continuously changing the image reference position, the images to be synthesized are changed according to the image reference position. Since the images are converted and synthesized each time, it is possible to obtain a moving image with the same sense of realism as a moving image actually shot by changing the camera angle.

[0016]

Embodiments Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 shows a configuration diagram of an image synthesizing apparatus according to a first embodiment of the present invention. In Figure 1,
1 and 2 are image input terminals, 3 is a pixel position correspondence detection means for detecting the correspondence of pixel positions between input images, 4 is an image reference position setting means for setting the reference position of the image, and 5 and 6 are image conversion means. A conversion matrix calculation means for calculating matrix coefficients, 7 to 8 are image conversion means, 9 to 10 are image cutting means, 11 is an image composition means, and 12 is a composite image output.

The operation of the image synthesizing apparatus according to the first embodiment of the present invention constructed as described above will be explained below.

The pixel position correspondence detection means 3 compares the two input images, detects an area where a common object exists, and detects the pixel position relationship between one image and the other image.

The image reference position setting means 4 sets the reference position of the image, which is equivalent to determining where to set the origin of the coordinate system in the combined image. The simplest example of setting this image reference position is, for example, to set one of the input images as the reference image, and to set the image origin, that is, the viewpoint, as the center point of this reference image.

Since the angles of the cameras in the two images are generally different, each point of the same object in the two images cannot be moved simply by parallel translation or rotation about an axis perpendicular to the screen. It is not always possible to match the pixel positions sufficiently. However, if the movement of the object or camera is expressed using a rotation matrix and a translation vector, and the image is transformed based on this, the positions of corresponding pixels can be matched. In particular, when the movement of the camera can be considered as rotation with the lens center fixed, or when the translational component of the camera can be ignored, such as when the target image is a distant view, this image transformation is performed using the coefficients Rij of the transformation matrix.
It can be easily expressed using (i, j = 1, 2, 3). When point (x, y) on the image is converted to point (X, Y),
When the focal length of the lens is f, X=f(R11x+R21y+R31f)/(R
13x+R23y+R33f) Y=f(R12x
+R22y+R32f)/(R13x+R23y+R3
3f).

The conversion matrix calculation means 5 and 6 calculate conversion matrix coefficients Rij using the corresponding information of pixel positions between the two images detected by the pixel position correspondence detection means 3 and the coordinate positions of the image reference position setting means 4. (i,j=1,2
, 3) can be detected. It is known that calculation of this conversion matrix is theoretically possible if there are two sets of corresponding pixels, and if there are three or more sets, there are several methods to reduce numerical calculation errors. Are known.

The transformation matrix coefficients Rij (i, j=1,
2, 3) and the conversion formula, the image conversion means 7 and 8
convert the input images respectively, and the image cutting means 9 and 1
Supply to 0. Based on the output of the pixel position correspondence detection 3, the image cutting means 9 and 10 cut out an image using a line segment connecting pixels that correspond to each other between the two images, and combine these images by an image synthesizing means 11 to connect them. It is possible to obtain a composite image in which the occurrence of discontinuity in parts is suppressed.

FIG. 2 is a diagram illustrating the processing of the image cutting means 9 and 10. As shown in FIG. 2(a), a case will be described in which two images taken by changing the camera as shown in A or B, that is, images in FIG. 2(b) and FIG. 2(c) are combined. Since the two images are best matched by a straight line connecting points P1 and P2, it is easiest to combine the two images using a cutting line as shown in Figure 2(d). Figure 2
If a cutout line such as that shown in FIG. 2(d) is used, an unnatural composite image as shown in FIG. 2(e) will result. Therefore, Figure 2 (
f) or (h), if you focus on pixels that correspond to each other between images and select a line connecting those pixels as the image cutting line, you can create a more natural image as shown in Fig. 2 (g) or (h). A composite image can be obtained. Note that, as in the example of a person and a car in Figure 2, depending on the selection of extraction lines, the temporal relationship between moving objects at distant positions in the image may not always be correctly reproduced. Compared to the case where the object is cut discontinuously, a much more natural composite image can be obtained.

FIG. 3 is an explanatory diagram showing an example of the operation of setting the image reference position in this embodiment. Similar to Figure 2, Figure 3(a)
are the respective images at two camera angles. When the image reference position is set to point PA in image A, the composite image near the image reference position will be as shown in FIG. 3(b), and the portion of image B will be greatly deformed in the composite image. Similarly, when the image reference position is set to point PB in image B, the composite image near the image reference position will be as shown in FIG. 3(d), and the portion of image A will be greatly deformed in the composite image. By setting the image reference position between PA and PB as shown in Figure 3(c), it is possible to obtain a composite image that suppresses extreme image deformation, or to obtain a composite image at a camera angle that is not actually taken. can be obtained.

In this embodiment, the pixel reference position is generated by continuously changing it as shown in FIGS. 4A to 4D, and images are synthesized accordingly, so that the output image consists of multiple images each time. Since it is obtained by combining images that have been subjected to different transformations, it is possible to obtain a moving image with a sense of realism equivalent to that obtained when the camera is actually photographed by changing its angle.

[0027] Also, if an arbitrary point in an image as shown in FIGS. 4A to 4D and an arbitrary point in another image among the plurality of images are
Any point on a line passing through the points may be selected as the image reference position.

As described above, according to this embodiment, the corresponding pixel positions of a plurality of images are detected, the change in the image due to the change in camera angle is calculated as a motion parameter, and the input image is adjusted using this motion parameter. Since images are converted and synthesized using a wide-angle lens, for example, when an image of an object with three-dimensional depth is taken, the angle between two straight lines changes due to changes in the camera angle. Even when the target is an image, it becomes possible to transform the image according to the camera angle, and it becomes possible to synthesize images that could not be processed simply by similar transformations such as parallel translation or rotation. In addition, when performing image cropping, the image cropping line is a line that connects pixels that correspond to each other between multiple images, so even if an object moves between two images, the image A more natural composite image can be obtained by suppressing discontinuities at connected parts. Furthermore, since the image reference position is flexibly set, it is possible to obtain a composite image with reduced image deformation, or to obtain an image equivalent to an image taken at a camera angle that was not actually taken.

FIG. 5 is a part of a block diagram of an image synthesizing apparatus according to a second embodiment of the present invention. FIG. 6 is a diagram showing the relationship between the input image and the subject in this embodiment. In Figure 5, 2
1 to 25 are image input terminals; 26 is image arrangement detection means; 2
7 is a reference image selection means, 28 is a reference image, 29 to 32 are pixel position correspondence detection means, 33 to 36 are conversion parameter calculation means, 37 to 40 are image conversion means, and 41 to 44 are converted images. The contents are as follows. Regarding the second embodiment of the present invention configured as described above, the first
The parts that are different from those in the embodiment will be explained. Input images 21 to 25 include a subject in a part of the image that is common to a part of other images, and among the images inputted by the image arrangement detection means 26, the image arranged in the center is selected. To detect. As shown in FIG. 6(b), since the input image 5 of the image input terminal 25 is placed in the center, the reference image selection means 2
7 selects the input image 5 as the reference image 28. The pixel position correspondence detection means 29 to 32 detect corresponding pixel positions using the reference image 28 and each image. The conversion parameter calculation means 33 to 36 obtain parameters for converting each input image to the reference image 5 from the reference image 28 and the pixel position correspondence information. Then, each image 1 converted by the image conversion means 37 to 40 using each conversion parameter.
, 2, 3, and 4 are input image 5 in the same manner as in FIG.
It can be used as an extended image of the surrounding area. Since the image placed at the center of each input image is used as a reference, the deformation of the image that occurs in the composite image is not so large, and it is possible to automatically perform natural image composition.

As described above, according to this embodiment, since the reference image is selected by determining the relative positions of a plurality of images, it is possible to automatically synthesize images without significantly deforming each of the input images. It can be carried out.

Note that the present invention is not limited to the above-mentioned embodiments, and various modifications can be made based on the spirit of the present invention.

[0032]

Effects of the Invention As explained above, the present invention calculates the change in the image due to the change in camera angle as a motion parameter, transforms the input image using this motion parameter, and synthesizes the image. It is possible to combine even images with

Furthermore, since the present invention flexibly sets the image reference position, it is possible to synthesize images with little deformation or to obtain images at angles that are not actually photographed.

Furthermore, since the present invention selects a reference image by determining the relative positions of a plurality of images, it is possible to automatically synthesize images without significantly deforming each of the input images.

Furthermore, according to the present invention, when images are cut out and synthesized, corresponding pixels in a plurality of images are connected and cut out, thereby realizing image synthesis without discontinuity.

Furthermore, the present invention generates images by continuously changing the pixel reference position, and synthesizes images accordingly, thereby creating a sense of realism that is equivalent to when the camera is actually photographed by changing its angle. You can obtain moving images with

[Brief explanation of the drawing]

FIG. 1 is a configuration diagram of an image synthesis apparatus in a first embodiment of the present invention.

FIG. 2 is a diagram illustrating the processing of image cutting means 9 and 10 in the first embodiment of the present invention.

FIG. 3 is an explanatory diagram showing an example of operation for setting an image reference position in the first embodiment of the present invention.

FIG. 4 is a diagram showing pixel reference positions in the first embodiment of the present invention.

FIG. 5 is a part of a configuration diagram of an image synthesis device according to a second embodiment of the present invention.

FIG. 6 is a diagram showing the relationship between an input image and a subject in a second embodiment of the present invention.

FIG. 7 is a configuration diagram of a conventional image synthesis device.

[Explanation of symbols]

1, 2 Input image 3 Pixel position correspondence detection means 4
Image reference position setting means 5, 6
Conversion parameter calculation means 7, 8 Image conversion means 9, 10 Image cutting means 11 Image composition means 12 Image composition output 21-25 Input image 29-32 Pixel position correspondence detection means 33-36 Conversion parameter calculation means 37-40
Image conversion means 41 to 44 Converted image

Claims (5)

[Claims] 1. Image input means for inputting a plurality of images in which a part of the image includes a common object with a part of another image, and a pixel position correspondence detection means for detecting corresponding pixel positions between the plurality of images. , an image reference position setting means for setting a reference coordinate position of the image, and a conversion parameter representing the transformation of the two-dimensional position of the image using the output of the image position correspondence determining means and the output of the image reference position setting means. a conversion calculation means for calculating the coordinates of the plurality of input images using the conversion parameters; and a plurality of image conversion means for converting the coordinates of the plurality of input images using the conversion parameters; An image synthesizing device comprising an image cutting means and an image synthesizing means for synthesizing and outputting the plurality of cut out images. 2. The image reference position setting means is configured to set a reference position on a line passing through two points, an arbitrary point in one of the plurality of images and an arbitrary point in another image among the plurality of images. 2. The image synthesis apparatus according to claim 1, wherein a point is selected as an image reference position. 3. The image synthesizing apparatus according to claim 1, wherein the image cutting means uses a line connecting pixels that correspond to each other among the plurality of images as an image cutting line. 4. The image reference position setting means continuously generates image reference position information using any point in the plurality of images as a starting point and an end point, thereby obtaining a moving image. The image synthesis device according to claim 1, characterized in that: 5. Image input means for inputting a plurality of images in which a portion of the image includes a common object with a portion of another image; and image position detection means for detecting relative positions between the plurality of input images. , reference image selection means for selecting one of the plurality of input images as a reference image based on the detection result of the image position detection means; pixel position correspondence detection means for detecting corresponding pixel positions between each pixel; and conversion calculation for calculating conversion parameters representing transformation of image positions using the output of the image position correspondence determination means and the reference image information. means, an image conversion means for coordinate-transforming the plurality of input images using the conversion parameters, and a plurality of image cutting means for cutting out each of the plurality of coordinate-transformed images according to an output of the pixel position correspondence detection means. , an image synthesizing means for synthesizing and outputting the plurality of cut out images.