JP2019046213A - Method and program for generating an image obtained by interpolating a plurality of images - Google Patents

Abstract

To provide a method and program for creating a bisected image.SOLUTION: The method allows a computer to execute a first estimation procedure on an area a included in an image A and an area c included in an image C and corresponding to the area a in such a way that a display image B at time tB between time tA and time tC is generated from the image A and the image C at the time tC later than the time tA for displaying the image A, wherein a correspondence relationship includes rotation of an area about an axis substantially perpendicular to the image. The method includes: a central coordinate acquisition step of acquiring a central coordinate Oa of rotation of the area a and a central coordinate Oc of rotation of the area c; a rotation recognition step of recognizing a direction and an angle of rotation; and an area forming step of forming, through interpolation of a shape of the area a and a shape of the area c, a shape of an area b which is included in the image B and which corresponds to the area a based on the time tA, the time tB, the time tC, the center coordinate Oa, the center coordinate Oc, and the direction and angle of rotation.SELECTED DRAWING: None

Description

  The present invention relates to a method and program for generating an image obtained by interpolating a plurality of images.

  It is generally performed to create a two-dimensional animation by creating a plurality of two-dimensional images and sequentially displaying each of the plurality of two-dimensional images in chronological order. Two-dimensional animation is an expression technique that composes a moving image by reproducing a still image created by handwriting or the like in association with each frame on the time axis.

  In the old days, cartoons (two-dimensional animation) such as televisions and movies are performed by creating multiple handwritten images, shooting multiple images sequentially on movie films, and projecting movie films. It has Each of the plurality of images is an image in which the motion of the character or the like is changed little by little, so by displaying the plurality of images continuously, it is possible to generate a moving image of the character with motion. Also, as a more primitive technique, there has been a Parapara cartoon or the like that expresses a moving character by flipping a plurality of images drawn on a plurality of sheets of paper with a finger.

  A general, two-dimensional animation is often created in the following steps.

Process (1) Creation of original image: An image included in a key frame which is a necessary frame scattered at a predetermined timing in a series of moving image images is called an original image. The designer mainly creates the original.

-Process (2) creation of a split image: An image which is located between a key frame and the next key frame on the time axis and which smoothly interpolates the preceding and following original images (this image is called a "split image") Create
Process (3) coloring: coloring of a plurality of closed regions present in the original image and the split image.
Process (4) shooting: A combination of the original image and the background, and a combination of the split image and the background are performed to create each frame.

  About the original picture of the above-mentioned process (1), it is basic that a designer creates. The original image is created by a designer by handwriting or using computer graphics software or the like for the characteristic movement part of the moving character. Usually, the designer does not create every frame of the animation, but draws the original for the key frame. The original picture is drawn with several frames apart.

  The above-mentioned process (2) is creation of a middle division image. A person other than the designer often creates a split image between one original image and the original image positioned next in time. The split image is an image used for a blank frame located between adjacent originals on the time axis. The purpose of creating a split image is to fill a blank frame by creating an image obtained by interpolating key frames including adjacent originals on a time axis in order to create a smooth moving image.

  Therefore, the split image is drawn so that the entire moving image is smooth. The split image is often created by handwriting, but since the number of images to be created is larger than that of the original image, more resources are required. Also, although it has been attempted to automatically create split images, it is difficult to automatically create split images in creating natural moving images.

  The above step (3) is coloring. Since both the original image and the split image are often drawn as line drawings, an operation (coloring) is performed to color the respective closed regions formed by the line drawings with appropriate colors. Digitized processing is also widespread for coloring.

  The above step (4) is photographing. In the old days, each frame was photographed on a silver halide film. In the photographing process (4), digitized processing is widespread, and digital moving images are generated.

  In the above steps (2) and (3), there are many parts that require manual work, and the ratio at which the operator performs complicated work is high. This is because the images of a plurality of objects present in the original image overlap or are deformed. That is, the corresponding closed regions in the two original images are deformed, or the corresponding intersection points (such as the intersection points where the boundaries of the closed regions intersect) are moved, which makes automatic matching difficult. ing.

  The following techniques exist as techniques for finding corresponding intersection points present in two images.

  Control points for indicating the corresponding positional relationship between the two bitmap images are specified for each of the two key frame bitmap images, and they are determined on a straight line connecting the corresponding control points. There exists a technique which produces | generates an interpolation frame by changing a control point to an internal division point (for example, patent document 1).

In addition, in order to efficiently perform matching processing, there is a technique in which the matching unit obtains a contour point of the second shape data that matches the contour point of the first shape data using dynamic programming (for example, patent document 2).
The following techniques exist as techniques for finding corresponding closed regions present in two images.

  For example, there is a technique of tracking and storing the closed area such as the head and the hand as a chain and tracking the characteristics of the closed area (see, for example, Non-Patent Document 1).

  In addition, there is a technique of performing contour matching to associate similar image closed regions (see, for example, Non-Patent Document 2).

Japanese Patent Application Laid-Open No. 11-7539 JP, 2015-115047, A

P. Garcia Trigo, H. Johan, T. Imagire, and T. Nishita. Interactive Region Matching for 2D Animation Coloring Based on Feature ’s Variation. IEICE ransactions (E92-D), No. 6, pp. 1289-1295 (2009). J. S. Madeira, A. Stork, and M. H. Gross. An approach to computer-supported cartooning. The Visual Computer, Vol. 12, No. 1, pp. 1-17 (1996).

  As described above, in the split image creation step (2), a lot of resources are required to create the split image. Also, in the coloring step (3), there is a coloring step involving manual work. In this coloring process, it is necessary to apply the same coloring to the corresponding closed regions for each of the original image and the split image. Therefore, in the case of coloring, it is necessary to specify a corresponding closed area present in each frame of a plurality of original images and a plurality of middle-cut images. Usually, the determination of the corresponding closed area requires a visual task by the operator.

  The embodiments disclosed below aim at enabling the creation of two-dimensional animation to be performed more smoothly by reducing the work of the operator, particularly in the process of creating split images and the coloring process. I assume.

  In one embodiment, a time tB between the time tA and the time tC from the known image A and the known image C displayed at the time tC after the time tA when the image A is displayed To generate an image B to be displayed on the first estimation in a computer with respect to an area a included in the image A and an area c included in the image C and having a correspondence with the area a A method of performing a procedure, wherein the first estimation procedure includes rotation of a region about an axis in a direction substantially perpendicular to an image, and the central coordinate Oa of the rotation of the region a A central coordinate acquisition step of acquiring the central coordinate Oc of the rotation of the area c; a rotation recognition step of recognizing the direction of the rotation and the angle of the rotation; the time tA, the time tB, and the time tC. And the center coordinates Oa and the center coordinates An area included in the image B by interpolating the shape of the area a and the shape of the area c based on c, the direction of rotation, and the angle of rotation, and an area corresponding to the area a And b. Forming a shape of the region b.

  By creating an image obtained by interpolating a plurality of images, it is possible to create animation more smoothly by reducing the human work load in the animation creation process.

It is a figure showing an example which constitutes a two-dimensional animation by a plurality of frames containing an original picture and a middle division picture. It is a figure which shows the example of feature extraction of each pixel which comprises an original image. It is a figure which shows the example of the table which manages each closed area which comprises an original drawing. It is a figure which shows the example of the method of matching each closed area of two images which contain several closed area. It is a figure which shows the result of matching of the closed area of each of two images containing a several closed area. It is a flowchart which shows the example of matching of an area | region. It is a figure which shows the example of matching of the intersection which exists in each of two images which contain several closed area | regions. It is a figure which shows the result of matching of each intersection of two images which contain several closed area | regions. It is a figure which shows the example of the 1st subject generate | occur | produced in the case of the matching of the intersection which exists in each of two images which contain several closed area | regions. It is a figure which shows the example of the process which corrects the intersection and boundary line which exist in the image containing several closed area | regions. It is a figure which shows the example of matching of an intersection based on the result of the process which corrects the intersection and boundary which exist in the image containing a several closed area. When two intersection points are regarded as a group, it is a flow chart which discovers the case where the pattern of a plurality of closed regions which exist around them is the same. It is a flowchart which shows the process which matches the intersection. It is a figure which shows the 1st example which produces a middle division image automatically. It is a figure showing an example which generates a plurality of middle division images. It is a figure which shows the example which provides an intersection, when there is no intersection in the isolated closed path. It is a figure which shows the example of a process of the intersection in case one of two original pictures has a closed circuit isolated. FIG. 8 is a diagram showing an example of performing processing of dividing an original image into layers, and an example of processing when a part of closed regions correspond to each other. It is a figure which shows the example of the hardware constitutions of this embodiment. It is a figure which shows the example which produces | generates a middle division image using interpolation, when an area | region involves rotation. It is a figure which shows notionally the example which produces | generates the area | region of a middle division image. It is a figure which shows notionally the other example which produces | generates the area | region of a middle division image. It is a figure which shows the example of the procedure which produces | generates the area | region of a middle division image. It is a figure which shows the example which corrects one of area | regions, when two area | regions are connected by the joint. It is a figure which shows the example of two images in which two area | regions are connected by the joint. It is a figure which shows the example which divides | segments an area | region and divides it into another layer of a layer. It is a figure which shows the shift | offset | difference of the area | region in a half-divided image when a half-divided image is formed from each of two layers. It is a figure which shows the example which eliminates the shift | offset | difference of two area | regions in a middle division image by giving a joint point.

  FIG. 1 is a diagram showing an example in which a two-dimensional animation is configured by a plurality of frames including an original image and a split image.

  A time axis t is shown in FIG. A designer draws an original 1 and an original 2. From time t1 to t2, a frame including the original picture 1 is displayed. Then, a frame including the original image 2 is displayed from time t3. In this case, since a frame from time t2 to t3 can not be a blank image, a middle split image N is created, and a frame including the middle split image N is displayed from time t2 to t3 to create a moving image. Can be displayed smoothly. In the example of FIG. 1, although the case where one middle division image N is added is illustrated, in fact, two or more middle division images may be created between original picture 1 and original picture 2 . FIG. 1 shows one split image N in order to simplify the explanation.

  If it is possible to generate an image that represents an intermediate movement between the original image 1 and the original image 2, it is possible to automatically draw the middle divided image N. In order to draw the middle divided image N, for example, if the correspondence between the main intersections in the original image 1 and the original image 2 is known, for example, the intersection point (for example, midpoint) obtained by interpolating the positions of corresponding intersection points is obtained. The corresponding intersection points may be determined in the middle divided image. For example, it is understood that the intersection 11 of the original image 1 and the intersection 21 of the original image 2 are corresponding intersection points. Therefore, for example, the coordinates of the intersection N1 of the middle divided image N are determined by determining the coordinates of the intersection 11 and the coordinates of the intersection (for example, the midpoint of the frame) obtained by interpolating the coordinates of the intersection 21 (for example, the pixel position of the frame). Can.

  The correspondence between the intersections of the original 1 and the original 2 is as follows. That is, the intersection 11, the intersection 12, the intersection 13 and the intersection 14 in the original image 1 correspond to the intersection 21, the intersection 22, the intersection 23 and the intersection 24 in the original image 2 respectively. If it is possible to find the correspondence between the plurality of intersections in the original image 1 and the original image 2, the coordinates of the plurality of intersection points N1, intersection points N2, intersection points N3, and intersection points N4 of the middle divided image N are, for example, the original 1 and the original 2 It can obtain | require by interpolating the corresponding intersection of and.

  The above process is described on the premise that the correspondence between each intersection of the original image 1 and each intersection of the original image 2 is known. However, in reality, relying on the instruction of the operator on the correspondence between the respective intersections of the original image 1 and the respective intersections of the original image 2 forces the operator to do a lot of work. Therefore, it is required to automatically estimate the correspondence between each intersection of the original 1 and each intersection of the original 2.

  Turning to FIG. FIG. 7 is a diagram showing an example of the correspondence of the intersection points present in each of two images including a plurality of closed regions in order to estimate that the intersection point 13 of the original image 1 and the intersection point 23 of the original image 2 are in correspondence. It is.

  The figure which expanded the periphery of the intersection 13 of FIG. 7 (A) is FIG. 7 (C). Moreover, the figure which expanded the periphery of the intersection 23 of FIG. 7 (B) is FIG. 7 (D).

  As shown in FIG. 7C, it can be seen that the intersection point 13 is an intersection point on the boundary line formed by the background Y, the closed region 1B, and the closed region 1C. As shown in FIG. 7D, it can be seen that the intersection point 23 is an intersection point on the boundary line formed by the background Y, the closed region 2B, and the closed region 2C.

  Here, it is assumed that it is known that the closed region 2B is in correspondence with the closed region 1B and that the closed region 1C and the closed region 2C are in correspondence. It is assumed that background Y is the same kind of background in original picture 1 and original picture 2 and it is known that they are in correspondence. A plurality of closed regions existing around the intersection 13 in FIG. 7C exist in the order of the closed region 1B, the background Y, and the closed region 1C in the counterclockwise direction, and similarly, the intersections 23 in FIG. The plurality of closed regions present in the periphery are present in the order of the closed region 2B corresponding to the closed region 1B counterclockwise, the background Y, and the closed region 2C corresponding to the closed region 1C. From this, it is possible to automatically estimate that the intersection point 13 and the intersection point 23 are in a corresponding relationship.

  In the above estimation, it is assumed that the correspondence between a plurality of closed regions existing around the intersection 13 of the original image 1 and a plurality of closed regions existing around the original image 2 is known. However, relying on the instruction of the operator for the correspondence between each of the plurality of closed regions of the original image 1 and the plurality of closed regions of the original image 2 imposes much labor on the operator. Therefore, it is required to automatically estimate, as much as possible, the correspondence between each of the intersections of the original image 1 and each of the intersections of the original image 2.

  Returning to FIG. 1, the above automatically determines that the closed area 1A, the closed area 1B and the closed area 1C of the original image 1 correspond to the closed area 2A, the closed area 2B and the closed area 2C of the original image 2, respectively. It is required to estimate.

From the above, in order to automatically generate a split image N located midway on the time axis of two originals 1 and 2 continuous in time, the process of estimation reverse to the above procedure is used. It is required to execute as follows.
(A) Estimating the correspondence between closed regions existing in original picture 1 and original picture 2;
(B) Estimating the correspondence of the intersection points of the boundaries dividing the closed regions present in the original image 1 and the original image 2 using the information on the correspondence of the closed regions.
(C) Determining the coordinates of an intersection that interpolates the associated intersection.
(D) Generating a split image using the shapes of the corresponding closed regions of the original image 1 and the original image 2 and the coordinates of the interpolated intersection point.

  In the process of (a), since the closed regions are associated, if, for example, the coloring of the original image 1 is specified, the corresponding closed regions in the original image 2 can be almost automatically colored. Furthermore, since the closed area corresponding to the closed area in the original image is known even in the intermediately divided image generated in the step (d), the coloring of the closed area of the intermediately divided image can also be performed substantially automatically.

  Therefore, in the above description of the prior art, the coloring process is located after the process of creating the split image, but in the present embodiment, the creating process of the split image and the coloring process are performed in the reverse order. Know that it may be

  Hereinafter, detailed embodiments will be described in order.

<1> Embodiment 1: Correspondence of Closed Areas <1-1> Definition of Terms

  The first embodiment shows an example of processing for automatically correlating a plurality of closed regions present in two original images. The following first embodiment is an example of processing, and is not limited to this example.

  FIG. 2 is a diagram showing an example of feature extraction of each pixel constituting an original image.

  Here, the following definition is performed using FIG. 2 (A).

  "Closed region": In the image, a region closed by a line is called "closed region". For example, in FIG. 2A, the closed area 1A, the closed area 1C, and the closed area 1B are respectively "closed areas" closed by lines. The background of the original is not a closed area. Also, neither straight lines or points nor images are closed regions.

  "Border": A line surrounding a closed area is called a border. The closed area and the closed area or background are adjacent to each other across the boundary.

  “Adjacent”: when the closed region A and the closed region B are in contact with each other at the boundary line X, the closed region A and the closed region B are said to be adjacent at the boundary line X.

  “Intersection point”: A point at which the boundaries intersect, or an end point of the boundaries (end point) is called an intersection.

  "Pixel": Meshes are defined at equal intervals in a two-dimensional plane present in an image, and a component of the image present at the coordinate position of each intersection of the mesh is called a pixel. A pixel may correspond to each pixel that constitutes a digital image.

  "Direction": In FIG. 2A, for example, four directions from the pixel P1 in the upper, lower, left, and right directions are referred to as "direction". Then, semi-lines U, D, L, R extending in the respective directions of up, down, left, and right are defined. The half lines U, D, L, R are half lines extending upward, downward, leftward, and right, respectively, from the pixel P1. In the example of FIG. 2A, the half straight line is defined for each of the four directions in the upper, lower, left, and right directions, but the number of directions is not limited to four.

  “Number of crossings”: In FIG. 2A, for example, a half straight line U extending in the upper direction from the pixel P1 crosses at a boundary line U11 with the closed region 1A constituting the original image 1. In this case, the number of crossings in the upper direction of the pixel P1 is defined as one. Similarly, a half straight line L extending in the left direction of the pixel P1 intersects with the boundary line with the closed regions 1A, 1C, 1B constituting the original picture 1 at L11, L12, L13. In this case, the number of crossings in the left direction of the pixel P1 is defined as three. Similarly, a half straight line D extending in the lower direction of the pixel P1 intersects with the boundary with the closed region A1 constituting the original 1 at D11. In this case, the number of downward crossings of the pixel P1 is defined as one. Similarly, a half straight line R extending in the right direction of the pixel P1 intersects the boundary line with the closed region 1A constituting the original 1 at R11. In this case, the number of crossings in the right direction of the pixel P1 is defined as one.

  “Distance to boundary”: In FIG. 2A, for example, a distance from the pixel P1 to a closed region including the pixel P1 for the first time is defined as “distance to the boundary”. In FIG. 2A, the distance to the upper azimuth boundary of the pixel P1 is α1. The unit of the distance to the boundary may be, for example, the number of pixels.

  For example, when a pixel belongs to a certain closed region, the pixel always crosses the boundary of the closed region in any orientation, so the number of crossings is 1 or more. If the pixel belongs to the background, the number of crossings may be zero.

<1-2> Constituent Elements of Original Picture The original picture shown in FIGS. 1 and 18 is a line drawing in many cases, and is constituted of, for example, the following elements.
(A) Closed area (area closed by a boundary line)
(B) Line segment branched from closed region (branch line segment)
(C) Isolated segment included in closed region (isolated segment)

  The branch line segment existing between the two original images and the isolated line segment can be associated by comparing relative coordinates of the boundary of the closed region. Therefore, it is desirable to first associate the closed region between the two original images. In the first embodiment, an example of the correspondence between closed areas is shown.

  <1-3> Order of generation of coloring and split images

  The above-described prior art example shows the procedure in which the original image and the split image are colored after the split image is created. However, in the first embodiment, since the closed regions of the original image are first associated, it is possible to color the corresponding closed regions of the original image at this stage. First, closed regions are associated between the original images. Then, in the plurality of original images, the same color is applied to the corresponding closed regions in the coloring. In this way, it is possible to save the trouble of performing coloring for each original.

  In addition, if the closed region is already associated between the two original images, then even if a split image located between the two original images on the time axis is subsequently generated, it is included in the split image. It is possible to easily associate the closed area with the closed area of the two original images.

  Therefore, after the closed regions in the original image are first associated, coloring can be performed, and substantially automatic coloring can be performed on the closed regions included in the split image generated thereafter.

<1-4a> Definition of Feature Vector of Pixel of Original Image In the present embodiment, the feature V (P) of the pixel P present in the original image is defined as an eight-dimensional vector as follows.

_{V (P) = (c 0} , c 1, c 2, c 3, c 4, c 5, c 6, c 7) ( Equation 1)
Here, c _{0 to} c ₇ are defined as follows. Here, β is a constant.
Let αu, αd, αl, and αr be the distances from the pixel P to the upper, lower, left, and right boundaries, respectively.
c ₀ = 1 / (α u + β) where the number of crossings in the upper direction is 1. In other cases c ₀ = 0
c ₁ = 1 / (αr + β) where the number of crossings in the right direction is 1. In other cases c ₁ = 0
c ₂ = 1 / (αd + β) where the number of lower order crossings is one. In other cases c ₂ = 0
c ₃ = 1 / (α 1 + β) where the number of crossings in the left direction is 1. In other cases c ₃ = 0

c ₄ = 1 / (α u + β) However, when the number of crossings of the four directions in the upper, lower, left, and right directions is 2 or more, and the number of crossings in the upper direction is the minimum among the number of crossings of the four directions in the upper, lower, left, and right directions. In other cases c ₄ = 0

c ₅ = 1 / (α r + β) However, when the number of crossings of the four directions in the upper, lower, left, and right directions is 2 or more, and the number of crossings in the right direction is the minimum among the number of crossings of the four directions in the upper, lower, left, and right directions. In other cases c ₅ = 0

c ₆ = 1 / (α d + β) However, when the number of crossings of the four azimuths in the upper, lower, left, and right directions is 2 or more, and the number of crossings in the lower rank is the smallest among the number of crossings of the four azimuths in the upper, lower, left, and right. In other cases c ₆ = 0

c ₇ = 1 / (α 1 + β) However, when the number of intersections of the four azimuths in the upper, lower, left, and right directions is 2 or more, and the number of intersections in the left azimuth is the smallest among the number of intersections of the four azimuths in the upper, lower, left, or right. In other cases c ₇ = 0

As for c ₀ , c ₁ , c ₂ , and c ₃ , when there are a plurality of orientations in which the number of crossings is 1, the elements of the feature vector corresponding to the orientations in which the number of crossings is 1 have values. Further, for c ₄ , c ₅ , c ₆ and c ₇ , when there are a plurality of orientations with the minimum number of crossings of 2 or more, the elements of the feature vector corresponding to the plurality of orientations of the minimum number of crossings It will have a value.

If the pixel P is a pixel belonging to the background or the boundary line, c ₀ = c ₁ = c ₂ = c ₃ = c ₄ = c ₅ = c ₆ = c ₇ = 0.

  In addition, the above-mentioned rule is only an illustration, and is not limited to this. As a method of determining other elements of the feature vector, for example, the following rule (a) or (b) may be mentioned.

  (A) In the above rule, when there are a plurality of the same number of crossings, it is defined that all of the plurality of orientations have elements of the feature vector. However, without doing so, priorities are provided in advance in the upper, lower, left, and right azimuths, and when there are a plurality of azimuths with the same number of crossings, among the azimuths having the same number of crossings, Only the distance to the border may be adopted. And the other elements of the feature vector may be zero.

(B) Among the feature vectors, the c ₀ , c ₁ , c ₂ and c ₃ components have a distance to the boundary line in the direction in which the number of crossings is 1 when there is a direction in which the number of crossings is 1 The feature amount 1 / (α + β) using α with the smallest α may be defined as an element of the orientation of α with the smallest distance to the boundary. Then, among the feature vectors, the components c ₄ , c ₅ , c ₆ and c _{7 have four} orientations when there is no orientation with 1 as the number of intersections and the number of intersections is 2 or more in all the orientations. Among them, the feature quantity 1 / (α + β) using α, which has the smallest distance to the boundary, may be provided as an element of the orientation of α, which has the smallest distance to the boundary. And, elements of feature vectors in other orientations may be zero.

In each of the above-mentioned examples, c ₀ and c ₄ represent components of upward orientation. c ₁ and c ₅ represent components in the right direction. c ₂ and _{c 6} represent the component of the lower position. c ₀₃ and c ₇ represent components in the left direction.

  The components of the feature vector of the pixels present in the background may be all zero.

  Note that the above-mentioned feature vector is an example as already pointed out, and the feature vector for achieving the task may be in another format. Furthermore, for example, the feature vector may be a four-dimensional vector having four orientations in the upper, lower, left, and right directions. The function based on the distance to the borderline of the orientation with the least number of intersections in the four orientations may be set as the element of the dimension of the corresponding orientation, and the values of the elements of other dimensions may be zero.

<1-4b> Properties of Feature Vectors The above feature vectors have the following properties.

  In the first embodiment, it is assumed that the original image is a line drawing composed of lines. The object on which the original is drawn is called an object. An object is drawn by a plurality of line drawings and includes a plurality of closed regions.

  The boundary between the object and the background is more distinctive than the boundary between the closed regions present in the object. This is that the border in contact with the background is a line that can also be called a contour that constitutes the silhouette of the object, and is a border that more strongly expresses the features of the object than lines existing inside the object. It is because it is known in the rule of thumb.

  If the number of crossings is 1, it indicates that the boundary existing in the direction is the boundary with the background. On the other hand, the boundary existing in the orientation in which the number of crossings is 2 or more is a boundary which has entered the object and is a boundary which is not in contact with the background.

  If at least one feature vector of each pixel defined above has an orientation where the number of crossings is 1, the distance to the boundary in that orientation is preferentially used. When there are a plurality of orientations with the number of intersections being 1, a component of the feature vector is generated using the shortest distance to the boundary line among the plurality of orientations. If the number of crossings does not have an orientation of 1, the distance to the boundary with the smallest value is adopted among the orientations of which the number of crossings is 2 or more.

  Thus, the reason for preferentially using the distance to the border where the number of crossings is 1 is to emphasize the border in contact with the background.

  Then, assuming that the distance to the adopted boundary line is α, 1 / (α + β) is calculated, and the value of the predetermined vector component of the 8-dimensional vector is set, and the values of the other vector components are set to zero. Here, β is a constant. The reason for adding the constant β to α is set so as to avoid the value of the vector component becoming infinite because α is zero on the border line. It also depends on how much the size of the closed region is to be targeted, and to what extent a component of a large value is provided as a feature in the vicinity of the boundary. Since the distance α to the boundary exists in the denominator, the component 1 / (α + β) of the vector takes a larger value (increasing function) as the pixel is closer to the boundary in the closed region. By placing α in the denominator in this way, it is possible to give larger feature quantities to the part close to the boundary line in the closed region. As described above, giving a larger feature amount to a part close to the boundary can give a characteristic so as to place more importance on the shape of the boundary. This is because the feature of the shape appears in the part closer to the boundary line than near the center of the closed region of the object, and by using this feature vector, the correspondence between the closed regions of the originals using the cost function described later This is to make it possible to estimate the sticking with high probability. Note that the feature amount 1 / (α + β) is an example, and is not limited to this feature amount. The feature value is preferably a function that emphasizes the vicinity of the boundary of the closed region by selecting a function that takes a large value near the boundary within the closed region.

<1-5> Example of feature vector of pixel (1) Feature vector of pixel P1 (FIG. 2 (A))
The number of crossings of the four azimuths of the pixel P1 is as follows.

  Since the pixel P1 intersects at U11 in the upper azimuth, the number of intersections in the upper azimuth is one.

  Since the pixel P1 intersects at R11 in the right orientation, the number of intersections in the right orientation is one.

  Since the pixel P1 intersects at D11 in the lower position, the number of intersections in the lower position is one.

  Since the pixel P1 intersects at L11, L12, and L13 in the left azimuth, the number of intersections in the left azimuth is three.

Then, the pixel P1 has an azimuth of the number of times of intersection in the upper azimuth, the right azimuth, and the lower position. The distance from the upper azimuth boundary is α1U. The distance from the boundary to the right is α1R. The distance from the lower position boundary is α1D. Therefore, the feature vector V (P1) of the pixel P1 is as follows.
V (P1) = (1 / (α1U + β), 1 / (α1R + β), 1 / (α1D + β), 0, 0, 0, 0, 0)
(2) Feature vector of pixel P2 (FIG. 2 (B))
The number of crossings of the four orientations of the pixel P2 is as follows.

  Since the pixel P2 intersects at U21 and U22 in the upper azimuth, the number of times of intersection in the upper azimuth is two.

  Since the pixel P2 intersects with R21 and R22 in the right azimuth, the number of intersections in the right azimuth is two.

  Since the pixel P2 intersects at D21 and D22 in the lower position, the number of lower intersections is two.

  Since the pixel P2 intersects at L21 and L22 in the left azimuth, the number of intersections in the left azimuth is two.

Then, in FIG. 2B, there is no bearing whose number of crossings is 1, and the value of the smallest number of crossings among the bearings whose number of crossings is 2 or more is 2, and the bearing is 4 All three orientations. The distance from the upper azimuth boundary is α2U. The distance from the right-facing boundary is α2R. The distance from the lower boundary is α2D. The distance from the left-facing boundary is α2L. Therefore, the feature vector V (P2) of the pixel P2 is as follows.
V (P2) = (0, 0, 0, 0, 1 / (α2U + β), 1 / (α2R + β), 1 / (α2D + β), 1 / (α2L + β))
(3) Feature vector of pixel P3 (FIG. 2 (C))
The number of crossings of the four orientations of the pixel P3 is as follows.

  Since the pixel P3 intersects at U31 and U32 in the upper azimuth, the number of times of intersection in the upper azimuth is two.

  Since the pixel P3 intersects with R31 and R32 in the right azimuth, the number of intersections in the right azimuth is two.

  Since the pixel P3 intersects at D31 in the lower position, the number of intersections in the lower position is one.

  Since the pixel P3 intersects at L31 and L32 in the left azimuth, the number of intersections in the left azimuth is two.

Then, in FIG. 2C, since there is an azimuth having the number of crossings of 1, when calculating the feature of the pixel P3, the distance from the lower-order boundary line is α3D of the azimuth of the number of crossings 1. To adopt. Therefore, the feature vector V (P3) of the pixel P3 is as follows.
V (P3) = (0, 0, 1 / (α3D + β), 0, 0, 0, 0, 0)
<1-6> Creation of a management table in which the closed areas constituting two original images are arranged in descending order

  As pre-processing for correlating a plurality of closed regions included in the original image 1 and the original image 2 shown in FIG. 1, a table may be created in which a plurality of closed regions present in each original image are arranged in descending order of area. By making this table, it is possible to evaluate the correspondence from the large closed area. The reason for creating such a table is that, for closed areas with a large area, it is better to establish the correspondence early and remove them from the correspondence candidates than to establish the correspondence of the closed area with a smaller area first. Also, there is a rule of thumb that it is easy to find the correspondence in the closed region and the accuracy of the correspondence is also high. Therefore, it makes sense to create such a table in order to check the correspondence in the order of the corresponding possibility.

  FIG. 3A is a management table in which three closed areas (closed area 1A, closed area 1B, and closed area 1C) of the original image 1 of FIG. 1 are arranged in the order of increasing area. No. Indicates the descending order of the closed region, and the closed region index is assigned an index (for example, 1A) labeled in the closed region. The index may automatically assign a unique identification to each closed region. The number of pixels represents the number of pixels included in each closed region, and since it is a numerical value proportional to the area, it may be substituted as a number representing the area. The number of pixels may be used, for example, in the case of sorting in the descending order of the area of the closed region. In the column of “association”, all zeros (0) are stored as initial values. For example, 1 is stored in the closed area where the association of the closed area described later is completed. Then, the closed region in which 1 is stored may be excluded from the objects of the subsequent association processing in order to indicate that the association processing has ended. Also, as will be described later, a closed area in which the association is only partially completed should not change “association” to 1 and remain 0 so that it becomes a target of subsequent association processing. Good. An example of this process will be described later.

  FIG. 3B is a management table in which the closed regions in the original image 2 of FIG. 1 are arranged in descending order of area.

<1-7> Correspondence of Closed Region An example of estimating the correspondence of closed regions respectively present in the original image 1 and the original image 2 will be shown below.

  When correlating closed regions, as described above, as a rule of thumb, a closed region with a large area of the closed region of one of the originals (for example, original 1) may correspond to the other of the originals. It is easy to find closed regions with high Therefore, in the following, first, the closed area of the largest area in the original image 1 is taken out in order, and the close area taken out is calculated using the cost function E exemplified below. For example, it is determined that the smaller the cost function E is, the motion vector (optimum motion vector) that minimizes the cost function E is determined, and the closed motion vector is extracted from the original image 1 using the optimal motion vector. The closed region of the original image 2 that has the largest overlap with the region may be found, and this combination may be estimated as a corresponding closed region pair. Alternatively, the pair of closed regions thus found may be displayed to prompt the operator for confirmation. At this time, the operator may cancel the association of the pair by inputting that the estimation of the association is incorrect. In this case, a corresponding closed area with a large overlap may be found next and displayed as a pair candidate. Alternatively, the next optimal vector may be used to find the closed region of the original image 2 having the largest overlap with the closed region of the original image 1 and displayed as a candidate for the next pair.

For example, a closed area 1A that occupies the largest closed area among _i closed areas D _i (three closed areas 1A, 1B, and 1C) present in the original image 1 of FIG. The cost function E (D _i , δ) for the closed region A1 is calculated according to Equation 2 while moving the movement vector δ, overlapping the original image 2 in FIG. 3B for all the pixels p included in the region 1A. It is also good.

Here, p + [delta] is shifted original 2 and its background by the movement vector [delta], when superimposed on the closed region D _{i (e.g.} closed region 1A), the pixel p in the closed region D _{i (e.g.} closed region 1A) It is the position of the original image 2 or background that overlaps the position. V1 (p) is a feature vector of the pixel at the pixel position p of the closed region D _i (for example, the closed region 1A) of the original image 1. Then, V2 (p + δ) is a feature vector of the pixel at the pixel position p + δ of the original image 2 and its background. If the original image 2 overlapping the pixel position p of the closed region D _i of the original image 1 (for example, the closed region 1A) is inside the closed region (for example, inside the closed region 2A), the square of the difference between the feature vectors of the original image 1 and the original image 2 Is calculated and integrated (equation 2 upper stage). If the original image 2 overlapping the pixel position p of the closed region D _i of the original image 1 (for example, the closed region 1A) is not inside the closed region (for example, background or border), the lower part of Equation 2 is calculated and the constant γ is integrated. Do. The meaning of integrating γ is that the cost function becomes large and a penalty is given by adding a relatively large constant γ for overlapping with the background portion. γ selects an appropriate value according to the total number of pixels of the original image or the number of pixels of the closed region. For example, a constant such as γ = 2 may be set.

  In the case where the position of p + δ of the original image 2 is not inside the closed region, calculation in the lower part of Equation 2 is performed. For example, the closed region in the original image 2 already associated is treated the same as the background etc. The lower part of Equation 2 may be calculated.

  The feature vector is already shown in Equation 1.

The movement vector δ is, for example, a two-dimensional movement vector in units of pixels, takes out the closed region D _i (for example, the closed region 1A) of the original image 1 and moves the original image 2 and its background by the movement vector δ It is a vector used to overlap. Then, a two-dimensional movement vector δ ^* _i which minimizes the cost function E of the equation 2 is obtained. For example, while changing the value of δ between the closed region 1A, the original image 2 and the background thereof, δ ^* _i which minimizes the cost function E is obtained as in the following equation.
δ ^* _i = argmin _δ E (D _i , δ) (Equation 3)
Here, arg min _δ E (D _i , δ) means to obtain an argument δ which minimizes the value of E (D _i , δ).

Then, for example, with respect to one closed region D _i of the original image 1 (for example, the closed region 1A), the cost function E is calculated by overlapping the original image 2 and the background while changing δ exhaustively to minimize the cost function E Thus, δ ^* _i which satisfies the above equation 3 is determined. Alternatively, if the minimum value (minimum value during calculation) of the cost function E is not updated within the predetermined number of calculations in the middle of the exhaustive calculation of the cost function E, the comprehensive calculation of δ is performed. It is possible to stop halfway and adopt the minimum value of the cost function obtained so far. Alternatively, in the calculation of the cost function E for a large area closed area, for example, the cost function E may be calculated to reduce the amount of calculation by limiting to every other pixel of the original 1 closed area. , You may speed up the calculation speed.

When this movement vector δ ^* _i is used, the closed region of original image 2 in which the overlapping area (for example, the number of pixels) of the closed region is the largest between closed region D _i (for example, closed region 1A) and original image 2 is It can be estimated that the closed region of the original image 2 corresponds to the closed region D _i of the original image 1.

  Hereinafter, an example of estimation of correspondence is concretely shown using FIG.

  In order to make the explanation easy to understand, an example in which the closed region of the original image 2 corresponding to the closed region 1C of the original image 1 is obtained will be shown using FIG.

  In FIG. 4A, the closed region 1C of the original image 1 is taken out and superimposed on the original image 2 as it is. Then, in this state, the cost function E (C1, δ) is determined for the closed region C1. The initial value of δ is a zero vector. The movement vector δ is drawn so as to move 1 C in FIG. 4, but in Expression 2, it is added to the coordinate position of the term of the original image 2. Since the movement vector δ is a vector that defines the relative positional relationship of superposition, in the figure, the point drawn as a vector for moving the closed region extracted from the original image 1 for easy intuitive understanding It should be noted. The descriptions of Equation 2 and FIG. 4 are synonymous with respect to relative expressions.

Thereafter, the movement vector δ is comprehensively changed, and δ ^* which minimizes the cost function E (C1, δ) is determined as shown in FIG. 4B.

  As shown in FIG. 4B, among the closed regions of the original image 2, the closed region of the original image 2 in which the area overlapping with the closed region C1 is the largest is determined. In this case, the closed region 2C and the closed region 1C overlap in the region S1. The closed area 2A and the closed area 1C overlap in the area S2. Among these overlaps, it can be seen that the largest overlap is S2. Therefore, it is estimated that the closed area 1C and the closed area 2C sharing the largest overlapping area correspond to each other.

  Thus, the correspondence between the closed region 1C of the original image 1 and the closed region 2C of the original image 2 was estimated. This estimate may be displayed to prompt the operator for confirmation. Alternatively, the association may be determined without obtaining confirmation from the operator. If the operator inputs that the estimation of the association is incorrect, in the case of the area where the overlapping area is the second largest (FIG. 4 (B)), the closed area 1C and the closed area 2A The pair may be displayed to estimate and display the correspondence of the next candidate.

Alternatively, if the operator inputs that the estimation of the association is incorrect, the area of overlap with C1 is the largest, using the movement vector δ, which is next smaller than the movement vector δ ^*. The closed region of the large original image 2 may be estimated and displayed as a candidate for the next closed region corresponding to the closed region C1 to prompt the operator for confirmation.

  FIG. 5 shows a closed area correspondence table. By the above results, the correspondence between the closed regions present in the original picture 1 and the original picture 2 is determined, and the correspondence table of FIG. 5 is stored. If coloring is performed in the original picture 1 using this correspondence table, coloring in the original picture 2 can be automatically performed.

<1-8> Operation of Correspondence of Regions FIG. 6 is a flowchart showing an example of correspondence of regions.

  In step S100, parameters i and j are initialized. The parameter i may be, for example, the order (No) when the closed regions of the original image 1 are arranged in descending order of area as shown in FIG. 3 (A). i may be a number that can uniquely identify each closed region of the original image 1. The parameter j may be, for example, the order (No) when the closed regions of the original image 2 are arranged in descending order of area as shown in FIG. 3 (B). j may be a number that can uniquely identify each closed region of the original image 2.

  In step S102, one closed region i (for example, closed region 1A) is extracted from one image (original image 1).

In step S104, a cost function E (D _i , δ) is calculated from the closed region D _i and the other original image.

  In step S106, it is checked whether δ is comprehensively changed. If the check is "No", the process proceeds to step S132. If the check is "yes", the process proceeds to step S108.

  In step S132, δ is changed by a predetermined value.

In step S108, the movement vector δ ^* ₁ in the case where the cost function E (D _i , δ) is minimized is stored.

In step S110, the closed region D _j of the other original image having the largest area overlapping with the closed region D _i is estimated to be a region corresponding to the closed region D _i using δ ^* ₁ .

  In step S152, the correspondence of the closed area may be displayed to prompt the operator for confirmation. If it is input that the correspondence is incorrect, the next pair of candidates may be displayed. Also, the correct correspondence of the closed regions may be input by the instruction of the operator. At this time, an instruction to color may be prompted.

In step S154, of the closed region D _j closed region D _i and original 2 of the original 1 correspondence relationship is determined, may be excluded from the scope of the subsequent processing of the correspondence. This operation may use the association flag shown in FIG.

  In step S112, it is checked whether i = I. That is, it is checked whether the process of association has been completed for all closed regions in the original image 1. If this check is "No", it will move to step S134. If this check is "Yes", the entire process may be terminated.

  In step S134, i is incremented. By this operation, the next closed region of the original image 1 can be extracted in step S102.

  The outline of the process flow of the first embodiment is as described above.

<2> Embodiment 2: Correspondence of intersections <2-1> Details of processing of correspondence of intersections

  The correspondence between the intersections present in each of the original image 1 and the original image 2 will be described below. The information on the correspondence of the intersections can be used to generate an image obtained by interpolating the original 1 and the original 2.

  FIG. 7 is a diagram showing an example of correspondence of intersection points present in each of two images including a plurality of closed regions.

  It is assumed that the correspondence between each of the closed regions between the original image 1 and the original image 2 has been completed by the processing already described. An enlarged view of the circled closed region of the original picture 1 of FIG. 7 (A) is shown in FIG. 7 (C). And the enlarged view of the enclosed area | region enclosed with the circle of the original image 2 of FIG. 7 (B) is shown by FIG. 7 (D). It is known that the closed region 2B corresponds to the closed region 1B and the closed region 2C corresponds to the closed region 1C. Further, the same background Y exists in the original picture 1 and the original picture 2.

  As already described, the plurality of closed regions existing around the intersection 13 in FIG. 7C exist in the order of the closed region 1B, the background Y, and the closed region 1C in the counterclockwise direction. A plurality of closed regions present around the intersection 23 in D) are present in the order of the closed region 2B corresponding to the closed region 1B counterclockwise, the background Y, and the closed region 2C corresponding to the closed region 1C. Therefore, it is understood that the pattern of the positional relationship of the closed region in contact with the intersection point 13 matches the pattern of the positional relationship of the closed region in contact with the intersection point 23. From this, it is possible to automatically estimate that the intersection point 13 and the intersection point 23 have a corresponding relationship. In this way, by using the information on the associated closed region, it is possible to automatically estimate the correspondence of the corresponding intersection points. The estimation result may be displayed on the screen to prompt the operator to determine the final correspondence. Alternatively, the correspondence relationship between the estimated intersection points may be determined automatically.

  FIG. 8 is a diagram showing the result of the mapping of the intersection points of each of two images including a plurality of closed regions. The information of the corresponding intersections can be stored by the correspondence table of the intersections shown in FIG.

  FIG. 9 is a diagram showing an example of a first problem that occurs in association of intersection points present in each of two images including a plurality of closed regions. In FIG. 9, it is assumed that the closed regions 1A and 2A, the closed regions 1B and 2B, the closed regions 1C and 2C, and the closed regions 1D and 2D are respectively corresponding closed regions.

  In this case, in the original image 1 of FIG. 9A, closed regions 1A, 1D, and 1B exist counterclockwise around the intersection point P11. On the other hand, in the original image 2 of FIG. 9B, closed regions 2A, 2D and 2C exist at the intersection point P21. In this example, it is determined that the intersection point P11 and the intersection point P21 are not corresponding intersection points because there is no corresponding closed region with the same pattern in the periphery. The intersection point corresponding to the intersection point P11 of the original image 1 does not exist in the original image 2.

  However, when the two intersections (intersection P11 and intersection P12) and FIG. 9 (B) (intersection P21 and intersection P22) in FIG. 9A are regarded as a group, the periphery of the intersection P11 and the intersection P12 of the original image 1 There are closed areas 1A, 1D, 1C, and 1B counterclockwise, and closed areas 2A, 2D, 2C, and 2B exist counterclockwise around the intersection point P21 and the intersection point P22 of the original image 2. In consideration, it can be seen that closed regions exist in the same pattern.

  Also in the example of FIG. 9, the intersection points can be associated by performing the following processing.

  FIG. 10 is a diagram showing an example of a process of correcting intersection points and boundaries present in an image including a plurality of closed regions.

  In FIG. 9A, an original image 1 may be selected, and an original image 2 may be selected instead of the original image 1. As shown in FIG. 10A, one of the two intersections (intersection P11, intersection P12) present in the selected original image 1 is deleted. In the case of FIG. 10A, the intersection point P12 is erased. The intersection to be deleted may be any intersection. This leaves the boundaries Ldb, Lbc, Lcd of the closed region. It is to be noted that among the intersections, the smaller one of the intersection angles (that is, the angle that Lcd and Lbc make on the area 1C side at the intersection P12 is smaller than the angle that Lad and Lab make on the area 1A side at the intersection P11) In this case, the intersection point P12) may be deleted in priority.

  As shown in FIG. 10 (B), the boundary line Lbc is extended along Ldb to the intersection point P11. An intersection point of the position of the intersection point P11 of the extended Lbc is taken as an intersection point P111.

  As shown in FIG. 10C, the boundary line Lcd is extended along Ldb to the intersection point P11. An intersection point of the position of the intersection point P11 of the extended Lcd is taken as an intersection point P112.

  After the above operation, Ldb may be deleted. Note that the timing of deletion of Ldb is not limited at this time, and may be deleted at other timings.

  As a result, as shown in FIG. 10D, the intersection point P12 is deleted, the boundary Lbc is extended, and the intersection point P111 is set at the position of the intersection point P11. The boundary line Lbc is extended, and the intersection point P112 is set at the position of the intersection point P11. Therefore, it can be considered that the intersection point P11 is divided into two, the intersection point P111 and the intersection point P112. Although the extended border Lbc and the extended border Lcd are drawn apart, they are drawn apart for the sake of clarity of the figure, and in fact, they exist on the same line. Do. The same applies to the other figures.

  FIG. 11 is a diagram showing an example of the correspondence of the intersection points based on the result of the process of correcting the intersection points and the boundary lines present in the image including a plurality of closed regions.

  In FIG. 11A, the intersection point P12 is deleted, the boundary line Lbc is extended, and the intersection point P111 is set at the position of the intersection point P11 by the process of FIG. 10 described above. The boundary line Lbc is extended, and the intersection point P112 is set at the position of the intersection point P11. For this reason, the intersection P11 is divided into two, the intersection P111 and the intersection P112.

  As shown in FIG. 11B, the original image 2 has two intersection points P21 and P22 as it is. As shown in the correspondence table of the intersections in FIG. 11C, the intersection P111 of the original 1 and the intersection P21 of the original 2 are associated with each other. Then, the intersection point P112 of the original image 1 and the intersection point 22 of the original image 2 are associated with each other. Then, it is assumed that a boundary of length zero exists between the intersection point P111 and the intersection point P112 in FIG. The intersection point P111 is connected to the boundary line Lab, and the intersection point P112 is connected to the boundary line Lad. The intersection point P11 may be deleted. The timing of deleting the intersection point P11 is not limited to this time, and may be deleted at another timing.

  By performing the above-described processing, one intersection point of any of the original images is deleted even for the intersection points for which association can not be performed as shown in FIG. 9, and association of the intersection points is performed by performing the above processing. It can do.

  [Operation of correspondence of intersection point]

  FIG. 12 shows the case where one intersection point of each of the original image 1 and the original image 2 can not be associated as shown in FIG. 9, and when two intersection points are regarded as a group, they exist around them. It is a flowchart which discovers the case where the pattern of the several closed area to perform is identical. In order to aid understanding, in the flow chart of FIG. 12 and the following description, reference numerals shown in FIG. The flowchart of FIG. 12 is not limited to the example of FIG.

  In step S300, among a plurality of points for which points can not be associated in one original image (original image 1), two adjacent intersection points (Ldb) existing on the boundary line (Ldb) of two closed regions (1D, 1B) The set Q of the set q of P11, P12) is extracted.

  Two or more adjacent intersections which exist on the boundary line (Lac) of two closed regions (2A, 2C) among a plurality of points which could not be matched in the other original image (original image 2) in step S302 The set R of the set r of (P21, P22) is extracted.

  Step S304 shows the process of repetition which performs processing by taking out the elements q included in the set Q one by one between step S314 and step S314.

  Step S306 shows the process of repetition which takes out element r included in the set R one by one and performs processing with step S312.

  In step S308, two of the patterns included in r are a plurality of closed region array patterns (1A, 1D, 1C, 1B counterclockwise) in contact with at least one of two intersection points (P11, P12) included in q. Pattern of an array of closed regions of one original corresponding to each of a plurality of closed regions (2A, 2D, 2C, 2B) in contact with at least one of the intersections (P21, P22) (counterclockwise counterclockwise 1A, 1D, 1C, Check if it matches with 1B).

  Step S310 determines whether the checks in step S308 match. If the checks match, the process proceeds to step S320. If the checks do not match, the process proceeds to step S312.

  At step S320, it is recognized that r and q correspond to each other, and the element s is added to the set S as s = (r, q) and stored.

  By performing the above processing, it is possible to extract the correspondence between two intersection points as shown in FIG.

  FIG. 13 is a flowchart showing a process of taking out the elements s one by one from the set having the set of two intersections extracted in FIG. 12 as the element s, and correlating the intersections. In order to aid understanding, in the flow chart of FIG. 12 and the following description, reference numerals shown in FIG.

  Step S400 indicates that elements s are extracted one by one from the set S, and the process is repeated from step S412.

In step S402, one intersection point is deleted. The flowchart of FIG. 12 is not limited to the example of FIG. In addition, as shown below, a set is defined.
s∈S
s = (q, r)
q∈ intersection point P11, intersection point P12 (intersection point included in one original (original 1))
r∈ intersection point P21, intersection point P22 (intersection point included in the other original (original 2))
In step S402, one of the two intersections (P11, P12) included in the pair (q) of intersections of one original image is deleted.

  In step S404, two border lines (Lbc, Lcd) connected to the deleted intersection (P12) and not connected to the remaining intersection (P12) are identified.

  Intersections remaining along the border line (Ldb) connecting each of the two border lines (Lbc, Lcd) specified in S406 with the eliminated intersection point (P12) and the remaining intersection point (P11) Extend to (P11).

  In step S408, a first intersection point (P111) is generated at the extended end of one of the two border lines (Lbc, Lcd) that has been extended, and the one border line (Lbc) is generated. The point (P21) on the boundary (L2bc) of the closed region (2B, 2C) of the other original (original image 2) corresponding to the two closed regions (1B, 1C) adjacent to the boundary It corresponds to the intersection point of 1 (P111). The intersection point P111 is connected to the boundary line Lab. Similarly, a second intersection point (P112) is generated at the extended end of the other boundary (Lcd) of the two extended boundaries (Lbc, Lcd), and the one boundary (Ldc) is separated. The point (P22) on the border line (L2 cd) of the closed region (2D, 2C) of the other original (original image 2) corresponding to the two closed regions (1D, 1C) adjacent to the Corresponding to the intersection point (P112) of The intersection point P112 is connected to the boundary line Lad.

  At step S410, the remaining intersection point (P11) is deleted. Two generated points of intersection (P111, P112) are connected by a boundary of zero length. The boundary Ldb may be deleted. The intersection point P11 may be deleted.

  By performing the above process, the process of associating intersections can be performed on the pattern of FIG.

  FIG. 14 shows an example of generating a split image N for two original images to which the processing of FIGS. 11, 12 and 13 is applied.

  As shown in the middle divided image N of FIG. 14, the intersection P111 of the original image 1 corresponds to the intersection NP111 of the middle divided image, and the intersection P112 of the original 1 corresponds to the intersection NP112 of the middle divided image. Then, it can be seen that the boundary line whose length is zero in the original image 1 is displayed as the boundary line Lac between the intersection point NP111 and the intersection point NP112 in the middle divided image N. In this way, it is possible to automatically generate a mid-division image by correlating the intersection points.

  In addition, in the case of generating the middle divided image N, interpolation may be performed on the shape of each boundary to generate a smooth middle divided image. Alternatively, the middle divided image N may be generated by the morphing technique using the corresponding intersection point, the corresponding boundary line, the information of the corresponding closed region, and the like. In addition, with regard to lines and points that do not constitute closed regions respectively present in two original images, it is often possible to infer the correspondences from the correspondences between the closed regions and the intersections. The operator may be prompted to specify the correspondence among closed regions, lines, points, etc. existing in the two original images for which the correspondence can not be estimated. Finally, with regard to closed regions, lines, points, etc. for which the operator's instruction is not given and the correspondence relationship is unknown, the positional relationship in the middle divided image from the positional relationship of interpolations such as surrounding closed regions, lines, points, etc. You may decide As seen in the morphing technology, it is possible to generate a split image by using the morphing technology etc., even if the correspondence between all the components present in the two original images is not known. is there.

  FIG. 15 is an example similar to the example shown in FIG. 14, and shows an example in which a plurality of middle division images are generated. Between original image 5 and original image 6, ten split images N01 to N10 are generated along the time axis. The original image and the split image respectively correspond to one frame of the moving image. In the case of FIG. 15, 12 frames of the original image 5, 10 middle-split images and the original image 6 are continuously displayed, and a smooth moving image is reproduced.

  FIG. 16 is a diagram showing an example of forming an intersection by giving an end point to a boundary of a closed region having no intersection. In the case of this example, one intersection point (intersection point P51 and intersection point P61) is shown for each of the original image 7 and the original image 8. The number of intersections of the closed regions in each original image may be plural. In this case, first, two closed regions are placed on one plane. Determine the Hausdorff distance between these two closed regions. Next, for all points on the boundary of one closed region, respective points at which the distance from the point on the boundary of the other closed region is minimum are determined, and the points of one closed region and the points of the other closed region Create a set of 2 point pairs that correspond to In each set, a set in which the distance between two points is equal to the Hausdorff distance is given to each closed region as a corresponding intersection point. When there are a plurality of sets of points where the distance between two points is equal to the Hausdorff distance, any set may be selected as a corresponding intersection point.

  In general, if there are two sets A and B, the Hausdorff distance dH can reach any point of the set B by advancing at least the distance dH at any point of the set A. Similarly, any point in the set B can be reached at any point in the set A by advancing at least the distance dH. Such a distance dH is called the Hausdorff distance. In the case of FIG. 16, in the original image 7 and the original image 8 placed on the same plane, a plurality of pixels included in the original image 7 correspond to the set A, and pixels included in the original image 8 correspond to the set B.

  Note that the Hausdorff distance does not have to be necessarily adopted, and a point at a similar position in the closed region may be adopted as a corresponding intersection point. Alternatively, points existing on similar patterns may be set as corresponding intersection points by using a method such as pattern recognition.

  FIG. 17 is a diagram showing an example of processing of an intersection when one of the two original images has a closed circuit isolated. The original picture 9 has an isolated closed circuit and a straight line. In the original image 10A, the straight line and the circle are in contact with each other, and an intersection point P101 and an intersection point P102 exist.

  In this case, the intersection of the original image 9 and the original image 10A can not be associated. Therefore, it is desirable to do as follows.

  That is, as shown in the original image 10B, the intersection point P101 and the intersection point P102 are deleted. Then, the straight line connected to the intersection point P101 and the straight line connected to the intersection point P102 are extended into one straight line. Furthermore, the end point of the circle connected to the intersection point P101 and the end point of the circle connected to the intersection point P102 are connected to make one circular figure. Next, new points of intersection are provided in the original image 9 and the original image 10B using the method shown in FIG. For example, an intersection point P093 and an intersection point P103 are provided. In this case, when two circular figures are placed on a plane as described with reference to FIG. 16, the Hausdorff distance between these two circles may be used to provide an intersection point. .

  FIG. 18 is a diagram showing an example in which an original image is divided into layers and processing is performed. In the original picture 11, an upper portion 11A of the eyebrow and a lower portion 11B of the eyebrow are present. And, in the original drawing 12, the upper part 12A of the brow and the lower portion 12b of the brow are present. In the periphery of the image of such a portion of the eyebrow, there may be cases where the correspondence of the intersection points in the two original images can not simply be made. That is, even by the process of associating the intersections illustrated with reference to FIGS. 7 and 9, there may be a closed region including the intersection where the association can not be performed. For such a closed area, it is effective to cut out the closed area, move it to another layer (layer), and separately process the correspondence of intersection points for each layer.

  That is, the eyelid portions of the original image 11 and the original image 12 are copied to different layers, and the image portion of the eyelid is deleted from the original original image. When this operation is performed, the original image can be divided into the layer of the human part and the layer of the part of the eyebrow, and the process of associating the intersection can be performed. As a method of determining whether or not to divide into layers, it is effective to move a closed region where many patterns in FIG. 9 occur around to another layer.

  It is desirable to move some closed regions to another layer and delete the image transferred to another layer from the original image. At this time, the periphery of the deleted portion may be blank, and a portion in which a closed region does not exist may occur. That is, there may be a case where there is no closed area and a branch line in the form of an open closed area in the blank part of the remaining image after moving a part of the closed area to another layer. In this case, the branch line may be left as it is, the end point (end point) of the branch line may be specified, and the end points of one original may be associated with the other original. Alternatively, the operator may be prompted to give an instruction if there remains an intersection where the association can not be performed.

Alternatively, in the above case, it is possible to form a closed region by extending the line segments existing around the portion from which the closed region has been deleted and connecting them together. When the original image is divided into layers and the intersections are associated, it may be necessary to perform the process of associating the newly generated closed area. In this case, it can often be dealt with by combining the closed regions that existed prior to transfer to the layer. In addition, when it is not possible to cope with it, it is also effective to partially re-associate the closed area again.
<3> Third Embodiment: Modified Example of Formation of Intermediately Divided Image As described above, by sequentially correlating the closed regions of the two original images and correlating the intersection points, correlating the respective portions of the two original images. Can do

  The split image can be performed by interpolating the position of the intersection of the two original images, but taking into consideration the correspondence of the closed region, the shape of the boundary of the closed region, the correspondence of the boundary connecting the corresponding intersections, etc. Therefore, it is advantageous to create a more natural split image by performing interpolation of the original image comprehensively.

  Further, by prompting the operator to instruct coloring when matching the image of the closed area, it is possible to urge the operator to collectively perform the process of coloring and the confirmation of the closed area.

  For example, the closed regions of the original image 1 and the original image 2 may be associated with each other, and the closed regions of the original image 2 and the original image 3 may be subsequently associated with each other. In this case, if the operator gives an instruction for coloring when the closed regions of the original image 1 and the original image 2 are associated, coloring when the closed regions of the original image 2 and the original image 3 are associated can be substantially automatically performed. It becomes. Moreover, since the coloring of the middle divided image can be easily estimated from the coloring of the original image, the coloring of the middle divided image can also be performed substantially automatically.

<4> Fourth Embodiment (Modified Example of Correlating Closed Areas)
An example of processing in the case where two closed regions of one original correspond to one closed region in the other original will be described below using FIG.

  In FIG. 18, the closed region 110 and the closed region 111 in the original image 11 correspond to the closed region 121 in the original image 12. In this case, the correspondence between closed regions is in a many-to-one format. Such a situation is likely to occur in the case of an original image in which many objects overlap in front of the closed region. In order to cope with such a case, for example, the following process may be performed.

  First, the closed region of the original image 12 corresponding to the closed region 110 of the original image 11 is estimated in the first embodiment described above. By this estimation, 121 is estimated as a closed region corresponding to the closed region 110. At this point, the corresponding closed area 110 and the closed area 121 are displayed on the screen to prompt the operator to confirm. The operator finds that the closed area 111 of the original image 11 remains as the association of the closed areas. For this reason, the estimation inputs that it is partially correct. In this input, the operator may instruct the region 110 of the original image 11 to be excluded from the targets of the subsequent closed region and the closed region 121 not to be excluded from the subsequent target of the closed region. This instruction can be realized by the processing of the flag in the column of table correspondence shown in FIG. That is, the flag of the correspondence of the area 110 is set to 1, and it is excluded from the target of the subsequent correspondence. On the other hand, in the field of the association in the area 121, the flag may be left as 0 and be taken into consideration in the subsequent association. At this time, the operator may also give instructions for coloring. Although the closed area 110 and the closed area 121 are colored in the same color by the operator's instruction of coloring, the closed area 121 can hold the state as the estimation target of the closed area.

  Thereafter, in the process of associating the closed area 111 of the original image 11, the closed area 121 of the original image 12 is estimated to correspond again, so that the association is displayed, and the operator is notified. When prompted for an instruction, the operator inputs that the correspondence is correct. According to the instruction of the operator, the flag of the correspondence of the closed area 111 and the flag of the correspondence of the closed area 121 shown in FIG. 3 are both replaced with 0 to 1, and these closed areas are the subsequent closed areas. It will be excluded from the target of the process of estimation of the association.

  Note that if the association column in FIG. 3 is not used, it is possible to allow one-to-many, many-to-one, many-to-many closed regions to be associated as closed regions. Also in this case, the operator may be prompted to input an instruction as to whether the association is correct.

<5> Hardware Configuration FIG. 19 is a diagram showing a hardware configuration 1700 of each embodiment.

  The hardware configuration 1700 includes a CPU 1710, a memory 1720, a communication control unit 1730, an input / output interface 1740, a display control unit 1750, a drive control unit 1760, and a print control unit 1770.

  The communication control unit 1730 is connected to a network such as the Internet. An input / output device 1742 is connected to the input / output interface 1740. A display device 1752 is connected to the display control unit 1750. The drive control unit 1760 can read and write the storage medium 1762. A printer 1772 is connected to the print control unit 1770.

  The storage medium 1762 may be a RAM, a ROM, a CD-ROM, a DVD-ROM, a hard disk, a memory card, or the like.

  The method of the embodiment described above can be implemented by a computer. Also, the method of the embodiment may be implemented as a program to be executed by a computer. Part or the front of the program may be executed by the operating system. Also, part of the program may be realized by hardware. The program may be stored in storage medium 1762.

  In the above embodiment, the method steps or program steps may be performed simultaneously or in reverse order, as long as no contradiction arises.

  The above embodiments can be implemented as hardware devices.

  It goes without saying that the above embodiments do not limit the invention described in the claims, and are treated as examples.

  The closed region and the background included in the original image and the split image are an example of the region. The original image, the split image, and the background are examples of elements constituting the image. The boundary can be an element that constitutes a part of the outline of the closed region.

<6> Fifth Embodiment
The fifth embodiment shows an example of an area interpolation method in the case where the area involves rotation.

  In the fifth embodiment, which will be described in detail below, the word “region” is used because the image to be processed does not necessarily have to be a closed region bounded by an outline. For example, the "region" may be a set of "closed regions", a part of "closed regions", or a region not bound by an outline. In the following embodiments, the "area" may be expressed as an "image".

  FIG. 20 is a diagram illustrating an example of generating an indivisible image using interpolation when the area involves rotation. The image of each plastic bottle is shown by a set of a plurality of areas. The following description will focus on a plastic bottle, which is an assembly of a plurality of areas. The following description also applies to each of a plurality of areas included in the plastic bottle.

  FIG. 20A shows an example of generating the middle division image 2002 at time tB when the image of the plastic bottle 2001 at time tA and the image 2003 of the plastic bottle at time tC are given. In FIG. 20A, the middle-cut image 2002 is formed in an appropriate size, and an example of an ideal middle-cut image is shown.

FIG. 20 (B) shows an image 2011 at time tA and an intermediate image 2012 in the case of using a method of linearly interpolating corresponding points in the image 2013 at time tC. The position of the cap of the plastic bottle is connected by a straight line 2022, and the cap of the plastic bottle 2012 of the split image is drawn at the midpoint between the position of the cap of the plastic bottle 2011 image and the position of the cap of the plastic bottle 2013 image. Similarly, for the bottom of the plastic bottle, a straight line 2021 is used to draw the bottom of the plastic bottle 2012 of the split image.
When the simple linear interpolation as described above is applied to a rotating image (region), there arises a disadvantage that the middle divided image (region) is drawn unnaturally small.

The present embodiment shows an example of a method for generating a split image in an image accompanied by such rotation into a natural image (as in the split image 2002 in FIG. 20A). The details will be described in detail below.
FIG. 21 is a diagram conceptually showing an example of generating an area of the middle division image.

  In FIG. 21A, the operator designates that the centers of rotation of the plastic bottle 2101 and the plastic bottle 2103 are the center of rotation Oa and the center of rotation Oc, respectively. The vector Va is a vector indicating the direction of the plastic bottle 2101, and the vector Vc1 is a vector indicating the direction of the plastic bottle 2103. These vectors can also be recognized by a machine such as a computer by recognizing the feature points Fa and Fc by image recognition, given the center of rotation. Alternatively, the vector gravity center Ga of the plastic bottle 2101 and the gravity center Gc of the plastic bottle 2103 are determined instead of the vector directed to the characteristic point of the plastic bottle, and the vector directed from the rotation center Oa and center Oc to the gravity center Ga and Gc The orientation (area orientation) of each plastic bottle may be recognized by respectively determining. Alternatively, the orientation of each plastic bottle may be given by the instruction of the operator. By this operation, the direction of rotation with time and the angle θ can be obtained.

  In any case, the angle θ shown in FIG. 210A can be obtained by recognizing the vector Va indicating the direction of the plastic bottle and the vector Vc1.

  FIG. 21B uses the recognized angle θ to rotate the plastic bottle 2013 in the opposite direction (counterclockwise) by the angle θ as shown in the plastic bottle 2103 c. By this operation, the directions (the directions of the regions) of the plastic bottle 2101 and the plastic bottle 2103 c coincide with each other. The vector Vc2 of the plastic bottle 2103c turns in the same direction as the vector Va. As described above, by matching the directions of the plastic bottle 2101 and the plastic bottle 2103 c, a state in which the influence of the rotation of the plastic bottle is removed can be obtained. Note that the rotation of the plastic bottle may be performed not on the plastic bottle 2103 but on the plastic bottle 2101. In this case, the direction of rotation of the plastic bottle 2101 may be rotated clockwise by an angle θ. Also by this operation, the direction of the plastic bottle can be adjusted. Alternatively, both of the plastic bottle 2101 and the plastic bottle 2103 may be rotated so that the directions of the both coincide with each other. The operation in this case will be described later with reference to FIG.

  FIG. 21C is a diagram showing that a plastic bottle 2102 b of an image obtained by interpolating the plastic bottle 2101 and the plastic bottle 2103 c is generated. In this case, in order to simplify the description, the plastic bottle 2102b generated by interpolation has the same shape as the plastic bottle 2101 and the plastic bottle 2103c, but the interpolated image has the original two Since the images are interpolated, it is customary for the two given images to be separate images.

  In the interpolated plastic bottle 2102b, the center of rotation Ob and the feature point Fb are specified by interpolation. And vector Vb1 which goes to feature point Fb from center Ob obtained by interpolation is specified. The interpolation is generally performed by linear interpolation based on the displayed time of the middle divided image, but is not limited to linear interpolation. Given corresponding pairs of points in two given images, based on the displayed temporal position of the split image, in the split image, the points corresponding to these pairs are displayed. It is identified.

FIG. 21D illustrates an example in which a middle-cut image is formed by rotating the interpolated image 2102b by an angle φ (clockwise) in an angle return direction according to the temporal position of the middle-cut image. FIG. In order to rotate the plastic bottle 2102b, for example, the plastic bottle 2102d may be obtained such that the vector Vb1 extending from the center of rotation Ob is rotated by φ and the vector Vb2 is obtained. Assuming that the time of the image 2101 is tA, the time of the image 2103 is tC, and the time of the middle divided image 2102d is tB, the angle φ is
φ = θ × (tB−tA) / (tC−tA)
It can be determined by

When the above processing is performed, the plastic bottle 2102 d of the middle-cut image in FIG. 21D becomes a natural middle-cut image than the plastic bottle 2012 of the middle-cut image shown in FIG. 20 (B). Although the example of the plastic bottle containing a some area | region was shown in the above description, it is as having stated already that the said process may be given with respect to each of the some area | region contained in a plastic bottle.
FIG. 22 is a diagram conceptually showing another example of generating a region included in the middle-cut image.

  In FIG. 22A, a region 2251 of a known image displayed at time tA (not shown) and a region 2253 of a known image displayed at time tC (not shown) are shown.

In the area 2251, there is a vector Va1 directed from the central coordinates Oa1 of the rotation of the area 2251 to the feature point Fa1. Then, in the area 2253, there is a vector Vc1 directed to the feature point Fc1 from the central coordinate Oc1 of the rotation of the area 2253. The rotation center coordinates Oa1 and Oc1 are generally given by the operator, but the shapes of the given two regions 2251 and 2253 are similar, and in consideration of the positional relationship of the surrounding image, The center coordinates of the rotation may be estimated by pattern recognition. It should be noted that the center coordinates of the rotation do not necessarily exist in the area.
Hereinafter, another example will be described in which a corresponding area in the middle-cut image at time tB (not shown) is determined.

  As shown in FIG. 22B, for example, by setting the vector Va1 and the vector Vc1 vertically and in the upward direction, the directions of the region 2251 and the region 2253 are made to coincide, and the regions 2261 and 2263 are obtained, respectively. There is. Note that such upward vertical direction is an example, and the directions of the two vectors may be made to match in the other direction. Here, the vector Va1 is rotated clockwise by δ1 to obtain the vector Va2. Further, Vc2 is obtained by rotating the vector Vc1 counterclockwise by δ2.

  FIG. 22C shows that the area 2262 included in the middle-cut image at time tB (not shown) is obtained using the area 2261 and the area 2263 in which the directions are matched. Region 2262 can be generated using linear interpolation based on time tB. For example, feature points Fb2 in a split image corresponding to feature points Fa and feature points Fc can be obtained by linear interpolation of feature points Fa and corresponding feature points Fc. Other feature points in the half-cut image, the shape of the closed region, vectors, etc. can be similarly generated by linear interpolation. Alternatively, the vector Vb2 may be created in parallel with the vector Va2 or the vector Vc2 without using the feature point Fb2. The vector Vb2 corresponds to the vector Va2 and the vector Vc2, and characterizes the direction of the area 2262.

  FIG. 22D shows a process of obtaining a final area 2272 in the middle-cut image. That is, by rotating the vector Vb2 counterclockwise by the angle γ1, the vector Vb3 is obtained. In addition, feature points Fb3 corresponding to the feature points Fa1 and Fc1 can be obtained. Other feature points, the shape of the closed region, and the like can also be acquired as the vector Vb3 rotates. The area 2272 obtained in this manner is an area corresponding to the area 2251 and the area 2253, which is included in the middle division image at time tB.

FIG. 22D shows the relationship of the rotation angles of the vectors. Assuming that the angle between the vector Va1 and the vector Vc1 is θ1, the angle φ1 between the vector Va1 and the vector Vb3 can be obtained by the following calculation.
φ1 = θ1 × (tB−tA) / (tC−tA)
The angle γ1 between the vector Vb2 and the vector Vb3 can be obtained by the following calculation.
γ1 = δ1-φ1 or γ1 = θ1-φ1-δ2
However, θ1 = δ1 + δ2

FIG. 23 is a diagram showing an example of a procedure for generating the area of the middle division image. A procedure for applying the present embodiment to an area including rotation will be described below.
[Step S2200] Based on the instruction of the operator, two original images for creating the middle-division image are acquired. Next, the process proceeds to step S2202.

[Step S2202] From each of the two images, the rotating area is arranged in a layer different from the area of the other part of the image based on the instruction of the operator. This step S2202 is performed to separate the area to be rotated from the other parts of the image for processing. The rotating area is arranged in an appropriate layer in consideration of the order of overlapping of the areas included in the image. In this case, the order of the layers of the corresponding areas of the two images may be aligned or the layers may correspond to each other so that the layers for arranging the corresponding areas of the two images are associated with each other. Is desirable. The method of obtaining the corresponding area pair is as described above. Of course, the corresponding area may be identified by the operator.

  Note that the division into layers as described above is an example of separately processing a rotating area and a part of another image. Therefore, in order to be able to process the rotating area separately from other image parts, another method is adopted in which operations such as grouping are performed so that the rotating area can be distinguished from other image areas. It goes without saying that it is acceptable. Next, the process proceeds to step S2204.

[Step S2204] The position of the center of rotation of the area to be rotated is specified based on the instruction of the operator.

  The corresponding regions in the regions of the two images are, for example, mostly regions of parts of the same object depicted in the images. In this case, with respect to the center of rotation, in most cases, the relative position with respect to the object does not change, but the object itself may be moving or the viewpoint may be moving. The central coordinates of the rotation of the area to be Therefore, it is desirable to acquire the center coordinates of each of the two corresponding areas by the instruction of the operator.

  There may be an embodiment in which the center coordinates Oc of one of the two corresponding regions are not acquired from the operator. For example, if FIG. 21 is taken as an example, if the area 2101 and the area 2103 have a sufficiently similar shape, the center coordinates Oc can be determined by the program based on the center coordinates Oa. Alternatively, for example, when the area 2101 and the area 2103 do not have similar shapes, the user designates a set (one set or a plurality of sets) of corresponding points of the area 2101 and the area 2103 to search the user, and a program based on the corresponding points The central coordinates Oc can also be determined by Alternatively, the center of rotation of one region may be determined based on the center of rotation of one of the regions by examining the positional relationship between the two regions using image recognition. It should be noted that the center of rotation of the region may be outside the region. Next, the process proceeds to step S2206.

[Step S2206] Based on the instruction of the operator, specify the angle of rotation θ and the direction of rotation, or from the positional relationship between the center of gravity of the rotating area or the characteristic position and the center of rotation, Identify.

  The rotation angle and rotation direction of the vector may be determined by, for example, determining a vector from the center of rotation of two corresponding regions to the position of the area gravity center of the region. The area gravity center is an example, and not the area gravity center, but the corresponding feature points present in two corresponding areas are recognized, and the vector from the center of rotation to the feature points is obtained in each of the two areas. The two obtained vectors may be used. Next, the process proceeds to step S2208.

[Step S2208] By rotating one or both of the two rotating areas by an appropriate angle, the two rotating areas are aligned. An appropriate angle may be calculated so that the orientations of the two regions coincide.

  For example, in the case where there is an image A displayed at time tA and an image C displayed at subsequent time tC, a case where the region rotates clockwise by θ by time is taken as an example. Around the center of rotation of the area of image C, rotate the area of image C by -θ. By this rotation, the directions of the two regions coincide. Note that, contrary to the above example, the area of the image A may be rotated by θ, and the directions of both areas may be aligned. Next, the process proceeds to step S2210.

[Step S2210] The two regions whose directions coincide are interpolated to generate a region of the m-th middle-division image. In step S2208, since the direction of the area is matched, for example, even if linear interpolation is used, processing of interpolation of the area of the m-th middle-division image can be performed without being affected by the rotation of the area. Next, the process proceeds to step S2212.

[Step S2212] Since the area interpolated at Step S2210 excludes the influence of rotation, at Step S2212, an operation to return the rotation by an appropriate amount is performed. The rotation angle φ, which returns the rotation, is calculated according to the following equation.
φ = θ × m / (n + 1)
However, n is the number of middle division images

  Further, when both of the two rotating regions are rotated to align the orientations of the two regions to form an interpolated region, the interpolated region is obtained by the operation using the above-mentioned rotation angle φ. The interpolated area may be directed so as to turn in the same direction as the direction.

  As the area of the known image at both ends from which the split image is formed, when the final moving image is to be generated, the image given first may be used, and the region of the split image is calculated. Because of this, the operation to reverse the rotation is unnecessary.

For example, when the image C is rotated by -θ, the area of the generated split image is rotated by φ. By this operation, the interpolated region included in the m-th middle divided image among the n middle divided images is rotated to an appropriate angle. When the image A is rotated by θ, the interpolated middle divided image is rotated by −θ + φ.
The above is an example of interpolation processing of the middle division image including rotation.

  FIG. 24 is a diagram showing an example of correcting the position of a region when two regions are connected by a joint. A joint refers to a connection point where two regions are connected at the joint and each of the two regions can rotate about the joint.

  The case where there are two rotating areas in each image will be described as an example. Even when there are three or more such regions in each image, similar processing can be performed by sequentially combining the two existing examples.

  Since the joint points of the regions do not separate from each other, processing for matching the positions of the joint points is performed according to the following procedure. The procedure will be described below with reference to FIG. 24, but a detailed specific example will be described with reference to FIG.

[Step S2302] The regions including the rotation are arranged in different layers. This operation is performed to operate each area separately. That is, the center of rotation and the angle of rotation are different for each area. It is not necessary to use layers, as long as each area can be operated separately. Next, the process proceeds to step S2304.

[Step S2304] The order of layers in layers and joint points O are specified. The joint point O may be designated by the operator. In general, the layer hierarchy is often, but not limited to, a user interface in which a layer to be displayed in front of an image is positioned on top. In addition, since a joint point has a function to connect two areas at a joint point, for example, a joint point is specified in each of the two areas, but in an image, these two joint points appear to overlap It becomes. Next, the process proceeds to step S2306. Correspondence of the area corresponding to the corresponding area existing in each of the two images or the correspondence between the layers in the layer hierarchy of the two images is set for the corresponding area among the layers of the two images. It is desirable to understand the relationship.
[Step S2306] An interpolated area for each hierarchy is created, and a joint point Omi after interpolation is obtained. (M: index of interpolation image, i: layer number)

  In this step S2306, if the order of the hierarchy is the same, it is assumed that the area has a correspondence. Each layer is subjected to the interpolation processing of the area shown in FIG. 23 described above. Next, the process proceeds to step S2308.

[Step S2308] The difference v = Omi−Omj of the positions of the joint points in the hierarchy i and the hierarchy j is obtained. In this case, it is assumed that the area of hierarchy i and the area of hierarchy j are connected at joint point O. The joint point of the interpolated area in the hierarchy i is represented by Omi. The joint point of the interpolated region in the hierarchy j is represented by Omj. Even in the interpolated image, the joint points should originally match, but since the interpolation process is performed for each layer hierarchy, the positions Omi and Omj of the joint points in the interpolated region match. May not. The processing proceeds to step S2310.

[Step S2310] The interpolated region of the layer j is translated by v. If the positions Omi and Omj of the joint points in the interpolated area do not match, the area of the hierarchy j is moved by the difference V = Omi−Omj to make the joint positions Omi and Omj coincide.

  Note that it is desirable to arrange which interpolated region is moved so that the entire image does not become unnatural. For example, in the case of humans, it is often natural to fix the bone joint closest to the body and move the bone joint away from the body so that both joints coincide. If there are two or more joints, the joints in the interpolated area may be made to coincide in order from the joint closest to the body.

  As another example, in the animation of walking of a person, the foot in contact with the ground is fixed and the joints of the interpolated regions are matched in the order of "foot, shin, thigh, hips" in reverse to the above example. You should do it.

  FIG. 25 is a diagram showing an example of two images in which two regions are jointed. A time axis is set from left to right, and at time tA, the user swings up the arm 2420 and tries to strike the drum 2400. At time tC, the arm 2440 beats the drum 2400.

  The state of the arm 2420 at time tA will be described below. First, in the upper arm 2424, the center of rotation P2a is present and is directed in the direction of the vector V2a. The forearm 2422 has a center of rotation P1a, which is directed in the direction of the vector V1a.

  The state of the arm 2440 at time tC will be described below. First, in the upper arm 2444, the center of rotation P2b is present and is directed in the direction of the vector V2b. The forearm 2442 has a center of rotation P1b and points in the direction of the vector V1b.

  Therefore, the upper arm 2424 at time tA rotates clockwise by θ2 until reaching the upper arm 2444 at time tC. The forearm 2422 at time tA rotates clockwise by θ1 until it reaches the forearm 2442 at time tC.

FIG. 26 is a diagram showing an example of dividing a region into different layers of layers. The arm 2520 at time tA is divided into two regions of the upper arm 2522 and the forearm 2524. Then place each one on another layer layer. Although layers are not illustrated, it is sufficient if two areas can be operated separately, and it is not necessary to necessarily use a user interface called a layer hierarchy.
The arm 2540 at time tC is divided into two regions of an upper arm 2542 and a forearm 2544. Then place each one in the hierarchy of another layer.

  FIG. 27 is a diagram showing the shift of the area in the middle-cut image when the middle-cut image is formed from each of the two layers. The arm 2620 at time tA and the arm 2640 at time tC are divided into different layers of layers as shown in FIG. 26, and the corresponding areas are interpolated to create the arm 2630 at time tB. The arm 2630 is a combination of the upper arm 2632 and the forearm 2634, which are interpolated regions, but it can be seen that a shift has occurred at the elbow. The cause of this deviation is caused by dividing the layer into layers and interpolating the area of each corresponding layer.

  FIG. 28 is a diagram showing an example of eliminating the deviation of two regions in a half-cut image by giving joint points. In order to eliminate the deviation of the elbow portion generated in the synthesis of the interpolated regions in FIG. 27 as described above, the following processing is performed. The flow of this process is described in FIG.

  For the image of the arm 2720 at time tA, the operator designates the joint part of the upper arm 2724 and the forearm 2722. As shown, the articulated position 2725 of the upper arm and the articulated position 2723 of the forearm are shown. Since the coordinates of the joints coincide, the joint position 2725 of the upper arm 2724 and the joint position 2723 of the forearm 2722 are shown at the same position. The operator specifies the joint position of the elbow to obtain the joint position 2725 of the upper arm 2724 and the joint position 2723 of the forearm 2722.

  The operator designates the joint portion of the upper arm 2744 and the forearm 2742 with respect to the image of the arm 2740 at time tC. As shown, the articulation position 2745 of the upper arm 2744 and the articulation position 2743 of the forearm 2742 are shown. Since the coordinates of the joints coincide, the joint position 2745 of the upper arm 2744 and the joint position 2743 of the forearm 2742 are shown at the same position. The operator specifies the joint position of the elbow to obtain the joint position 2745 of the upper arm 2744 and the joint position 2743 of the forearm 2742.

  Interpolation is applied to each layer layer to obtain an interpolated image 2730a. In this case, the joint position 2735a of the upper arm 2734a and the joint position 2733a of the forearm 2732a are deviated. In order to eliminate this deviation, the forearm 2732a is translated in the direction of the vector 2738.

The result is shown on arm 2730b. The positions of the joint position 2735b of the upper arm 2734b and the joint position 2733b of the forearm 2732b coincide with each other. In this case, it is desirable to translate the forearm 2732b further away from the body.
The joint position was matched by the above processing. The upper arm 2734 b and the forearm 2732 b, which are interpolated regions, have a desirable connection without deviation.
The above processing can be performed by sequentially processing in the same manner also in the case of three or more areas.
Note that the feature points and the joint points may exist outside the area.
A feature point is an example of an arbitrary point. The joint position and the coordinates of the joint are examples of joint points.

  In the claims, main elements are denoted by reference numerals. It should be noted that this is a code to help the understanding of the claims and does not necessarily reflect the code of the drawings exactly. Further, reference numerals in the claims do not limit the scope of the invention and the technical scope of the invention described in the claims to the contents of the description of the drawings.

Lab boundary line Lac boundary line Lbc boundary line Lcd boundary line Ldb boundary line Lde boundary line P12 deletion point intersection point P111 intersection point set at the position of P11 intersection point set at the position of P112 P11

Anyway, by the vector Va and vector Vc1 indicating the bottles orientation is recognized, the angle θ is obtained as shown in FIG. 2 1 (A).

Claims (7)

An image displayed at time tB between the time tA and the time tC from the known image A and the known image C displayed at the time tC after the time tA when the image A is displayed A method of causing a computer to execute a first estimation procedure on an area a included in the image A and an area c included in the image C and having a correspondence with the area a so as to generate B. And
The first estimation procedure is
The correspondence includes the rotation of the region about the axis in the direction substantially perpendicular to the image, and obtains the central coordinates Oa of the rotation of the region a and the central coordinates Oc of the rotation of the region c. Acquisition step,
A rotation recognition step of recognizing the direction of the rotation and the angle of the rotation;
Interpolate the shape of the area a and the shape of the area c based on the time tA, the time tB, the time tC, the center coordinates Oa, the center coordinates Oc, the direction of rotation and the angle of rotation An area forming step of forming a shape of an area b which is an area included in the image B and is an area corresponding to the area a;
How to have it. The area forming step is
Depending on the direction of rotation and the angle of rotation, one or both of rotation of the region a about the central coordinate Oa and rotation of the region c about the central coordinate Oc are performed, Forming an area a1 and an area c1 by respectively aligning the directions of the area a and the area c;
An interpolation step of forming a shape of a region x interpolated using the shape of the region a1 and the shape of the region c1 based on the time tA, the time tB and the time tC;
Based on the time tA, the time tB and the time tC
Centering on center coordinates Ob in the image B obtained by interpolation from the center coordinates Oa and the center coordinates Oc,
The angle obtained by dividing the angle of rotation is
By rotating so that the interpolated region x points,
Acquiring the shape of the area b;
The method of claim 1 comprising The rotation recognition step is
Determining a center of gravity Ga of the area a and a center of gravity Gc of the area c;
Estimating the direction of rotation and the angle of rotation from the vector from the center coordinate Oa toward the center of gravity Ga and the vector from the center coordinate Oc toward the center of gravity Gc;
The method according to claim 1 or 2, comprising The rotation recognition step is
Acquiring from the operator a feature point Fa of the area a, and a feature point Fc existing in the area c and corresponding to the feature point Fa;
Estimating the direction of rotation and the angle of rotation from the vector from the central coordinate Oa toward the feature point Fa and the vector from the central coordinate Oc toward the feature point Fc;
The method according to claim 1 or 2, comprising The rotation recognition step is
The method according to claim 1 or 2, comprising the steps of: obtaining from the operator the direction of the rotation and the angle of the rotation. A known region p connected to the region a at the joint point Ja in the image A, and the joint point Ja being a central coordinate of rotation, and a joint point Jc to the region c in the image C, the joint point If there is a known region r where J c is the center of rotation coordinates
An area q interpolated in the image B is further formed by further applying the area p, the area r, and the coordinates of each of the joint point Ja and the joint point Jc to the first estimation procedure, and Identifying a joint point Jb corresponding to the joint point Ja and the joint point Jc in the area b, and specifying a joint point Jq corresponding to the joint point Ja and the joint point Jc in the area q;
If the coordinates of the joint point Jb and the coordinates of the joint point Jq do not match, either the area b or the area q so that the coordinates of the joint point Jb coincide with the coordinates of the joint point Jq Translating one or both of the steps;
The method according to any one of claims 1 to 5, further causing the computer to execute a second estimation procedure including. A program that causes a computer to execute the method according to any one of claims 1 to 6.

Images