JP2013183874A - Image processing device, program, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, a program, and an image processing method capable of preventing the generation of a region which is not observed due to deletion of a part of images or controlling the frequency of generation when an image summarization process is carried out by deleting a part of the images.SOLUTION: An image processing device includes: an image sequence acquisition unit 200 for acquiring an image sequence having a plurality of images; and a processing unit 100 for carrying out an image summarization process for deleting a portion of the plurality of images in the acquired image sequence to acquire a summarized image sequence. The processing unit 100 selects a reference image and an image to be assessed from the plurality of images, computes, in accordance with the reference image, the coverage ratio of the image to be assessed, on the basis of transform information between the reference image and the image to be assessed, and assesses, on the basis of the coverage ratio, whether the image to be assessed can be deleted.

Description

  The present invention relates to an image processing apparatus, a program, an image processing method, and the like.

  When still image capturing is continued in time series at a given time interval, when a subject having a spatial spread is covered by a large number of images, or after each moving image is captured, each moving image is configured. When an image is captured as a still image, a very large amount of images (hereinafter also referred to as image sequences) that are temporally or spatially continuous are acquired. In such a case, there is a high possibility that images that are close in the image sequence (that is, close in time or space) are similar images. The need to check everything is not high. In the first place, it is not uncommon for the number of images to be tens of thousands or more, and checking everything with the user's hand itself is a heavy burden.

  Therefore, there is a demand for deleting a part of the image from the image sequence and summarizing it into an image sequence having a smaller number than the original image sequence (hereinafter, this process is referred to as an image summarization process). For example, Patent Document 1 discloses an image summarization processing method that leaves an image that makes it easy to grasp the contents of an image sequence by extracting an image of a boundary where the scene in the image sequence changes or an image representative of the image sequence. Has been.

JP 2009-5020 A JP 2011-24763 A

  For example, when an image summarization technique is applied to the medical field, it is necessary to suppress the occurrence of an area that cannot be observed by deleting an image from the viewpoint of avoiding oversight of a disease.

  However, if only the image of the boundary where the scene changes is left as in the method of Patent Document 1 or if image summarization is performed from the viewpoint of whether the image sequence after summarization is intuitively easy to see, the image is deleted. This may cause an area that cannot be observed, which is not preferable. Further, since the degree of occurrence of the region that cannot be observed depends on the contents of the image, it is difficult to control the degree of oversight of the disease by the conventional image summarization processing method.

  According to some aspects of the present invention, when image summarization processing is performed by deleting a part of an image, an image that suppresses the generation of an area that cannot be observed due to the deletion of the image and controls the generation degree. A processing device, a program, an image processing method, and the like can be provided.

  According to an aspect of the present invention, an image sequence acquisition unit that acquires an image sequence having a plurality of images, and a summary image sequence by deleting a part of the plurality of images of the image sequence acquired by the image sequence acquisition unit. A processing unit that performs an image summarization process to be acquired, wherein the processing unit selects a reference image and a determination target image from the plurality of images, and is based on deformation information between the reference image and the determination target image. Thus, the present invention relates to an image processing apparatus that calculates a coverage of the determination target image by the reference image and determines whether or not the determination target image can be deleted based on the coverage.

  In one aspect of the present invention, a reference image and a determination target image are selected from the acquired image sequence, and the coverage of the determination target image by the reference image is calculated based on deformation information between the selected reference image and the determination target image. Then, it is determined whether the determination target image can be deleted. Therefore, since it is possible to determine whether or not the image to be determined is covered by a reference image (in a narrow sense, an image to be left in the summary image sequence), it is possible to determine whether or not the image can be observed by deleting the image. It is possible to deter doing this.

  In one aspect of the present invention, when the first to Nth images (N is an integer of 2 or more) are input as an input image sequence, the processing unit is configured to output the pth (p is 1 ≦ p ≦ N). Image) is selected as the first reference image, the qth image (q is an integer equal to or greater than p + 2) is selected as the second reference image, and the rth image (where r is p + 1 ≦ r ≦ q−). 1) is selected as the determination target image, the deformation information between the first reference image and the determination target image, and the second reference image and the determination target image. The coverage may be calculated based on deformation information, and whether the determination target image is to be deleted may be determined based on the coverage.

  This makes it possible to set a reference image in front of and behind the determination target image.

  In one aspect of the present invention, the processing unit selects the second reference image from a second reference image selection section in which a start point and an end point corresponding to the p + 2 to Nth images are set, It is determined whether or not the determination target image can be deleted based on the first reference image and the second reference image, and when it is determined that the p + 1 to q−1th images can be deleted, the first reference image and the second reference image are deleted. The xth image (x is an integer satisfying x> q) included in the second reference image selection section is selected as a new second reference image, and the start point of the second reference image selection section is selected as the second reference image selection section. You may update to the qth image.

  As a result, it is possible to set the second reference image selection section, update the second reference image to the back image, update the start point of the selection section, and the like based on the determination result of the deletion possibility. Become.

  In the aspect of the invention, the processing unit may be based on a value of (q + j) / 2 when a j-th image (j is an integer) corresponds to the end point of the second reference image selection section. The value of x may be set.

  This makes it possible to apply a binary search method when updating the second reference image to the rear image.

  In one aspect of the present invention, the processing unit is included in the second reference image selection section when it is determined that at least one of the p + 1 to q-1th images cannot be deleted. The y-th image (y is an integer satisfying y <q) is selected as the new second reference image, and the end point of the second reference image selection section is updated to the q-th image. Good.

  As a result, the second reference image can be updated to the front image, and the end point of the selected section can be updated.

  In the aspect of the invention, the processing unit may be based on a value of (i + q) / 2 when an i-th image (i is an integer) corresponds to the start point of the second reference image selection section. The value of y may be set.

  This makes it possible to apply a binary search method when updating the second reference image to the front image.

  In the aspect of the invention, the processing unit may update the first reference image when the start point and the end point are adjacent to each other as a result of updating the start point or the end point of the second reference image selection section. One of the plurality of images selected as the image is included in the summary image sequence, and one of the plurality of images corresponding to the start point and the one corresponding to the start point among the plurality of images. A partial image sequence consisting of the rear one in the input image sequence rather than one of a plurality of images is set as the input image sequence, and the value of p is set to 1 for the set input image sequence The processing may be performed again.

  Accordingly, when the start point and the end point of the second reference image selection section are adjacent to each other, it is possible to end the processing for the input image sequence and set the partial image sequence.

  In the aspect of the invention, the processing unit may select the q + 1th image as the second reference image when it is determined that the p + 1st to q−1th images can be deleted. Good.

  As a result, it is possible to update the second reference image to an image one backward.

  In one aspect of the present invention, the processing unit is selected as the first reference image when it is determined that at least one of the p + 1 to q−1 images is not erasable. A process of including one of a plurality of images in the summary image sequence, a partial image sequence composed of the q-1st to Nth images is set as the input image sequence, and the set input image sequence On the other hand, the value of p may be set to 1 and the process may be performed again.

  As a result, the first reference image can be left in the summary image sequence, an appropriate partial image sequence can be set as a new input image sequence, and the first reference image and the second reference image can be updated. Therefore, more efficient image summarization processing becomes possible.

  In one aspect of the present invention, when the first to Nth images (N is an integer of 2 or more) are input as an input image sequence, the processing unit selects the first image as the reference image. In addition, a k-th image (k is an integer satisfying 2 ≦ k ≦ N−1) is selected as the determination target image, and the coverage is determined based on deformation information between the reference image and the determination target image. And determining whether or not the determination target image can be deleted based on the coverage ratio. If it is determined that the kth image can be deleted, the k + 1th image is selected as the determination target image. May be.

  As a result, it is possible to perform an image summarization process based on the coverage after setting a reference image in front of the determination target image.

  In one aspect of the present invention, when it is determined that the k-th image cannot be deleted, the processing unit selects one of the plurality of images selected as the reference image as the summary image sequence. The partial image sequence including the kth to Nth images may be set as the input image sequence, and the set input image sequence may be processed again.

  Thereby, since a reference image can be newly set, it is possible to perform image summarization processing based on the coverage.

  In one aspect of the present invention, when the first to Nth images (N is an integer equal to or greater than 2) are input as an input image sequence, the processing unit performs the sth (s is 2 ≦ s ≦ N). -1) is selected as the reference image, and a t-th image (t is an integer satisfying 1 ≦ t ≦ N and t ≠ s) is selected as the determination target image. The coverage may be calculated based on deformation information between the determination target images, and whether or not the determination target image can be deleted may be determined based on the coverage.

  As a result, it is possible to select a determination target image from both the front and the rear with respect to the reference image and perform a determination process of whether or not deletion is possible.

  Further, in one aspect of the present invention, the processing unit determines whether the first to first s-1 images are determined to be undeletable in the determination of whether or not deletion is possible. A partial image sequence made up of the s-1th images may be set as the input image sequence, and the set input image sequence may be processed again.

  As a result, when the input image sequence is divided by the reference image, an image summarization based on the coverage is performed by a method of determining whether or not the front part of the reference image can be deleted and setting a new input image sequence as necessary. Processing can be performed.

  In the aspect of the invention, the processing unit may determine the s + 1 to Nth when the determination on whether or not deletion is possible determines that at least one of the s + 1 to Nth images cannot be deleted. A partial image sequence consisting of the images may be set as the input image sequence, and the set input image sequence may be processed again.

  As a result, when the input image sequence is divided by the reference image, an image summarization based on the coverage is performed by a method for determining whether or not a portion behind the reference image can be deleted and setting a new input image sequence as necessary. Processing can be performed.

  In the aspect of the invention, the processing unit is adjacent to the reference image, the determination target image, and the plurality of images that are between the reference image and the determination target image in the image sequence. The deformation information between the images may be obtained, and the deformation information between the reference image and the determination target image may be obtained based on the obtained deformation information between the obtained adjacent images.

  Thereby, the deformation information between the reference image and the determination target image can be calculated based on the deformation information between adjacent images.

  In one aspect of the present invention, the processing unit obtains a covered area that is an area where the determination target image is covered by the reference image, based on deformation information between the reference image and the determination target image. As the coverage, a ratio of the coverage area in the determination target image may be calculated.

  This makes it possible to calculate the coverage based on the coverage area.

  In the aspect of the invention, the processing unit selects first to Mth (M is an integer of 2 or more) reference images from the plurality of images as the reference image, and the uth (1 ≦ u) ≦ M) is obtained based on deformation information between the reference image and the determination target image, and the union area of the first to Mth covering areas is set as the covering area and set. The coverage may be calculated based on the coverage area.

  Accordingly, it is possible to calculate the coverage based on the coverage area after setting a plurality of reference images.

  In the aspect of the invention, the processing unit may set a weighting factor according to an on-image position of the determination target image, and calculate the first weight sum calculated based on the weighting factor and the covering region. The coverage may be calculated based on a ratio between the weighting coefficient and the second weight sum calculated based on the determination target image.

  Thereby, it is possible to calculate the coverage using the weighting factor.

  In one aspect of the present invention, the processing unit sets a plurality of points for the determination target image, and converts the plurality of points according to deformation information between the reference image and the determination target image. In this case, the coverage may be calculated based on the number of converted points included in the reference image.

  Thereby, it becomes possible to calculate a coverage based on a plurality of points.

  In one aspect of the present invention, the processing unit selects first to Mth reference images (M is an integer of 2 or more) from the plurality of images as the reference image, and sets the plurality of points to the first point. When conversion is performed according to deformation information between the reference image u (1 ≦ u ≦ M) and the determination target image, the number of the converted points included in the u-th reference image is set to the u-th cover. The coverage may be calculated based on the first to Mth covering information obtained from each of the first to Mth reference images.

  As a result, it is possible to calculate the coverage based on a plurality of points after setting a plurality of reference images.

  In the aspect of the invention, the processing unit may set a weighting factor according to an on-image position of the determination target image, and use the plurality of points as deformation information between the reference image and the determination target image. When conversion is performed accordingly, the coverage may be calculated based on the converted point included in the reference image and the weighting factor.

  Thereby, it is possible to calculate the coverage using the weighting factor.

  In the aspect of the invention, the processing unit sets 0 as a value of the weighting coefficient in a first region of the determination target image and is different from the first region of the determination target image. 1 may be set in the second area.

  Thereby, 0 or 1 can be set as the weighting coefficient.

  Another aspect of the present invention provides an image sequence acquisition unit that acquires an image sequence having a plurality of images, and a summary image sequence by deleting a part of the plurality of images of the image sequence acquired by the image sequence acquisition unit The computer functions as a processing unit that performs image summarization processing to acquire the image, and the processing unit selects a reference image and a determination target image from the plurality of images, and deforms between the reference image and the determination target image The present invention relates to a program that calculates a coverage of the determination target image based on the reference image based on the information and determines whether or not the determination target image can be deleted based on the coverage.

  According to another aspect of the present invention, an image sequence having a plurality of images is acquired, a reference image and a determination target image are selected from the plurality of images in the image sequence, and between the reference image and the determination target image Based on deformation information, the coverage of the determination target image by the reference image is calculated, the determination target image is determined to be deleted based on the coverage, and based on the determination result of the deletion permission, The present invention relates to an image processing method for performing image summarization processing for obtaining a summary image sequence by deleting a part of the plurality of images in the image sequence.

1 is a system configuration example of an image processing apparatus according to a first embodiment. The flowchart for demonstrating the process of 1st Embodiment. The figure explaining the coverage calculation process using a covering area | region. FIG. 4A to FIG. 4D are diagrams for explaining image summarization processing according to the first embodiment. The figure explaining the coverage calculation process using a some point. The flowchart for demonstrating the coverage calculation process using several points. The figure explaining the modification which selects a some reference | standard image. The figure explaining the process which calculates | requires a covering area | region based on a some reference | standard image. FIG. 9A and FIG. 9B are diagrams illustrating a modification example in which a plurality of reference images are selected. The figure explaining the coverage calculation process using weighting. 9 is a system configuration example of an image processing apparatus according to a second embodiment. The flowchart for demonstrating the process of 2nd Embodiment. FIGS. 13A and 13B are views for explaining image summarization processing according to the second embodiment. FIG. 10 is a diagram for explaining an example of a method for selecting a first reference image in the second embodiment. 9 is a system configuration example of an image processing apparatus according to a third embodiment. The flowchart for demonstrating the process of 3rd Embodiment. The other flowchart for demonstrating the process of 3rd Embodiment. The figure explaining the image summarization process of 3rd Embodiment. 1 is a system configuration example of image summarization processing according to the present embodiment. FIG. 20A to FIG. 20G are diagrams illustrating a modification of the second embodiment. The flowchart for demonstrating the process of the modification of 2nd Embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. First, the method of this embodiment will be described. When an image sequence composed of a large number of images that are temporally or spatially continuous is acquired, the user uses the image sequence to perform some processing (for example, medical practice such as diagnosis if an endoscopic image sequence). When performing, it is desirable to perform image summarization processing. This is because the number of images included in the image sequence is very large, and it is necessary for the user to make a determination after viewing all of the images. In addition, there is a high possibility that there are images that are similar to each other in the image sequence, and even if all such similar images are checked, the amount of information that can be obtained is limited and is not commensurate with the effort. .

  As a specific example, an image sequence captured using a capsule endoscope can be considered. The capsule endoscope is a capsule-shaped endoscope with a built-in small camera, and takes an image at a given time interval (for example, twice per second). Since the capsule endoscope requires several hours (in some cases, several tens of hours) from taking the clothes to discharging, tens of thousands of captured images are acquired in one examination by one user. In addition, the capsule endoscope stays in the same place or returns in the opposite direction due to the influence of the movement of the living body when moving in the living body. For this reason, in a large number of images, the same subject as that of other images is imaged, and there are many images that are not highly useful in finding lesions.

  In the conventional image summarization processing, an image at a boundary where the scene changes and an image representing an image sequence are extracted. However, with such a method, when deleting an image, the relationship between the subject captured in the image to be deleted and the subject captured in the remaining image is not particularly considered. For this reason, it is possible that the subject captured on the image included in the pre-summarization image sequence is not captured on any image included in the post-summary image sequence. In addition, since the degree to which a subject that is not included in any image in the image sequence is generated by the image summarization process depends on the image sequence to be processed, it is difficult to control the degree in the conventional method. .

  This is not preferable particularly in image summarization processing in the medical field. In the medical field, for the purpose, oversight of a notable area (for example, a lesion) must be suppressed as much as possible. For this purpose, it is desirable to image the widest possible range in the living body, and in the image summarization process, it should be prevented that a subject range that cannot be observed by deleting a given image is generated.

  Therefore, the present applicant performs an image summarization process based on the coverage ratio between the reference image (the image to be left, an image to be a candidate to be left depending on the embodiment) and the determination target image (the target image to determine whether to delete). Suggest a method. Specifically, as shown in FIG. 3, the covering area is calculated on the determination target image by deforming the reference image. The subject imaged with the reference image corresponds to the subject imaged on the covering area of the determination target image. That is, the range outside the covering region in the determination target image is a region that cannot be covered even if the reference image is left when the determination target image is deleted.

  Therefore, the ratio of the coverage area in the determination target image is calculated as the coverage ratio, and by determining whether to delete the determination target image based on the calculated coverage ratio, the degree of occurrence of the subject range that cannot be observed is determined. Control. For example, if the image to be judged is deleted when the coverage is equal to or higher than the threshold, and the image to be judged is not deleted when the coverage is less than the threshold, the degree of occurrence of an area that cannot be covered is controlled according to the threshold setting. it can. If the threshold value is increased, the degree of occurrence of an area that cannot be covered can be reduced, so that a deterrent effect such as oversight of a lesion is enhanced. Also, if the threshold value is reduced, an area that cannot be covered is likely to occur, but the number of images included in the summarized image sequence can be reduced to reduce the subsequent burden on the user.

  As an embodiment of the image processing apparatus here, an apparatus including a processing unit 100 and an image sequence acquisition unit 200 as shown in FIG. The image sequence acquisition unit 200 acquires an image sequence having a plurality of images. Then, the processing unit 100 performs image summarization processing that acquires a summary image sequence by deleting some of the plurality of images included in the image sequence acquired by the image sequence acquisition unit 200. Specifically, the processing unit 100 selects a reference image and a determination target image from a plurality of images, and covers the determination target image with the reference image based on deformation information between the selected reference image and the determination target image. Calculate the rate. Then, it is determined whether or not the determination target image can be deleted based on the calculated coverage.

  As a result, image summarization processing based on the coverage can be performed. When it is determined that the determination target image can be deleted by determining whether or not deletion is possible, it may be determined that the determination target image is deleted (first and second embodiments), or other images. It may be determined that the deletion is not confirmed until the determination result of whether or not deletion is possible when the image is determined as the determination target image (third embodiment).

  In the following, in the first embodiment, a method of performing image summarization processing for guaranteeing the coverage of the image behind the reference image in order of sequence will be described. Further, as a modification, a method of calculating the coverage using a plurality of points instead of the coverage region, a method of calculating the coverage using a plurality of reference images, and weighting according to the position on the image is used for obtaining the coverage. A method and a method for obtaining desired deformation information based on deformation information between adjacent images will also be described. Note that the modification described in the first embodiment may be applied to the second and third embodiments.

  In the second embodiment, a method will be described in which image summarization processing for guaranteeing the coverage of front and rear images is performed in sequence based on reference images set in front and rear of the determination target image. In the third embodiment, a method of setting a reference image other than the end points of the image sequence and determining whether to delete the front partial image sequence and the rear partial image sequence of the reference image will be described.

2. First Embodiment As a first embodiment, an image summarization process for guaranteeing a coverage for an image behind a reference image will be described. First, the basic method will be described, and then four modifications will be described. Each modification may use one of them or a combination of two or more.

2.1 Method of First Embodiment FIG. 1 shows a system configuration example of an image processing apparatus according to this embodiment. The image processing apparatus includes a processing unit 100, an image sequence acquisition unit 200, and a storage unit 300.

  The processing unit 100 performs image summarization processing on the image sequence acquired by the image sequence acquisition unit 200 by deleting some of the plurality of images included in the image sequence. The function of the processing unit 100 can be realized by hardware such as various processors (CPU and the like), ASIC (gate array and the like), a program, and the like.

  The image sequence acquisition unit 200 acquires an image sequence that is an object of image summarization processing. In addition to storing the image sequence acquired by the image sequence acquisition unit 200, the storage unit 300 serves as a work area for the processing unit 100 and the like, and its function can be realized by a memory such as a RAM or an HDD (hard disk drive).

  Further, as illustrated in FIG. 1, the processing unit 100 includes a reference image selection unit 1001, a determination target image selection unit 1002, a coverage area calculation unit 1003, a coverage rate calculation unit 1004, a deletion possibility determination unit 1005, A partial image sequence setting unit 1008 and a summary image sequence determination unit 1009 may be included. Note that the processing unit 100 is not limited to the configuration of FIG. 1, and various modifications such as omitting some of these components or adding other components are possible. Each of the above-described units is set to explain each subroutine when the image summarization process executed by the processing unit 100 is divided into a plurality of subroutines, and the processing unit 100 does not necessarily include the above-described units. Does not have as.

  The reference image selection unit 1001 selects a reference image from a plurality of images in the image sequence. The determination target image selection unit 1002 selects an image different from the reference image from among the plurality of images in the image sequence as the determination target image.

  The covering area calculation unit 1003 uses the deformation information (deformation parameters) between the reference image and the determination target image to project the reference image onto the determination target image to obtain a covering area. The coverage calculation unit 1004 calculates the coverage based on the coverage area.

  Deletability determination unit 1005 determines whether or not the determination target image can be deleted based on the calculated coverage. Specifically, a comparison process between the coverage and a given threshold value may be performed.

  When the deletion permission determination unit 1005 determines that the determination target image cannot be deleted, the partial image sequence setting unit 1008 determines a part of the image sequence based on the position in the image sequence of the determination target image at that time. Thus, an image sequence composed of one or more images is set as a partial image sequence.

  The summary image sequence determination unit 1009 determines a summary image sequence that is an image sequence after the summary processing. In the present embodiment, the reference image selected by the reference image selection unit 1001 is included in the summary image sequence. Further, it is assumed that an image determined to be deleteable among the determination target images is deleted and not included in the summary image sequence.

  FIG. 2 shows a flowchart for explaining the image summarization processing of this embodiment. When this processing is started, first, an image sequence to be subjected to image summarization processing is acquired (S101). The image sequence is acquired by the image sequence acquisition unit 200, and an RGB 3-channel image arranged in time series can be considered. Alternatively, it may be an image sequence that is spatially continuous, such as an image sequence that is spatially arranged and photographed by imaging devices arranged in a horizontal row. Further, the “space” in the spatially continuous image sequence may be a space representing a two-dimensional or three-dimensional position, or may be a space such as a color space.

  When the image sequence is acquired, the reference image selection unit 1001 selects the first image of the input image sequence (the image sequence acquired in S101 in the first process and then the partial image sequence set in S107 described later). A reference image is selected (S102). The reference image selected here is left in the summary image sequence. Note that when the reference image cannot be selected from the input image sequence due to an error or the like (for example, when there is no image in the image sequence), the process ends.

  Then, the determination target image selection unit 1002 selects a determination target image from images included in the input image sequence (S103). If the determination target image is not set, the next image of the reference image (second image in the input image sequence) is selected as the determination target image. If the kth image in the input image sequence has already been selected as the determination target image, the selection position is shifted by 1 and the k + 1th image in the input image sequence is selected as the new determination target image. If the determination target image cannot be selected (for example, if the number of images included in the input image sequence is less than 2 or k + 1), the process ends.

  When the reference image and the determination target image are selected, the covering region calculation unit 1003 uses the deformation parameter between the reference image and the determination target image to project the reference image onto the determination target image to obtain the covering region ( S104). Here, the deformation parameter may be a non-rigid deformation parameter estimated by the method described in Patent Document 2. An example of the covering region is shown in FIG. Here, the deformation parameter represents how the subject imaged in the reference image is deformed on the determination target image. In other words, the subject imaged in the reference image and the subject imaged in the covering area on the determination target image are corresponding (same in the narrow sense).

  When the coverage area is calculated, the coverage ratio calculation unit 1004 calculates the coverage ratio based on the coverage area and the determination target image (S105). For example, the coverage can be obtained from the ratio of the area of the covering region to the area of the entire determination target image as shown in FIG.

  Then, the deletion possibility determination unit 1005 performs a comparison process between the calculated coverage and a preset threshold value (which may be set by the system or may be determined based on an input from the user). Perform (S106). If the coverage is less than the threshold, it is determined that the determination target image cannot be deleted, and the process proceeds to a partial image sequence setting process. If the coverage is equal to or greater than the threshold, it is determined that the determination target image can be deleted, and the process returns to S103 to select the determination target image again.

  If it is determined in S106 that the determination target image cannot be deleted, the partial image sequence setting unit 1008 sets a partial image sequence (S107). Specifically, an image sequence composed of a determination target image determined to be unremovable and subsequent images may be set as a partial image sequence. When the partial image sequence is set, the process returns to S102, and the above-described processing is executed using the partial image sequence as the input image sequence.

  FIGS. 4A to 4D illustrate the above image summarization processing. As shown in FIG. 4A, when an image sequence having N images is acquired by the image sequence acquisition unit 200, first, the first image is selected as the reference image, and the second image is selected. Selected as a determination target image. Then, the coverage is calculated between the reference image and the determination target image, and it is determined whether the determination target image can be deleted.

  If it is determined that the determination target image can be deleted, a new determination target image is selected. Specifically, the position of the determination target image is shifted backward, and the third image is selected as the determination target image as shown in FIG. Then, the coverage is calculated between the reference image and the determination target image, it is determined whether or not the determination target image can be deleted, and the image selected as the determination target image is updated until a determination target image that is determined not to be deleted is found. I will do it.

  As shown in FIG. 4C, when it is determined that the second to k−1th images can be deleted and the kth image is determined to be undeleteable, the second to k−1th images are deleted. Since an image means that it is covered to some extent by the reference image (to the extent set by the threshold), it is deleted and not included in the summary image sequence. On the other hand, the k-th image cannot be sufficiently covered by the reference image, so it is necessary to leave it in the summary image sequence. For this purpose, the k-th image and the subsequent images (k to N-th images) are set as partial image sequences.

  Then, the processes shown in FIGS. 4A to 4C may be repeated for the partial image sequence. Specifically, as shown in FIG. 4D, a partial image sequence composed of N−x + 1 images is set as an input image sequence, and the first image (k-th in FIG. 4C) is the reference image, The second (k + 1th in FIG. 4C) image is processed as a determination target image. The subsequent processing is the same. When it is determined that the determination target image can be deleted, the next image is selected as a new determination target image. Further, if it is determined that the determination target image cannot be deleted, the reference image is left in the summary image sequence, the image determined to be deleteable is deleted, and the image after the determination target image at that time is replaced with a new partial image sequence. Set. Eventually, if it is determined that all the images in the input image sequence can be deleted, or if only one image is included in the input image sequence and the determination target image cannot be set, the process ends. It will be.

  In the above-described embodiment, as illustrated in FIG. 1, the image processing apparatus includes an image sequence acquisition unit 200 that acquires an image sequence having a plurality of images, and a plurality of images in the image sequence acquired by the image sequence acquisition unit 200. A processing unit 100 that performs image summarization processing for deleting a part of the image and acquiring a summary image sequence. Then, the processing unit 100 selects a reference image and a determination target image from a plurality of images. This process is performed by, for example, the reference image selection unit 1001 and the determination target image selection unit 1002. Further, the coverage of the determination target image by the reference image is calculated based on the deformation information between the selected reference image and the determination target image, and whether or not the determination target image can be deleted is determined based on the calculated coverage. . The coverage rate calculation process is performed by, for example, the coverage rate calculation unit 1004, and the deletion permission determination process is performed by, for example, the deletion permission determination unit 1005.

Here, the coverage is information indicating how many of the subjects captured on the determination target image are captured on the reference image. For example, when an image having an aspect ratio of 1: 1 is acquired, a subject having a square shape of 10 m in length and breadth in real space is captured in the determination target image, and the reference image includes the aforementioned subject. Assume that a square area of 5 m in length and width is captured in full. In this case, an area of 100 m ² in real space is imaged in the determination target image, and an area of 25 m ² in real space (and an area included in the area of 100 m ² described above) is imaged in the reference image. . Therefore, since the reference image covers 25% of the determination target image, the coverage may be 25 or 0.25. In addition, since it is rare that an image is captured facing a planar subject, generally, the shape of the reference image and the determination target image are different even for the same subject. In the present embodiment, deformation information corresponding to such deformation is acquired by a method such as Patent Document 2, and the coverage is calculated using the deformation information. The coverage may be information indicating the degree of coverage of the determination target image by the reference image, and is not limited to the ratio / ratio.

  Further, the determination process of whether or not deletion is possible is, for example, a comparison process with a given threshold value. If the threshold value is increased (for example, if it is set to a value corresponding to 100%), it can be expected to improve the suppression effect against the occurrence of a region that cannot be observed by deleting the image. On the other hand, if the threshold value is lowered, the number of images included in the summary image sequence after the summary processing can be reduced. There is a trade-off between improving the suppression effect and reducing the number of images, and control is possible by setting the threshold value. Therefore, it is desirable to set the threshold value appropriately according to the situation.

  As a result, it is possible to suppress the occurrence of a subject region that cannot be observed as a result of deleting an image by image summarization processing, and to control the degree (intensity) of the suppression. By using the method of the present embodiment, if a value corresponding to x% is used as the threshold used for the determination process of whether or not deletion is possible, the subject imaged on the determination target image even if the determination target image is deleted This is because it can be guaranteed that x% is covered by the reference image (the subject range of x% is captured on the reference image). Since it is difficult to obtain the deformation on the subject image as deformation information without causing any error, even if x is set as the threshold, the region covered by the reference image in the determination target image Can be less than x%.

  When first to Nth images (N is an integer equal to or greater than 2) are input as an input image sequence, the processing unit 100 selects the first image as a reference image, and the kth (k is An integer satisfying 2 ≦ k ≦ N−1) may be selected as the determination target image. Then, the coverage is calculated based on the deformation information between the reference image and the determination target image, and whether or not the determination target image can be deleted is determined based on the coverage. Further, when it is determined that the kth image can be deleted, the (k + 1) th image is selected as the determination target image.

  Here, the input image sequence is an image sequence that is a target of this processing (selection of a reference image / determination target image, determination of whether or not to delete, and update of a determination target image when deletion is possible). The image sequence acquired by the unit 200 may be used, or an image sequence including a part of the image sequence may be used.

  As a result, when an input image sequence is input, processing as shown in FIGS. 4A to 4B can be performed. Here, the determination target image is selected from the rear of the reference image (for example, the image adjacent to the reference image at first), and when the selected determination target image can be deleted, the determination target image is further updated to the rear one. . That is, it is determined whether it can be deleted in order from a place close to the reference image (whether it can be sufficiently covered by the reference image), and a place where it cannot be deleted is searched. Note that the determination target image determined to be deleteable here is basically deleted and is not included in the summary image sequence, but is not limited to this, and the determination target image determined to be deleteable May be included in the summary image sequence.

  In addition, the processing unit 100 may perform processing for including the image selected as the reference image in the summary image sequence. At the same time, when it is determined that the kth image selected as the determination target image cannot be deleted, the partial image sequence including the kth to Nth images is set as a new input image sequence and set. Processing may be performed again on the input image sequence.

  As a result, the process shown in FIG. 4C can be performed. Since the deletion permission / inhibition determination process is performed based on how much the determination target image is covered by the reference image as described above, the reference image remains even if the determination target image determined to be deleteable is deleted. Therefore, it is possible to suppress the occurrence of an unobservable region. That is, in this embodiment, the reference image is included in the summary image sequence. If the k-th image is determined not to be deleted, it means that the k-th image cannot be sufficiently covered by the reference image, and therefore the k-th image should remain in the summary image sequence. That is, the kth image may be set as the next reference image. Specifically, for example, a partial image sequence including the kth to Nth images may be set as a new input image sequence. In this way, as shown in FIG. 4D, the head image of the input image sequence, that is, the kth image in the image sequence acquired by the image sequence acquisition unit 200 is selected as the reference image, and the (k + 1) th image is selected. Images after the image are sequentially selected as determination target images. Note that a partial image sequence including a part of the images in the image sequence of FIG. 4D may be newly set as an input image sequence, and this process is repeated (or recursively).

  Further, the processing unit 100 may obtain a covered region that is a region where the determination target image is covered by the reference image based on deformation information between the reference image and the determination target image. Then, the ratio of the coverage area in the determination target image is calculated as the coverage ratio.

  Thereby, it is possible to calculate the coverage based on the coverage area. The covering region is specifically shown in FIG. 3 and represents a region projected on the determination target image after the reference image is deformed based on the deformation information. The subject area imaged in the reference image corresponds to the subject area imaged in the obtained covering area (they match in an ideal situation where there is no error in the deformation information). Therefore, it is possible to obtain the coverage from the ratio of the covered area to the determination target image (specifically, the ratio of the respective areas). The covered area may be obtained by deforming the reference image based on the deformation information, and the obtained covered area is not necessarily limited to the one projected onto the determination target image. Further, the covering region is not limited to the one obtained based on the entire reference image, and may be obtained by deforming a part of the reference image with the deformation information.

  Further, in the present embodiment described above, an image sequence acquisition unit 200 that acquires an image sequence having a plurality of images, and a summary image sequence by deleting some of the plurality of images in the image sequence acquired by the image sequence acquisition unit 200 It can be applied to a program that causes a computer to function as the processing unit 100 that performs image summarization processing. Then, the processing unit 100 selects a reference image and a determination target image from a plurality of images. Further, the coverage of the determination target image by the reference image is calculated based on the deformation information between the selected reference image and the determination target image, and whether or not the determination target image can be deleted is determined based on the calculated coverage. .

  As a result, the above-described image processing can be realized by a program. For example, it is conceivable that an image sequence acquired by a capsule endoscope or the like is input to a system such as a PC, and image summarization processing is performed by a program executed by a processing unit (CPU, GPU, etc.) of the system. The program is recorded on an information storage medium. Here, as the information recording medium, various recording media that can be read by a system such as a PC such as an optical disk such as a DVD or a CD, a magneto-optical disk, a hard disk (HDD), a memory such as a nonvolatile memory or a RAM can be assumed. .

2.2 Modification (Coverage calculation based on multiple points)
Next, a modification of this embodiment will be described. In the above-described method, the reference image itself is deformed by the deformation parameter and the covering area is calculated, but the covering rate calculating method is not limited to this. For example, when a plurality of points are set on the determination target image instead of the image itself, and the points are moved based on the deformation parameter, based on the positional relationship between the plurality of moved points and the reference image The coverage may be calculated. This will be described in detail below.

  The configuration of the processing unit 100 is a configuration in which the covering region calculation unit 1003 is removed as compared to FIG. Further, the processing in the coverage rate calculation unit 1004 is different. In addition, since the processing content is the same about other structures, detailed description is abbreviate | omitted and the coverage calculation process is described.

  As illustrated in FIG. 5, the coverage calculation unit 1004 arranges a plurality of points on the determination target image at regular intervals, for example. Then, a point on the determination target image is projected onto the reference image using a deformation parameter between the reference image and the determination target image (similar to the method described above, the non-rigid body deformation parameter of Patent Document 2). Of the plurality of points projected onto the reference image, the ratio of the number of points included in the reference image may be the coverage.

  By doing in this way, compared with the method which deform | transforms the image itself and calculates | requires a covering area | region, a coverage rate can be calculated easily and reduction of a processing load etc. is attained.

  FIG. 6 shows a flowchart for explaining this process. S201 to S203 and S206 to S207 are the same as S101 to S103 and S106 to S107. In addition, since there is no need to calculate the covering area, there is no block corresponding to S104. S205 corresponds to S105, but the processing contents are different as described above, and S105 calculates the coverage based on the coverage area calculated in S104, whereas S205 projects the plurality of points. The coverage is calculated based on the above.

  In the above modification, the processing unit 100 sets a plurality of points for the determination target image, and converts the plurality of points according to the deformation information between the reference image and the determination target image into the reference image. Based on the number of converted points included, the coverage is calculated.

  As a result, it is possible to calculate the coverage by performing processing on the set points instead of performing processing based on the deformation information on the reference image itself. Compared to this, the processing load can be reduced. Specifically, a plurality of points are set on the determination target image, and it is determined which image position (including a position outside the reference image here) corresponds to each set point in the reference image. Then, the coverage is calculated by comparing the number of points set in the determination target image with the number of points included in the reference image among the plurality of points after processing based on the deformation information. In this way, it is not necessary to calculate the area after processing using the deformation information, and the processing can be lightened. The problem here is the case where the reference image cannot sufficiently cover the determination target image. In such a case, even if a plurality of points are set on the reference image and moved to the determination target image. The converted points are all included in the determination target image, and do not serve as an index representing the degree of cover. Therefore, if a method of setting a plurality of points is used, it is desirable to set the plurality of points on the determination target image.

2.3 Modification (Example of setting a plurality of reference images)
Next, an example using a plurality of images as a reference image will be described. The configuration of the processing unit 100 here is the same as in FIG. As shown in FIG. 7, the covering area calculation unit 1003 calculates and calculates candidate areas of the covering area between each reference image of a plurality of reference images (reference image group) and the determination target image. A covering area is calculated based on a plurality of candidate areas. In this case, the processing in the coverage rate calculation unit 1004, the deletion possibility determination unit 1005, and the like is the same as the above-described method.

  Alternatively, the coverage area is calculated between each reference image of the plurality of reference images (reference image group) and the determination target image, and the coverage calculation unit 1004 calculates the determination target image and the calculated plurality of coverage areas. It may be considered that the coverage is calculated based on the above.

  Specific processing will be described below. Since the flowchart is the same as FIG. 2, the drawing is omitted. Since S101 is the same, description thereof is omitted. In S102, the first image in the input image sequence is selected as a reference image and added to the reference image group. When S102 is performed for the first time, there is only one reference image included in the reference image group, and thus the processes in S103 to S107 are the same as those described above.

  When the partial image sequence is set in S107 and the process returns to S102, the reference image selection unit 1001 selects the first image of the partial image sequence as the reference image and adds it to the reference image group. That is, if the process of S102 is performed for the mth time, a total of m reference images are held in the reference image group. Then, by calculating the i-th candidate area from the i-th (1 ≦ i ≦ m) reference image and the determination target image in the reference image group, m candidate areas are obtained, and the covering area is calculated therefrom. . For example, as shown in FIG. 8, a region corresponding to the union of a plurality of candidate regions may be set as a covering region.

  Specific examples are shown in FIGS. 9A and 9B. When the first image is a reference image, it is determined that the 2nd to (x-1) th images can be deleted, and when the xth image is determined to be undeleteable, as described above, the xth to Nth images are determined. An image sequence having an image is set as a partial image sequence. Then, in order to set the partial image sequence as the input image sequence, the head of the partial image sequence, that is, the xth image in the original image sequence is selected as the reference image. Therefore, the reference image group holds the first and xth images as reference images. When a determination target image (selected in order from x + 1) is selected, a candidate area is obtained based on the first image and the determination target image, and based on the xth image and the determination target image. Find candidate areas. Then, the final coverage area is calculated by the method described with reference to FIG.

  Similarly, if it is determined that the yth image cannot be deleted, the image sequence including the yth to Nth images becomes a partial image sequence, and thus the yth image is added to the reference image group. Then, the y + 1th and subsequent images are selected as the determination target images, candidate areas are obtained from the three images of the first image, the xth image, and the yth image, respectively, and the covering region is calculated. Thereafter, the same processing is repeated.

  Although the coverage area is used for calculating the coverage here, the present invention is not limited to this. You may combine with the method of using the several point mentioned above.

  In the above modification, the processing unit 100 may select the first to Mth reference images (M is an integer of 2 or more) from a plurality of images as the reference image. Then, the u-th covering region is obtained based on deformation information between the u-th (1 ≦ u ≦ M) reference image and the determination target image, and the union region of the first to M-th covering regions is determined as the union region. The coverage is set as the coverage area, and the coverage is calculated based on the set coverage area.

  As a result, as shown in FIGS. 9A and 9B, a plurality of reference images can be set as reference images. In the embodiment in which one reference image is set, the latest reference image (for example, the xth image in FIG. 9A) is used for determining whether or not the determination target image (for example, the (x + 1) th image in FIG. 9A) can be deleted. ) Was used. However, from the viewpoint of suppressing the occurrence of a region that cannot be observed by deleting the image, the subject captured in the image to be deleted does not have to be captured in the most recent image that remains, and the image that remains It is sufficient that at least one of the images is captured. Therefore, the reference image need not be limited to one, and a plurality of reference images may be used. For example, in the example of FIG. 9A, the x-th to N-th images are set as the partial image sequence so that the x-th image becomes a new reference image. The first image that has been used as the reference image may be held and used to determine whether or not the determination target images after the (x + 1) th image can be deleted. In that case, as shown in FIG. 8, it is conceivable that the area representing the union of the areas obtained by deforming the respective reference images based on the deformation information is used as the covering area. By doing so, there is a high possibility that it is possible to determine that the determination target image can be deleted, and it is possible to reduce the number of images included in the image sequence after the summary processing. Therefore, it is possible to reduce the load on the user. By repeating the process, the number of images that can be used as reference images increases (in the example of FIG. 9B, the first and xth images can be used in addition to the yth image), and it is determined that deletion is possible. The possibility to do is further improved. However, since the processing load increases by increasing the number of reference images, it is not necessary to use all of the usable reference image candidates, and some of them may be used.

  Further, the processing unit 100 may select the first to Mth (M is an integer equal to or greater than 2) reference images from a plurality of images as a reference image even in a method using a plurality of points instead of the covering region. When a plurality of points set on the determination target image are converted according to deformation information between the u th (1 ≦ u ≦ M) reference image and the determination target image, the u th reference image is obtained. The number of converted points included is obtained as the u-th covering information, and the covering ratio is calculated based on the first to M-th covering information.

  Thereby, even when a plurality of points are used, a plurality of reference images can be set. However, when a plurality of points are used, the number of converted points included in the reference image is used for the coverage calculation, so that information corresponding to the union of the coverage information obtained from the plurality of reference images must be calculated. It is difficult (it is possible to obtain the position of the point after conversion, so that it is possible to perform some processing based on the position information, but this may lose the advantage that the processing is easier than the method using the covered region). Therefore, it is conceivable that the coverage in this case is calculated using the maximum value among the number of points obtained from each reference image. By doing so, the effect of reducing the number of images as much as the method using a plurality of reference images and coverage areas cannot be obtained, but the processing load is lighter than that of the method and the method using one reference image is compared. Thus, the number of images after the summarization process can be reduced.

2.4 Modification (weighting)
In the above-described method, the area of the covering region with respect to the area of the determination target image is simply set as the covering ratio, but the present invention is not limited to this. For example, a coverage factor may be obtained by setting a weighting factor according to the position on the image and performing weighting using the weighting factor. Here, a modified example will be described in which image summarization that guarantees a weighted coverage according to the distance from the center of the image is performed in sequence. Detailed description of the same processes as those described above will be omitted.

  The coverage area calculation unit 1003 obtains a weight map of the reference image in which the area weight is applied according to the position in the image. Then, as shown in FIG. 10, using the deformation parameter between the reference image and the determination target image, the weight map of the reference image is projected onto the determination target image to form a weighted coverage region. For example, in a capsule endoscopic image having blackened areas at the four corners, a weight with a predetermined area weight of 0 and a remaining area weight of 1 is applied.

  Then, the coverage calculation unit 1004 acquires a weight map of the determination target image to which the region weight is applied according to the position in the image. The ratio of the sum of the coverage weight maps obtained by multiplying the weighted coverage region and the weight map of the determination target image to the total sum of the weight maps of the determination target image is defined as the coverage ratio. For example, in a capsule endoscopic image having blackened areas at the four corners, a weight with a predetermined area weight of 0 and a remaining area weight of 1 is applied.

  In the example of FIG. 10, among the covered areas, the four corner areas whose weights are set to 0 in the reference image weight map are not treated as covered areas. In addition, an area projected on the four corners having a weight of 0 in the weight map of the determination target image in the covering area corresponds to being not handled as a covering area. In addition, the area of the determination target image corresponding to the denominator of the coverage calculation formula does not include the four corner regions whose weights are set to 0 in the determination target image weight map.

  In the example of FIG. 10, 0 or 1 is used as the weight, but a value between 0 and 1 may be set. The weight is typically between 0 and 1, but it does not preclude setting a negative value or a value greater than 1.

  The weighting is not limited to the method using the covering region, and a plurality of points may be set and a weighting coefficient may be set for each point. Further, it can be applied to a method using a plurality of reference images.

  In the above modification, the processing unit 100 may set the weighting coefficient according to the position on the image of the determination target image. Then, the coverage ratio is calculated based on the ratio between the first weight sum calculated based on the weight coefficient and the coverage area and the second weight sum calculated based on the weight coefficient and the determination target image. To do.

  Thereby, weighting according to the position on the image of the determination target image can be performed, and the degree of contribution to the coverage ratio calculation can be set according to the position of the determination target image on the image. For example, when an optical system such as a fish-eye lens that has a large influence of distortion is used, the distortion in the peripheral portion of the image is larger than that in the central portion of the image, which is not suitable for observing the subject. In such a case, it is possible to determine whether or not deletion is possible with an emphasis on the central part by increasing the weighting coefficient in the central part and reducing the weighting coefficient in the peripheral part. Specifically, even if the equivalent area is covered on the image, the coverage ratio greatly contributes when the central portion is covered, whereas the coverage ratio is even if the peripheral portion is covered. The contribution to is small.

  The weight sum is determined based on the region to be processed and the weighting factor. For example, if the weighting factor is set for each pixel, the first weighting sum may be the sum of the weighting factors set for each pixel for the pixels included in the covering region. The weight sum may be the sum of the weight coefficients set for each pixel for the pixels of the entire determination target image. Further, if the weighting factor is set for each given region (the area of each region may be different), the second weight sum is the sum of the products of the weighting factor setting region and the weighting factor. Thus, the first weight sum is the sum of the products of the area of the area included in the covering area and the weight coefficient in the weight coefficient setting area. Depending on the weight coefficient setting method, the weight sum may be obtained by another method.

  In addition, the processing unit 100 may set the weighting coefficient according to the position of the determination target image on the image, using a method that uses a plurality of points instead of the covering region. When a plurality of points are converted according to deformation information between the reference image and the determination target image, the coverage is calculated based on the converted points included in the reference image and the weighting factor.

  This makes it possible to use the weighting factor even when using a plurality of points. As a specific method, when a plurality of points are set on the determination target image, a weighting coefficient corresponding to the position on the image may be given to each point. Then, when a plurality of points are projected on the reference image based on the deformation information, the number of points included in the reference image is not simply counted, but the value of the weighting factor assigned to the points included in the reference image The coverage is obtained by integrating. In this case, the denominator of the formula for calculating the coverage shown in FIG. 5 is not simply the total number of points, but a value obtained by summing up the weighting factors for all the set points.

  Further, the processing unit 100 sets 0 as the weighting factor value in the first area of the determination target image and sets 1 in the second area different from the first area of the determination target image. Good.

  As a result, as an extreme example of weighting, it is possible to perform processing that uses a partial region of the determination target image and does not use other regions.

  In the above weighting example, the weighting factor is set for the determination target image, but the weighting factor may be set on the reference image side. For example, referring to an example using a covering area, in FIG. 10, the weighting coefficient 0 is set at the four corners of the reference image, and the weighting coefficient 1 is set at the other areas. In this case, the second weight sum is the same as that described above, but the first weight sum calculation method is different. In this case, a weighting coefficient 0 is set in the area corresponding to the four corners of the reference image in the covering area, and a weighting coefficient 1 is set in the other areas of the covering area. Furthermore, as described above, since the covering region is projected onto the determination target image, a weighting coefficient corresponding to the position of the determination target image on the image is also set. In other words, if the weighting factor is set in units of pixels, two pixels, that is, the weighting factor on the reference image side and the weighting factor on the determination target image side are set for each pixel included in the covering region. Will be. In this case, as the first weight sum, for example, for each pixel included in the covering region, a product of the reference image side weight coefficient and the determination target image side weight coefficient is obtained, and a process for obtaining the sum of the values is performed. Good.

  Also, when using a plurality of points as shown in FIG. 5, a weighting factor may be set on the reference image side. In this case, the point that is located in the reference image by the conversion among the plurality of points can be associated with the weighting coefficient set on the reference image side, and as a result, it becomes one point. On the other hand, two weight coefficients are set for the reference image side and for the determination target image side. Therefore, for example, the coverage may be calculated based on the sum of the values obtained by obtaining the product of the reference image side weight coefficient and the determination target image side weight coefficient for each of the converted points included in the reference image.

2.5 Modification (Method using deformation parameters between adjacent images)
In the above-described method, the covering region is directly calculated using the deformation parameter between the reference image and the determination target image. However, the present invention is not limited to this. For example, by using a deformation parameter between all adjacent images from the reference image to the determination target image, the covering region is cumulatively projected between the adjacent images, and the covering region obtained by projecting the reference image onto the determination target image may be obtained. Good.

  In the above modification, the processing unit 100 obtains deformation information between adjacent images for the image between the reference image and the determination target image in the image sequence, and based on the obtained deformation information between the adjacent images, Deformation information between the image and the determination target image may be calculated.

  Thereby, when the reference image and the determination target image are not adjacent to each other, it is possible to calculate by accumulating the deformation information obtained between the adjacent images instead of directly obtaining the deformation information between the two images. Become. The deformation information can be calculated by the method shown in Patent Document 2 or the like, but generally, the process of combining a plurality of pieces of deformation information is much lighter than the process of calculating the deformation information from scratch. For example, if the deformation information is a matrix or the like, the process of obtaining the matrix from two pieces of image information is heavy, but combining a plurality of already obtained matrices (for example, it is only necessary to take the matrix product) ) It will be very easy.

  This method is particularly effective in a process where the number of times of using deformation information is used, for example, as in a second embodiment described later. For example, in the second embodiment, a reference image (second reference image) is set not only in front of but also behind the determination target image, and the second reference image is updated according to conditions. Specifically, when the first image is the first reference image and the k-th image is the second reference image, it is determined whether or not deletion is possible between the second to (k-1) -th images and each reference image. Depending on the conditions, the second reference image is updated to the (k + 1) th image without changing the first reference image. In this case, deformation information between each of the second to kth images and the (k + 1) th image that is the second reference image is necessary, and it is necessary to obtain the deformation information k−1 times. And since it is different from the deformation information between the 2nd to (k-1) th images used in the immediately preceding process and the kth image (the immediately preceding second reference image), it must be newly determined. . For example, the image sequence acquisition unit 200 acquires N images as an image sequence, and the second reference image is sequentially updated from the third to the Nth image without changing the first reference image. If a case is assumed, it is necessary to obtain deformation information 1 + 2 + 3 +... + N−2 = (N−2) (N−1) / 2 times. In other words, the number of times of calculating deformation information with a large load is large and inefficient.

  In that respect, if adjacent deformation information is used, if the image sequence acquisition unit 200 acquires N images as an image sequence, it is sufficient to obtain the deformation information N-1 times between adjacent images. In this case, when the reference image and the determination target image are selected from the N images, a process of combining necessary ones of the N-1 pieces of deformation information is necessary. The processing is lighter than the deformation information calculation processing.

3. Second Embodiment In the second embodiment, a method will be described in which image summarization processing that guarantees the coverage of front and rear images is performed in sequence. First, the basic method will be described, and then two modifications will be described.

3.1 Method of Second Embodiment FIG. 11 shows a system configuration example of an image processing apparatus according to this embodiment. A second reference image selection unit 1007 is added to the processing unit 100 of FIG.

  The reference image selection unit 1001 selects the first reference image. The second reference image selection unit 1007 selects two or more images behind the first reference image as the second reference image. Further, the determination target image selection unit 1002 selects an image that is behind the reference image and ahead of the second reference image as the determination target image. In FIG. 11, the reference image selection unit 1001 and the second reference image selection unit 1007 are separated. However, the present invention is not limited to this, and the reference image selection unit 1001 uses the first reference image and the second reference image. You may select both.

  FIG. 12 shows a flowchart for explaining the image summarization processing of this embodiment. S301 to S302 are the same as S101 to S102. After S302, two or more images behind the first reference image selected in S302 are selected as second reference images (S308). Then, a determination target image is set (S303). When the determination target image is not set, the image next to the first reference image (second image in the input image sequence) is selected as the determination target image. If the kth image in the input image sequence has already been selected as the determination target image, the selection position is shifted by 1 and the k + 1th image in the input image sequence is selected as the new determination target image. However, the selection range of the determination target image is not up to the last image in the input image sequence but until the second reference image matches.

  When the determination target image does not coincide with the second reference image, the process proceeds to S304, and the coverage area is calculated. Here, the first candidate region is calculated based on the deformation parameter between the first reference image and the determination target image, and the first candidate region is calculated based on the deformation parameter between the second reference image and the determination target image. 2 candidate regions are calculated. Then, similarly to the processing described with reference to FIG. 8, a region corresponding to the union of the first candidate region and the second candidate region may be set as the covering region.

  S304 to S306 are the same as S104 to S106. If it is determined in S306 that the image can be deleted, the process returns to S303, and the determination target image is updated to the next image. If the determination target image matches the second reference image as a result of the update, the process returns to S308, and the second reference image is updated to the next image. In addition, when the second reference image is updated, the selection state of the determination target image is reset. If the determination target image does not match the second reference image, the processing from S304 is performed.

  If it is determined in S306 that the determination target image cannot be deleted, the reference image and the current second reference image cannot cover all the images sandwiched between them. Therefore, it is necessary to leave an image one previous to the current second reference image in the summary image sequence. Therefore, an image sequence including an image immediately before the current second reference image and the subsequent image is set as the partial image sequence (S307), and the process returns to S302.

  FIGS. 13A and 13B illustrate the above image summarization processing. Here, a method of cumulatively using the deformation parameters between adjacent images when projecting the reference image onto the determination target image will be described, but the present invention is not limited to this.

  It is assumed that the entire image sequence kth image has been selected as the first reference image (note that the processing has been completed for the 1st to (k-1) th images, and the kth to Nth images are the partial image sequences. Corresponds to the case where it is set as). Then, the k + 2nd image is selected as the second reference image.

  Further, a determination target image is selected from the front of the image between the first reference image and the second reference image, and a covering area calculation process based on the first reference image and the determination target image, and determination as the second reference image are performed. The coverage ratio is calculated from the coverage area calculation process based on the target image, and whether or not deletion is possible is determined.

  As shown in FIG. 13A, when it is determined that all the images between the first reference image and the second reference image can be deleted, the first reference image is the first reference image. Since there is a possibility that an image further away from the reference image may be selected, the second reference image is reselected as shown in FIG. 13B. Specifically, the second reference image that was k + 2 may be updated to k + 3.

  Then, it is determined again whether or not the image between the first reference image and the second reference image can be deleted. As shown in FIG. 13B, when there is a determination target image that is determined not to be deleted, all of the first reference image and the current second reference image are included between them. This means that it is impossible to cover the image (specifically, the determination target image that is determined not to be deleted cannot be covered). It is thought.

  Therefore, an image sequence including an image immediately before the current second reference image (corresponding to the second reference image in FIG. 13A) and subsequent images is set as a partial image sequence. As a result, the second reference image at the time of FIG. 13A is selected as the reference image in the next process, and the image to be deleted is covered by the image remaining in the summary image sequence. Is guaranteed.

  In the above description, the top image of the input image sequence is selected as the reference image in S302. However, this is not the case when the processing in S302 is performed for the first time. In the present embodiment, the determination target image can be deleted if it is covered by the second reference image behind the determination target image. That is, as shown in FIG. 14A, for example, if the first and second images are covered by the third image, it is necessary to leave the first and second images in the image after the summary processing. There is no. Therefore, it is not always necessary to leave the top image, and there is a possibility that the number of images included in the summary image sequence will be unnecessarily increased in the method using the top image as a reference image.

  Therefore, in the present embodiment, the first reference image does not have to be the top image of the image sequence acquired by the image sequence acquisition unit 200. An example of a specific method is shown. As shown in FIG. 14B, the 0th image that does not actually exist is selected as the first reference image (the selection process here is for convenience, and the 0th image is actually selected). Preparation or other processing is not necessary). If it does in that way, the 2nd picture will be selected by the 2nd standard picture selection processing in S308, and the picture in the meantime (only the 1st picture here) will be chosen one by one. The processes of S304 to S306 are performed between the determination target image and the second reference image as long as the first reference image does not actually exist. If the first image is covered by the second image, the second reference image is updated as shown in FIG. 14C in accordance with the processing of FIG. 12, and the third image is transferred. It is determined whether the first and second images can be covered by the second image. Hereinafter, if the process is repeated, as shown in FIG. 14D, if the k-1st image is set as the second reference image, all of the 1st to k-2th images can be covered. Even if the 1st image is used as the second reference image, k that cannot cover all the 1st to (k-1) th images can be found. In this case, it is determined in S306 that the image cannot be deleted. In S307, the image sequence including the k-1 to Nth images is set as the partial image sequence, and the process returns to S302. In the second and subsequent processing in S302, as described above, the first image of the input image sequence is selected as the reference image, so that the (k−1) -th image remains in the summary image sequence as the reference image. As described above, since the 1st to 2nd images can be covered by the (k-1) th image, the 1st to 2nd images can be deleted, and the images included in the summary image sequence The number of sheets can be reduced.

  In the present embodiment described above, when the first to Nth images (N is an integer of 2 or more) are input as the input image sequence, the processing unit 100 selects the pth image as the first reference image. The qth image (q is an integer satisfying p + 2 ≦ q ≦ N−1) is selected as the second reference image, and the rth image (r is an integer satisfying p + 1 ≦ r ≦ q−1) is selected. Select as a determination target image. Then, the coverage is calculated based on the deformation information between the first reference image and the determination target image and the deformation information between the second reference image and the determination target image, and based on the coverage It is determined whether or not the determination target image can be deleted. Further, when it is determined that the p + 1 to q−1th images can be deleted, the q + 1th image is newly selected as the second reference image.

  As a result, as shown in FIGS. 13A and 13B, the image summarization processing based on the coverage can be performed after setting the reference images in front and behind the determination target image. In this case, since two reference images are used, there is a high possibility that it is possible to determine that the determination target image can be deleted, and the number of images after the summarization process can be reduced. In addition, as described in the modification of the first embodiment, an image that is close in time (or spatial) to the determination target image is set as the reference image as compared to the method of setting a plurality of reference images only in the front. it can. Since the closer the image is, the higher the possibility that the captured subject is more similar, so the possibility of deleting the determination target image can be further increased. In this embodiment, a plurality of reference images may be set in the front, rear, or both as in the above-described modification. When it is determined that the determination target image set between the first reference image and the second reference image can be deleted (in a narrow sense, all determination target images can be deleted. (It is not necessary to limit the second reference image.) In this case, even if the space between the first reference image and the second reference image is further widened, there is a possibility that the image between them may be covered. Is updated to an image later than the current second reference image.

  Note that, as described above, in the present embodiment, the reference image may be set behind the determination target image, and therefore it is not necessary to set the first reference image as the first image in the first process. This is because if all the previous images are covered by the second and subsequent given images, the previous images can be deleted by using the given image as the reference image.

  In addition, the processing unit 100 may perform a process of including the image selected as the first reference image in the summary image sequence. If it is determined that at least one of the p + 1 to q-1th images cannot be deleted, a partial image sequence composed of the q-1st to Nth images is set as an input image sequence and set. The input image sequence is processed again after setting the value of p to 1.

  As a result, the first reference image can be included in the summary image sequence in this embodiment as well as the reference image is included in the summary image sequence in the first embodiment. The case where at least one of the determination target images between the first reference image and the second reference image cannot be deleted is a case where the space between the first reference image and the second reference image is excessively widened. Therefore, the image that is in front of the second reference image at that time (in the narrow sense, in front and closest) should remain in the summary image sequence. Therefore, the partial image sequence including the q-1st to Nth images is set as an input image sequence, and the first and second reference images and the determination target image are selected from the set input image sequence. Depending on the conditions for determining whether or not to delete, the second image update process or the like is performed again. Note that, for the set partial image sequence, the head image should remain in the summary image sequence.

3.2 Modification (Other updating method of second reference image)
Next, a modification of this embodiment will be described. In this modified example, the same notation is repeated for the determination of whether or not deletion is possible because the second reference image selection method is described. Therefore, as a result of selecting the qth image as the second reference image and determining whether or not to delete in order to simplify the sentence, all images between the first reference image and the second reference image can be deleted. Is described as “the qth image is OK”, and at least one image between the first reference image and the second reference image cannot be deleted. “The image is NG”.

  In the above method, when the q-th image is OK, the second reference image is selected again. However, the new second reference image to be selected is limited to the q + 1-th image. It was.

When the first to Nth images are input as the input image sequence and the first image is selected as the first reference image, the determination target image candidate is selected when the qth image is selected as the second reference image Q-2 images (second to q-1th images) can be considered, and the determination process is performed q-2 times. Assuming that the image summarization process is completed without determining that the determination target image has been deleted once, q is selected from 3 to N (may include N + 1 if a virtual image is considered). Therefore, at least 1 + 2 + 3 +... + N−2 = (N−2) (N−1) / 2 processes are required, and the order of calculation amount is N ² . That is, in the above-described method, if N becomes very large, the amount of calculation increases dramatically, which is not preferable.

  Therefore, here, when the second reference image is selected again, the amount of calculation is reduced by expanding the selection range without limiting the target to adjacent images. Specifically, when the q-th image is OK, the new second reference image is not limited to the q + 1-th image, but is allowed to be selected from the q + 2 image and the image behind it. In this case, even if the qth image is NG, it is not known whether or not the q-1th image is OK (the q-1st image may not be selected as the second reference image). Because there is). Therefore, unlike the basic method of the second embodiment, the fact that the q-th image is NG does not immediately leave the q-1-th image as a summary image, but basically the q-th image. By selecting an image ahead of the first image as a new second reference image, the front image is determined.

  That is, in this modification, until the end condition is satisfied, the second reference image is updated backward in the case of OK, and forward in the case of NG, thereby searching for the next summary image of the first reference image. It will be. By appropriately setting the position of the new second reference image, the number of images selected as the second reference image until the next summary image is found can be reduced, and the amount of calculation can also be reduced. Note that the calculation amount reduction here is only to reduce the expected value of the calculation amount, and depending on the position of the summary image next to the first reference image, the above-described method requires less calculation amount. The possibility cannot be denied. Hereinafter, the method of the modification will be described in detail.

  The system configuration example of the image processing apparatus is the same as that in FIG. 11, and the second reference image selection process (update process) in the second reference image selection unit 1007 is different. Therefore, detailed description of similar parts is omitted, and different points will be described.

  When the input image sequence is input, the reference image selection unit 1001 selects the first reference image. Here, as described above, the top image (first image) of the input image sequence is selected. When the input image sequence is an image sequence acquired by the image sequence acquisition unit 200 (when the first first reference image selection process is performed), an image other than the head (for example, a virtual 0th image) However, in the following description, the first reference image is assumed to be the first image unless otherwise specified.

  Thereafter, the second reference image is selected. Here, a second reference image selection section (actually equivalent to a range for searching for a summary image next to the first reference image) corresponding to an image to be selected for the second reference image is set. The half-open interval [i, j) corresponding to the i-th image to the j-th image is set as the second reference image selection interval, and i is made to correspond to the next image of the first reference image (i = 2 in a narrow sense). ), J = N + 2. Note that j = N + 2 is set even if a virtual N + 1th image is set as the second reference image, so that a virtual 0th image may be set as the first reference image. Because it is good. The case where the second reference image is the (N + 1) th image corresponds to the case where it is possible to cover all the images behind the first reference image alone and determine whether the second reference image is unnecessary. To do.

  Then, the second reference image is selected from the set second reference image selection section. Here, in order to perform processing efficiently, the second reference image is determined based on a given condition. First, when the second reference image is selected for the first time after setting the first reference image, the i + 1th image (third image in a narrow sense) is selected as the second reference image. This is the same as the basic method of the second embodiment.

  FIG. 20A illustrates the processing so far. Here, N = 12 image sequences are considered, the first reference image is the first image, the second reference image selection section is the second image to the 14th image (i = 2, j = 14). The second reference image is the third image.

  Since the determination target image selection process, the coverage rate calculation process, the deletion permission determination process, and the repetition of these processes after the second reference image is selected are the same as those described above, detailed description thereof is omitted. .

  If a given image (initially the third image) is selected as the second reference image and the image is OK, the position of the second reference image can be further separated from the first reference image. Since it is good, an image behind the current one is selected as a new second reference image. This idea itself is the same as the basic method of the second embodiment. However, although the second reference image is moved backward one by one in the above-described method, it may be shifted to two or more backward images here.

  As an example, if the current second reference image is the a-th image counted from the first reference image, the 2 × a-th image counted from the first reference image is replaced with the new second reference image. It may be an image. Specifically, as shown in FIG. 20B, when the third image (second from the first reference image) is selected as the second reference image, the third image is If OK, the next second reference image is selected as the fifth (fourth counted from the first reference image) image.

  By the way, if the q-th image is OK, it is not necessary to select the q−1-th and previous images as the summary images left in the summary image sequence. Therefore, since there is no merit to select an image ahead of the current position (qth) as the second reference image, the second reference image selection section may be updated. Specifically, the starting point i of the selected section may be i = q. In this modification, since the second reference image is selected from the second reference image selection section, this prevents the image ahead of the current position from being selected. For example, as shown in FIG. 20B, when the third image is OK, the second image is not a summary image, so it may be removed from the selected section, and the start point of the selected section is the third point. Update to an image.

  Similarly, if the fifth image is OK, the ninth image is selected as a new second reference image as shown in FIG. 20C, and the start point of the second reference image selection section is set to 5. Update to the second image.

  However, as can be understood from the case where the ninth image is OK in FIG. 20C, when the qth image is the second reference image and the qth image is OK, q If the value increases, the new second reference image may become extremely rearward. For example, the image behind N + 1 becomes a candidate and the second reference image cannot be selected, or the interval between the second reference images before and after the update becomes too wide, The search for the summary image becomes inefficient.

  Therefore, when selecting an image behind the current position as a new second reference image, another method may be used in combination. As an example, a new second reference image is determined based on the value of (q + j) / 2. For example, when the 9th image is OK, the start point of the second reference image selection section is updated to the 9th image, and thus the [9, 14) half-open section. That is, by setting the image near the center as a new second reference image, the center of the search range is set as a processing target. The method of halving the search range by determining the center of the search range is not limited to the widely known binary search, and it is also widely known that the binary search has an advantage in terms of computational complexity. Yes. The second reference image selection section of the present embodiment is that if a given image is OK, all images ahead of it may be considered OK, and if a given image is NG, the image behind it is All have the property of being considered NG, and a binary search technique can be applied. In other words, efficient processing can be expected by selecting a new second reference image from the vicinity of the middle between the second reference image before update and the end point of the second reference image selection section.

Here, the method of doubling the distance of the first reference image starting point and the method corresponding to the binary search are used together. For example, when the q-th image is the second reference image before update, the k-th image that satisfies the following expression (1) may be used as the next second reference image. Here, min (a, b) represents the smaller one of a and b.

  On the other hand, when the q-th image is NG as described above, a new second reference image is selected from the front of the current position, contrary to the case of OK. How much forward image is selected can be determined by various methods. For example, a method corresponding to the binary search may be used here. In this case, since the starting point of the second reference image selection section is the i-th image, a new second reference image is determined based on the value of (i + q) / 2. Further, as long as the qth image is NG, the qth image and the image behind it are not selected as the summary image. Therefore, the end point of the second reference image selection section may be updated, and j = q may be set. An example in which the ninth image is NG is shown in FIG. The seventh image is selected as a new second reference image, and the end point j of the second reference image selection section is updated to j = 9.

  Note that the reason why the second reference image selection section is a half-open section is for convenience of explanation here. That is, when the q-th image is OK, there remains a possibility that the q-th image is selected as a summary image. Therefore, when the starting point i of the second reference image selection section is i = q. , I may be included in the second reference image selection section. On the other hand, when the q-th image is NG, the q-th image is not selected as the summary image. Therefore, when the end point j of the second reference image selection section is j = q, j is the second reference image. It is better not to include it in the image selection section. From the above, the second reference image selection section is merely [i, j), and depending on the notation of the sign and the expression, there is nothing to represent the second reference image selection section by an open section or a closed section. No problem.

  With the above processing, the second reference image selection section (in the narrow sense, the search range of the next summary image) is narrowed. The next summary image is a k-th image when the k-th image is OK and the k + 1-th image is NG. Therefore, when a location where the OK image and the NG image are adjacent to each other is found. Processing will be terminated. In the above example, it is assumed that the process is performed in a binary search immediately before the end, for example, as shown in FIG. The i-th image is OK, the j-th image next to it is NG, and the q-th image between them is the second reference image. In this case, if the q-th image is OK, the image is as shown in FIG. 20F, and if it is NG, the image is as shown in FIG. 20G. In any case, the start point and the end point of the second reference image selection section are adjacent to each other. The image corresponding to the start point is OK, and the image corresponding to the end point is NG. Therefore, since the image corresponding to the start point may be selected as the next summary image, the search process for the input image sequence ends.

  If the next summary image is found, it is the same as the basic method of the second embodiment in that a partial image sequence including the image and subsequent images may be set as an input image sequence. Therefore, the partial image sequence setting unit 1008 sets the start point of the second reference image selection section and the subsequent images as the partial image sequence, and sets the partial image sequence as a new input image sequence. When a new input image sequence is given, the subsequent processing is the same, and detailed description thereof is omitted.

  FIG. 21 shows a flowchart for explaining this process. S601 and S602 are the same as S301 and S302. After the first reference image is selected in S602, a second reference image selection section is set (S609). As the processing of S609 performed immediately after S602, for example, as described above, a half-open section of [i, j) that satisfies i = 2 and j = N + 2 may be set. As will be described later, when the process of S609 is performed after S603 or S606, the second reference image selection section that has already been set is updated.

  If the second reference image selection section is set (or updated) in S609, it is determined whether the start point and the end point are adjacent (j = i + 1 is satisfied) (S610). In the case of Yes in S610, as shown in FIG. 20F, since the i-th image has been found to be a summary image next to the first image, the i-th image and the subsequent images Is set as a partial image sequence (S607), and the process returns to S602.

  If No in S610, the next summary image has not yet been found, so the second reference image is selected from the second reference image selection section set in S609 (S608). When the process of S608 is performed for the first time after the setting of the first reference image in S602, for example, the i + 1th image (two images behind the first reference image) may be selected. In other cases, a process of selecting a new second reference image is performed according to the position of the immediately preceding second reference image.

  When the second reference image is selected in S608, a determination target image is selected (S603). The coverage area calculation process (S604), the coverage rate calculation process (S605), and the image deletion possibility determination process (S606) after selection of the determination target image are the same as S304 to S306. If it is determined in step S606 that the image can be deleted, the process returns to step S603, the determination target image is updated to the next image, and the same processing is performed. By repeating the processing of S603 to S606, it is determined whether all the images between the first reference image and the second reference image can be deleted, or at least one of them cannot be deleted. If all can be deleted, it is determined in S603 that the determination target image is the second reference image, and the process returns to S609. If at least one image cannot be deleted, the determination in S606 indicates that the image cannot be deleted, and the process returns to S609. Although not shown in FIG. 21, it is necessary to retain information indicating whether the process has returned from S603 to S609 or from S606 to S609, and the process in the next S609 or the like needs to be changed accordingly. is there.

  When the process returns from S603 to S609, all the images can be deleted. Therefore, a process for updating the start point of the second reference image selection section is performed. As a result, the previous second reference is selected in S608. An image behind the image is selected as a new second reference image. On the other hand, when returning from S606 to S609, since at least one image cannot be deleted, a process of updating the end point of the second reference image selection section is performed. An image ahead of the second reference image is selected as a new second reference image.

  In the above modification, when the pth image is selected as the first reference image and the qth image is selected as the second reference image from the input image sequence including the first to Nth images, The processing unit 100 selects the second reference image from the second reference image selection section in which the start point and the end point corresponding to the p + 2 to Nth images are set. Then, it is determined whether the determination target image can be deleted based on the first reference image and the second reference image. If it is determined that the p + 1 to q−1th images can be deleted, The xth image (x is an integer satisfying x> q) included in the reference image selection section is selected as a new second reference image. At the same time, the starting point of the second reference image selection section may be updated to the qth image.

  Here, the second reference image selection section includes the (p + 2) th to Nth images in view of the property of being an image that is a candidate for the second reference image. However, since a virtual image such as the (N + 1) th image may be selected as the second reference image, the end point of the second reference image selection section may be larger than N. In addition, since the second reference image selection section also has an aspect of a search range for the next summary image, an image that can be selected as a summary image is included in the selection section even if the image is not selected as the second reference image. It may be included. In that case, an image (p + 1-th image) one behind the first reference image may be set as the starting point of the second reference image selection section.

  Thereby, when the second reference image is updated, the position of a new second reference image can be determined flexibly. In other words, the basic method of the second embodiment is a method of checking the search range one by one from the top and reducing the search range, so that the amount of calculation becomes very large depending on the position of the correct answer. End up. In that respect, by making it possible to select images that are not adjacent to each other as a new second reference image, the search range can be greatly reduced by one unit of determination (determination of whether the qth image is OK or NG). Therefore, an effect of reducing the amount of calculation can be expected, and the load on the system can be reduced and the processing time can be shortened.

  In addition, when it is determined that at least one of the p + 1 to q−1th images cannot be deleted, the processing unit 100 determines that the yth (y is y <q) included in the second reference image selection section. An integer that satisfies (integer integer) may be selected as a new second reference image. At the same time, the end point of the second reference image selection section is updated to the qth image.

  As a result, when the second reference image is updated, an image ahead of the current second reference image can be selected as a new second reference image. As described above, as long as the backward search is not limited to selecting adjacent images, an unsearched range may remain ahead of the current second reference image. Can be considered that there is a correct answer in the unsearched range. In that case, it is possible to perform appropriate processing by performing a forward search. Further, as in the backward search, selection of a new second reference image is not limited to adjacent images.

  Further, the processing unit 100 may set the value of x based on the value of (q + j) / 2 when the j-th image (j is an integer) corresponds to the end point of the second reference image selection section. Good. Alternatively, when the i-th (i is an integer) image corresponds to the start point of the second reference image selection section, the value of y may be set based on the value of (i + q) / 2.

This makes it possible to use a binary search method in selecting a new second reference image. In the case of a backward search, an image that is intermediate between the current second reference image and the end point is selected, and in the case of a forward search, an intermediate between the current second reference image and the start point is selected. Will be selected. Therefore, the search range (corresponding to the length of the second reference image selection section) can be halved. If logN images are selected as the second reference image, the entire search range can be searched. Expected to end. Therefore, the order of the calculation amount can be suppressed to N × logN, and when N is very large, the calculation amount is reduced as compared with the basic method of the second embodiment (the order of calculation amount is N ² ). Great effect. Since (q + j) / 2 and (i + q) / 2 are not necessarily integers, there may be cases where there is no image corresponding to each value. In that case, for example, a maximum integer not exceeding (q + j) / 2, or an integer larger by 1 may be considered.

  In addition, the processing unit 100 performs processing for including the image selected as the first reference image in the summary image sequence when the start point and the end point are adjacent as a result of updating the start point or the end point of the second reference image selection section. May be performed. Also, an image corresponding to the start point and a partial image sequence consisting of images behind the image corresponding to the start point are set as the input image sequence, and the value of p is set for the set input image sequence. The processing may be performed again by setting it to 1.

  Here, that the start point and the end point are adjacent indicates that the image corresponding to the start point and the image corresponding to the end point are adjacent in the input image sequence. When N images are given as an input image sequence, it is assumed that the input image sequence is a set of images that are continuous in time series or spatially. You can define forward and backward. For example, an image acquired at an earlier time in time series is an image ahead of an image acquired at a later time. Specifically, each image in the input image sequence is represented as first to Nth images, and an image with a smaller shaken numerical value is assumed to be in front. Therefore, the i-th image and the j-th (> i) image in the image sequence are adjacent to each other in a situation where j = i + 1 is satisfied.

  This makes it possible to set a condition based on the start point and end point (or length) of the second reference image selection section as the process end condition for the input image sequence. By setting such an end condition, an image that is predicted to be the farthest from the first reference image in the image group determined to be OK when selected as the second reference image It can be selected as the first image (corresponding to the next summary image) of the image sequence. This is because, as shown in FIG. 20F and the like, this end condition is equivalent to searching for a position where an OK image and an NG image are adjacent to each other. Therefore, the number of summary images included in the finally output summary image sequence can be reduced, and the burden on the user can be reduced.

3.3 Modification (Initial setting of second reference image)
In the above second embodiment and its modifications, an input image sequence (which may be an image sequence acquired by the image sequence acquisition unit 200 or a partial image sequence consisting of a part of the images) is input. In this case, the second reference image that is initially set is limited to an image that is two images behind the first reference image.

  However, the initial position of the second reference image may be different. For example, in an actual image sequence, a section in which similar images continue for a long time and a section in which there are not many similar images are not often adjacent. In other words, the length of the next summary section (representing how far the adjacent summary images are separated) is predicted to be close to the length of the previous summary section. Therefore, if a plurality of summary images have already been obtained and information corresponding to “the length of the previous summary section” has been acquired, the initial position of the second reference image is determined from the first reference image. By setting it at a position separated by “the length of the previous summary section”, it is expected that the correct answer will be reached sooner, and a reduction in the amount of calculation can also be expected.

  Specifically, the length g of the summary section is acquired from the immediately preceding summary image and the previous summary image. Then, when the second reference image selection section is [i, j), the second reference image may be set to the i + g image instead of the i + 1th image. Note that g cannot be acquired when the number of summary images already determined is zero or one. In this case, the initial position of the second reference image is set without using g. For example, if there are no summary images, the second reference image may be the (i + 1) th image as described above. If there is one summary image, the length from the first image in the image sequence acquired by the image sequence acquisition unit 200 to the summary image is g ′, and the first second reference image is i + g. It may be '

  Note that when the second reference image needs to be updated, various updating methods can be considered. For example, the next second reference image may be selected using a binary search method as in the above-described modification.

  However, in view of the assumption that there is a high possibility that the next summary image is likely to be in the vicinity of the i + g image, the fact that the updated second reference image is far away from the i + g image is until the next summary image is found. It may increase the number of searches and may be considered undesirable. In that case, as described as the basic method of the second embodiment, an image adjacent to the previous second reference image may be selected as a new second reference image. However, as long as the determination is not made for the (i + 1) -th to i + g-1 images, there is a possibility that the next summary image exists in the range. Therefore, the update of the second reference image is not limited to moving backward one by one, and may be shifted forward one by one depending on the determination result of deletion availability.

4). Third Embodiment In the third embodiment, a technique will be described in which image summarization processing that guarantees the coverage of the front and rear images of the reference image is performed in order of sequence. FIG. 15 shows a system configuration example of the image processing apparatus in the present embodiment. An image sequence dividing unit 1006 is added to the processing unit 100 of FIG.

  In the present embodiment, the reference image selection unit 1001 selects a reference image from images other than the first and last images in the input image sequence (this is not necessarily the case if the number of images in the input image sequence is less than 3). In a narrow sense, the center image of the input image sequence may be used as the reference image. The determination target image selection unit 1002 sequentially selects all images other than the reference image among images included in the input image sequence.

  The image sequence dividing unit 1006 divides the input image sequence into a first image sequence composed of images before the reference image and a second image sequence composed of images after the reference image. The partial image sequence setting unit 1008 performs the partial image sequence setting process based on the determination result of whether or not the image included in the first image sequence can be deleted, and the determination result of whether or not the image included in the second image sequence can be deleted. The partial image sequence setting process is performed based on the above.

  FIG. 16 shows a flowchart for explaining the image summarization processing of this embodiment. When this process is started, first, an image sequence to be subjected to image summarization processing is acquired by the image sequence acquisition unit 200 in the same manner as in S101 and the like (S501). Then, the image sequence acquired by the image sequence acquisition unit 200 is added to the input image sequence group.

  Next, processing described later with reference to FIG. 17 is executed on the input image sequence included in the input image sequence group (S502). When the process of S502 is performed for the first time, the process is performed on the image sequence acquired in S501. Note that the input image sequence to be processed in S502 is deleted from the input image sequence group. Thereafter, it is determined whether or not the input image sequence remains in the input image sequence group (S503). As will be described later, in the process in S502, one or two partial image sequences are set depending on circumstances, and the set partial image sequence is added to the input image sequence group. Therefore, Yes is determined in S503. It is possible. In the case of No in S503, the process ends.

  FIG. 17 is a flowchart for explaining the processing in S502 of FIG. When this process is started, first, an input image sequence to be processed is selected from the input image sequence group (S401). S402 to S406 are generally the same as S102 to S106, although there are differences in the selection position of the reference image. In the present embodiment, regardless of the determination result of whether or not deletion is possible, all images (except for the reference image) included in the input image sequence are sequentially selected and processed. When the determination of whether or not deletion is possible is completed for all the images, it is determined in S403 that there is no image to be selected, and the process proceeds to S409.

  In step S409, the input image sequence is divided into a first divided image sequence composed of images before the reference image and a second divided image sequence composed of images after the reference image. Then, partial image sequence setting processing is performed based on the first divided image sequence and the second divided image sequence (S407). Specifically, in S407, first, it is determined whether there is at least one image determined to be undeleteable in S406 in the images included in the first divided image sequence (S4071). The first divided image sequence is set as a partial image sequence and added to the input image sequence group (S4072). Similarly, it is determined whether there is at least one image determined to be undeleteable in S406 in the images included in the second divided image sequence (S4073). The divided image sequence is set as a partial image sequence and added to the input image sequence group (S4074). That is, in the process of S502, 0 to 2 partial image sequences are added to the input image sequence group as a result of processing performed using one input image sequence as an input.

  FIGS. 18A to 18B illustrate the above image summarization processing. As shown in FIG. 18A, when an image sequence having N images is acquired by the image sequence acquisition unit 200, the image sequence is added to the input image sequence group, and is processed in S502. It becomes. In S502, an x-th image (2 ≦ x ≦ N−1, and in a narrow sense, x is a value close to N / 2) is first selected as a reference image, and the 1st to (x−1) th images and x + 1 to 1 are selected. The Nth image is sequentially selected as a determination target image, and whether or not deletion is possible is determined.

  If it is determined that at least one of the 1-x-1th images cannot be deleted, all of the 1-x-1th images cannot be covered with the reference image. Therefore, it is necessary to set the reference image in the 1st to (x-1) th images. Specifically, the process of S502 is executed again with the 1st to (x-1) th images as the divided image sequence and the divided image sequence as the input image sequence. For example, as shown in FIG. 18B, y-th (2 ≦ y ≦ x−2) of the image sequence composed of x−1 images, and y in the narrow sense is (x−1) / An image having a value close to 2 is selected as a reference image, and other images are sequentially selected as determination target images to determine whether or not deletion is possible. As a result, in some cases, a divided image sequence consisting of the 1st to (y-1) th images and a divided image sequence consisting of the (y + 1) to (x-1) th images are added to the input image sequence group, and are subject to the processing in S502. . Thereafter, it is determined that all the images included in the divided image sequence can be deleted, and the process of S502 is repeated until the divided image sequence cannot be included in the summary image sequence.

  Further, the same processing is performed on the divided image sequence including the (x + 1) to (N + 1) th images as shown in FIG.

  In the present embodiment described above, when the first to Nth (N is an integer of 2 or more) images are input as the input image sequence, the processing unit 100 determines that the sth (s is 2 ≦ s ≦ N−1). An image satisfying (integer satisfying) is selected as the reference image, and a t-th image (t is an integer satisfying 1 ≦ t ≦ N and t ≠ s) is selected as the determination target image. Then, the coverage is calculated based on the deformation information between the reference image and the determination target image, and whether or not the determination target image can be deleted is determined based on the coverage. Further, when it is determined that at least one of the first to s−1 images cannot be deleted, a partial image sequence including the first to s−1 images is set as an input image sequence and processed again. I do. If it is determined that at least one of the s + 1 to Nth images cannot be deleted, a partial image sequence including the s + 1 to Nth images is set as an input image sequence, and the process is performed again.

  As a result, the processing shown in FIGS. 18A and 18B becomes possible. Here, the s-th image is preferably an image near the center of the input image sequence. That is, the input image sequence is divided into two divided image sequences with the reference image as a boundary, and whether or not deletion is possible is determined for each. If the deletion is not possible, the divided image sequence is set as the input image sequence. Therefore, in the present embodiment, when one input image sequence is input, two new input image sequences may be set as outputs. In this embodiment, when it is determined that at least one image in the divided image sequence cannot be deleted, the entire divided image sequence is set as a new input image sequence, and a reference image is selected from the set. That is, even if it is determined that a given image can be deleted, the deletion is not always confirmed immediately. When a divided image sequence including the given image is set as a new input image sequence, It should be noted that a given image may be selected as a reference image and eventually included in the summary image sequence (ie not deleted by the image summarization process).

  The three embodiments 1 to 3 to which the present invention is applied and the modifications thereof have been described above, but the present invention is not limited to the embodiments 1 to 3 and the modifications as they are, The constituent elements can be modified and embodied without departing from the spirit of the invention. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described first to third embodiments and modifications. For example, some constituent elements may be deleted from all the constituent elements described in the first to third embodiments and the modifications. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. In addition, a term described together with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term anywhere in the specification or the drawings. Thus, various modifications and applications are possible without departing from the spirit of the invention.

100 processing unit, 200 image sequence acquisition unit, 300 storage unit, 1001 reference image selection unit,
1002 Determination target image selection unit, 1003 coverage area calculation unit, 1004 coverage rate calculation unit,
1005 Deletability determination unit, 1006 Image sequence division unit,
1007 Second reference image selection unit, 1008 Partial image sequence setting unit,
1009 Summary image sequence determination unit

Claims (24)

An image sequence acquisition unit for acquiring an image sequence having a plurality of images;
A processing unit that performs image summarization processing for acquiring a summary image sequence by deleting a part of the plurality of images of the image sequence acquired by the image sequence acquisition unit;
Including
The processor is
A reference image and a determination target image are selected from the plurality of images,
Based on deformation information between the reference image and the determination target image, the coverage of the determination target image by the reference image is calculated,
An image processing apparatus that determines whether or not the determination target image can be deleted based on the coverage. In claim 1,
When the first to Nth images (N is an integer of 2 or more) are input as the input image sequence,
The processor is
The pth (p is an integer satisfying 1 ≦ p ≦ N) image is selected as the first reference image, the qth image (q is an integer equal to or greater than p + 2) is selected as the second reference image, and the first an image of r (r is an integer satisfying p + 1 ≦ r ≦ q−1) is selected as the determination target image;
Calculating the coverage based on the deformation information between the first reference image and the determination target image and the deformation information between the second reference image and the determination target image; An image processing apparatus that determines whether or not the determination target image can be deleted based on the image. In claim 2,
The processor is
The second reference image is selected from the second reference image selection section in which the start point and the end point corresponding to the p + 2 to Nth images are set, and the first reference image and the second reference image are selected. Based on the determination whether the determination target image can be deleted,
When it is determined that the (p + 1) -th to (q-1) -th images can be deleted, an x-th image (x is an integer satisfying x> q) included in the second reference image selection section is newly added to the first image. The image processing apparatus is selected as a second reference image and updates the start point of the second reference image selection section to the qth image. In claim 3,
The processor is
The x value is set based on the value of (q + j) / 2 when the j-th image (j is an integer) corresponds to the end point of the second reference image selection section. Processing equipment. In claim 3 or 4,
The processor is
If it is determined that at least one of the p + 1 to q−1 images cannot be deleted, the y th image (y is an integer satisfying y <q) included in the second reference image selection section. Is selected as a new second reference image, and the end point of the second reference image selection section is updated to the qth image. In claim 5,
The processor is
The y value is set based on the value of (i + q) / 2 when the i-th image (i is an integer) corresponds to the start point of the second reference image selection section. Processing equipment. In any one of Claims 3 thru | or 6.
The processor is
As a result of updating the start point or the end point of the second reference image selection section, when the start point and the end point are adjacent,
A process of including one of the plurality of images selected as the first reference image in the summary image sequence;
One of the plurality of images corresponding to the start point, and among the plurality of images, a partial image sequence composed of a rear one in the input image sequence than one of the plurality of images corresponding to the start point An image processing apparatus that is set as an input image sequence and performs the process again with the p value set to 1 for the set input image sequence. In claim 2,
The processor is
An image processing apparatus, wherein when it is determined that the (p + 1) -th to (q-1) -th images can be deleted, the (q + 1) -th image is selected as the second reference image. In claim 8,
The processor is
If it is determined that at least one of the p + 1 to q−1 images cannot be deleted, one of the plurality of images selected as the first reference image is included in the summary image sequence. And a partial image sequence composed of the q-1st to Nth images is set as the input image sequence, and the value of p is set to 1 for the set input image sequence. An image processing apparatus that performs processing again. In claim 1,
When the first to Nth images (N is an integer of 2 or more) are input as the input image sequence,
The processor is
A first image is selected as the reference image, and a k-th image (k is an integer satisfying 2 ≦ k ≦ N−1) is selected as the determination target image;
Calculating the coverage based on deformation information between the reference image and the determination target image, determining whether the determination target image can be deleted based on the coverage,
When it is determined that the k-th image can be deleted, an k + 1-th image is selected as the determination target image. In claim 10,
The processor is
If it is determined that the k-th image cannot be deleted, a process of including one of the plurality of images selected as the reference image in the summary image sequence is performed, and the k-th to N-th images are performed. An image processing apparatus, wherein a partial image sequence composed of images is set as the input image sequence, and the set input image sequence is processed again. In claim 1,
When the first to Nth images (N is an integer of 2 or more) are input as the input image sequence,
The processor is
The s-th (s is an integer satisfying 2 ≦ s ≦ N−1) image is selected as the reference image, and the t-th (t is an integer satisfying 1 ≦ t ≦ N and t ≠ s) image is determined. Select as target image,
An image processing apparatus that calculates the coverage based on deformation information between the reference image and the determination target image, and determines whether the determination target image can be deleted based on the coverage. In claim 12,
The processor is
In the determination of whether or not deletion is possible, if it is determined that at least one of the first to s-1th images cannot be deleted, a partial image sequence including the first to s-1th images is input. An image processing apparatus which is set as an image sequence and performs the process again on the set input image sequence. In claim 12,
The processor is
In the determination of whether or not deletion is possible, if it is determined that at least one of the s + 1 to Nth images cannot be deleted, a partial image sequence including the s + 1 to Nth images is set as the input image sequence. An image processing apparatus that performs the process again on the set input image sequence. In any one of Claims 1 thru | or 14.
The processor is
For the reference image, the determination target image, and the plurality of images between the reference image and the determination target image in the image sequence, the deformation information between the adjacent images is obtained, and the obtained adjacent ones are obtained. An image processing apparatus that obtains the deformation information between the reference image and the determination target image based on the deformation information between the images. In any one of Claims 1 thru | or 15,
The processor is
Based on deformation information between the reference image and the determination target image, a covering region that is a region where the determination target image is covered by the reference image is obtained, and the covering region occupies the determination target image as the covering rate An image processing apparatus that calculates a ratio of In claim 16,
The processor is
As the reference image, first to Mth reference images (M is an integer of 2 or more) are selected from the plurality of images,
Obtaining a u-th covering region based on deformation information between the u-th (1 ≦ u ≦ M) reference image and the determination target image;
An image processing apparatus, wherein an union area of first to Mth covering areas is set as the covering area, and the coverage is calculated based on the set covering area. In claim 16 or 17,
The processor is
A weighting factor is set according to the position of the determination target image on the image, based on the first weight sum calculated based on the weighting factor and the coverage area, and on the weighting factor and the determination target image. An image processing apparatus that calculates the coverage based on a ratio with the calculated second total weight. In any one of Claims 1 thru | or 15,
The processor is
When a plurality of points are set for the determination target image and the plurality of points are converted according to deformation information between the reference image and the determination target image, the converted image included in the reference image An image processing apparatus that calculates the coverage based on the number of points. In claim 19,
The processor is
As the reference image, first to Mth reference images (M is an integer of 2 or more) are selected from the plurality of images,
When the plurality of points are converted according to deformation information between the u-th (1 ≦ u ≦ M) reference image and the determination target image, the converted points included in the u-th reference image The number is obtained as the u-th covering information,
An image processing apparatus that calculates the coverage based on first to Mth cover information obtained from each of the first to Mth reference images. In claim 19 or 20,
The processor is
A weighting factor is set according to the position of the determination target image on the image, and the plurality of points are converted according to deformation information between the reference image and the determination target image, and are included in the reference image. An image processing apparatus that calculates the coverage based on a point after conversion and the weighting factor. In claim 18 or 21,
The processor is
As the value of the weighting factor, 0 is set in the first area of the determination target image, and 1 is set in a second area different from the first area of the determination target image. Image processing device. An image sequence acquisition unit for acquiring an image sequence having a plurality of images;
As a processing unit that performs image summarization processing for acquiring a summary image sequence by deleting a part of the plurality of images of the image sequence acquired by the image sequence acquisition unit,
Make the computer work,
The processor is
A reference image and a determination target image are selected from the plurality of images,
Based on deformation information between the reference image and the determination target image, the coverage of the determination target image by the reference image is calculated,
A program for determining whether or not the determination target image can be deleted based on the coverage. Obtain an image sequence with multiple images,
A reference image and a determination target image are selected from the plurality of images in the image sequence,
Based on deformation information between the reference image and the determination target image, the coverage of the determination target image by the reference image is calculated,
Based on the coverage, it is determined whether the determination target image can be deleted,
An image processing method for performing image summarization processing for acquiring a summary image sequence by deleting a part of the plurality of images of the image sequence based on the determination result of the deletion possibility.

Images