JP2011160354A - Image processing device, image processing method, and program therefor - Google Patents

Abstract

An object of the present invention is to improve the accuracy of splicing between a plurality of divided image data and generate panoramic image data with higher accuracy than before.
An image processing apparatus acquires panoramic image data of a subject (step S201) and acquires divided image data of the subject (step S202). Then, the image processing apparatus calculates a difference between each divided image data and the whole-view image data in an area corresponding to the divided image data (step S206), and performs a correction process according to the difference on each divided image data. (Step S207). Next, the image processing apparatus generates panoramic image data by combining the corrected divided image data (step S212).
[Selection] Figure 5

Description

  The present invention relates to a technique for generating panoramic image data using a plurality of image data.

  Conventionally, as a technique for executing panorama synthesis processing using image data of a plurality of images, a technique for extracting corresponding points between a plurality of image data to be combined and combining the corresponding points is known. (For example, refer to Patent Document 1). In addition, there is a technique for automatically determining a rough position of a plurality of pieces of image data corresponding to a partial image of a subject by photographing an entire view of the subject in advance (see, for example, Patent Document 2).

JP 2000-312287 A Japanese Patent Laid-Open No. 5-130339

  However, in the method of extracting corresponding points as disclosed in Patent Document 1, there is a case where panorama synthesis fails. This can occur when a subject in which a specific pattern (pattern) continues is photographed. That is, since there are a plurality of similar patterns in the captured image data, the correlation between corresponding points cannot be obtained between the two captured image data. For this reason, there is a problem in that the corresponding corresponding points are connected to each other and the panoramic image data is greatly distorted. In addition, even when the arrangement of each captured image data is determined using the method disclosed in Patent Document 2, there is a distortion of a lens such as a barrel distortion or a pincushion distortion when an object is photographed from an oblique direction. Therefore, it is difficult to synthesize accurately.

  Therefore, an object of the present invention is to improve the accuracy of joining between a plurality of divided image data, and to generate panoramic image data with higher accuracy than before.

  The image processing apparatus according to the present invention is acquired by a first acquisition unit that acquires panoramic image data of a subject, a second acquisition unit that acquires divided image data of the subject, and the second acquisition unit. A calculation unit that calculates a difference value between each divided image data and the panoramic image data in an area corresponding to the divided image data, and a correction process according to the difference value calculated by the calculation unit And a generating unit that generates panoramic image data by combining the divided image data corrected by the correcting unit.

  According to the present invention, by using the panoramic image data of the subject, it is possible to improve the accuracy of joining between a plurality of divided image data, and it is possible to generate panoramic image data with higher accuracy than before. .

1 is a block diagram illustrating a system configuration of an image processing apparatus according to a first embodiment. It is a figure for demonstrating the outline of imaging | photography operation | movement of a panoramic image. It is a figure for demonstrating the outline of imaging | photography operation | movement of a panoramic image. It is a figure for demonstrating the outline | summary of a perspective correction process and a mesh correction process. It is a figure for demonstrating the outline | summary of a perspective correction process and a mesh correction process. It is a figure for demonstrating the method of detecting the optimal parameter in a perspective correction process and a mesh correction process. It is a figure for demonstrating the method of detecting the optimal parameter in a perspective correction process and a mesh correction process. It is a flowchart which shows the panoramic image data synthetic | combination process in 1st Embodiment. It is a figure for demonstrating the phenomenon in which sufficient accuracy cannot be obtained even if it calculates | requires the difference value of whole view image data and divided image data, since the whole view image data is in the state expanded. It is a flowchart which shows the synthesis process of the panoramic image data in 2nd Embodiment. It is a figure for demonstrating the outline of a projector projection system. It is a flowchart which shows the synthesis process of the panorama image data in 3rd Embodiment. It is a figure which shows the result of an edge extraction process. It is a figure for demonstrating the outline of the synthesizing process of the panorama image data in 4th Embodiment. It is a flowchart which shows the synthesis | combination process of the panorama image data in 4th Embodiment.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  First, the first embodiment will be described. In the present embodiment, a method for generating panoramic image data based on full-view image data will be described.

  FIG. 1 is a block diagram illustrating a system configuration of the image processing apparatus according to the first embodiment. The input device 101 is a device for inputting instructions and data from a user, and includes a pointing system such as a keyboard and a mouse. The display device 102 is a device that displays a GUI or the like, and usually represents a monitor such as a CRT or a liquid crystal display. The storage device 103 is a device for storing image data and programs, and normally a hard disk is used. Reference numeral 104 denotes a CPU, which executes control related to all the processes of the above-described components. The RAM 106 provides programs, data, work areas, and the like necessary for the processing to the CPU 104. Further, it is assumed that the control program necessary for the processing of the subsequent flowcharts is stored in the storage device 103, and is once executed after being read into the RAM. The camera device 105 is connected to a digital camera or the like and acquires input image data. As for the system configuration, there are various components other than the above, but a description thereof will be omitted.

  FIGS. 2A and 2B are diagrams for explaining an outline of the panoramic image shooting operation. FIG. 2-1 (a) shows a subject when generating panoramic image data. In the present embodiment, high-resolution panoramic image data for a picture as shown in FIG. FIG. 2-1 (b) shows a method for capturing a panoramic image of the subject shown in FIG. 2-1 (a). In this case, the camera is placed at a position away from the subject so that the entire area of the subject is within the angle of view of the camera, or the camera lens is replaced with a lens with a wider angle. FIG. 2-1 (c) shows a method for capturing a divided image. First, a camera is installed so as to capture the center of the subject. Then, the telephoto of the lens is adjusted so that the necessary area of the subject can be photographed, or the lens is exchanged for photographing. The angle of view at this time depends on the resolution of the panoramic image data to be created. When the resolution of panoramic image data is high, the number of divisions at the time of shooting increases. Therefore, it is necessary to reduce the angle of view. Conversely, when the resolution of panoramic image data is low, the number of divisions at the time of shooting may be small, so the angle of view is increased. Next, the camera is rotated along two rotation axes. Actually, the camera is rotated by installing the camera on a rotatable turntable and rotating the turntable. Since the turntable is equipped with a tachometer, the rotation angle of the camera can also be acquired. By rotating the camera, it is possible to photograph not only the central part of the subject but also the entire subject. Specifically, shooting is started from the upper left of the subject, and the entire subject is shot while gradually rotating. The rotation angle at this time depends on the number of divisions at the time of shooting described above. When the number of divisions is small, the rotation angle is large, and when the number of divisions is large, the rotation angle is small. At this time, the rotation angle is adjusted so that overlapping regions (overlapping regions) are formed between adjacent divided images. A specific example of adjustment is shown below.

  Here, as an example, consider the case where the desired panoramic image data resolution is 7000 × 5000, the camera resolution is 3000 × 3000, and 1/3 of the divided image data is used as the overlap region. At this time, when the two pieces of divided image data are combined in the horizontal direction, the divided image data 301 and 302 overlap each other by 1/3 (1000 pixels) as shown in FIG. As a result, the size of the composite image data is 5000 × 3000. When the number of divisions is calculated based on this size, the number of divided images is six as in the division result 304. In this embodiment, the entire area of the subject is photographed by rotating the camera. However, an imaging system in which the camera itself translates vertically and horizontally by a rail or the like may be used. It does not matter even if it is a system. FIG. 2-2 (a) shows the result of taking a panoramic image using the method of FIG. 2-1 (b). In this case, since there is lens distortion called barrel distortion or pincushion distortion, the distortion is corrected in advance using an existing method. It should be noted that since the panoramic image is stored in one piece of image data, the resolution of the panoramic image data is not higher than the resolution of the divided image data.

  FIG. 2-2 (b) shows the result of taking a divided image using the method of FIG. 2-1 (c). A divided image taken in three vertical and horizontal divisions is shown. Since each piece of divided image data has a lens distortion similar to the whole-view image data, it is necessary to correct in advance using an existing method. Further, as shown in the right diagram of FIG. 2-2 (c), in each divided image data, a phenomenon in which vertical lines and horizontal lines are distorted in an oblique direction is seen. This is because the subject is photographed from an oblique direction by rotating the camera. Such distortion also occurs when the subject and the camera cannot be installed in parallel. A process for correcting such distortion is called a parse correction. By performing the parse correction, a plurality of divided images can be synthesized with high accuracy. Further, in order to remove the influence of the minute movement of the subject that cannot be corrected only by the perspective correction, it is necessary to execute mesh correction processing for dividing each divided image data into meshes and minutely deforming the mesh.

FIGS. 3A and 3B are diagrams for explaining the outline of the perspective correction process and the mesh correction process. FIG. 3A illustrates the perspective correction process. Reference numeral 403 denotes a subject, and a case where the subject 401 is photographed by the camera 401 is considered. The camera 401 is photographing the subject 403 from an oblique direction. The relationship between the pixel position of the image data and the subject position at this time can be expressed by the following Expression 1.
ImagePos (X, Y) = Screen (Proj * View * Pos (x, y, z)) Equation 1
Here, Pos (x, y, z) indicates the three-dimensional position of the subject. View is a viewpoint conversion matrix and indicates the position of the camera 401 and the shooting direction. Proj is a projection matrix from 3D to 2D, and indicates the angle of view of the camera. Screen () is a function used to convert the coordinate system. X and Y indicate a two-dimensional arrangement of image data and are represented by integer values. A virtual camera 402 is installed in the front direction with respect to the subject 403. The perspective correction process can be performed by reproducing the image data captured by the virtual camera 402. Also in the virtual camera 402, the relationship between the pixel position of the image data and the subject position can be expressed by the following Expression 2.
ImageVPos (X, Y) = Screen (Proj * ViewV * Pos (x, y, z)) Equation 2
ViewV is a viewpoint conversion matrix, and indicates the position and shooting direction of the virtual camera 402. Parameters other than ViewV are the same as those in Equation 1. Here, by merging Equation 1 and Equation 2,
ImageVPos (X, Y) = Func (ImagePos (X ₀ , Y ₀ ))...
Is obtained. Func () is a function that associates ImageVPos () with ImagePos (), and is obtained from Equation 1 and Equation 2. X ₀ and Y ₀ are the coordinates of the image data of the camera 401 corresponding to X and Y, and take real values. Now, if the pixel value of ImagePos (X ₀ , Y ₀ ) is known, the pixel value of ImageVPos (X, Y) can be set. That is, the captured image data of the virtual camera 402 can be reproduced. At this time, since X ₀ and Y ₀ are real values, interpolation processing is required between the real values when obtaining pixel values.

  In FIG. 3B, camera image data 404 is image data captured by the camera 401. In the figure, the gray region which is the subject region is distorted in a trapezoidal shape. The virtual camera image data 405 is obtained by reproducing the captured image data photographed by the virtual camera 402 using Expression 3. It can be seen that the distortion of the subject area is corrected. Theoretically, if lens correction and perspective correction are performed, image data can be synthesized simply by arranging divided image data. However, in actuality, it cannot be synthesized as it is because of a rotation error of the swivel base. Therefore, it is necessary to correct parameters such as the camera rotation angle at the time of perspective correction to optimum values. Furthermore, in order to remove the influence of the minute movement of the subject, the mesh correction shown in FIG. The mesh correction is a process for minutely deforming each divided image data. The divided image data is divided into fine meshes, and each lattice point of the mesh is moved in an arbitrary direction to correct a fine shift at the time of synthesis. FIG. 3-2 (b) shows a method for acquiring the shooting position of the divided image. It is assumed that the distance between the camera and the subject is L, and the rotational frequency of the camera is θ in the horizontal direction and φ in the vertical direction. At this time, since the center of the divided image faces the point (L × tan θ, L × tan φ), the photographing position can be determined using these values. At the same time, it is possible to associate the position with the panoramic image.

  FIGS. 4A and 4B are diagrams for explaining a technique for detecting an optimum parameter in the perspective correction process and the mesh correction process. As a flow of the entire processing, first, rough correction is performed by the perspective correction, and then fine adjustment is performed by the mesh correction. Therefore, first in FIG. 4A, the CPU 104 determines the optimum parameter for the perspective correction. Consider a case where there are divided image data 1102 and panoramic image data 1101 corresponding to the divided image data 1102. First, the CPU 104 inputs the rotation angle of the camera obtained from the tachometer of the swivel base and the distance information between the subject and the camera acquired by laser measurement or the like as parameters, and executes the perspective correction on the divided image data 1102. Then, divided image data 1104 is generated. Then, the CPU 104 generates difference image data 1103 obtained by taking the difference between the divided image data 1104 and the panoramic image data 1101. In the difference image data 1103, the black area indicates that the difference value is 0, and the non-black area indicates that the difference value is not 0. Theoretically, by performing the parse correction, the difference image data should be all black areas. However, in reality, when the accuracy of the swivel base is poor, the rotational angle is shifted, so that a region other than black also occurs. Therefore, it is necessary to correct the rotation for the amount of deviation. Therefore, the CPU 104 corrects the deviation by changing the parameter so that the rotation angle of the camera obtained from the tachometer tachometer varies before and after the rotation angle.

  First, the CPU 104 performs perspective correction on the divided image data 1102 using a certain changed parameter, and generates divided image data 1105. At the same time, the CPU 104 generates difference image data 1106 by taking the difference between the divided image data 1105 and the panoramic image data 1101. Then, it can be seen that the difference image data 1106 has more black areas than the difference image data 1103. That is, the difference image data 1106 has a smaller sum of difference values. After that, it is possible to correct the rotational angle deviation by repeating the perspective correction while changing the parameters, and determining the parameter with the smallest sum of the difference values. In most cases, the difference image data is all black in this state. However, when the subject is moving minutely due to air-conditioning wind or the like, an area other than black may occur even if the perspective correction is performed. Therefore, mesh correction as shown in FIG. The left figure shows a state in which the panoramic image data and the divided image data are superimposed before mesh correction. Each point where the vertical line and the horizontal line intersect is a mesh lattice point, and the position of each lattice point can be calculated from the position of the divided image data. In the left figure, there is a difference between the contour 1107 of the divided image data and the contour 1108 of the full-view image data. Therefore, if difference image data is generated in this state, the sum of the difference values becomes large. Therefore, when the mesh lattice point 1109 near the portion where the contour is shifted is moved as shown in the right figure, the contours of the divided image data and the whole-view image data coincide. That is, the sum of the difference values is reduced. In this way, by moving the grid points of each mesh and determining the grid point position where the sum of the difference values is minimized, the influence of the subject moving slightly can be corrected. In this way, the adjustment accuracy can be improved by adjusting so that the difference value of each correction is minimized.

FIG. 5 is a flowchart showing the panoramic image data synthesis process in the present embodiment. In step S <b> 201, the CPU 104 causes the camera device 105 to capture a panoramic image using the method described with reference to FIG. 2A. The panoramic image data is input from the camera device 105 and developed on the RAM 106. At this time, the CPU 104 performs lens distortion correction on the full-view image data. In step S202, the CPU 104 determines the division number N from the resolution of the desired panoramic image data, and shoots N pieces of divided image data using the method described with reference to FIG. The divided image data group is also input from the camera device 105 and developed on the RAM 106. At this time, the CPU 104 performs lens distortion correction on the divided image data. In step S203, the CPU 104 determines a correction process to be performed. The correction process includes a parse correction process and a mesh correction process. A rough correction is performed by the parse correction process, and a fine adjustment is performed by the mesh correction process. Therefore, the first process selects the perspective correction process, and the second process selects the mesh correction process. In step S204, the CPU 104 generates a variable i on the RAM 106 and initializes it to 0. The CPU 104 also initializes parameters used for each correction here. For example, parameters used in the perspective correction are the camera position (X _C , Y _C , Z _C ), the distance L between the camera and the subject, and the rotation angle (θ, φ) of the camera. There are measured values measured with a tachometer or the like. Further, the CPU 104 determines a range in which each parameter is changed from the measured value, and initializes each parameter with the minimum value. Also in the case of mesh correction, since the coordinate values (X _m , Y _m , Z _m ) of each mesh are parameters, the CPU 104 determines a changing range from the parameters at the time of shooting and initializes them with a minimum value. In addition, the CPU 104 determines a fixed increment value for each parameter at the same time. Step S201 is a processing example of the first acquisition unit, and step S202 is a processing example of the second acquisition unit.

  In step S205, the CPU 104 performs the correction process selected in step S203 on the i-th divided image data using the designated parameters. The correction result is stored in another area on the RAM 106. In step S <b> 206, the CPU 104 calculates a difference value between the i-th divided image data subjected to the correction process and the corresponding partial area of the entire scene image data. At this time, as shown in FIG. 3B, by obtaining the shooting area of the divided image data, it is possible to take correspondence between the divided image data and the full view image data. Since the resolution is different between the panoramic image data and the divided image data, the CPU 104 enlarges the panoramic image data according to the divided image data so that a difference value can be calculated. Then, the CPU 104 adds the difference values for each pixel, takes the sum of the difference values, and stores it on the RAM 106. Step S205 is a processing example of the correction unit, and step S206 is a processing example of the calculation unit.

  In step S207, the CPU 104 determines whether difference values have been calculated for all parameters within the changing range with respect to the parameters used for each correction. If Yes, the process proceeds to step S208. If No, the process returns to step S205, and the CPU 104 adds the increment value calculated in step S204 to each correction process parameter, and executes the correction process again. In step S208, the CPU 104 selects the smallest one from the difference values calculated in step S206, and determines a correction parameter at that time as an optimum parameter. In step S209, the CPU 104 increments i by 1, and proceeds to step S210. In step S210, the CPU 104 determines whether correction processing has been performed on all the divided image data. If Yes, the process proceeds to step S211. If No, the process returns to step S205, and the CPU 104 processes the next divided image data. In step S <b> 211, the CPU 104 determines whether all processing of the perspective correction process and the mesh correction process has been performed. If Yes, the process proceeds to step S212. If No, the process returns to step S203, and the CPU 104 proceeds to the next correction process. In step S212, the CPU 104 performs all correction processes on all the divided image data using the optimum parameter group. Then, the CPU 104 connects the corrected divided image data groups to generate panoramic image data. The stitching process here can be performed simply by superimposing the divided image data groups. As shown in FIG. 3B, the CPU 104 obtains the shooting area of the divided image data and puts the divided image data at that position. Repeat. By performing the processing control described above, it is possible to synthesize high-accuracy panoramic image data. Step S212 is a processing example of the generation unit.

  In the present embodiment, each correction process is executed using all possible parameters, and the optimum parameter is determined from among them. For example, the optimum parameter is efficiently used using a genetic algorithm or the like. Can also be determined. In addition, although the description has been made using a cultural object such as a picture as a subject, the same effect can be obtained even if it is realized using a general wide-view landscape.

  Next, a second embodiment will be described. In the present embodiment, a method for improving accuracy and resolution by repeating panorama synthesis processing will be described. Since the system configuration, the image capturing method, and the image correction method of the present embodiment are the same as those of the first embodiment, description thereof will be omitted.

  Specifically, let us consider a case where 30000 × 20000 pixel panoramic image data is generated using a camera capable of photographing at a resolution of 3000 × 2000 pixels. Here, for simplicity of explanation, the overlap region is not considered. At this time, if considered simply, the number of divisions is 10 × 10, but since the panoramic image data is captured with a resolution of 3000 × 2000 pixels, the resolution is 1/10 compared to the divided image data. End up. In this state, even if the difference value between the panoramic image data and the divided image data is obtained, the panoramic image data has been enlarged ten times, so that sufficient accuracy cannot be obtained.

  This phenomenon will be described with reference to FIG. First, consider the case where there is divided image data 1201 and corresponding panoramic image data 1203. Here, in order to calculate the difference value between the two image data, it is necessary to enlarge the divided image data 1201 and to match the sizes of the two image data. Therefore, the CPU 104 enlarges the divided image data 1201 and obtains divided enlarged image data 1202. However, the edge of the divided enlarged image data 1202 is crushed due to enlargement processing, and it is difficult to directly compare with the whole-view image data 1203. Therefore, the CPU 104 first generates panorama composite data P (0) by setting the number of divisions to 4 (2 × 2). In this case, even in the calculation of the difference value, there is almost no problem in accuracy because the enlargement ratio of the whole-view image data is twice. Next, the CPU 104 uses the generated P (0) as the panoramic image data, performs panorama synthesis processing with a division number of 16 (4 × 4), and obtains panorama synthesis data P (1). In this case, as in the previous case, there is no problem in accuracy because the enlargement rate of the panoramic image data P (0) is twice. Next, the CPU 104 uses the generated P (1) as the panoramic image data, performs panorama synthesis processing with a division number of 64 (8 × 8), and obtains panorama synthesis data P (2). In this case as well, there is no problem in accuracy because the enlargement ratio of the panoramic image data P (1) is twice as in the previous case. Finally, the CPU 104 uses the generated P (2) as the panoramic image data, performs panorama synthesis processing with a division number of 100 (10 × 10), and obtains panorama synthesis data P (3). P (3) is the final output panoramic image data in this case, and the resolution and accuracy can be improved.

  FIG. 7 is a flowchart collectively showing the processes in the second embodiment described above. In step S501, as in the first embodiment, the CPU 104 causes the camera device 105 to capture the panoramic image data of the subject. In step S502, the CPU 104 determines the number M of executions of the panorama synthesis process and the division number N (i) in each synthesis process. In the above example, M is 4 and N (i) = [4, 16, 64, 100]. In step S503, the CPU 104 initializes a variable i to 0. In step S504, the CPU 104 captures the i-th divided image data. Here, N (i) pieces of image data are acquired. In step S505, the CPU 104 generates panoramic image data P (i) using the captured divided image data by the same method as in the first embodiment. In step S506, the CPU 104 replaces the panoramic image data with P (i). In step S507, the CPU 104 increments i by 1. In step S508, the CPU 104 determines whether i = M-1. In the case of Yes, the process is terminated, and P (M−1) is obtained as final panorama composite data. In No, it returns to step S504 and continues a process. By performing the processing control described above, it is possible to synthesize panoramic image data with higher accuracy and higher resolution. Step S506 is a processing example of the replacement unit.

  Next, a third embodiment will be described. In this embodiment, a method for generating panoramic image data using a projector will be described. Since the system configuration, the image capturing method, and the image correction method of the present embodiment are the same as those of the first embodiment, description thereof will be omitted.

  FIG. 8 is a diagram for explaining the outline of the projector projection system. A panorama image projection projector 603 and a capture camera 604 for capturing a projection result are connected to an information terminal 605 that executes image processing such as panorama synthesis. Further, the projection result from the projector 603 is projected onto the projection area 602 so as to overlap the subject 601. At this time, calibration processing is performed in order to correct distortion at the time of projection caused by the lens performance of the projector and installation errors.

  FIG. 9 is a flowchart showing panorama image data synthesis processing according to this embodiment. In step S901, the CPU 104 of the information terminal 605 determines the division number N from the resolution of the desired panoramic image data, and shoots N pieces of divided image data using the method described with reference to FIGS. . The captured divided image data is input from the camera device 105 and developed on the RAM 106 of the information terminal 605. In step S902, the CPU 104 generates a variable i on the RAM 106 and initializes it to 0. Further, the CPU 104 sets parameters to initial values for each of the correction processes of the perspective correction process and the mesh correction process. Further, the CPU 104 sets a fixed increment value for each correction parameter. Since the initialization process has been described in the first embodiment, the details are omitted. In step S <b> 903, the CPU 104 combines the divided image data groups and generates panoramic image data on the RAM 106. The process at this time will be specifically described.

  First, the CPU 104 performs correction processing using the specified parameters for each correction. Then, as shown in FIG. 3B, the CPU 104 acquires the shooting area of the divided image data, and simply superimposes the divided image data on the position. Thereby, panorama synthesis data can be generated. In step S <b> 904, the CPU 104 projects the generated panoramic image data onto the subject using the projector 603. In step S905, the CPU 104 captures the projection result using the capture camera 604, and obtains captured image data. In step S906, the CPU 104 performs edge extraction processing on the captured image data. For example, a Laplacian filter or a high-pass filter is used for the edge extraction process. The result at this time is shown in FIG. In step S907, the CPU 104 performs edge comparison processing. This scans the vicinity of an edge and detects similar edges. As the edge similarity, a parameter when the edge is approximated to an nth-order curve, each inclination and length when the edge is divided into fine line segments, and the like are used.

  Here, for example, as shown in FIG. 10B, it is assumed that an edge 701 and an edge 702 are detected. At this time, the CPU 104 calculates the distance between the two edges and calculates the distance 703 between the edges. The CPU 104 executes this process for all the edges and obtains the sum value. In step S908, the CPU 104 determines whether or not the edge total value has been calculated for all parameters within the changing range with respect to the parameters used for each correction. If Yes, the process proceeds to step S909. If No, the process returns to step S903, and the CPU 104 adds the increment value calculated in step S902 to the parameters of each correction process, and executes the panorama synthesis process again. In step S909, the CPU 104 selects the smallest one from the total edge values calculated in step S907, and determines a correction parameter at that time as an optimum parameter. In step S910, the CPU 104 increments i by 1, and proceeds to step S911. In step S911, the CPU 104 determines whether each correction process has been performed on all the divided image data (i = N−1). If yes, the process proceeds to step S912. If no, the process returns to step S903 to process the next divided image data. In step S912, the CPU 104 performs each correction process on each divided image data using the optimum parameter group, and generates panorama composite data.

  By performing the processing control described above, it is possible to synthesize high-accuracy panoramic image data. Furthermore, since the synthesis result is projected onto the actual subject, the faithful reproducibility for the subject can be seen at a glance.

  Next, a fourth embodiment will be described. In this embodiment, a method for improving accuracy and resolution by changing the projection area of the projector will be described. Since the system configuration, the image capturing method, and the image correction method of the present embodiment are the same as those of the first embodiment, description thereof will be omitted.

  Consider a case where panoramic image data of 30000 × 20000 pixels is generated using a projector capable of projecting at a resolution of 3000 × 2000 pixels. Here, it is assumed that the overlap region is not considered for easy understanding. Simply, when the projection is performed on the entire object by the projector, the resolution of the projected image is 3000 × 2000 pixels, so that the resolution is 1/10 compared to the generated panoramic image data. Even if the distance between the edges is obtained in this state, the projection image is magnified 10 times, so that sufficient accuracy cannot be obtained. Since this phenomenon is the same as in the second embodiment, a detailed description thereof is omitted. Therefore, the CPU 104 divides the projection area of the projector into 100 areas of 10 × 10. Then, the CPU 104 projects a part of the corresponding panoramic image data in each area. This is shown in FIG. The projector 803 narrows down and projects the projection area 802. At this time, the image to be projected is a projection image 806 which is an area corresponding to the projection area 802 in the panoramic image data 805 as shown in FIG. After that, the projected image is changed while moving the projection area up, down, left, and right, and the same processing as in the third embodiment is repeated.

  FIG. 12 is a flowchart collectively showing the processes in the above-described fourth embodiment. In step S1001, the CPU 104 sets the division number N of the panorama synthesis process n and the projector projection area R (i) corresponding thereto. In step S1002, the CPU 104 initializes a variable i to 0. In step S <b> 1003, the CPU 104 causes the camera device 105 to capture the divided image data. Here, N pieces of divided image data are acquired. In step S1004, the CPU 104 generates panoramic image data using the captured divided image data. In step S1005, the CPU 104 projects a part of the panoramic image data corresponding to the projection area R (i) onto the projector. In step S1006, the CPU 104 performs each correction process similar to that in the third embodiment. In step S1007, the CPU 104 increments i by 1. In step S1008, the CPU 104 determines whether i = N−1. In the case of Yes, the process is terminated and final panoramic image data is obtained. In No, it returns to step S1005 and continues a process.

  By performing the processing control described above, it is possible to synthesize panoramic image data with higher accuracy and higher resolution. Furthermore, since the synthesis result is projected onto the actual subject, the faithful reproducibility for the subject can be seen at a glance.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (6)

First acquisition means for acquiring panoramic image data of a subject;
Second acquisition means for acquiring divided image data of the subject;
Calculation means for calculating a difference value between each divided image data acquired by the second acquisition means and the whole-view image data in an area corresponding to the divided image data;
Correction means for executing correction processing according to the difference value calculated by the calculation means for each of the divided image data;
An image processing apparatus comprising: generating means for generating panoramic image data by combining the divided image data corrected by the correcting means. A replacement means for replacing the panoramic image data with the panoramic image data;
The image processing apparatus according to claim 1, wherein the second acquisition unit acquires divided image data obtained by dividing the panoramic image data replaced by the replacement unit.   The image processing apparatus according to claim 1, wherein the correction unit executes a correction process that minimizes a difference value calculated by the calculation unit for each of the divided image data.   The calculation means extracts the edge of each divided image data acquired by the second acquisition means and the edge of the panoramic image data of the area corresponding to the divided image data, and the distance between the extracted edges The image processing apparatus according to claim 1, wherein the difference is calculated as a difference value. An image processing method by an image processing apparatus,
A first acquisition step of acquiring panoramic image data of a subject;
A second acquisition step of acquiring divided image data of the subject;
A calculation step of calculating a difference value between each of the divided image data acquired by the second acquisition step and the whole-view image data of an area corresponding to the divided image data;
A correction step of executing correction processing corresponding to the difference value calculated in the calculation step for each of the divided image data;
And a generating step of generating panoramic image data by combining the divided image data corrected in the correcting step. A first acquisition step of acquiring panoramic image data of a subject;
A second acquisition step of acquiring divided image data of the subject;
A calculation step of calculating a difference value between each of the divided image data acquired by the second acquisition step and the whole-view image data of an area corresponding to the divided image data;
A correction step of executing correction processing corresponding to the difference value calculated in the calculation step for each of the divided image data;
A program for causing a computer to execute a generation step of generating panoramic image data by combining the divided image data corrected in the correction step.

Images