JP2016208294A - Multiple-image photographing system, photographing apparatus, and image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve higher accuracy in image composition by adding a marker having higher luminance than the surroundings to an object.SOLUTION: A photographing apparatus comprises: a camera device having at least one camera for photographing an object in a photographing area; a marker adding device that adds at least one marker having higher luminance than the luminance of the object to a common area where adjacent photographing areas overlap; and a control unit. The control unit alternately performs a first control process for moving and stopping the marker adding device so that the marker is arranged at the foremost part in a moving direction in the photographing area and causing the camera device to photograph the photographing area, and a second control process for moving the camera device from the photographing position of the photographing area in the moving direction and stopping it so that the next photographing area adjacent to the photographing area in the moving direction and including the marker in the common area with the photographing area can be photographed, and causing the camera device to photograph the next photographing area.SELECTED DRAWING: Figure 1A

Description

  The present invention relates to a multi-image photographing system, a photographing device, and an image processing device that continuously photograph a subject with a camera and combine them into one image.

  In recent years, structures such as bridges and tunnels built during the period of high growth have reached the end of their useful lives, and there is an increasing need to check the damage status of these structures and determine the need for repairs and repairs. . In the conventional inspection of the damaged state of the structure, if the inspection worker cannot inspect the structure from the road, check the structure by building a scaffold or getting on the deck at the end of the crane of the bridge inspection car. An operator went near the inspection object such as a high place or under a bridge and observed it visually. However, in recent years, observation techniques based on camera photography have been introduced to reduce the cost of building a scaffold and ensure the safety of inspection workers.

  In the inspection of the damage state of the structure, the area of the inspection range is wide, for example, several [m] to several tens [m], and the crack width to be inspected is, for example, 0.2 [mm] narrower than the hair. So fine. In a single photographed image, the resolution is insufficient when photographing a wide area. On the other hand, if the resolution is increased, only a narrow range can be photographed. Therefore, in the observation technique based on camera photography, a single camera can be moved or a combination of multiple cameras (multi-cameras) can be used to synthesize and observe a plurality of high-resolution images taken. Necessary. The above shows the situation where small cracks in a vast building structure are confirmed. However, in other fields, analysis of fine structures and scratches of large products and large art objects is taken over a vast surface. Have similar needs.

  As a method for synthesizing images taken by a plurality of cameras into one high-resolution image, for example, there is a technique described in Patent Document 1. In Patent Document 1, a group of images obtained by adding a marker to a common image region and a group of images without a marker are acquired by a plurality of photographing units having a plurality of adjacent photographing regions, and the images calculated from the marker image are joined together. It is described that a markerless image group is synthesized by applying a correction parameter for use to a markerless image. Further, the present invention acquires a group of images before and after attaching an adhesive marker to the front part of the imaging region in the moving direction, and sets the camera so that the adhesive marker is the rear part of the imaging region. It is described that an image group captured by moving is acquired, an image stitching correction parameter is calculated from these image groups, and an image without a marker is synthesized.

  With regard to claim 7 of Patent Document 1, after photographing a markerless image, an external view is shown in FIG. 15 for a method of photographing an image with a marker attached by a marker applying means for attaching a marker to the first half of the photographing region. This is described in [0001] to [0002] using a schematic configuration diagram in FIG. Specifically, the marker applying means is integrated with the camera, and the marker applying is configured such that the attached droplet marker is ejected from the nozzle on the side of the camera and attached to the target position on the subject. Yes.

JP 2014-164363 A

  The prior art described above is a configuration in which a droplet marker is attached to a subject. Therefore, in order to improve the photographing accuracy, it is necessary to make the marker more conspicuous, less likely to be displaced, and not to change the surface state of the subject. SUMMARY OF THE INVENTION An object of the present invention is to provide a marker with higher brightness than the surroundings to a subject to improve the accuracy of image composition.

  A multi-image shooting system according to one aspect of the invention disclosed in the present application is a multi-image shooting system including a shooting device and an image processing device, and the shooting device includes at least one camera for shooting a subject in a shooting area. A camera-providing device, a marker-providing device that imparts at least one marker having a luminance higher than the luminance of the subject to a common area where the adjacent imaging regions overlap, and a predetermined moving direction of the camera device and the marker-applying device. And a control unit that controls movement so as to move or stop each independently intermittently, and controls timing of the marker application by the marker application device and photographing by the camera device, The control unit adds the marker so that the marker is arranged at the forefront part of the moving direction in the imaging region. The marker is included in a common area that is adjacent to the moving direction of the imaging area and the imaging area, and the first control process of moving the camera to stop and capturing the imaging area by the camera device A second control process in which the camera device is moved from the shooting position of the shooting area in the moving direction to stop so that the next shooting area can be shot, and the next shooting area is shot by the camera device. Are alternately executed, and the image processing apparatus extracts the coordinate value of the position uniquely specified by the common definition of the marker image of the captured image group captured by the imaging apparatus as the coordinate value of the marker image. An extraction unit, a first photographed image obtained by photographing a first photographing region in the photographed image group, and a second photographed region that is the second photographing region with respect to the first photographing region. Images taken The first marker extracted from the first photographed image by the extraction unit with respect to the same marker given to the common area of the first photographing area and the second photographing area for each combination Based on the coordinate value of the image and the coordinate value of the second marker image extracted from the second photographed image, a correction value for combining the first photographed image and the second photographed image is obtained. An image obtained by removing the image of the common area from the first photographed image and an image of the common area from the second photographed image using the calculation unit to calculate, and the correction value calculated by the calculation unit. Combining the removed image with the remaining image that is the shooting region of one of the first and second captured images and does not have the marker image And a portion.

  According to the representative embodiment of the present invention, it is possible to improve the accuracy of image composition by providing a marker with a higher brightness than the surroundings to a subject. Problems, configurations, and effects other than those described above will become apparent from the description of the following embodiments.

1 is an explanatory diagram illustrating an appearance example 1 of a multi-image shooting system according to Embodiment 1. FIG. FIG. 3 is an explanatory diagram illustrating an appearance example 2 of the multi-image capturing system according to the first embodiment. FIG. 3 is an explanatory diagram illustrating an appearance example 3 of the multi-image capturing system according to the first embodiment. 6 is an explanatory diagram illustrating an appearance example 4 of a multi-image capturing system according to Embodiment 1. FIG. 6 is an explanatory diagram illustrating an appearance example 5 of a multi-image capturing system according to Embodiment 1. FIG. FIG. 6 is an explanatory diagram illustrating an appearance example 6 of the multi-image capturing system according to the first embodiment. FIG. 10 is an explanatory diagram illustrating an appearance example 7 of the multi-image capturing system according to the first embodiment. FIG. 3 is an explanatory diagram illustrating a configuration example of a moving mechanism of the photographing apparatus according to the first embodiment. FIG. 6 is an explanatory diagram illustrating an operation example of the multi-image capturing system according to the first embodiment. FIG. 6 is a functional block diagram of a multi-image capturing system common to Embodiments 1 to 3. It is a functional block diagram of the imaging device shown in FIG. 3 is a timing chart of the photographing apparatus according to Embodiment 1. It is explanatory drawing which shows the example of a setting of the marker provision space | interval in Example 1. FIG. FIG. 6 is an explanatory diagram illustrating a relationship between an imaging region and a marker application position according to the first embodiment. FIG. 3 is an explanatory diagram illustrating a shooting area and a shot image group on a subject according to Example 1; 3 is a flowchart illustrating an example of an operation processing procedure of the imaging apparatus according to the first embodiment. FIG. 4 is a functional block diagram of an image processing apparatus common to Embodiments 1 to 3. FIG. 6 is an explanatory diagram of a reference image-based image processing example 1 according to the first embodiment. FIG. 6 is an explanatory diagram of a reference image-based image processing example 2 according to the first embodiment. FIG. 6 is an explanatory diagram illustrating a first example of image processing based on a real coordinate of a marker image according to Example 1; FIG. 6 is an explanatory diagram illustrating a second example of image processing based on a real coordinate of a marker image according to Example 1; FIG. 6 is an explanatory diagram illustrating a third example of image processing based on a real coordinate of a marker image according to Example 1; FIG. 10 is an explanatory diagram of a real image-based image processing example 4 of a marker image according to Example 1; FIG. 6 is an explanatory diagram illustrating an example of image processing between adjacent images according to the first embodiment. FIG. 6 is an explanatory diagram illustrating an example of image processing between adjacent images according to the first embodiment. It is a flowchart which shows the example of a marker coordinate value calculation process procedure common in Example 1- Example 3. FIG. 10 is a flowchart illustrating an example of an image composition processing procedure common to Embodiments 1 to 3. 17 is a flowchart illustrating a detailed processing procedure example of a correction value calculation process (step S57) of the entire unified coordinate system illustrated in FIG. It is a flowchart which shows the detailed process sequence example of the reference | standard image unified coordinate correction value calculation process (step S68) shown to FIG. 17A. FIG. 17 is a flowchart showing a detailed processing procedure example of a composite image creation process (step S62) based on a correction value process between adjacent images shown in FIG. FIG. 6 is an explanatory diagram illustrating an example of an appearance of a multi-image capturing system according to a second embodiment. FIG. 6 is an explanatory diagram illustrating a shooting route and a shooting area in the multi-image shooting system according to the second embodiment. 6 is a functional block diagram of a photographing apparatus according to Embodiment 2. FIG. 6 is a timing chart of the imaging apparatus according to the second embodiment. FIG. 6 is an explanatory diagram 1 illustrating a shooting sequence on a subject and a shooting region group according to the second embodiment. FIG. 9 is an explanatory diagram illustrating a photographing order on a subject and a photographing region group according to the second embodiment. 6 is a flowchart illustrating an example of an operation processing procedure of the imaging apparatus 5 according to the second embodiment. It is a flowchart which shows the detailed process sequence example of a pan direction moving imaging | photography process (step S106) shown to FIG. 23A. It is a flowchart which shows the example of a detailed process sequence of the tilt direction operation process (step S107) shown to FIG. 23A. FIG. 10 is an explanatory diagram illustrating an example of a captured image group according to the second embodiment. FIG. 9 is an explanatory diagram of a reference image-based image processing example 1 according to a second embodiment. FIG. 6 is an explanatory diagram of a reference image-based image processing example 2 according to a second embodiment. FIG. 10 is an explanatory diagram of a reference image-based image processing example 3 according to the second embodiment. FIG. 10 is an explanatory diagram of a reference image-based image processing example 4 according to the second embodiment. FIG. 10 is an explanatory diagram (part 1) illustrating an example of image processing between adjacent images according to the second embodiment. FIG. 10 is an explanatory diagram (part 2) of an image processing example between adjacent images according to the second embodiment. FIG. 17 is a flowchart illustrating a detailed processing procedure example of a correction value calculation process (step S57) of the entire unified coordinate system illustrated in FIG. 16 according to the second embodiment; It is explanatory drawing which shows the external example 1 of the multi-image imaging system concerning Example 3. FIG. FIG. 10 is an explanatory diagram illustrating an appearance example 2 of the multi-image capturing system according to the third embodiment. FIG. 10 is an explanatory diagram illustrating an appearance example 3 of the multi-image capturing system according to the third embodiment. FIG. 10 is an explanatory diagram illustrating an external appearance example 4 of the multi-image capturing system according to the third embodiment. FIG. 6 is a functional block diagram of a camera apparatus according to a third embodiment. It is explanatory drawing which shows the relationship between the imaging | photography area | region concerning Example 3, and a marker provision position. 9 is a timing chart of the photographing apparatus according to the third example. FIG. 6 is an explanatory diagram 1 illustrating a shooting area and a shot image group on a subject according to a third embodiment. FIG. 6 is an explanatory diagram illustrating a shooting area and a shot image group on a subject according to a third embodiment. 12 is a flowchart illustrating an example of an operation processing procedure of the imaging apparatus according to the third embodiment. FIG. 10 is an explanatory diagram (part 1) illustrating an example of image processing based on a reference image according to the third embodiment. FIG. 12 is an explanatory diagram (part 2) of a reference image-based image processing example according to the third embodiment. FIG. 10 is an explanatory diagram (part 3) of a reference image-based image processing example according to the third embodiment. FIG. 14 is an explanatory diagram (part 4) of a reference image-based image processing example according to the third embodiment. 17 is a flowchart showing a detailed processing procedure example of a correction value calculation process (step S57) of the entire unified coordinate system shown in FIG. 16 according to the third embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each embodiment, as an example, a case where a surface of a concrete structure as a subject structure is photographed while moving the camera is shown. In each embodiment, a multi-image photographing system including a photographing device and an image processing device will be described as an example. However, the photographing device and the image processing device may be configured independently. In other words, the image processing apparatus may be connected to the image processing apparatus via wired or wireless communication, and an image captured by the image capturing apparatus after image capturing by the image capturing apparatus may be transmitted to the image processing apparatus at a point away from the shooting site. Further, by connecting a portable storage medium such as a USB memory to the photographing apparatus to store the photographed image in the portable storage medium, and connecting the portable storage medium storing the photographed image to the image processing apparatus, the image processing apparatus It is good also as acquiring a picked-up image.

  Further, in the following description, when there are a plurality of identical constituent elements, branch numbers are given to the reference numerals. Taking the marker 31 as an example, if there are two markers 31, they are shown separately as a marker 31-1 and a marker 31-2. In addition, for the sake of explanation, when a plurality of identical components are not distinguished, the branch number is omitted, and for example, the marker 31 is described.

  Also, branch numbers are assigned to the same components in time series. In the branch number in this case, “1” indicates the initial position. Taking the marker 31-1 as an example, the markers 31-1-1, 31-1-2,..., 31-1-n, 31-1- (n + 1),. Note that n is a value indicating a time series of movement and is an integer of 1 or more.

<Appearance of multi-image shooting system>
1A to 1G are explanatory diagrams illustrating an example of an appearance of a multi-image shooting system according to the first embodiment. 1A is a plan view of the multi-image shooting system, FIG. 1B is a side view of the multi-image shooting system, FIG. 1C is a front view of the multi-image shooting system, FIG. 1D is a plan view of the basic part of the camera unit, and FIG. FIG. 1F is a plan view of the basic part of the marker applying unit, and FIG. 1G is a plan view of the additional unit of the marker applying unit.

  The multi-image capturing system 1 moves over, for example, two track rails 105-1 and 105-2. The track rails 105-1 and 105-2 are not components of the multi-image photographing system 1. The movement processing unit 132 and the camera unit guiding unit 134 mounted on the track rail 105-1 and the arm connecting the movement processing unit 132 and the camera unit guiding unit 134 constitute a camera unit basic unit 140. The camera unit basic unit 140 has a configuration in which the camera unit extension unit 141 is connected to the width of the shooting range as necessary (see FIG. 1D).

  In the example of FIG. 1A, three camera unit expansion units 141 are connected to the camera unit basic unit 140. Cameras 2-1 to 2-3 are arranged in each of the three camera unit expansion units 141. The cameras 2-1 to 2-3 shoot the shooting areas 113-1 to 113-3.

  In FIG. 1F, the marker assigning unit 130 includes a marker assigning unit basic unit 145 and a marker assigning unit additional unit 146. In this example, the three marker assignment unit basic units 145 and the marker addition unit extension unit 146 constitute the marker addition unit 130. The three sets of marker providing unit basic unit 145 and marker adding unit adding unit 146 are configured on the movement processing unit 133 and the marker guiding unit 135.

  When the marker base is added, marker lasers 131-1 to 131-4 are added. The marker lasers 131-1 to 131-4 are alternately arranged with the cameras 2-1 to 2-3 along the arrangement direction of the cameras 2-1 to 2-3. More specifically, the marker lasers 131-1 and 131-2 form the markers 31-1 and 31-2 at both ends in the arrangement direction of the cameras 2-1 to 2-3 on the imaging region 113-1. Placed in. A “marker” is a beam spot formed in an imaging region by laser light from a marker laser.

  The marker lasers 131-2 and 131-3 are arranged so as to form the markers 31-2 and 31-3 at both ends in the arrangement direction of the cameras 2-1 to 2-3 on the imaging region 113-2. The marker lasers 131-3 and 131-4 are arranged so as to form markers 31-3 and 31-4 at both ends in the arrangement direction of the cameras 2-1 to 2-3 on the imaging region 113-3. The marker laser 131-2 is arranged so that the marker 31-2 is formed in an overlapping area of the imaging areas 113-2 and 113-2. The marker laser 131-3 is arranged so that the marker 31-3 is formed in an overlapping area of the imaging areas 113-2 and 113-3.

  The multi-image shooting system 1 further includes a control unit 7, a power source 136, and an image processing device 9. The control unit 7, the power source 136, and the image processing apparatus 9 are connected by a power source / communication cable 137. Further, the control unit 7 is connected to the photographing apparatus 5 by a power / communication cable 137. The control unit 7 controls the movement and stop of the movement processing unit 132 and the movement processing unit 133 included in the photographing apparatus 5, photographing by the camera 2, lighting and extinguishing of the marker laser 131, and the like. The power source 136 supplies power necessary for various operations in the multi-image photographing system 1. The image processing device 9 performs image processing on the captured image.

  A reflector 106 is installed in front of the track rail 105-1. The reflector 106 reflects the lateral laser beam 107 used for laser distance measurement for measuring the position of the movement processing unit 132. The reflector 106 is not a component of the multi-image shooting system 1.

<Configuration Example of Moving Mechanism of Imaging Device 5>
FIG. 2 is an explanatory diagram illustrating a configuration example of a moving mechanism of the photographing apparatus 5 according to the first embodiment. In FIG. 2, the photographing apparatus 5 includes a movement processing unit 132 and a movement processing unit 133. Each of the movement processing unit 132 and the movement processing unit 133 includes a drive motor 151, a gear 152, a drive wheel 153, a brake 154, a control IF 155, and guide wheels 156-1 to 156-5. The gear 152 transmits the force of the drive motor 151 to the drive wheel 153. The brake 154 is fixed to the track rail 105-1 when stopped. The control IF 155 includes a circuit that controls a series of operations and performs communication with the control unit 7. The guide wheels 156-1 to 156-3 support the position in the horizontal direction with respect to the track rail 105-1. The guide wheels 156-4 and 156-5 support the position in the vertical direction.

  The movement processing unit 133 includes a laser distance measurement unit 157. The laser distance measuring unit 157 irradiates the reflecting plate 106 installed in front of the track rail 105-1 of FIG. 1 with the side distance laser beam 107, and measures the relative position with mm accuracy. The laser distance measurement unit 157 is used to control the movement distance of the movement processing unit 133 and to acquire position information.

  Further, the movement processing unit 133 and the movement processing unit 132 are indirectly connected by an adjustment cylinder 158. The adjustment cylinder 158 has a trigger that indicates a distance at a predetermined position of the extended distance of the adjustment cylinder 158. The trigger is detected by the sensor 159 and used for control to make the proximity distance and the remote distance between the movement processing unit 133 and the movement processing unit 132 constant. Of the two adjustment cylinders 158, one is for the proximity point and the other is for the remote point.

<Operation of Multi-Image Shooting System 1>
FIG. 3 is an explanatory diagram of an operation example of the multi-image capturing system 1 according to the first embodiment. (A) is a plan view showing an operation example of the multi-image shooting system 1, and (b) is a side view showing an operation example of the multi-image shooting system 1.

  The multi-image shooting system 1 (1) forms the marker 3 at the end of the shooting area 113 to take a picture of the shooting area 113, and (2) advances the marker applying device 6 to place the marker at the forefront of the shooting area 113. Then, the imaging region 113 is imaged, and (3) the camera device 8 is advanced to form the marker 3 again at the end of the imaging region 113 and the imaging region 113 after the advancement is imaged.

  The marker applying device 6 includes marker lasers 131-1 to 131-4 and a movement processing unit 133. The camera device 8 includes a movement processing unit 132 and the camera 2. While the camera device 8 is fixed by the brake 154 from (1) to (2), the marker applying device 6 follows the movement of the marker lasers 131-1 to 131-4, and the imaging regions 113-1 to 113 are detected. -3 and move forward so that the markers 31-1 to 31-4 are formed at the foremost part. When the laser distance measuring unit 157 and the interval position sensor 159 detect a predetermined movement distance of the marker applying device 6, a signal is received from the control unit 7, and the marker applying device 6 is fixed by the brake 154. In this fixed state, the camera device 8 performs (2) photographing with the camera 2.

  Thereafter, the camera device 8 releases the brake 154 according to a signal from the control unit 7, advances the camera device 8 so that the fixed marker 3 is formed at the tail in the next imaging region 113, and again Fixing with the brake 154, the camera apparatus 8 performs (3) imaging | photography.

<Functional Block of Multi-Image Shooting System 1>
FIG. 4 is a functional block diagram of the multi-image shooting system 1 common to the first to third embodiments. The multi-image photographing system 1 includes a photographing device 5 that photographs a subject by moving the camera 2 and the marker 31, and an image processing device 9 that performs image processing on the photographed image. The imaging device 5 includes a marker applying device 6, a control unit 7, and a camera device 8. The marker applying device 6 and the camera device 8 are controlled by the control unit 7. An image photographed by the camera device 8 and photographing-related information are sent to the image processing device 9.

  The image processing apparatus 9 includes a basic processing unit 10 that primarily processes an acquired image, and an image composition processing unit 11 that performs image processing for output. Each function expresses a conceptual function and does not prescribe the hardware configuration.

<Functional blocks of the photographing device 5>
FIG. 5 is a functional block diagram of the photographing apparatus 5 shown in FIG. The imaging device 5 includes a camera device 8, a control unit 7, and a marker applying device 6. The camera device 8 includes a camera unit 102 including at least one camera 2 and a movement processing unit 132 that moves or stops the camera unit 102. The marker imparting device 6 includes a marker imparting unit 130 such as the marker laser 131 in FIG. 1, a movement position measuring unit 138 such as the laser distance measuring unit 157 in FIG. 2, and a movement processing unit 133.

  The control unit 7 includes a timing control unit 182, a movement control unit 186, and a monitor control unit 185. The timing control unit 182 includes an imaging control unit 183 that performs imaging timing and a marker application control unit 184 that performs marker application timing. The movement control unit 186 generates a movement or stop signal for the movement processing unit 132 or the movement processing unit 133, and from each driving unit associated with the camera movement / stop signal I3 or the marker movement / stop signal I4 from both. Receive status information. The monitor control unit 185 monitors the position of the marker 31 in the imaging area 113. When the shooting control signal I1 is transmitted from the shooting control unit 183 to the camera 2, the camera 2 performs shooting and generates shot image information I10 that is a set of the shot images 114. The captured image information I10 is sent to the image processing device 9 in FIG.

  When the position of the marker 31 is monitored and controlled in the imaging device 5, the monitor image information I12 from the camera 2 is input to the monitor control unit 185. The monitor control unit 185 processes the monitor image obtained from the monitor image information I12 to form the marker 31 at an appropriate position. When the control signal from the timing control unit 182 is input to the movement control unit 186, the movement control unit 186 performs timing control on the movement processing unit 132 and the movement processing unit 133, as shown in FIG. The movement of the camera device 8 and the marker applying device 6 is controlled.

  In addition, when the marker 31 is arranged at an appropriate position while being observed and confirmed by the user outside the photographing apparatus 5, monitor image information I12 from the camera 2 to the outside is output to an external monitor screen, and the user Manual position correction information I6 by monitor observation is input to the monitor control unit 185.

  In shooting by moving or stopping the camera device 8 or the marker applying device 6, the timing control unit 182 outputs a shooting control signal I 1 to the camera 2 according to the control sequence, or turns on / off the marker 31 to the marker applying unit 130. Timing control for outputting a necessary image is performed by outputting I2. For example, the timing control unit 182 performs timing control when receiving a camera movement / stop signal I3 or a marker movement / stop signal I4 related to the movement processing unit 132 and the movement processing unit 133 from the movement control unit 186. Further, the timing control unit 82 executes timing control when receiving the marker moving distance information I5 from the moving position measuring unit 138.

  Further, the timing control unit 182 transmits meta information of the image information I10 as the shooting-related information I11 to the outside in accordance with the timing of shooting and device control. The shooting related information I11 includes, for example, a shooting time, a shooting number, a camera movement or stop associated with shooting, a movement or stop of the marker attaching unit 130, a front-rear relationship between the camera movement and the marker movement, lighting or extinguishing of the marker 31, and a shooting camera. 2, the moving distance information of the adjustment cylinder 158 shown in FIG. 2, the measured value of the laser distance measuring unit 157, the marker interval distance information in the arrangement direction of the camera 2, and the marker 31 corresponding to the marker applying unit 130. Various information such as physical spacing information is included. Here, since the shooting related information I11 is aggregated in the timing control unit 182, the shooting related information I11 is transmitted from the timing control unit 182, but a function different from the timing control unit 182 transmits the shooting related information I11. It is good also as a structure.

<Timing chart of photographing apparatus 5>
FIG. 6 is a timing chart of the photographing apparatus 5 in the first embodiment. In FIG. 6, (1) is the movement of the camera device 8, (2) is the movement of the marker applying device 6, (3) is the marker on or off, (4) is still image shooting, (5) is the image or shooting related Information transmission, (6) shows the image ID of the photographed image group: nZ. Here, still image shooting indicates that it is different from the monitor image. Image ID: nZ specifies a captured image. Note that n in the image ID: n−Z indicates a time series of movement of the n-th imaging region 113, and n = 1 at the initial stage. The branch number “Z” of the image ID: n−Z indicates the state of the captured image 114 specified by the image ID, and is “O”, “B”, or “F”. For example, “O” indicates a captured image in which all the markers 31 in the imaging region are turned off. “F” indicates a captured image provided with a marker 31 at the forefront of the imaging region. “B” indicates a captured image in which a marker 31 is added at the end of the imaging region.

  The image ID: n-Z is uniquely determined based on (1) movement information of the camera device 8 and (2) movement information of the marker imparting device 6, shooting time, or shooting number. At the initial position, the camera 2 turns off the marker 31 and takes a still image. This captured image does not include the marker 31. Therefore, the image ID is “1-O”. When the camera device 8 captures a still image with the image ID: 1-O, the camera device 8 transmits (5) image information I10 and imaging related information I11 as image / imaging related information.

  Next, the camera device 8 (3) turns on the marker 31 and shoots to obtain a shot image with an image ID: 1-B. Next, the movement control unit 186 moves the marker applying device 6 by a predetermined interval L0 and stops it. At this time, the marker 31 is formed at the forefront of the imaging region 113. The camera device 8 performs the third shooting in this state, and obtains a captured image with the image ID: 1-F.

  Next, the movement control unit 186 moves and stops the camera device 8 by a predetermined interval L0, and the camera device 8 captures an image at the stop position to obtain a captured image with an image ID: 2-B. Thereafter, while the marker 31 is lit, the movement and stop of the marker applying device 6 at a predetermined interval L0, the photographing with the camera device 8, the movement and the stop of the camera device 8 with the predetermined interval L0, and the photographing with the camera device 8 are sequentially executed. By doing so, the image after image ID: 2-F is obtained.

<Setting example of marker interval>
FIG. 7 is an explanatory diagram illustrating a setting example of the marker applying interval in the first embodiment. 7A shows a state before the movement of the marker 31 in which the two markers 31-1-1 and 31-2-1 are located at the end of the imaging region 113-1. (B) shows the state after the movement of the markers 31-1-2 and 31-2-2 moved to the forefront part of the same imaging region 113-1 at the movement interval L0. Two basic conditions that are important in the image composition of multi-image shooting that moves and shoots are shown below.

(R1) In order to perform the joint synthesis of the captured images before and after the movement, the marker assigned to the previous imaging area must be included in the next imaging area.

(R2) In order to conceal the marker image in the next photographed image with the previous photographed image, the marker image can be moved only by a distance shorter than the distance from the forefront part of the photographing region to be photographed next to the marker image.

  From the first condition (R1), in order to reduce the number of shootings by making the moving distance before and after the movement as long as possible, a marker to be given at the time of the previous photographing is attached to the forefront of the photographing region as much as possible. It is necessary to place the marker at the end of the imaging region as much as possible when shooting after. Thereby, since the movement interval L0 is increased, the movement distance before and after the movement is increased, and the number of photographing is reduced.

  From the second condition (R2), it is necessary to apply the marker as far as possible from the foremost part, that is, at the end. Thereby, the movement interval L0 becomes large.

  In order to obtain the optimum marker interval, from the interpretation of the first condition (R1) and the second condition (R2), the marker assigned in the previous shooting is arranged at the end as much as possible in the next shooting. By arranging the marker to be assigned to the front part as much as possible, the movement interval L0 can be increased. Respective error ranges (allowable width of grounding error) for marker application are x1 on the last side, x2 on the foremost side, and x3 on the diameter of the marker 31. When the movement direction width of the imaging region 113-1 is X, the maximum movement interval L0 is expressed by the following formula (1).

  L0 = (X−x1−x2−x3) (1)

  By setting L0 as an alternate movement distance between the camera device 8 and the marker applying device 6, it is possible to take a picture with the maximum movement interval L0.

<Relationship between shooting area and marker position>
FIG. 8 is an explanatory diagram illustrating the relationship between the imaging region and the marker application position according to the first embodiment. In FIG. 8, (1) to (3) are imaging areas 113 and markers 31-1 and 31 captured by the arrangement of the camera 2 and the marker lasers 131-1 and 131-2 in (1) to (3) of FIG. -2. (4) further shows the positional relationship between the imaging region 113 and the marker 31 at the time of the next imaging of (3).

  Here, it is assumed that the photographing region 113-1-n is taken at an arbitrary position. In (1) and (2), the imaging region 113-1-n is fixed as in FIG.

  In (1), the camera 2 shoots with the markers 31-1-n and 31-2-n added to the shooting region 113-1-n.

  In (2), thereafter, the movement control unit 186 moves the markers 31-1-n and 31-2-n by the interval L0 and moves them to the forefront of the same imaging region 113-1-n. The camera 2 images the imaging region 113-1-n including the markers 31-1- (n + 1) and 31-2- (n + 1) moved to the forefront.

  In (3), markers 31-1- (n + 1), 31-2- (n + 1) are fixed. In this state, the movement control unit 186 moves the camera device 8 by the interval L0. Accordingly, the markers 31-1- (n + 1) and 31-2- (n + 1) are arranged at the end of the new imaging region 113-1- (n + 1). In this state, the camera 2 captures an imaging region 113-1- (n + 1).

  In (4), after (3), the movement control unit 186 performs the same operation as the operation shown in (1) and (2), so that the movement control unit 186 has the markers 31-1- (n + 1), 31-2- (N + 1) is moved by an interval L0 and moved to the forefront of the same imaging region 113-1- (n + 1). The camera 2 images the imaging region 113-1- (n + 1) including the markers 31-1- (n + 2) and 31-2- (n + 2) moved to the forefront.

<Shooting area and photographed image group on subject>
FIG. 9 is an explanatory diagram of a shooting area and a shot image group on the subject according to the first embodiment. 9A shows the shooting area on the subject 4 and the position of the marker 31. FIG. Specifically, (a) is a view of the imaging areas 113 (1, 1) to 113 (3, 2) by the cameras 2-1 to 2-3 and the markers 31-1-1-1 to 3-4-3. The positional relationship is shown including the crack 41. A parenthesis (m, n) at the end of the code of the shooting area 113 (also for a shot image 114 described later) indicates an image matrix based on the upper left shooting area 113 in the camera 2-1. That is, m is a branch number of the camera 2, and n is a time series of shooting. For example, the shooting area 113 (1, 1) indicates the first shooting area shot by the camera 2-1, and the shooting area 113 (1, 2) is the second (second) shot by the camera 2-1. ).

  In FIG. 9, (b) shows a captured image group. The parenthesized numerals <1> to <3> in (b) correspond to the operations (1) to (3) in FIG. The photographed image 114 (m, n) Z is a group of images photographed along the timing chart of FIG. 6 while alternately moving the photographing region 113 and the marker 31 as shown in FIG. As described above, m is a branch number indicating the arrangement position of the cameras 2, and n is a time series of movement. Z indicates one of the states O, F, and B of the marker 31. For example, “O” indicates a state in which all the markers 31 in the imaging region 113 are turned off. “F” indicates a state in which the marker 31 is provided at the forefront of the imaging region 113. “B” indicates a state in which the marker 31 is provided at the end of the imaging region 113.

  The marker image 103-mn-XY is an image of the marker 31-mn. X is either state B or F of the captured image 114 described above. Y is a marker ID indicating either S or R. “S” indicates one end (left side) in the arrangement direction of the cameras 2 orthogonal to the moving direction, and “R” indicates the other end (right side).

  For example, the marker image 103-2-1-1-BR is an image of the marker 31-2-1 attached to the tail of the moving direction in the shooting area 113 (1, 1), and the shooting area 113 (1, 1). Is the marker image on the right side of the two marker images included in the captured image 114 (1, 1) B. In addition, the marker image 103-2-1-BS is an image of the marker 31-2-1 added to the end of the moving direction in the imaging region 113 (2, 1), and the imaging region 113 (2, 1). Is a marker image on the left side of the two marker images included in the captured image 114 (2,1) B. The marker images 103-2-1-1-BR and 103-2-1-BS are images obtained by photographing the same marker 31-2-1 in different photographing regions 113 (1, 1) and 113 (2, 1). .

<Example of Operation Processing Procedure of Imaging Device 5>
FIG. 10 is a flowchart illustrating an example of an operation processing procedure of the photographing apparatus 5 according to the first embodiment. This flowchart shows the flow from the initial shooting of the timing chart of FIG. The user initializes the photographing device 5 by manual operation (step S1) and installs the camera unit 102 at the initial position (step S2). The imaging device 5 performs imaging with the marker 31 turned off under the control of the imaging control unit 183, and obtains a captured image 114 having an image ID: 1-O (step S3). Steps S3 and after are automatic execution unless otherwise specified.

  Next, the imaging device 5 lights the marker 31 under the control of the marker application control unit 184. And the imaging device 5 moves the marker provision part 130 by control of the movement control part 186, distribute | arranges the marker 31 to the predetermined position of the tail of the imaging region 113, and stops the movement of the marker provision part 130 ( Step S4).

  Next, the imaging device 5 captures the imaging region 113 including the marker 31 at the position where the movement of the marker applying device 6 is stopped in step S4, and acquires the captured image 114 of the image ID1-B (step S5). And the imaging device 5 judges whether the marker provision apparatus 6 is moved (step S6). Specifically, for example, the imaging device 5 moves the marker applying device 6 depending on whether or not the current position of the multi-image imaging system 1 is within a preset imaging range based on the imaging related information I11. Judging by all means. That is, the photographing device 5 determines that the marker applying device 6 should be moved if the current position of the multi-image photographing system 1 is within the photographing range (step S6: Yes), and moves if it is outside the photographing range. It is judged that it should not be (step S6: No). Step S6: If Yes, the process proceeds to Step S7.

  Next, the imaging device 5 moves the marker imparting device 6 to a position that is the forefront portion of the imaging region 113 under the control of the marker imparting control unit 184, and stops at that position (step S7). Next, the photographing device 5 photographs the photographing region 113 including the marker 31 at the position where the movement of the marker applying device 6 is stopped in step S7, and obtains the photographed image 114 having the image ID: n−F (step S8). ). Here, n−F is the image ID used in (6) of FIG. n represents a time series of movement, that is, n-th movement of the imaging region 113, and initially n = 1. “F” indicates a state in which the marker 31 is provided at the forefront of the imaging region.

  Next, under the control of the movement control unit 186, the imaging device 5 moves and stops the camera device 8 so that the marker 31 is positioned at the end of the imaging region 113 (step S9).

  And the imaging device 5 image | photographs the image group (n + 1) -B containing the marker 31 (step S10), and returns to step S6. (N + 1) -B is the image ID used in (6) of FIG. 6, and is the next image group after the image group n-F. “B” indicates a state in which the marker 31 is provided at the end of the imaging region. When returning to step S6, the (n + 1) th movement in step S10 is read as the nth movement in step S8. Step S6: In the case of No, the operation of the photographing device 5 is finished.

<Functional Block of Image Processing Device 9>
FIG. 11 is a functional block diagram of the image processing apparatus 9 common to the first to third embodiments. The image processing device 9 includes a basic processing unit 10 and an image composition processing unit 11. The basic processing unit 10 acquires the photographed image information I10 and the photographing related information I11 from the photographing device 5, and processes the information so that it can be easily used. The image composition processing unit 11 executes processing for obtaining the display area instruction I52 and processing for processing the image information into the instructed output information I53.

  The basic processing unit 10 includes an information acquisition unit 190, a marker coordinate value extraction unit 191, a marker information holding unit 192, and a marker route information generation unit 193. The information acquisition unit 190 acquires and holds the captured image information I10 and the imaging related information I11 in association with the image information I21 in the order of the transmission time from the imaging device 5.

  The marker coordinate value extraction unit 191 extracts the coordinate value (hereinafter referred to as marker coordinate value) of the marker image 103 of each captured image 114 in the image information I21. The marker information holding unit 192 holds the marker coordinate value in association with the image ID (see FIG. 6). In the second and third embodiments, when using laser markers of different colors for different purposes, the marker coordinate value extraction unit 191 associates marker coordinate values according to the application. For example, in Example 2, a red marker is a pan direction marker and a green marker is a tilt direction marker, in Example 3, a red marker is a marker related to a planar arrangement, and a green marker is a marker related to a moving direction. As described above, an ID associated with the application is assigned, and the marker information holding unit 192 holds the ID.

  The marker route information generation unit 193 specifies the position of the marker image 103 for each image ID from the ID / related information I28 and the marker coordinate information I19, and is specified by the image ID related to the processing of the image composition processing unit 11. Marker route information I29 corresponding to the marker position between the captured images 114 is generated. The ID / related information I28 is information including an image ID and shooting related information I11. The marker coordinate information I19 is information held by the marker information holding unit 192. That is, it is a combination of a marker coordinate value and an image ID. The marker route information I29 is marker information obtained by assigning an image ID to position information of the marker image 103 in each captured image 114 (for example, any one of upper right, lower right, upper left, and lower left in the captured image 114). Information associated with adjacent image IDs.

  In addition, the image composition processing unit 11 includes a composition correction value calculation unit 194, a whole area image display unit 195, a composite image selection unit 196, a correction value calculation unit 197, and a composition unit 198.

  The composition correction value calculation unit 194 obtains marker coordinate information I26 which is a marker coordinate value from the basic processing unit 10 and, optionally, additional information from the outside, for example, shooting environment information I50 such as marker arrangement interval and lens aberration. Then, a correction value for image synthesis for each captured image 114 is calculated.

  The entire area image display unit 195 obtains correction value information I23 that is a correction value calculated by the compositing correction value calculation unit 194, obtains image information I21 from the basic processing unit 10, and synthesizes and displays the entire area image. Display information I51 indicating the combined and displayed whole area image is output.

  The composite image selection unit 196 obtains the image region information I24 that is linked to the entire region image display unit 195, obtains the display region instruction I52 from the outside, and selects an instruction target image from the entire region image.

  The correction value calculation unit 197 acquires and selects the image ID signal I25 including the image ID of the selected image in the composite image selection unit 196, the marker coordinate information I26 and the marker route information I29 from the basic processing unit 10. An image composition correction value for composing the image is calculated.

  The synthesizing unit 198 generates a marker concealment image using the combination information I27 including the correction value calculated by the correction value calculation unit 197 and the marker route information I29, and outputs including the generated marker concealment image Information I53 is output.

<Method for extracting marker coordinate values>
Here, a specific method of extracting marker coordinate values by the marker coordinate value extracting unit 191 will be described. The marker coordinate value extraction unit 191 sets a threshold value for the luminance of the image so that the center-of-gravity coordinate value can be easily obtained as the image matching coordinates of the marker image 103 included in each captured image 114 in the image information I21. The image 103 is binarized. Thereby, the marker image 103 with high luminance is, for example, “white”, and the other image portions are “black”. Accordingly, if the barycentric coordinate value of the “white” pixel group (white pixel linked set) is simply obtained, it becomes the coordinate value of the marker image 103. In the present embodiment, a description is given of a mode in which the barycentric coordinate value of the marker image obtained by binarization is used as the reference coordinate value for image alignment. However, the coordinate value of the marker image for image alignment is a binarized marker image. It is not limited to the barycentric coordinate value.

  For example, a center coordinate value to which a binarized marker image distribution load is added may be used as the coordinate value of the marker image. Further, any coordinate value in the rectangular area including the white pixel linked set may be used as the coordinate value of the marker image. Further, when the luminance distribution of the marker image is stable regardless of the shooting position, the coordinate value of the luminance peak of the marker image may be simply used. In this case, there is an advantage that the marker coordinate value extraction process is reduced. The coordinate value of the marker image means a coordinate value depending on the marker image as described above.

  Further, the marker coordinate value extraction unit 191 divides the photographed image 114 into a left half and a right half, and sets an ID (marker ID) for identifying the marker image 103 in a divided region to which the marker image 103 belongs. For example, when the marker image 103 belongs to the left half of the divided area, the marker coordinate value extraction unit 191 sets “S” as the marker ID, and when the marker image 103 belongs to the right half of the divided area, the marker ID 103 Set “R” to. Then, the marker coordinate value extraction unit 191 associates the set marker ID with the barycentric coordinate value of the marker image 103.

  The actual captured image 114 includes a high-luminance image in addition to the high-luminance marker image 103. For example, when shooting under a bridge, leaked light with high brightness is reflected from the hole in the damaged part or from the edge. In such a case, since the photographic image 114 to which the marker image 103 is added and the photographic image 114 to which the marker image 103 is not attached are photographed at the same pixel position at the same photographing point, both the photographed images 114 are captured. The marker image 103 can be specified by obtaining the difference in luminance.

<Method for Constructing Image ID: (m, n) Z>
Next, a configuration method of the image ID: (m, n) Z by the information acquisition unit 190 will be described. Note that the image ID: n-Z shown in FIG. 6 is substantially the same as the image ID: (m, n) Z, except that the code m indicating the position of the camera 2 in the arrangement direction is not defined. ID.

  The information acquisition unit 190 sets an image ID to (6) the photographed image 114 shown in the timing chart of FIG. 6 from the photographing related information I11. For example, the information acquisition unit 190 associates with the shooting-related state information of (1) to (4) in FIG. 6, shooting time, shooting number, movement or stop of the camera device 8, movement or stop of the marker applying device 6, The image ID: (m, n) Z is set from the front-rear relationship between the movement of the camera device 8 and the movement of the marker 3, the lighting or extinction of the marker 3, and the relationship between the camera 2 number and the image information.

  For example, as shown in the timing chart (6) of FIG. 6, in the case of the captured image 114 (m, n) Z immediately after the stop of the marker 3, “F” is set to “Z” and the camera 2 is stopped. In the photographed image 114 (m, n) Z photographed immediately after, “B” is set to “Z”, and in the photographed image 114 (m, n) Z in which the related marker 3 is turned off, “Z” is displayed. “O” is set.

  Further, with the time series in the moving direction of the camera 2, the value of “n” (n = 1, 2, 3,...) Of the image matrix (m, n) for each captured image 114 (m, n) Z. Is set, and the value of “m” in the image matrix (m, n) (m = 1, 2, 3,...) Is set corresponding to the arrangement position from the left end of the camera 2 and the camera 2. Thereby, the image ID: (m, n) Z of the captured image 114 is set.

<Machiner route processing>
Next, marker route processing will be described. The marker route information generation unit 193 obtains the marker movement / stop signal I4, the marker point / light-off signal I30, and the marker movement distance information I5 from the imaging-related information I11 as marker addition related information, and the marker coordinate extraction unit 191 receives the marker Get an ID. The marker route information generation unit 193 creates the marker route information I29 by linking the marker ID to the image ID indicating the photographing order of the photographed image 114, the marker presence / absence information, and the marker actual coordinate position.

  For example, the marker image 103-1-2-BS and the marker image 103-1-2-FS shown in (b) of FIG. 9 are present in the rear half of the captured image 114 (1,1). In the case where “B” indicating the marker ID exists at the end of the code in the divided area in the front half of the captured image 114 (1, 1), “F” is assigned. From these information and the time-series relationship of the captured image 114, the captured image 114 of the same marker is specified to obtain the marker image 103-1-2 having the same marker ID, and the marker of the marker image 103-1-2 A marker route is created by sequentially arranging IDs along the image ID.

  The correction value calculation unit 197 calculates a correction value for composition of the image selected by the composite image selection unit 196. First, the correction value calculation unit 197 sets a reference image for composition correction value. For example, the correction value calculation unit 197 sets the center image of the selected image group as the reference image. An optimum route from the reference image to each image to be image-combined is set using the marker route information I29. Specifically, for example, the correction value calculation unit 197 calculates a matrix error between adjacent images on the route from the reference image to the target image, thereby calculating the quantization error inherent in the coordinate value of the marker image. In addition to accumulating, the lens aberration is acquired from the shooting environment information I50, so that the lens distortion and the matrix product based on the lens aberration are calculated.

  The image that is the target of image synthesis that is the end point of the route is the remaining image other than the reference image in the selected image group, but the correction value calculation unit 197 is the outermost image group of the selected image group. May be selected as an image to be synthesized as an end point of the route.

  For example, in FIG. 14A, the marker image 123 (C, C), which is the reference image, and the marker image 123 (C, C + 1) have a matrix product from the marker image 123 (C-1, C + 1) on the upper right. -1, C) and a route via the left marker image 123 (C, C + 1). In addition to these, there is also a route that passes through the marker image 123 (C + 1, C) → the marker image 123 (C + 1, C + 1) → the marker image 123 (C, C + 1). The correction value calculation unit 197 selects a route having the shortest distance (the smallest number of images passing through the route) as the optimum route from among these route groups.

  For example, the route of the marker image 123 (C, C) → the marker image 123 (C−1, C) → the marker image 123 (C−1, C + 1) and the marker image 123 (C, C) → the marker image 123 (C, C The route of (C + 1) → marker image 123 (C−1, C + 1) is selected. Further, of these two routes, in FIG. 14A, since the accuracy of alignment in the vertical direction is further important as crack information, the correction value calculation unit 197 takes into account the importance of the direction in advance, and the correction value calculation unit 197 displays the marker image 123 (C, C) → marker image 123 (C, C + 1) → marker image 123 (C−1, C + 1) route is set as the optimum route.

  Further, the correction value calculation unit 197 may select the optimum route in consideration of quantization error and lens distortion. Since the quantization error depends on the marker interval and the number of pixels, the correction value calculation unit 197 calculates the quantization error from the marker interval and the pixel density as the uncertainty of the coordinate value of the marker image for each link constituting the route. Turn into. Since the lens distortion depends on the position in the image, the correction value calculation unit 197 sets the lens distortion as a degree of uncertainty of the coordinate value of the marker image from the camera lens aberration measured in advance for each link constituting the route. Turn into. Then, the correction value calculation unit 197 calculates the cumulative uncertainty for each route by summing the uncertainty of each link, and selects the route with the smallest cumulative uncertainty as the optimum route. When the number of images is large and the number of route options increases, the influence of cumulative error or cumulative distortion for each route appears, so that an optimal route that does not depend only on the number of images on the route is selected.

  Finally, the correction value calculation unit 197 follows the set optimum route in the coordinate system of the reference image (marker image 123 (C, C)) (marker image 123 (C-1, C + 1). )) Coordinate transformation matrix product for matching the coordinate system (see equation (9) described later) is calculated as a composition correction value for image composition.

<Reference image-based image processing example>
FIG. 12A and FIG. 12B are explanatory diagrams illustrating a reference image-based image processing example according to the first embodiment. 12A shows the processing of the marker coordinate value extraction unit 191 in the basic processing unit 10 of FIG. (B) in FIG. 12A and (c) in FIG. 12B show processing of the correction value calculation unit 194 for synthesis of the image synthesis processing unit 11. (D) of FIG. 12B shows the process of the whole area image display part 195. FIG. However, the actual image processing is not a calculation for all the pixel coordinates, but is a process for obtaining a correction value for coordinate conversion based on the coordinate value in the captured image 114 of the marker image 103 as a base point. (C) is the figure which showed the image of the coordinate system to be converted as a binarized marker coordinate system in an easy-to-understand manner.

  In FIG. 12A (a), the marker image 115 is an image that has been binarized by providing a threshold value for the brightness of the captured image 114 shown in FIG. 9B. A white point in the marker image 115 is the marker image 103. (A) shows the marker images 115 (m, 1) O to 115 (m, n) F corresponding to m rows of the photographed image group in FIG. 9. Image IDs of the marker images 115: 115 (m, 1) O to 115 (m, n) F and marker images 103-m-2-FS to 103-m- (n + 1) -FS in the marker image 115 The barycentric coordinate value is extracted by the marker coordinate value extracting unit 191 and held in the marker information holding unit 192. The marker coordinate system conversion process will be described below using affine transformation display.

  In general, when a coordinate matrix for converting a coordinate value expressed in a coordinate system of (bx, by) to a coordinate system of (ax, ay) is [α], the following affine transformation equation (2 ).

  (Ax, ay, 1) = [α] (bx, by, 1) (2)

  Here, the transformation matrix [α] is expressed by the following formula (3).

  [Α] = [(A, −B, tx) (B, A, ty) (0, 0, 1)] (3)

  The transformation matrix [α] includes correction values that are four unknowns A, B, tx, and ty. A and B are expressed as A = R × cos θ and B = R × sin θ, using a polar coordinate expression when coordinate conversion to (ax, ay) is performed with reference to the coordinate system of (bx, by). The Here, R is a change in magnification of the dimension between coordinate systems when the coordinate system of (bx, by) is used as a reference, and θ is a rotation angle of the coordinate axes. Further, (tx, ty) represents the movement amount of the origin of both coordinate systems. For this reason, if the magnification and the shooting direction are the same between adjacent camera shooting areas, only one marker image 103 may be used to obtain a correction value for conversion between both coordinate systems. In this case, the coordinate value of one marker image 103 is substituted into equation (2) and A = 1 and B = 1, so that (tx, ty) is obtained to obtain the necessary coordinate conversion value, that is, synthesis. A correction value is obtained.

  In FIG. 12A, (b) is a diagram showing an image group of m rows in a coordinate system unified with the coordinate system of the reference image in m rows of (a). The reference image is, for example, a marker image 115 (1, 1) with m = 1 and n = 1. In (b), between the marker images 115 adjacent in m rows, a common marker image 103, for example, the first marker image 115 (m, 1) F and the next marker image 115 (m, 2) B The barycentric coordinate values of the marker images 103-m-2-FS, 103-m-2-BS, 103-m-2-FR, and 103-m-2-BR of the common markers are expressed by the following equation (4). By substituting into, four unknowns A, B, tx, ty of the transformation matrix [α1] are obtained.

(Am1x, am1y, 1) = [α1] (am2x, am2y, 1)
.... (4)

The barycentric coordinate value of the marker image 103-m-2-FS is (af, bf). The barycentric coordinate value of the marker image 103-m-2-BS is (ab, bb). The barycentric coordinate value of the marker image 103-m-2-FR is (xf, yf). The barycentric coordinate value of the marker image 103-m-2-BR is (xb.yb). Substituting these into equation (4) above,
(Af, bf, 1) = [α1] (xf, yf, 1)
(Ab, bb, 1) = [α1] (xb, yb, 1)
And 4 variables are obtained.

  Similarly, a transformation matrix [α (n−) indicating a correction value between the (n−1) th binarized image 115 (m, n−1) and the nth binarized image 115 (m, n). 1)] is expressed by the following equation (5).

(Am (n-1) x, am (n-1) y, 1)
= [Α (n-1)] (amnx, amny, 1) (5)

  The coordinate values of the n-th marker image 115 (m, n) are affine transformation coefficients from the first marker image 115 (m, 1), which is the reference image, to the n-th marker image 115 (m, n). Multiply the matrix. Thereby, the second and subsequent marker images 115 (m, 2),..., 115 (m, n) are unified into the pixel coordinate system of the first marker image 115 (m, 1) which is the reference image. When the coordinate system of the n-th marker image 115 (m, n) is uniformly expressed by the coordinate system of the marker image 115 (m, 1), the following equation (6) is obtained.

(Am1x, am1y, 1)
= [Α1] [α2]... [Α (n−1)] (amnx, amny, 1)
.... (6)

  Marker image groups displayed in the in-row unified coordinate system of the m-th row and n-th marker image 115 (m, n) thus obtained are the marker images 116 (m, 1) to 116 (m, n) in (b). n). Note that, in (b), the marker images 116 (m, 1) to 116 (m, n) for m rows are shown, but the markers for m = 1, 2, 3,. An image 116 is generated.

  In FIG. 12B (c), the composition correction value calculation unit 194 performs the in-row unified coordinate system and the m-row marker image of the m-1-row marker image group 116- (m-1) obtained in (b). The group 116-m is synthesized with the in-line unified coordinate system. The compositing correction value calculation unit 194 uses the marker in the marker image 103- (m−1) -2 at both ends in the common moving direction in the in-row unified coordinate system of the m−1th row and the in-row unified coordinate system of the mth row. The coordinate values of the images 103- (m-1) -2-FR and 103- (m-1) -2-FS and the marker image 103- (m-) in the marker image 103- (m-1)-(n + 1). 1)-(n + 1) -FR, 103- (m-1)-(n + 1) -FS coordinate values are substituted into affine transformation equation (2), and the transformation matrix [β (m-1 )]. The in-line unified coordinate system of the (m-1) th row is (a (m-1) 1x, a (m-1) 1y), and the in-row unified coordinate system of the mth row is (am1x, am1y). The formula assigned to the formula (2) is expressed by the following formula (7).

(A (m-1) 1x, a (m-1) 1y, 1)
= [Β (m−1)] (am1x, am1y, 1) (7)

  The equation for transforming the m-row unified coordinate system in the first row unified coordinate system (a11x, a11y) is expressed by the following equation (8) by sequentially multiplying the affine transformation matrix between the rows.

(A11x, a11y, 1)
= [Β1] [β2]... [Β (m−1)] (am1, am1y, 1)
.... (8)

  The n-th binarized image 116 (m, n) is a unified coordinate system of the entire matrix of the reference image, for example, the reference image of the binarized image 116 (1, 1). ) Is expressed by the following formula (9).

  (A11x, a11y, 1) = [β1] [β2] ... [β (m-1)] [α1] [α2] ... [α (n-1)] (amnx, amny, 1) ... (9)

  FIG. 12B (c) shows marker images 117 (1, 1) to 117 (m, n) in the unified coordinate system of the entire matrix. Multiplication affine transformation coefficients [β1] [β2]... [Β (m−1)] [α1] [α2]... [Α (n−1)] (= [[Beta]] [[alpha]]) corresponds to the final composition correction value for each of the marker images (1, 1) to 117 (m, n).

  In FIG. 12B (c), two marker images 103 having a wide interval in the arrangement direction of the cameras 2 are used. Pixel coordinateization includes cumulative errors such as particleization errors. Since the error element becomes smaller in inverse proportion to the interval, it is better that the interval between the two marker images 103 to be used is wider. When performing synthesis by multiplying affine transformation from the correction value of the marker image 103 of another image instead of the coordinate value of the marker image 103 that is directly shared between adjacent images in image synthesis, ideally the marker It is desirable to derive the shortest route from the route information and avoid accumulation of multiplication errors.

  In (d) of FIG. 12B, the image information I21 is obtained from the information acquisition unit 190 of the basic processing unit 10 using the correction value (affine transformation coefficient [β] [α] obtained in (c) of FIG. 12B). The operation of the entire area image display unit 195 that acquires and synthesizes the entire image will be specifically described.

  The entire area image display unit 195 converts each marker image 117 (m, n) indicated by the conversion of the entire matrix of all the images into the unified coordinate system (that is, the coordinate system of the reference image) as the original captured image 114 (m , N), the coordinate values of each marker image 117 (m, n) are replaced with the coordinate values as a series of synthesized images via the marker coordinate values.

  When synthesizing the captured image 114 (m, n), the entire area image display unit 195 uses the imaging time corresponding to the image ID from the information acquisition unit 190, and the imaging time is late, and the marker image 103 is at the end. The photographed image 114-B is overwritten with the photographed image 114 with the marker image 103 with the previous photographing time at the end. Finally, the entire area image display unit 195 overwrites the initial captured image 114-O without the marker image 103.

  In this way, the entire area image display unit 195 can synthesize a subject image in which the marker images 103 are sequentially hidden. The entire area image display unit 195 converts the coordinate value of each captured image 114 (m, n) into a matrix unified coordinate system, and generates a composite image 118 of the images 118 (1, 1) O to 118 (m, n) B. obtain. Here, it is easy to understand, and processing for displaying the composite image 118 has been shown, but the display range may be limited as necessary. In addition, the coordinate conversion image 118-B of the captured image 114-B with the earlier shooting time on the time series is overwritten with the coordinate converted image 118-B of the captured image 114-B with the later shooting time on the time series, and finally the marker image. Instead of the marker concealment procedure by overwriting with the coordinate conversion image 118-O of the initial photographed image 114-O without 103, the coordinate conversion image 118-F of the photographed image 114-F that is slow in time series The coordinate-converted image 118-F of the early captured image 114-F may be overwritten, and overwritten with the image captured without the marker image 103 at the end.

<Example of Real Coordinate-Based Image Processing of Marker Image 103>
13A to 13D are explanatory diagrams illustrating an example of image processing based on real coordinates of the marker image 103 according to the first embodiment. FIG. 12 illustrates correction value calculation and image synthesis for synthesizing the entire image using the coordinate values of the marker image 103 extracted from the captured image 114. 13A to 13D, the synthesis of the entire image with reference to the marker position coordinate system that is the actual coordinate system of the marker image 103 will be described. The marker image 103 in FIG. 13A is the same as (a) in FIG. 12. However, in FIG. 13A, since the coordinate unification processing between adjacent images is not performed, the marker image 115 (1, 1, 1) B to 115 (m, n) B are selected.

  FIG. 13B will be described. FIG. 13B shows marker positions 120-1-1-1 to 120-4-n in the real coordinate system (marker position coordinate system). In the functional block of the image processing apparatus 9 in FIG. 11, the physical actual interval size information of the marker laser 131 corresponding to the interval between the markers 31 and the marker movement distance information included in the imaging related information I11 are used as the imaging environment information I50. .

  The marker moving distance information includes marker moving distance information used in control, measured marker moving distance information, or marker moving distance information calculated from indirect parameters. The marker movement distance information used in the control is marker movement distance information when the interval between the movement processing unit 133 and the movement processing unit 132 is controlled using the adjustment cylinder 158 shown in FIG.

  The marker movement distance information measured by the laser distance measurement unit 157 is a measurement value by the laser distance measurement unit 157 shown in FIG. In addition to this, there are various methods for controlling and measuring the movement distance, such as reading a scale engraved on the track rail 105 and the like, and reading a rotational movement distance linked to the drive wheel.

  At this time, in order to associate the marker image 103 and the marker 3 in the photographed image 114 with each other, the photographing device 5 is identical to the photographing related information I11 so that the relationship between the marker moving distance information and the photographed image information I10 can be understood. Is given a shooting time or shooting number. The composition correction value calculation unit 194 configures the marker positions 120-1-1-1 to 120-4-n as the actual coordinate arrangement of the marker group on the lattice based on the physical actual interval size information and the marker movement distance information. .

  FIG. 13C is a diagram illustrating the transformation of the coordinate system of the marker image 103 with reference to the marker position 120 illustrated in FIG. 13B. The marker route information generation unit 193 identifies the identity between the marker position 120-mn of the marker position coordinate system of FIG. 13B and the marker image 103-mn-XY of FIG. The image 103 is associated. Specifically, if m and n have the same value, they are associated with each other. For example, the marker image 103-1-1-BS and the marker position 120-1-1 are associated with each other, and the marker image 103-2-1-1-BS and the marker position 120-2-1 are associated with each other. More specifically, the marker route information generation unit 193 refers to the relationship between the position of the physical camera 2 and the position of the marker providing unit 130 and the time or order of the captured image 114 and the marker movement distance information. Thus, the identity with the marker image 103 is identified, and the marker position 120-mn and the marker image 103-mn-XY are associated with each other.

  The pixel coordinate values of the marker images 103-1-1-BS and 103-1-1-BR of the marker image 115 (m, 1) B to the marker image 115 (m, n) B in FIG. The coordinate values of the coordinate positions 120-1-1 and 120-2-1 in the marker position real coordinate system of the marker 3-1 corresponding to the images 103-1-1-BS and 103-1-1-BR, Substituting into Equation (2), a transformation matrix [γ11] for affine transformation is calculated. The real coordinate system (cx, cy) of the marker image 115 (1, 1) is expressed by the following equation (10) by an affine transformation matrix [γ11].

  (Cx, cy, 1) = [γ11] (a11x, a11y, 1) (10)

  Similarly, this conversion is performed by using marker coordinates 115-mn-BS and 115-mn-BR and marker positions 120-mn and 120-m- (n + 1) of an arbitrary marker image 115 (m, n) B. ) To obtain [γmn], and the marker images 121 (1, 1) B to 121 (m, n) as a conversion system from the pixel coordinate value of each image to the entire unified coordinate in the actual coordinate system of the marker position are obtained. B) In FIG. 13C, conversion correction values [γ11] to [γmn] for 121 (1,1) B to 121 (3, n) B are obtained.

  FIG. 13D applies the marker image 121 (m, n) B of each image indicated by the conversion to the overall unified coordinate system to the original captured image 114 (m, n) B, as in (d) of FIG. 12B. By doing so, a series of synthesized images obtained by converting into the overall unified coordinates via the marker coordinates can be obtained. The entire image display unit 195 sequentially overwrites the captured image 114 with the marker image 103 at the tail end and the captured image 114 with the marker image 103 at the tail end of the previous captured image. Eventually, the entire area image display unit 195 overwrites the initial captured image 114 (1,1) O to 114 (3,1) O without the marker image 103.

  In addition, the coordinate conversion image 122-B of the time-sequentially captured image 114-B is overwritten with the coordinate conversion image 122-B of the chronologically-slowly captured image 114-B, and finally the initial imaging without the marker image 103 is performed. In place of the marker concealment procedure by overwriting the coordinate conversion image 122-O of the image 114-O with the coordinate conversion image 122-O, the coordinate conversion image 122-F of the photographed image 114-F which is not shown in the figure but is slow in time series is time series. The coordinate-converted image 122-F of the upper captured image 114-F may be overwritten and overwritten with an image captured without the marker image 103 at the end.

  In this way, the entire area image display unit 195 can synthesize the original subject image in which the marker images 103 are sequentially hidden. The entire area image display unit 195 converts the real coordinates into a coordinate system having a unified coordinate system, and obtains a composite image 122 of the selected images 122 (1, 1) O to 122 (3, n) B. Here, the processing intended to display the composite image 122 is easy to understand, but the display range may be limited as necessary.

  In the image composition using the marker image 103 of the common marker 3, a particleization error always occurs due to the pixelization of the pixel, and further, imaging distortion is added. Cumulative misalignment occurs. When a full HD of 1920 × 1080 pixels is used as the camera 2 for photographing, if the resolution for photographing the crack 41 of 0.2 [mm] is set to the pixel interval, the photographing region 113 is 384 [mm] × 216 [ mm].

  For example, in the correction value processing of only the marker image shown in FIG. 12, the error is accumulated in the conversion to the unified coordinate system, and the particleization error due to the pixelation at the center of gravity of the marker image 103 in the captured image 114 is the maximum. Causes an error of ± 0.2 mm × n. When a commercially available laser distance measuring device is applied as the laser distance measuring unit 157 shown in FIG. 2, the position accuracy of the marker moving distance information is about ± 1 [mm]. Simply, when the number n of moving shooting is n = 5 or more, the correction value using the actual marker unified coordinate system by the laser distance measuring device is more than the image synthesis by the correction value between adjacent images based on the marker image. The image is less distorted. Conversely, if the display range is less than or equal to the range of five images, the accuracy of alignment between images at the time of image synthesis is higher when the correction value between adjacent images based on the marker image is used.

  However, in the technique shown in FIG. 12, in order to display the entire image, the initial assigned marker and the final assigned marker are used as a reference for the image synthesis of the unified coordinate system of (m−1) and m rows. The image group located in the middle of the row is affected by the shift due to the accumulation of the graining error. As described above, in the display image composition, the calculation means for the composition correction value with the optimum alignment accuracy between images differs depending on the number of images included in the display range, the measurement accuracy of the marker coordinate position, the distortion accompanying the photographing, and the like.

  In the image processing in the composite image selection unit 196, the correction value calculation unit 197, and the synthesis unit 198 in FIG. 11, the range of the image group that is the target in the composite image selection unit 196 based on the display area instruction I52 is, for example, five images. In the wide case, if there is no correction value or real coordinate information for the real coordinate unified coordinate system, the output information I53 is output by applying the correction value for the standard image coordinate unified coordinate system.

  However, when the display range is narrow, for example, when the number of images is 5 or less, the synthesis process for obtaining the alignment accuracy at the pixel level in a narrow range by the adjacent image group is more suitable. Further, when the actual coordinate value is applied to the marker position coordinate system, the coordinate system of the composite image corrected in the marker position coordinate system reflects the actual size as it is. Therefore, the crack size and the size of the object in the image can be directly obtained as the pixel distance.

  14A and 14B are explanatory diagrams illustrating an example of image processing between adjacent images according to the first embodiment. FIG. 14A is a configuration diagram of marker images 123 (C−1, C−1) to 123 (C + 1, C + 1) when a 3 × 3 image is selected corresponding to the display range. The image processing apparatus 9 acquires the common marker coordinate information 126 adjacent to the related image ID from the marker information holding unit 192. Further, the image processing apparatus 9 acquires the marker route information I29 from the marker route information generation unit 193, and sets the shortest route to each image in the display range with reference to the marker image 123 (C, C) that is the center of display. To do.

  The correction value calculation unit 197 sequentially converts the coordinate values of the common marker image 103 into the central image unified coordinate system based on the pixel coordinates of the central marker image 123 (C, C) according to the marker route using the formula ( Substituting into 2), the value of the affine transformation matrix is obtained. Further, the correction value calculation unit 197 performs matrix product multiplication on the route, and converts the marker images 123 (C-1, C-1) to 123 (after the conversion of the image in the display area into the central image unified coordinate system. C + 1, C + 1) is calculated.

  In FIG. 14B, as in (d) and FIG. 13D of FIG. 12B, a series of composite images can be obtained by applying coordinate transformation values to the central image unified coordinate system to the original photographed image 114 (m, n). . The entire image display unit 195 sequentially overwrites the captured image 114 with the marker image 103 at the tail end and the captured image 114 with the marker image 103 at the tail end of the previous captured image. Finally, the entire area image display unit 195 overwrites the initial captured image 114 without the marker image 103. The entire area image display unit 195 obtains a composite image 124 in which the original subject images 124 (C−1, C−1) B / O to 124 (C + 1, C + 1) B in which the marker images 103 are sequentially hidden are combined.

  “B / O” means that the image in which the marker image 103 is arranged at the tail is a B-type image or an O-type image without the marker image 103. Here, a 3 × 3 image configuration is shown in an easy-to-understand manner, but the procedure is the same for m × n. In addition, the coordinate conversion image 124-B of the time-sequentially captured image 114-B is overwritten with the coordinate conversion image 124-B of the chronologically-slowly captured image 114-B, and finally the initial imaging without the marker image 103 is performed. In place of the marker concealment procedure by overwriting with the coordinate conversion image 124-O of the image 114-O, the time series of the coordinate conversion image 124-F of the captured image 114-F that is not shown in the figure is slow in time series. The coordinate-converted image 124-F of the upper captured image 114-F may be overwritten and overwritten with the image captured without the marker image 103 at the end.

<Example of marker coordinate value calculation processing procedure>
FIG. 15 is a flowchart illustrating an example of a marker coordinate value calculation processing procedure common to the first to third embodiments. This flowchart is processing executed by the basic processing unit 10 of the image processing apparatus 9 of FIG. In the basic processing unit 10, first, the information acquisition unit 190 acquires the captured image information I 10 (a set of the captured images 114) and the capturing related information I 11 output from the capturing device 5 and assigns an image ID to the captured image 114. (Step S50).

  Next, the marker coordinate value extraction unit 191 performs a binarization process by setting a threshold value for the luminance of each captured image 114, and creates a marker image 115 (step S51). Specifically, for example, the marker coordinate value extraction unit 191 acquires the imaging related information I11 from the information acquisition unit 190. In addition, the marker coordinate value extraction unit 191 acquires the monitor image information I12 output from the imaging device 5 and the ID / related information I28 output from the information acquisition unit 190. Then, the marker coordinate value extraction unit 191 uses the acquired information I11, I12, and I28 to set a threshold value for the luminance of each captured image 114, perform binarization processing, and create a marker image 115.

  Next, the marker coordinate value extraction unit 191 detects the marker image 103 from the marker image 115 and calculates the barycentric coordinate value of the marker image 103. The marker coordinate value extraction unit 191 includes the ID of the image to be processed, the position of the marker centroid coordinate value in the image (for example, upper right, lower right, upper left, lower left, etc.), the shooting time or the shooting order, etc. Using the imaging related information I11, a marker ID is assigned as information related to the marker image 103 (step S52).

  Next, the marker coordinate value extraction unit 191 holds the marker information holding unit 192 in association with the image ID, the marker ID, and the barycentric coordinate value of the marker image 103 (step S53). Thereby, the basic processing unit 10 ends the series of processes.

<Example of image composition process>
FIG. 16 is a flowchart illustrating an example of an image composition processing procedure common to the first to third embodiments. This flowchart is processing executed by the image composition processing unit 11 of the image processing apparatus 9 of FIG. In the image composition processing unit 11, first, the composition correction value calculation unit 194 acquires the shooting environment information I50 (step S55). As described above, the shooting environment information I50 is additional information from the outside such as the actual size information of the marker arrangement interval. The shooting environment information I50 is used for image processing based on the marker coordinate position.

  Next, the composition correction value calculation unit 194 receives the image ID, the marker ID, and the barycentric coordinate value of the marker image 103 from the marker coordinate information unit 192, and the information indicating the relationship between the marker IDs from the marker route information generation unit 193. get. Next, the composition correction value calculation unit 194 executes a correction value calculation process for the entire unified coordinate system (step S57). Details of the correction value calculation process (step S57) of the overall unified coordinate system are shown in FIG. 17A in the first embodiment, FIG. 25 in the second embodiment, and FIG. 34 in the third embodiment.

  Next, the entire area image display unit 195 converts the composite image obtained by the correction value calculation process (step S57) of the entire unified coordinate system into an output coordinate form and outputs it (step S58). Then, the composite image selection unit 196 receives a display area designation by an instruction operation based on the display information I51 or a manual operation (step S59).

  Next, the composite image selection unit 196 selects an image ID in the designated area designated in step S59 (step S60). For example, when the designated area is a wide range as in the entire image display (step S61: Yes), the correction value calculation unit 197 performs correction in the entire image unified coordinate system such as the marker position actual coordinate system calculated in step S57. A value is selected, and the process proceeds to step S63. On the other hand, when it is not a wide range (step S61: No), it transfers to step S62.

  In step S62, the correction value calculation unit 197 executes a composite image creation process by a correction value process between adjacent images (step S62). Details of the composite image creation process (step S62) by the correction value process between the close-up images are shown in FIG. 17C in the first embodiment and in FIG. 26B in the second embodiment. Thereafter, the process proceeds to step S64.

  In step S63, the correction value calculated in step S57 is applied to the image in the designated area to create a composite image that hides the marker image 103 (step S63), and the process proceeds to step S64. Then, the synthesis unit 198 converts the synthesized image created in step S62 or S63 into an output coordinate form and outputs it (step S64).

  Thereafter, the composite image selection unit 196 determines whether or not there is a change in the designated area (step S65). That is, the composite image selection unit 196 determines whether or not there is a display area instruction I52. If there is a change in the designated area (step S65: Yes), the process returns to step S60. On the other hand, when there is no change in the designated area (step S65: No), the image composition processing unit 11 ends the series of processes.

<Correction value calculation process of entire unified coordinate system (step S57)>
FIG. 17A is a flowchart showing a detailed processing procedure example of the correction value calculation processing (step S57) of the entire unified coordinate system shown in FIG. The overall unified coordinate system correction value calculation process (step S57) is an all-image correction value calculation process based on the coordinate position of the marker image 103.

  First, the composition correction value calculation unit 194 obtains marker arrangement interval information included in the shooting environment information I50 or the marker route information I29 (step S66). Next, the composition correction value calculation unit 194 determines whether or not the actual size is included in the marker arrangement interval information (step S67). If the actual dimension is included (step S67: Yes), the marker actual dimension coordinate system can be configured, and the process proceeds to step S69. In this case, the composition process shown in FIG. 13 is executed. On the other hand, when the actual dimension is not included (step S67: No), the process moves to step S68 because the marker actual dimension coordinate system is not applied. In this case, the composition process shown in FIG. 12 is executed.

  In step S68, the composition correction value calculation unit 194 executes a standard image unified coordinate correction value calculation process (step S68), and proceeds to step S71. Details of the reference image unified coordinate correction value calculation process (step S68) will be described with reference to FIG. 17B.

  In the case of Step S67: Yes, the composition correction value calculation unit 194 sets a marker position coordinate system based on the actual dimension coordinates of the marker corresponding to FIG. 13B (Step S69). Then, the composition correction value calculation unit 194 calculates the actual image corresponding to the marker image 121 in FIG. 13C from the in-image coordinate values of the marker image 103 identified by the image ID corresponding to the barycentric coordinate values of the marker images 103. A correction value for one coordinate system coordinate is calculated (step S70).

  Thereafter, the entire area image display unit 195 generates a composite image and stores the correction value in step S70 used for generating the composite image (step S71). Specifically, for example, when the correction value is calculated in step S70, the entire area image display unit 195 sequentially changes the captured image 114 to the captured image 114 with the latest capturing time and the marker image 103 as shown in FIG. 13D. The marker image 103 with the previous shooting time is overwritten with the shot image 114 at the end. Finally, the entire area image display unit 195 overwrites the initial captured image 114 (1, 1) O to 114 (3, 1) O without the marker image 103. Thereby, the composite image 122 is obtained.

  Further, when the correction value is calculated in step S68, the composition correction value calculation unit 194, as shown in (d) of FIG. 12B, the captured image is delayed and the marker image 103 is at the end. B is overwritten with the captured image 114 with the marker image 103 with the previous capturing time at the end. Finally, the entire area image display unit 195 overwrites the initial captured image 114-O without the marker image 103. Thereby, the composite image 118 is obtained. Thereby, the correction value calculation process (step S57) of the entire unified coordinate system is ended.

<Reference Image Unified Coordinate Correction Value Calculation Processing (Step S68)>
FIG. 17B is a flowchart illustrating a detailed processing procedure example of the reference image unified coordinate correction value calculation process (step S68) illustrated in FIG. 17A. FIG. 17B is a total image correction value calculation process using only the marker image 103 in the captured image 114. In FIG. 17B, steps S75 to S81 correspond to (a) and (b) in FIG. 12A, and steps S82 to S86 correspond to (c) and (d) in FIG. 12B. In the initial state, m = 1 and n = 1.

  The composition correction value calculation unit 194 acquires a marker image 115 that is a binarized image of the captured image 114 captured by the m-th camera 2-m (step S75). Referring to (a) of FIG. 12A, the marker image 115 is a marker image 115 (m, 1) O, 115 (m, 1) F, 115 (m, 2) B, 115 (m, 2) F,. , 115 (m, n) B, 115 (m, n) F. Then, the composition correction value calculation unit 194 acquires the barycentric coordinate value of each marker image 103 of the marker image 115.

  Next, the composition correction value calculation unit 194 performs marker coordinate information (marker images 103-m-2-FS, 103-m-2-FR) related to the first marker image 115 (m, 1) F in the moving direction. Is obtained) (step S76). Then, the composition correction value calculation unit 194 increments n, and uses the coordinate value of the marker image 103 that is common between adjacent images in FIG. ] Is calculated (step S77). For example, when n = 1, the marker image 115 (m, 1) F and the incremented (n = 2) marker image 115 (m, 2) B are adjacent images. The marker images 103 common to the adjacent images are the marker image 103-m-2-FS and the marker image 103-m-2-BS, and the marker image 103-m-2-FR and the marker image 103-m-. 2-BR.

  Therefore, the composition correction value calculation unit 194 substitutes the coordinate value of the marker image 103-m-2-FS for the left side of Expression (5), and sets the coordinate value of the marker image 103-m-2-BS to Expression (5). ) To calculate the [α1]. Similarly, the composition correction value calculation unit 194 substitutes the coordinate value of the marker image 103-m-2-FR for the left side of Expression (5), and sets the coordinate value of the marker image 103-m-2-BR to Expression ( Substituting into the right side of 5), [α1] is calculated. The composition correction value calculation unit 194 outputs a composition correction value [α1], which is a transformation matrix in which elements are finally determined from both calculation results.

  Then, the composition correction value calculation unit 194 determines whether or not the nth marker image 115 (m, n) exists (step S78). When the n-th marker image 115 (m, n) exists (step S78: Yes), the composition correction value calculation unit 194 re-executes step S77. On the other hand, when the n-th marker image 115 (m, n) does not exist (step S78: No), the composition correction value calculation unit 194 calculates the correction value matrix product [α1] [α2]. −1)] (= [α]). The correction value matrix product [α] is an affine transformation matrix from (a) to (b) in FIG. 12A.

  Thereafter, the composition correction value calculation unit 194 increments m (step S80), and determines whether or not the m-th camera exists (step S81). When the m-th camera exists (step S81: Yes). The composition correction value calculation unit 194 re-executes step S75. On the other hand, when the m-th camera does not exist (step S81: No), the process proceeds to step S82. Thereby, the processing up to (b) in FIG. 12A is completed, and m in-row unified images 116-1, 116-2,..., 116-m connected in the moving direction are generated.

  Next, the composition correction value calculation unit 194 sets the coordinate system of the specific camera 2 (for example, the leftmost camera 2-1 where m = 1) as the reference coordinate system (step S82), and sets m to m = Reset to 1.

  Then, the composition correction value calculation unit 194 increments m, and uses the coordinate value of the marker image 103 common between adjacent in-line unified images in FIG. -1)] is calculated (step S83). For example, the in-row unified image 116- (m-1) and the in-row unified image 116-m are adjacent in-row unified images. The marker image 103 common between the adjacent in-row unified images is the marker image 103- (m−1) -2 of the marker image 116 (m−1,1) and the marker image 103 of the marker image 116 (m, 1). -(M-1) -2, the marker image 103- (m-1)-(n + 1) of the marker image 116 (m-1, n) and the marker image 103- () of the marker image 116 (m, n). m-1)-(n + 1).

  Therefore, the composition correction value calculation unit 194 substitutes the coordinate value of the marker image 103- (m−1) -2 of the marker image 116 (m−1,1) for the left side of the expression (7), and the marker image 116 [Β (n-1)] is calculated by substituting the coordinate value of the marker image 103- (m-1) -2 of (m, 1) into the right side of Expression (7). Similarly, the composition correction value calculation unit 194 assigns the coordinate value of the marker image 103- (m−1) − (n + 1) of the marker image 116 (m−1, n) to the left side of Expression (7), [Β (n-1)] is calculated by substituting the coordinate value of the marker image 103- (m-1)-(n + 1) of the marker image 116 (m, n) into the right side of Expression (7). The composition correction value calculation unit 194 outputs a composition correction value [β (n−1)], which is a transformation matrix whose elements are finally determined from both calculation results.

  Thereafter, it is determined whether or not the camera 2-m exists (step S84). When the camera 2-m exists (step S84: Yes), the composition correction value calculation unit 194 re-executes step S83. To do. On the other hand, when the camera 2-m does not exist (step S84: No), the composition correction value calculation unit 194 calculates the correction value matrix product [β1] [β2]... [Β (m−1)] (= [ β]) is calculated (step S85). The correction value matrix product [β] is an affine transformation matrix in (c) of FIG. 12B.

  Then, the composition correction value calculation unit 194 combines the correction value matrix product [α] calculated in step S79 and the correction value matrix product [β] calculated in step S85, for example, a reference image of all images, for example, the marker image 115. A conversion correction value [β] [α] to a matrix unified coordinate system using the pixel coordinates of (1, 1) F as a coordinate system is calculated (step S86). Thereby, the reference image unified coordinate correction value calculation process (step S68) is completed, and the process proceeds to step S71.

<Composite Image Creation Processing by Correction Value Processing Between Close-Up Images (Step S62)>
FIG. 17C is a flowchart illustrating a detailed processing procedure example of the composite image creation process (step S62) based on the correction value process between the adjacent images illustrated in FIG. FIG. 17C corresponds to the combining process of FIGS. 14A and 14B.

  The composite image selection unit 196 sets a captured image ID corresponding to the display information I15, and selects an image ID corresponding to the center image in FIG. 14A from the set captured image ID (step S90). Next, the correction value calculation unit 197 selects the marker ID of the marker image 103 that is common to the center image ID and the adjacent image ID, and acquires the marker coordinate information I26 from the marker information holding unit 192 (step S91). .

  The correction value calculation unit 197 calculates a composite correction value between the images from the coordinates of the common marker between the images adjacent to the center image (step S92). Next, the correction value calculation unit 197 acquires the marker route information I29 of the related marker from the marker route information generation unit 193, and sets the shortest conversion route from the central image for the image to be output / displayed (step) S93).

  After that, the correction value calculation unit 197 obtains a correction value between adjacent images along the conversion route, and multiplies the correction value matrix along the route to perform unified coordinate conversion into a central image corresponding to FIG. 14A. A correction value is calculated (step S94). Then, the synthesis unit 198 applies the unified coordinate conversion correction value calculated in step S94, and converts the image group (124 (C-1, C-1) -B / O converted into the central image unified coordinates shown in FIG. 14B. ˜124 (C + 1, C + 1) B), the previous image is overwritten on the subsequent image in time series, thereby creating the composite image 124 which is the marker concealed image shown in FIG. 14B (step S95). Thereafter, the process returns to step S62 in FIG. 17A.

  In coordinate conversion based on common marker coordinates between adjacent images, accumulation of errors due to pixel particleization errors and distortion is small. Further, in comparison with the conversion to the central image unified coordinate system in the entire image system, in the image alignment at the pixel level, for example, in the case of photographing the crack 41, the image synthesis with an accuracy of about 0.2 [mm] is performed. In addition, a high-accuracy image combination can be obtained as compared with the entire image alignment of 1 [mm] order.

  In the first embodiment, when obtaining a matrix unified coordinate system, a correction value [α] is first calculated between images in the moving direction taken by the same camera to form a unified coordinate system, and then a correction value [ β] was obtained, but this order may be reversed.

  Here, the camera unit 102 and the marker applying unit 130 are alternately moved on the track rails 105 disposed on both sides of the subject by the driving wheel 153 and the guide wheel 156 of the movement processing unit 132 and the movement processing unit 133. Although the configuration has been described, the track rail 105, the drive wheel 153, and the guide wheel 156 may be replaced with wheels or a crawler belt that travels on the subject. When contact with the subject is permitted, the effect of the first embodiment can be obtained in the same manner.

<Effect of Example 1>
The embodiment 1 described above has the following effects (A) to (H).
(A) The multi-image photographing system 1 that photographs by moving the camera 2 is a mechanism that alternately moves or stops the camera device 8 and the marker applying device 6 independently. Thereby, the marker provision apparatus 6 can be independently fixed to the subject. Therefore, it is not limited to low-brightness markers with adhesive properties such as droplets, but a high-brightness marker image that can irradiate a subject with a high-brightness marker using an active light source such as a laser and can be clearly distinguished from the surroundings. Can be obtained. In addition, it is possible to execute a binarized image that can be easily processed in order to detect the barycentric coordinate value of the marker image in the image processing.

(B) By using a mechanism that alternately moves or stops the camera device 8 and the marker applying device 6, the marker applying interval can be maximized. Therefore, the number of shots can be reduced, and efficient operation is possible.

(C) By adopting a configuration in which a plurality of cameras 2 move in parallel on the subject, close-up shooting is possible in an environment where close-up shooting is possible, and the moving distance of the camera device 8 and the area of the shot image 114 are linear. Therefore, the captured image 114 with less distortion can be acquired.

(D) Necessary number of markers 3 is given by adding the marker 3 to the common area of the adjacent imaging areas 113 in the arrangement direction of the cameras 2 and the common area of the adjacent imaging areas 113 moving back and forth in the moving direction. Can be reduced.

(E) When performing image synthesis between the in-row unified images 116- (m−1) and 116-m adjacent in the arrangement direction of the camera 2, the in-row unified images 116- (m−1) and 116-m have an interval. By selecting a wide common marker image 103 (see FIG. 12C) or by selecting so that the longest distance of the marker route between adjacent images in image matching is shortened, image misalignment can be reduced. it can.

(F) Also, as shown in FIG. 13, the number of synthesized images increases by acquiring information defining the marker position related to the marker interval size, moving distance, etc. and using it for calculating the correction value of the image synthesis process. In the image composition in a wide range, the influence of the particleization error due to pixelation and the accumulation of photographing distortion is eliminated at the time of composition of the photographed image generated by multiplying only the correction values between adjacent images, and the image misalignment is reduced. Information equivalent to actual measurement such as crack dimensions can be automatically calculated from the image.

(G) The multi-image shooting system 1 applies a correction value between adjacent images when performing pixel-level synthesis between adjacent images when the output range of the synthesized image is narrow, so that the necessary image In the case of a wide range where the number increases, a correction value using a distant marker based on the marker route is applied, or a correction value based on the marker position is used. As a result, a composite image with less displacement and distortion between images can be provided using an image composition correction value suitable for the output image.

(H) When the captured image 114 is converted into a unified pixel coordinate and image synthesis is performed, there is no marker image at the position of the marker image of the image with the marker, the relationship between the chronological relationship of the shooting time and the arrangement of the marker image. By overwriting the image, it is possible to form a composite image of the subject in which the marker image is hidden.

  In the first embodiment, the camera 2 and the marker providing unit 130 move in parallel on the subject and shoot images. On the other hand, in the second embodiment, the camera 2 and the marker applying unit 130 are mounted on a pan head that moves to a pan angle (horizontal rotation) and a tilt angle (vertical rotation), and shooting is performed by changing the pan angle and the tilt angle. It is an example. Specifically, the multi-image shooting system 1 according to the second embodiment is shot from a fixed point away from the subject, and is shot by changing the pan angle and the tilt angle instead of the parallel movement of the camera 2 and the marker providing unit 130. To do. With this configuration, for example, there is a river flow around and it is difficult to approach, and it is possible to take a picture even when the parallel movement configuration as in the first embodiment is not applicable. For example, marker-based image composition can be performed even when shooting from a distant point where a bridge pier can be seen.

  In the second embodiment, the “pan marker” is a marker given in the pan direction, and the “tilt marker” is a marker given in the tilt direction.

<Appearance of multi-image shooting system 1>
FIG. 18 is an explanatory diagram of an appearance example of the multi-image capturing system 1 according to the second embodiment. When visually inspecting the crack 41 on the side wall of the bridge pier 43 of the bridge 42, etc., a river 44 flows underneath and is a scene where the camera 2 takes a picture remotely from the opposite shore. The camera 2 is mounted on a camera head 225 that can rotate in the pan direction and the tilt direction, and is fixed by a tripod.

  In order to provide a marker, the tripod includes a pan marker laser 231, a tilt marker laser 232-S, a tilt marker laser 232-R, a pan marker head 226, tilt marker heads 227-S and 227-R, Is fixed.

  The pan marker laser 231 gives the subject a pan marker set 3P that moves following the rotation of the camera 2 in the pan direction. The tilt marker laser 232-S follows the rotation in the tilt direction and applies the left end tilt marker set 3T-S to the subject. The tilt marker laser 232-R follows the rotation in the tilt direction and gives the right end tilt marker set 3T-R. The pan marker pan 226 rotates each marker laser in the pan direction. The tilt marker heads 227-S and 227-R rotate the marker lasers in the tilt direction. The pan marker laser 231 and the tilt marker lasers 232 -S and 232 -R may be the same color laser. For example, a red laser is used for the pan marker 231 and a green laser is used for the tilt marker laser 232. It is also possible to use different colors.

  When different colors are used, there is an advantage that it is possible to easily classify the image alignment marker in the pan direction and the image alignment marker in the tilt direction in the same image by using a red filter or a green filter in the image processing. Note that the wavelength filter may be provided in the optical system of the photographing camera, or red, green, and blue color element separation of the photographed image may be used. Hereinafter, in this embodiment, a technique that does not stick to the color usage of the marker laser will be described.

  It is connected to a control unit 7 that controls movement of each pan head, lighting / extinguishing of a marker, photographing of the camera 2, and the like, and is operated by an operator.

<Shooting route and shooting area in the multi-image shooting system 1>
FIG. 19 is an explanatory diagram of a shooting route and a shooting area in the multi-image shooting system 1 according to the second embodiment. In FIG. 19, in photographing the side wall of the pier 43, a pan marker set 3P that is a set of a plurality of pan markers, a left tilt marker set 3T-S that is a set of a plurality of left tilt markers, and a right tilt that is a set of a plurality of right tilt markers. The relationship between the initial position of the marker set 3T-R, the shooting route 212, and the shooting area 213 is shown.

  Here, as an example, the shooting route 212 is a route in which the marker 3 is moved from the upper left to the right side, and then sequentially moved to the end and turned back. The shooting route 212 is a route that places importance on the efficiency of movement of the camera 2, and may be a route that returns to the left end and repeats in the horizontal direction. Further, the route may not be continuously moved in the horizontal direction, but may be continuously moved in the vertical direction and turned back. The functional block diagram of the multi-image shooting system 1 of the second embodiment is similar to that shown in the first embodiment and conforms to FIG.

<Functional blocks of the photographing device 5>
FIG. 20 is a functional block diagram of the photographing apparatus 5 according to the second embodiment. The description will focus on the differences from the functional blocks of the imaging device 5 in the first embodiment shown in FIG. Moreover, the same code | symbol is attached | subjected to the same function and description is abbreviate | omitted.

  The camera device 8 includes a camera unit 202 including one camera 2 and a movement processing unit 132. The movement processing unit 132 includes a camera pan pan operation unit 241 and a camera tilt pan operation unit 242. The camera pan head operating unit 241 rotates and moves the camera unit 202 in the pan angle direction. The camera tilt pan head operating unit 242 rotates and moves the camera unit 202 in the tilt angle direction.

  The marker imparting device 6 includes a marker imparting unit 230 and a movement processing unit 133. The marker providing unit 230 includes a pan marker light source 235 such as a pan marker laser 231, a tilt marker S light source 236 such as a tilt marker laser 232 -S, and a tilt marker R light source 237 such as a tilt marker laser 232 -R. The movement processing unit 133 includes a pan marker pan pan operation unit 243, a pan marker tilt pan operation unit 244, a tilt marker S pan head operation unit 245, and a tilt marker R pan head operation unit 246.

  The control unit 7 is basically the same as that of the first embodiment, but the contents of the control signal and timing control are changed when the configuration of the movement processing unit 132 and the marker applying device 6 is changed to the pan head. After the initial shooting direction and marker application direction are set in the camera unit 202, the control unit 7 transmits a shooting control signal I1 from the shooting control unit 183 to the camera 2 at an external operation instruction or at a predetermined timing. Upon receiving the shooting control signal I1, the camera starts shooting, and transmits the shot image information I10 that is a set of the shot images 114 to the image processing device 9 shown in FIG.

  Further, in the case where the position of the marker 3 is monitored and controlled in the photographing apparatus 5, when the monitor image information I12 from the camera 2 is received by the monitor control unit 185, the control unit 7 processes the monitor image and performs the marker operation. A control signal is output to the movement control unit 186 so that 3 is arranged at an appropriate position. When the movement control unit 186 receives the control signal, each pan head (camera pan head 225, pan marker pan head 226, tilt marker pan head 227-S, 227-R) so that the marker is arranged at a predetermined position. To control.

  Specifically, the movement control unit 186 transmits the camera pan pan angle movement / stop signal I31 to the camera pan head operation unit 241 of the movement processing unit 132 as a movement during monitor control and photographing. The camera head pan angle movement / stop signal I31 is a signal that causes the camera pan head operation unit 241 to rotate the camera head 225 in the pan direction and to stop the rotation.

  In addition, the movement control unit 186 transmits the camera head tilt angle operation / stop signal I31 to the camera tilt head operation unit 242. The camera head tilt angle movement / stop signal I32 is a signal that causes the camera tilt head operating unit 242 to rotate the camera head 225 in the tilt direction and stop the rotation.

  The movement control unit 186 transmits a pan marker pan angle movement / stop signal I33 to the pan marker pan head operation unit 243. The pan marker pan angle movement / stop signal I33 is a signal for causing the pan marker pan head operating unit 243 to move the pan marker in the pan direction by rotating the pan marker pan head 226 and to stop the movement.

  The movement control unit 186 transmits a pan marker tilt angle movement / stop signal I34 to the pan marker tilt pan operation unit 244. The pan marker tilt angle movement / stop signal I34 is a signal that causes the pan marker tilt pan operation unit 244 to move the pan marker in the tilt direction by rotating the tilt angle of the tilt marker pan head 227 and to stop the movement.

  The movement control unit 186 transmits a tilt marker S angle movement / stop signal I35 to the tilt marker S pan head operation unit 245. The tilt marker S angle movement / stop signal I35 is a signal for causing the tilt marker S pan head operating unit 245 to move the tilt marker in the tilt direction by rotating the tilt angle of the tilt marker pan head 227-S and to stop the movement.

  The movement control unit 186 transmits a tilt marker R angle movement / stop signal I36 to the tilt marker R pan head operation unit 246. The tilt marker R angle movement / stop signal I36 is a signal for causing the tilt marker R pan head operating unit 246 to move the tilt marker in the tilt direction by rotating the tilt angle of the tilt marker pan head 227-R and to stop the movement.

  The monitor control unit 185 controls the camera shooting position and the marker application position. Specifically, for example, the monitor control unit 185 detects the marker position of the marker image of the high-intensity marker from the monitor image of the monitor image information I12 of the camera 2 by image recognition. Then, the monitor control unit 185 compares the detected marker position with a predetermined position in the imaging region that has been set in advance, identifies the positional deviation, and sends a correction value for the positional deviation to the movement control unit 186. By notifying, marker position control is executed. Also, the monitor control unit 185 performs position control by allowing the operator to view the monitor image information I12 output to the outside and inputting it to the monitor control unit 185 as monitor observation / manual position correction information I6. May be.

  Further, without using the monitor control, the control unit 7 matches the position setting by each camera platform (camera camera platform 225, pan marker camera platform 226, tilt marker camera platform 227-S, 227-R) with the environmental dimension information. The photographing position and the marker applying position may be set only by numerical control of the pan head.

  The timing control unit 182 controls the shooting timing by moving or stopping the camera device 8 or the marker applying device 6. Specifically, for example, the timing control unit 182 includes a camera head pan angle movement / stop signal I31, a camera head tilt angle movement / stop signal I32, and a pan marker pan related to the movement processing unit 132 and the movement processing unit 133. An angle movement / stop signal I33, a pan marker tilt angle movement / stop signal I34, a tilt marker S angle movement / stop signal I35, and a tilt marker R angle movement / stop signal I36 are received from the movement control unit 186.

  Upon receiving the camera movement / stop signal I3, the timing control unit 182 transmits a shooting control signal I1 to the camera 2 in accordance with the sequence shown in the timing chart of FIG. 21, and controls the shooting timing. Upon receiving the marker movement / stop signal I4, the timing control unit 182 transmits a marker on / off signal I30 to the pan marker light source 235, the tilt marker S light source 235, and the tilt marker R light source 237 of the marker applying unit 130, Control the timing of turning on or off. Thereby, a necessary marker can be added to an image that requires a marker and photographed.

  Further, the timing control unit 182 outputs the photographing related information I11 to the outside. The shooting related information I11 is meta information related to the image information. Specifically, for example, the shooting-related information I11 includes shooting time, shooting number, rotational movement / stop of the pan / tilt direction of the camera platform 225, pan / tilt direction of the pan marker platform 226, and the like. Rotation movement / stop, rotation movement / stop of tilt marker head 227-S equipped with left tilt marker laser 232-S, rotation movement / movement stop of tilt marker head 227-R equipped with right tilt marker laser 232-R, And information on the relationship between the camera number and the image information when there are a plurality of photographing cameras. The shooting related information I11 is state information related to setting the image ID as seen in the timing chart of FIG.

  Although not specifically shown in the second embodiment, each cloud is measured by separately measuring the distance to the subject using a laser distance measuring device or the like and obtaining the component angle with the subject from the three marker distances. The maximum movement interval value is converted into a distance from the setting angle of the camera platform (camera head 225, pan marker head 226, tilt marker head 227-S, 227-R) and the actual shooting area size and marker position interval information by trigonometry. However, it is also possible to include it in the imaging related information I11 as marker movement distance information.

<Timing chart of photographing apparatus 5>
FIG. 21 is a timing chart of the photographing apparatus 5 according to the second embodiment. In FIG. 21, (1) shows the timing of moving / stopping the pan angle of the camera 2 in the lateral direction, and (2) shows the timing of moving / stopping the tilt angle of the camera 2 in the downward direction. (3) shows the timing of moving / stopping the pan angle of the pan marker in the horizontal direction. (4) shows the timing of the downward movement / stop of the pan marker tilt angle. (5) shows the timing of turning on / off the pan marker.

  The tilt marker R angle in (6) indicates the timing of the downward movement / stop of the tilt angle of the tilt marker set 3T-R. (7) shows the timing of turning on / off the tilt marker R. The tilt marker S angle of (8) indicates the timing of moving / stopping the tilt angle of the tilt marker set 3T-S in the downward direction. (9) shows the timing of turning on / off the tilt marker S. (10) indicates the timing of still image shooting (different from the monitor image). (11) indicates the transmission timing of the image / photographing related information. (12) indicates a captured image ID that identifies a captured image captured by the still image capturing of (10). The captured image ID is based on information such as (1) the movement processing unit 132 or (2) the movement processing unit 133, such as the camera pan pan angle movement / stop signal I31 or the pan marker pan angle movement / stop signal I33. It is decided uniquely.

  The timing control unit 182 controls the timing of taking a still image by turning off all the markers 3 at the initial position. This captured image is the first captured image 214 (1,1) S that does not include a marker. Next, the timing control unit 182 shoots with the pan marker light source 235 and the tilt marker S light source 236 turned on to obtain a captured image 214 (1,1) R. Next, the panning angle of the camera platform is moved and stopped so that the pan marker set 3P is positioned at the left end of the photographing region 213, and the photographed image 214 (1,2) S is obtained.

  Next, the timing control unit 182 controls the pan marker pan head operation unit 243 to move and stop so that the pan marker set 3P is positioned at the right end of the imaging region 213, and the captured image 214 (1, 2). Get R. The timing control unit 182 alternately repeats the movement / stop of the camera pan angle and the movement / stop of the pan angle of the pan marker.

  When the captured image 214 (1, 4) S is obtained, the timing control unit 182 turns on the tilt marker R light source 237 disposed at the lower end of the imaging region 213 to obtain the captured image 214 (1, 4) R. Next, the timing control unit 182 moves and stops the camera tilt angle and the pan marker tilt angle so that the tilt marker set 3T-R comes to the upper end of the imaging region 213, turns off the pan marker light source 235, and performs imaging. An image 214 (2,4) R is obtained.

  Next, the timing control unit 182 moves and stops the tilt angle so that the tilt angle of the tilt marker R comes to the lower end of the imaging region 213, turns on the pan marker light source 235, and shoots the captured image 214 (2,4) S. Get. From the next time, the timing control unit 182 repeats photographing until the left end while alternately moving the camera pan angle and the pan marker pan angle horizontally in the reverse direction. When the captured image 214 (2,1) S at the left end is obtained, the flow in the tilt direction at the right end is reversed, the tilt marker R is replaced with the tilt marker S, and a similar sequence is executed.

<Shooting order on the subject and its shooting area group>
FIG. 22A and FIG. 22B are explanatory diagrams showing a shooting sequence on a subject and a shooting region group according to the second embodiment. FIG. 22A shows a flow of shooting by moving the camera 2 and the marker set 3 along the shooting route 212 as indicated by arrows and fixing them. FIG. 22B shows the relationship between the imaging regions 213 in the entire imaging range, the positions assigned to all marker sets 3, and the relationship between the IDs assigned to the respective imaging regions 213. Each marker set 3 is always assigned to a common area where adjacent shooting areas overlap. For example, in the shooting areas 213 (1, 1) and 213 (1, 2), two pan marker sets 3P-1-1U and 3P-1-1L are assigned to the common area. In addition, in the imaging regions 213 (1, 1) and 213 (2, 1), tilt marker sets 3T-1-1S and 3T-1-1R are added to the common region. The control unit 7 controls the angles of the camera head 225 and the marker heads 226 and 227, so that the tilt direction is set between the shooting area adjacent to the pan direction (left and right in FIG. 22) and the shooting areas at both ends of the pan direction. The marker set 3 is assigned to a common area where adjacent shooting areas overlap (upper and lower in FIG. 22), and shooting is performed to obtain a shot image.

  The pan marker set 3P moves horizontally along the moving route of the shooting area 213 of the camera 2 and goes down when reaching the end. However, the tilt marker set 3T remains at both ends, and the shooting area 213 descends. Move down to match. Hereinafter, the movement of the imaging region 213 of the camera 2 and each marker 3 will be specifically described with reference to (b).

  At the initial position of the camera 2, the control unit 7 turns off the left tilt marker set 3T-S and the pan marker set 3P. In this unlit state, the control unit 7 controls the camera 2 to photograph the photographing region 213 (1, 1).

  Next, the control unit 7 sets the left tilt marker set 3T-S and the pan marker set 3P in a common area of the shooting areas 213 (2, 1) and 213 (1, 2) scheduled to be adjacently shot. To light up. The illuminated left tilt marker set 3T-S and pan marker set 3P are left tilt marker set 3T-1-S1, 3T-1-1R and pan marker set 3P-1-1U, 3P-1-1L. In this lighting state, the control unit 7 controls the camera 2 to photograph the photographing region 213 (1, 1).

  Next, the control unit 7 moves the shooting area 213 of the camera 2 along the solid arrow, and the pan angle of the camera platform 225 so that the pan marker set 3P is positioned at the left end of the shooting area 213 (1, 2). To fix the camera head 225. The pan marker sets 3P at this fixed position are the pan marker sets 3P-1-1U and 3P-1-1L. In this fixed state, the control unit 7 controls the camera 2 and images the imaging region 213 (1, 2).

  Next, the control unit 7 adjusts the pan angle of the pan marker head 226 so that the pan marker set 3P is arranged at the right end of the imaging region 213 (1, 2) with the camera 2 fixed. The pan marker head 226 is fixed. The pan marker sets 3P at this fixed position are the pan marker sets 3P-1-2U and 3P-1-2L. In this fixed state, the control unit 7 controls the camera 2 to photograph the imaging region 213 (1, 2).

  Sequentially, the control unit 7 performs control so that the camera 2 and the pan marker set 3P are alternately moved to repeatedly execute the above-described imaging mode. When the shooting area 213 moves to the right end and becomes the shooting area 213 (1, 4), the right tilt marker platform 227 is arranged so that the right tilt marker set 3T-R is arranged at the lower end of the shooting area 213 (1, 4). -R is adjusted to turn off the right tilt marker set 3T-R. In this unlit state, the control unit 7 controls the camera 2 to photograph the photographing area 213 (1, 4), and then turns on the right tilt marker set 3T-R again, and again captures the photographing area 213 (1 , 4).

  The second embodiment is an example in which the position of the first embodiment in photographing is replaced with an angle. The maximum movement interval L0 in the pan direction and the tilt direction in the second embodiment is obtained by calculating the movement amount of the camera unit 202 and the movement amount of the marker providing unit 130 in the expression (1) shown in FIG. The pan angle or tilt angle of the camera head 225 corresponding to the movement amount of 213, the pan angle of the pan marker head corresponding to the movement amount of the pan marker set 3P in the pan direction, or the tilt directions of the tilt markers 3T-R and 3T-S Are replaced with tilt angles of tilt marker heads 227-R and 227-S corresponding to the movement amount of X, and X is a shooting angle range in the pan direction and tilt direction corresponding to the shooting area length, and a setting error is a setting angle of the head It is obtained by setting the error and the marker diameter to an equivalent angle.

<Example of Operation Processing Procedure of Imaging Device 5>
FIG. 23A is a flowchart illustrating an example of an operation processing procedure of the imaging apparatus 5 according to the second embodiment. First, the imaging device 5 initializes the functions of the camera device 8, the marker applying device 6, and the control unit 7 (step S100). Next, the photographing apparatus 5 sets the angle of the camera unit 202 at the initial photographing position, sets the angle of the pan marker laser 231 so that the pan marker set 3P is arranged at the initial predetermined position on the subject, and the tilt marker. The angle of the tilt marker laser 232 is set so that the set 3T is arranged at the initial setting position on the subject (step S101).

  Next, the imaging device 5 turns off the light sources 235 to 237 of the marker applying unit 230 and performs the first imaging (step S102). Then, the imaging device 5 includes pan marker sets 3P-1-1U and 3P-1-1L set in front of the moving direction of the imaging area 213 (1, 1), which is the initial imaging position, and the imaging area 213 (1, 1). The tilt marker sets 3T-1-1S and 3T-1-1R arranged in (1) are turned on (step S103).

  Next, the photographing device 5 photographs the photographing region 213 (1, 1) that is the initial photographing position, and obtains a photographed image (step S104). Then, the imaging device 5 sets an initial pan movement direction for moving the imaging region 213 and the pan marker set 3P (step S104). The pan movement direction is one of the pan directions (left and right) (right or left). Thereafter, the photographing apparatus 5 executes pan direction moving photographing processing (step S106). Details of the pan-direction moving shooting process (step S106) will be described with reference to FIG. 23B. After the pan direction moving photographing process (step S106), the photographing apparatus 5 executes a tilt direction operation process (step S107). Details of the tilt direction operation processing (step S107) will be described with reference to FIG. 23C.

  After the tilt direction operation process (step S107), the photographing apparatus 5 determines whether or not the final photographing within the photographing target range (step S108). Specifically, for example, when the imaging device 5 uses the imaging-related information I11, it is determined whether or not the final imaging is performed by detecting the final position within a movement range set in advance in the camera device 8 or the marker applying device 6. Judging. Further, when using the monitor image information I12, the photographing device 5 determines whether or not the final photographing is performed by instructing the final image photographing while the operator looks at the monitor image. If it is not the final photographing (step S108: No), the photographing device 5 changes the pan movement direction of the photographing region 213 and the pan marker set 3P to the reverse direction, and returns to step S106. On the other hand, if it is the final shooting (step S108: Yes), the shooting device 5 ends the series of processes.

  FIG. 23B is a flowchart illustrating a detailed processing procedure example of the pan-direction moving photographing process (step S106) illustrated in FIG. 23A. The imaging device 5 rotates the camera pan head 225 on which the camera 2 is mounted in the pan movement direction, thereby moving the pan marker set 3P already assigned to the imaging area 213 in the pan movement direction, and the same imaging area 213. The pan marker set 3P is stopped at a predetermined position at the end of the pan movement direction at (the end opposite to the pan movement direction) (step S110).

  Next, the photographing apparatus 5 determines whether or not the photographing region 213 of the camera 2 has arrived at the photographing region 213 at the end of the entire photographing range in the pan movement direction (step S111). Specifically, for example, as an initial setting, the imaging device 5 sets an upper limit movement amount in the pan direction as a movement limit in the movement control unit 186 of the control unit 7, and sets the pan movement direction in the pan movement direction. When the movement amount reaches the upper limit movement amount, the photographing apparatus 5 determines that the photographing region 213 of the camera 2 has arrived at the photographing region 213 at the end of the entire photographing range in the pan movement direction.

  If it has not arrived (step S111: No), the imaging device 5 captures the imaging region 213 at the current position and acquires a captured image (step S112). Then, the imaging device 5 rotates the pan marker light source 235 in the pan movement direction, thereby moving the pan marker set 3P in the pan movement direction while fixing the imaging region 213, so that the same imaging region 213 is moved forward in the pan movement direction. The pan marker set 3P is stopped at a predetermined position of the section (end on the pan movement direction side) (step S113). Thereafter, the photographing device 5 photographs the photographing region 213 at the current position, acquires a photographed image (step S114), and returns to step S110.

  In step S111, when the shooting area 213 of the camera 2 has arrived at the shooting area 213 at the end of the entire shooting range in the pan movement direction (step S111: Yes), the shooting apparatus 5 moves in the pan direction. The process (step S106) ends, and the process proceeds to a tilt direction operation process (step S107).

  FIG. 23C is a flowchart illustrating a detailed processing procedure example of the tilt direction operation processing (step S107) illustrated in FIG. 23A. The operator determines whether or not the tilt angle of the camera platform 225 on which the camera 2 is mounted is set to an angle (initial angle) that is the initial position of the scheduled shooting range (step S120). If it is the initial angle (step S120: Yes), the process proceeds to step S124. If it is not the initial angle (step S120: No), the process proceeds to step S121.

  When the initial photographing is finished and the tilt angle is not the initial angle (step S120: No), the photographing device 5 is disposed at a predetermined position on the final tail (opposite end of the tilt moving direction) in the tilt moving direction in the photographing region 213. The angle of the camera pan head 225 is controlled so as to include the tilt marker set 3T, and shooting is performed with the tilt marker set 3T and the pan marker set 3P turned on (step S121).

  Next, the photographing apparatus 5 determines whether or not the photographing region 213 of the camera 2 has arrived at the final position of the entire photographing range (step S122). Specifically, for example, as an initial setting, the photographing apparatus 5 sets an upper limit movement amount in the tilt direction that is a movement limit in the movement control unit 186 of the control unit 7, and the movement amount in the tilt movement direction is the upper limit. When the movement amount is reached, the photographing apparatus 5 determines that the photographing region 213 of the camera 2 has arrived at the final position of the entire photographing range.

  When the final position has been reached (step S122: Yes), the imaging device 5 ends the tilt direction operation process (step S107), and proceeds to step S108. On the other hand, when the final position has not been reached (step S122: No), the imaging apparatus 5 has the tilt marker set 3T at a predetermined position at the forefront part in the tilt movement direction (end on the tilt movement direction side) in the imaging region 213. The tilt angle of the tilt marker heads 227-S and 227-R is lowered so as to come to the tilt marker set 3T, the tilt marker set 3T is stopped, and the imaging region 213 at the current position is imaged (step S123). Then, the process proceeds to step S126.

  On the other hand, when the final position has not been reached (step S122: No), the photographing apparatus 5 turns off the tilt marker set 3T and turns on the pan marker set 3P, and sets the photographing area 213 at the current position. A photograph is taken and a photographed image is acquired (step S124). Next, the imaging device 5 turns on the tilt marker set 3T, images the imaging region 213 at the current position, acquires a captured image (step S125), and proceeds to step S126.

  In step S126, the imaging device 5 lowers the tilt angle of the camera unit 202 so that the tilt marker set 3T comes to a predetermined position in the imaging region 213 at the end of the tilt movement direction (opposite end of the tilt movement direction). The pan marker set 3P is stopped by lowering the tilt angle of the pan marker head 226 so that the marker set 3P is at a predetermined position, and the pan marker set 3P is turned off and photographed (step S126).

  Next, the imaging device 5 determines whether or not the tilt angle of the camera 2 has reached the final angle in the tilt movement direction over the entire target imaging range (step S127). Specifically, for example, the photographing apparatus 5 sets an upper limit movement amount in a tilt direction that is a movement restriction as an initial setting in the movement control unit 186 of the control unit 7 and sets the movement amount in the tilt movement direction. When the movement amount reaches the upper limit movement amount, the imaging device 5 determines that the imaging region 213 of the camera 2 has reached the final angle in the tilt movement direction in the entire target imaging range.

  If the tilt angle has not reached the final angle (step S127: No), the photographing apparatus 5 places the tilt marker set 3T at a predetermined position in the forefront portion (end on the moving direction side) of the tilt movement direction in the photographing region 213. The tilt marker pan heads 227-S and 227-R are lowered and stopped so that they are positioned, and the pan marker set 3P is also turned on to photograph (step S128), and the process returns to step S107 in FIG. 23A. When the tilt angle reaches the final angle (step S127: No), the photographing apparatus 5 turns off the pan marker set 3P and turns on the tilt marker set 3T to photograph the current photographing region 213. FIG. Return to step S107.

<Example of photographed image group>
FIG. 24 is an explanatory diagram of an example of a captured image group according to the second embodiment. FIG. 24 is a timing chart of FIG. 21 obtained by photographing the photographing region group along the photographing route on the subject shown in FIG. 22 and the operation processing procedure of the photographing device 5 shown in FIGS. 23A to 23C. A photographed image group corresponding to the ID of the photographed image 214 is shown. The captured image 214 (m, n) is an image obtained by capturing the imaging region 213 (m, n).

  The captured image 214 (1,1) O at the initial position is a captured image without a marker image in which all the markers 3 are turned off in the imaging region 213 (1,1) at the initial position. The photographed image 214 (1,1) R has a marker image 203P-1-1-R of the pan marker set 3P and a tilt marker set 3T-S because the marker is lit in the same photographing region 213 (1,1). It is a picked-up image including the marker image 203T-1-1-L.

  The photographed image 214 (1,2) S is a photographed image obtained by photographing the photographed area 213 (1,2) after moving the photographed area 213 (1,1) in the pan direction, and the photographed area 213 (1,1). 2) The left end marker image 203P-1-1-S corresponding to the tail end in the pan movement direction is included.

  The photographed image 214 (1.2) R moves the pan marker set 3P in the pan movement direction in the same photographing area 213 (1,2), and the foremost part in the pan movement direction in the photographing area 213 (1,2). It is the picked-up image image | photographed in the state arrange | positioned at the right end which hits. The photographed image 214 (1.2) R includes a marker image 203P-1-2-R at the right end corresponding to the frontmost part in the pan movement direction within the photographing region 213 (1, 2).

  A captured image 214 (1, n) S is an image captured when the imaging region 213 arrives at the right end, which is the end in the pan direction of the entire imaging range. Although the tilt marker set 3T-R is also included in the shooting region 213, the shot image 214 (1, n) S is shot in a state where the marker image is turned off to hide the marker image in the image composition processing.

  The captured image 214 (1, n) R is arranged at the lower end portion on the tilt movement direction side of the imaging region 213 for image synthesis in the movement in the tilt direction, and the lit image of the illuminated tilt marker set 3T-R. It includes a marker image 203T-1-n-L.

  The captured image 214 (2, n) R includes a marker image 203T-1-n-U for image alignment in the tilt direction at the upper end, which is the last in the movement direction, but a pan marker set for marker concealment processing. 3P is an image taken with the light turned off.

  The captured image 214 (2, n) S includes a marker image 203T-2-n-1 that is lit in the same imaging region 213 (2, n) and a marker image 203P-2- ( n-1) -S. In the following, along with the shooting route shown in FIG. 22, a group of all shot images shot in the flowchart shown in FIG. 23 is shown.

  The functional blocks for performing the image processing in the second embodiment conform to the functional blocks of the image processing apparatus 9 shown in FIG. 11 of the first embodiment. Differences from the first embodiment will be described below.

  The information acquisition unit 190 acquires the state information from (1) to (9) as the shooting related information I11 related to the identifier of the shot image group 214 of (12) shown in the timing chart of FIG. The shooting-related information I11 in the second embodiment includes shooting time, shooting number, rotational movement / stop of the camera-mounted pan head, rotational movement / stop of the camera-mounted pan head, and pan marker mounting unit-mounted pan head. Rotation movement / stop, Rotation movement / stop of tilt pan head with pan marker attachment unit, Rotation movement / stop of tilt pan head with left tilt marker attachment unit, Rotation movement / movement stop of tilt pan head with right tilt marker provision unit Information such as movement / stop operation and time information thereof, rotation angle information of each pan head, lighting / extinction of each marker, and relationship between camera number and image information if there are a plurality of photographing cameras.

  The information acquisition unit 190 refers to the shooting related information I11, specifies an image shot immediately after the pan marker is moved or stopped, and assigns the identifier “F” to the shot image. The information acquisition unit 190 refers to the shooting-related information I11, specifies an image shot immediately after the camera 2 is moved or stopped in the pan direction, and assigns an identifier “B” to the shot image. The information acquisition unit 190 refers to the shooting related information I11, identifies an image in which the related marker is turned off, and assigns an identifier “O” to the shot image. The information acquisition unit 190 refers to the imaging related information I11 and assigns “n” of the image matrix (m, n) according to the movement order of the camera 2 in the pan movement direction.

  The information acquisition unit 190 refers to the shooting-related information I11, applies “m” of the image matrix corresponding to the movement order in the tilt direction, and sets an image ID such as the shot image group of FIG. Give.

  The marker route information generation unit 193 creates marker route information that links the marker assignment related information and the marker ID indicating the position in the image of the marker image 203 in conjunction with the shooting order of the images. The marker addition related information is information including a pan marker pan angle move / stop signal I33, a tilt angle move / stop signal I34, tilt angle move / stop signals I35 and I36 of a tilt marker, and a marker point / light off signal I30. The marker route information generation unit 193 extracts the marker coordinate information I19 from the marker coordinate value extraction unit 191, and specifies the captured image 214 to which the marker image 203 belongs. Then, the marker route information generation unit 193 identifies the marker ID from the marker image ID and the image ID obtained when the marker image 203 is obtained from the position in the captured image 214.

  For example, regarding the marker 203P in the pan direction, if two markers arranged at the top and bottom in the image are near the left end of the left half, they are identified as “203P-S”, and if they are near the right end of the right half, “203P− R ". The marker 203T in the tilt direction is identified as “203T-U” from the two markers arranged on the left and right near the upper end near the center of the image area, and “203T-L” from the two markers arranged on the left and right near the lower end. ] And the marker image ID are derived, the ID of the marker “3P” in the pan direction and the marker “3T” in the tilt direction are derived from the relationship between the marker ID and the image ID, and the left and right are aligned along the image ID. The marker route and the upper and lower marker routes are configured.

  The correction value calculation unit 197 calculates the correction value for composition of the image selected by the composite image selection unit 196, as in the first embodiment. First, the correction value calculation unit 197 sets a reference image for composition correction value. For example, the correction value calculation unit 197 sets the center image of the selected image group as the reference image. An optimum route from the reference image to each image to be image-combined is set using the marker route information I29. Specifically, for example, the correction value calculation unit 197 calculates a matrix error between adjacent images on the route from the reference image to the target image, thereby calculating the quantization error inherent in the coordinate value of the marker image. In addition to accumulating, the lens aberration is acquired from the shooting environment information I50, so that the lens distortion and the matrix product based on the lens aberration are calculated.

  The image that is the target of image synthesis that is the end point of the route is the remaining image other than the reference image in the selected image group, but the correction value calculation unit 197 is the outermost image group of the selected image group. May be selected as an image to be synthesized as an end point of the route.

  Further, the correction value calculation unit 197 may select the optimum route in consideration of quantization error and lens distortion. Since the quantization error depends on the marker interval and the number of pixels, the correction value calculation unit 197 calculates the quantization error from the marker interval and the pixel density as the uncertainty of the coordinate value of the marker image for each link constituting the route. Turn into. Since the lens distortion depends on the position in the image, the correction value calculation unit 197 sets the lens distortion as a degree of uncertainty of the coordinate value of the marker image from the camera lens aberration measured in advance for each link constituting the route. Turn into. Then, the correction value calculation unit 197 calculates the cumulative uncertainty for each route by summing the uncertainty of each link, and selects the route with the smallest cumulative uncertainty as the optimum route. When the number of images is large and the number of route options increases, the influence of cumulative error or cumulative distortion for each route appears, so that an optimal route that does not depend only on the number of images on the route is selected.

  Finally, the correction value calculation unit 197 performs a coordinate transformation matrix product for matching the coordinate system of the target image with the coordinate system of the reference image along the optimal route that has been set (see the above equation (9)). Is calculated as a composition correction value for image composition.

  Also, the combining unit 198 selects and overwrites the original image without the marker image at the marker image position represented in the same unified coordinate system from the time series of the image on the marker route and the relationship of the marker arrangement. By doing so, the marker image is concealed. Thereby, the composition unit 198 outputs image output information in which the marker image is concealed.

<Reference image-based image processing>
FIG. 25A to FIG. 25D are explanatory diagrams illustrating a reference image-based image processing example according to the second embodiment. The relationship between the image processing in FIGS. 25A to 25D and the image processing apparatus 9 in FIG. 11 will be described. FIG. 25A shows the processing of the marker coordinate value extraction unit 191, FIGS. 25B and 25C show the processing of the correction value calculation unit for synthesis 194 of the image synthesis processing unit 11, and FIG. 25D shows the processing of the whole area image display unit 195. The corresponding marker image is shown.

  However, in actual image processing, it is not processing for calculating all the pixel coordinates, but processing for obtaining a correction value according to the equation (1) for coordinate conversion using the marker coordinates as a base point. FIGS. FIG. 6 is a diagram showing the image of the coordinate system to be converted as a binarized marker coordinate system in an easy-to-understand manner.

  FIG. 25A shows marker images 215 (m, 1) S to 215 (m, n) R in the m-th row of the image group obtained by binarizing the captured image 214 shown in FIG. Show. Here, “S” in ID (symbol) notation indicates left, “R” indicates right, “L” indicates below, and “U” indicates above. Specifically, the m rows are a group of images taken at the moving position in the m-th tilt direction in FIG. 22B, and n indicates the n-th moving position in the pan direction.

  The marker coordinate value extraction unit 191 includes marker images 203-m-1-R to 203-m- (n-1) in the marker images 215 (m, 1) S to 215 (m, n) R in the m-th row. The barycentric coordinate values such as −S and marker images 203T-m-1-L and 203T-mn-L are extracted and held in the marker information holding unit 192. In the present embodiment, a mode is described in which the barycentric coordinate of the marker image obtained by binarization is used as the reference coordinate for image matching, but the coordinate of the marker image for image matching is the barycentric coordinate of the binarized marker image For example, the marker image is uniquely determined in common regardless of the image, such as the center coordinate to which the distribution load of the binarized marker image is added and the quarter position on the X coordinate axis or the y coordinate axis. Coordinates are used to coordinate the marker image. Further, when the luminance distribution of the marker image is stable regardless of the shooting position, the coordinates of the luminance peak of the marker image may be simply used. In this case, there is an advantage that the marker coordinate value extraction process is reduced. The coordinates of the marker image mean the coordinates of the position specified by the uniquely determined definition related to the assigned marker.

  The marker coordinate system conversion process will be described below using affine transformation display. The processing flow is the same as in the first embodiment, and the conversion formula to be used is the same as in the first embodiment. FIG. 25B shows conversion to the in-row unified coordinate system of the m-th row, that is, correction value calculation. In the marker images of two markers that are common between adjacent images in the m rows in FIG. 25A, for example, in the marker images 215 (m, 1) R and 215 (m, 2) S, the marker image 203-m-1- By substituting the pixel coordinate values of SU, 203-m-1-RU and marker images 203-m-1-SL, 203-m-1-RL into equation (3), [α in equation (3) ] Are obtained.

  Assuming that the obtained affine transformation matrix [α] is [α1], the transformation from the coordinate system of the marker image 215 (m, 2) to the coordinate system of the marker image 215 (m, 1) is expressed by Expression (4). The Similarly, the marker image 215 (m, n-1) R and the marker image 215 (m, n) S are expressed by Expression (5) using [α (n-1)]. The coordinates of an arbitrary marker image are multiplied by an affine transformation coefficient matrix to obtain a matrix product [α1] [α2] as a transformation from a reference image, for example, a pixel coordinate system of 215 (m, 1) to in-line unified coordinates. ... [α (n-1)] can be used to express as in equation (6). In this way, the display in the in-line unified coordinate system with m rows becomes the marker images 216 (m, 1) to 216 (m, n).

  FIG. 25C shows conversion to a matrix unified coordinate system (overall unified coordinate system) between in-row unified coordinate systems, that is, correction value calculation. FIG. 25C shows a process of displaying and synthesizing the in-row unified coordinate system of the m−1 row image group obtained in FIG. 25B in the overall unified coordinate system of the m row image group.

  The combination correction value calculation unit 194 includes a marker image of the markers 3T- (m−1) −1S at both ends in the common moving direction in the in-row unified coordinate system of the m−1th row and the in-row unified coordinate system of the mth row. 203T- (m-1) -1-LS, 203T-m-1-US and marker image 203T- (m-1) -n-LR, 203T- of marker 3T- (m-1) -n-LR The coordinate value of (m-1) -n-UR is substituted into affine transformation equation (2) to obtain a transformation matrix [β (m-1)] of both. The in-line unified coordinate system of the (m-1) th row is (a (m-1) 1x, a (m-1) 1y), and the in-row unified coordinate system of the mth row is (am1x, am1y). The formula assigned to the formula (2) is expressed by the following formula (7).

  When transforming the in-row unified coordinate system of m rows using the in-row unified coordinate system of the first row, [β1] [β2]... [Β (m−1)] multiplied by the affine transformation matrix between the rows is It can be expressed as shown in equation (8). An arbitrary marker image 216 (m, n) in the matrix is expressed as a reference image, for example, a marker image 217 (m, n) at the unified coordinates in the matrix of the marker image 216 (1, 1), From equation (8), the matrix product [β1] [β2]... [Β (m−1)] [α1] [α2]. It can be expressed as follows.

  FIG. 25D shows transformation of the unified coordinate system of the matrix calculated by the processing up to FIG. 25C, that is, synthesis of the entire image using the correction value. The entire area image display unit 195 applies the marker image 217 (m, n) of the coordinate system of each image indicated by the conversion of the entire image to the matrix unified coordinate system to the original captured image 214 (m, n) S. Thus, the marker image 217 (m, n) is replaced with coordinate values as a series of synthesized images via the marker coordinates.

  When superimposing and synthesizing the captured images 214 (m, n), the synthesizing unit 198 uses the imaging time corresponding to the image ID from the information acquisition unit 190 and has a marker image with the latest imaging time at the end. The captured image 214 is overwritten with the image having the marker image with the earliest photographing time at the end. The composition unit 198 finally composes the image so as to overwrite the image without the initial marker image. In this way, the synthesis unit 198 can synthesize the original subject image in which the marker images are sequentially hidden.

  Using a group of captured images having a marker image behind the moving direction of the captured image, a coordinate-converted image 218-R of the captured image 214-R that is earlier in time series and a coordinate-converted image of the captured image 214-R that is later in time series. 218-R is overwritten, the coordinate conversion image 218-S of the time-series earlier captured image 214-S is overwritten with the coordinate-converted image 218-S of the later time-series captured image 214-S, and the tilt marker is also time-series. Instead of the marker concealment procedure by overwriting the coordinate conversion image of the upper image with the coordinate conversion image of the upper image in the time series, use the captured image group with the marker image ahead in the moving direction of the captured image. The coordinate conversion image 218-R of the late photographic image 214-R is overwritten with the coordinate conversion image 218-R of the chronologically early photographic image 214-R by the coordinate conversion image 218-R of the slow photographic image 214-R. -S overwrites the coordinate conversion image 218-S of the captured image 214-S that is earlier in time series, and also overwrites the tilt marker with the coordinate conversion image of the image that is earlier in time series and the coordinate conversion image of the earlier image in time series, and finally Similarly, a composite image in which the marker is concealed can be obtained even in the reverse procedure of overwriting the former with an image without a marker image.

  FIG. 25D shows the process of overwriting and the flow of overwriting, and the marker image used for image alignment and concealed is indicated by a broken line. Finally, the synthesis unit 198 generates a synthesized image by converting the images 218 (1,1) O to 218 (m, n) S into the matrix unified coordinate system. Here, processing that is easy to understand and is intended for display of the entire image is shown, but the display range may be limited as necessary.

<Example of image processing between adjacent images>
FIG. 26A and FIG. 26B are explanatory diagrams illustrating an example of image processing between adjacent images according to the second embodiment. FIG. 26A shows marker images 219 (c−1, c−1) to 123 (c + 1, c + 1) when a 3 × 3 image is selected corresponding to the display range. In the functional block of the image processing device 9 in FIG. 11, the composition correction value calculation unit 194 acquires the common marker coordinate information 126 adjacent to the related image ID from the marker information holding unit 192. Further, the correction value calculation unit 197 acquires the marker route information I29 from the marker route information generation unit 193, and uses the marker image 219 (c, c) that is the center of display as a reference to the shortest marker up to each image in the display range. Set the route.

  Sequentially, the composition correction value calculation unit 194 converts the marker image of the common marker into the central image unified coordinate system based on the pixel coordinates of the marker image 219 (c, c) of the central image according to the marker route. The value of the affine transformation matrix is obtained by substituting the coordinate values of The composition correction value calculation unit 194 multiplies the matrix product on the route, and converts the images in the display area into the central image unified coordinate system 219 (c-1, c-1) to 123 (c + 1, c + 1) is calculated.

  FIG. 26A shows a route when the tilt marker set 3T is included, but in the case where the marker image of the tilt marker set 3T is not included in the selected image from the display area instruction I52 in the composite image selection unit 196, the display range is shown. The shortest marker route to the inner image is derived and the conversion correction value is applied.

  In FIG. 26B, the synthesis unit 198 converts the converted value into the central image unified coordinate system by the matrix product of Expression (6) along the marker route with reference to the coordinates of the central image shown in FIG. By applying to 214 (m, n), the conversion value to the central image unified coordinate system by the matrix product of Expression (6) is replaced with the coordinate values to a series of synthesized images via the marker coordinate values.

  As for the composition, the composition unit 198 superimposes the images so that the image having the marker image with the oldest shooting time is overwritten, and finally overwrites the image with the initial (oldest) image without the marker image. To synthesize. In this way, the synthesis unit 198 obtains a synthesized image 220 that is a combination of the original subject images 220 (c−1, c−1) to 220 (c + 1, c + 1) in which the marker images are sequentially hidden. In addition, when the uppermost image 220 (c-1, c-1) has a marker image regarding the image overwriting by the marker image concealment of the synthesized image 220, the synthesizing unit 198 further displays the previous captured image. Conceal again.

  In the flowchart of the image processing of the second embodiment, the flowchart corresponding to the derivation of the marker coordinates in the basic processing unit 10 of the functional block of the image processing apparatus 9 of FIG. 11 is the same as that of the first embodiment shown in FIG. The basic flowchart is the same as FIG. 16 shown in the first embodiment. Here, a detailed flowchart of image composition correction value calculation and image composition is shown.

<Correction value calculation process of entire unified coordinate system (step S57)>
FIG. 27 is a flowchart of a detailed process procedure example of the correction value calculation process (step S57) of the overall unified coordinate system illustrated in FIG. 16 according to the second embodiment. In the initial state, m = 1 and n = 1.

  The composition correction value calculation unit 194 acquires the marker coordinate value of the marker image group at the m = 1st tilt position (step S175). Specifically, the marker image group at the m = 1st tilt position is the marker image 214 (1,1) to 214 (1, n) in the m = 1st row shown in FIG. 25A. The combination correction value calculation unit 194 includes marker images 203P-1-1-R, 203T-1-1-L, and 203P-1-1 included in the marker images 214 (1,1) to 214 (1, n). The marker coordinate values of -S, 203P-1-2-R,..., 203P-1- (n-1) -S, 203T-1-n-L are acquired.

  Next, the composition correction value calculation unit 194 acquires the marker coordinate value of the nth marker image in the pan movement direction from the marker image group at the mth tilt position (step S176). Specifically, for example, when m = 1 and n = 1, as the n-th marker image, marker images 203P-1-1-1R and 203T-1- included in the marker image 215 (1,1) are used. A 1-L coordinate value is acquired.

  Then, the composition correction value calculation unit 194 increments n and calculates the composition correction value [α (n−1)] using the coordinate value of the common marker image between adjacent images in the pan movement direction. (Step S177). For example, when m = 1 and n = 1, the marker image 214 (1,1) R and the incremented (n = 2) marker image 214 (1,2) S are adjacent images. The marker image 203 common to the adjacent images is the marker image 203P-1-1-RU of the marker image 214 (1,1) R and the marker image 203P-1-1-1 of the marker image 214 (1,2) S. A marker image 203P-1-1-RL of the marker image 214 (1,1) R and a marker image 203P-1-1-1SL of the marker image 214 (1,2) S.

  Therefore, the composition correction value calculation unit 194 substitutes the coordinate value of the marker image 203P-1-1-RU of the marker image 214 (1,1) R into the left side of the equation (5), and the marker image 214 (1,1, 2) Substituting the coordinate value of the S marker image 203P-1-1-SU into the right side of equation (5), [α (n-1)] = [α1] is calculated. Similarly, the composition correction value calculation unit 194 substitutes the coordinate value of the marker image 203P-1-1-RL of the marker image 214 (1,1) R for the left side of the equation (5), and the marker image 214 (1 , 2) Substituting the coordinate value of the marker image 203P-1-1-SL of S into the right side of equation (5), [α (n−1)] = [α1] is calculated. The composition correction value calculation unit 194 outputs a composition correction value [α1], which is a transformation matrix in which elements are finally determined from both calculation results.

  Then, the composition correction value calculation unit 194 determines whether or not the n-th marker image 214 (m, n) exists (step S178). When the n-th marker image 214 (m, n) exists (step S178: Yes), the composition correction value calculation unit 194 re-executes step S177.

  On the other hand, when the n-th marker image 214 (m, n) does not exist (step S178: No), the composition correction value calculation unit 194 calculates the correction value matrix product [α1] [α2]. −1)] (= [α]). The correction value matrix product [α] is an affine transformation matrix from FIG. 25A to FIG. 25B.

  Thereafter, the composition correction value calculation unit 194 increments m (step S180), and determines whether or not there is a marker image group at the m-th tilt position (step S181). A marker image group at the m-th tilt position exists (step S181: Yes). The composition correction value calculation unit 194 re-executes step S175. On the other hand, when the m-th camera does not exist (step S181: No), the process proceeds to step S182. Thereby, the processing up to FIG. 25B is completed, and m in-row unified images 216-1, 216-2,..., 216-m connected in the pan movement direction are generated.

  Next, the composition correction value calculation unit 194 sets a coordinate system of a specific tilt position m (for example, m = 1) as a reference coordinate system (step S182), and resets m to m = 1.

  Then, the composition correction value calculation unit 194 increments m, and uses the coordinate value of the marker image 203 that is common between adjacent in-line unified images in FIG. 25C, and the composition correction value [β (m−1)]. Is calculated (step S183). For example, the in-row unified image 216- (m-1) and the in-row unified image 216-m are adjacent in-row unified images. The marker images 203 common to the adjacent in-row unified images are the marker images 203T- (m-1) -1-LS, 203T- (m-1) -n- of the in-row unified images 216- (m-1). LR and marker images 203T- (m-1) -1-US, 203T- (m-1) -n-UR of the in-row unified image 216-m.

  Therefore, the compositing correction value calculation unit 194 substitutes the coordinate value of the marker image 203T- (m−1) -1-LS of the in-line unified image 216- (m−1) for the left side of the equation (7). [Β (n-1)] is calculated by substituting the coordinate value of the marker image 203T- (m-1) -1-US of the unified image 216-m into the right side of Expression (7). Similarly, the composition correction value calculation unit 194 substitutes the coordinate value of the marker image 203T- (m-1) -n-LR of the in-row unified image 216- (m-1) for the left side of the equation (7), [Β (n-1)] is calculated by substituting the coordinate value of the marker image 203T- (m-1) -n-UR of the in-row unified image 216-m into the right side of Expression (7). The composition correction value calculation unit 194 outputs a composition correction value [β (n−1)], which is a transformation matrix whose elements are finally determined from both calculation results.

  Thereafter, it is determined whether or not the composition correction value [β (n−1)] can be calculated (step S184). If the composition correction value [β (n−1)] can be calculated (step S184: Yes), the composition correction value calculation unit 194 performs step S183 is re-executed. On the other hand, when the composition correction value [β (n-1)] cannot be calculated (step S184: No), since there is no more in-line unified image 216-m, the composition correction value calculation unit 194 The correction value matrix product [β1] [β2]... [Β (m−1)] (= [β]) is calculated (step S185). The correction value matrix product [β] is the affine transformation matrix in FIG. 25C.

  Then, the composition correction value calculation unit 194 combines the correction value matrix product [α] calculated in step S179 and the correction value matrix product [β] calculated in step S185, for example, a reference image of all images, for example, the marker image 215. A conversion correction value [β] [α] to a matrix unified coordinate system using the pixel coordinates of (1, 1) as a coordinate system is calculated (step S186). Thereby, the reference image unified coordinate correction value calculation process (step S68) is completed, and the process proceeds to step S71.

  Note that the flowchart corresponding to the image processing between adjacent images in the second embodiment is the processing shown in FIGS. 26A and 26B, and is the same as FIG.

  In the second embodiment, the camera 2 is sequentially moved in the pan direction and moved in the tilt direction at the time of folding. However, the camera 2 may be sequentially moved in the tilt direction and moved in the pan direction to be turned back. And tilt configuration are switched. Further, the return may be repeated by returning to the original position. Furthermore, in the image composition process, the in-row unified coordinate system is first obtained, and the matrix unified coordinate system is configured by obtaining the correction value for composition of the adjacent in-row unified coordinate system, but the column unified coordinate system is first obtained, The matrix unified coordinate system may be configured by obtaining the composition correction value of the adjacent column unified coordinate system.

<Effect of Example 2>
The second embodiment described above has the following effects (I) and (J).

(I) Image composition using a marker can be easily performed even from a remote place, and image composition in which a marker is concealed can be performed by combining marker concealment methods.

(J) The camera and the marker are alternately rotated in one direction (eg, pan direction), and the image of the marker is continuously photographed at the forefront or tail of the photographing area, and a series of photographing operations. Are repeatedly moved in the vertical direction (example: tilt direction), and a marker is given to two points that are appropriately separated for vertical image alignment. Therefore, by capturing an image included in the forefront or rearmost part of the imaging region in the vertical direction, it is possible to improve the accuracy of combining a plurality of planar images in the vertical direction.

  Example 3 will be described. In the first embodiment, the camera 2 and the marker adding unit are used in the process of shooting by moving a plurality of cameras 2 arranged in parallel to the subject or arranged in a row to the subject and combining the plurality of captured images. A multi-image shooting system 1 for shooting by moving 130 alternately is shown.

  In the third embodiment, a planar imaging device including a plurality of cameras 2 arranged in a vertical and horizontal plane and a marker providing unit arranged in a vertical and horizontal plane for image alignment of captured images of each camera 2 is used. The multi-image shooting system 1 according to the third embodiment is configured to move and capture a planar image capturing apparatus and combine a captured image with a plurality of cameras 2 arranged in a planar shape and a marker for image composition in the moving direction. Move the parts alternately to shoot. Since the third embodiment can shoot a wider shooting area at a time than the first embodiment, the shooting time can be shortened.

<Appearance of multi-image shooting system 1>
28A to 28D are explanatory diagrams illustrating an appearance example of the multi-image capturing system 1 according to the third embodiment. FIG. 28A is a plan view before movement of the planar camera unit / planar marker applying unit, FIG. 28B is a plan view after movement of the planar camera unit / planar marker providing unit, and FIG. 28C is a state of FIG. 28A. FIG. 28D is a front view of the state of FIG. 28A. Since the basic configuration is the same as that of FIG. 1 of the first embodiment, different points will be mainly described.

  On the movement processing unit 132, in the first embodiment, the camera unit 102 is configured by a plurality (three in FIG. 1) of cameras 2 arranged in a direction orthogonal to the moving direction. In the third embodiment, the camera unit 102 is configured by the planar camera unit 302 and the planar marker providing unit 333.

  The planar camera unit 302 is a plurality of cameras 2-mq arranged in a planar shape. m is a number indicating the arrangement position of the cameras 2 arranged in the direction orthogonal to the movement direction, and q is a number indicating the arrangement position of the cameras 2 arranged in the movement direction. In FIG. 28A, m = 2, q = 3, that is, a 2 × 3 planar configuration. In addition, Example 1 becomes the same structure as the case where q = 1.

  A shooting area where each camera 2-mq captures a subject is referred to as an individual shooting area 113-m-q. The cameras 2-mq are arranged in the planar camera unit 302 so that the individual photographing regions 113-mq of the adjacent cameras 2-mq overlap.

  The imaging area of the camera unit 102 is referred to as a planar parallel imaging area 313-n. As in the first embodiment, n is a number indicating a time series of movement. Since (a) shows the initial position, n = 1, and (b) shows the state moved from (a), so n = 2. The planar parallel imaging region 313-n is an imaging region in which the individual imaging regions 113-mq of the individual cameras 2-mq are combined.

  The planar marker providing unit 333 includes a plurality of marker lasers 131. The planar camera unit 302 captures the planar parallel imaging region 313-n depending on whether the markers 31 of the planar marker providing unit 333 are turned on or off collectively. The plurality of marker lasers 131 are arranged at positions where the markers 31 are added to a common area where the individual imaging areas 113-mq of the adjacent cameras 2-mq overlap.

  For example, the marker laser 131-1-2 gives the marker 31 to a common area where the individual imaging areas 113-1-1 and 113-1-2 overlap. The marker laser 131-2-1 gives the marker 31 to a common area where the individual imaging areas 113-1-1 and 113-2-1 overlap. The marker laser 131-2-2 gives the marker 31 to a common area where the individual imaging areas 113-1-1, 113-1-2, 113-2-1 and 113-2-2 overlap. The marker laser 131-2-3 gives the marker 31 to a common area where the individual imaging areas 113-1-2 and 113-2-2 overlap.

  The marker laser 131-3-1 gives the marker 31 to a common area where the individual imaging areas 113-2-1 and 113-3-1 overlap. The marker laser 131-3-2 gives the marker 31 to a common area where the individual imaging areas 113-2-1, 113-2-2, 113-3-1, and 113-3-2 overlap. The marker laser 131-3-3 gives the marker 31 to the common area where the individual imaging areas 113-2-2 and 113-3-2 overlap. The marker laser 131-4-1 gives the marker 31 to a common area where the individual imaging areas 113-3-1 and 113-3-2 overlap.

  Thus, by arranging the marker laser 131 so that the marker 31 is provided only in the common region, the number of marker lasers 131 can be reduced.

  The movement processing unit 133 includes two movement marker lasers 331-R and 331-S. The movement processing unit 133 assigns movement markers 33-S and 33-R to the forefront part in the movement direction in the planar parallel imaging region 313-n photographed by the planar camera unit 302. First, as shown in (a), the movement marker lasers 331-S, 331-S are located at the forefront in the movement direction of the planar parallel imaging region 313-1 and the movement marker 33- S, 33-R is given almost directly below.

  Thereafter, as illustrated in FIG. 28B, the planar parallel imaging region 313-n is changed from the planar parallel imaging region 313-1 to the planar parallel imaging region 313-by the movement processing unit 132 moving in the moving direction of the camera unit 102. Move to 2. Accordingly, the movement markers 33-S and 33-R are arranged at the tail end of the planar parallel imaging region 313-2. In the movement stop state of the movement of the camera unit 102, the planar camera unit 302 has a planar parallel imaging region 313-2 when the markers 31 of the planar marker imparting unit 333 are turned on collectively, and a surface when the collective light is turned off. The parallel imaging region 313-2 is imaged.

  Moreover, the camera part guidance | induction part 134 will be three units. In FIG. 28A, camera part guidance | induction parts 134-1 to 134-3 are comprised. The camera unit guiding units 134-1 to 134-3 are combined with the movement processing unit 132 to support both sides of the planar camera unit 302 / planar marker applying unit 333, and move and stop. The movement processing unit 133 supports the arms holding the movement marker lasers 331S and 331-R from both sides by the guide unit 135, and moves and stops. While the movement processing unit 132 and the movement processing unit 133 move and stop alternately, the planar camera unit 302 captures the planar parallel imaging regions 313-1 and 313-2.

  The planar laser beam 131 and the moving marker lasers 331-R and 331-S may be the same color laser. For example, a red laser is used for the planar laser beam 131, and the moving marker laser is used. It is possible to use a different color for 331 such as using a green laser. When the latter different color is used, there is a merit that it is possible to easily distinguish between a planar arrangement marker and a movement marker in the same image by using a red filter or a green filter in image processing. Note that the wavelength filter may be provided in the optical system of the photographing camera, or red, green, and blue color element separation of the photographed image may be used. Hereinafter, in this embodiment, a technique that does not stick to the color usage of the marker laser will be described.

  The functional blocks of the multi-image shooting system 1 according to the third embodiment conform to the contents shown in FIG. In addition, the functional blocks of the photographing apparatus 5 basically conform to the configuration shown in FIG. One difference is that the marker applying unit 130 in FIG. 5 is replaced with the moving marker laser 331 in FIG. 28A in the third embodiment. Another difference is the camera device 8 as shown in FIG.

<Functional Block of Camera Device 8>
FIG. 29 is a functional block diagram of the camera device 8 according to the third embodiment. In the third embodiment, the camera 2 of the camera unit 102 in FIG. 5 is replaced with a planar camera unit 302 and a planar marker providing unit 333. The planar marker imparting unit 333 receives the imaging control signal I1 including information on turning on / off the marker laser 131 arranged in a planar shape, so that the marker laser 131 is turned on or off in conjunction with imaging.

  The shooting related information I11 output by the timing control unit 182 includes shooting time, shooting number, movement or stop of the planar camera unit 102, movement or stop of the moving marker laser 331, movement of the camera unit 102 and moving marker laser 331. Of the movement of the sheet marker, the lighting or extinguishing of the planar marker providing unit 333, the lighting or extinguishing of the moving marker laser 331, the relationship between the arrangement number of the camera 2 and the image, the arrangement number of the planar marker providing unit 333 and the image Various information such as the relationship, the physical distance between the vertical and horizontal marker lasers, the measured value of the laser distance measuring unit 157, and the distance of the marker laser 331 for movement are included.

<Relationship between shooting area and marker position>
FIG. 30 is an explanatory diagram illustrating the relationship between the imaging region and the marker application position according to the third embodiment. Basically, it is the same as FIG. 8 of the first embodiment, except that the imaging area 113 is a planar parallel imaging area 313. In FIG. 30, (1) shows a state in which movement markers 33-Sn and 33-Rn are added to the tail of the n-th planar parallel imaging region 331-n in time series of movement. . The movement markers 33-Sn and 33-Rn are markers provided by the movement marker lasers 331-S and 331-R.

  In (2), the movement markers 33-Sn and 33-Rn are moved by an interval L1, and the plane parallel photographing is performed as the movement markers 33-S- (n + 1) and 33-R- (n + 1). The state given to the forefront part of field 313-n is shown.

  In (3), the planar parallel imaging region 313-n is moved by the interval L1, and the planar markers 33-S- (n + 1) and 33-R- (n + 1) for movement are moved after the planar parallel imaging region 313-n is moved. The state arranged at the end of (n + 1) is shown.

  (4) shows a state in which, in the state of (3), the same operations as the operations of (1) to (2) are executed to photograph the planar parallel photographing region 313- (n + 1). Further, the interval L1 between the markers 3 is an equation in which the width X of the imaging region 113 for one camera 2 in FIG. 7 is replaced with the width of the planar parallel imaging region 313, and L0 in equation (1) is replaced with L1. It is expressed by

<Timing chart of photographing apparatus 5>
FIG. 31 is a timing chart of the photographing apparatus 5 according to the third embodiment. In FIG. 31, (1) indicates the movement or stop of the camera unit 102 (the planar camera unit 302 and the planar marker providing unit 333). (2) indicates the movement or stop of the movement marker lasers 331-S, 331-R by movement or stop control of the movement processing unit 133. (3) indicates whether the marker 31 is turned on or off. (4) indicates whether the moving marker 33 is turned on or off. (5) shows the shooting timing of a still image. (6) shows the transmission timing of the image / shooting related information (image information I10 and shooting related information I11). (7) indicates the image ID of the planar photographed image 314. The image ID is based on the movement or stop state of (1) the movement processing unit 132 that moves the planar camera unit 302 and (2) the movement processing unit 133 that moves the movement marker lasers 331-S and 331-R. , Uniquely determined.

  The imaging device 5 turns on the planar marker 31 and the moving marker 33 at the initial position to check the initial position and operation, and then displays the planar captured image 314 with the marker 31 and the moving marker 33 turned off. Take a picture. The planar photographed image 314 photographed at this time is a planar photographed image 314-1-O that does not include a marker image.

  The photographing device 5 transmits image information I10 and photographing related information I11 as shown in (6) every time a still image is photographed. Next, the imaging device 5 turns on the planar marker 31 and the moving marker 33 to obtain a planar captured image 314-1-F. Next, the imaging device 5 moves the planar camera unit 302 by a predetermined interval L1 and stops it, turns off the planar marker 31 and shoots to obtain a planar captured image 314-2-B.

  Next, the photographing device 5 obtains the planar photographed image 314-2F by moving the movement processing unit 133 while moving the predetermined distance L1 while stopping the movement marker 33 and stopping and photographing at the stop position. To do. Thereafter, the planar camera unit 302 is sequentially moved and stopped at a predetermined interval L1, the planar marker 31 is turned off, photographing, the moving marker 33 is moved and stopped at a predetermined interval L1, and the planar marker 31 is turned on and photographed. By repeating the above, images after the planar photographed image 314-2-F are obtained.

  The functional blocks for performing image processing in the third embodiment substantially conform to the functional blocks of the image processing apparatus 9 shown in FIG. 11 of the first embodiment, but differences from the first embodiment will be described below. Specifically, the information acquisition unit 190 acquires the state information (1) to (4) as the photographing related information I11 related to the identifier of the photographed image group 214 (6) shown in the timing chart of FIG. . The shooting-related information I11 in the third embodiment includes shooting time, shooting number, movement / stop of the planar camera unit 102, movement / stop of the moving marker laser 331, movement of the camera unit 102, and movement marker laser 331 associated with shooting. Pre- and post-movement relations, lighting / extinguishing of the planar marker attaching unit 333, lighting / extinguishing of the moving marker laser 331, relationship of the arrangement number image information of the photographing camera, relationship between the arrangement number of the planar marker providing unit 333 and the image It becomes.

  The information acquisition unit 190 refers to the imaging related information I11 to obtain a matrix element (m, q) from the corresponding IDs in the vertical and horizontal directions of the planar camera 2-mq. The information acquisition unit 190 creates and holds a marker ID corresponding to the vertical and horizontal alignments of the planar marker 31 with reference to the imaging related information I11.

  The information acquisition unit 190 refers to the imaging related information I11 and assigns the identifier “F” to the captured image immediately after the movement of the moving marker laser 331 to the ID corresponding to the imaging position of the planar camera unit 102. The information acquisition unit 190 refers to the shooting related information I11 and assigns an identifier “B” to an image shot immediately after the movement of the planar camera unit 302 is stopped. The information acquisition unit 190 refers to the imaging related information I11 and assigns an identifier “O” to an image in which all markers are turned off. The information acquisition unit 190 refers to the shooting-related information I11 and creates an image group (314) that is a group of image matrices (m, q) shot by the planar camera unit 302 according to the movement order of the cameras 2-m-q. A movement ID is assigned.

  The marker route information generation unit 193 generates marker route information I29 that is linked to the marker addition related information and the position in the image of the marker image, and is linked to the image capturing order. The marker addition related information includes information such as the movement / stop of the movement marker laser 331, the front-rear relationship between the movement of the camera unit 102 and the movement of the movement marker laser 331, and the turning on / off of the movement marker laser 331. Specifically, for example, the marker route information generation unit 193 extracts the marker coordinate information I19 from the marker coordinate value extraction unit 191, and specifies the captured image 114 to which the marker image of the moving marker laser 331 belongs. Then, the marker route information generation unit 193 displays “S” when the marker image exists in the left half region in the captured image 114, and displays “R” when the marker image exists in the right half region. Set to the marker ID of the marker image in (b).

  Further, regarding the marker image 103 of the planar marker 31 included in the imaging region 113 of the camera unit 102, the traveling direction is determined based on the relationship between the image matrix (m, q) of the captured image 114 and the position in the image of the marker image 103. If it is the front side, “− (n + 1) −F”, if it is the rear side in the traveling direction, “−n−B”, and the position in the image of the marker image 103 is m in the image matrix (m, q). If the direction is to the left, “−m− (n + 1) −FS” is assigned to the image ID, and if the direction is to the right, “− (m + 1) − (n + 1) −FR” is assigned to the image ID.

  The correction value calculation unit 197 calculates the correction value for composition of the image selected by the composite image selection unit 196, as in the first embodiment. First, the correction value calculation unit 197 sets a reference image for composition correction value. For example, the correction value calculation unit 197 sets the center image of the selected image group as the reference image. An optimum route from the reference image to each image to be image-combined is set using the marker route information I29. Specifically, for example, the correction value calculation unit 197 calculates a matrix error between adjacent images on the route from the reference image to the target image, thereby calculating the quantization error inherent in the coordinate value of the marker image. In addition to accumulating, the lens aberration is acquired from the shooting environment information I50, so that the lens distortion and the matrix product based on the lens aberration are calculated.

  The image that is the target of image synthesis that is the end point of the route is the remaining image other than the reference image in the selected image group, but the correction value calculation unit 197 is the outermost image group of the selected image group. May be selected as an image to be synthesized as an end point of the route.

  Further, the correction value calculation unit 197 may select the optimum route in consideration of quantization error and lens distortion. Since the quantization error depends on the marker interval and the number of pixels, the correction value calculation unit 197 calculates the quantization error from the marker interval and the pixel density as the uncertainty of the coordinate value of the marker image for each link constituting the route. Turn into. Since the lens distortion depends on the position in the image, the correction value calculation unit 197 sets the lens distortion as a degree of uncertainty of the coordinate value of the marker image from the camera lens aberration measured in advance for each link constituting the route. Turn into. Then, the correction value calculation unit 197 calculates the cumulative uncertainty for each route by summing the uncertainty of each link, and selects the route with the smallest cumulative uncertainty as the optimum route. When the number of images is large and the number of route options increases, the influence of cumulative error or cumulative distortion for each route appears, so that an optimal route that does not depend only on the number of images on the route is selected.

  Finally, the correction value calculation unit 197 performs a coordinate transformation matrix product for matching the coordinate system of the target image with the coordinate system of the reference image along the optimal route that has been set (see the above equation (9)). Is calculated as a composition correction value for image composition.

  Also, the combining unit 198 selects and overwrites the original image without the marker image at the marker image position represented in the same unified coordinate system from the time series of the image on the marker route and the relationship of the marker arrangement. By doing so, the marker image is concealed. Thereby, the composition unit 198 outputs image output information in which the marker image is concealed.

<Shooting area and photographed image group on subject>
FIG. 32A and FIG. 32B are explanatory diagrams illustrating a shooting area and a shot image group on a subject according to the third embodiment. FIG. 32A shows the marker 31 and the moving marker 33 in the planar parallel imaging regions 313-1 and 313-2 before and after the movement. The planar parallel imaging region 313-1 includes imaging regions 113-1 (1, 1) to 113-1 (3, 2). The planar parallel imaging region 313-2 includes imaging regions 113-2 (1,1) to 113-2 (3,2).

  FIG. 32B shows a photographed image 114-1 (1, 1) taken along the timing chart of FIG. 31 while the planar camera unit 302 / planar marker applying unit 333 and the moving marker laser 331 are alternately moved and stopped. ) Planar captured images 314-1-O to 314-2-F including O to 114-2 (3,2) F are shown.

  The planar photographed image 314-1-O is an image photographed with the marker 31 turned off at the initial position. The planar captured image 314-1-F is an image captured by lighting the marker 31 at the initial position. The planar photographed image 314-1-F is a marker image in which the movement markers 33-1-S and 33-1-R are lit by the movement marker laser 331 at the forefront of the planar photographed image 314-1-F. It is the image which produced | generated 303-1-FS and 303-1-FR (it displays with the square for convenience in FIG. 32B).

  The planar captured image 314-2-B is captured by the planar parallel imaging region 313-n from the planar parallel imaging region 313-1 by the movement of the planar camera unit 302 / planar marker providing unit 333. It is the image which image | photographed the planar parallel imaging | photography area | region 313-2 when transfering to the area | region 313-2. In the planar photographed image 314-2-B, the markers 31 are turned off, and the movement markers 33-1-S and 33-1R are moved by the movement marker laser 331 at the end of the planar parallel photographing region 313-2. Illuminated to generate marker images 303-1-BS and 303-1-BR.

  The planar photographed image 314-2-F turns on the marker 31 and moves the moving marker laser 331 to the forefront part of the planar parallel photographing region 313-2, thereby moving markers 33-2-S and 33. This is an image in which the marker images 303-2-FS and 303-2-FR are generated by turning on -2-R.

<Operation Processing Procedure of Imaging Device 5>
FIG. 33 is a flowchart illustrating an example of an operation processing procedure of the imaging apparatus 5 according to the third embodiment. FIG. 33 is a processing flow along the flow of the timing chart of FIG. First, the imaging device 5 initializes the imaging device 5 (step S301). Next, the imaging device 5 installs the planar camera unit 302 / planar marker providing unit 333 at the initial position (step S302). And the imaging device 5 image | photographs in the state which turned off the marker lasers 131 and 331, and acquires the planar picked-up image 314-1-O (step S303).

  Next, the imaging device 5 turns on the movement marker laser 331 to move the movement marker 33 and stops it at a position where the movement marker 33 is arranged in the forefront of the planar parallel imaging region 313-n (step). S304). And the imaging device 5 lights the marker laser 131 in the same position, and image | photographs the planar image 314-1-F (step S305).

  Next, the imaging device 5 determines whether to move the planar camera unit 302 and the planar marker providing unit 333 (step S306). Specifically, for example, the imaging device 5 determines whether the current position of the multi-image imaging system 1 is within a preset imaging range based on the imaging related information I11, and the planar camera unit 302 and the plane. The right or wrong of the movement of the marker-like marking unit 333 is determined. That is, the imaging device 5 determines that the planar camera unit 302 and the planar marker providing unit 333 should be moved if the current position of the multi-image imaging system 1 is within the imaging range (step S306: Yes), and imaging. If it is out of the range, it is determined that it should not move (step S306: No). Step S306: If Yes, the process moves to step S307.

  Next, the imaging device 5 moves the planar camera unit 302 / planar marker applying unit 333 while holding the position of the movement marker 33, so that the movement marker 33 is in the planar parallel imaging region 313-n. It stops at the position arranged at the end (step S307). Then, the imaging device 5 turns off all the marker lasers 131 and captures the planar captured image 314- (n + 1) -B at a position where the moving marker 33 is arranged at the tail of the planar parallel imaging region 313-n. (Step S308). n is the number of movements of the planar camera unit 302 / planar marker providing unit 333.

  Next, the imaging device 5 moves and stops the movement processing unit 133 so that the movement marker 33 is arranged at the forefront of the planar parallel imaging region 313 (step S309). Then, the imaging device 5 turns off all the marker lasers 131, images the planar parallel imaging region 313-n in which the moving marker 33 is disposed in the forefront of the planar parallel imaging region 313, and the planar captured image 314. -(N + 1) -F is acquired (step S310). Then, the process returns to step 306. Step S306: In the case of No, the operation of the imaging device 5 ends.

<Reference image-based image processing example>
FIG. 34A to FIG. 34D are explanatory diagrams illustrating a reference image-based image processing example according to the third embodiment. FIG. 34A shows planar unified coordinate system marker images 319-1 and 319-2 converted into a unified coordinate system for the reference photographed image 114 in the planar photographed image 314. The planar unified coordinate system marker images 319-1 and 319-2 are binarized images. The planar unified coordinate system marker images 319-1 and 319-2 are obtained by replacing the captured images 114-1 and 114-2 in the planar captured images 314 with the marker image 103-1 in the planar captured image 314 of FIG. 32B. The flow shown in FIG. 12 of Example 1 or each marker image using -1-2 to 103-1-4-2 and marker images 103-1-1-2 to 103-1-4-2. If the interval size information 103 is known, the image is processed in the flow shown in FIG.

  In the present embodiment, a description will be given of a mode in which the binarized marker image centroid coordinates are used as reference coordinates for image alignment. However, the image alignment marker coordinates are limited to the binarized marker image centroid coordinates. Rather than the binarized marker image, for example, the coordinate of the marker image that is uniquely determined by the marker image, such as the center coordinate with the distributed load applied, or the quarter position on the X coordinate axis or the y coordinate axis. As good as When the brightness distribution of the marker image is stable regardless of the shooting position, the brightness peak coordinates of the marker image may be simply used as the reference coordinates for image alignment. In this case, there is an advantage that the marker coordinate value extraction process is reduced. As described above, the coordinates of the marker image mean the coordinates of the position specified by the uniquely determined definition related to the assigned marker.

  Further, the planar unified coordinate system marker images 319-1 and 319-2 are marker images 303-1-FS and 303-, which are captured images of the common movement markers 33-1-S and 33-1-R, respectively. 1-FR, 303-1-BS, 303-1-BR.

  FIG. 34B shows the movement marker images 303-1-FS, 303-1-FR, 303, which are necessary for unifying the coordinate systems of the planar unified coordinate system marker images 319-1, 319-2. It is the figure which displayed only -1-BS and 303-1-BR in an easy-to-understand manner. By substituting the pixel coordinate value of the common marker image for movement in the planar unified coordinate system marker images 319-1 and 319-2 into the equation (3), four [α] in the equation (3) are obtained. An unknown number can be obtained.

  Assuming that the obtained affine transformation matrix [α] is [α1], the coordinates of the planar unified coordinate system marker image 319-2 are unified in the coordinate system of the planar unified coordinate system marker image 319-1, and the equation (4) Similarly, it is possible to convert between adjacent images. Similarly, if there are a plurality of planar uniform coordinate system marker images, the matrix product [α1] [α2]... [Α (n−1)] in the movement direction is converted into the standard planar uniform coordinate system. Can be expressed as shown in Equation (6). The coordinates of an arbitrary marker image are multiplied by an affine transformation coefficient matrix to obtain a reference plane unified coordinate system marker image, for example, a marker image of the entire unified coordinate system in the coordinate system of the plane unified coordinate system marker image 319-1. 320.

  FIG. 34C shows a marker image 320 of the entire unified coordinate system when n = 2 as Example 3. By repeating the above affine transformation coefficient matrix product n-1 times for a general planar imaging area number n, the nth reference planar uniform coordinate system can be converted into an overall unified coordinate system.

  FIG. 34D shows the synthesis of the whole image using the conversion correction value to the whole unified coordinate system calculated by the processing up to FIG. 34C. First, the composition unit 198 performs image composition within the planar photographed image 314. The synthesizing unit 198 uses the planar photographed images 314-1-O and 314-2-B without the reflection of the planar marker 31 for image synthesis. The synthesizing unit 198 synthesizes the planar captured image in which the marker image is concealed by arranging the captured images 114 by applying the coordinates of the image in the overall unified coordinate system.

  Next, the synthesizing unit 198 performs bonding between the planar shot images 314. By overwriting the composite image of the planar photographed image 314-1-O whose photographing time is before the planar photographed image 314-2-B, the movement marker images 303-1-BS and 303 are displayed. An overall image 321 for output in which -1-BR is concealed is obtained. In the third embodiment, the case where the number n of the planar photographed images is two is shown. However, in general, the photographing time is late using the photographing time corresponding to the ID of the n planar photographed images 314, and the movement is performed. The marker image 303 of the moving marker 33 of the same moving marker 33 is overwritten with the composite image of the planar captured image 314 with the marker image 303 of the moving marker 33 at the end and the same marker image 303 of the moving marker 33 is overwritten at the end. The image information is superimposed and finally combined so that the image 314-1-0 without the initial marker image is overwritten to obtain an overall image 321 for output.

  Further, in the third embodiment, the case where the planar parallel imaging region 313 moves in one direction and the number of the planar imaging images 314 in the imaging region in the moving direction is two, and the case where it is enlarged to n in general are shown. However, as in the second embodiment, there may be enlargement in which moving imaging is performed m times in a direction orthogonal to the moving direction.

<Correction value calculation process of entire unified coordinate system (step S57)>
FIG. 35 is a flowchart of a detailed process procedure example of the correction value calculation process (step S57) of the overall unified coordinate system shown in FIG. 16 according to the third embodiment. In the initial state, m = 1 and n = 1. The composition correction value calculation unit 194 acquires the coordinate value of the marker image 103 of the first planar photographed image group (step S375). Next, the composition correction value calculation unit 194 performs the same processing as that in FIG. 17A of the first embodiment on the captured image 114 in the direction orthogonal to the moving direction, and unifies the planar image of the planar captured image 314. A conversion value to the coordinate system is obtained (step S376).

  Next, the composition correction value calculation unit 194 determines the presence or absence of the next planar photographed image 314 (step S377). When there is the next planar photographed image 314 (step S377: Yes), the composition correction value calculation unit 194 acquires the marker coordinates of the next planar photographed image 314 (step S378) and returns to step 376. On the other hand, when there is no next planar image 314 (step S377: No), the process proceeds to step S379.

  Next, the compositing correction value calculation unit 194 uses the uniform surface coordinates as a reference for the unified coordinate of the entire photographing area from the unified coordinate system group to the reference photographed image 114 of the plurality of uniform planar coordinate systems. A system is selected (step S380). For example, the coordinate system of the planar photographed image 314-1 with n = 1 is selected as the standard planar coordinate system. Then, the composition correction value calculation unit 194 sets the n = 1st planar unified coordinate system selected in step S380 as a processing target and increments n. Then, the composition correction value calculation unit 194 calculates the marker image 303 of the common movement marker 33 in the (n−1) th planar unified coordinate system after increment and the nth planar unified coordinate system. A coordinate conversion matrix [α (n)] of the composite correction value is calculated from the coordinate values and held (step S381). Here, n is the number of movements.

  Next, the composition correction value calculation unit 194 determines whether or not there is a next planar image 314 (step S382). If there is a next planar photographed image 314 (step S382: Yes), the process returns to step S381 and is executed again. On the other hand, when there is no next planar photographed image 314 (step S382: No), the composition correction value calculation unit 194 multiplies the correction values between the uniform planar coordinate systems for the entire photographing region image to unify the whole. Correction value matrix products [α1] [α2]... [Α (n−1)] that are conversion correction values for the coordinate system are calculated (step S383). Then, the process returns to step S57 in FIG.

<Effect of Example 3>
The third embodiment described above has the following effects (K) to (N).

(K) With a plurality of cameras arranged in a plane and a planar camera unit 302 that assigns a marker to a common region of adjacent shooting regions in the shooting region, a wider shooting range than the camera unit 102 of the first embodiment can be collectively shot. Therefore, the shooting time can be shortened.

(L) The number of marker lasers can be reduced by providing a marker in a common area of adjacent photographing areas by the planar camera unit 302 and not providing a marker in a non-common area.

(M) The movement processing unit 132 that supports the planar camera unit 302 that constitutes the camera unit 102 and the movement processing unit 133 that supports the moving marker laser 331 are alternately moved to capture images between the planar captured images. Therefore, it is possible to easily realize the image alignment between the planar parallel imaging regions with high accuracy at the pixel level.

<Modification>
The modification of the present Examples 1-3 is given.
(A) In all the embodiments, an image processing apparatus that acquires an image photographed by a photographing apparatus having a control unit for alternately moving a camera unit and a marker providing unit and photographing related information and performs image processing has been described. However, the image processing apparatus can perform the same image processing as in the embodiment even if an image is taken by manually moving the alternate moving mechanism of the camera unit and the marker providing unit and using the image and the shooting-related information provided manually. Is possible.

(B) In the first embodiment, information relating to the dimension that defines the position of the marker 3, the interval dimension of the moving marker, and the moving distance is acquired in a form in which a camera or a marker applying unit is installed in parallel with the subject and moved in parallel. The technique used for calculating the correction value of the image composition process has been described. Even when a camera or a marker attaching unit is mounted on a pan head installed at a point away from the subject as in the second embodiment, and the pan and head are rotated to move the imaging region and the marker attaching position, the distance to the subject It is also possible to derive the dimension of the marker interval from this distance information and the pan head angle information by laser distance measurement, and synthesize the image by applying it to the dimension information of the marker position using this.

(C) In the first and third embodiments, a plurality of cameras and a plurality of marker applying units are arranged in a line in a line or vertically and horizontally in a plane with a parallel relationship to the subject, and wide imaging in width or plane. The configuration for shooting the area at once is described. Even when a camera or a marker attaching unit is mounted on a pan head installed at a point away from the subject as in the second embodiment, and the pan and head are rotated to move the imaging region and the marker attaching position, a plurality of cameras The configuration of the marker applying unit may be arranged linearly on the camera platform in a radial pattern or in a planar pattern in a radial pattern so that a wide area on the subject can be imaged collectively. It is also possible to provide a moving marker in the angle moving direction and use the moving marker to efficiently synthesize an image by combining such image joining and combining.

(D) In all the embodiments, regarding the movement marker concealment, the marker image is concealed by overwriting the marker image of the image with the earlier photographing time or with the smaller photographing number or the photographing time later or with the larger photographing number. However, the same effect can be obtained by overwriting the marker image of the image with the earlier shooting time or with the smaller shooting number with the marker image with the later shooting time or with the larger shooting number. Instead of photographing an image without a marker image at the first photographing position, it is also possible to compose a marker concealed image by photographing an image without a marker image at the last photographing position to obtain a final composite image.

(E) In all the embodiments, the technique of hiding the marker using the captured image itself has been described. However, when the first captured image and the second captured image include a marker image for the same captured area, By applying a procedure for combining the marker images of the first captured image and the second captured image by combining the portions without the marker images, an image without the marker image is formed, and this is used for image synthesis. It is also possible to apply.

  As described above, according to the present embodiment, a high-intensity marker for image synthesis can be stably provided, and a detailed image of a wide range of subjects can be obtained as an image of the subject itself without any marker reflection. it can. In addition, it is possible to increase the accuracy of image composition by providing a marker with higher brightness than the surroundings to the subject.

  The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the scope of the appended claims. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and the present invention is not necessarily limited to those having all the configurations described. A part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Moreover, you may add the structure of another Example to the structure of a certain Example. Moreover, you may add, delete, or replace another structure about a part of structure of each Example.

  In addition, each of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them, for example, with an integrated circuit, and the processor realizes each function. It may be realized by software by interpreting and executing the program to be executed.

  Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, and an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, and a DVD.

  Further, the control lines and the information lines are those that are considered necessary for the explanation, and not all the control lines and the information lines that are necessary for the mounting are shown. In practice, it can be considered that almost all the components are connected to each other.

DESCRIPTION OF SYMBOLS 1 Multi-image imaging system 2 Camera 5 Imaging device 6 Marker providing device 7 Control unit 8 Camera device 9 Image processing device 11 Image composition processing unit

Claims (15)

A multi-image photographing system having a photographing device and an image processing device,
The imaging device
A camera device having at least one camera for photographing a subject in a photographing region;
A marker imparting device that imparts at least one marker having a brightness higher than the brightness of the subject to a common area where adjacent shooting areas overlap;
The camera device and the marker applying device are controlled to move independently and intermittently in a predetermined movement direction, and the marker is applied by the marker applying device and the image is taken by the camera device. A control unit for controlling the timing of
The controller is
A first control process for causing the marker applying device to move and stop so that the marker is arranged in the forefront portion of the moving direction in the shooting region, and shooting the shooting region with the camera device;
The camera device is moved from the shooting position of the shooting area so that the next shooting area that is adjacent to the moving direction of the shooting area and includes the marker in a common area with the shooting area can be shot. And moving in the direction to stop, and alternately executing a second control process for photographing the next photographing area by the camera device,
The image processing apparatus includes:
An extraction unit that extracts a coordinate value of the marker image from each captured image based on a set of pixels constituting a marker image that is an image of the marker in each captured image of the captured image group captured by the imaging device;
A first photographed image obtained by photographing a first photographed area in the photographed image group, and a second photographed image obtained by photographing a second photographed area as the next photographed area with respect to the first photographed area. The first marker extracted from the first photographed image by the extraction unit with respect to the same marker given to the common area of the first photographing area and the second photographing area for each combination Based on the coordinate value of the image and the coordinate value of the second marker image extracted from the second photographed image, a correction value for combining the first photographed image and the second photographed image is obtained. A calculation unit for calculating,
Using the correction value calculated by the calculation unit, an image obtained by removing the image of the common area from the first photographed image, an image obtained by removing the image of the common area from the second photographed image, A synthesizing unit that synthesizes the remaining image where the marker image does not exist and is the imaging region of one of the first captured image and the second captured image;
A multi-image shooting system comprising: A camera device having at least one camera for photographing a subject in a photographing region;
A marker imparting device that imparts at least one marker having a brightness higher than the brightness of the subject to a common area where adjacent shooting areas overlap;
The camera device and the marker applying device are controlled to move independently and intermittently in a predetermined movement direction, and the marker is applied by the marker applying device and the image is taken by the camera device. A control unit for controlling the timing of
The controller is
A first control process for causing the marker applying device to move and stop so that the marker is arranged in the forefront portion of the moving direction in the shooting region, and shooting the shooting region with the camera device;
The camera device is moved from the shooting position of the shooting area so that the next shooting area that is adjacent to the moving direction of the shooting area and includes the marker in a common area with the shooting area can be shot. And a second control process for alternately moving and stopping in the direction and causing the camera device to photograph the next imaging region. In the imaging device according to claim 2,
In the same shooting area, the interval between the position of the first marker applied to the rearmost position and the position of the second marker applied to the frontmost portion is the moving direction in the imaging area. An imaging apparatus, wherein the interval is obtained by removing an allowable width of an arrangement error of the first marker, an allowable width of an arrangement error of the second marker, and the width of the marker from a width length. In the imaging device according to claim 2 or 3,
The camera device is fixed on the subject so as to be in a positional relationship for photographing in a direction perpendicular to the subject,
The camera apparatus and the marker applying apparatus have a structure that moves parallel to the surface of the subject. In the imaging device according to claim 2 or 3,
The camera device includes a first camera platform that can rotate the camera in at least a first rotation direction;
The marker imparting device includes a marker imparting unit that imparts the marker and that is rotatable in the first rotation direction, and a first marker imparting platform that mounts the marker imparting unit,
The controller is
A third control process for rotating and stopping the first marker application platform so that the marker is applied to the forefront part of the imaging region of the camera device, and causing the camera device to perform imaging,
And alternately executing a fourth control process in which the first camera pan head is rotated and stopped so that the marker is arranged at the tail end of the next imaging region, and the camera device performs imaging. An imaging device characterized by the above. In the imaging device according to claim 5,
A second camera head that is rotatable in a second rotation direction that is 90 degrees different from the first rotation direction together with the first camera head;
A second marker-carrying pan that is rotatable in the second rotation direction together with the first marker-carrying pan;
At least one other marker providing unit different from the marker applying unit;
A third marker attachment platform that can rotate the other marker application portion in the second rotation direction;
The controller is
In some imaging areas of the series of imaging areas in the first rotation direction imaged by the camera device, another marker is provided by the other marker applying unit at the forefront part in the second rotation direction of the imaging area. A third control process for rotating and stopping the third marker attaching pan head to be assigned, and causing the camera device to shoot,
Rotating and stopping the second marker-attached pan head in the second rotation direction so that the second marker is arranged at the final tail in the second rotation direction of the next imaging region, And a fifth control process for causing the camera device to shoot alternately. A planar camera unit in which a plurality of cameras that shoot subjects in each imaging area are arranged in a plane;
A plurality of marker assigning units for assigning at least one marker to a common region where adjacent photographing regions overlap in the photographing region group of the plurality of cameras;
A control unit that controls the timing of marker application by the plurality of marker applying units and the timing of imaging by the plurality of cameras;
The controller is
When the markers are assigned by the plurality of marker assigning units, the imaging region group is imaged by the plurality of cameras, and when the marker is not assigned, the imaging region group is imaged by the plurality of cameras. An imaging apparatus characterized by causing the image to be captured. In the imaging device according to claim 7,
The plurality of marker providing units are:
Among the plurality of cameras arranged in the direction orthogonal to the moving direction, among the common shooting area where the shooting areas of adjacent cameras overlap, and the common shooting area where the shooting areas overlap in the moving direction, 3 An imaging apparatus, wherein the marker is added to a common area where two or more adjacent imaging areas overlap. In the imaging device according to claim 7,
The plurality of marker providing units are:
Among the plurality of cameras arranged in a direction orthogonal to the moving direction, a common photographing region where the photographing regions of adjacent cameras overlap each other, and the photographing region group of the plurality of cameras in which the plurality of cameras are arranged in a plane. The marker is attached to a common area where three or more adjacent imaging areas overlap among the common areas where adjacent imaging areas overlap. An image processing device that cooperates with a photographing device,
The imaging device
A camera device including a plurality of cameras arranged in a row direction and a column direction orthogonal to the row direction;
A marker imparting device including a plurality of marker imparting units each imparting a marker to a subject;
The plurality of cameras and the plurality of marker imparting units are moved and stopped in the row direction, and at least a common region overlapping between a photographing region adjacent to the column direction and a photographing region adjacent to the row direction, By capturing the imaging region group to which the marker is added by the plurality of marker applying units with the plurality of cameras, the captured image group for the imaging region group to which the marker is applied is stored,
The image processing apparatus includes:
An acquisition unit for acquiring the captured image group;
An extraction unit that extracts a coordinate value of a position uniquely specified by a common definition of a marker image in each captured image of the captured image group acquired by the acquisition unit, as a coordinate value of the marker image;
Based on the coordinate value of the first marker image and the coordinate value of the second marker image respectively included in the captured image of the adjacent imaging region for the same marker extracted by the extraction unit, the adjacent imaging region A first calculation unit that calculates a first correction value for sharing a coordinate system of captured images for each captured image in the adjacent imaging region;
A reference image in an in-line photographed image group belonging to the same row in the photographed image group is selected for each row, and a first correction for each photographed image in the adjacent photographing region calculated by the first calculating unit. By using the values, the coordinate system of the in-line captured image group is unified with the pixel coordinate system of the reference image, so that the in-line captured image group is synthesized for each row, and the in-line captured image adjacent in the column direction is combined. Select a marker image of two markers that are common between groups, and sequentially calculate a second correction value using the in-row unified coordinate value of the selected marker image for each in-row captured image group adjacent in the column direction. A second calculation unit that
A third in-line coordinate system is selected from the plurality of in-line captured image groups, and a third coordinate system of the plurality of in-line captured image groups is unified using the second correction value. A third calculation unit for calculating a correction value;
A synthesizing unit that synthesizes the plurality of in-line captured image groups using the third correction value calculated by the third calculating unit;
An image processing apparatus comprising: The image processing apparatus according to claim 10,
A selection unit for selecting a display target region from the synthesized image synthesized by the synthesis unit;
Among the composite images, any one of the captured images included in the display target region selected by the selection unit is set as a reference image, and each of the captured images in the adjacent shooting region extracted for the same marker is extracted by the extraction unit. Based on the combination of the coordinate value of the first marker image and the coordinate value of the second marker image included, and the time series of shooting of the shot image, for each shot image of the adjacent shooting area, A generation unit that generates marker route information indicating a joining order for connecting the first marker image and the second marker image;
Based on the marker route information generated by the generation unit, a route group from the reference image to another captured image in the display target area is set, and the number of captured images constituting the route of the route group is minimized. A specific route to be selected from the group of routes, and a coordinate system of another captured image constituting the specific route is converted into a coordinate system of the reference image along the specific route. A fourth calculation unit that calculates a correction value of
The synthesis unit is
The image processing apparatus, wherein the display target area is synthesized using the fourth correction value calculated by the fourth calculation unit. An image processing device that cooperates with a photographing device,
The imaging device
A camera device including a plurality of cameras arranged in a row direction and a column direction orthogonal to the row direction;
A marker applying device including a plurality of marker applying units for applying a marker to a subject and a measuring unit for measuring a value related to the interval or position between the marker applying points;
The plurality of cameras and the plurality of marker imparting units are moved and stopped in the row direction, and at least a common region overlapping between a photographing region adjacent to the column direction and a photographing region adjacent to the row direction, A control unit that causes the plurality of cameras to shoot the imaging region group to which the marker is added by the plurality of marker applying units;
A marker position coordinate relationship information transmission unit for transmitting a value related to the interval or position of the marker by the marker applying device;
The image processing apparatus includes:
An acquisition unit that acquires the imaging region group and the marker position coordinate relationship information captured by the imaging device;
An extraction unit that extracts a coordinate value of the marker image from each captured image based on a pixel set that constitutes a marker image that is an image of the marker in each captured image of the captured image group acquired by the acquisition unit;
The coordinate value of the marker image corresponding to the marker position coordinate value of the marker derived from the marker position coordinate value of the marker or marker interval information included in the marker position coordinate relationship information along the time series of shooting of the captured image And calculating a correction value for converting the coordinate value of the marker image into the marker position coordinate value;
A combining unit that combines the plurality of captured image groups using the correction value calculated by the calculating unit;
An image processing apparatus comprising: In the image processing device according to any one of claims 10 to 12,
The synthesis unit is
When there is a first photographed image in which the marker image of the marker exists and a second photographed image in which the marker image does not exist for the same photographing region, photographing of the photographing regions adjacent to each other in the second photographed image An image processing apparatus characterized by combining images. In the image processing device according to any one of claims 10 to 12,
The synthesis unit is
In each photographed image group of the plurality of photographed image groups, a first photographed image in which the marker image of the marker exists on the rear side in the row direction and a second photographed image in which the marker image does not exist in the same photographing region. The second photographed image is adopted, and the first photographed image and the third photographed image in which the marker image of the marker exists on the front side in the row direction for the same photographing region, If there is, the third photographed image is adopted, and using the correction value, the adopted photographed image group is overlapped with the photographed image having the slower photographing order on the photographed image having the earlier photographing order. An image processing apparatus characterized by combining the images as described above. In the image processing device according to any one of claims 10 to 12,
The synthesis unit is
In each captured image group of the plurality of captured image groups, a first captured image in which the marker image of the marker exists on the front side in the row direction and a second captured image in which the marker image does not exist in the same captured area. When the second captured image is used, the first captured image and the third captured image in which the marker image of the marker exists on the rear side in the row direction for the same captured area If there is, the third photographed image is adopted, and using the correction value, the adopted photographed image group overlaps the photographed image with the earlier photographing order with the photographed image with the later photographing order. An image processing apparatus characterized by combining the images as described above.