JP2021069032A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a photographing device suitable for a narrow portion. An imaging device 100 generates a side view image of a narrow portion. The camera 110 shoots while moving the narrow portion in the depth direction. The image processing unit 120 has a trained model 122. The trained model is derived from the correlation between the image of the gaze part in the camera image when the gaze part and the camera are relatively far from each other and the image of the gaze part in the camera image when the gaze part and the camera are relatively close to each other. It will be taken. During the actual operation, the image processing unit generates a front view image based on the camera images sequentially output from the camera. [Selection diagram] Fig. 2

Description

The present invention relates to a photographing device suitable for a narrow portion.

In narrow areas such as tunnels, narrow alleys, indoors, structures with narrow recesses, and spaces sandwiched between two narrow walls, you may want to photograph the surroundings and internal structure (hereinafter referred to as the subject). is there. In a narrow area, it is not possible to keep a sufficient distance between the camera and the subject, so if the camera and the subject are facing each other, even if a wide-angle lens is used, a part of the subject will not be captured by the camera. Situations can occur.

In Patent Document 8, while moving the camera along the subject, the subject is photographed from an oblique direction nearby, a portion including the subject is cropped from the photographed image, converted into an image photographed from the front, and the converted image is obtained. Disclosed is a technique for synthesizing images by connecting them in order according to the scanning direction. According to this method, the field of view can be widened as compared with the case where the camera faces the subject.

Japanese Unexamined Patent Publication No. 2004-31549 International Publication WO08 / 102898 Japanese Unexamined Patent Publication No. 2007-323616 Japanese Unexamined Patent Publication No. 2010-073035 Japanese Unexamined Patent Publication No. 2012-022652 Japanese Unexamined Patent Publication No. 2012-109737 Japanese Unexamined Patent Publication No. 2013-250891 Japanese Unexamined Patent Publication No. 2011-126998

R. Timofte, V.D. Smet, and L. V. Gool., "A +: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution. ACCV, 2014" C. Dong, C. C. Loy, K. He, and X. Tang. "Image Super-Resolution Using Deep Convolutional Networks", TPAMI, 2016

Even with the technique described in Patent Document 8, a situation may occur in which the imaging range in the height direction is not sufficient. As an approach to solve this problem, a method of taking a picture while tilting the camera in the height direction (pitching direction) and synthesizing a plurality of images can be considered. However, in this method, since it is necessary to tilt the camera, the entire photographing apparatus becomes structurally complicated. In addition, it takes time because multiple shootings are required.

As another approach to solve this problem, it is conceivable to provide a plurality of cameras facing different directions or arranged at different heights for shooting. However, in this case, since a plurality of cameras are required, the system becomes complicated and the cost increases.

The present invention has been made in view of the above problems, and one of the exemplary purposes of the present invention is to provide a photographing device suitable for a narrow portion.

One aspect of the present invention relates to a photographing device that generates a lateral front view image of a narrow portion. The photographing device includes a camera that shoots while moving the narrow part in the depth direction, an image of the gaze part in the image of the camera when the gaze part and the camera are relatively far from each other, and when the gaze part and the camera are relatively close to each other. It has a trained model derived from the correlation with the image of the gaze part in the camera image, and is an image processing unit that generates a front view image based on the camera images sequentially output from the camera during actual operation. And.

The image processing unit crops a predetermined area from the camera image, inputs the cropped image to the trained model to generate a high-resolution image, synthesizes the high-resolution images sequentially generated as the camera moves, and views the front view. An image may be generated.

The image processing unit crops a first region that is included in one of the pair of camera images taken at different times and is not included in the other, and inputs the cropped image to the trained model to generate a high resolution partial image. Then, the high-resolution partial image is combined with the image of the other second region of the pair of camera images to generate a high-resolution image, and the high-resolution images sequentially generated as the camera is moved are combined for front view. An image may be generated.

The image processing unit may generate a high resolution image based on an image set including a plurality of camera images taken at different times.

A predetermined area is defined at a different position for each of the plurality of camera images included in the image set, and the image processing unit crops the corresponding predetermined area from each camera image and inputs the cropped image to the trained model. A high-resolution intermediate image may be generated, and a plurality of high-resolution intermediate images obtained for a plurality of camera images may be combined to generate a high-resolution image.

The high resolution image may be vector data.

It should be noted that any combination of the above components or components and expressions of the present invention that are mutually replaced between methods, devices, systems, and the like are also effective as aspects of the present invention.

According to the present invention, it is possible to provide a photographing device suitable for a narrow portion.

FIG. 1A is a diagram for explaining a photographing target of the photographing apparatus according to the embodiment, and FIG. 1B is a diagram showing a final image which is an output of the photographing apparatus. It is a block diagram of the photographing apparatus which concerns on embodiment. FIG. 3A is a diagram illustrating the movement of the camera, and FIGS. 3B and 3C are two images IMG_A and IMG_B obtained from the viewpoints A and B of the camera of FIG. 3A. It is a figure which shows. 4 (a) and 4 (b) are views showing front view images obtained in the first comparative technique and the second comparative technique. It is a figure explaining learning. 6 (a) and 6 (b) are diagrams for explaining super-resolution processing. 7 (a) to 7 (c) are diagrams for explaining the operation of the photographing apparatus. It is a block diagram of the photographing apparatus which concerns on Example. 9 (a) and 9 (b) are views for explaining the processing of the front view image synthesizing unit. It is a figure which shows the user interface of a display. It is a figure explaining the super-resolution processing which concerns on modification 1. FIG. It is a figure explaining the super-resolution processing which concerns on modification 2.

Hereinafter, the present invention will be described with reference to the drawings based on preferred embodiments. The same or equivalent components, members, and processes shown in the drawings shall be designated by the same reference numerals, and redundant description will be omitted as appropriate. Further, the embodiment is not limited to the invention but is an example, and all the features and combinations thereof described in the embodiment are not necessarily essential to the invention.

FIG. 1A is a diagram illustrating a photographing target of the photographing apparatus 100 according to the embodiment. The photographing target of the photographing device 100 is the side of the narrow portion 2 (the road in this example). The photographing device 100 photographs both sides or one side while advancing through the narrow portion 2 in the depth direction. In the following, in order to simplify the explanation and facilitate understanding, the left side of the photographing apparatus 100 will be focused on.

FIG. 1B is a diagram showing a final image IMGf which is an output of the photographing apparatus 100. The final image IMGf is a front view image of the side of the narrow portion (here, the left side), and includes a plurality of objects OBJ1 to OBJ3 existing on the left side of the narrow portion 2.

FIG. 2 is a block diagram of the photographing apparatus 100 according to the embodiment. The photographing device 100 includes a camera 110 and an image processing unit 120. The camera 110 is directed to the front of the photographing device 100 in the traveling direction. Alternatively, when measuring only the left side, the camera 110 may be arranged at a slight inclination to the left side.

The camera 110 continuously shoots while moving in the narrow portion in the depth direction. The camera 110 may be a still camera or a video camera.

FIG. 3A is a diagram illustrating the movement of the camera. FIG. 3A shows a state in which the narrow portion 2 is viewed from the side. 3 (b) and 3 (c) are diagrams showing two camera images IMG_A and IMG_B obtained at the viewpoints A and B of the camera of FIG. 3 (a).

Focus on the central object OBJ2 (building in this example). The entire front surface of the object OBJ2 is shown in the image IMG_A taken from the viewpoint A on the front side, but since the size is small, it can be said that the resolution of the object OBJ2 is low.

In the image IMG_B taken from the viewpoint B closer to the object OBJ2 which is the subject, the object OBJ2 is larger than the image IMG_A, and the details thereof are shown in high resolution. However, the entire front of the object OBJ2 is not shown.

Before explaining the image processing of the photographing apparatus 100 according to the embodiment, some comparative techniques will be described.

In the first comparative technique, an area including the subject (area RGN1 in FIGS. 3B and 3C) is cropped from an image taken at a position close to the subject (gaze portion), and the area is converted to front view. And synthesize. FIG. 4A is a diagram showing a front view image obtained in the first comparative technique. In the first comparison technique, a high-resolution front-view image can be obtained, but the upper side of the subject is chipped.

In the second comparison technique, the area (area RGN2 in FIGS. 3B and 3C) in which the entire subject is captured is cropped from the image taken at a position far from the subject, and the area is converted to front view. , Synthesize. FIG. 4B is a diagram showing a front view image obtained in the second comparative technique. In the second comparison technique, the entire front surface of the subject can be observed, but the image is stretched by the front view conversion, so that the resolution is lowered and the edges and contours appear blurred like out-of-focus.

Returning to FIG. 2, the image processing in the present embodiment will be described. The image processing unit 120 generates the final image IMGf based on the output of the camera 110. The image processing unit 120 can be implemented by combining a calculation processing means such as a CPU (Central Processing Unit) and a software program.

A trained model 122 generated in advance by machine learning is mounted on the image processing unit 120. The machine learning of the trained model 122 will be described. The trained model 122 is used for super-resolution processing of an image.

For the super-resolution processing, a classical method that does not require training data (for example, the bilinear method or the Lanczos method) may be used, but in that case, accuracy becomes a problem. Therefore, in the present embodiment, a learning-based method such as the A + method disclosed in Non-Patent Document 1 and the SRCNN method disclosed in Non-Patent Document 2 will be adopted. The A + method is a method of constructing a learning model called a sparse dictionary from learning data. The SRCNN method is a method using deep learning. As the learning image, a general-purpose corpus may be used, or a general-purpose corpus may be prepared independently. Training of the trained model 122 requires a pair of low-resolution images and high-resolution images, but it is often used for high-resolution images and images that have been reduced in resolution using the classical method described above. is there. A predetermined value (assumed to be r) is used as the enlargement ratio of the high-resolution image corresponding to the low-resolution image.

On the other hand, as the learning data, an image obtained in an environment in which the photographing apparatus 100 is used or an environment close thereto can be used. FIG. 5 is a diagram illustrating learning. In the learning stage, a plurality of sample image SIs are taken while moving the camera 110. This sample image SI is used as training data. An object or part (called a gaze part) existing in a narrow part appears in a plurality of sample images taken from different positions in different sizes and in different positions.

In FIG. 5, X _# and Y _# (# = 1, 2, ...) Represent a region including the same gaze portion. When focusing on a certain gaze portion, a sample image when the gaze portion and the camera are relatively far from each other is referred to as a first image. Further, a sample image when the same gaze portion and the camera are relatively far from each other is referred to as a second image. The trained model is derived based on the correlation _{between the image X #} of the gaze portion in the first image _{and the image Y #} of the gaze portion in the second image. Image X _# is a wide-range, low-resolution image of _{the gaze portion, and image Y #} is a narrow-range, high-resolution image of the gaze portion. The trained model 122 can be grasped as executing the conversion process from X to Y. This conversion process is represented by a map f.
Y _# = f (X _# )
As described above, _{the pair of images X #} and Y _# may be a one-dimensional image (vector) having a width of 1 pixel. In this case, _{the height (number of pixels) of the image Y #} is r times the height (number of pixels) of the image X _#.

When the pair of images X _# and Y _# is a two-dimensional image having a width of 2 pixels or more, their shapes may be different. This is because, depending on the optical system of the camera, perspectives are added according to the distance and position to the same subject, so their shapes are photographed differently.

From a plurality of sample images, a converter Y = f (X) can be obtained as a trained model by inputting a pair of _{images X #} and Y _# into the learner for each of the plurality of gaze portions. In the image processing unit 120, a denoising process may be performed in addition to the super-resolution process.

Return to FIG. During the operation of the photographing device 100, the image processing unit 120 crops a predetermined area from the output image of the camera 110, and inputs the low resolution image included in the predetermined area to the converter which is the trained model. As a result, an image (high resolution image) obtained by super-resolution processing the entire predetermined region can be obtained. 6 (a) and 6 (b) are diagrams for explaining super-resolution processing. FIG. 6A shows a case where a one-dimensional low-resolution image IMGl is input. The high-resolution image IMGh can be grasped as an estimate of the image obtained when the camera position approaches the gaze portion, and its height h'is r times the height h of the low-resolution image IMGl (r>. 1).

FIG. 6B shows a case where a two-dimensional low-resolution image is input, and the height h'of the high-resolution image IMGh is r times the height h of the low-resolution image IMGl. The width w'of the high-resolution image IMGh may be equal to or larger than the width w of the low-resolution image IMGl.

Return to FIG. The image processing unit 120 combines the high-resolution images sequentially generated with the movement of the camera 110 to generate the final image IMGf which is a front view image.

The above is the configuration of the photographing device 100. Next, the operation will be described. 7 (a) to 7 (c) are diagrams for explaining the operation of the photographing apparatus 100. FIG. 7A shows an output image of the camera 110 obtained at a certain time. The image processing unit 120 extracts a predetermined region RGNc from the image of FIG. 7A. In this example, the width of the predetermined area RGNc is 2 pixels or more, and the cropped low-resolution image is two-dimensional image data. However, the width of the predetermined area RGNc is one pixel, and the low-resolution image is one-dimensional vector data. May be.

FIG. 7B shows a high-resolution image IMGh obtained by inputting the image data in the region RGNc into the trained model 122. For comparison, the frame of the image IMG_B in FIG. 3 (c) is shown as an alternate long and short dash line. This high-resolution image IMGh has abundant details even in the portion protruding from the frame of the image IMG_B.

FIG. 7 (c) shows an image IMGg obtained by converting the high-resolution image IMGh of FIG. 7 (b) into a front view. By repeating the processes of FIGS. 7 (a) to 7 (c) with respect to the camera images obtained at different times while changing the position of the camera, a plurality of front-view high-resolution images IMGg are generated. Then, by combining them, the final image IMGf of FIG. 1 (b) can be obtained.

Subsequently, a specific configuration of the photographing apparatus 100 will be described with reference to Examples.

FIG. 8 is a block diagram of the photographing apparatus 200 according to the embodiment. The photographing device 200 includes an image photographing unit 202, an input image determining unit 204, a sharpening processing unit 206, a front view image synthesizing unit 208, and an image output unit 210.

The input image determination unit 204 is an image acquisition means and corresponds to the camera 110 of FIG. The input image determination unit 204 may include a camera, a means for moving the camera, a means for acquiring the position or moving distance of the camera, a means for controlling the posture of the camera, and the like. The image taken by the input image determination unit 204 is input to the input image determination unit 204 together with the position information.

The input image determination unit 204, the sharpening processing unit 206, and the front view image composition unit 208 can be associated with the image processing unit 120 of FIG.

The input image determination unit 204 receives the image and position information obtained by the image capturing unit 202, and selects a part of the input image (corresponding to the predetermined area RGNc in FIG. 7B, hereinafter referred to as a selection area). , Output it. The selected area is an area in which a relatively wide range located far from the camera is photographed, and is an area in which the sharpness (effective resolution) is low because the area is too far from the camera. The selected area may be a linear area (vector) or a wide area (image). The image data of the selected area is supplied to the sharpening processing unit 206 together with the position information.

The sharpening processing unit 206 receives a selection area from the input image determination unit 204 and outputs an image with enhanced sharpness. "Clarification" includes a function of enlarging an image and a function of removing noise (denosing), such as super-resolution processing. In the case of super-resolution processing, if the selected area is vector data, the output is a vector whose size is the number of elements multiplied by a constant. In the case of two-dimensional image data in which the selected area has a width, an image having a size obtained by multiplying the number of pixels in the vertical direction and the horizontal direction by a constant shall be output. The output of the sharpening processing unit 206 includes a high-resolution image (one-dimensional vector or two-dimensional image) and its position information. As a means for generating a high-resolution image, the trained model 122 of FIG. 2 can be used.

9 (a) and 9 (b) are views for explaining the processing of the front view image synthesizing unit 208. The front view image synthesizing unit 208 joins the high-resolution images output from the sharpening processing unit 206 together by using the position information output together with the sharpening processing unit 206 to generate one large final image IMGf. When the high-resolution image is a one-dimensional image (vector), as shown in FIG. 9A, a plurality of high-resolution images L1 to Ln may be combined in the horizontal direction. When the high-resolution image is a two-dimensional image, as shown in FIG. 9B, a margin portion 4 may be provided and bonded. In this case, as the pixel value in the margin portion 4, the average value of the corresponding pixels may be used, or the maximum value may be used. By this process, a front view image can be obtained. If an unusual edge or the like is generated at the joint as a result of the bonding process, a post-treatment for smoothing the edge or the like may be performed, and for example, a Gaussian filter or a Wiener filter can be used.

As pre-processing or post-processing of bonding, conversion processing such as camera position information correction may be performed in order to bring the image closer to the front view image.

Returning to FIG. 8, the image output unit 210 outputs the final image IMGf obtained by the front view image composition unit 208. The image output unit 210 may include a display and visually present the final image IMGf to the user. The image output unit 210 may have an enlargement / reduction function. Further, when there are a plurality of final images, a function of displaying them at the same time and a function of displaying them selectively may be provided. When the size of the final image is large, the display portion may be changed by a GUI (Graphical User Interface) such as a slide bar.

Further, the image output unit 210 may have a function of adding a marker to a portion of interest to the user (for example, an abnormal portion or a damaged portion) and highlighting the portion. Further, it may have a function of calculating the position information of the portion of interest to the user and displaying the value. The image output unit 210 may store the data of the final image IMGf in a storage medium.

FIG. 10 is a diagram showing a user interface of the display. The output image of the camera is displayed in the area 302 of the display 300. Along with this area 302, buttons 306 for play, stop, and rewind are provided so that the image (or frame) displayed in the area 302 can be controlled.

Marking 308 can be displayed in the area 302. The marking 308 is an area designated by the user, and the presence or absence of display of the marking 308 can be controlled by the marking button 310. For example, when the photographing apparatus 200 is a diagnostic apparatus, the area surrounded by the marking 308 is the target of the abnormality detection process, and the abnormal portion or the damaged portion in the region is detected.

The final image IMGf obtained by image processing is displayed in the area 310. Region 310 may be divided into a plurality of columns 312. For example, different parts of the final image IMGf may be selectively displayed in the plurality of columns 312. Further, when generating both the right side and left side front view images of the narrow portion, the final image of the left side surface may be displayed on one of the plurality of columns 312, and the final image of the right side surface may be displayed on the other side. If the entire final image cannot be displayed in column 312, the slide bar 314 may be displayed so that the user can select the display range. In addition to or in place of the slide bar 314, buttons for enlarging or reducing the image may be added.

The marking 316 can also be displayed in the area 310, and the display unit (that is, 310) of the marking 316 can be controlled to be enlarged or reduced by the buttons 320 and 322.

The above is the configuration of the photographing device 200. According to the photographing device 200, it is possible to generate a front-view image in which a wide range is captured, and it becomes easy for a human being to visually discover what is present in the image. Furthermore, since there is almost no distortion, it becomes easy to automatically find out where and what is present by the image recognition process. Therefore, it is possible to improve the accuracy of visual processing and automatic image recognition processing using the output of the photographing apparatus 200.

The application of the photographing device 100 and the photographing device 200 is not particularly limited, but it is widely applied when the measurement target is long and narrow such as a tunnel, a clay pipe, a coke oven, etc. it can.

The present invention has been described above based on examples. It will be understood by those skilled in the art that the present invention is not limited to the above embodiment, various design changes are possible, various modifications are possible, and such modifications are also within the scope of the present invention. By the way. Hereinafter, such a modification will be described.

(Modification example 1)
In the conventional super-resolution processing, a high-resolution image of one gaze portion is generated using one image, but in the modified example 1, a set of a plurality of camera images taken at different distances is used. Then, a high-resolution image of one gaze part is generated. FIG. 11 is a diagram illustrating the super-resolution processing according to the first modification. FIG. 11 shows a plurality of (three images in this example) camera images obtained at different times, in other words, the same gaze portion taken from different distances. For each of the plurality of camera images, a unique area is defined so as to include the same gaze portion. From each camera image, an image of a unique region is cropped to generate three low-resolution images IMGl1 to IMGl3. By performing super-resolution processing on the three low-resolution images IMGl1 to IMGl3, three high-resolution intermediate images IMGh1 to IMGh3 can be obtained. Three high-resolution intermediate images IMGh1 to IMGh3 are combined to generate one high-resolution image IMGh.

(Modification 2)
FIG. 12 is a diagram illustrating the super-resolution processing according to the second modification. In the super-resolution processing of FIGS. 6A and 6B, the cropping region was defined to include the entire gaze portion. On the other hand, in the modified example of FIG. 12, a part of the gaze portion is cropped and super-resolution processing is performed. The left side of FIG. 12 shows an image obtained when it is far from the gaze portion 400, and the right side of FIG. 12 shows an image obtained when it is close to the gaze portion 400. In the image of FIG. 12, reference numeral 402 represents a portion that is framed out when the camera approaches the gaze portion. In this modification, this portion 402 is cropped, and the cropped image is subjected to super-resolution processing. As a result, a high-resolution image portion (high-resolution partial image) 404 of a portion that is framed out when the camera approaches the gaze portion is obtained. Further, the range 406 of the gaze portion framed in the image obtained when the camera is close to the gaze portion is selected, and the high resolution image IMGh can be generated by combining the two portions 404 and 406.

2 Narrow area 100 Imaging unit 110 Camera 120 Image processing unit 122 Learned model 200 Imaging device 202 Image imaging unit 204 Input image determination unit 206 Clarification processing unit 208 Front view image composition unit 210 Image output unit

Claims (6)

It is a photographing device that generates a front view image of the side of a narrow part.
A camera that shoots while moving the narrow part in the depth direction,
Correlation between the image of the gaze portion in the image of the camera when the gaze portion and the camera are relatively far from each other and the image of the gaze portion in the image of the camera when the gaze portion and the camera are relatively close to each other. An image processing unit that has a trained model derived from, and generates the front view image based on the camera images sequentially output from the camera during actual operation.
A photographing device characterized by being provided with. The image processing unit crops a predetermined area from the camera image, inputs the cropped image to the trained model to generate a high-resolution image, and sequentially generates the high-resolution image as the camera moves. The photographing apparatus according to claim 1, wherein the image is combined to generate the front view image. The image processing unit crops a first region included in one of the pair of camera images taken at different times and not included in the other, and inputs the cropped image to the trained model to provide a high resolution portion. Generate an image and
The high resolution partial image is combined with the image of the other second region of the pair of camera images to generate a high resolution image.
The photographing apparatus according to claim 1, wherein the high-resolution images sequentially generated with the movement of the camera are combined to generate the front-view image. The imaging device according to claim 1, wherein the image processing unit generates a high-resolution image based on an image set including a plurality of the camera images captured at different times. Predetermined areas are defined at different positions for each of the plurality of camera images included in the image set.
The image processing unit crops a corresponding predetermined area from each camera image, inputs the cropped image to the trained model, and generates a high-resolution intermediate image.
The photographing apparatus according to claim 4, wherein a plurality of high-resolution intermediate images obtained from the plurality of camera images are combined to generate the high-resolution image. The photographing apparatus according to any one of claims 2 to 5, wherein the high-resolution image is vector data.

Images