KR102085070B1 - Apparatus and method for image registration based on deep learning - Google Patents

Abstract

The present invention relates to an image matching device and a method, and an image matching device according to an embodiment of the present invention includes a visible light image input unit for receiving a visible light (RGB) image; An infrared image input unit for receiving an infrared (IR) image; A feature extractor which extracts an object from a visible light image using a deep learning technique and extracts a background image from an infrared image; And a matching unit matching the extracted object with the background. According to the present invention, an object recognition and identification capability may be improved even in an image captured at night.

Description

Deep learning based image registration device and method {Apparatus and method for image registration based on deep learning}

The present invention relates to a deep learning based image registration device and method, and more particularly, to accurately detect an object by matching a visible light image and an infrared image in a situation where it is difficult to detect an object such as a pedestrian or a vehicle at night. Deep learning-based image registration device and method.

It is difficult to check a person's identity because the image quality and outline are blurred in the video taken by CCTV at night. That is, the image photographed with visible light at night is difficult to recognize the object well. On the other hand, the infrared image, unlike the visible light image, clearly shows the outline of an object such as a pedestrian or a vehicle, so that the object and the background are clearly distinguished. In the visible image, it is difficult to accurately distinguish the dark part of the object, whereas in the infrared image, objects such as pedestrians or vehicles are relatively visible in the dark part, and thus it is easy to distinguish the object. This feature is more prominent in poorly lit environments such as night vision. However, infrared images are monochromatic so it is difficult to see the characteristics of the object.

CCTVs installed for security and safety of facilities require high object identification. In particular, there is a need for a method for clearly distinguishing objects such as pedestrians or vehicles and clearly identifying the characteristics of the images captured at night.

An image matching device and method for clearly detecting an object such as a person or a vehicle even with an image captured at night are proposed.

Image matching apparatus according to an embodiment of the present invention is a visible light image input unit for receiving a visible light (RGB) image; An infrared image input unit for receiving an infrared (IR) image; A feature extractor which extracts an object from a visible light image using a deep learning technique and extracts a background image from an infrared image; And a matching unit matching the extracted object with the background.

The feature extractor may include first and second Bbox-preds for calculating coordinate values of a bounding box for separating an object in a box offset regressor (x, y, w, h) background image for a visible light image and an infrared image, respectively. (Bounding box-prediction) unit; And first and second classification layers (CLS) -scores in the process of separating the object region from the background image by determining whether the object is formed (true, false) using the Softmax function for the visible light image and the infrared image, respectively. It includes;

The image matching device further includes a morphological analysis filter for acquiring color features in a visible light (RGB) image frame.

The image matching device further includes a morpheme analyzer for separating the morphological vector matrix for each visible light (RGB) image frame and the infrared (IR) image frame.

The image registration device includes a space vector generator for generating an empty vector; And a frame tracker intersection generator for generating an image frame track to contain the object matched with the ball track to be intersected.

The image matching device includes: a morphological analysis piece generator for outputting image information; And an object frame vector separator that separates the visible light image and the infrared image into pipes to generate space allocation.

The image matching device may further include a frame matching generator for allocating a frame received by the object frame vector separator to a vector space.

The present invention relates to an apparatus and method for acquiring color information from a visible light image, acquiring contour information from an infrared image, and matching the same. The present invention uses only the background image in the infrared image and discards the shape of the object.On the contrary, in the visible image, the background image is discarded and only the morphemes such as pedestrians or vehicles are extracted and matched to the infrared image. Use the method. Accordingly, it is possible to provide an image processing apparatus and method capable of real-time matching high resolution visible light images and infrared images using deep learning image analysis techniques.

The visible and infrared images pass through four convolutional layers through four parts of the deep CNN (Convolutional Neural Network) to calculate the characteristics of the object.

The Region CNN Feature section is a Region Proposal Network that connects widget dataweight analysis tools necessary for CNN learning to create feature maps of objects that reflect the characteristics of visible and infrared images that pass through a fourth convolution layer. One that contains detailed visual information through the fifth convolutional layer, combining the features of objects formed through the NIN (Network in Network) Converter, which controls the dimension of the feature map through the 1 × 1 convolution. Feature map is created.

 The Bbox-pred (Bounding box-prediction) section calculates the coordinates of the bounding box for separating objects in the background image of the Box Offset Regressor (x, y, w, h), and the Cls (Classification Layers)-score section is the Softmax function. Use to determine whether the object is formed (true, false) and separate the object area from the background image.

 The matching part is an object separated from the background through the Cls (Classification Layers) -score part of the visible light image, and the object is a box offset regressor (x, y, w, h) of the bounding box-prediction part in the infrared image. Match the background images except for

The present invention accurately recognizes an object, such as a pedestrian or a vehicle, and acquires and reads color information with respect to an image photographed in a low light illumination environment at night. In reading or feature maps, two factors are important in identifying objects by morphologically identifying objects and recognizing color. The characteristics of the pedestrian can be divided into colors such as short-sleeved clothes, long pants, red short-sleeved clothes, and black long pants, so the accuracy of object recognition can be improved. As the characteristics of the object are subdivided, the accuracy of tracking and finding the same object can be improved by comparing weights and similarities in a large storage space. In addition, even when visually identified, when objects such as pedestrians and vehicles are displayed in color, the characteristics of the objects can be easily distinguished and easily remembered.

According to the present invention, it is possible to accurately detect an object such as a person or a vehicle even with an image photographed at night.

By simultaneously capturing and matching visible and infrared images in a low visibility environment at night, it is possible to accurately read objects such as pedestrians and vehicles, and clearly distinguish between foreground and background in object detection using deep learning image analysis algorithms. There is an effect that can increase the detection rate.

In addition, the color of the object can not be recognized at most by shooting in infrared mode or by thermal image at night, but color information can be obtained even at night by matching visible and infrared images.

Also, when shooting a forest in the daytime, you can't identify objects such as people or vehicles in the forest, but it is easy to recognize by matching the visible and infrared images, even in a vehicle that is heavily tanned invisible to the naked eye. You can easily recognize the object.

1 shows an image matching device according to an embodiment of the present invention.
FIG. 2 illustrates one embodiment of the matching unit 108 shown in FIG. 1.
3 is a diagram illustrating a memory space allocation and registration recording process of a vector frame in an image matching device according to an embodiment of the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting the invention. In this specification, the singular also includes the plural unless specifically stated in the text. As used herein, “comprises” and / or “comprising” does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be the second component within the technical idea of the present invention. In addition, the terms "... unit", "module", etc. described in the specification mean a unit for processing at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software. .

Hereinafter, with reference to the drawings will be described in more detail with respect to the present invention. The embodiments introduced below are provided as examples to sufficiently convey the spirit of the present invention to those skilled in the art to which the present invention pertains.

1 shows an image matching device according to an embodiment of the present invention.

Referring to FIG. 1, the image matching device 100 according to the present invention includes a visible light (RGB) image input unit 101a, an infrared (IR) image input unit 101b, an image feature extractor 110, and an image matcher ( 108). The visible light (RGB) image input unit 101a and the infrared (IR) image input unit 101b respectively receive images obtained from an image sensor. That is, the visible ray (RGB) image input unit 110a receives a visible ray (RGB) image, and the infrared ray (IR) image input unit 110b receives an infrared image.

The image feature extractor 110 extracts a feature required from a visible light (RGB) image using a deep learning technology and extracts a feature required from an infrared (IR) image. In more detail, the image feature extractor 110 may extract an object from a visible light (RGB) image and extract a background from an infrared image (IR).

 The image feature extraction unit 110 includes four deep CNN (Convolutional Neural Network) units 102a and 102b, a Region Proposal Network (RPN) unit 103a and 103b, a Region CNN feature unit 104a and 104b, and a network in NIN. Network Converter 105, Bbox-pred (Bounding box-prediction) 106a, 106b, Classification Layers (Cls) -score 107a, 107b, Image Matching 108, Image Output 109 ).

The Deep CNN 4 units 102a and 102b use a convolutional neural network (CNN), which is a deep learning image analysis algorithm, to the images received from each of the visible (RGB) image input unit 101a and the infrared (IR) image input unit 101b. Apply. At this time, the structure of the two CNNs are the same and only the inputs are different. The Deep CNN 4 parts 102a and 102b are composed of multiple layers of convolutional layers, of which two feature maps passing through the fourth convolutional layer are joined along the channel dimension. Calculate each feature.

Convolutional Neural Network (CNN) is a type of deep learning algorithm that is specialized for processing data arranged in a grid and is an effective neural network for identifying patterns of data. Convolutional Neural Network (CNN) is basically a neural network with layers that use convolution operations on images. CNN (Convolutional Neural Network) utilizes multiple filters that can be used as shared parameters to effectively extract and learn features from adjacent images by maintaining spatial information of images in two dimensions. Convolutional Neural Networks (CNNs) have the advantage of enabling easier learning with minimal parameters and preprocessing.

The RPN (Region Proposal Network) units 103a and 103b each learn CNN to create feature maps reflecting the characteristics of the visible light (RGB) and infrared (IR) images from two feature maps passing through the fourth convolutional layer. Connect widget data set analysis tools needed for. In general, the Region Proposal Network (RPN) part plays a key role in object detection and uses three elements: basic anchor box, delta, and probaility. The anchor box is a fixed area initially set up to check if an object is found in this area, which are small rectangular areas partitioned inside the input image. Delta is the value to adjust the size and position of the basic anchor. This is learned through the AI model. One Delta corresponds to one Anchor. Probability is the probability that an object exists inside each anchor. After all, the output of RPNs is that there is a high probability that the object exists, and the bounding boxes are deduplicated. This is called the ROI (Region of Interest).

Region CNN Feature section 104a, 104b combines the features of the object formed in the fourth convolutional layer and Region Proposal Network (RPN) section 103a, 103b to provide semantic and detailed visual information, which is the fifth convolutional layer. One feature map is created.

The NIN (Network in Network) Converter 105 adjusts the dimension of the feature map through 1 × 1 convolution, which must be matched in order to use the weight of the pre-trained model. need.

The bounding box-prediction (Bbox-pred) units 106a and 106b are used to calculate coordinate values of a bounding box for separating objects in a box offset regressor (x, y, w, h) background image.

Cls (Classification Layers) -score unit 107a, 107b separates the object area from the background image by determining whether the object is formed (true, false) using the Softmax function.

The image matching unit 108 is an object separated from the background through a classification layer (SCI) -score unit 107a of a visible light (RGB) image, and a boxing box-prediction (Bbox-pred) in an infrared (IR) image. Through the Box Offset Regressor (x, y, w, h) of the unit 106b, the backgrounds other than the objects are matched with each other.

FIG. 2 illustrates an embodiment of the image matcher 108 illustrated in FIG. 1.

1 and 2, an image matching unit 200 according to an embodiment of the present invention may include a classification layer (Score) 107a (CLS) illustrated in FIG. 1 in a visible light (RGB) image frame unit 201a. Receives an object (or foreground) frame required for image frame registration from the apparatus. In addition, the infrared (IR) image frame unit 201b receives a background image as a box offset regressor (x, y, w, h) of Bbox-Pred (Bounding box-prediction, 107b) shown in FIG. Receive a background space vector frame without the outline of the foreground) (step 201b).

The Morpheme filter 202 performs a filtering operation to remove noise of the color image, and acquires color features from the visible light (RGB) image frame unit 201a.

In the morpheme analyzer 203, the color object received through the received background space vector frame received from the infrared (IR) image frame unit 201b, the visible light (RGB) image frame unit 201a, and the morpheme filter 202. Analyze each feature using morphemes. After separating the morpheme vector matrix for each frame through the morpheme analyzer 203, the vector elements are generated by the spatial vector generator 206-1.

The spatial vector generator 206-1 generates an empty vector, and generates an image frame track to contain the object matched with the empty track to intersect the track of the frame tracker intersection generator 206-3.

The object frame vector separator 204 analyzes only the digitized vector information of the object analyzed by the morphological analyzer 203. The object frame vector separator 204 generates a space allocation by dividing the image information received through the morphological frame pipe generator 206-2 into a visible light (RGB) image and an infrared (IR) image pipe.

The stem frame type determiner 204-1 provides an image frame feature type for stemming when the object frame vector separator 204 operates. In this case, the type refers to a widget data set required for feature analysis. In order to provide image frame feature types required for morphological analysis, pre-learning is performed through a widget data set necessary for feature analysis.

 The frame match generator 205 assigns the received frame to vector space. It handles functions such as cross frame tracker required for frame matching.

 The image matching initializer 206 is a space vector generator 206-1, a stem vector frame pipe generator 206-2, and a frame tracker cross generator that are output from the frame matching generator 205 and a space vector frame processing apparatus. 206-3) and the image generator 206-5 to connect the devices to activate the image matching device. An object vector block is generated based on a color feature object extracted from a morpheme by taking a color object image.

The registered image generator 207 performs object (or foreground) image registration on the vector space forming the background in real time using the image registration initializer 206.

3 is a diagram illustrating a memory space allocation and registration recording process of a vector frame in an image matching device according to an embodiment of the present invention.

2 and 3, an image container space vector is created by the space vector generator 206-1 (step 301). The cross vector is then generated through the stem frame pipe generator 206-2 (step 302) and then the tracker cross generator 206-3 is created (step 304). The feature track is an object (or foreground) of visible (RGB) images and an object (or foreground) of infrared (IR) images in the stem frame pipe generator 206-2 and the tracker cross generator 206-3. After receiving the background image and filling it with the matched image stream data of the cross vector 302 and the tracker cross generator 304, the feature data is recorded (step 303). The group 206 is created (step 305).

An image matching device and method according to an embodiment of the present invention is an image matching technology using a CNN (Convolutional Neural Network), which is an artificial neural network suitable for image analysis and processing, and detects an object such as a pedestrian or a vehicle in real time with a high resolution image. In addition, color information can be obtained, thereby maximizing the reading effect in identification or recognition. This technology can improve the recognition rate of objects because deep learning image analysis can distinguish and classify objects (or foreground) and backgrounds such as pedestrians and vehicles.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, in a software module executed by hardware, or by a combination thereof. Software modules may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may realize the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

101a: visible light image input unit 101b: infrared image input unit
102a, 102b: Deep CNN Part 4 103a, 103b: RPN
104a, 104b: Region CNN Feature 105: NIN Converter
106a, 106b: Box Offset Regressor 107a, 107b: Softmax
108: matching unit 109: image output unit

Claims (8)

delete delete A visible light image input unit configured to receive a visible light (RGB) image;
An infrared image input unit for receiving an infrared (IR) image;
A feature extractor which extracts an object from a visible light image using a deep learning technique and extracts a background image from an infrared image; And
And a matching unit for matching the extracted object with a background.
The feature extraction unit
Box Offset Regressor (x, y, w, h) First and second Bbox-preds (Bounding box-) for calculating coordinate values of bounding boxes for separating objects in a background image for visible and infrared images prediction unit; And
First and second classification layers (Cls) -score part of the process of separating the object area from the background image by determining whether the object is formed (true, false) using the Softmax function for the visible light image and the infrared image, respectively. Including;
The matching part
Box Offset Regressor (x, y) of the object separated through the first Classified Layers (Cls) -score part of the visible light (RGB) image and the second Bbox-pred (bounding box-prediction) part in the infrared (IR) image , w, h) image matching device to match the background image excluding the object. The method of claim 3,
And a morphological analysis filter for obtaining color features in a visible light (RGB) image frame. The method of claim 3,
And a morpheme analyzer for separating morphological vector matrices for each visible light (RGB) image frame and an infrared (IR) image frame. The method of claim 5,
A space vector generator for generating a ball vector; And
And a frame tracker intersection generator for generating an image frame track to contain the object matched with the ball track to be intersected. The method of claim 6,
A morphological analysis piece generator for outputting image information; And
And an object frame vector separator for separating a visible light image and an infrared image by a pipe to generate a space allocation. The method of claim 7, wherein
And a frame matching generator for allocating a frame received from the object frame vector separator to a vector space.

Images