CN103996181B - A kind of big view field image splicing system and method based on bionical eyes - Google Patents

Abstract

The invention discloses a bionic binocular-based large-field image mosaic system and method. Including high-definition cameras, connected to the onboard fast processor SECO CARMA DevKit; through Zeroconf technology and robot operating system ROS (Robot Operating System), the onboard fast processor SECO CARMA DevKit and high-performance computers form a distributed computing network; high-definition camera shooting After the high-definition image is imported, the image is transferred to the onboard fast processor SECO CARMA DevKit through the USB interface, and the feature points of the high-definition image are extracted, and then the high-performance computer matches and stitches the image. In the method of the invention, a high-performance computer and a fast image processor CARMA DevKit are formed into a distributed computing network to create different nodes, firstly the airborne embedded CARMA DevKit is used to extract feature points, and then image splicing is completed on the host computer. Embodiments of the present invention are mainly used for large-field-of-view stitching in images, especially for large-field-of-view environment perception in mobile robots.

Description

A large field of view image stitching system and method based on bionic binoculars

technical field

The invention discloses a bionic binocular-based large-field image mosaic system and method, and relates to the fields of robot vision, feature point extraction technology and CUDA parallel computing.

Background technique

Robot vision is the most efficient way to perceive the surrounding environment and plays a vital role in the navigation of mobile robots. Real-time, large field of view image information is very important for the navigation operation of ground mobile robots.

Image stitching refers to the synthesis of multiple small-sized images from the same scene with a certain overlapping area into a large-sized image with a large viewing angle. Image stitching can break through the limitations of the shooting angle of the lens and expand the field of view. It is widely used in remote sensing image processing, medical image analysis, cartography, computer vision, video surveillance, virtual reality, super-resolution reconstruction, and robot navigation. At present, due to the large amount of data involved in the image stitching process and the intensive calculation, many serial processing methods are difficult to meet the real-time requirements in practical applications.

At present, complex image algorithms are directly calculated on the airborne platform. At this time, the airborne platform needs to be equipped with exclusive FPGA or DSP hardware, and the algorithm is solidified on the relevant hardware, and relevant optimization is performed to reduce the economy; or mobile In the image servo system of the robot airborne platform, the image information is transmitted to the upper server machine for processing through the network, which requires high processing of the upper server.

Most of the image processing resolutions on the airborne platform are 640×480, which cannot be processed in real time for high-definition images. Due to the large amount of calculation and the large amount of data, the industrial control computer mounted on the mobile robot airborne platform cannot meet the calculation requirements for direct image stitching. The image stitching process is mainly divided into three steps: image acquisition, image matching and image fusion, and the core technology is image matching. The key to image matching is to accurately determine the position of the overlapping parts in the two images, so as to obtain the transformation relationship of the two images. Image stitching has an image matching algorithm based on feature points. The extraction of feature points in this process is computationally complex and can be accelerated by GPU parallelization.

In March 2010, Willow Garage (http://www.willowgarage.com/) released the robot operating system Robot Operating System (ROS). The robot operating system ROS provides various libraries and tools to help software developers develop robot applications, including hardware abstraction layer, hardware drivers, virtualization tools, message passing, and software package management. At the same time, SECO, an embedded hardware solution provider, and NVIDIA jointly released an embedded GPGPU platform solution product CARMA DevKit for researchers. high-performance computing platform.

Contents of the invention

In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a system and method for large-field image stitching based on bionic binoculars, which solves the problems of slow processing speed of high-definition real-time image stitching or the need for dedicated hardware equipment.

In order to achieve the above-mentioned purpose, the idea of the present invention is: firstly, the image input system collects high-definition images; The result is returned to the host computer for the final processing of image stitching.

The large field of view image stitching system based on bionic binoculars of the present invention comprises:

(1) High-definition image input, ARTAM-1400MI-USB3 high-definition camera is transmitted to the processor through the USB interface;

(2) Fast image processing system: Through the parallel computing technology of SECO CARMA DevKit embedded CUDA software and hardware platform, real-time image feature detection is performed on the high-definition images collected in real time.

(3) The upper computer subscribes to the calculation results of the SECO CARMA DevKit embedded CUDA software and hardware parallel computing platform for final image matching and fusion.

According to above-mentioned inventive concept, the present invention adopts following technical scheme:

A large field of view image stitching system based on bionic eyes, including the two high-definition cameras, characterized in that: the two high-definition cameras are respectively connected to the corresponding fast image processor CARMA DevKit; the fast image processing The CARMA DevKit is connected to a switch through the network; the switch is connected to a main control machine and a DSP processor.

A bionic binocular-based large field of view image stitching method, using the above-mentioned bionic binocular-based large field of view image stitching system for image stitching, characterized in that the stitching steps are as follows:

Step ①: Set the main control machine as the host MASTER of the robot operating system ROS, create an image acquisition node by CARMA DevKit, and obtain high-definition images from the high-definition camera and transmit them to the image fast processor CARMADevKit in real time through the USB interface;

Step ②: CARMA DevKit utilizes the distributed computing characteristics of the robot operating system ROS to create a SURF algorithm node to extract feature points from the obtained image;

Step ③: The computing node using CARMA DevKit publishes the data described by the feature points to the matching node of the master computer for optimal matching of feature points.

Step ④: The best matching result of the feature points calculated by the matching node of the master computer is released to the affine stitching computing node for affine calculation, and then the data of the left and right cameras are stitched into one image.

The described large-field-of-view image stitching method based on bionic binoculars is characterized in that, the specific steps of the image acquisition of the step ① are as follows:

Step 1-1: Use Zeroconf technology to interconnect the main control machine and two CARMA DevKits as network nodes, and then set the main control machine as the MASTER of the robot operating system ROS distributed operating system.

Step 1-2: Use the camera operation package uvc_camera integrated in the robot operating system ROS system to create a camera opening node and publish the image data.

The described large field of view image stitching method based on bionic binoculars is characterized in that, the specific steps of the image feature point detection of the step 2. are as follows:

Step 2-1: Install the robot operating system ROS in CARMA DevKit, and connect CARMA DevKit to the main control machine through Zeroconf.

Step 2-2: Use the OpenCV image processing library integrated in the ROS system to create a distributed SURF algorithm node, extract feature points from the obtained image, and publish the calculation results to the matching node of the host.

The described large field of view image stitching method based on bionic binoculars is characterized in that, the specific steps of the image feature point matching of the step 3. are as follows:

Step 3-1: Install the robot operating system ROS in the master computer, and use Zeroconf technology to set the master computer as the MASTER of the entire distributed computing of the robot operating system ROS.

Step 3-2: Create the best matching node, subscribe to the feature point description information of the image feature point extraction published by the SURF algorithm node, and perform the best matching of the feature points.

The described large-field-of-view image mosaic method based on bionic binoculars is characterized in that, the specific steps of the image affine mosaic of the step ④ are as follows:

Step 4-1: Create an image affine stitching node on the host computer, subscribe to the best matching information of feature points published by the best matching node and subscribe to the image data published by the CARMA DevKit image acquisition node.

Step 4-2: Find the homography matrix of the left and right images according to the best matching information of the feature points, perform affine transformation, stitch them into one image, and perform data fusion.

Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages: the present invention can quickly detect the feature points in the input high-definition image on the airborne platform. In this way, there is no need for on-board dedicated image processing hardware and optimized hardware algorithms, and the image feature point detection and image matching system can be quickly deployed on the mobile robot platform.

Description of drawings

Fig. 1 is a hardware block diagram of the system of the present invention.

Fig. 2 is a flow chart of the method of the system of the present invention.

Figure 3 is an image stitching effect diagram.

detailed description

The preferred embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention.

Embodiment one:

Referring to Fig. 1, this large-field-of-view image stitching system based on bionic binoculars includes two high-definition cameras 1 and 2, which are characterized in that they are respectively connected to two corresponding fast image processors CARMA DevKit 3 and 4 (embedded hardware solution Provider SECO and NVIDIA jointly released an embedded GPGPU platform solution product CARMA DevKit for researchers, which uses graphics card parallel computing technology to improve the speed of computing and builds a mobile high-performance computing platform); The two image fast processors CARMA DevKit 3 and 4 are connected to a switch 5 through the network; the switch 5 is connected to a main control machine 6 and a DSP 7 processor. Two high-definition cameras 1 and 2 are ARTAM-1400MI-USB3 14 million USB3 industrial cameras from ARTRAY Corporation of Japan; two fast image processors CARMA DevKit 3 and 4 are produced by SECO Corporation of the United States and are mainly used for embedded CUDA Calculation; the switch 5 model is TP-LINKTL-SF1008+ 8-port 100M switch of Pulian Company; the main control machine 6 model is HP 8470w mobile workstation, and the DSP 7 processor uses TMS320F2812 DSP2812 development board.

Embodiment two:

Referring to Figure 1, Figure 2 and Figure 3, this large-field image stitching method based on bionic binoculars uses the above-mentioned system for image stitching, and the stitching steps are as follows:

Step ①: Set the master computer as the robot operating system ROS (in March 2010, Willow Garage released the robot operating system Robot Operating System, referred to as ROS. The robot operating system ROS provides various libraries and tools to help software developers develop robots Application, including hardware abstraction layer, hardware driver, virtualization tool, message delivery, software package management) host MASTER, image acquisition node is created by CARMA DevKit, and high-definition images are acquired by high-definition cameras and transmitted to image fast processor CARMA in real time through USB interface DevKit;

Step ②: CARMA DevKit utilizes the distributed computing characteristics of the robot operating system ROS to create a SURF algorithm node to extract feature points from the obtained image;

Step ③: The computing node using CARMA DevKit publishes the data described by the feature points to the matching node of the master computer for optimal matching of feature points.

Step ④: The best matching result of the feature points calculated by the matching node of the master computer is released to the affine stitching computing node for affine calculation, and then the data of the left and right cameras are stitched into one image.

Embodiment three:

This embodiment is basically the same as Embodiment 2, and the special feature is that the specific steps of image acquisition in step ① are as follows:

Step 1-1: Use Zeroconf technology to interconnect the main control machine and two CARMA DevKits as network nodes, and then set the main control machine as the MASTER of the robot operating system ROS distributed operating system.

Step 1-2: Use the camera operation package uvc_camera integrated in the robot operating system ROS system to create a camera opening node and publish the image data.

The concrete steps of the image feature point detection of described step 2. are as follows:

Step 2-1: Install the robot operating system ROS in CARMA DevKit, and connect CARMA DevKit to the main control machine through Zeroconf.

Step 2-2: Use the OpenCV image processing library integrated in the ROS system to create a distributed SURF algorithm node, extract feature points from the obtained image, and publish the calculation results to the matching node of the host.

The concrete steps of the image feature point matching of described step 3. are as follows:

Step 3-1: Install the robot operating system ROS in the master computer, and use Zeroconf technology to set the master computer as the MASTER of the entire distributed computing of the robot operating system ROS.

Step 3-2: Create the best matching node, subscribe to the feature point description information of the image feature point extraction published by the SURF algorithm node, and perform the best matching of the feature points.

The concrete steps of the image affine mosaic of described step ④ are as follows:

Step 4-1: Create an image affine stitching node on the host computer, subscribe to the best matching information of feature points published by the best matching node and subscribe to the image data published by the CARMA DevKit image acquisition node.

Step 4-2: Find the homography matrix of the left and right images according to the best matching information of the feature points, perform affine transformation, stitch them into one image, and perform data fusion.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes and substitutions that can be easily imagined by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (5)

1. A large-field image stitching method based on bionic binoculars, using a bionic binocular-based large-field image stitching system for image stitching, the system includes two high-definition cameras, and the two high-definition cameras are respectively connected to corresponding fast image processing Device CARMA DevKit (3,4); Described fast image processor CARMA DevKit (3,4) is connected to a switchboard (5) through network; Described switchboard (5) is connected to a main control machine and a DSP ( 7) processor; It is characterized in that, splicing steps are as follows: Step ①: Set the main control machine as the host MASTER of the robot operating system ROS, create an image acquisition node by CARMA DevKit (3, 4), and acquire high-definition images from the left and right high-definition cameras and transmit them to the image fast processor in real time through the USB interface CARMA DevKit (3, 4); Step ②: CARMA DevKit (3, 4) uses the distributed computing characteristics of the robot operating system ROS to create a SURF algorithm node to detect the high-definition image feature points of the obtained high-definition image; Step ③: Create a SURF algorithm node using the distributed computing characteristics of the robot operating system ROS by CARMA DevKit (3, 4), and publish the data described by the feature points to the matching node (10) of the host computer to optimize the feature points of the high-definition image match; Step ④: The best matching result of the feature points calculated by the matching node (10) of the master computer is published to the affine stitching computing node for affine calculation, and then the high-definition images of the left and right two high-definition cameras are affinely stitched into one image. 2. the large field of view image stitching method based on bionic binoculars according to claim 1, is characterized in that, described step 1. the specific steps of obtaining high-definition images are as follows: Step 1-1: Use Zeroconf technology to interconnect the main control machine and two CARMA DevKit (3, 4) as network nodes, and then set the main control machine as the MASTER of the robot operating system ROS distributed operating system; Step 1-2: Use the camera operation package uvc_camera integrated in the robot operating system ROS distributed operating system to create a camera opening node and publish high-definition image data. 3. the large field of view image stitching method based on bionic binoculars according to claim 1, is characterized in that, the concrete steps of the high-definition image feature point detection of described step 2. are as follows: Step 2-1: Install the robot operating system ROS distributed operating system in CARMA DevKit (3, 4), and connect CARMA DevKit (3, 4) to the master computer through Zeroconf technology; Step 2-2: Utilize the OpenCV image processing library integrated in the robot operating system ROS distributed operating system to create a SURF algorithm node, extract feature points from the obtained high-definition image, and publish the calculation results to the matching node of the host computer (10 ). 4. the large-field-of-view image stitching method based on bionic binoculars according to claim 1, is characterized in that, the specific steps of the best matching of the high-definition image feature points of described step 3. are as follows: Step 3-1: Install the robot operating system ROS in the master computer, and use Zeroconf technology to set the master computer as the MASTER of the entire distributed computing of the robot operating system ROS; Step 3-2: Create a matching node (10), subscribe to the feature point description information of the high-definition image feature point extraction released by the SURF algorithm node, and perform the best feature point matching. 5. the large field of view image stitching method based on bionic binoculars according to claim 1, is characterized in that, the concrete steps of the high-definition image affine stitching of described step ④ are as follows: Step 4-1: Create an affine splicing computing node on the host computer, subscribe to the best matching information of feature points issued by the matching node (10) and subscribe to the high-definition image data issued by the image acquisition node of CARMA DevKit (3, 4); Step 4-2: Find the homography matrix of the left and right high-definition images according to the best matching information of feature points, perform affine transformation, stitch them into one image, and perform data fusion.