CN114623818A - High-precision wearable helmet data acquisition equipment - Google Patents

Abstract

The invention provides high-precision wearable helmet data acquisition equipment, which integrates a camera, an IMU (inertial measurement unit) and a laser scanner into a helmet, performs hardware time synchronization, and acquires accurate sensor timestamps while acquiring data; the data acquisition process is realized by integrating all sensor data according to the hardware timestamp and dividing a data frame; removing motion distortion of laser observation in a data frame according to the observation of an inertial measurement unit, and giving color information of an image; dividing laser points in the data frame into three types of boundary characteristic points, surface characteristic points and no characteristic points according to the point cloud geometric curvature and the image gray gradient; and performing feature association on the continuous data frames according to different features, calculating relative positions and postures, and calculating coordinates of map points.

Description

High-precision wearable helmet data acquisition equipment

technical field

The invention relates to hardware design and data processing of a wearable helmet mobile measurement system, and belongs to the field of computer vision and the research field of laser point cloud measurement data processing automation.

Background technique

As a new type of basic surveying and mapping data, high-precision 3D point cloud has been widely used in highway, water conservancy, forestry, electric power and other industries. Mobile measurement technology is the main means to obtain high-precision 3D point cloud data. These methods can satisfy large-scale mapping tasks, but there are still problems such as poor accuracy in areas where GNSS signals are missing, difficult and time-sensitive operations in complex environments, and the overall size of the equipment is too large. Under the development trend of ubiquitous surveying and mapping, more and more industries need to apply mobile surveying equipment to areas where GNSS signals are missing or limited accessibility, and can operate quickly and easily to obtain high-precision data.

At present, there have been some researches on wearable mobile measurement systems at home and abroad, but they mainly focus on backpacks and handheld devices, such as the zebedee handheld mobile measurement system designed by Bosse et al. (2012), the star scout launched by Beike Tianhui Handheld mobile measurement system, LiBackPack backpack mobile measurement system launched by Digital Green Earth. The backpack or handheld mobile measurement method limits the observation angle of the mobile measurement equipment, and it is impossible to adjust the observation target in real time according to the observation angle of the operator. In addition, the existing real-time localization mapping (SLAM) methods (Shan and Englot, 2018; Zhang and Singh, 2014) are mostly used in vehicles, backpacks, and handheld platforms. Compared with wearing on the head, the motion of these platforms is more difficult to move. more gentle. In the process of walking, there are high dynamic motions such as turning and ups and downs, which brings huge challenges to the existing SLAM methods.

Related information and literature:

https://www.isurestar.com/starscan.html

https://www.lidar360.com/archives/6358.html

Bosse, M., Zlot, R., Flick, P., 2012. Zebedee: Design of a spring-mounted 3-drange sensor with application to mobile mapping. IEEE Transactions on Robotics 28, 1104-1119.

Shan, T., Englot, B., 2018. Lego-loam: Lightweight and ground-optimizedlidar odometry and mapping on variable terrain, 2018IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).IEEE, pp.4758-4765.

Shin, E.-H., El-Sheimy, N., 2004. An unscented Kalman filter for in-motion alignment of low-cost IMUs, Position Location and Navigation Symposium, 2004. PLANS 2004. IEEE, pp. 273-279.

Yang, Z., Shen, S., 2016. Monocular visual–inertial state estimation with online initialization and camera–imu extrinsic calibration. IEEE Transactionson Automation Science and Engineering 14, 39-51.

Zhang, J., Singh, S., 2014. LOAM: Lidar Odometry and Mapping in Real-time, Robotics: Science and Systems.

SUMMARY OF THE INVENTION

Based on the existing research, the present invention designs a novel wearable helmet movement measurement system.

The invention provides a high-precision wearable helmet data acquisition equipment, which integrates a camera, an IMU and a laser scanner into a helmet, performs hardware time synchronization, and collects accurate sensor time stamps while collecting data; the data acquisition process The implementation is as follows,

Step 1, according to the hardware timestamp, integrate all sensor data and divide the data frame;

Step 2, the laser observation in a data frame is based on the observation of the inertial measurement unit, the motion distortion is removed, and the color information of the image is given;

Step 3, according to the geometric curvature of the point cloud and the image grayscale gradient, the laser points in the data frame are divided into three categories: boundary feature points, surface feature points, and no feature points;

In step 4, the continuous data frames are characterized according to different characteristics, the relative position and attitude are calculated, and the coordinates of the map points are calculated. The implementation method includes the following processing:

The state quantities that need to be maintained in the helmet movement measurement are:

分别为位置、速度与姿态。

与

是IMU的加速度与角速度零偏；where x _k is the kth system state,

position, velocity, and attitude, respectively.

and

is the zero offset of the acceleration and angular velocity of the IMU;

The MEMS IMU model is used to model the system state x _k , and the pre-integration model is constructed according to the IMU state equation;

Laser matching constraints are divided into three categories: geometric boundary feature point constraints, geometric plane feature point constraints, and spectral boundary feature point constraints. The constraints are established as point-to-surface distances.

Moreover, in step 1, the implementation of integrating all sensor data is as follows: the control unit receives the 200Hz TOV signal sent by the IMU, records the signal reception time, and linearly interpolates the observations aligned with the system time, so as to realize the IMU time Synchronization; the control unit sends a 10Hz pulse to the camera, and records the pulse sending time, so as to realize the camera time synchronization; the control unit sends a 1Hz pulse to the lidar, and records the pulse sending time, so as to realize the lidar time synchronization.

Moreover, in step 2, according to the pre-integration principle, the IMU observation data within 0.1s is integrated, and then the laser point cloud data within 0.1s is projected into the initial time coordinate system of the data frame according to the integration result; The point cloud in the frame initial time coordinate system is projected into the camera plane coordinate system to obtain the corresponding image spectral information.

Moreover, in step 3, the geometric plane feature points and the geometric boundary feature points in the point cloud of a single frame are extracted according to the curvature. In addition, according to the spectral information obtained from the image for each point, the larger gray gradient is extracted. The points are used as spectral boundary feature points, and the remaining points are used as featureless points.

The mobile measurement equipment provided by the present invention can be applied to areas with missing GNSS signals or limited accessibility, can operate quickly and conveniently to obtain high-precision data, is convenient to use, has low cost, fills a market gap, and has important market value.

Description of drawings

Fig. 1 is the method flow chart of the embodiment of the present invention;

Fig. 2 is the hardware composition of the helmet movement measurement system of the embodiment of the present invention;

FIG. 3 is the multi-sensor synchronization principle of the helmet movement measurement system according to the embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be specifically described below with reference to the accompanying drawings and embodiments.

Based on the existing research, the present invention designs a novel wearable helmet movement measurement system. The system is divided into two parts: hardware and method flow: hardware design of wearable helmet movement measurement system, and high dynamic SLAM flow design. In the hardware design part, the camera, IMU, and laser scanner are integrated into a helmet, and hardware time synchronization is performed to collect accurate sensor timestamps while collecting data. High dynamic SLAM process design part: The first step is to integrate all sensor data and divide the data frame according to the hardware timestamp. Each frame of data contains 1 image, laser observations within 0.1s, and inertial measurements within 0.1s. In the second step, the laser observation within 0.1s in a data frame is based on the observation of the inertial measurement unit, the motion distortion is removed, and the color information of the image is given. The third step is to divide the laser points in the data frame into three categories: boundary feature points, surface feature points, and no feature points according to the geometric curvature of the point cloud and the image grayscale gradient. The fourth step is to associate the continuous data frames according to different features, calculate the relative position and attitude, and solve the coordinates of the map points.

The embodiment of the present invention provides high-precision wearable helmet data acquisition equipment, and the data processing flow is shown in Figure 1, including the following steps:

Step 1: Integrate all sensor data into data frames according to hardware timestamps.

The schematic diagram of hardware integration in the embodiment of the present invention is shown in Figure 2. The system includes three types of sensors: IMU, camera, and laser scanner (or lidar), and uses STM32 microcontroller as the control unit. The camera, IMU, and laser scanner are respectively Connect the control unit, select the type of sensor and control unit as shown in Table 1.

The time synchronization principle in the embodiment of the present invention is shown in FIG. 3 . The control unit receives the TOV (Time of Valid) signal of 200 Hz sent by the IMU, records the signal reception time, and linearly interpolates the observation value aligned with the system time, thereby realizing IMU time synchronization. The control unit sends 10Hz pulses to the camera, and records the time when the pulses are sent out, so as to achieve camera time synchronization. The control unit sends 1Hz pulses to the lidar and records the time when the pulses are sent out, thereby realizing the time synchronization of the lidar.

Table 1. Sensor and Control Unit Description

Step 2: The laser observation within 0.1s in a data frame is based on the observation of the inertial measurement unit, the motion distortion is removed, and the color information of the image is given. According to the pre-integration principle (Yang and Shen, 2016), the IMU observation data within 0.1s is integrated, and then the laser point cloud data within 0.1s is projected into the initial time coordinate system of the data frame according to the integration results. Then, the point cloud in the initial time coordinate system of the data frame is projected into the camera plane coordinate system to obtain the corresponding image spectral information.

Step 3: According to the geometric curvature of the point cloud and the image grayscale gradient, the laser points in the data frame are divided into three categories: boundary feature points, surface feature points, and no feature points. In the embodiment, preferably according to the method in (Zhang and Singh, 2014), the geometric plane feature points and the geometric boundary feature points in the single frame point cloud are extracted according to the curvature. In addition, according to the spectral information obtained from the image at each point, points with larger grayscale gradients are extracted as spectral boundary feature points. The remaining points are regarded as featureless points.

Step 4: Correlate the continuous data frames according to different features, calculate the relative position and attitude, and solve the map point coordinates. The state quantities that need to be maintained in the helmet movement measurement system are:

分别为IMU坐标系(body)每一帧相对世界坐标系(world)的位置、速度与姿态。

与

分别是IMU的加速度计(accelerometer)与陀螺仪(gyroscope)的零偏(bias)。本发明采用MEMS IMU模型(Shin and El-Sheimy,2004)对系统IMU状态进行建模：where x _k is the kth system state,

They are the position, velocity and attitude of each frame of the IMU coordinate system (body) relative to the world coordinate system (world).

and

They are the biases of the IMU's accelerometer and gyroscope, respectively. The present invention adopts the MEMS IMU model (Shin and El-Sheimy, 2004) to model the system IMU state:

与

是加速度计误差、陀螺仪误差、加速度随机游走与角速度随机游走，

为第k个系统状态时从IMU坐标系转到世界坐标系的旋转矩阵，

为

的四元数形式，g^w为世界坐标系下的重力向量，这样就可以对系统状态进行递推。 _where a _m and w _m are acceleration and angular velocity measurements, an , _wn ,

and

are the accelerometer error, gyroscope error, acceleration random walk and angular velocity random walk,

is the rotation matrix from the IMU coordinate system to the world coordinate system when it is the kth system state,

for

The quaternion form of , g ^w is the gravity vector in the world coordinate system, so that the system state can be recursive.

In order to use the laser matching information to correct the system state and suppress the system state divergence, the pre-integration model of the IMU state equation is constructed as follows:

与

分别为预积分相对位置、相对速度、相对旋转误差项，积分从t_k计算到t_k+1时刻，

为t时刻相对k帧的旋转矩阵。这样，所有的积分都是在IMU坐标系下从第k个系统状态开始积分，与世界坐标系没有关系，与起始系统状态没有关系，便于求导与计算残差，进而优化位姿。in,

and

are the relative position, relative velocity, and relative rotation error terms of pre-integration, respectively. The integration is calculated from t _k to t _k+1 time,

is the rotation matrix relative to k frames at time t. In this way, all the integrals start from the kth system state in the IMU coordinate system, which has nothing to do with the world coordinate system, and has nothing to do with the initial system state, which is convenient for derivation and residual calculation, and then optimize the pose.

Laser matching constraints are divided into three categories: geometric boundary feature point constraints, geometric plane feature point constraints, and spectral boundary feature point constraints. The constraints between spectral boundary points and geometric boundary points are established by the distance from point to line. The constraints of geometric surface feature points are established by the distance from the point to the surface. The formula is as (10). In the existing kth frame, p is the coordinate of each feature point. (n ^T , d) are parameters of a line or plane, where d is a constant. If it is a straight line, n ^T is the vector from the feature point to the midpoint of the straight line. If it is a plane, n ^T is the plane normal vector.

According to the above solution, in the embodiment of the present invention, a wearable helmet movement measurement system is first designed and integrated, and the time synchronization of the camera, the IMU, and the laser scanner is realized. After that, a high dynamic SLAM method flow design was designed for the wearable helmet mobile measurement system, and finally the real-time solution of the system position and attitude was realized, and the map point coordinates were solved. During specific implementation, computer software technology can be used to realize the automatic operation of the method process.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which the present invention pertains can make various modifications or additions to the described specific embodiments or substitute in similar manners, but will not deviate from the spirit of the present invention or go beyond the definition of the appended claims range.

Claims (4)

1. A high-precision wearable helmet data acquisition equipment, characterized in that: a camera, an IMU and a laser scanner are integrated into a helmet, and hardware time synchronization is performed to collect accurate sensor time stamps while collecting data; The acquisition process is implemented as follows, Step 1, according to the hardware timestamp, integrate all sensor data and divide the data frame; Step 2, the laser observation in a data frame is based on the observation of the inertial measurement unit, the motion distortion is removed, and the color information of the image is given; Step 3, according to the geometric curvature of the point cloud and the image grayscale gradient, the laser points in the data frame are divided into three categories: boundary feature points, surface feature points, and no feature points; In step 4, the continuous data frames are characterized according to different characteristics, the relative position and attitude are calculated, and the coordinates of the map points are calculated. The implementation method includes the following processing: The state quantities that need to be maintained in the helmet movement measurement are:

where x _k is the kth system state,

position, velocity, and attitude, respectively.

and

is the zero offset of the acceleration and angular velocity of the IMU; The MEMS IMU model is used to model the system state x _k , and the pre-integration model is constructed according to the IMU state equation; Laser matching constraints are divided into three categories: geometric boundary feature point constraints, geometric plane feature point constraints, and spectral boundary feature point constraints. The constraints are established as point-to-surface distances. 2. a kind of high-precision wearable helmet data acquisition equipment according to claim 1, is characterized in that: in step 1, the realization mode that all sensor data is integrated is, control unit receives the TOV signal of 200Hz that IMU sends, records Signal reception time, and linearly interpolate the observations aligned with the system time to achieve IMU time synchronization; the control unit sends 10Hz pulses to the camera, and records the pulse sending time to achieve camera time synchronization; the control unit sends 1Hz to the lidar Pulse, and record the time when the pulse is emitted, so as to realize the time synchronization of lidar. 3. A high-precision wearable helmet data acquisition equipment according to claim 1, wherein in step 2, according to the pre-integration principle, the IMU observation data within 0.1s is integrated, and then the laser within 0.1s is integrated. The point cloud data is projected into the initial time coordinate system of the data frame according to the integration result; the point cloud in the initial time coordinate system of the data frame is then projected into the camera plane coordinate system to obtain the corresponding image spectral information. 4. a kind of high-precision wearable helmet data acquisition equipment according to claim 1 or 2 or 3, is characterized in that: in step 3, according to the curvature to extract the geometric plane feature points and geometric boundary feature points in the single frame point cloud , in addition, according to the spectral information obtained from the image at each point, the points with larger grayscale gradients are extracted as the spectral boundary feature points, and the remaining points are regarded as non-feature points.

Images