CN111754566A - Robot scene positioning method and construction operation method - Google Patents

Abstract

The invention discloses a robot scene positioning method and a construction operation method. The invention relates to the field of robot control, wherein, the robot scene positioning method generates a grid map of the completed state of the building by extracting structural information according to BIM data, obtains the scanning data of the current scene from the lidar installed on the robot, and uses an adaptive Monte Carlo algorithm to scan the current scene. The data and the completed state grid map are used to locate the robot in the current scene to obtain the current positioning information of the robot. Based on the BIM model information as a priori information, the information about the size of the building structure is extracted from it, and converted into a completed state grid map that can be used for robot positioning. The completed map positioning method with redundant information can be used in the entire engineering cycle, simplifying the process. The positioning process improves the positioning efficiency of the robot in changing scenarios.

Description

Robot scene positioning method and construction operation method

technical field

The invention relates to the field of robot control, in particular to a robot scene positioning method and a construction operation method.

Background technique

The main principle of simultaneous localization and mapping (SLAM) is to detect the surrounding environment through sensors on the robot, and construct an environment map while estimating the pose (the pose includes position and orientation). Taking the visual SLAM currently widely used in indoor robot positioning as an example, after arriving at the construction site, it is necessary to manually control the robot to construct the visual feature map of the scene, especially when the site is large and the structure is complex, it will consume a lot of working time and manpower and material resources, reducing The autonomy of robots loses the significance of using robots to improve decoration automation. In addition, due to the particularity of building decoration work, there is such a large-scale interaction between the construction environment and the robot. With the progress of the construction process, the actual environment where the robot is located will continue to change, such as gradually building walls, and finally with the completed drawings. match etc. In the face of changing scenes, traditionally, it is necessary to constantly recreate the map to ensure the positioning accuracy of the robot in changing scenes, resulting in a decrease in positioning efficiency. Therefore, it is necessary to propose a robot scene localization method that can improve the robot's localization efficiency in changing scenes.

SUMMARY OF THE INVENTION

The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention proposes a robot scene positioning method, which can improve the positioning efficiency of the robot under changing scenes, and realizes a positioning method that does not require prior mapping and can be used in the entire engineering cycle.

In a first aspect, an embodiment of the present invention provides: a method for locating a robot scene, including:

Extracting structural information from BIM data to generate a raster diagram of the completed state of the building;

Obtain the scanning data of the current scene by the lidar installed on the robot;

Using an adaptive Monte Carlo algorithm to locate the robot in the current scene according to the scan data and the completed state grid map, the current positioning information of the robot is obtained.

Further, the use of the adaptive Monte Carlo algorithm to locate the robot in the current scene according to the scan data and the completed state grid map to obtain the current positioning information of the robot, including:

Obtain the engineering structure information contained in the scan data;

Update the likelihood field of each particle in the adaptive Monte Carlo algorithm in real time according to the engineering structure information;

Screen the particles using the likelihood field as a weight;

The current positioning information of the robot is obtained by positioning the robot in the current scene according to the screening result and the completed state grid map.

The embodiments of the present invention have at least the following beneficial effects: simplifying the positioning process and improving the positioning efficiency of the robot in changing scenarios

In the second aspect, an embodiment of the present invention provides: a construction operation method, comprising:

Obtain the current positioning information of the robot by using the robot scene positioning method according to any one of the first aspects;

Using the current positioning information to perform inter-frame pose tracking to obtain the estimated pose to be optimized;

Perform joint optimization on the estimated pose to be optimized to obtain a joint optimized pose;

The construction operation is performed according to the joint optimized pose and the three-dimensional position information obtained through feature extraction.

Further, performing the inter-frame pose tracking using the current positioning information as the initial pose to obtain the estimated pose to be optimized, including:

Obtain the current camera positioning information according to the transformation matrix between the robot and the camera and the current positioning information;

The camera positioning information is used as the initial pose to obtain the positioning difference between camera frames, and the estimated pose to be optimized is obtained by using the camera projection error and the positioning difference between the camera frames.

Further, the joint optimization is optimized by using a nonlinear optimization beam adjustment method.

Further, obtaining the camera projection error through feature matching, including:

Using a laser projector to perform environmental projection to obtain several projection spots;

Use a camera to obtain the projected light spot for ORB feature matching;

The camera projection error is obtained according to the feature matching result.

Further, it also includes using a neural network to perform operation recognition, determining the three-dimensional position information of the operation target according to the operation recognition result, and the robot performs construction operations according to the joint optimized pose and the three-dimensional position information.

The embodiments of the present invention have at least the following beneficial effects: using the current positioning information obtained from the BIM model information to perform joint optimization of the front-end fusion BIM information inter-frame pose tracking and the back-end fusion BIM information, and improve the accuracy of traditional visual SLAM.

In a third aspect, an embodiment of the present invention provides: a construction operating system, comprising:

Obtaining the current positioning information unit: used to obtain the current positioning information of the robot by using the robot scene positioning method according to any one of the first aspects;

Inter-frame pose tracking unit: used for using the current positioning information to perform inter-frame pose tracking to obtain the estimated pose to be optimized;

Joint optimization unit: used to jointly optimize the estimated pose to be optimized to obtain a joint optimized pose;

Construction operation unit: used for construction operation according to the joint optimized pose and the operation position information obtained through feature extraction.

In a fourth aspect, an embodiment of the present invention provides: an electronic device, characterized in that it includes:

at least one processor; and, a memory communicatively coupled to the at least one processor;

Wherein, the processor is configured to execute the method according to any one of the first aspect, or to execute the method according to any one of the second aspect, by calling the computer program stored in the memory.

In a fifth aspect, an embodiment of the present invention provides: a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, the computer-executable instructions are used to cause a computer to execute the first aspect The method of any one, or, for performing the method of any one of the second aspects.

The beneficial effects of the embodiments of the present invention are:

In the embodiment of the present invention, the completed state grid map of the building is generated by extracting structural information according to the BIM data, the scanning data of the current scene from the laser radar installed on the robot is obtained, and the self-adaptive Monte Carlo algorithm is used according to the scanning data and the completed state grid map. Position the robot in the current scene to obtain the current positioning information of the robot. There is no need to remotely control the robot to construct the visual feature map of the scene at the beginning of the project, avoid consuming a lot of working time and manpower and material resources when the site is large and complex in structure, and reduce the autonomy of the robot. Based on the BIM model information as a priori information, extract information about The size information of the building structure is converted into a completed state grid map that can be used for robot positioning. The completed map positioning method with redundant information can be used in the entire project cycle, which simplifies the positioning process and improves the positioning efficiency of the robot in changing scenarios. .

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort. In the attached image:

1 is a schematic flowchart of a specific embodiment of a robot scene positioning method in an embodiment of the present invention;

2 is a schematic diagram of a BIM model diagram and a completed state grid diagram of a specific embodiment of a robot scene positioning method in an embodiment of the present invention;

3 is a schematic diagram of laser scanning of a specific embodiment of a robot scene positioning method in an embodiment of the present invention;

4 is a schematic flow chart of a specific embodiment of the construction operation method in the embodiment of the present invention;

5 is a schematic diagram of inter-frame pose tracking according to a specific embodiment of the construction operation method in the embodiment of the present invention;

Fig. 6-Fig. 9 is the visual enhancement effect schematic diagram of a specific embodiment of the construction operation method in the embodiment of the present invention;

10 is a schematic diagram of reprojection error in the related art;

FIG. 11 is a schematic diagram of the inter-frame constraint reprojection error based on BIM information in a specific embodiment of the construction operation method in the embodiment of the present invention;

Figure 12 is a schematic diagram of two kinds of wall defects;

13 is a schematic diagram of the detection effect of the YOLO convolutional neural network according to a specific embodiment of the construction operation method in the embodiment of the present invention;

FIG. 14 is a structural block diagram of a specific embodiment of the construction operating system in the embodiment of the present invention.

Detailed ways

In order to more clearly describe the embodiments of the present invention or the technical solutions in the related art, the specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative efforts, and obtain other implementations.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terms used herein in the description of the present invention are for the purpose of describing specific embodiments only, and are not intended to limit the present invention.

The flowcharts shown in the figures are only exemplary illustrations and do not necessarily include all contents and operations/steps, nor do they have to be performed in the order described. For example, some operations/steps can be decomposed, and some operations/steps can be combined or partially combined, so the actual execution order may be changed according to the actual situation.

Example:

The working environment of construction robots is different from general indoor mobile robots. It mainly has the following three characteristics: 1. It can obtain the prior information of the working environment BIM (Building Information Modeling, referred to as BIM); 2. The working environment is mostly indoor weak texture rough wall 3. The working environment will undergo structural changes with the construction. The mapping and positioning process framework of traditional visual SLAM is inefficient and cannot work in a weak texture environment. In addition, when the environmental structure changes, the feature map created by traditional visual SLAM will not be available, and the map needs to be rebuilt, which is similar to the work of construction robots. There are irreconcilable contradictions in the scene.

In this embodiment, repeatable and high-precision positioning work is performed by pre-acquiring the prior information BIM of the building environment and combining binocular vision and 2D lidar.

An embodiment of the present invention provides a method for locating a robot scene. FIG. 1 is a schematic flowchart of the method for locating a robot scene according to an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:

S11: Extract the structural information from the BIM data to generate a grid diagram of the completed state of the building.

The full name of BIM is: Building Information Modeling. BIM integrates various information from the design, construction, operation of the building until the end of the whole life cycle of the building into a 3D model information database. , which provides a complete and actual construction engineering information base for this model. The database not only contains the geometric information, professional attributes and state information describing building components, but also the state information of non-component objects (such as space and motion behavior). Since BIM includes an information storage solution in the construction field and has rich building information, this embodiment extracts the available information in the BIM for the positioning requirements of the construction robot, and uses it as the prior information of the building environment.

In one embodiment, the BIM is stored in the standard exchange format IFC (IndustryFoundationClass). The information description of the IFC is divided into four levels: the resource layer (IFC-Resource Layer), the core layer (IFC-CoreLayer), and the shared layer (IFC). -Interoperability Layer) and domain layer (IFC-Domain Layer).

The resource layer is the basic layer of the entire system, and any layer of the IFC can reference entities in the resource layer. This layer mainly defines the general information of the engineering project, which is independent of the specific building and has no overall structure, but is scattered basic information. The core content of this layer mainly includes attribute resources, performance resources, and structure resources. These entity resources are mainly used for the definition of upper-level entity resources to display the attributes of the upper-level entity.

The core layer defines the overall framework of the information model, such as product, process, control and other related information. and abstract processes. It plays a linking role in the whole system. This layer abstracts and defines abstract concepts applicable to the entire construction industry, such as the IFCProduct entity can describe the construction site, construction space, construction components, etc. of a construction project.

The shared layer mainly serves the domain layer, enabling the information exchange between various domains, and at the same time refining the components of the system, such as specific building components such as slabs (IFCSlab), columns (IFCColumn), beams (IFCBeam), walls (IfcWall) All are defined at this level. If a building has 100 walls, there will be 100 instances of IfcWall in the neutral file.

The domain layer is the top layer of the IFC system architecture and mainly defines the entity types for each professional domain. These entities have specific concepts for each area of expertise. Such as boilers and pipes in the field of HVAC.

As shown in Figure 2, it is a schematic diagram of the BIM model diagram and the completed state grid diagram in an embodiment. It can be seen from the figure that the structure and size information in the project are extracted according to the inheritance relationship between the IFC entities in the BIM model diagram, and then Generates a raster map of the as-built state of the building at a preset resolution.

S12: Obtain the scanning data of the current scene by the lidar installed on the robot.

In one embodiment, a 2D lidar is installed on the robot to scan the construction scene in real time to obtain scan data on the site, where the scan data includes engineering structure information, such as load-bearing walls, partition walls, and the like.

S13 : using the adaptive Monte Carlo algorithm to locate the robot in the current scene according to the scan data and the completed state grid map to obtain the current positioning information of the robot.

In one embodiment, the implementation process of step S13 is as follows:

S131: Obtain the engineering structure information contained in the scanned data. According to the characteristics of environmental changes during the construction process, the initial state is only the load-bearing wall. As the construction progresses, the partition wall is built. When all the partition walls are built, the completion is achieved. The effect of the drawing.

S132: Update the likelihood domain of each particle in the adaptive Monte Carlo algorithm in real time according to the obstacle information.

S133: Screen the particles using the likelihood domain as a weight.

S134: Position the robot in the current scene according to the screening result and the completed state grid diagram to obtain the current positioning information of the robot.

For example, update the scan layer scan_layer, load the raster image of the completed state in the potential layer through the potential_map_server, and obtain all the engineering structure information in the lidar scan data of the current position in the scan layer scan_layer, such as obstacles such as clapboard walls. The particles are screened by calculating the likelihood field (that is, the above-mentioned probability) of each particle as the weight. The smaller the weight of the particle, the less consistent the observed data obtained based on it is with the actual map, so the particle is easier to be screened. Lose.

As shown in Figure 3, it is a schematic diagram of laser scanning in one embodiment. For all particles, if there is an invisible wall in the map (as shown by the dotted line in the figure), the laser will pass through the wall. If If the end can illuminate other walls, the weight can be calculated according to the likelihood domain of other walls (as shown on the left). If the end does not illuminate any wall, it will be discarded as an invalid value (as shown on the right). . When an object appears in both the scan layer scan_layer and the potential layer potential_map_server, it is considered to be a wall.

In one embodiment, the particle filter of the adaptive Monte Carlo algorithm is used to realize positioning. The Monte Carlo particle filter positioning can be regarded as the following process: for example, a group of particles are evenly distributed in the scene at the beginning, and the robot moves through the movement of the robot. For example, if the robot moves forward one meter, all the particles move forward one meter, and then use the position of each particle to simulate a sensor position information and compare the scanning data obtained by the robot through 2D lidar scanning in this example. Yes, so as to give each particle a probability, that is, the likelihood domain, and regenerate the particles according to the generated probability. Generally speaking, the higher the probability, the greater the generation probability, that is, the likelihood domain is weighted for screening, leaving For the particles with heavy weights, after many iterations, all the particles will slowly converge, and the robot will be positioned according to the result of the particle convergence to obtain the current positioning information of the robot.

In one embodiment, there are many ways to calculate the weight, and the weight can be simply considered as the difference between the measured value of the real sensor and the actual sensor. The larger the weight is, the smaller the weight indicates that it is far from the real position.

For example, the robot is initially located near a wall, but the robot does not know where it is. At this time, all particles are initialized, and the weights of all particles are the same. At this time, the robot knows that there is a wall in front of it through the laser scanning data, and the weight of the particles closer to the wall is increased at this time. After resampling, the robot moves, and it can be seen that the sampled particles are concentrated near the particle points that were originally close to the wall. At this time, the robot re-perceives and knows that there is still a wall in front of it. At this time, the originally sampled particles close to the wall The weight becomes larger, and the particles with greater weight are retained after screening. At this time, the main sampling points have been concentrated on the wall. The robot knows that its position is next to the wall through the screening results, and realizes the positioning of the robot.

Since any construction stage map is a subset of the map in the completed state, and the particle likelihood field of this embodiment can be updated in real time, the current positioning information of the robot can be obtained in real time through weight screening, so the completed state grid is used in this embodiment. Lattice can be used for navigation in different construction stages. It is not necessary to remotely control the robot to construct a visual feature map of the scene at the beginning of the project. This avoids consuming a lot of work time and manpower and material resources when the site area is large and the structure is complex, and reduces the autonomy of the robot. At the same time, based on the BIM model information as a priori information, this embodiment extracts information about the size of the building structure and converts it into a completed state grid map that can be used for robot positioning. The completed map positioning with redundant information can be used in the entire engineering cycle. The method simplifies the positioning process, can directly locate in the face of changing scenes, and improves the positioning efficiency of the robot in changing scenes.

In another embodiment, a construction operation method is provided, which is improved based on the traditional visual SLAM method. It solves the problem that traditional visual SLAM cannot meet the needs of efficient and high-precision positioning and mapping in the case of changing environments and weak texture interior decoration, and at the same time, through deep learning, the constructed SLAM map has semantic information.

The advantages and disadvantages of traditional laser SLAM and visual SLAM are very obvious: in the face of weak texture environment, the stability and accuracy of laser SLAM are better than traditional visual SLAM; while in perceiving high-dimensional environmental information (such as: identifying wall defect points) , 3D reconstruction, etc.), visual SLAM is better. However, the robustness of visual SLAM to environmental changes is poor, and it cannot be directly applied to building construction scenarios, nor can it work well in a decoration environment with weak interior wall textures. Therefore, this embodiment combines visual and 2D lidar fusion SLAM The construction operation method effectively combines the respective advantages of traditional laser SLAM and traditional visual SLAM.

As shown in Figure 4, it is a schematic flowchart of a construction operation method provided by an embodiment of the present invention, as shown in Figure 1, the method includes the following steps:

S21: Obtain the current positioning information of the robot by using any one of the robot scene positioning methods described in the foregoing embodiments, that is, obtain the current positioning information of the robot through the above-mentioned embodiments.

S22 : Using the current positioning information to perform inter-frame pose tracking to obtain the estimated pose to be optimized, the inter-frame pose tracking in this embodiment may be regarded as inter-frame pose tracking in which BIM information is fused at the front end.

In traditional visual SLAM, its front end passes through a uniform motion model. This model considers that the camera is always moving at a constant speed, so the transformation matrix of the k+1th frame relative to the kth frame and the transformation matrix of the kth frame relative to the k-1th frame The same, but this assumption obviously does not meet the working requirements of the construction robot, so in this embodiment, the uniform velocity model is improved by fusing BIM information.

S23: Perform joint optimization on the estimated pose to be optimized to obtain a joint optimized pose. For example, in one embodiment, joint optimization is performed using a nonlinear optimized beam adjustment method, which is also called a BA method.

S24: The construction operation is performed according to the joint optimized pose and the three-dimensional position information obtained by feature extraction, wherein the three-dimensional position information is a kind of semantic information, and the map is constructed according to the semantic information, and the operated target is identified, for example, when the wall is repaired, the wall The three-dimensional position information of surface defects, the construction operation is to repair wall defects for the robot, etc., and the application scope of the robot is expanded.

In an embodiment, the specific process of the inter-frame pose tracking of the front-end fusion BIM information in step S22 is as follows, including:

S221: Obtain the current camera positioning information according to the transformation matrix between the robot and the camera and the current positioning information, which is expressed as:

T _camera = T _{current robot position} * T _{robot - camera}

Among them, T _camera represents the current camera positioning information, T _{current robot positioning} represents the current positioning information of the robot, T _robot-camera represents the transformation matrix between the robot and the camera, and the transformation matrix can be obtained as a priori parameter.

S222 : using the camera positioning information as the initial pose to obtain the positioning difference between camera frames, and using the camera projection error and the positioning difference between the camera frames to obtain the estimated pose to be optimized.

In one embodiment, the current camera positioning information is used as the initial pose to replace the initial pose of the uniform motion model in the related art, and then based on the optimized camera frame positioning difference from the previous frame to the current frame, combined with The projection error of the 3D feature points observed in the current frame to the current camera.

As shown in FIG. 5 , a schematic diagram of the inter-frame pose tracking of this embodiment, where P is a three-dimensional space point, and P1 and P2 are three-dimensional space points in the kth frame (previous frame that has been optimized) and the kth point of the camera, respectively. The observation matching point pair of 1 frame (the current frame to be optimized), that is, the observation matching point pair is obtained through feature matching, P2' is the projection point of the three-dimensional space point P on the camera in the k+1th frame, and the e _projection represents the camera. The camera plane distance error between the observation point P2 of the current frame and the projection point P2', that is, the above-mentioned camera projection error, e _BIM represents the positioning difference between the camera frames, that is, the estimated pose of the current frame of the camera to be optimized and obtained through the BIM information. The error between the initial camera poses (the camera positioning information obtained in step S221).

The estimated pose to be optimized belongs to inter-frame tracking, and the current frame is optimized through observation, which is expressed as:

Among them, ζ* represents the estimated pose to be optimized, that is, the Lie algebra of the camera pose, u represents the pixel coordinates of the observation point P2 in the k+1th frame of the camera, and Kexp(ζ^)P represents the three-dimensional space point P in the camera parameter K. In this case, the frame plane projection of the k+1th frame of the camera, its projection value is represented as P2', _{Tw_curr} represents the estimated pose to be optimized of the k+1th frame of the camera, and Tw_BIM represents the k+ _1th obtained through the BIM information Frame camera positioning information, that is, the initial pose, e _projection represents the camera plane distance error between the observation point P2 and the projection point P2' of the k+1th frame of the camera, that is, the above camera projection error, e _BIM represents the camera inter-frame positioning difference , that is, the error between the estimated pose of the camera's current frame to be optimized and the initial pose of the camera obtained through the BIM information.

In one embodiment, the camera projection error is obtained through feature matching, including:

S2221: Use a laser projector to perform environmental projection to obtain several projection spots;

S2222: Use the camera to obtain the projected light spot for ORB feature matching;

S2223: Obtain the camera projection error according to the feature matching result.

Due to the lack of texture features on the walls in the decoration environment, it is impossible to extract visual features for feature matching at this time. Therefore, in this embodiment, the visual feature enhancement of artificial projection is performed for this situation. The specific method is to use a laser projector to form a dot matrix to perform environmental projection on a wall lacking visual features to obtain several projection spots, and capture these artificially projected projection spots through a camera device, such as a binocular camera, to extract ORB features for ORB feature matching. The feature matching result obtains the camera projection error according to the above method, so as to calculate the estimated pose to be optimized.

In an embodiment, the purpose of setting the laser projector dot matrix is to meet the requirements for the placement of visual features of the entire site when some construction scenes have large walls and a large scene range. Therefore, an omnidirectional laser dot matrix projection device is designed, such as The device is set up as 8 laser dot matrix projectors, which are evenly distributed on a 360-degree circular base. Each laser projector is equipped with a rotating base with 2 degrees of freedom, which can respectively realize rotation and rotation perpendicular to the horizontal direction. Pitch to meet the projection needs of different housing structures. At the same time, a tripod can be installed at the bottom to increase the height of the projection point and prevent debris from blocking the light path.

As shown in FIGS. 6-9 , it is a schematic diagram of a visual enhancement effect in an implementation manner of this embodiment. Figure 6 shows the visual features extracted from the indoor wall, and Figure 7 shows the features successfully matched between the two images. It can be found that, because the wall is relatively flat and the texture features are sparse, the feature points that can be successfully matched between adjacent frames rare. When the wall that needs to be operated in the indoor painting process is a white wall, the visual features that can be extracted will be more sparse. FIG. 8 is a schematic diagram of a laser projector spot, and FIG. 9 is a feature matching situation after visual feature enhancement using a laser projector. It can be seen that after laser projection, the ORB features that can be extracted are significantly increased. Even on a completely blank wall, feature matching can be achieved, and positioning and mapping in a weak texture decoration environment can be achieved. According to the feature matching results, the above method can be obtained. The camera projection error is used to calculate the estimated pose to be optimized, and finally meet the requirements of the construction robot construction operating system.

The above process of this embodiment completes the frame-to-frame pose tracking of the front-end fusion of BIM information. The motion of the camera can be estimated through the detection images between adjacent frames, and the motions at adjacent moments are connected in series to form the motion trajectory of the robot. achieve positioning. On the other hand, according to the camera pose at each moment, the position of the three-dimensional space point corresponding to each pixel can be calculated, and the scene map can be obtained.

Since any measurement data has noise, the above-mentioned front-end inter-frame pose tracking with BIM information fusion can estimate the motion of the camera from the image, but it is necessary to reduce the influence of noise in the estimation process for back-end optimization, otherwise cumulative drift will inevitably occur. , the drift will cause the inability to establish a consistent map, so the estimated pose to be optimized at the front end and the initial values of these data are sent to the back end for overall optimization. Generally speaking, the optimization method is a filtering or nonlinear optimization algorithm.

In this embodiment, step S23 performs joint optimization on the estimated pose to be optimized, and the joint optimization in the joint optimization pose is obtained, that is, the above-mentioned back-end optimization process. In one embodiment, the nonlinear optimization beam adjustment method is used for joint optimization. Bundle Adjustment, also known as BA method, is a nonlinear optimization method.

In the related art, the back-end optimization mostly uses the reprojection error method. This method projects the coordinates of the three-dimensional feature points corresponding to the pixel coordinates to the adjacent frame according to the estimated pose, and the pixel coordinates obtained on the adjacent frame are matched with the pixels obtained by the feature. error between coordinates.

As shown in Figure 10, it is a schematic diagram of the reprojection error in the related art, where P is a three-dimensional space point, P1 and P2 are the observation matching point pairs of the three-dimensional space points in the kth frame and k+1 frame of the camera, respectively, and P2' is The projection point of the three-dimensional space point P in the k+1th frame of the camera, and e is the camera plane distance error between the observation point P2 and the projection point P2' in the k+1th frame of the camera. The solution is solved using the nonlinear optimal beam adjustment method BA.

However, the optimization of the existing nonlinear optimized beam adjustment method BA relies too much on the initial pose of the camera. Due to the inaccuracy of the initial pose, the optimization process of the nonlinear optimized beam adjustment method BA requires multiple iterations to achieve sufficient accuracy. The required iteration period is longer, and better optimization results cannot be obtained. Therefore, this embodiment improves the reprojection error method, uses the fusion BIM information, and uses the 2D lidar to perform pre-positioning to obtain the camera positioning information as the initial pose of the current frame of the camera to implement inter-frame constraints, which can reduce the number of optimization iterations. Get more accurate optimization results in cycles.

As shown in FIG. 11 , a schematic diagram of the inter-frame constraint reprojection error based on BIM information in this embodiment is shown. Among them, P is the three-dimensional space point, P1 and P2 are the observation matching point pair of the three-dimensional space point in the kth frame and the k+1th frame of the camera respectively, that is, the observation matching point pair is obtained through feature matching, and P2' is the three-dimensional space point. P is the projection point of the k+1th frame of the camera, e _projection represents the camera plane distance error between the observation point P2 of the camera k+1 frame and the projection point P2', that is, the above-mentioned camera projection error, e _BIM represents the camera inter-frame positioning The difference is the error between the estimated pose of the camera's current frame to be optimized and the initial pose of the camera obtained from the BIM information.

The joint optimized pose after joint optimization belongs to the inter-frame optimization, so it is necessary to traverse several co-view frames of the current frame for joint optimization, which is expressed as:

表示遍历第k+1帧之前的帧，例如取20个共视帧，即k从1遍历到20，u表示相机第k+1帧中观测点P2的像素坐标，Kexp(ζ^{^})P表示三维空间点P在相机参数K的情况下对相机第k+1帧的帧平面投影，其投影值表示为P2’,T_{w_k+1}表示相机第k+1帧的待优化估计位姿，T_{w_BIM1}表示通过BIM信息获得的第k+1帧相机定位信息，即初始位姿，e_投影表示相机第k+1帧的观测点P2与投影点P2’之间的相机平面距离误差，即上述相机投影误差，e_BIM表示相机帧间定位差，即相机第k+1帧的待优化估计位姿和通过BIM信息获得的相机初始位姿之间的误差。Among them, ζ* ₁ represents the joint optimization pose, which is a Lie algebra,

Indicates to traverse the frames before the k+1th frame, for example, take 20 common view frames, that is, k traverses from 1 to 20, u represents the pixel coordinates of the observation point P2 in the k+1th frame of the camera, and Kexp(ζ ^{^} )P represents The three-dimensional space point P projects the frame plane of the k+1th frame of the camera under the condition of the camera parameter K, and its projection value is expressed as P2', _{Tw_k+1} represents the estimated pose to be optimized of the k+1th frame of the camera, T _{w_BIM1} represents the camera positioning information of the k+1th frame obtained through the BIM information, that is, the initial pose, and e _projection represents the camera plane distance error between the observation point P2 and the projection point P2' of the k+1th frame of the camera, that is, the above camera Projection error, e _BIM represents the positioning difference between camera frames, that is, the error between the camera's k+1th frame's estimated pose to be optimized and the camera's initial pose obtained through BIM information.

In the process of obtaining the joint optimization pose above, in the graph optimization nonlinear optimization beam adjustment method BA, the constraint edge between the kth frame of the camera and the k+1th frame of the camera is added through the injection of BIM information. Compared with related technologies In the medium nonlinear optimization, only the edge constraints of the camera and the landmark three-dimensional space point are concerned, which improves the optimization effect and reduces the number of optimization iterations.

In one embodiment, step S24 performs the construction operation according to the joint optimized pose and the three-dimensional position information obtained by feature extraction, wherein the three-dimensional position information is a kind of semantic information, and the map is constructed according to the semantic information to identify the operated target, such as When the wall is repaired, the three-dimensional position information of the wall defect expands the application scope of the robot.

For example, the neural network is used for operation recognition, the three-dimensional position information of the operation target is determined according to the operation recognition result, and the robot performs construction operations according to the three-dimensional position information. Its own position, the position of objects in the environment, it is also necessary to identify the object to be operated, such as the convex hull that needs to be polished on the wall, etc., and even some common objects on the construction site, such as ladders, buckets, sandbags, etc. need to be identified, and These operation objects are located, and corresponding construction operations are carried out according to the operation objects.

Taking wall repair as an example, repair of wall defects is a major process in decoration construction. This task requires robots to identify wall defects and determine their positions, so as to achieve operations such as grinding and plastering of defects. In this embodiment, target detection is performed through the YOLO convolutional neural network to realize the identification of wall defects, and the ORB feature corresponding to the defect is determined according to the defect identification result to determine the three-dimensional position information of the defect in world coordinates, and the repaired defect is indicated on the map. The three-dimensional position information of the operation target is registered and registered in the visual feature map, which is convenient for the robot to interact with the environment for construction operations, so as to meet the work needs of the construction robot.

As shown in Figure 12, it is a schematic diagram of two kinds of wall defects, namely pit defects and convex pit defects, which are common wall defects. In one embodiment, model training is performed by collecting several photos at the decoration site and manually annotating a data set. For example, 500 photos are collected as samples, of which 420 are used for the training data set and 80 are used for the test data. Set, through the software environment: Ubuntu+Darknet+Cuda+Cudnn to conduct model training to obtain appropriate model parameters.

As shown in FIG. 13 , a schematic diagram of the detection effect of the YOLO convolutional neural network in this embodiment is shown. The upper diagram is a schematic diagram of a collected wall defect, which includes two convex pit defects and one pit defect. A schematic diagram of the detection results after the YOLO convolutional neural network detection. As can be seen from the figure, the three defects in the left picture were identified after detection.

The above-mentioned deep learning-based YOLO convolutional neural network establishes semantic information about defects, identifies the operated target, and labels the operation target into the feature map created by visual SLAM to realize an operable semantic map.

In one embodiment, it also includes performing loopback detection on the joint optimized pose of the back-end to solve the problem of position estimation drifting over time. Loopback detection is also called closed-loop detection, which is essentially an algorithm for detecting the similarity of observed data. It refers to the ability of the robot to recognize the scene that has reached the scene. If the detection loopback is successful, the detection information is provided to the backend for processing. The backend adjusts the trajectory and map to match the loopback detection result based on this information, and optimizes the correction of the accumulated error. Eliminating redundant map points, optimizing the essential map, and updating all map points can significantly reduce the cumulative error, achieve global pose consistency, and obtain globally consistent trajectories and maps.

In this embodiment, based on BIM model information as a priori information, information about the size of the building structure is extracted from it, and converted into a completed state grid map that can be used for robot positioning, and a completed map positioning method with redundant information can be used in the entire engineering cycle. , simplifies the positioning process, and can directly locate in the face of changing scenarios, improving the positioning efficiency of the robot in changing scenarios. Then in the construction operation, in view of the problem that traditional visual SLAM cannot perform mapping and positioning in a weak texture environment, the visual features of the wall are increased by setting a laser projector lattice for manual projection, which solves the problem that the visual SLAM front end cannot work in a weak texture environment. It uses YOLO convolutional neural network to detect operable objects on the wall, integrates operable objects into visual SLAM feature maps, and constructs semantic maps that contain three-dimensional location information of operations. At the same time, the current positioning information obtained from the BIM model information is used to perform the joint optimization of the front-end fusion BIM information inter-frame pose tracking and the back-end fusion BIM information to improve the accuracy of traditional visual SLAM.

In another embodiment of the present disclosure, a construction operating system is provided for executing the above-mentioned construction operation method. As shown in FIG. 14 , a structural block diagram of the construction operating system in this embodiment includes:

Obtaining the current positioning information unit 100: used to obtain the current positioning information of the robot by using the robot scene positioning method as described above;

Inter-frame pose tracking unit 200: used for performing inter-frame pose tracking using current positioning information to obtain an estimated pose to be optimized;

Joint optimization unit 300: used to jointly optimize the estimated pose to be optimized to obtain a joint optimized pose;

Construction operation unit 400: used to perform construction operations according to the joint optimized pose and the operation position information obtained through feature extraction.

The specific details of each unit module in the above construction operating system have been described in detail in the construction operation method corresponding to the above embodiment, and thus are not repeated here.

In addition, an embodiment of the present invention also provides an electronic device, including:

at least one processor, and a memory communicatively coupled to the at least one processor;

Wherein, the processor is configured to execute the method according to the first embodiment by calling the computer program stored in the memory. A computer program is a program code. When the program code runs on the device, the program code is used to make the device execute the steps in the robot scene positioning method or the construction operation method described in the above-mentioned embodiments of this specification.

In addition, the present invention also provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are used to make the computer execute the robot scene positioning method or construction described in the above-mentioned embodiments of this specification. steps in how-to.

Without loss of generality, the computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid state storage technology, CD-ROM, DVD or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art know that the computer storage medium is not limited to the above-mentioned ones.

It should be noted that: the above-mentioned order of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. And the foregoing describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited may be performed in a different order than in the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, storage medium and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for related parts.

The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it is still possible to implement the foregoing embodiments. The technical solutions described in the examples are modified, or some or all of the technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and all of them should cover within the scope of the claims and description of the invention.

Claims (10)

1. A robot scene positioning method, comprising:

extracting structural information according to BIM data to generate a complete state grid diagram of the building;

acquiring scanning data of a laser radar installed on the robot on a current scene;

and positioning the robot in the current scene by using a self-adaptive Monte Carlo algorithm according to the scanning data and the completion state grid diagram to obtain the current positioning information of the robot.

2. The method as claimed in claim 1, wherein the using an adaptive monte carlo algorithm to locate the robot in the current scene according to the scan data and the completion status raster map to obtain the current location information of the robot includes:

acquiring engineering structure information contained in the scanning data;

updating the likelihood domain of each particle in the self-adaptive Monte Carlo algorithm in real time according to the engineering structure information;

screening the particles by taking the likelihood domain as weight;

and positioning the robot in the current scene according to the screening result and the completion state grid diagram to obtain the current positioning information of the robot.

3. A construction operation method, characterized by comprising:

obtaining current positioning information of the robot by using the robot scene positioning method according to any one of claims 1 or 2;

tracking the pose between frames by using the current positioning information to obtain an estimated pose to be optimized;

performing joint optimization on the estimated pose to be optimized to obtain a joint optimization pose;

and performing construction operation according to the combined optimization pose and the three-dimensional position information obtained by feature extraction.

4. The construction operation method according to claim 3, wherein the inter-frame pose tracking of the current positioning information as an initial pose to obtain an estimated pose to be optimized comprises:

obtaining current camera positioning information according to a transformation matrix between the robot and the camera and the current positioning information;

and obtaining camera inter-frame positioning difference by taking the camera positioning information as an initial pose, and obtaining the pose to be optimized and estimated by using camera projection errors and the camera inter-frame positioning difference.

5. A construction operation method according to claim 3, wherein said joint optimization is an optimization using a nonlinear optimization beam-balancing method.

6. The construction operation method according to claim 4, wherein obtaining the camera projection error through feature matching comprises:

carrying out environmental projection by using a laser projector to obtain a plurality of projection light spots;

utilizing a camera device to obtain the projection light spot for ORB feature matching;

and obtaining the camera projection error according to the feature matching result.

7. The construction operation method according to claim 3, further comprising performing operation recognition using a neural network, determining three-dimensional position information of an operation target according to the operation recognition result, and performing construction operation by the robot according to the joint optimization pose and the three-dimensional position information.

8. A construction work system, comprising:

acquiring a current positioning information unit: for obtaining current positioning information of a robot using a robot scene positioning method according to any of claims 1 or 2;

an inter-frame pose tracking unit: the current positioning information is used for tracking the pose between frames to obtain an estimated pose to be optimized;

a joint optimization unit: the system is used for carrying out joint optimization on the estimated pose to be optimized to obtain a joint optimization pose;

a construction operation unit: and the construction operation is carried out according to the joint optimization pose and the operation position information obtained by feature extraction.

9. An electronic device, comprising:

at least one processor; and a memory communicatively coupled to the at least one processor;

wherein the processor is adapted to perform the method of any one of claims 1 or 2, or to perform the method of any one of claims 3 to 7, by invoking a computer program stored in the memory.

10. A computer-readable storage medium having stored thereon computer-executable instructions for causing a computer to perform the method of any one of claims 1 or 2, or for performing the method of any one of claims 3 to 7.

Images