CN116300905A - A Constrained Multi-robot Reinforcement Learning Safe Formation Method Based on 2D Laser Observation - Google Patents

Abstract

The invention discloses a two-dimensional laser observation-based multi-robot reinforcement learning safety formation method. Each formation member generates a preliminary control instruction according to individual observation and intra-formation sharing information by using a navigation planning method based on reinforcement learning, and the preliminary control instruction is further optimized under the guidance of a control barrier function, so that the final control instruction meets safety constraint, and the running safety of the robot is ensured. According to the invention, under the condition of remarkably small observation and calculation cost, the multi-robot information is integrated, the multi-robot control is planned, the safe and efficient multi-robot formation collaborative planning is realized, the formation control instruction generation is realized by only utilizing the two-dimensional laser information, the execution safety is ensured, and the large-scale formation task under the unknown dynamic unstructured environment can be dealt with.

Description

A Constrained Multi-robot Reinforcement Learning Safe Formation Based on 2D Laser Observation method

technical field

The invention belongs to the field of formation navigation of mobile robots, and in particular relates to a safety formation method for multi-robot reinforcement learning with constraints based on two-dimensional laser observation.

Background technique

In recent years, with the continuous development of robot technology, the working environment of robots has gradually migrated from static structured scenes to dynamic unstructured scenes, and the work tasks have gradually transitioned from single-machine mobile operations to multi-machine collaborative operations. In the context of the current rapid development of robot technology, the problem of multi-robot formation control that takes into account lightweight, flexibility and safety needs to be solved urgently.

Most of the existing multi-agent formation navigation planning methods are centralized global planning methods from an omniscient perspective, relying on complete offline map construction, online robot positioning and fast formation planning solution, the method structure is bloated, and it is mostly suitable for static simple environments or Small-scale formation tasks in structured scenarios are difficult to migrate to large-scale formation tasks in complex dynamic unstructured environments. As the environment complexity increases and the number of robots increases, the effectiveness and real-time performance of the method will be difficult to be guaranteed.

In recent years, the multi-agent planning method based on deep learning adopts an end-to-end method to enhance the adaptability of the method, which can realize large-scale parallel computing and simultaneous planning for multi-agents, and has achieved remarkable results, but generally does not have explicit It is difficult to be popularized in practical applications.

Contents of the invention

The technical problem to be solved by the present invention is to solve the problem of navigation planning with safety constraints based on original observation information for large-scale formations in complex dynamic unstructured environments, and provides a multi-robot reinforcement learning with constraints based on two-dimensional laser observations In the safe formation method, the formation member robots can realize information sharing and environment perception only through sensor observation information communication with a small amount of data, and carry out scalable formation distribution planning on the basis of comprehensive environment perception, and finally through flexible navigation strategy Generate actual execution instructions with the controller with safety constraints to ensure the efficiency and safety of formation navigation tasks. The technical scheme adopted in the present invention is as follows:

The invention discloses a two-dimensional laser observation-based safety formation method for multi-robot reinforcement learning with constraints, including:

S1: Obtain formation task setting parameters, including formation formation, number of formation members and formation starting position, initialize according to the parameters, and obtain preliminary state perception information;

S2: Through the internal communication of the multi-robot formation, share the state perception information in real time, and obtain the state perception information obtained by the other robots in the formation through sensor observation, including two-dimensional laser observation information, speed state information and relative positional relationship of the target;

S3: Decouple and integrate multi-robot state perception information, obtain the environment distribution state based on the target position of the formation center, and establish an observation potential field according to the target position of the formation center, the positions of each formation member, and the distribution of obstacles;

S4: According to the observed potential field and the setting parameters of the formation task, split the formation level and calculate the formation target position within each layer, and allocate the target position according to the current formation member position;

S5: Each formation member robot generates an initial control instruction from the policy network obtained based on reinforcement learning training according to the assigned target position and comprehensive state perception information;

S6: According to the control obstacle function describing the safety state of the robot, combined with the observation prediction model and the gradient optimizer, iteratively optimize the initial control instruction until the safety constraints are met, and obtain the final control instruction that guarantees safety;

S7: Send the final control command to the robot controller for execution, reach a new state, and return to S2 until the formation reaches the final target position.

As a further improvement, in step S1, the present invention acquires formation task setting parameters, including formation formation, number of formation members and formation starting position, specifically:

This method supports multiple formation schemes including full encirclement formation, half encirclement formation and arbitrary angle envelopment, and can be extended to a large number of robot formation tasks.

As a further improvement, in step S2 of the present invention, the state perception information includes two-dimensional laser observation information, speed state information and relative positional relationship of the target, specifically:

The two-dimensional laser information can be determined as a fixed-length one-dimensional array information according to the resolution of the sensor, and the speed state information and the relative position information of the target can be determined as two-dimensional coordinate information. The integrated state perception information is determined by an array with a limited size, the data volume is small, the communication load occupies less, and the expansion of multiple robots can be realized.

As a further improvement, in step S3, the present invention decouples and integrates the multi-robot state perception information, obtains the environment distribution state based on the target position of the formation center, and according to the target position of the formation center, the positions of each formation member and the distribution of obstacles and other information to establish the observation potential field, specifically:

According to the target position, the distribution of obstacles and the positions of other formation members, the observation potential field is established, and the execution cost of each position as the target position of formation members under the current environment state is described. Among them, the specific value of the cost of each position in the potential field is determined by the resultant force F of the obstacle repulsion F _ro received by the current position, the formation teammate repulsion F _ra , and the target attraction F _a . The specific relationship is:

F＝F _ro +F _ra +F _a

As a further improvement, in step S4 of the present invention, according to the observed potential field and formation task setting parameters, formation hierarchy splitting and formation target position calculation within each layer are performed, and target position allocation is performed according to the current formation member positions, Specifically:

The formations are layered according to the observation potential field and task setting, and the overall shape of each layered formation is a nested scale-enlarged relationship to ensure the overall order of the formation and the safety of each formation member within the formation. Inside each layer, the method first determines the lowest position of the potential field as the position of the current formation member, then updates the overall environmental potential field according to the repulsive potential field of the member, and finally repeats the cycle iteratively to gradually find the formation members in the layer s position. After determining the position of the target formation, the method assigns the target according to the principle of distance priority and avoiding track interlacing, combined with the current position of each formation member.

As a further improvement, in step S5 of the present invention, according to the assigned target position and comprehensive state perception information, the policy network obtained based on reinforcement learning training generates initial control instructions, specifically:

The strategy network learns through autonomous trial and error in the simulation environment, and iteratively optimizes iteratively according to the reward and punishment information fed back from the environment until it converges to an autonomous navigation strategy with relatively stable performance. The policy network can batch process large-scale multi-robot formation decision-making tasks, with higher efficiency and stronger scalability.

As a further improvement, in step S6 of the present invention, according to the control obstacle function describing the safety state of the robot, combined with the observation prediction model and the gradient optimizer, the initial control instruction is iteratively optimized until the safety constraints are satisfied, and the final control instruction that guarantees safety is obtained. ,Specifically:

According to the control obstacle function based on two-dimensional laser observation, the safety evaluation of the initial command is carried out, and the gradient direction of the command optimization is judged by combining the observation prediction model, and the gradient optimizer is used for iterative optimization, and finally the final control command that ensures safety is obtained.

The observation-based control obstacle function describes the safety degree of the current robot state in the environment, including two parts: the robot's own safety and the internal safety of the formation. The safety of the robot itself mainly describes the safety relationship between the robot and static or dynamic obstacles inside the environment when the robot is moving at the current speed state, and the internal safety of the formation mainly describes the safety between the robot and other members of the formation when it is moving at the current speed state relation.

The present invention has following beneficial effect:

1) Lightweight. The present invention shares environment perception information through internal communication of formations with a small amount of data, uses a decision network based on reinforcement learning to perform batch processing on local decision-making tasks of larger formations, and improves method efficiency from communication and information processing levels.

2) Flexibility. The present invention expresses the dynamic environment by establishing an observed potential field, assigns targets to any number of formation members in an iterative update manner, and uses a deep network with batch processing capabilities to help any number of formation members make decisions. Raw sensor information such as 3D LiDAR data is used as input to the method, so that the method can be flexibly migrated in complex real environments.

3) Security. The present invention uses the safety control obstacle function based on two-dimensional laser observation as a control instruction constraint, and performs safety evaluation and optimization on the policy network output based on reinforcement learning before the instruction is executed, until it meets the safety constraints and sends it to the controller to ensure the robot's operation. safety.

Therefore, the present invention is a very practical and effective multi-robot formation navigation method and has good application prospects.

Description of drawings

Figure 1 is the experimental frame diagram of the safety formation method of reinforcement learning with constrained multi-robots based on two-dimensional laser observation;

Figure 2 is an information flow diagram of a multi-robot reinforcement learning safety formation method with constraints based on two-dimensional laser observation;

Fig. 3 is a flowchart of a method for safe formation of multi-robot reinforcement learning with constraints based on two-dimensional laser observation.

Detailed ways

The invention discloses a two-dimensional laser observation-based safety formation method with constrained multi-robot reinforcement learning. The specific scheme is as follows:

First of all, before the task starts, the user needs to determine the formation formation, the number of formation members, initialize the starting position of each formation member and the final target position of the formation according to the task setting, perform parameter initialization settings on each formation member robot, and Position and activate multiple robots in simulation or real-world scenarios. After the task is executed, the robots of each formation member share the perception information through internal communication in the multi-agent formation, and obtain the state information of the remaining agents, including two-dimensional laser observation information, speed state information and relative positional relationship of the target. Subsequently, each formation member robot decouples and integrates the multi-agent state information, obtains the environment distribution state based on the target position of the formation center, and establishes an observation potential field based on the target position of the formation center, the position of each formation member, and the distribution of obstacles. . According to the distribution of the observed potential field, the selected formation and the number of formation members, each formation member robot splits the formation level and calculates the formation target position within each layer, and according to the current formation member position according to the unified rules Make target assignments. Subsequently, each formation member robot generates an initial control instruction by a strategy network based on reinforcement learning training based on the assigned target position, two-dimensional laser observation information, and speed state information. According to the control obstacle function that describes the safety state of the robot, the robots of each formation member combine the observation prediction model and the gradient optimizer to iteratively optimize the initial control instructions until the safety constraints are satisfied, and the final control instructions that guarantee safety are issued to the robot controller for execution , reaching a new state. Each formation member robot continues to execute the current process until the formation reaches the final target position and completes the formation navigation task.

Fig. 1 is an execution frame diagram of the method of the present invention, Fig. 2 is an information flow diagram of the method of the present invention, Fig. 3 is an overall flow chart of the method of the present invention, the specific execution steps of the scheme are as follows.

(1) The user specifies the formation formation, the number of formation members, and the starting position of the formation before the multi-robot formation operation according to the task requirements, and initializes the parameters on each formation member robot, and performs multi-robot operation in the simulation or actual scene. Placement and start-up are carried out, and then the multi-robot formation performs mobile operations under the specified task parameter settings.

In terms of formation formation, this method supports multiple formation formation schemes including but not limited to full-encirclement formation and semi-encirclement formation, and can be extended to a larger number of multi-robot formation tasks. The starting position of the formation needs to be in a relatively open place, and the multi-robots conform to the pre-set formation and spread out safely and evenly. The robot parameter initialization includes but is not limited to the setting of the safety distance, the upper limit setting of the robot speed, etc., and is initialized according to the task requirements or the execution capability of the robot platform.

(2) Multi-robots share the original state perception information of the sensor through internal communication, including two-dimensional laser observation information, speed state information and relative positional relationship of the target.

Among them, the two-dimensional laser information can be determined as a fixed-length one-dimensional array information according to the actual selection and setting of the sensor, and the specific implementation generally depends on the observation angle and resolution of the laser radar. The meaning of each number in the array is the obstacle distance at the current angle. If it exceeds the radar observation range, it will be marked as infinity. Velocity status information is the linear velocity information and angular velocity information of the current robot, which can be determined as a fixed-length one-dimensional array information. The relative position information of the target can be determined as two-dimensional coordinate information, and the specific communication information type is also fixed-length one-dimensional array information. The integrated state perception information is determined by an array with a limited size, the data volume is small, the communication load occupies less, and the expansion of multiple robots can be realized.

(3) According to the perception information obtained through communication sharing, each robot is decoupled based on the target position of the formation center, and a local occupancy map with the target position of the formation center as the midpoint is obtained. According to the target position, the distribution of obstacles and the positions of other formation members, an observation potential field can be established to describe the execution cost of each position in the current environment state as the target position of formation members. Among them, the specific value of the cost of each position in the potential field is determined by the resultant force F of the obstacle repulsion F _ro received by the current position, the formation teammate repulsion F _ra , and the target attraction F _a . The specific relationship is:

F＝F _ro +F _ra +F _a

Among them, the obstacle repulsion force Fro and the teammate repulsion force F _ra are determined by the inverse proportional relationship of the Euclidean distance between the current position and the obstacle position or the existing robot position on the field. The closer the distance between the two, the greater the repulsion force. The attractive force F _a of the target is determined by the proportional relationship of the square term of the Euclidean distance between the current position and the target position, the closer the distance between the two, the smaller the attractive force. Such a potential field setting can guide the formation to gradually approach the target position while maintaining a distance from obstacles.

(4) Each formation member stratifies the formation according to the environment state, the number of formation members and the safety distance of the robot, and then determines the target position of each formation member within the hierarchy, and finally assigns the target position of each formation member to each robot.

The overall shape of each layered formation is a nested and enlarged relationship to ensure the overall order of the formation and the safety of each formation member within the formation. Inside each layer, the method first determines the lowest position of the potential field as the position of the current formation member, then updates the overall environmental potential field according to the repulsive potential field of the member, and finally repeats the cycle iteratively to gradually find the formation members in the layer s position. After determining the position of the target formation, the method assigns the target according to the principle of distance priority and avoiding track interlacing, combined with the current position of each formation member. Since all robots use the same set of decision-making planning rules and parameter settings, and the perception information is unified through sharing, the allocation results can ensure the consistency of multiple machines.

(5) Each formation robot generates an initial control command based on the assigned target position, two-dimensional laser observation information, and velocity state information through a policy network based on reinforcement learning training.

The strategy network learns through autonomous trial and error in the simulation environment, and iteratively optimizes iteratively according to the reward and punishment information fed back from the environment until it converges to an autonomous navigation strategy with relatively stable performance. In practice, deep networks play strategic self-games in a stochastic multi-agent environment with shared strategies, and learn dynamic obstacle avoidance strategies.

The input of the deep strategy network is the sequential local obstacle occupancy map, speed state information and target position information converted from time-series two-dimensional laser information. The specific method of two-dimensional laser sequence conversion is that the method decouples the robot’s own pose changes obtained by the odometer, and obtains multiple frames of laser data in the same observation coordinate system, which can be transformed into a map form under the observation perspective of the robot. Get a sequence of stacked local obstacle maps. In this form, static obstacles and dynamic obstacles can be clearly distinguished, in which static obstacles appear as stacked ghosts, while dynamic obstacles appear as a moving track. The network uses a three-dimensional convolutional network to process sequence image information, a linear layer to process array information, and finally outputs decision-making instructions.

In terms of reward and punishment functions, the reward and punishment feedback information of the environment is mainly composed of two parts, the reward for approaching the target and the penalty for approaching obstacles.

R(s _t )＝R _g (s _t )+R _c (s _t )

R _g (s _t ) is used to guide the agent to gradually approach and finally reach the designated formation target position.

Where p ^t refers to the position of the robot at time t, and p ^* refers to the target position.

In order to ensure the safety of the movement in the process of approaching the formation target, the robot will be punished to a certain extent when approaching or encountering obstacles.

Among them, r refers to the safe radius of the robot, and d refers to the minimum obstacle distance at the current moment.

Under the guidance of the reward and punishment signal, the performance of the policy network will gradually converge to gradually approach the target position while ensuring the safety of the robot itself, so that the formation member robots have preliminary navigation performance, and are flexibly applicable to changing environmental distribution and multi-intelligence body interaction state.

The policy network can batch process large-scale multi-robot formation decision-making tasks, with higher efficiency and stronger scalability, and can meet large-scale formation task processing in practice.

(6) Each formation member robot evaluates the safety of the initial command according to the control obstacle function based on two-dimensional laser observation, and judges the gradient direction of the command optimization in combination with the observation prediction model, and uses the gradient optimizer to perform iterative optimization, and finally obtains a safety-guaranteed final control order.

The safety evaluation function is based on the control barrier function theory, and evaluates the degree of danger of the environment. According to the state of the robot and the evaluation results, the environment can be divided into initial state, safe state, and dangerous state. The transition between states can also be expressed as the effect of control decisions on the current environment, such as:

In the present invention, the observation-based control obstacle function describes the safety degree of the current robot state in the environment, including two parts: the safety of the robot itself and the internal safety of the formation. The safety of the robot itself mainly describes the safety relationship between the robot and static or dynamic obstacles inside the environment when the robot is moving at the current speed state, and the internal safety of the formation mainly describes the safety between the robot and other members of the formation when it is moving at the current speed state relation. In practice, the method chooses the collision time as the description of the safety assessment, that is, the length of time for a collision to occur after the robot receives and executes a given command in its current state.

The invention utilizes a world model based on two-dimensional lidar observations to perform observation predictions of any length. The world model is trained by means of supervised learning, using the trial-and-error data of the robot's autonomous exploration in the simulation environment, and fitting the observation change mechanism of the robot under different motion commands, so as to realize the prediction of future observations. The network uses the LSTM network for processing time-series information and the convolutional neural network for processing image information to process the sequential obstacle occupancy map sequence obtained from the recovery of time-series radar data, introduce the current control instruction information into the image hidden variable space, and obtain the comprehensive information of the current state. perception, and finally output the map prediction of obstacle occupancy under the control command.

After the observation-based safety evaluation function and an effective world model are established, according to the influence of dynamic constraints and the influence of control decisions on the current state, the method is selected to use numerical approximation methods to establish optimization iterations in the form of differences according to discrete changes in the environment gradient.

According to the gradient and the corresponding initial control decision, the augmented Lagrangian optimization is used to perform multi-step optimization iterations until the policy meets the security evaluation constraints, and the final security decision can be obtained to ensure the security of the execution policy.

Apparently, the present invention is not limited to the above embodiments, and many modifications can be made, and all modifications that can be directly derived or associated by those skilled in the art from the content disclosed in the present invention should be considered as the protection scope of the present invention.

Claims (7)

1. A multi-robot reinforcement learning safe formation method with constraints based on two-dimensional laser observation, characterized in that it comprises: S1: Obtain formation task setting parameters, including formation formation, number of formation members and formation starting position, initialize according to the parameters, and obtain preliminary state perception information; S2: Through the internal communication of the multi-robot formation, share the state perception information in real time, and obtain the state perception information obtained by the other robots in the formation through sensor observation, including two-dimensional laser observation information, speed state information and relative positional relationship of the target; S3: Decouple and integrate multi-robot state perception information, obtain the environment distribution state based on the target position of the formation center, and establish an observation potential field according to the target position of the formation center, the positions of each formation member, and the distribution of obstacles; S4: According to the observed potential field and the setting parameters of the formation task, split the formation level and calculate the formation target position within each layer, and allocate the target position according to the current formation member position; S5: Each formation member robot generates an initial control instruction from the policy network obtained based on reinforcement learning training according to the assigned target position and comprehensive state perception information; S6: According to the control obstacle function describing the safety state of the robot, combined with the observation prediction model and the gradient optimizer, iteratively optimize the initial control instruction until the safety constraints are met, and obtain the final control instruction that guarantees safety; S7: Send the final control command to the robot controller for execution, reach a new state, and return to S2 until the formation reaches the final target position. 2. The band-constrained multi-robot reinforcement learning safety formation method based on two-dimensional laser observation as claimed in claim 1, characterized in that, in the step S1, the formation task setting parameters are obtained, including formation formation, formation members The number and starting position of the formation are as follows: It supports multiple formation schemes including full encirclement formation, half encirclement formation and arbitrary angle envelopment, and can be extended to a larger number of robot formation tasks. 3. The band-constrained multi-robot reinforcement learning safe formation method based on two-dimensional laser observation as claimed in claim 1, wherein, in the step S2, the state perception information includes two-dimensional laser observation information, speed state information and The relative positional relationship of the target, specifically: The two-dimensional laser information is determined as a fixed-length one-dimensional array information according to the resolution of the sensor, the speed state information and the relative position information of the target are determined as two-dimensional coordinate information, and the integrated state perception information is determined by an array of a limited size, with a small amount of data , the communication load occupies less, and the expansion of multiple robots is realized. 4. As claimed in claim 1 or 2 or 3, the multi-robot reinforcement learning safe formation method based on two-dimensional laser observation is characterized in that, in the step S3, the multi-robot state perception information is decoupled and integrated , to obtain the environment distribution state based on the target position of the formation center, and establish the observation potential field according to the target position of the formation center, the positions of each formation member, and the distribution of obstacles, specifically: According to the target position, the distribution of obstacles and the positions of other formation members, an observation potential field is established, and each position is described as the execution cost of the formation member target position under the current environment state. The specific value of the cost of each position in the potential field is determined by the current position The resultant force F of obstacle repulsion F _ro , formation teammates repulsion F _ra and target attraction F _a is determined, and its specific relationship is: F=F _ro +F _ra +F _a . 5. The multi-robot reinforcement learning safe formation method with constraints based on two-dimensional laser observation as claimed in claim 4, characterized in that, in the step S4, according to the observed potential field and the formation task setting parameters, the formation Layer splitting and formation target position calculation within each layer, and target position allocation based on the current formation member positions, specifically: According to the observation potential field and task setting, the formations are layered. The overall shape of each layered formation is a nested proportional magnification relationship to ensure the overall order of the formation and the safety of each formation member within the formation. Inside each layer, First determine the lowest position of the potential field as the position of the current formation member, then update the overall environmental potential field according to the repulsive potential field of the member, and finally repeat the cycle through iteration to gradually find the position of each formation member in the layer; determine the target formation After the position, according to the principle of distance priority and avoiding track interlacing, the target is allocated according to the current position of each formation member. 6. The method for constrained multi-robot reinforcement learning safe formation based on two-dimensional laser observation as claimed in claim 4, characterized in that, in the step S5, according to the assigned target position and comprehensive state perception information, based on The policy network obtained by reinforcement learning training generates initial control instructions, specifically: The policy network is learned through autonomous trial and error in the simulation environment, and iteratively optimized according to reward and punishment information fed back from the environment until it converges to an autonomous navigation policy with relatively stable performance. 7. As claimed in claim 1 or 2 or 3 or 5 or 6, the band-constrained multi-robot reinforcement learning safety formation method based on two-dimensional laser observation is characterized in that, in the step S6, according to the description of the safety state of the robot Control the barrier function, combine the observation prediction model and the gradient optimizer to iteratively optimize the initial control instructions until the safety constraints are met, and obtain the final control instructions that guarantee safety, specifically: According to the control obstacle function based on two-dimensional laser observation, the safety evaluation of the initial instruction is carried out, and the gradient direction of the instruction optimization is judged by combining the observation prediction model, and the gradient optimizer is used for iterative optimization, and finally the final control instruction ensuring safety is obtained; The observation-based control obstacle function describes the safety degree of the current robot state in the environment, including two parts: the safety of the robot itself and the internal safety of the formation. The safety relationship between dynamic obstacles and the internal safety of the formation mainly describe the safety relationship between the robot and other members of the formation when it is moving at the current speed state.

Images