CN114815816A - An autonomous navigation robot - Google Patents

Abstract

The invention discloses an autonomous navigation robot, which comprises a sensor, a controller and a traveling mechanism, wherein the sensor is arranged on the controller; the sensor is used for detecting the distance and the angle of an obstacle relative to the robot to form state data; the controller is used for classifying scenes where the robot is located according to the state data and the relative position of the target point, and if the scenes are simple scenes, executing a PID control strategy; if the scene is a complex scene, executing a reinforced simulation learning control strategy; if the scene is an emergency scene, executing a constraint reinforcement simulation learning control strategy; the controller controls the traveling mechanism to drive the robot to travel by executing a corresponding control strategy, so that the aim of navigation and obstacle avoidance is fulfilled. The autonomous navigation strategy of the robot is designed for various complex scenes with dynamic and static obstacles, and the defects that the traditional path planning method cannot avoid the dynamic obstacles, the supervised learning method has poor generalization capability, and the reinforcement learning method has unsatisfactory strategy output under simple and emergency conditions can be overcome.

Description

An autonomous navigation robot

technical field

The invention belongs to the technical field of mobile robots, and in particular relates to an autonomous navigation robot that can reasonably plan a walking path according to environmental changes and can automatically avoid obstacles.

Background technique

Navigation and obstacle avoidance are the basic functions for mobile robots to complete various tasks. The mobile robot perceives the environment through external sensors, and obtains the information of each dimension of the geometric space. According to the obtained geometric space information combined with the obstacle avoidance algorithm, the robot can avoid obstacles and plan the path autonomously during the walking process.

Implementing the autonomous learning obstacle avoidance function on a mobile robot is an important step to improve the intelligence of the robot, which can enable the mobile robot to have a human-like behavior strategy and avoid dynamic or static obstacles in front of the robot in an unknown environment. Thus, the mobile robot has the ability to navigate autonomously.

At present, the autonomous navigation methods for robots mainly include path planning methods, supervised learning methods and reinforcement learning methods. Among them, the path planning method needs to accurately perceive the robot and its environment to ensure that the planned path length is optimal. However, this method requires centralized computing on a central server, which is difficult to use in large-scale robot swarms and unknown environments with dynamic obstacles. Supervised learning methods can make decisions based on sensor data, allowing mobile robots to avoid dynamic obstacles. However, it is difficult to collect the data required by this method. If the environmental state observed by the robot has not appeared in the training data set, it cannot make a correct decision, so the generalization ability is poor. The reinforcement learning method is trained by the interaction between the robot and the environment, and does not require a data set, and the strategy adopted in the environment has a certain degree of randomness. However, the automatic navigation robot designed based on the reinforcement learning model cannot drive in a straight line and walk out of the shortest path in a simple scene. Also, when approaching the target point position, the robot may linger near the target point instead of rapidly approaching the target point. At the same time, in emergency scenarios such as the surrounding obstacles are very dense and the obstacles suddenly appear in front of them, the robot cannot respond to the obstacles in time and perform emergency obstacle avoidance.

SUMMARY OF THE INVENTION

The invention designs the autonomous navigation strategy of the mobile robot for various complex scenes with dynamic and static obstacles, which can make up for the inability of the traditional path planning method to avoid dynamic obstacles, the poor generalization ability of the supervised learning method, and the reinforcement learning method in simple and urgent. In case the output strategy is not ideal.

In order to achieve the above-mentioned purpose of the invention, the present invention adopts the following technical solutions to realize:

An autonomous navigation robot, comprising a sensor, a controller and a walking mechanism; wherein, the sensor is used to detect the distance and angle of an obstacle relative to the robot to form state data; the controller is used to relative to the target point according to the state data and The location classifies the scene where the robot is located. If it is a simple scene, execute the PID control strategy; if it is a complex scene, execute the reinforcement imitation learning control strategy; if it is an emergency scene, execute the constraint reinforcement imitation learning control strategy; A corresponding control strategy is executed to calculate the linear velocity and angular velocity of the robot; the walking mechanism is used to drive the robot to walk according to the linear velocity and angular velocity calculated by the controller.

In some embodiments of the present application, in order to avoid the robot from colliding with obstacles during walking as much as possible, a collision prediction model may be configured in the controller, and the collision prediction model may be based on the state data and the robot's collision prediction model. Its own speed predicts whether the robot will collide.

In some embodiments of the present application, the simple scene is a scene in which there is no obstacle in front of the robot or the robot reaches around the target point; the emergency scene is a scene in which the collision prediction model machine predicts that the robot will collide; The complex scene is a scene other than the simple scene and the emergency scene.

In some embodiments of the present application, when the controller executes the PID control strategy, the controller can set the angle between the forward direction of the robot and the target point as a deviation, and substitute it into the PID calculation formula to calculate the angular velocity of the robot, and Keep the linear velocity of the robot constant. The PID control strategy can control the robot to travel on the shortest path and quickly reach the target point when approaching the target point position.

In some embodiments of the present application, the reinforcement imitation learning control strategy executed by the controller may include:

imitating the learning process, which uses the data in the expert dataset to train the Actor network;

The reinforcement learning process uses the Actor network and the Critic network trained by the imitation learning process, combines the state data, the robot's own speed and the relative position of the target point to calculate the output action a, and controls the walking mechanism to adjust the robot according to the action a. Linear and angular velocity of walking.

In some embodiments of the present application, the autonomous navigation robot is further configured with a memory for storing the action a calculated and output by the controller when executing the reinforcement imitation learning control strategy, and the robot arrives after the robot executes the action a in the environment. The state s and the reward r obtained by the robot performing action a, and the collected (s, a, r) data are stored in the experience pool; the number of data stored in the experience pool by the controller satisfies the set conditions When , the loss value of the reinforcement learning model is calculated, and then the Actor network and the Critic network in the reinforcement learning model are updated to achieve network optimization.

In some embodiments of the present application, the constrained reinforcement imitation learning control strategy may be configured to be the same as the Actor network and the Critic network used in the reinforcement imitation learning control strategy; the controller implements the constrained reinforcement imitation learning control strategy When , first determine whether the linear speed of the robot is greater than the set threshold; if it is greater than the set threshold, set the speed of the robot to 0, that is, control the robot to stop to achieve emergency obstacle avoidance; if it is less than or equal to the set threshold, reduce The distance data detected by the sensor is input, and the reduced distance data is input into the reinforcement learning model, so that the numerical value representing the speed of the robot in the output action a calculated by the reinforcement learning model is reduced. By reducing the walking speed of the robot and navigating with the help of reinforcement imitation learning control strategy, the design purpose of effective obstacle avoidance can be achieved.

Compared with the prior art, the advantages and positive effects of the present invention are: the present invention utilizes a scene classification model based on collision prediction to classify the environment where the robot is located at any time. For simple scenarios, the PID control strategy is used to control the robot to reach the target point in a straight line and quickly, avoiding the situation that the robot lingers near the target point instead of rapidly approaching the target point. For complex scenes, the reinforcement imitation learning control strategy is used for navigation to control the robot to avoid obstacles safely. For emergencies, a constraint-enhanced imitation learning control strategy is used to control the robot to respond to sudden obstacles in time to avoid collisions. The combined application of the three control strategies can make the robot reach the target point safely in a shorter time and path length, and improve the efficiency.

Other features and advantages of the present invention will become more apparent upon reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings.

Description of drawings

1 is a main hardware architecture diagram of an embodiment of an autonomous navigation robot proposed by the present invention;

2 is an overall architecture diagram of an embodiment of a navigation strategy executed by an autonomous navigation robot proposed by the present invention;

Fig. 3 is the scene classification flow chart based on collision prediction;

Fig. 4 is the update flow chart of the navigation model;

5 is a flowchart of an embodiment of a reinforcement imitation learning control strategy;

6 is a flow chart of an embodiment of a constrained reinforcement imitation learning control strategy;

Figure 7 is a schematic diagram of the trajectories of eight robots walking in a circular scene.

Detailed ways

The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

As shown in FIG. 1 , in order to realize the autonomous navigation function, the mobile robot of this embodiment is mainly configured with functional components such as sensors, controllers, memories, and traveling mechanisms in hardware configuration.

Among them, the sensor is used to observe the environmental state of the robot, such as the distance and angle of the obstacle relative to the robot, to form the state data and provide it for the navigation strategy. In some embodiments, the sensor may be a lidar sensor to enable fast and accurate acquisition of status data.

The controller is used to execute the navigation strategy, receive the state data detected by the sensor, and combine the relative position of the target point to classify the scene where the robot is located. Then, the control strategy corresponding to the scene is executed according to the type of scene the robot is in, so as to control the robot to reach the target point position safely and quickly, and realize autonomous navigation.

The memory is connected to the controller to store various data generated by the robot during operation, such as the action a that the controller calculates and outputs when executing the reinforcement imitation learning control strategy, the state s the robot reaches after executing action a in the environment, the robot execution The reward r obtained by action a, etc. The controller stores the collected (s, a, r) data into the experience pool (a storage space divided in the memory) for updating and optimizing the data model in the navigation strategy.

The walking mechanism is used to drive the robot to walk according to the angular velocity and linear velocity calculated by the controller, and finally reach the position of the target point.

The navigation strategy in this embodiment is formulated by a combination of a PID control strategy, a reinforcement imitation learning control strategy, and a constrained reinforcement imitation learning control strategy. Among them, when the robot is in a simple scene, the PID control strategy can be used for navigation to control the robot to reach the target point position in a straight line and quickly, so as to avoid the situation that the robot lingers near the target point instead of reaching the target point quickly. When the robot is in a complex scene, the reinforcement imitation learning control strategy can be used for navigation, so that the robot can walk out of an optimized route while avoiding obstacles safely. When the robot is in an emergency situation, the constraint reinforcement imitation learning control strategy can be used for navigation, so that the robot can respond to the sudden obstacle in time and avoid collision.

In order to enable the robot to automatically invoke the corresponding control strategy according to the type of environment in which it is located, a scene classification model based on collision prediction is configured in the controller in this embodiment, as shown in FIG. 2 and FIG. 3 . During the operation, the robot observes the environment state through sensors in real time, and inputs the detected state data and the robot's own speed (including but not limited to linear speed and angular speed) into the collision prediction model to predict the possible collision of the robot probability, and then control the robot to respond in advance to achieve safe obstacle avoidance.

The collision prediction model here can adopt the existing mature collision prediction model. When training the collision prediction model, you can use a variety of previous robot navigation algorithms to test in a variety of different simulation environments, and collect a large number of types of "sensor data, the robot's own speed, whether there is a collision" data. ; Then, use the collected data of the above types to train the collision prediction model; finally, apply the trained collision prediction model to the actual scene.

In this embodiment, the environment where the robot is located may be classified according to the prediction result generated by the collision prediction model. For example, if it is detected by the sensor that there is no obstacle in front of the robot or the robot has reached the target point, the scene at this time can be defined as a simple scene; if the prediction result generated by the collision prediction model is that a collision will occur (for example, the robot The surrounding obstacles are very dense or there are obstacles suddenly appearing), the scene at this time can be defined as an emergency scene; the rest of the scenes can be defined as complex scenes.

Referring to Fig. 3, during the actual operation, the robot periodically inputs the state data observed by the sensor and its own speed information into the collision prediction model to predict the collision. Of course, historical collision data can also be input into the collision prediction model to improve the accuracy of the prediction results.

If a collision is predicted by the collision prediction model, the robot executes the control strategy in the emergency scenario to avoid obstacles, that is, executes the constraint reinforcement imitation learning control strategy. If it is predicted that no collision will occur, the current scene is further classified. For example, if the sensor detects that the robot has reached the target point or that there is no obstacle in front of the robot, the PID control strategy in the simple scene is executed; otherwise, it is determined that the robot is in In complex scenarios, the reinforcement imitation learning control strategy is implemented.

In the PID control strategy, the angle between the forward direction of the robot and the target point can be set as the deviation, and the PID calculation formula can be substituted to calculate the angular velocity of the robot and keep the linear velocity of the robot unchanged. In the PID calculation formula, the three parameters of proportional, integral and differential can be respectively set as: k _P =0.15, k _I =0.08, k _D =0.01.

In the reinforcement imitation learning control strategy, the combination of imitation learning and reinforcement learning is used to formulate the control strategy.

Reinforcement Learning (RL) is an important branch of machine learning. Reinforcement learning is usually described by Markov Decision Process (MDP), specifically: the machine is in an environment, and each state is the machine's perception of the current environment; the machine can only affect the environment through actions , when the machine performs an action, it will cause the environment to transition to another state with a certain probability; at the same time, the environment will feed back a reward to the machine according to the potential reward function. In summary, reinforcement learning mainly includes four elements: state, action, transition probability and reward function, namely:

State s: the machine's perception of the environment, all possible states are called state space;

Action a: The action taken by the machine, all actions that can be taken constitute the action space;

Transition probability p: When an action is performed, the current state will transition to another state with a certain probability;

Reward function r: At the same time as the state transition, the environment feeds back a reward to the machine.

Therefore, the main task of reinforcement learning is to continuously try in the environment, adjust the strategy according to the feedback information obtained by the attempt, and finally generate a better strategy π, and the machine can know what action should be performed in what state according to this strategy. The quality of a strategy depends on the cumulative reward after long-term implementation of the strategy, in other words: the cumulative reward can be used to evaluate the quality of the strategy, and the optimal strategy means that after the strategy has been implemented in the initial state, the final cumulative reward The highest reward.

Imitation learning is a branch of reinforcement learning, which can well solve the multi-step decision-making problem in reinforcement learning.

In the process of imitation learning in this embodiment, expert data is used to train the Actor network, which can accelerate the convergence speed of the model in the subsequent training process while optimizing the navigation performance.

Referring to Figure 5, during the imitation learning phase, proven navigation algorithms (such as ORCA, RL, Hybrid-RL algorithms) can be used, tested in several different simulation environments, and data of the form (s, a) are collected As expert data, an expert dataset is formed.

Actor networks are trained using data from the expert dataset. Through training, the Actor network is fitted to the expert data, so that in the subsequent environment interaction process, the robot will not blindly explore in the environment, thereby accelerating the convergence speed of the model. At the same time, the data in the expert data set is the data corresponding to the robot's perfect performance in simple and emergency situations and the ability to execute perfect strategies. Therefore, Actor networks trained with expert datasets can perform better than traditional methods in simple and emergent situations.

The optimization goal of imitation learning is to reduce the error with the expert data set. Therefore, in this embodiment, the objective optimization function of imitation learning can be configured as:

表示求结果最小值所对应的θ。Among them, s _i and a _i are the data in the expert data set, and _si represents the state observed by the robot through the sensor, a _i represents the action performed by the robot in the state _si ; π _θ represents the Actor network; N represents the expert The number of samples in the dataset; θ represents the weight of the Actor network;

Represents the θ corresponding to the minimum value of the result.

The Actor network is optimized using the calculated weights θ.

Next, in the reinforcement learning phase, the Actor network can be further trained using the data collected by the robot during the actual operation.

The training of the reinforcement learning model is achieved by the robot constantly exploring the environment, collecting data, and aiming at maximizing the cumulative reward. The reinforcement learning model does not plan a path for the robot, but directly outputs control commands for navigation according to the state observed by the robot in the environment.

Combined with Figure 2 and Figure 4, the reinforcement learning model outputs the probability distribution of an action according to the state observed by the robot's own sensors in the environment, the robot's own speed, and the relative position of the target point. After that, randomly sample from the probability distribution to get the action a that the robot should perform. After the robot performs action a in the environment, it reaches state s. At the same time, performing action a results in an immediate reward r (ie, a reward) and a new observed state. The robot will store the data collected in the environment into the experience pool in the form of (s, a, r). When the amount of data stored in the experience pool satisfies a certain condition, the loss value of the reinforcement learning model is calculated to update the Critic network and the Actor network in the reinforcement learning model to optimize the navigation strategy.

In this embodiment, the reward function of the reinforcement learning model can be configured as follows:

Initialize reward function r=0;

The reward function r is calculated according to the positions pt ^-1 and pt of the robot at the ^t -1th time step and the ^t -th time step, the position g of the target point and the angular velocity wt of the robot at the t-1th time step, including The following situations:

If |p ^t -g||＜0.1, then r=r+r _arrival ; otherwise r=r+w _g (||p ^t-1 -g||-||p ^t -g||);

If the robot is predicted to collide, then r=r+r _collision ;

其中，||||表示取模运算；r_arrival、r_collision、w_g、w_w均为超参数。超参数是在开始学习过程之前设置的参数，而不是通过训练得到的参数数据。通常情况下，需要对超参数进行优化，给学习模型选择一组最优超参数，可以提高学习的性能和效果。本实施例的超参数可以赋予经验值。If |w ^t |>0.7, then

Among them, |||| represents the modulo operation; r _arrival , r _collision , w _g , and w _w are all hyperparameters. Hyperparameters are parameters that are set before starting the learning process, not parameter data obtained through training. Usually, hyperparameters need to be optimized, and selecting a set of optimal hyperparameters for the learning model can improve the performance and effect of learning. The hyperparameters of this embodiment can be given empirical values.

The optimization goal of the Critic network is to minimize the error with the accumulated reward value in the experience pool. Therefore, the objective optimization function of the Critic network configured in this embodiment is:

表示求结果最小值所对应的φ。Among them, φ is the weight of the Critic network; γ is the discount factor; t is the time step; T is the maximum number of steps; s _t represents the state of the robot at the t-th time step; r _t' is the robot at the t'-th time step. The reward obtained; V _φ is the Critic function, which is used to evaluate the quality of the state s;

Indicates the φ corresponding to the minimum value of the result.

The optimization objective of the Actor network is to maximize the expected value of the accumulated reward. Therefore, the objective optimization function of the Actor network configured in this embodiment is:

即：

which is:

λ、ε为超参数；θ为Actor网络的权重；n表示回合数；T_n表示第n回合的最大步数；a_t表示机器人在第t个时间步的动作；π_θ和π_old分别表示当前的Actor网络和更新前的Actor网络；E_t表示期望函数；

表示求结果最大值所对应的θ；clip表示限制函数，且

即，将r_t(θ)的值限制在[1-ε，1+ε]范围内。in,

λ and ε are hyperparameters; θ is the weight of the Actor network; n represents the number of rounds; T _n represents the maximum number of steps in the nth round; a _t represents the action of the robot at the t-th time step; π _θ and π _old respectively represent The current Actor network and the Actor network before the update; E _t represents the expectation function;

represents the θ corresponding to the maximum value of the result; clip represents the limit function, and

That is, the value of r _t (θ) is restricted to be in the range of [1-ε, 1+ε].

Using the calculated weights θ of the Actor network and φ of the Critic network, the Actor network and the Critic network are updated to further train the reinforcement learning model.

The trained Actor network and Critic network can be applied to the constrained reinforcement imitation learning control strategy in emergency scenarios.

Although the traditional deep learning model will take certain actions to avoid obstacles in more urgent scenarios, the range of actions taken is often insufficient, and the execution of actions is not timely enough, resulting in frequent collisions.

In order to solve the above problem, this embodiment configures a collision prediction model, which can predict possible collisions several steps in advance. After that, the reinforcement imitation learning control strategy of constraints is implemented, and the control strategy is used to constrain the actions performed by the robot, thereby avoiding the occurrence of collisions.

The constraint reinforcement imitation learning control strategy in this embodiment is based on the reinforcement imitation learning control strategy in complex scenarios, adding constraints, and by limiting the walking speed of the robot, the technical effect of safe obstacle avoidance is achieved.

As shown in Figure 6, when the robot is in an emergency environment, first determine whether the linear speed of the robot is greater than the set threshold; Set to 0, that is, control the robot to stop running to avoid sudden obstacles; otherwise, reduce the distance data detected by the sensor, and input the reduced distance data into the reinforcement learning model, so that the output action a is calculated by the reinforcement learning model The numerical value representing the speed of the robot (at least including the linear speed) is reduced in , that is, the robot is controlled to decelerate, so that a relatively safe action a can be obtained. The robot performs this action a and will not collide.

In some embodiments, when the linear speed of the robot in an emergency environment is less than or equal to a set threshold, the distance data detected by the sensor can be reduced by a factor of P, that is, the distance data detected by the sensor is divided by P, the P preferably takes a value between [1.25, 1.5]. At the same time, the reduced distance data is constrained to be within the set size threshold range [Dmin, Dmax], which can further ensure that the action a output by the reinforcement learning model will be a safe action, avoiding the robot and the Obstacle launch collision.

The autonomous navigation strategy executed by the mobile robot of this embodiment is described below through a specific example.

Eight robots are deployed in a circular scene with a radius of 3 meters, and each robot needs to drive to the position opposite to its own circle center, that is, the target point position. The eight robots all entered the running state within the same time period, forming obstacles to each other. Each robot executes the autonomous navigation strategy of this embodiment, and the formed motion trajectory is shown in FIG. 7 .

In the process of running, each robot only depends on the state of the environment it observes to decide the next action.

First, the robot inputs the observed environment information (state s) into the collision prediction module. The collision prediction module determines which type of environment the robot is currently in according to the environment information.

If the robot is currently in a simple environment, the PID control strategy is used to control the robot to move quickly along a straight line towards its target point position.

If the robot is in a complex environment, the reinforcement imitation learning control strategy is adopted to output the action a that the robot should perform according to the environment state s.

If the robot is in an emergency environment, there is a high possibility of a collision. At this time, if the current speed of the robot is greater than the set threshold, the robot is controlled to stop moving immediately; otherwise, the constrained reinforcement imitation learning control strategy is adopted, and the constrained environment state is input into the reinforcement learning model, and then the constrained action a is obtained for the robot to execute, to avoid collisions.

It can be seen from the above experimental scenarios that all robots can reach the target point with a short, smooth path and without collision, and successfully complete all navigation tasks.

Of course, the above is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. Improvements and modifications should also be considered within the scope of the present invention.

Claims (10)

1. An autonomous navigation robot, comprising:

the sensor is used for detecting the distance and the angle of the obstacle relative to the robot to form state data;

the controller is used for classifying scenes where the robot is located according to the state data and the relative position of the target point, and if the scenes are simple scenes, executing a PID control strategy; if the scene is a complex scene, executing a reinforced simulation learning control strategy; if the scene is an emergency scene, executing a constraint reinforcement simulation learning control strategy; calculating the linear speed and the angular speed of the robot walking by executing corresponding control strategies;

and the traveling mechanism is used for driving the robot to travel according to the linear velocity and the angular velocity calculated by the controller.

2. The autonomous navigation robot according to claim 1, wherein a collision prediction model that predicts whether or not the robot can collide based on the state data and a speed of the robot itself is provided in the controller.

3. The autonomous navigational robot of claim 2,

the simple scene is a scene that no obstacle exists in front of the robot or the robot reaches the periphery of a target point;

the emergency scene is a scene in which the collision of the robot is predicted by the collision prediction model machine;

the complex scene is a scene except the simple scene and the emergency scene.

4. The autonomous navigation robot of any of claims 1 to 3, wherein the controller sets an angle between a forward direction of the robot and a target point as a deviation when executing a PID control strategy, substitutes a PID calculation formula to calculate an angular velocity of the robot, and keeps a linear velocity of the robot constant.

5. The autonomous navigational robot of claim 1, wherein the augmented-mock-learning control strategy executed by the controller comprises:

simulating a learning process that trains an Actor network using data in an expert dataset;

and in the reinforcement learning process, an Actor network and a Critic network trained by the simulation learning process are utilized, state data, the self speed of the robot and the relative position of a target point are combined to calculate and output an action a, and the walking mechanism is controlled to adjust the linear speed and the angular speed of the robot walking according to the action a.

6. The autonomous navigational robot of claim 5, wherein the controller configures the objective optimization function of the mock learning to:

wherein s is _i 、a _i Is data in an expert data set, and s _i Representing the state of the robot as observed by the sensor, a _i Is shown in state s _i An action performed by the lower robot; pi _θ Representing an Actor network; n represents the number of samples in the expert data set; θ represents the weight of the Actor network;

represents theta corresponding to the minimum value of the calculation result;

the controller optimizes the Actor network using the weight θ.

7. The autonomous navigational robot of claim 5, further comprising:

a memory for storing an action a which the controller calculates and outputs when executing the reinforcement mimic learning control strategy, a state s which the robot reaches after executing the action a in the environment, and a reward r obtained when the robot executes the action a, and storing the collected (s, a, r) data into an experience pool;

when the quantity of the data stored in the experience pool meets a set condition, the controller calculates a loss value of the reinforcement learning model, and then updates an Actor network and a Critic network in the reinforcement learning model.

8. The autonomous navigational robot of claim 7, wherein the controller configures the reinforcement learned reward function as follows:

initializing a reward function r to be 0;

according to the position p of the robot at the t-1 time step and the t time step ^t-1 、p ^t And the position g of the target point and the angular velocity w of the t-1 time step of the robot ^t Calculating a reward function r, including the following:

if p ^t -g | < 0.1, then r ═ r + r _arrival (ii) a Otherwise r is r + w _g (||p ^t-1 -g||-||p ^t -g||)；

If the robot is predicted to collide, r is r + r _collision ；

If | w ^t If | is greater than 0.7, then

Wherein, | | | represents a modulo operation; r is _arrival 、r _collision 、w _g 、w _w Are all hyper-parameters.

9. The autonomous navigational robot of claim 7, wherein the controller configures the objective optimization function of the Critic network during the reinforcement learning process to be:

wherein phi is the weight of the Critic network; gamma is a discount factor; t is a time step; t is the maximum step number; s _t Representing the state of the robot at the t time step; r is _t' A reward obtained for the robot at the t' th time step; v _φ Is a Critic function and is used for evaluating the state s;

expressing phi corresponding to the minimum value of the calculation result;

configuring an objective optimization function of the Actor network as follows:

wherein θ is the weight of the Actor network; n represents the number of rounds; t is _n Representing the maximum number of steps of the nth round; a is _t Representing the motion of the robot at the t-th time step; ε represents a hyperparameter; pi _θ And pi _old Respectively representing a current Actor network and an Actor network before updating; e _t Representing a desired function;

represents theta corresponding to the maximum value of the calculation result; clip represents a restriction function, and

10. the autonomous navigation robot of any one of claims 5 to 9,

the constraint reinforcement modeling learning control strategy is the same as an Actor network and a Critic network used in the reinforcement modeling learning control strategy;

the constraint-reinforced mimic learning control strategy executed by the controller comprises:

judging whether the linear velocity of the robot is greater than a set threshold value or not;

if the current value is greater than the set threshold value, controlling the robot to stop walking;

if the distance data detected by the sensor is smaller than the set threshold, the distance data detected by the sensor is input to the reinforcement learning model, and the numerical value representing the robot speed in the calculation output action a by the reinforcement learning model is reduced.

Images