CN111857107A - Auxiliary mobile robot navigation control system and method based on learning component library - Google Patents

Abstract

The invention discloses a system and a method for controlling navigation of an auxiliary mobile robot based on a learning component library, wherein the system comprises the learning component library, and the learning component library comprises: the system comprises an initialization component, an environment modeling component, a path planning component, a core algorithm component, a testing component, an optimization component and a visualization component. The components in the invention are mutually interacted and flexibly called, and a plurality of simulation or practically applied reinforcement learning training and visualization closed-loop learning systems which are suitable for different algorithm strategies of navigation task scenes can be quickly constructed according to the type of the mobile robot in the using process.

Description

Auxiliary mobile robot navigation control system and method based on learning component library

technical field

The invention relates to an auxiliary mobile robot navigation control system and method based on a learning component library, and belongs to the technical field of robot control.

Background technique

In recent years, with the development of robot technology, function-assisted mobile robots have been widely used in various fields such as agriculture, commerce, logistics, medical assistance, and military industry. For example, during the domestic new crown virus epidemic, auxiliary mobile robots, with their autonomy, played an important role in hospitals, community disinfection, express logistics and distribution, body temperature detection, intelligent consultation in isolation areas, etc., and promoted my country's epidemic prevention protests process.

Assisted mobile robot is a comprehensive system that integrates environment perception, autonomous positioning, path planning, low-level navigation control and execution of specific auxiliary functions. Take a mobile robot that performs disinfection tasks in public places during the epidemic as an example. During the process of disinfection, it uses a variety of external sensors on its own, such as monocular vision cameras, binocular vision cameras, lidar, and millimeter-wave radar. Ultrasonic sensors, etc. obtain the environmental information of the area that needs to be sterilized; then combine their own internal sensors, such as inertial sensors, GPS, etc. to estimate their global position and attitude information in the current area; on the basis of the above two steps, combined with specific task requirements, Use path planning algorithms, such as artificial potential field method, heuristic fast expanding random tree, etc., to plan an optimal path from the initial position to the target position; finally, combine its own dynamics and kinematics characteristics, actuator characteristics and chassis drive Configuration, through the underlying navigation controller to carry out precise navigation and tracking control of the planned trajectory, so that the mobile robot travels according to the pre-planned path.

However, the current traditional navigation control method lacks a specific simulation platform for auxiliary mobile robots, the configuration and training process is complicated and tedious, and lacks systematicness; and each reinforcement learning navigation control algorithm at this stage is based on a specific robot and a specific scene. On top of that, the reinforcement learning environment construction methods for simulated scenarios and actual scenarios are different and lack flexibility.

SUMMARY OF THE INVENTION

Aiming at the above technical problems existing in the above-mentioned traditional system-assisted mobile robot navigation control method, the present invention provides an auxiliary mobile robot navigation control method based on a learning component library, which is convenient for users to quickly strengthen the learning closed-loop control system according to their own needs. A navigation control method for mobile robots that is easy to build and facilitate parameter debugging and performance optimization.

The present invention adopts the following technical solutions.

In one aspect, the present invention provides an auxiliary mobile robot navigation control system based on a learning component library, including a learning component library, and the learning component library includes: an initialization component, an environment modeling component, a path planning component, a core algorithm component, a test component component, optimization component and visualization component; the initialization component is used to complete the initialization of the state space and action space corresponding to a specific mobile robot type, and used to set up a reward function; the environment modeling component is used to read and process The sensor data carried by the mobile robot, as well as the global position data used to determine the location where the robot is located, and to establish a virtual environment for interacting with the mobile robot when performing a simulation task; the path planning component is used to provide selectable path planning algorithm to achieve optimal navigation path planning; the core algorithm component is used to provide a variety of reinforcement learning algorithms for selection, cooperate with the underlying control algorithm component or directly output controller commands, and obtain current information through the environment modeling component after the action. , to complete the reinforcement learning closed-loop control; the test component is used to provide a perturbation method for selection to test the performance of the reinforcement learning algorithm determined by the core algorithm component; the optimization component is used to provide an optimization algorithm for selection. The selected parameters of the reinforcement learning algorithm determined by the core algorithm component are used for adjustment to improve the performance of the navigation control algorithm; the visualization component is used to visualize the output values of the core algorithm component and the test component.

Further, the core component library includes a same strategy module, a different strategy module and a comprehensive strategy module, the same strategy module is used to encapsulate the reinforcement learning algorithm of the same strategy, and the different strategy is used to encapsulate the reinforcement learning algorithm of the different strategy; The integrated strategy module is used to encapsulate an integrated strategy algorithm, and the integrated strategy algorithm is a data-driven enhancement algorithm that integrates the same strategy and different strategies. The comprehensive strategy algorithm includes: by timely feeding back the learned new strategy to the mobile robot system, and collecting specific system data to optimize the adaptive ability of the reinforcement learning algorithm; at the same time, considering the original characteristics of the system, the re-collected data is compared with the previous playback experience. The data is combined and learned again to finally determine the reinforcement learning algorithm.

Further, the system further includes: a low-level control algorithm component, the low-level control algorithm component can be directly used to provide a benchmark component for comparison with the reinforcement learning algorithm, and can also be combined with the upper-level reinforcement learning algorithm to build a state from direct to execution. Closed-loop control of reinforcement learning systems for machine instructions. Further, the environment modeling component includes: a sensor data processing module, a mobile robot positioning module and a reinforcement learning environment modeling module, the sensor data processing module is used to read and process sensor data carried by the mobile robot, the mobile robot The robot positioning module is used for locating the global position data of the robot in real time; the reinforcement learning environment modeling module is used for establishing a virtual environment for interacting with the mobile robot when the simulation task is performed.

Further, the optional optimization algorithm provided by the optimization component includes a regularization algorithm, and the regularization algorithm includes an L1 and L2 regularization algorithm, an entropy regularization algorithm and/or an early stopping algorithm.

Further, an evaluation function module is respectively set in the path planning component and the core algorithm component to provide a performance evaluation function to realize the performance evaluation of the parameter adjustment and algorithm selection of the path planning component and the core algorithm component.

In a second aspect, the present invention provides an auxiliary mobile robot navigation control method based on a learning component library, the method is based on the learning component library-based auxiliary mobile robot navigation control system described in the above technical solution, and the method includes the following Steps: Select the state space and action space corresponding to the specific mobile robot type from the pre-built initialization components, and set up the reinforcement learning reward function to complete the initialization;

Use the pre-built environment modeling component to build a reinforcement learning simulation environment; obtain the relative position of the obstacle and the mobile robot's own position through the environment modeling component, use the pre-built path planning component to select the required path planning algorithm, and plan the optimal navigation path ; Adjust the reward function of the navigation control algorithm according to the path planning result;

Select and determine the reinforcement learning algorithm from the pre-built core algorithm components, combine the defined action space, state space, reward function and reinforcement learning environment, select the core algorithm module for training; perform actions through the underlying control module or directly output controller instructions, Then, obtain the relative position of the obstacle and the position of the mobile robot itself through the environment modeling component again, and repeat the steps to complete the output controller command to complete the reinforcement learning closed-loop control;

Select the perturbation method from the test component, and test the performance of the reinforcement learning algorithm selected from the core algorithm component;

Select and determine the optimization algorithm from the optimization component to adjust the selected parameters of the reinforcement learning algorithm determined by the core algorithm component, so as to improve the performance of the navigation control algorithm;

Use the visualization component to visualize the output values of the core algorithm component and the test component.

In a third aspect, the present invention provides a navigation control method for an assisted mobile robot based on a learning component library. The method is based on an assisted mobile robot navigation control system based on a learning component library. The system includes a learning component library, and the learning component The library includes: initialization component, environment modeling component, path planning component, core algorithm component, testing component, optimization component, visualization component and underlying control algorithm component; the initialization component is used to complete the state space corresponding to a specific mobile robot type, Initialization of the action space, and for setting up a reward function; the path planning component for providing a selectable path planning algorithm to achieve optimal navigation path planning; the core algorithm component for providing a variety of reinforcement learning algorithms for selection, so that the output controller instruction completes the reinforcement learning closed-loop control; the testing component is used to provide a perturbation method for selection to test the performance of the reinforcement learning algorithm determined by using the core algorithm component; the optimization component is used to provide The selected optimization algorithm adjusts the selected parameters of the reinforcement learning algorithm determined by the core algorithm component, so as to improve the performance of the navigation control algorithm; the visualization component is used to visualize the output values of the core algorithm component and the test component; so The underlying control algorithm components described above are used to provide benchmark components for comparison with reinforcement learning algorithms;

The method includes the following steps:

Select the state space and action space corresponding to the specific mobile robot type from the pre-built initialization components, and set up the reinforcement learning reward function to complete the initialization;

Call the environment modeling component to obtain the sensor data carried by the mobile robot and the global position data of the mobile robot;

Combine the defined action space, state space, reward function with the sensor data carried by the mobile robot and the global position data of the mobile robot, select and determine the reinforcement learning algorithm from the pre-built core algorithm components, cooperate with the underlying control components or directly make the output control After the action, each sensor value of the environment modeling component is used again, and the above process is repeated to complete the reinforcement learning closed-loop control;

Select the disturbance method from the test component, use the test component to evaluate and test the algorithm, and feedback the output state observation value of the sensor processing module in real time to judge whether the control requirements are met;

Select and determine the optimization algorithm from the optimization component to adjust the selected parameters of the reinforcement learning algorithm determined by the core algorithm component until the mobile robot obtains the predetermined execution effect in the navigation control task; use the visualization component to adjust the core algorithm component and the test component. The output value is visualized.

Further, a performance evaluation function module is respectively set in the path planning component and the core algorithm building, and the method further includes: using the evaluation function module to determine a performance evaluation function, and adjusting parameters and algorithms of the path planning component and the core algorithm component. The selected performance evaluation uses the visualization component to visualize the evaluation results of the performance evaluation function.

The beneficial technical effect obtained by the present invention: in the present invention, each component interacts with each other, and can be called flexibly. During the use process, a variety of simulation or practical application of reinforcement learning training and training can be quickly constructed according to the type of the mobile robot. Visual closed-loop learning system; by testing components, you can test the stability, robustness and generalization ability of the configuration algorithm. If you need to optimize the algorithm, you can easily and quickly perform parameter optimization and regularization by learning the optimization components in the component library operation, avoid algorithm overfitting and improve algorithm performance; at the same time, if you want to change the sensor configuration or drive configuration of the mobile robot, you don’t need to rebuild the entire navigation control algorithm workflow, you can directly replace the corresponding component modules, which has good flexibility with versatility.

The assisted mobile robot navigation and control method based on the learning component of the present invention can not only be applied to the actual mobile robot control through the complete workflow between components, but also can use the simulation environment to carry out the navigation control algorithm effect of the mobile robot. Simulation test; has good flexibility and versatility, and can be easily applied to the navigation control tasks of auxiliary mobile robots equipped with various sensor schemes and driving configurations. In the actual application process, the mobile robot can be controlled by traditional control algorithms to perform navigation tasks, and the reinforcement learning-based navigation control can also be performed by quickly building a reinforcement learning environment. In the simulation environment, you can also build a simulation environment to study the performance of the algorithm. During use, the path planning algorithm, the reward function construction method, the core learning algorithm module, etc. can be changed modularly, and various evaluation indicators of each algorithm can be easily monitored, and a benchmark algorithm for comparison can be established.

On the other hand, the navigation control learning component library provides low-level control algorithm components including mainstream control algorithms to facilitate performance comparison and verification of the learning algorithms.

Description of drawings

Fig. 1 is the overall structure of a kind of auxiliary mobile robot navigation control system based on learning component provided by the embodiment of the present invention;

2 is a first construction method of a learning component-based assisted mobile robot navigation control method provided in an embodiment of the present invention;

3 is a second construction method of a learning component-based assisted mobile robot navigation control method provided by an embodiment of the present invention;

FIG. 4 is an architecture diagram of a comprehensive strategy algorithm in an embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

Embodiment 1. An auxiliary mobile robot navigation control system based on a learning component library, including a pre-established learning component library for navigation control of an auxiliary mobile robot, the learning component library includes: an initialization component, an environment modeling components, path planning components, core algorithm components, testing components, optimization components and visualization components;

The initialization component is used to complete the initialization of the state space and action space corresponding to the specific mobile robot type, and to set up a reward function;

The environment modeling component is used to read and process sensor data carried by the mobile robot, and to determine the global position data where the positioning robot is located and to establish a virtual environment for interacting with the mobile robot when performing a simulation task;

The path planning component is used to provide a selectable path planning algorithm to realize optimal navigation path planning; the core algorithm component is used to provide a variety of reinforcement learning algorithms for selection, so that the output controller instruction can complete the reinforcement learning closed-loop control The optimization component is used to provide a regularization method so as to realize the optimization of the reinforcement learning algorithm; the test component is used to provide a perturbation method for selection to test the performance of the reinforcement learning algorithm determined by the core algorithm component; the The optimization component is used to provide a regularization algorithm for selection to adjust the selected parameters of the reinforcement learning algorithm determined by the core algorithm component, so as to improve the performance of the navigation control algorithm; the visualization component is used to adjust the core algorithm component and the test The output value of the component is visualized.

Learning Component Library for Navigation Control, a computer database, is a standardized computer software module for mobile robot navigation control. The navigation control learning component library calls the pre-packaged algorithms and modules according to the input information, and finally obtains the results returned by each component. The learning component library provided by the present invention can be directly applied to the navigation control of the actual mobile robot, and the core reinforcement learning algorithm can also be used as an upper-level control link in the control closed loop to learn the complex mobile robot behavior and output the underlying controller's behavior. reference volume.

In the navigation control problem of the mobile robot tracking the planned path, the components in this embodiment mainly fall into the following categories:

It includes the initialization components for the action space, state space, and reward function design of mobile robots with different driving configurations; the environment modeling component includes various modules required to build the environment model; the path planning component includes different path planning algorithms; The core algorithm components of strategy, heterogeneous strategy, and comprehensive strategy algorithms; include optimization components of regularization modules that improve the robustness and generalization of control algorithms and avoid overfitting; include test components for testing algorithm performance. Contains visualization components to visualize various performance parameters.

For example, in the path planning component, input the environmental information output by the previous environment modeling component, the position of the robot itself and the target point, and after selecting the required path planning algorithm, the planned path can be obtained.

For another example, in the core algorithm component, by inputting the selected algorithm type, the algorithm module is called for training, and the performance evaluation parameters during the training process are returned in real time to monitor the algorithm performance during the training process.

In a specific embodiment, optionally, the initialization component mainly includes a state space design module, an action space design module, and a reward function design module. The environment modeling component mainly includes a visual sensor processing module, a lidar sensor processing module, a robot positioning sensor processing module, and a reinforcement learning environment modeling module. The path planning component includes a heuristic path planning module, an artificial potential field path planning module, a machine learning path planning module, and the like. The core algorithm component includes a same-policy algorithm module, a different-strategy algorithm module, and a comprehensive strategy algorithm module. The optimization component includes a hyperparameter optimization module, a regularization module, etc., wherein the regularization module is encapsulated with commonly used regularization algorithms, such as L1/L2 regularization algorithm, entropy regularization algorithm, early stop algorithm, etc. Added to improve the generalization performance of reinforcement learning algorithms. The disturbance component includes a dynamic obstacle disturbance module, a wind disturbance module, a water flow disturbance module, and the like. The visualization component includes a learning curve visualization module, a navigation control error visualization module, an actuator numerical visualization module, and the like.

The various components in the learning component library proposed by the present invention are standardized computer software modules used for the learning algorithm of the assisted mobile robot navigation control. Each component calls the pre-packaged algorithms and modules according to the input information, and finally obtains the results returned by each component. Those skilled in the art can, based on the system architecture provided by the present invention and according to the requirements of practical applications, use the existing technology to realize the construction of each component and the invocation between components, that is, to encapsulate the intelligent algorithm and convert it into a standard module, and add it to the In the component that contains the corresponding function. The various components in the learning component library in the present invention can interact with each other and call each other.

The navigation control learning component library in this embodiment can be applied to the actual mobile robot control through the complete workflow between the components, and can also use the simulation environment to simulate the effect of the navigation control algorithm of the mobile robot. When the mobile robot is simulated and controlled, the virtual environment can be built through the reinforcement learning environment modeling module in the environment modeling component. The user can directly test the overall algorithm performance of the core algorithm library, and can also strengthen the design by himself. The learning algorithm is connected, and then the construction of the algorithm training environment is quickly completed. At the same time, it avoids some time and hardware cost losses caused by training directly on the actual robot.

If the usage scenario is relatively simple, it can be directly applied to the actual mobile robot control. On the one hand, through the observation values of each sensor, the strategy network outputs the actuator commands, and at the same time, in order to prevent damage to the actuators, the thresholds of each actuator command are limited. , to ensure actual runtime security. On the other hand, based on the navigation control learning component library, the core reinforcement learning algorithm can also be used as an upper control link in the control closed loop to learn complex mobile robot behavior and output the reference quantity of the underlying controller. Learning from the advantages of traditional closed-loop controllers ensures the final algorithm performance.

Preferably, in the navigation control learning component library, the parameter adjustment and algorithm selection of each component can be determined by the performance evaluation function of each component, and each performance evaluation function can be visualized through a visualization module to facilitate monitoring and evaluation of algorithm performance.

In the route planning component, the final planned route result is evaluated according to the time index and the energy consumption index.

Optionally, in the core algorithm component, the final navigation control effect is evaluated by the final learning curve, the tracking accuracy, and the value change curve of the actuator.

Preferably, in the component, whether each module is used or not is determined by task requirements; each module can be flexibly added, deleted and replaced according to different task conditions. For example, if you want to compare the navigation control effect of the deep deterministic strategy gradient algorithm and the comprehensive strategy algorithm, you only need to replace the core algorithm components and compare them according to the final evaluation index.

Embodiment 2 On the basis of Embodiment 1, this embodiment provides an auxiliary mobile robot navigation control system based on a learning component library. The system further includes: a low-level control algorithm component, and the low-level control algorithm component uses To provide benchmark components against which reinforcement learning algorithms are compared. The underlying control algorithm component is an optional component, including some commonly used control algorithm modules. This component is mainly used as a comparison benchmark or cooperates with the core algorithm component to construct a navigation control closed-loop system. On the one hand, the underlying control algorithm component can be directly used as a benchmark component for comparison with the reinforcement learning algorithm, and on the other hand, it can be combined with the upper-level reinforcement learning algorithm to build a closed-loop control reinforcement learning system from the state directly to the actuator instruction. The reinforcement learning output (x, y, psi..) upper layer instruction is used as the input of the controller, and the controller outputs the actuator instruction by calling the underlying control algorithm component for tracking. Such a layered architecture can effectively reduce the data latitude of reinforcement learning and improve efficiency.

Embodiment 3 On the basis of Embodiment 2, this embodiment provides an auxiliary mobile robot navigation control system based on a learning component library (as shown in FIG. 3 ). The core component library includes a same strategy module, a different A strategy module and an integrated strategy module, the same strategy module is used to encapsulate the reinforcement learning algorithm of the same strategy, and the different strategy is used to encapsulate the reinforcement learning algorithm of different strategies; the integrated strategy module is used to encapsulate the integrated strategy algorithm, so The comprehensive strategy algorithm described above is a data-driven reinforcement algorithm that integrates the same strategy and different strategies.

In the traditional navigation control method, the data read by the sensor must first perform feature extraction, fusion, and state estimation, and then perform the underlying actuator control or the upper-level mission control. Commonly used are feedback linear control, linear quadratic control, model predictive control, backstepping control, etc. These methods have some limitations that limit the application of mobile robots in complex scenarios. For example, the linearization of the motion model is difficult to accurately describe the dynamics of complex systems; and some nonlinear control methods rely on accurate mathematical and physical models of the controlled object, which often require a lot of prior knowledge and expert experience. The design process is cumbersome and time-consuming. With the rapid development of artificial intelligence technology, deep reinforcement learning has a wide range of applications in the field of intelligent body control. The control based on deep reinforcement learning directly avoids the tedious data processing process, and directly outputs the output according to the observation value of the sensor and through the policy network. Action to be performed. However, traditional deep reinforcement learning algorithms still have the following problems in robot control: 1. The mainstream reinforcement learning algorithms with the same strategy have strong adaptability to environmental changes, but because they rely on a large amount of real-time data, they require huge computational costs resources, the convergence speed is slow; 2. Although the reinforcement learning algorithm of different strategies has good computational efficiency, it has poor adaptability to environmental changes due to its repeated sampling of the original state sequence data; 3. It tends to overuse specific tasks. Fitting, the generalization ability of the algorithm is weak.

By providing a comprehensive strategy algorithm, this embodiment realizes the advantages of a comprehensive same strategy and a different strategy algorithm and can improve the generalization ability of the reinforcement learning control algorithm. The comprehensive strategy algorithm includes: by timely feeding back the learned new strategy to the mobile robot system, and collecting specific system data to optimize the adaptive ability of the reinforcement learning algorithm; at the same time, considering the original characteristics of the system, the re-collected data is compared with the previous playback experience. The data is combined and learned again to finally determine the reinforcement learning algorithm. The specific instructions are as follows:

The architecture diagram of the comprehensive strategy algorithm in this embodiment is shown in FIG. 4 . The mainstream reinforcement learning algorithms are based on the same strategy or different strategies. Both of these two methods have the above problems. The comprehensive strategy algorithm proposed in this embodiment combines the advantages of the above two types, and the details are as follows:

For example, the most typical heterogeneous strategy reinforcement learning algorithm, Q-Learning, the update process of its action state Q value is as follows:

Among them, R(s) is the reward function, and (s', a') is the optimal state-action pair; the algorithm always uses the optimal Q value when calculating the expected return of the next state, and selects the optimal action, but the current The strategy does not necessarily choose the optimal action, so it does not care what the strategy is. The strategy of generating samples is different from the strategy of learning, which is also called different strategy mechanism. The advantage of different strategies is that the global optimum can be obtained, and the general behavior is strong, but the training process is tortuous and the convergence speed is slow.

For another example, the most typical same-strategy reinforcement learning algorithm, the SARSA algorithm, the update process of its Q value is as follows

It can be found that in the typical same-strategy reinforcement learning algorithm, the strategy used when the network parameters are updated is the same as the strategy for generating samples. This strategy algorithm is relatively straightforward and has a fast calculation speed, but because it only uses the currently known The optimal choice of , may not learn the optimal solution and fall into a local optimum.

The comprehensive strategy algorithm encapsulated by the comprehensive strategy module in this embodiment is established on the basis of the above two strategy algorithms, and its main process is as follows:

S41 performs initialization operations such as states and actions; S42 performs initial actions, which are similar to different strategy algorithms such as DQN, and perform experience pool filling;

S43 strengthens the learning algorithm subject part, carries out strategy evaluation and strategy optimization, and judges whether the convergence is achieved through S44, if not, at this time, go to S45. The difference of the comprehensive strategy algorithm is that the experience pool is filled after the normal execution of the action to obtain the reward. It will extract specific data from the previous state sequence and fill it into the experience pool to form new sampled data, and then repeat the above steps again until convergence, which not only reduces the degree of correlation between the data, but also utilizes the previous useful information. data, which combines the algorithm advantages of different strategies and same strategies, thus improving the convergence performance.

Through the comprehensive strategy module provided by the core algorithm component provided in this embodiment, through the comprehensive strategy mechanism, compared with the different strategy algorithm, it has stronger adaptability to the transformation of the environment; on the other hand, compared with the same strategy algorithm, it has Better computational efficiency and convergence performance.

The comprehensive strategy module of this embodiment integrates a data-driven reinforcement learning algorithm for a comprehensive strategy, so that the method can be applied not only to general auxiliary mobile robot application scenarios, but also to complex scenarios with strong nonlinearity and environmental changes. At the same time, by testing the components, the stability, robustness and generalization ability of the algorithm can be easily tested, and the linkage of the optimized components can be easily adjusted and optimized.

Other features and advantages of the present invention will become more apparent upon reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings.

As can be seen from Figure 1, the assisted mobile robot navigation control system based on the learning component library has a total of eight components. S11 is the initialization component, including the S111 state space design module, the S112 behavior space design module, and the S113 reward function design module; state space Design and behavior space design are the first steps in the reinforcement learning workflow. According to task requirements, it can be designed as discrete space or continuous space; reward function design needs to be combined with specific path planning routes, and its forms mainly include end-state rewards, single Step reward, continuous reward, nonlinear reward, etc.

S12 is the environment modeling component, including S121 sensor data processing module, S122 mobile robot positioning module, S123 reinforcement learning environment modeling module; the sensor data processing module is used to read and process the sensor data carried by the mobile robot, such as for vision The sensor can enhance the observation information through processing algorithms such as noise reduction and dehazing; the mobile robot positioning module is used to locate the global position of the robot in real time; the reinforcement learning environment modeling module is used to establish virtual and intelligent body interaction environment.

S13 is the path planning component, which aims to use the environmental information to plan an optimal mobile robot motion path in real time according to the target task requirements; it mainly includes some commonly used path planning algorithm modules, such as the heuristic path planning module of S131, S132 Artificial Potential Field Path Planning Module, S133 Machine Learning Path Planning Module, etc.

S14 is the core algorithm component, and there are three main modules, S141 same-strategy algorithm module, S142 different-strategy algorithm module, S143 comprehensive strategy algorithm module; this component includes the packaging of various reinforcement learning algorithms, and integrates a combination of same-strategy and The data-driven comprehensive strategy reinforcement learning algorithm with different strategy advantages is used to deal with task scenarios with strong nonlinearity and strong adaptability to environmental changes.

S15 is a visualization component, including S151 path planning visualization module, S152 learning curve visualization module, S153 navigation control error visualization module, and S154 actuator numerical visualization module.

S16 is an optimization component, which is mainly used to optimize the stability and robustness of the algorithm and improve the generalization ability; it includes S161 parameter optimization module, S162 regularization module, etc.

S17 is a test component, which tests the performance of the algorithm by adding disturbance to the environment, such as the S171 dynamic obstacle module, the S172 wind disturbance module, and the S173 water flow disturbance module.

S18 is the underlying control algorithm component. This component is a benchmark component that can be compared with reinforcement learning algorithms. It can be used directly for actual mobile robot control, or it can be used in combination with reinforcement learning algorithm components to improve the algorithm performance of actual mobile robots; mainly There are S181 linear quadratic optimization control module, S182 model prediction control module, and S183 feedback linearization control module.

Embodiment 4. An auxiliary mobile robot navigation control method based on a learning component library. The method is based on the learning component library-based auxiliary mobile robot navigation control system provided in Implementation 3. The control method includes: establishing an auxiliary mobile robot. The navigation control learning component library of mobile robots; according to the characteristics of different auxiliary mobile robot drive configurations, sensor schemes, etc., select the state space and action space corresponding to different mobile robot types from the initialization components; according to the needs of real use scenarios, build Simulate the environment, and select the required path planning algorithm to plan the optimal navigation path; according to the characteristics of the actual task, set up the reward function; and select one of the same strategy, different strategy or comprehensive strategy algorithm combining the advantages of both from the algorithm components. As a learning algorithm, configure the hyperparameters of the algorithm; select the required regularization method from the optimization component according to the disturbance of the usage scenario; check the training effect, and add the disturbance component according to the needs of the usage scenario to test all the Select the stability, robustness and generalization ability of the algorithm; according to the control requirements, the optimization components can be used to adjust the parameters of the main components to improve the performance of the navigation control algorithm.

Embodiment 5. An auxiliary mobile robot navigation control method based on a learning component library, based on the auxiliary mobile robot navigation control system based on the learning component library provided in the third embodiment, and the method provided in this embodiment is directly based on the learning component library. The core algorithm component, which directly outputs controller instructions. The following will be introduced according to Figure 2 of the specification. This example will take a disinfection mobile robot equipped with vision sensors, lidar sensors, and positioning sensors to autonomously move to the target position and disinfect in indoor public places as an example.

As shown in Figure 2, the present invention directly passes through the core algorithm component of the learning component library, outputs the controller instruction, and completes the construction steps of the reinforcement learning closed-loop control:

Step S21, according to the chassis configuration and driving mode of the disinfection mobile robot, combined with the motion model, initialize the state space and action space of the robot by using a pre-built environment modeling component;

Step S22 is divided into two situations, one is to carry out the simulation research of the navigation control algorithm of the disinfection robot, at this time, only need to establish a reinforcement learning environment according to its motion model, and then the next step can be entered;

If it is applied to the actual control scene, the sensor data processing module and the positioning module need to be called to obtain the environmental information and the observation value of the disinfection mobile robot state, and at the same time obtain the positioning signal and update the status information;

Step S23, according to the environment information, obtain the relative position of the obstacle and the position of the disinfection mobile robot itself through the environment modeling component, and use the pre-built path planning component to call the path planning component to obtain the optimal path;

Step 24-A adjusts the reward function of the navigation control algorithm established according to the environment modeling component according to the path planning result;

Step 24-B, jointly define the action space, state space, reward function and reinforcement learning environment, select the core algorithm module from the pre-built core algorithm group, select and determine the reinforcement learning algorithm, conduct training, and output the controller through the underlying control module or directly Instruction completion reinforcement learning closed-loop control;

Step S25 is an optional step, using the underlying control algorithm component to select the mainstream control algorithm benchmark for final comparison;

Step S26 tests and evaluates S24, for example, in a simulation environment, the behavior of the disinfection robot when it encounters a pedestrian can be tested from the test component dynamic obstacle module;

S27 verifies whether the navigation control effect meets the requirements, otherwise, the optimization component can be used to optimize the reinforcement learning algorithm determined by the core algorithm component to improve the task execution effect.

In the same task, S24-S28 are repeatedly executed until the disinfection mobile robot obtains the ideal execution effect in the navigation control task. Use the visualization component to visualize the output values of the core algorithm component and the test component to monitor the learning and training process in real time.

Embodiment 6. An auxiliary mobile robot navigation control method based on a learning component library, based on the auxiliary mobile robot navigation control system based on the learning component library provided in the third embodiment, as shown in FIG. 3, the present invention provides a learning component. The core algorithm of the library is combined with the traditional control method to build a closed-loop control learning component system construction steps:

Step S31, initialize the state space and action space of the robot by using a pre-built initialization component;

Step S32, call the environment modeling component to obtain the sensor data carried by the mobile robot and the global position data where the mobile robot is located (optionally, the environment modeling component includes a sensor data processing module and a mobile robot positioning module, which are obtained through the sensor data processing module. The sensor data carried by the mobile robot, the global position data of the mobile robot is obtained through the mobile robot positioning module);

Step S33, combining the task requirements, using the environment modeling component to design the reward function, determining the core algorithm through the core algorithm component, and focusing on learning the motion strategy of the mobile robot by inputting the observed value, as the reference input of the traditional controller;

Steps S34 and S35, adding a traditional controller, constructing a closed loop of navigation control, combining the underlying control algorithm component or directly making the controller command output, after executing the command, obtain the current mobile robot information through the environment modeling module again, repeat the above steps to complete Reinforcement learning closed-loop control;

At the same time, the test component is used to evaluate and test the algorithm, and the sensor processing module output state observation value is fed back in real time; step S36 judges whether the control requirement is met;

Step S37: On the basis of the previous step, if the optimization needs to be continued, the optimization component is called to perform parameter optimization and regularization to perform algorithm optimization, so as to improve the task execution effect.

Likewise, S33-S38 are repeatedly executed until the sterilizing mobile robot obtains the ideal execution effect in the navigation control task. Use the visualization component to visualize the output values of the core algorithm component and the test component.

The method of combining the underlying control algorithm components is to use the reinforcement learning output (x, y, psi..) upper layer instructions as the input of the controller, and the controller can track by calling the underlying control algorithm components to output the actuator instructions. Such a layered architecture can effectively reduce the data latitude of reinforcement learning and improve efficiency. The algorithm encapsulated by the underlying control algorithm component may include LQR, PID, MPC or Backstepping.

In addition, it should be noted that, compared with the first method, the underlying control component in Figure 2 is only used as a link for comparing performance, and the underlying control algorithm component in Figure 3 is used as a part of the entire learning closed loop. The loop is used to output the underlying control instructions. The advantage of this is that it can combine the advantages of traditional control and reinforcement learning, reduce the dimensions of state space and action space, and make reinforcement learning focus on learning complex behavior strategies. Combine the advantages of traditional mainstream control algorithms , which improves the stability and performance of the algorithm, but on the other hand, the introduction of traditional control algorithms increases the overall algorithm complexity. In particular, by using the components in the control learning component library, various closed-loop learning control systems other than the above two can be constructed according to the task requirements.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowcharts and/or block diagrams, and combinations of flows and/or blocks in the flowcharts and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in one or more of the flowcharts and/or one or more blocks of the block diagrams.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions An apparatus implements the functions specified in a flow or flows of the flowcharts and/or a block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in one or more of the flowcharts and/or one or more blocks of the block diagrams.

The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are only illustrative rather than restrictive. Under the inspiration of the present invention, without departing from the scope of protection of the present invention and the claims, many forms can be made, which all belong to the protection of the present invention.

Claims (10)

1. An auxiliary type mobile robot navigation control system based on a learning component library is characterized by comprising the learning component library, wherein the learning component library comprises: the system comprises an initialization component, an environment modeling component, a path planning component, a core algorithm component, a testing component, an optimization component and a visualization component; the initialization component is used for completing initialization of a state space and an action space corresponding to a specific mobile robot type and setting up a reward function; the environment modeling component is used for reading and processing sensor data carried by the mobile robot, determining global position data of the positioning robot and establishing a virtual environment interacting with the mobile robot when a simulation task is performed; the path planning component is used for providing a selectable path planning algorithm to realize an optimal navigation path; the core algorithm component is used for providing a reinforcement learning algorithm with a plurality of strategies, so that the instruction of the output controller is used for finishing the reinforcement learning closed-loop control; the testing component is used for providing a disturbance method in a simulation environment for selection so as to test the performance of the reinforcement learning algorithm determined by the core algorithm component; the optimization component is used for providing a selected optimization algorithm to adjust the selected parameters of the reinforcement learning algorithm determined by the core algorithm component so as to improve the performance of the navigation control algorithm; the visualization component is used for visualizing the output values of the core algorithm component and the test component during simulation or actual learning tasks.

2. The system of claim 1, wherein the core component library comprises a same strategy module, a different strategy module and a comprehensive strategy module, the same strategy module is used for encapsulating reinforcement learning algorithms of the same strategy, and the different strategy module is used for encapsulating reinforcement learning algorithms of the different strategies; the comprehensive strategy module is used for packaging a comprehensive strategy algorithm, and the comprehensive strategy algorithm is a data-driven strengthening algorithm for synthesizing the same strategy and the different strategies.

3. The system of claim 2, wherein the integrated strategy algorithm comprises: the new learning strategy is fed back to the mobile robot system, and specific system data is collected to optimize the adaptability of the reinforcement learning algorithm; and combining the newly collected data with the experience data played back in the past, and learning again to finally determine the reinforcement learning algorithm.

4. The system of claim 1, wherein the system further comprises: the bottom layer control algorithm component can be directly used for providing a reference component for comparison with the reinforcement learning algorithm, and can also be combined with the upper layer reinforcement learning algorithm to build a closed-loop control reinforcement learning system from a state directly to an actuator instruction.

5. The system of claim 1, wherein the environment modeling component comprises: the system comprises a sensor data processing module, a mobile robot positioning module and a reinforcement learning environment modeling module, wherein the sensor data processing module is used for reading and processing sensor data carried by a mobile robot, and the mobile robot positioning module is used for positioning global position data of the robot in real time; the reinforcement learning environment modeling module is used for establishing a virtual environment interacting with the mobile robot when a simulation task is carried out.

6. A learning component library-based auxiliary mobile robot navigation control system as claimed in claim 1 wherein the optimization component provides alternative optimization algorithms including regularization algorithms L1 and L2, entropy regularization algorithms and/or early stop algorithms.

7. The navigation control system of an auxiliary mobile robot based on a learning component library as claimed in claim 1, wherein the path planning component and the core algorithm component are respectively provided with a performance evaluation function module for providing a performance evaluation function to realize performance evaluation of parameter adjustment and algorithm selection of the path planning component and the core algorithm component.

8. An auxiliary mobile robot navigation control method based on a learning component library is characterized in that the method is based on an auxiliary mobile robot navigation control system of the learning component library; the system includes a learning component library, the learning component library including: the system comprises an initialization component, an environment modeling component, a path planning component, a core algorithm component, a testing component, an optimization component, a visualization component and a bottom control algorithm component; the initialization component is used for completing initialization of a state space and an action space corresponding to a specific mobile robot type and setting up a reward function; the environment modeling component is used for reading and processing sensor data carried by the mobile robot, determining global position data of the positioning robot and establishing a virtual environment interacting with the mobile robot when a simulation task is performed; the path planning component is used for providing a selectable path planning algorithm to realize optimal navigation path planning; the core algorithm component is used for providing a plurality of reinforcement learning algorithms for selection, so that the instruction of the output controller is used for finishing reinforcement learning closed-loop control; the test component is used for providing a selected perturbation method for performance test under different working conditions so as to test the performance of the reinforcement learning algorithm determined by the core algorithm component; the optimization component is used for providing a selected optimization algorithm to adjust the selected parameters of the reinforcement learning algorithm determined by the core algorithm component so as to improve the performance of the navigation control algorithm; the visualization component is used for visualizing the output numerical values of the core algorithm component and the test component; the underlying control algorithm component is used for providing a benchmark component which is compared with a reinforcement learning algorithm;

The method comprises the following steps:

selecting a state space and an action space corresponding to a specific mobile robot type from a pre-constructed initialization component, and setting a reinforcement learning reward function to complete initialization;

constructing a reinforcement learning simulation environment by utilizing a pre-constructed environment modeling component; acquiring the relative position of the barrier and the position of the mobile robot through an environment modeling component, and selecting a required path planning algorithm by utilizing a pre-constructed path planning component to plan an optimal navigation path; adjusting a reward function of a navigation control algorithm according to a path planning result;

selecting and determining a reinforcement learning algorithm from a pre-constructed core algorithm component, combining a defined action space, a state space, a reward function and a reinforcement learning simulation environment, and selecting a core algorithm module for training; the bottom layer control module or the controller instruction is directly output to act, then the relative position of the barrier and the position of the mobile robot are obtained through the environment modeling component again, and the steps are repeated to complete the reinforcement learning closed-loop control;

selecting a perturbation method from the test component, and testing the performance of the reinforcement learning algorithm selected and determined from the core algorithm component;

Selecting and determining an optimization algorithm from the optimization components to adjust selected parameters of the reinforcement learning algorithm determined by the core algorithm component so as to improve the performance of the navigation control algorithm;

and the visualization component is used for realizing visualization of the output values of the core algorithm component and the test component so as to monitor the learning training process in real time.

9. A method for controlling navigation of an auxiliary mobile robot based on a learning component library, wherein the method is based on a system for controlling navigation of an auxiliary mobile robot based on a learning component library, the system comprises a learning component library, and the learning component library comprises: the system comprises an initialization component, an environment modeling component, a path planning component, a core algorithm component, a testing component, an optimization component, a visualization component and a bottom control algorithm component; the initialization component is used for completing initialization of a state space and an action space corresponding to a specific mobile robot type and setting up a reward function; the environment modeling component is used for reading and processing sensor data carried by the mobile robot, determining global position data of the positioning robot and establishing a virtual environment interacting with the mobile robot when a simulation task is performed; the path planning component is used for providing a selectable path planning algorithm to realize optimal navigation path planning; the core algorithm component is used for providing a plurality of reinforcement learning algorithms for selection, so that the instruction of the output controller is used for finishing reinforcement learning closed-loop control; the optimization component is configured to provide an optimization method to enable optimization of a reinforcement learning algorithm; the test component is used for providing a selected perturbation method for performance test under different working conditions so as to test the performance of the reinforcement learning algorithm determined by the core algorithm component; the optimization component is used for providing a selected optimization algorithm to adjust the selected parameters of the reinforcement learning algorithm determined by the core algorithm component so as to improve the performance of the navigation control algorithm; the visualization component is used for visualizing the output numerical values of the core algorithm component and the test component; the underlying control algorithm component is used to provide a baseline component as a comparison to a reinforcement learning algorithm

The method comprises the following steps:

selecting a state space and an action space corresponding to a specific mobile robot type from a pre-constructed initialization component, and setting a reinforcement learning reward function to complete initialization;

calling an environment modeling component to obtain sensor data carried by the mobile robot and global position data where the mobile robot is located;

combining the defined action space, state space, reward function, sensor data carried by the mobile robot and global position data of the mobile robot, selecting and determining a reinforcement learning algorithm from a pre-constructed core algorithm component, combining a bottom control algorithm component or directly outputting a controller instruction, obtaining current mobile robot information through an environment modeling module again after executing the instruction, and repeating the steps to complete reinforcement learning closed-loop control;

selecting a disturbance method from the test assembly, performing algorithm evaluation and test by using the test assembly, feeding back an output state observation value of the sensor processing module in real time, and judging whether the control requirement is met;

selecting and determining an optimization algorithm from the optimization component, and adjusting the selected parameters of the reinforcement learning algorithm determined by the core algorithm component until the mobile robot obtains a preset execution effect in the navigation control task;

And visualizing the output numerical values of the core algorithm component and the test component by utilizing the visualization component.

10. The navigation control method for the auxiliary mobile robot based on the learning component library of claim 9, wherein performance evaluation function modules are respectively arranged in the path planning component and the core algorithm component, the method further comprises the steps of determining a performance evaluation function by using the performance evaluation function modules, performing performance evaluation on parameter adjustment and algorithm selection of the path planning component and the core algorithm component, and visualizing the evaluation result of the performance evaluation function by using the visualization component.

Images