TWI876514B - Curriculum reinforcement learning unmanned vehicle autonomous navigation method and system and heterogeneous robot - Google Patents

Abstract

The invention discloses a curriculum reinforcement learning unmanned vehicle autonomous navigation method applied to a heterogeneous robot. The method includes: (a) performing unmanned vehicle autonomous navigation training for the heterogeneous robot through the curriculum reinforcement learning method, so that the heterogeneous robot can learn the function of real-time obstacle avoidance; and (b) controlling the heterogeneous robot that have completed the unmanned vehicle autonomous navigation training and learned the function of real-time obstacle avoidance to perform unmanned vehicle autonomous navigation tasks in virtual environment and real environment respectively.

Description

The course strengthens the learning of unmanned vehicle autonomous navigation methods and systems and heterogeneous robots

The present invention relates to autonomous navigation, and more particularly to a curriculum-enhanced learning method and system for autonomous navigation of unmanned vehicles and heterogeneous robots.

Generally speaking, robot navigation usually requires the ability to avoid obstacles and plan a path to a target. However, traditional robot navigation methods have limited performance when dealing with heterogeneous robots with different sensor modes or vehicle dynamics.

To solve this problem, deep reinforcement learning has been widely used. However, once dealing with a more complex exploration space, the convergence speed of deep reinforcement learning models may be very slow, which needs further improvement.

In view of this, the present invention proposes a curriculum-enhanced learning method and system for autonomous navigation of unmanned vehicles and heterogeneous robots to effectively solve the above-mentioned problems encountered by the prior art.

According to a preferred embodiment of the present invention, a method for learning autonomous navigation of unmanned vehicles through curriculum reinforcement learning is applied to a heterogeneous robot, which includes the following steps: (a) training the heterogeneous robot for autonomous navigation of unmanned vehicles through curriculum reinforcement learning so that the heterogeneous robot can learn the real-time obstacle avoidance function; and (b) controlling the heterogeneous robot that has completed the autonomous navigation training of unmanned vehicles and learned the real-time obstacle avoidance function to perform autonomous navigation tasks of unmanned vehicles in virtual environments and real environments respectively.

In one embodiment, in step (b), the unmanned vehicle autonomous navigation mission performed by the heterogeneous robot includes: transferring from the autonomous navigation of the land unmanned vehicle to the autonomous navigation of the water unmanned vehicle, and improving the obstacle avoidance and wind and wave resistance of the water unmanned vehicle autonomous navigation.

In one embodiment, heterogeneous robots have different sensor modalities and vehicle dynamics.

In one embodiment, the curriculum reinforcement learning method in step (a) uses a hierarchically integrated dynamic and perception model and a sample-efficient deep reinforcement learning model to solve the complex autonomous control problem of a heterogeneous robot in a dense obstacle environment using a reinforcement learning strategy trained with different fields of view.

In one embodiment, in step (a), the curriculum reinforcement learning method gradually increases the difficulty of the unmanned vehicle autonomous navigation training in multiple stages as the heterogeneous robot learns.

In one embodiment, the training difficulty increases gradually from basic navigation tasks such as obstacle avoidance and target navigation to advanced navigation tasks such as sensor modalities and vehicle dynamics.

In one embodiment, in step (b), the virtual environment includes a land environment and an aquatic environment.

In one embodiment, in step (b), the real environment includes a land environment and an aquatic environment.

Another preferred embodiment of the present invention is a curriculum-enhanced learning unmanned vehicle autonomous navigation system. In this embodiment, the curriculum-enhanced learning unmanned vehicle autonomous navigation system includes: a heterogeneous robot; a training device for connecting with the heterogeneous robot and performing unmanned vehicle autonomous navigation training on the heterogeneous robot through a curriculum-enhanced learning method, so that the heterogeneous robot learns the real-time obstacle avoidance function; and a control device for connecting with the heterogeneous robot that has completed the unmanned vehicle autonomous navigation training and learned the real-time obstacle avoidance function and controlling the heterogeneous robot to perform unmanned vehicle autonomous navigation tasks in a virtual environment and a real environment respectively.

In one embodiment, the control device controls the heterogeneous robot to perform unmanned vehicle autonomous navigation tasks including: transferring from the autonomous navigation of the land unmanned vehicle to the autonomous navigation of the water unmanned vehicle, and improving the obstacle avoidance and wind and wave resistance of the water unmanned vehicle during autonomous navigation.

In one embodiment, heterogeneous robots have different sensor modalities and vehicle dynamics.

In one embodiment, the curriculum reinforcement learning method adopted by the training device is to solve the complex autonomous control problem of heterogeneous robots in a dense obstacle environment by using a reinforcement learning strategy trained with different fields of view through a hierarchical integrated dynamics and perception model and a sample-effective deep reinforcement learning model.

In one embodiment, the curriculum reinforcement learning method adopted by the training device is to gradually increase the training difficulty of the unmanned vehicle autonomous navigation training in multiple stages along with the learning progress of the heterogeneous robot.

In one embodiment, the training difficulty increases gradually from basic navigation tasks such as obstacle avoidance and target navigation to advanced navigation tasks such as sensor modalities and vehicle dynamics.

In one embodiment, the virtual environment includes a land environment and an aquatic environment.

In one embodiment, the real environment includes a land environment and an aquatic environment.

Another preferred embodiment of the present invention is a heterogeneous robot. In this embodiment, the heterogeneous robot is applied to a curriculum-enhanced learning unmanned vehicle autonomous navigation system. The curriculum-enhanced learning unmanned vehicle autonomous navigation system also includes a training device and a control device. The heterogeneous robot includes: a connection module, which is used to connect with a training device and a control device respectively to receive training instructions from the training device and control instructions from the control device respectively; a training module, which is coupled to the connection module, and is used to perform unmanned vehicle autonomous navigation training according to the training instructions and the curriculum reinforcement learning method, so that the heterogeneous robot can learn the real-time obstacle avoidance function and issue a training completion notification; and a control module, which is coupled to the connection module and the training module. When the control module receives the training completion notification, the control module controls the heterogeneous robot according to the control instructions to perform the unmanned vehicle autonomous navigation tasks in the virtual environment and the real environment respectively.

In one embodiment, the control module controls the heterogeneous robot to perform unmanned vehicle autonomous navigation tasks including: transferring from the autonomous navigation of the land unmanned vehicle to the autonomous navigation of the water unmanned vehicle, and improving the obstacle avoidance and wind and wave resistance of the water unmanned vehicle during autonomous navigation.

In one embodiment, heterogeneous robots have different sensor modalities and vehicle dynamics.

In one embodiment, a curriculum reinforcement learning method uses a layered integrated dynamics and perception model and a sample-efficient deep reinforcement learning model to solve the complex autonomous control problem of a heterogeneous robot in a dense obstacle environment using a reinforcement learning strategy trained with different fields of view.

In one embodiment, the curriculum reinforcement learning method gradually increases the difficulty of the unmanned vehicle autonomous navigation training in multiple stages as the heterogeneous robot learns.

In one embodiment, the training difficulty increases gradually from basic navigation tasks such as obstacle avoidance and target navigation to advanced navigation tasks such as sensor modalities and vehicle dynamics.

In one embodiment, the virtual environment includes a land environment and an aquatic environment.

In one embodiment, the real environment includes a land environment and an aquatic environment.

The first preferred embodiment of the present invention is a method for learning autonomous navigation of unmanned vehicles through curriculum enhancement. In this embodiment, the method for learning autonomous navigation of unmanned vehicles through curriculum enhancement is applied to heterogeneous robots, and the heterogeneous robots have different sensor modalities and vehicle dynamics, but the invention is not limited thereto.

It should be noted that the curriculum-enhanced learning method for autonomous navigation of unmanned vehicles for heterogeneous robots disclosed in the present invention adopts curriculum-enhanced learning methods and related designs such as wind and wave resistance during autonomous navigation on water, so as to effectively improve the effect and speed of autonomous target navigation training of unmanned vehicles for heterogeneous robots on the ground and on the water surface, so as to avoid obstacles on the ground and on the water surface.

Please refer to FIG. 1, which shows a flow chart of the course-enhanced learning method for autonomous navigation of unmanned vehicles in this embodiment. As shown in FIG. 1, the course-enhanced learning method for autonomous navigation of unmanned vehicles includes the following steps:

Step 100: Conduct unmanned vehicle autonomous navigation training for heterogeneous robots through curriculum reinforcement learning methods;

Step 200: Determine whether the heterogeneous robot has completed unmanned vehicle autonomous navigation training and learned the real-time obstacle avoidance function;

Step 300: If the result of the determination in step 200 is yes, the heterogeneous robot that has completed the unmanned vehicle autonomous navigation training and learned the real-time obstacle avoidance function is controlled to perform the unmanned vehicle autonomous navigation task in the virtual environment and the real environment respectively; and

If the determination result of step 200 is no, step 100 is executed again.

In practical applications, the curriculum reinforcement learning method in step 100 can use a hierarchically integrated dynamic and perception model and a sample-efficient deep reinforcement learning model to use reinforcement learning strategies trained with different fields of view to solve complex autonomous control problems of heterogeneous robots in dense obstacle environments, but is not limited thereto.

It should be noted that in step 100, the curriculum reinforcement learning method can gradually increase the difficulty of the unmanned vehicle autonomous navigation training in multiple stages as the heterogeneous robot learns. For example, the difficulty of the unmanned vehicle autonomous navigation training can be gradually increased from the basic navigation tasks of obstacle avoidance and target navigation to the advanced navigation tasks of sensor mode and vehicle dynamics, but is not limited thereto.

In actual applications, in step 200, the virtual environment may include a land environment and a water environment and the real environment may include a land environment and a water environment, but not limited thereto. The unmanned vehicle autonomous navigation tasks performed by the heterogeneous robot may include: transferring from the autonomous navigation of the land unmanned vehicle to the autonomous navigation of the water unmanned vehicle, and improving the obstacle avoidance and wind and wave resistance of the unmanned vehicle autonomous navigation on the water, but not limited thereto.

The second preferred embodiment of the present invention is a curriculum-enhanced learning unmanned vehicle autonomous navigation system. In this embodiment, the curriculum-enhanced learning unmanned vehicle autonomous navigation system is applied to heterogeneous robots, and the heterogeneous robots have different sensor modes and vehicle dynamics, but are not limited thereto.

It should be noted that the curriculum-enhanced learning unmanned vehicle autonomous navigation system for heterogeneous robots disclosed in the present invention adopts curriculum-enhanced learning methods and related designs such as wind and wave resistance during autonomous navigation on water, so as to control the heterogeneous robots to perform unmanned vehicle autonomous target navigation on the ground and on the water surface and effectively avoid obstacles on the ground and on the water surface.

Please refer to FIG. 2 , which shows a schematic diagram of the curriculum-enhanced learning unmanned vehicle autonomous navigation system in this embodiment. As shown in FIG. 2 , the curriculum-enhanced learning unmanned vehicle autonomous navigation system CRS includes a heterogeneous robot HR, a training device TD, and a control device CD. When the training device TD is connected to the heterogeneous robot HR, the training device TD transmits a training instruction TC to the heterogeneous robot HR and performs unmanned vehicle autonomous navigation training on the heterogeneous robot HR through the curriculum-enhanced learning method, so that the heterogeneous robot HR learns the real-time obstacle avoidance function. When the heterogeneous robot HR has completed the unmanned vehicle autonomous navigation training and learned the real-time obstacle avoidance function, the control device CD is connected to the heterogeneous robot HR and controls the heterogeneous robot HR to perform the unmanned vehicle autonomous navigation tasks in the virtual environment and the real environment respectively according to the control command CC.

In practical applications, the curriculum reinforcement learning method adopted by the training device TD is to solve the complex autonomous control problem of the heterogeneous robot HR in a dense obstacle environment by using a layered integrated dynamics and perception model and a sample-effective deep reinforcement learning model using reinforcement learning strategies trained with different fields of view, but not limited to this.

It should be noted that the curriculum reinforcement learning method adopted by the training device TD can gradually increase the training difficulty of the unmanned vehicle autonomous navigation training in multiple stages as the learning progress of the heterogeneous robot HR. For example, the training difficulty of the unmanned vehicle autonomous navigation training can be gradually increased from the basic navigation tasks of obstacle avoidance and target navigation to the advanced navigation tasks of sensor mode and vehicle dynamics, but not limited to this.

In practical applications, the virtual environment may include a land environment and a water environment and the real environment may include a land environment and a water environment, but not limited thereto. The autonomous navigation tasks of the unmanned vehicle performed by the control device CD controlling the heterogeneous robot HR may include: transferring from the autonomous navigation of the land unmanned vehicle to the autonomous navigation of the water unmanned vehicle, and improving the obstacle avoidance and wind and wave resistance of the autonomous navigation of the water unmanned vehicle, but not limited thereto.

A third preferred embodiment of the present invention is a heterogeneous robot. In this embodiment, the heterogeneous robot is applied to curriculum reinforcement learning of unmanned vehicle autonomous navigation system and the heterogeneous robot has different sensor modes and vehicle dynamics, but is not limited thereto.

Please refer to FIG. 3, which shows a schematic diagram of the heterogeneous robot in this embodiment applied to the curriculum-enhanced learning unmanned vehicle autonomous navigation system. As shown in FIG. 3, the curriculum-enhanced learning unmanned vehicle autonomous navigation system CRS also includes a training device TD and a control device CD. The heterogeneous robot HR applied to the curriculum-enhanced learning unmanned vehicle autonomous navigation system CRS includes a connection module CNM, a training module TM and a control module CTM.

When the connection module CNM is connected to the training device TD and the control device CD respectively, the connection module CNM receives the training command TC from the training device TD and the control command CC from the control device CD respectively. The training module TD is coupled to the connection module CNM to perform unmanned vehicle autonomous navigation training on the heterogeneous robot HR according to the training command TC and the curriculum reinforcement learning method, so that the heterogeneous robot HR learns the real-time obstacle avoidance function and issues a training completion notification FN.

The control module CTM is coupled to the connection module CNM and the training module TM. When the control module CD receives the training completion notification FN sent by the training module TD, the control module CTM controls the heterogeneous robot HR to perform the unmanned vehicle autonomous navigation tasks in the virtual environment and the real environment respectively according to the control instruction CC, but not limited to this.

In practical applications, the curriculum reinforcement learning method adopted by the training module TD can solve the complex autonomous control problem of heterogeneous robots HR in dense obstacle environments by using reinforcement learning strategies trained with different fields of view through a hierarchical integrated dynamics and perception model and a sample-effective deep reinforcement learning model, but is not limited to this.

It should be noted that the curriculum reinforcement learning method can gradually increase the difficulty of the unmanned vehicle autonomous navigation training in multiple stages as the learning progress of the heterogeneous robot HR. For example, the training difficulty can be gradually increased from the basic navigation tasks of obstacle avoidance and target navigation to the advanced navigation tasks of sensor modality and vehicle dynamics, but not limited to this.

In actual applications, the virtual environment may include a land environment and a water environment and the real environment may include a land environment and a water environment, but not limited thereto. The control module CTM controls the unmanned vehicle autonomous navigation tasks performed by the heterogeneous robot HR to include: transferring from the autonomous navigation of the land unmanned vehicle to the autonomous navigation of the water unmanned vehicle, and improving the obstacle avoidance and wind and wave resistance of the unmanned vehicle autonomous navigation on the water, but not limited thereto.

Compared to the prior art, the curriculum-enhanced learning unmanned vehicle autonomous navigation method and system disclosed in the present invention for heterogeneous robots adopts curriculum-enhanced learning methods and related designs such as wind and wave resistance during autonomous navigation on water, so as to effectively improve the effect and speed of unmanned vehicle autonomous target navigation training for heterogeneous robots in different environments (such as on the ground or on the water), so as to effectively avoid obstacles on the ground and on the water.

100: Steps 200: Steps 300: Steps CRS: Curriculum Enhanced Learning Unmanned Vehicle Autonomous Navigation System TD: Training Device CD: Control Device HR: Heterogeneous Robot TC: Training Command CC: Control Command CNM: Connection Module CTM: Control Module TM: Training Module FN: Training Completion Notice

FIG. 1 is a flow chart showing a method for curriculum-enhanced learning of autonomous navigation of unmanned vehicles according to a first preferred embodiment of the present invention.

FIG. 2 is a schematic diagram of a curriculum-enhanced learning unmanned vehicle autonomous navigation system according to a second preferred embodiment of the present invention.

FIG. 3 is a schematic diagram of a heterogeneous robot according to a third preferred embodiment of the present invention.

100: Steps

200: Steps

300: Steps

Claims (18)

A curriculum-enhanced learning method for autonomous navigation of unmanned vehicles is applied to a heterogeneous robot, comprising the following steps: (a) training the heterogeneous robot for autonomous navigation of unmanned vehicles through a curriculum-enhanced learning method, so that the heterogeneous robot learns a real-time obstacle avoidance function; and (b) controlling the heterogeneous robot that has completed the autonomous navigation training of the unmanned vehicle and learned the real-time obstacle avoidance function to execute a virtual environment respectively. and autonomous navigation tasks of unmanned vehicles in real environments; wherein the heterogeneous robots have different sensor modalities and vehicle dynamics, and the curriculum reinforcement learning method in step (a) uses a layered integrated dynamics and perception model and a sample-effective deep reinforcement learning model to use a reinforcement learning strategy trained with different fields of view to solve the complex autonomous control problem of the heterogeneous robot in a dense obstacle environment. The course-enhanced learning method of unmanned vehicle autonomous navigation as described in claim 1, wherein in step (b), the unmanned vehicle autonomous navigation task performed by the heterogeneous robot includes: transferring from the autonomous navigation of the land unmanned vehicle to the autonomous navigation of the water unmanned vehicle, and improving the obstacle avoidance and wind and wave resistance during the autonomous navigation of the water unmanned vehicle. As described in claim 1, the curriculum-enhanced learning method for autonomous navigation of unmanned vehicles, wherein in step (a), the curriculum-enhanced learning method gradually increases the difficulty of the training of autonomous navigation of the unmanned vehicle in multiple stages as the learning progress of the heterogeneous robot progresses. A course as described in claim 3 to enhance learning of autonomous navigation methods for unmanned vehicles, wherein the difficulty of training gradually increases from basic navigation tasks of obstacle avoidance and target navigation to advanced navigation tasks of sensor modalities and vehicle dynamics. The course-enhanced learning method for autonomous navigation of unmanned vehicles as described in claim 1, wherein in step (b), the virtual environment includes a land environment and a water environment. The course-enhanced learning method for autonomous navigation of unmanned vehicles as described in claim 1, wherein in step (b), the real environment includes a land environment and a water environment. A curriculum-enhanced learning unmanned vehicle autonomous navigation system, comprising: a heterogeneous robot; a training device, used to connect with the heterogeneous robot and perform unmanned vehicle autonomous navigation training on the heterogeneous robot through a curriculum-enhanced learning method, so that the heterogeneous robot learns a real-time obstacle avoidance function; and a control device, used to connect with the heterogeneous robot that has completed the unmanned vehicle autonomous navigation training and learned the real-time obstacle avoidance function and control the heterogeneous robot The unmanned vehicle autonomous navigation tasks in virtual environment and real environment are respectively performed; wherein the heterogeneous robot has different sensor modalities and vehicle dynamics, and the curriculum reinforcement learning method adopted by the training device is to solve the complex autonomous control problem of the heterogeneous robot in a dense obstacle environment by using a reinforcement learning strategy trained with different fields of view through a hierarchical integrated dynamics and perception model and a sample-effective deep reinforcement learning model. As described in claim 7, the course reinforces the learning of the unmanned vehicle autonomous navigation system, wherein the control device controls the unmanned vehicle autonomous navigation task performed by the heterogeneous robot to include: transferring from the autonomous navigation of the land unmanned vehicle to the autonomous navigation of the water unmanned vehicle, and improving the obstacle avoidance and wind and wave resistance during the autonomous navigation of the water unmanned vehicle. As described in claim 7, the curriculum-enhanced learning unmanned vehicle autonomous navigation system, wherein the curriculum-enhanced learning method adopted by the training device is to gradually increase the training difficulty of the unmanned vehicle autonomous navigation training in multiple stages as the learning progress of the heterogeneous robot. The course as claimed in claim 9 enhances learning of an autonomous navigation system for an unmanned vehicle, wherein the difficulty of the training gradually increases from basic navigation tasks of obstacle avoidance and target navigation to advanced navigation tasks of sensor modalities and vehicle dynamics. The course as described in claim 7 enhances the learning of unmanned vehicle autonomous navigation system, wherein the virtual environment includes a land environment and a water environment. The course as described in claim 7 enhances the learning of unmanned vehicle autonomous navigation system, wherein the real environment includes a land environment and a water environment. A heterogeneous robot is applied to a curriculum-enhanced learning unmanned vehicle autonomous navigation system. The curriculum-enhanced learning unmanned vehicle autonomous navigation system further includes a training device and a control device. The heterogeneous robot includes: a connection module, which is used to connect to the training device and the control device respectively to receive a training instruction from the training device and a control instruction from the control device respectively; a training module, which is coupled to the connection module, and is used to perform unmanned vehicle autonomous navigation training according to the curriculum-enhanced learning method according to the training instruction, so that the heterogeneous robot learns the real-time obstacle avoidance function and issues a training completion notification; and a control module coupled to the connection module and the training module. When the control module receives the training completion notification, the control module controls the heterogeneous robot to perform autonomous navigation tasks of unmanned vehicles in virtual environments and real environments respectively according to the control instructions; wherein the heterogeneous robot has different sensor modalities and vehicle dynamics, and the curriculum reinforcement learning method uses a layered integrated dynamics and perception model and a sample-effective deep reinforcement learning model to use reinforcement learning strategies trained with different fields of view to solve the complex autonomous control problem of the heterogeneous robot in a dense obstacle environment. The heterogeneous robot as described in claim 13, wherein the control module controls the autonomous navigation mission of the unmanned vehicle performed by the heterogeneous robot to include: transferring from the autonomous navigation of the unmanned vehicle on land to the autonomous navigation of the unmanned vehicle on water, and improving the obstacle avoidance and wind and wave resistance during the autonomous navigation of the unmanned vehicle on water. A heterogeneous robot as described in claim 13, wherein the curriculum reinforcement learning method gradually increases the difficulty of the unmanned vehicle autonomous navigation training in multiple stages as the learning progress of the heterogeneous robot. A heterogeneous robot as described in claim 15, wherein the training difficulty gradually increases from basic navigation tasks such as obstacle avoidance and target navigation to advanced navigation tasks such as sensor modalities and vehicle dynamics. A heterogeneous robot as described in claim 13, wherein the virtual environment includes a terrestrial environment and an aquatic environment. A heterogeneous robot as described in claim 13, wherein the real environment includes a land environment and an aquatic environment.

Images