CN116578113A - A method and system for aircraft intelligent cooperative confrontation decision-making based on reinforcement learning - Google Patents

Abstract

The invention discloses a decision-making method and system for intelligent cooperative confrontation of aircraft based on reinforcement learning. The method includes observation value design: simulation modeling of flight dynamics, weapons, radar, etc. The action space of an aircraft, including the target aircraft number and the command value made by the four headings, the command action value includes the angle of attack, roll angle, throttle amount; reward function design: design inventory reward, distance reward/penalty and radar lock item; Reinforcement learning environment design: use training mode and application mode to carry out dynamic control of aircraft and opponents, and realize the data interface function of state, action and reward value; the invention customizes the intelligent cooperative confrontation decision system of aircraft, and the objective function is reasonable. After a certain amount of training, it has a significant effect, can ensure the effectiveness of the model and algorithm, and can be used to formulate a suitable countermeasure strategy for the aircraft.

Description

A method and system for aircraft intelligent cooperative confrontation decision-making based on reinforcement learning

technical field

The invention relates to the technical field of aircraft intelligent cooperative confrontation, in particular to a decision-making method and system for aircraft intelligent cooperative confrontation based on reinforcement learning.

Background technique

With the continuous advancement of military technology, the low-altitude defense system has gradually developed towards the integrated defense of sky, air, sea, submarine and road. This means that the combat method of future air combat is no longer limited to stand-alone combat, but gradually transformed into a confrontation between systems and systems, systems and systems. In this context, the concept and tactics of multi-aircraft cooperative operations continue to develop, and the technology of cooperative task allocation has become a hot issue that researchers are increasingly concerned about. Task allocation refers to the establishment of an allocation plan that maximizes the overall operational efficiency based on information such as task requirements, battlefield environment, and target configuration, and satisfies certain constraints. Appropriate task allocation scheme plays a key role in multi-aircraft cooperative operations.

Realize the customized confrontation of different numbers of aircraft clusters within 4 (inclusive) of each side, and the aircraft type can be specified. In order to make the distributed real-time simulation system achieve a realistic simulation effect, it is often necessary not only to solve various data models in real time in the system, but also to transfer data between systems with a deterministic network with extremely low delay time. Let the various subsystems work in harmony. Traditionally, distributed environment simulation generally uses the solution of "high-speed Ethernet + upper and lower computer" to meet the needs of these two aspects. Limited by the TCP/IP protocol, the traditional Ethernet cannot meet the needs of real-time and deterministic data transmission between real-time simulation subsystems. Although some measures (such as increasing network speed, reducing network load, etc.) can be taken to reduce delay, it is still difficult to fundamentally solve the inherent defects that Ethernet does not have real-time and deterministic, and doing so will increase additional costs .

The JSBSim model is a general-purpose flight dynamics model developed abroad. It can simulate various aircraft types and has high real-time performance, which meets the requirements of this project for the flight dynamics model. JSBSim is an open source cross-platform six-degree-of-freedom nonlinear flight dynamics model. It is written in object-oriented C++ language and supports dynamic modeling of different types of aviation or aerospace vehicles. The aircraft dynamics in the model are expressed by the extensible markup language, without compiling and linking the code, you can build your own six-degree-of-freedom aircraft model and simulate it.

The MAPPO algorithm is a variant that applies the PPO algorithm to multi-agent tasks, and also uses the ActorCritic architecture. The difference is that in the Actor part, in order to further reduce the variance of the advantage function, the generalized advantage estimation function is used instead. Estimate the advantage function in a way similar to TD, and balance its variance and bias, which can significantly reduce the variance of the evaluation under a certain bias.

Contents of the invention

In order to solve the above-mentioned problems in the prior art, the present invention provides a method and system for aircraft intelligent cooperative confrontation decision-making based on reinforcement learning, which can ensure the effectiveness of models and algorithms, and can be used for aircraft to formulate appropriate countermeasures. The specific plan is as follows:

A method for intelligent cooperative confrontation decision-making of aircraft based on reinforcement learning, comprising the following steps:

Step 1: Observation value design: Simulate and model different models, their weapons, and radars based on flight dynamics; the information obtained by the aircraft includes: its own position information, the relative position relationship between itself and the enemy aircraft, the speed of its own aircraft, its own The speed difference between the aircraft and the enemy aircraft;

Step 2: Action space design: Design the action space of each aircraft, including the target aircraft number and the command values made by the four headings. The command action values include angle of attack, roll angle, and throttle, and are written in the following form:

a＝[target,x _t ,y _t ,z _t ,v _t ]

Among them, target represents the number of the target aircraft selected by the aircraft, and x _t , y _t , z _t , v _t represent the command values made by the agent on the four track dimensions respectively;

Step 3: Reward function design: Design inventory rewards, distance rewards/penalties and radar locking items, and the reward function is written as follows:

r＝ω _α r _s +ω _β r _d +ω _γ r _r

Among them, r _s is the survival reward part, r _d is the distance reward/penalty item, r _r is the radar lock item; ω _α , ω _β and ω _γ are the proportions of each part;

Step 4: Reinforcement learning environment design: Use training mode and application mode to dynamically control the aircraft and opponents, and realize the data interface functions of state, action and reward value;

A training round consists of training steps, and each training round contains a finite number of training steps;

The agent uses the state information as the input of the deep neural network, and generates actions after calculation;

After the action is format converted, it forms an executable command for the aircraft and sends it to the environment.

Further, the simulation modeling of the radar is used to simulate the air interception and air combat in the air/air function, including:

Step 1.1: Radar Data Processing Modeling

Step a: Perform data preprocessing on the target detection point trace information;

Step b: Enter the track management module, judge that the point track information is a real new target, then open a new track; if the point track can be associated with the existing track information at this time, it will become a stable motion point track;

Step c: Reversely convert the distance, azimuth, and pitch angle information of the quasi-target in the spherical coordinate system to the three-way position coordinates in the Cartesian coordinate system, thereby performing filtering and prediction, and sending the filtering result to the output interface;

Step d: If the target is lost for a period of time, it will be judged as the end of the track, and the interface will be cleared;

Step 1.2: Data Preprocessing

Combine the transmitter module, receiver module and target processing module in the overall design of the radar system to simplify the processing; for the transmission and reception of the beam, it is determined by the scanning range of the radar system, the transmission power, the target distance, and the target radar scattering cross-sectional area. The theoretical maximum detectable distance under the condition of no electronic jamming, fine weather, and target falling in the clutter-free area.

Further, the track association performed by the track management module is specifically:

Calculate whether the currently detected target point information falls within the set range centered on the point track from the last prediction of the existing track to this moment:

1) If the target detection module detects the target track, and the detected target track fails to be associated with the established track, it is considered to be a new target, and at the same time the radar can successfully associate the target track twice consecutively , then enter the track start;

2) If the target detection module detects the target information, and the association between the quasi-target information and the established track fails, and in the following period of time, the target point track detected by the airborne radar is different from the previously established track. If the association is successful, it will be judged as a false alarm and the track will be terminated;

3) If the target detection module does not detect the target point track, the track association fails at this time, and arrange to execute the small search event at the next moment. When the radar performs the small search and does not detect the lost target, the track will be terminated;

4) If the target detection module detects the target point track, and the detected target point track is successfully associated with the established track in three dimensions of distance, azimuth, and pitch, then it is judged that the point track is a new observation point of this track , that is, to maintain the track.

An aircraft intelligent cooperative confrontation decision-making system based on reinforcement learning, including an upper-layer architecture and a lower-layer architecture;

The upper layer structure includes guidance and control simulation nodes, tactical command simulation nodes, battlefield environment management nodes, tactical deduction nodes and tactical simulators, and data communication and interaction between each simulation node through the DIS network;

The lower layer structure is located in a single tactical simulator. Through the hybrid real-time communication network, the fire control calculation module, flight control calculation module, visual calculation module, visual display module, and instrument calculation module in the simulator are multifunctional The display module and the device control are connected with the acquisition module.

further,

1) The pilot control simulation node is the management and monitoring center of the entire system, used to coordinate and control the operation of the entire system, monitor the system status and record and playback data for evaluation; it communicates with other simulation nodes through the DIS network Interaction, including order issuing, status inquiry and data transmission;

2) The tactical command simulation node is responsible for the command and coordination of the aircraft, and is used to realize the communication and collaboration between the aircraft, to ensure teamwork and to achieve the specified mission objectives; it receives instructions from the guidance control simulation node through the DIS network, Send instructions to the battlefield environment management node and tactical deduction node, and receive data from the aircraft to update the status;

3) The battlefield environment management node builds a simulated air combat environment by establishing a JSBSim dynamic model, and is responsible for managing and monitoring the entire battlefield environment, for realizing environmental modeling and simulation, and positioning and tracking the position of the aircraft on the battlefield; Receive instructions from the guidance control simulation node through the DIS network, update the battlefield environment information, and send it to the tactical deduction node;

4) The tactical deduction node is responsible for tactical deduction and planning, and is used to collect information from other nodes and analyze it, formulate tactical strategies and plan routes; it receives information from battlefield environment management nodes and tactical command simulation nodes through the DIS network information, analyze it and generate a corresponding action plan;

5) The tactical simulator is responsible for simulating the behavior of the aircraft, and is used to predict the behavior and performance of the aircraft through simulation, so as to guide the actions of the aircraft; it receives information from the battlefield environment management node and the tactical deduction node through the DIS network, and based on This information simulates the behavior of the aircraft.

further more,

1) The fire control calculation module is responsible for calculating the fire control data of the aircraft, including missile launch azimuth and elevation angle, target distance, ballistic correction; this module receives data from the flight control calculation module and the visual scene calculation module, through Calculate and generate corresponding fire control data, and send it to the flight control calculation module;

2) The flight control calculation module block is responsible for calculating the flight control data of the aircraft, including flight speed, height and attitude; this module receives data from the fire control calculation module, the visual scene calculation module and the instrument calculation module, through Calculate and generate corresponding flight control data, and send it to the equipment control and acquisition module;

3) The visual scene calculation module is responsible for calculating the scene rendering of the aircraft. This module receives data from the fire control calculation module, the flight control calculation module and the equipment control and acquisition module, generates corresponding image data through calculation, and It is sent to the visual display module;

4) The visual display module is responsible for displaying the visual data generated by the visual calculation module in the form of an image. This module receives data from the visual calculation module and renders it as a visualized image;

5) The instrument calculation module is responsible for calculating various instrument data of the aircraft, including speed, altitude and attitude; this module receives data from the flight control calculation module, generates corresponding instrument data through calculation, and sends it to multiple Function display module;

6) The multifunctional display module is responsible for displaying the instrument data generated by the instrument calculation module, and other data related to the aircraft, including fire control data, mission information, and battery status; this module receives information from the instrument calculation module and equipment control and Collect module data and render it as visualized information;

7) The device control and acquisition module is responsible for communication and data acquisition with various devices of the aircraft.

Furthermore, in the above-mentioned upper-layer architecture and lower-layer architecture, the following four technologies are organically integrated to form a simulation architecture based on HLA and hybrid real-time network:

1) Using DIS distributed management, time advancement mechanism and load balancing control technology

In the upper layer structure of the system, the distributed management and data communication are realized through the DIS network, so that the simulation nodes can work together efficiently; at the same time, the system uses the time advancement mechanism to ensure the accuracy and synchronization of the simulation results, and through Load balance control technology to ensure the stability and reliability of the system;

2) Utilize the high real-time characteristics and deterministic delay of the reflective memory network

In the underlying architecture of the system, reflective memory network technology is used to achieve high real-time and deterministic delays, enabling fast and accurate data interaction and collaborative work between modules;

3) Use RTX's precise clock and preemptive task scheduling mechanism

In the lower layer architecture of the system, the precise clock and preemptive task scheduling mechanism of the RTX real-time operating system are used to enable the system to perform fine control and scheduling of tasks, thereby ensuring the efficiency of data interaction and collaborative work between modules. accuracy;

4) Using the data communication mechanism of CAN bus

In the underlying architecture of the system, the CAN bus data communication mechanism is used to realize efficient data transmission and communication between modules, thus ensuring the stability and reliability of the system.

Further, the basic features of the JSBSim dynamic model: including wingspan, chord length, wing area, pilot eye position, aerodynamic reference point, center of gravity position, moment of inertia, product of inertia, front start and main touchdown point Location and engine thrust lines, landing gear model.

The present invention is based on the reinforcement learning method, customizes the aircraft intelligent cooperative confrontation decision-making system, the objective function is reasonable, has a remarkable effect after certain training, can ensure the validity of the model and algorithm, and can be used for the aircraft to formulate a suitable confrontation strategy.

Description of drawings

Fig. 1 is the basic architecture of the aircraft tactical confrontation simulation system in the present invention.

Detailed ways

In order to explain the features and technical content of the present invention in more detail, the present invention is described below in conjunction with the accompanying drawings. The features and technical content here are only used to illustrate and explain the present invention, and are not intended to limit the present invention. Those skilled in the art may modify the technical solutions of the foregoing examples according to applications, but such modifications do not make the essence of the technical solutions still within the scope of the disclosed examples.

As shown in Fig. 1, the present invention provides a kind of aircraft intelligent cooperative confrontation decision-making system based on reinforcement learning, including:

1. Upper layer structure: including guidance and control simulation nodes, tactical command simulation nodes, battlefield environment management nodes, tactical deduction nodes, tactical simulators, etc. These simulation nodes communicate and interact with each other through the DIS network.

(1) Tuning control simulation node: The tuning control simulation node is the management and monitoring center of the entire system. Its main role is to coordinate and control the operation of the entire system, monitor system status and record and playback data for evaluation. This node performs data interaction with other simulation nodes through the DIS network, including command issuance, status query, data transmission, etc.

(2) Tactical command simulation node: The tactical command simulation node is responsible for the command and coordination of aircraft. Its main role is to enable communication and collaboration between aircraft, ensuring teamwork and achieving assigned mission objectives. This node receives instructions from the guidance control simulation node through the DIS network, sends instructions to the battlefield environment management node and tactical deduction node, and receives data from the aircraft to update the status.

(3) Battlefield environment management node: establish a JSBSim dynamic model to build a simulated air combat environment, and is responsible for managing and monitoring the entire battlefield environment. Its main role is to enable environmental modeling and simulation, as well as to locate and track the position of aircraft on the battlefield. This node receives instructions from the guidance control simulation node through the DIS network, updates the battlefield environment information, and sends it to the tactical deduction node.

(4) Tactical deduction node: The tactical deduction node is responsible for tactical deduction and planning. Its main role is to collect information from other nodes and analyze it, formulate tactical strategies and plan routes. This node receives information from battlefield environment management nodes and tactical command simulation nodes through the DIS network, analyzes the information and generates corresponding action plans.

(5) Tactical simulator: The tactical simulator node is responsible for simulating the behavior of the aircraft. Its main role is to predict the behavior and performance of the aircraft through simulation in order to better guide the actions of the aircraft. This node receives information from the battlefield environment management node and the tactical deduction node through the DIS network, and simulates the behavior of the aircraft based on these information.

2. Lower layer architecture: located in a single tactical simulator, by using the hybrid real-time communication network proposed in this paper, the fire control calculation module, flight control calculation module, visual calculation module, and visual display module in the simulator , instrument calculation module and so on.

(1) Fire control calculation module: The fire control calculation module is responsible for calculating the fire control data of the aircraft, including missile launch azimuth and elevation angle, target distance, trajectory correction, etc. This module receives data from the flight control calculation module and the visual calculation module, generates corresponding fire control data through calculation, and sends it to the flight control calculation module.

(2) Flight control calculation module: The flight control calculation module is responsible for calculating the flight control data of the aircraft, including flight speed, altitude, attitude, etc. This module receives data from the fire control calculation module, visual calculation module and instrument calculation module, generates corresponding flight control data through calculation, and sends it to the equipment control and acquisition module.

(3) Visual calculation module: The visual calculation module is responsible for calculating the scene rendering of the aircraft. This module receives data from the fire control calculation module, the flight control calculation module and the equipment control and acquisition module, generates corresponding image data through calculation, and sends it to the visual display module.

(4) Visual display module: The visual display module is responsible for displaying the visual data generated by the visual calculation module in the form of images. This module receives data from the View Rendering module and renders it into an image for visualization.

(5) Instrument calculation module: The instrument calculation module is responsible for calculating various instrument data of the aircraft, including speed, altitude, attitude, etc. This module receives data from the flight control calculation module, generates corresponding instrument data through calculation, and sends it to the multi-function display module.

(6) Multi-function display module: The multi-function display module is responsible for displaying the instrument data generated by the instrument calculation module and other data related to the aircraft, including fire control data, mission information, battery status, etc. This module receives the data from the instrument calculation module and the equipment control and acquisition module, and renders it into visualized information.

(7) Equipment control and acquisition module: The equipment control and acquisition module is responsible for communication and data acquisition with various equipment of the aircraft.

3. The establishment of the simulated air combat environment: JSBSim, an open-source cross-platform six-degree-of-freedom nonlinear flight dynamics model, is mainly used.

The establishment of JSBSim dynamic model mainly includes the following contents:

Basic features: including wingspan, chord length, wing area, pilot's eye position, aerodynamic reference point, center of gravity position, moment of inertia, product of inertia, position of front takeoff and main takeoff and touchdown points, engine thrust line, landing gear model

Flight control scheme: In the present invention, since the state of the rudder surface of the aircraft is directly controlled end-to-end by the agent, no stability enhancement design is adopted.

The present invention is based on the aircraft intelligent cooperative confrontation decision-making method based on reinforcement learning, specifically as follows:

Step 1: Observation value design: Simulate and model different models, their weapons, and radars based on flight dynamics; the information obtained by the aircraft includes: its own position information, the relative position relationship between itself and the enemy aircraft, the speed of its own aircraft, its own The speed difference between the aircraft and the enemy aircraft;

Step 2: Action space design: Design the action space of each aircraft, including the target aircraft number and the command values made by the four headings. The command action values include angle of attack, roll angle, and throttle amount;

Can be written as follows:

a＝[target,x _t ,y _t ,z _t ,v _t ]

Among them, target represents the number of the target aircraft selected by the aircraft, and x _t , y _t , z _t , and v _t represent the command values made by the agent on the four track dimensions respectively.

Step 3: Reward function design: design inventory rewards, distance rewards/penalties and radar locking items;

The return function can be written as follows:

r＝ω _α r _s +ω _β r _d +ω _γ r _r

Among them, r _s is the survival reward part, r _d is the distance reward/penalty item, and r _r is the radar lock item. ω _α , ω _β , ω _γ are the proportions of each part.

Step 4: Reinforcement learning environment design: Use training mode and application mode to dynamically control the aircraft and opponents, and realize the data interface functions of state, action and reward value.

The training round is composed of training steps, and each training round contains a limited training step; the agent uses the state information as the input of the deep neural network, and generates actions after calculation; the actions are converted into executable instructions for the aircraft, and sent to the environment.

In the present invention, since the agent needs to interact with the JSBSim simulation environment, the continuous operation mode is not used for simulation, but after each decision of the agent is made, a function is called to complete the step-by-step operation of the simulation.

Radar system simulation selects radar function-level simulation, and designs the modules of the radar system. Aiming at the typical air-to-air mode of airborne phased-array radar, the functional-level system simulation of airborne phased-array radar is carried out for the air combat intelligence in this project. Provide a fast and accurate battlefield situation awareness.

Airborne Phased Array Radar Modeling:

(1) Modeling of main working methods: The modeling of this system mainly simulates air interception (AIC) and air combat (ACM) in the air/air function

(2) Radar data processing modeling:

Firstly, data preprocessing is performed on the target detection point information.

Next, enter the track management module, judge that the point track information is a real new target, and open a new track; if the point track can be associated with the existing track information at this time, it becomes a stable motion point track.

After that, the distance, azimuth, and pitch angle information of the quasi-target in the spherical coordinate system are converted back to the three-way position coordinates in the Cartesian coordinate system, so as to perform filtering and prediction, and send the filtering results to the output interface.

If the target is lost for a period of time, it will be judged as the end of the track, and the interface will be cleared at this time.

(3) Data preprocessing:

In this system, in order to improve the real-time performance of radar processing, the transmitter module, receiver module and target processing module in the overall design of the radar system are combined to simplify the processing.

For the transmission and reception of the beam, the scanning range of the radar system, the transmission power, the target distance, and the target radar scattering cross-sectional area are used to determine the theoretical maximum reliability under the condition that no electronic interference is received, the weather is fine, and the target falls in the clutter-free area. Detection distance.

Track association: Calculate whether the currently detected target point information falls within the set range centered on the point track that was predicted last time to this moment. When the aircraft target makes a large maneuver, choose a larger threshold to keep the correct update of the track:

1) If the target detection module detects the target track and fails to associate the detected track with the established track, it means that it is a new target, and at the same time the radar can successfully associate the track twice consecutively , you can enter the track start.

2) If the target detection module detects the target information, and the association between the quasi-target information and the established track fails, and in the following period of time, the target point track detected by the airborne radar is different from the previously established track. If the association is successful, it will be judged as a false alarm, and the track must be terminated.

3) If the target detection module does not detect the target track, the track association must fail at this time, and when the radar executes the tracking event, it is originally necessary to maintain the track, but the target track required for this track maintenance is equal to The prediction result of the target point trace at the previous moment is arranged to execute the small search event at the next moment. In the absence of beam modeling, the small search behavior is equivalent to expanding a certain threshold. When the radar performs a small search and does not detect a lost target, the track can be terminated.

4) If the target detection module detects the target point track, and the detected target point track is successfully associated with the established track in three dimensions of distance, azimuth, and pitch, then it is judged that the point track is a new observation point of this track , that is, to maintain the track.

Case analysis of swarm air combat intelligent decision-making at track control level:

Observation space setting: According to the motion settlement equation under six freedoms, with Respectively represent the pitch angle, yaw angle and roll angle, reflecting the attitude of the aircraft relative to the ground relative coordinate system, [x, y, z] represents the three-dimensional space coordinate value of the aircraft with the ground as the reference system, and [vx, vy, vz] Respectively represent the split of the flight speed of the aircraft in three directions.

Action space setting: Based on the characteristics of the track control task and the design of the simulation environment, the north-east sky coordinates of the target point and the target speed are selected as the decision-making dimension for continuous space decision-making control. At the same time, in order to narrow the search space, the location range of the decision-making is concentrated within a certain range of the target machine, and the number of the target machine is also generated through the network. The decision result is represented by a one-dimensional vector such as a=[target,x _t ,y _t ,z,v _t ].

Organically synthesize the following four technologies to form a simulation architecture based on HLA and hybrid real-time network:

1) Using DIS distributed management, time advancement mechanism and load balancing control technology

In the upper layer structure of the system, the distributed management and data communication are realized through the DIS network, so that the simulation nodes can work together efficiently. At the same time, the system uses the time advance mechanism to ensure the accuracy and synchronization of simulation results, and ensures the stability and reliability of the system through load balancing control technology.

2) Utilize the high real-time characteristics and deterministic delay of the reflective memory network

In the underlying architecture of the system, reflective memory network technology is used to achieve high real-time and deterministic delays, enabling fast and accurate data interaction and collaborative work between modules.

3) Use RTX's precise clock and preemptive task scheduling mechanism

In the lower layer architecture of the system, the precise clock and preemptive task scheduling mechanism of the RTX real-time operating system are used to enable the system to perform fine control and scheduling of tasks, thereby ensuring the efficiency of data interaction and collaborative work between modules. accuracy.

4) Using the data communication mechanism of CAN bus

In the underlying architecture of the system, the CAN bus data communication mechanism is used to realize efficient data transmission and communication between modules, thus ensuring the stability and reliability of the system.

Claims (8)

1. A method for intelligent cooperative confrontation decision-making of aircraft based on reinforcement learning, characterized in that, comprising the following steps: Step 1: Observation value design: Simulate and model different models, their weapons, and radars based on flight dynamics; the information obtained by the aircraft includes: its own position information, the relative position relationship between itself and the enemy aircraft, the speed of its own aircraft, its own The speed difference between the aircraft and the enemy aircraft; Step 2: Action space design: Design the action space of each aircraft, including the target aircraft number and the command values made by the four headings. The command action values include angle of attack, roll angle, and throttle, and are written in the following form: a＝[target,x _t ,y _t ,z _t ,v _t ] Among them, target represents the number of the target aircraft selected by the aircraft, and x _t , y _t , z _t , v _t represent the command values made by the agent on the four track dimensions respectively; Step 3: Reward function design: Design inventory rewards, distance rewards/penalties and radar locking items, and the reward function is written as follows: r＝ω _α r _s +ω _β r _d +ω _γ r _r Among them, r _s is the survival reward part, r _d is the distance reward/penalty item, r _r is the radar lock item; ω _α , ω _β and ω _γ are the proportions of each part; Step 4: Reinforcement learning environment design: Use training mode and application mode to dynamically control the aircraft and opponents, and realize the data interface functions of state, action and reward value; A training round consists of training steps, and each training round contains a finite number of training steps; The agent uses the state information as the input of the deep neural network, and generates actions after calculation; After the action is format converted, it forms an executable command for the aircraft and sends it to the environment. 2. The aircraft intelligent cooperative confrontation decision-making method based on reinforcement learning according to claim 1, wherein the simulation modeling of the radar is used to simulate air interception and air combat in the air/air function, specifically comprising: Step 1.1: Radar Data Processing Modeling: Step a: Perform data preprocessing on the target detection point trace information; Step b: Enter the track management module, judge that the point track information is a real new target, then open a new track; if the point track can be associated with the existing track information at this time, it will become a stable motion point track; Step c: Reversely convert the distance, azimuth, and pitch angle information of the quasi-target in the spherical coordinate system to the three-way position coordinates in the Cartesian coordinate system, thereby performing filtering and prediction, and sending the filtering result to the output interface; Step d: If the target is lost for a period of time, it will be judged as the end of the track, and the interface will be cleared; Step 1.2: Data Preprocessing Combine the transmitter module, receiver module and target processing module in the overall design of the radar system to simplify the processing; for the transmission and reception of the beam, it is determined by the scanning range of the radar system, the transmission power, the target distance, and the target radar scattering cross-sectional area. The theoretical maximum detectable distance under the condition of no electronic jamming, fine weather, and target falling in the clutter-free area. 3. The aircraft intelligent cooperative confrontation decision-making method based on reinforcement learning according to claim 2, wherein the track management module carries out track association specifically as follows: Calculate whether the currently detected target point information falls within the set range centered on the point track from the last prediction of the existing track to this moment: 1) If the target detection module detects the target track, and the detected target track fails to be associated with the established track, it is considered to be a new target, and at the same time the radar can successfully associate the target track twice consecutively , then enter the track start; 2) If the target detection module detects the target information, and the association between the quasi-target information and the established track fails, and in the following period of time, the target point track detected by the airborne radar is different from the previously established track. If the association is successful, it will be judged as a false alarm and the track will be terminated; 3) If the target detection module does not detect the target point track, the track association fails at this time, and arrange to execute the small search event at the next moment. When the radar performs the small search and does not detect the lost target, the track will be terminated; 4) If the target detection module detects the target point track, and the detected target point track is successfully associated with the established track in three dimensions of distance, azimuth, and pitch, then it is judged that the point track is a new observation point of this track , that is, to maintain the track. 4. An aircraft intelligent cooperative confrontation decision-making system based on reinforcement learning, characterized in that it includes an upper-level architecture and a lower-level architecture; The upper layer structure includes guidance and control simulation nodes, tactical command simulation nodes, battlefield environment management nodes, tactical deduction nodes and tactical simulators, and data communication and interaction between each simulation node through the DIS network; The lower layer structure is located in a single tactical simulator. Through the hybrid real-time communication network, the fire control calculation module, flight control calculation module, visual calculation module, visual display module, and instrument calculation module in the simulator are multifunctional The display module and the device control are connected with the acquisition module. 5. the aircraft intelligent cooperative confrontation decision system based on reinforcement learning according to claim 4, is characterized in that, 1) The pilot control simulation node is the management and monitoring center of the entire system, used to coordinate and control the operation of the entire system, monitor the system status and record and playback data for evaluation; it communicates with other simulation nodes through the DIS network Interaction, including order issuing, status inquiry and data transmission; 2) The tactical command simulation node is responsible for the command and coordination of the aircraft, and is used to realize the communication and collaboration between the aircraft, to ensure teamwork and to achieve the specified mission objectives; it receives instructions from the guidance control simulation node through the DIS network, Send instructions to the battlefield environment management node and tactical deduction node, and receive data from the aircraft to update the status; 3) The battlefield environment management node builds a simulated air combat environment by establishing a JSBSim dynamic model, and is responsible for managing and monitoring the entire battlefield environment, for realizing environmental modeling and simulation, and positioning and tracking the position of the aircraft on the battlefield; Receive instructions from the guidance control simulation node through the DIS network, update the battlefield environment information, and send it to the tactical deduction node; 4) The tactical deduction node is responsible for tactical deduction and planning, and is used to collect information from other nodes and analyze it, formulate tactical strategies and plan routes; it receives information from battlefield environment management nodes and tactical command simulation nodes through the DIS network information, analyze it and generate a corresponding action plan; 5) The tactical simulator is responsible for simulating the behavior of the aircraft, and is used to predict the behavior and performance of the aircraft through simulation, so as to guide the actions of the aircraft; it receives information from the battlefield environment management node and the tactical deduction node through the DIS network, and based on This information simulates the behavior of the aircraft. 6. The aircraft intelligent cooperative confrontation decision-making system based on reinforcement learning according to claim 4, characterized in that, 1) The fire control calculation module is responsible for calculating the fire control data of the aircraft, including missile launch azimuth and elevation angle, target distance, ballistic correction; this module receives data from the flight control calculation module and the visual scene calculation module, through Calculate and generate corresponding fire control data, and send it to the flight control calculation module; 2) The flight control calculation module block is responsible for calculating the flight control data of the aircraft, including flight speed, height and attitude; this module receives data from the fire control calculation module, the visual scene calculation module and the instrument calculation module, through Calculate and generate corresponding flight control data, and send it to the equipment control and acquisition module; 3) The visual scene calculation module is responsible for calculating the scene rendering of the aircraft. This module receives data from the fire control calculation module, the flight control calculation module and the equipment control and acquisition module, generates corresponding image data through calculation, and It is sent to the visual display module; 4) The visual display module is responsible for displaying the visual data generated by the visual calculation module in the form of an image. This module receives data from the visual calculation module and renders it as a visualized image; 5) The instrument calculation module is responsible for calculating various instrument data of the aircraft, including speed, altitude and attitude; this module receives data from the flight control calculation module, generates corresponding instrument data through calculation, and sends it to multiple Function display module; 6) The multifunctional display module is responsible for displaying the instrument data generated by the instrument calculation module, and other data related to the aircraft, including fire control data, mission information, and battery status; this module receives information from the instrument calculation module and equipment control and Collect module data and render it as visualized information; 7) The device control and acquisition module is responsible for communication and data acquisition with various devices of the aircraft. 7. The aircraft intelligent cooperative confrontation decision-making system based on reinforcement learning according to claim 4 is characterized in that, in the upper-level architecture and the lower-level architecture, the following four technologies are organically synthesized to form a set of HLA-based and hybrid Simulation architecture of real-time network: 1) Using DIS distributed management, time advancement mechanism and load balancing control technology In the upper layer structure of the system, the distributed management and data communication are realized through the DIS network, so that the simulation nodes can work together efficiently; at the same time, the system uses the time advancement mechanism to ensure the accuracy and synchronization of the simulation results, and through Load balance control technology to ensure the stability and reliability of the system; 2) Utilize the high real-time characteristics and deterministic delay of the reflective memory network In the underlying architecture of the system, reflective memory network technology is used to achieve high real-time and deterministic delays, enabling fast and accurate data interaction and collaborative work between modules; 3) Use RTX's precise clock and preemptive task scheduling mechanism In the lower layer architecture of the system, the precise clock and preemptive task scheduling mechanism of the RTX real-time operating system are used to enable the system to perform fine control and scheduling of tasks, thereby ensuring the efficiency of data interaction and collaborative work between modules. accuracy; 4) Using the data communication mechanism of CAN bus In the underlying architecture of the system, the CAN bus data communication mechanism is used to realize efficient data transmission and communication between modules, thus ensuring the stability and reliability of the system. 8. The aircraft intelligent cooperative confrontation decision-making system based on reinforcement learning according to claim 5, wherein the basic features of the JSBSim dynamic model include wingspan, chord length, wing area, pilot's eye position, air Dynamic reference point, position of center of gravity, moment of inertia, product of inertia, position of front takeoff and main takeoff and touchdown points, engine thrust line, landing gear model.