CN118629234A - Coordinated control method, device and readable storage medium for traffic lights at multiple intersections - Google Patents

Abstract

The invention relates to a method and equipment for cooperatively controlling traffic signal lamps at multiple intersections and a readable storage medium. The method comprises the following steps: acquiring traffic flow information, and inputting the traffic flow information into a traffic signal lamp control model based on a multi-agent reinforcement learning algorithm to acquire and store the state, action and rewarding function of each agent; adopting an interpretable influence mechanism based on a high-efficiency linked neural network to calculate importance coefficients of input data on different traffic routes and calculate traffic flow hidden variables after weighted aggregation; adopting a bias ReLU neural network to approximate a valence function and a strategy function in actor-critic reinforcement learning algorithm so as to construct a frame of the piecewise linearity actor-critic; adopting a centralized training distributed execution method, wherein actor of each intelligent agent trains through traffic flow information to obtain respective strategy functions; and training the centralized critic according to the traffic flow hidden variable to obtain a joint cost function to obtain the optimal solution of the multi-agent reinforcement learning algorithm. The invention can improve the efficiency and the safety of road traffic flow.

Description

Coordinated control method, device and readable storage medium for traffic lights at multiple intersections

Technical Field

The present invention relates to a method and device for collaboratively controlling traffic lights at multiple intersections, and a readable storage medium, and belongs to the technical field of intelligent traffic light control.

Background Art

Transportation is a key driving force for economic and social development and one of the manifestations of urban economic competitiveness. The development of transportation infrastructure has brought many conveniences to people's lives, but it has also brought a series of problems. The most common problem in urban transportation is congestion, which brings many inconveniences to people's travel, increases driving risks, reduces logistics efficiency, causes energy waste, causes overall economic losses to society, and brings huge pressure on the environment.

Traffic lights can adjust the traffic flow on different roads by controlling the phases of red and green lights in different directions at intersections, thereby achieving the purpose of traffic diversion, reducing congestion, and reducing the waiting time of vehicles at intersections. Generally speaking, traffic light control methods are mainly divided into three types: timing control, sensing control, and adaptive control. Among them, timing control updates the phase of intersection lights based on a predetermined schedule, and sensing control uses sensors to obtain the current traffic flow status and uses predetermined rules to set the green light phase of the intersection. Both methods do not take into account long-term traffic conditions, cannot optimize signal light control in real time, and lack the ability to dynamically alleviate traffic congestion.

With the development of urbanization and artificial intelligence, data-driven control methods play an increasingly important role in intelligent transportation systems. Therefore, many researchers focus on using adaptive methods to solve the problem of traffic light control. In this context, reinforcement learning methods are widely used in traffic light control. Traffic light control is actually a sequential decision problem, which can be modeled as a Markov decision process and solved by reinforcement learning algorithms. The purpose of traffic light control is to minimize the waiting time of vehicles in the entire traffic network by adjusting the signal lights at intersections. In the control of single-intersection signals, Genders et al. were the first to apply deep reinforcement learning methods to traffic light control and used convolutional neural networks to approximate the Q-value function. However, in real life, traffic networks are interconnected, and the control of a certain intersection signal will inevitably affect the traffic conditions of its upstream and downstream intersections, thereby triggering a chain reaction at surrounding intersections. Therefore, many studies have explored the application of multi-agent reinforcement learning algorithms in the coordinated control of multi-intersection traffic lights, with the goal of finding a balance between centralized and decentralized models to reduce traffic congestion and minimize the waiting time of vehicles on the entire road network. For example, Van der Pool et al. proposed a multi-intersection traffic light control mechanism based on deep reinforcement learning, defined a new reward function, and adopted a migration planning technique for a smaller intersection set, linking the learning results with the maximum plus coordination algorithm for a larger intersection set, and achieved highly efficient coordinated control between multiple intersections. For another example, Casas applied the Deep deterministic policy gradient (DDPG) algorithm to the traffic light control system. DDPG uses the policy gradient method. Compared with the value update method, such as Deep Q-Network (DQN), DDPG directly optimizes the agent strategy and uses a deterministic strategy to directly give a certain action value, reducing the variance of action selection and accelerating algorithm optimization. For another example, Tian, Tan, Feng et al. optimized the global learning model based on the idea of self-region division, proposed the Coder algorithm, and performed a good aggregation of the output of the self-region, optimizing the coordination effect between different intersections. In order to further explore the spatiotemporal dependence of traffic data between different intersections, many scholars have proposed traffic light control based on graph reinforcement learning. Nishi et al. proposed a reinforcement learning algorithm based on graph convolutional neural networks, modeling multiple traffic intersections as a network graph. For another example, Devailly et al. proposed an inductive graph reinforcement learning algorithm that combines graph convolutional networks with independent DQNs. For another example, Wang et al. proposed a spatiotemporal multi-agent reinforcement learning framework that uses graph neural networks to extract spatial structure information and uses recurrent neural networks to model system dynamics. Finally, the coordinated control of traffic lights at multiple intersections is achieved through decentralized DQNs and attention mechanisms.

Although the above works have dealt with the control problem of traffic lights at multiple intersections through collaborative reinforcement learning, the impact of coupled agents on the global road network has not been explicitly considered. Therefore, modeling the relationship between multiple agents and extracting the relationship between input traffic flow data through an interpretable network is of great research significance.

Summary of the invention

The present invention provides a method and device for collaboratively controlling traffic lights at multiple intersections, and a readable storage medium, aiming to solve at least one of the technical problems existing in the prior art.

The technical solution of the present invention relates to a method for coordinated control of traffic lights at multiple intersections in a dynamic scene, wherein each lane at each traffic intersection is provided with a road sensor for transmitting vehicle information to a traffic control center; the method according to the present invention comprises the following steps:

S100, acquiring traffic flow information through the road sensor and inputting it into a traffic light control model based on a multi-agent reinforcement learning algorithm to obtain and store the state, action and reward function of each agent;

S200, according to the traffic flow information of each control interval, using an interpretable influence mechanism based on an efficient link neural network, to find the importance coefficient of input data on different traffic road networks, and to find the weighted aggregated traffic flow latent variables;

S300, using biased ReLU neural network to approximate the value function and policy function in the actor-critic reinforcement learning algorithm to construct a piecewise linear actor-critic framework;

S400, using a centralized training distributed execution method, each intelligent agent actor obtains the local observed traffic flow information through step S100, and trains to obtain its own strategy function; the centralized critic trains to obtain a joint value function based on the weighted aggregated traffic flow hidden variables obtained in step S200;

S500. According to the strategy function and joint value function obtained through training, the optimal solution of the multi-agent reinforcement learning algorithm is obtained for application in the coordinated control of traffic lights at multiple intersections.

Further, in step S100,

The traffic flow information includes the number of vehicles on lane _j , the density of vehicles on lane j, The waiting time _Wi (k) at intersection i at time k is expressed as follows:

In the formula, u _j , d _j represent the upstream node and downstream node of lane j respectively. represents the traffic flow into lane j, represents the traffic flow leaving lane j and heading to the next intersection v, represents the adjacent intersection of the downstream node d _j , τ represents the interval coefficient between vehicles, L(e _j ) represents the lane length of lane j, Represents the waiting time of vehicles on the lane with upstream node u _j and downstream node _vi .

Furthermore, in step S100, based on the partially observable Markov decision mechanism, at time step k, the local observation o _i of each agent i used to represent the joint state S = o ₁ ×o ₂ ×…o _N of the agents is expressed as follows:

In the formula, the observation value of each agent i is the vehicle queue length of the adjacent road of the traffic intersection _vi ρ _i = a _i is the signal light phase of the current intersection used to represent the agent's action. It is a four-bit one-hot encoding that effectively encodes the current green light phase state; Indicates the road density of the current intersection; wherein each traffic intersection is provided with a separate controller for providing a signal light phase;

Among them, the reward function of the agent is expressed as follows:

Where k ₁ , k ₂ , k ₃ represent the hyperparameters of the reward function under different traffic intersection conditions, and ΔQ _i (k) represents the change of the vehicle queue between adjacent time steps;

The global reward on the entire traffic network is set to the linear sum r(k) of each agent’s reward _ri , which is expressed as follows:

In the formula, N represents the number of agents.

Further, in step S200,

All hidden layer nodes of the efficient link neural network are connected to the output layer, and the neurons in the hidden layer of the efficient link neural network include source nodes D and intermediate nodes C;

Among them, the importance coefficients of input data on different traffic road networks are expressed as follows:

In the formula, the variance _σm is used as the input data of different traffic networks. The importance coefficient of VAR(·) represents the input variable The corresponding mth output component The corresponding variance, where f(·) is the output function of the efficient link neural network; represents the input of the efficient link neural network at time step k obtained according to the traffic flow information; α _p,s is the weight coefficient of the efficient link neural network, and z _p,s is the output of the intermediate node C;

Among them, the traffic flow latent variable after weighted aggregation is expressed as follows:

Where H _i,k is the latent variable output by the efficient link neural network at the i-th intersection at time step k.

Further, in step S300,

The neuron z(x) of the biased ReLU neural network is expressed as follows:

Where, _qi represents the number of segmentation biases in the i-th dimension of the input data, and the biased ReLU neural network uses different bias parameters βi _,qi in different dimensions;

Further, in step S300,

The optimal solution π ^* of the reinforcement learning algorithm is expressed as follows:

u∈π(x)

Where x represents the state in reinforcement learning, w represents a random noise with probability distribution P(·|x,u), π is the policy function, g(x,u,w) is the cost of each step, and γ is the discount factor; is the optimal value function, and f(x,u,w) represents the state transition function of reinforcement learning.

Further, in step S400,

By summing up the loss functions of all critics, we get a joint value function L(φ). The goal of the critic is to minimize the joint value function, which is expressed as follows:

In the formula, represents the advantage function, _Nb represents the training batch size, b′ is any sampling sequence after sample b, γ is the discount factor, λ is the regularization parameter, and W is the estimated joint value function The weight coefficients of the neural network;

Each decentralized actor updates its policy function π _i,θ according to the state variables observed locally in the traffic network, and uses the clip function to limit the change ratio of the new and old policies _Li (θ):

In the formula, ∈ represents the truncation coefficient in the clip function, The coding variable entered for the local observations.

Furthermore, in step S500, the average waiting time AVE and traffic flow stability STA are used to quantitatively evaluate the status of the traffic network, which is expressed as follows:

Where _Ts represents the test time, and _Ei (t) represents the average waiting time of the vehicle of agent i during the test time _Ts .

The technical solution of the present invention also relates to a computer-readable storage medium on which program instructions are stored, and the above-mentioned method is implemented when the program instructions are executed by a processor.

The technical solution of the present invention also relates to a coordinated control device for traffic lights at multiple intersections, wherein the device includes a computer device, and the computer device includes the above-mentioned computer-readable storage medium.

The beneficial effects of the present invention are as follows:

The present invention provides an interpretable influence mechanism to reveal the spatiotemporal dependency of input traffic flow data, and proposes a multi-agent reinforcement learning algorithm based on the influence mechanism. In the scenario of multi-intersection traffic lights, the multi-agent reinforcement learning algorithm based on the interpretable influence mechanism proposed by the present invention is used. On the one hand, the spatiotemporal correlation of traffic characteristics is extracted through the interpretable influence mechanism, and the influence of different input traffic data on the output is modeled. On the other hand, the multi-agent reinforcement learning algorithm is used to realize the coordinated control of traffic lights at multiple intersections, alleviate intersection congestion, and effectively improve the efficiency and safety of road traffic flow.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a basic flow chart of the method according to the present invention.

FIG. 2 is a schematic diagram of the structure of a highly efficient linked neural network according to the method of the present invention.

FIG3 is a schematic diagram of the structure of a biased ReLU neural network according to the method of the present invention.

FIG4 is a flow chart of a multi-agent reinforcement learning algorithm according to the method of the present invention.

FIG. 5 is a schematic diagram of the structure of a simulated irregular traffic network according to the method of the present invention.

FIG. 6 is a graph showing the comparison of the intersection waiting time between the method according to the present invention and various existing methods.

FIG. 7 is a comparison chart of the average reward per round between the method according to the present invention and the existing advanced algorithm.

FIG8 is a comparison diagram of the stability of road network traffic flow in each round between the method according to the present invention and the existing advanced algorithm.

DETAILED DESCRIPTION

The concept, specific structure and technical effects of the present invention will be clearly and completely described below in combination with the embodiments and drawings to fully understand the purpose, scheme and effect of the present invention.

It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. The singular forms "a", "said" and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit the invention. The term "and/or" used herein includes any combination of one or more of the related listed items.

It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided herein is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

1 to 4, in some embodiments, the method for coordinated control of traffic lights at multiple intersections according to the present invention includes at least the following steps:

S100, obtaining traffic flow information through road sensors and inputting it into a traffic light control model based on a multi-agent reinforcement learning algorithm to obtain and store the state, action and reward function of each agent;

S200, according to the traffic flow information of each control interval, using an interpretable influence mechanism based on an efficient link neural network, to find the importance coefficient of input data on different traffic road networks, and to find the weighted aggregated traffic flow latent variables;

S300, using biased ReLU neural network to approximate the value function and policy function in the actor-critic reinforcement learning algorithm to construct a piecewise linear actor-critic framework;

S400, using a centralized training distributed execution method, each intelligent agent actor obtains the local observed traffic flow information through step S100, and trains to obtain its own strategy function; the centralized critic trains to obtain a joint value function based on the weighted aggregated traffic flow hidden variables obtained in step S200;

S500. According to the strategy function and joint value function obtained through training, the optimal solution of the multi-agent reinforcement learning algorithm is obtained for application in the coordinated control of traffic lights at multiple intersections.

The present invention proposes a method for collaborative control of traffic lights at multiple intersections based on an explainable influence mechanism, which is a method for collaborative control of large-scale traffic lights in an irregular road network. It can better establish the relationship between different intersections and realize collaborative control of traffic lights in the global traffic network.

The method of the present invention can coordinately control the entire traffic light including multiple traffic networks, effectively alleviate the congestion at the intersection, and improve the efficiency and safety of the road traffic flow. The traffic network can be an irregular traffic network, a multi-multi regular traffic network, etc. Each traffic network can have multiple traffic intersections, and the traffic intersections can be three-way intersections and/or crossroads, etc. Each traffic intersection can be provided with multiple two-way lanes and/or one-way lanes in different directions. Among them, the lane length between two adjacent traffic intersections, the speed limit of each lane, and the number of external input lanes can be obtained through the traffic control center. Furthermore, the signal light time of the traffic intersection can be obtained through the traffic control center, including the yellow light time and the minimum and maximum green light time.

Specific implementation of step S100

In the method of the present invention, when a vehicle enters the coverage area of the intelligent transportation system, the road sensor collects statistics on the vehicle information and transmits the information to the traffic control center, defines the variables of the state, action, and reward function in the reinforcement learning, and stores them in the memory bank.

Specifically, the traffic flow information on each traffic network at time k is obtained through the road sensors on each lane on the traffic network. The present invention models the traffic network as a graph network, where each intersection is a node in the graph network and the connecting section of adjacent intersections is an edge in the graph network. This includes the number of vehicles on lane j, the density of vehicles on lane _j , and the number of vehicles on lane j. The waiting time _Wi (k) at intersection i is expressed as follows:

In the formula, u _j , d _j represent the upstream node and downstream node of lane j respectively. represents the traffic flow into lane j, represents the traffic flow leaving lane j and heading to the next intersection v, represents the adjacent intersection of the downstream node d _j , τ represents the interval coefficient between vehicles, and L(e _j ) represents the lane length of lane j, that is, the lane length between two adjacent traffic intersections. Represents the waiting time of vehicles on the lane with upstream node u _j and downstream node _vi .

In the traffic light control model based on the multi-agent reinforcement learning algorithm of the present invention, the multi-intersection traffic light control problem is defined as a partially observable Markov decision problem, and the state, action and reward function of each agent are given.

In some application embodiments, in order to better represent the inbound and outbound traffic flow on the road and the number of vehicle queues at the intersection, the present invention defines the observation value of each agent i as the vehicle queue length of the adjacent road of the traffic intersection _vi (i.e., the downstream node _vi of lane j) The signal light phase ρ and road density at the current intersection At time step k, the local observation o _i (k) of each agent i can be defined as:

Here, o _i (k) is used to represent the local observation of the joint state S = o ₁ ×o ₂ ×…o _N of agent i. Each traffic intersection has a separate controller to provide the phase of the traffic light. The traffic light phase ρ is used to represent the agent action, which is a four-bit one-hot encoding that effectively encodes the current green light phase state. At each time step k, the controller of the traffic intersection v _i selects a discrete action, which is represented as The joint action grows exponentially with the number of agents. Among them, the present invention only considers feasible phase configurations in the action set and uses four-stage green phase control.

In some application embodiments, the present invention proposes a new reward function. Considering the characteristics of large-scale traffic networks and the optimization goal of traffic light control, the reward function of the agent is defined as:

Where κ ₁ ,κ ₂ ,κ ₃ are the hyperparameters of the reward function under different traffic intersection conditions, and ΔQ _i (k) is the change of the vehicle queue between adjacent time steps.

The global reward on the entire traffic network is set to the linear sum r(k) of each agent’s reward _ri , which is expressed as follows:

Where N is the number of agents, that is, the number of traffic intersections.

Specific implementation of step S200

The traffic light control model is a nonlinear system, which is very challenging to process complex spatiotemporal traffic flow data. Therefore, the present invention proposes an interpretable influence mechanism based on an efficient link neural network (EHHNN) to extract the spatiotemporal dependency of traffic flow data to enhance the coordinated control of traffic lights at different traffic intersections. Based on the interpretable influence mechanism, the present invention calculates the importance coefficient of input data on different road networks based on the road network traffic flow data of each control interval in the memory bank, and calculates the weighted aggregated traffic flow hidden variables.

First, the present invention inputs the traffic flow data obtained from step S100 into a linear transformation layer as the input of the efficient link neural network in step k. Among them, EHHNN can be regarded as a directed acyclic graph, and its network structure is shown in Figure 2. All hidden layer nodes in EHHNN are connected to the output layer. The hidden layer includes two types of neurons, including source nodes D and intermediate nodes C.

In the Efficient Link Neural Network, the output of the source node D can be described as:

z _1,s ＝max{0,x _m -β _m,qm }

Among them, m represents the dimension of the input data, and β _m,qm represents the q _m -th bias parameter on the input variable x _m .

In the hidden layer of EHHNN, the output zp _,s of the intermediate node C is obtained by taking the minimum value of the neurons generated in the previous layers:

in, It is defined as including all neurons generated by the previous layer network, and |J _p,s |=p represents the number of intermediate nodes in the p-th layer network.

Finally, the output of the EHHNN is the weighted sum of all neurons in the hidden layer:

Among them, α _1,s ,α _p,s are the weight coefficients of EHHNN, α ₀ is the bias constant, n ₁ ,n _p represent the number of neurons in the first and pth layers of the network respectively.

In this paper, a two-way analysis of variance (ANOVA) is used to determine the main effects of different traffic flow information as individual factors, as well as the interactive effects of bivariate factors on intersection congestion. This feature of EHHNN provides insights into how different variables contribute to the overall prediction and helps to gain a deeper understanding of the underlying relationships in the data, so that hidden nodes that affect each output component in the ANOVA can be identified:

In the formula, VAR(·) represents the input variable The corresponding mth output component The corresponding variance, where the variance _σm is used as the input data on different traffic networks The importance coefficient of f(·) is the output function of the efficient link neural network. The larger the value of σ _m , the larger the corresponding input variable The greater the impact on the prediction output.

The present invention realizes accurate prediction based on the influence mechanism of EHHNN on the one hand, and obtains the influence coefficient of different inputs on the output through ANOVA decomposition on the other hand, thereby deriving the weighted aggregate embedding of traffic flow information from other lanes of the traffic network, that is, the weighted aggregated traffic flow latent variable is expressed as follows:

Where Hi _,k is the latent variable obtained by EHHNN at the i-th intersection in the k-th step.

Specific implementation of step S300

The Actor-Critic algorithm is a strategy learning method in reinforcement learning. This paper designs a new piecewise linear neural network, namely a biased ReLU neural network, to approximate the joint value function in the actor-critic method. and policy functionπ _i,θ .

See the network structure of the biased ReLU neural network in Figure 3, where the input data First it passes through the source node D (source neurons), then through the intermediate node C (intermediate neurons) before outputting The biased ReLU neural network of the present invention is a piecewise linear neural network similar to the ReLU neural network. Unlike the ReLU neural network, the biased ReLU uses different bias parameters in different dimensions.

Specifically, the bias parameter set in each dimension is defined as It is determined based on the distribution of the input data:

Where v and η are the variance and expectation of the input data respectively.

The neuron z(x) of the biased ReLU neural network can be expressed as:

Where q _i represents the number of segmentation biases on the i-th dimension of the input data.

For a reinforcement learning problem, the optimal solution π ^* can be expressed as:

u∈π(x)

Where x represents the state in reinforcement learning, w is a random noise with probability distribution P(·|x,u), π is the policy function, g(x,u,w) is the cost of each step, and γ is the discount factor. is the optimal value function, and f(x,u,w) represents the state transfer function in reinforcement learning.

According to the conclusions in the prior art, maximizing or minimizing a piecewise linear value function will result in a piecewise linear policy solution. The present invention constructs a piecewise linear actor-critic framework by using biased ReLU to approximate the policy function and value function in the actor-critic.

Specific implementation of step S400

See the flowchart of the multi-agent reinforcement learning algorithm of the present invention in Figure 4. Different from the independent proximal policy optimization method (IPPO) of the prior art, the multi-agent actor-critic algorithm proposed in the present invention is a method using centralized training and distributed execution, where each agent actor is trained to obtain its own strategy function based on the traffic flow information obtained through local observation (o _i (k) in step S100), and the centralized critic is trained to obtain a joint value function based on the weighted aggregated traffic flow latent variables obtained in step S200.

Specifically, the present invention updates all critics together to minimize the joint value function, that is, the goal of the critic is to minimize the joint value function. represents the advantage function, which is a truncated estimate of the generalized advantage estimate of the prior art, _Nb represents the training batch size, b′ is an arbitrary sampling sequence after sample b, γ is the discount factor, λ is the regularization parameter, and W is the estimated joint value function The weight coefficient of the neural network is obtained by summing the loss functions of all critics to obtain a joint value function L(φ), which is expressed as follows:

Among them, L(φ) is the function that minimizes the joint value, is the weighted aggregated traffic flow hidden variable obtained in step S200. Each decentralized actor updates its strategy function π _i,θ according to the state variables observed locally in the traffic network, and uses the proximal strategy optimization method and the clip function to limit the change ratio of the new and old strategies _Li (θ):

In the formula, ∈ represents the truncation coefficient in the clip function, The coding variable entered for the local observations.

Specific implementation of step S500

Repeat steps S100 to S400, and apply the strategy obtained in each time step k (the optimal solution π ^* of reinforcement learning in step S300) to the traffic light control until each round of multi-intersection traffic light coordinated control is completed. It should be noted that a round specifically refers to a complete sequence starting from the initial state of the traffic network, that is, time step k=0, until the set end state is reached. The end state of the present invention is that the simulation network time reaches 2500s.

Specifically, by controlling the signal light phase at each traffic intersection and collecting traffic flow information at each time step k through sensors on the traffic lanes, the average waiting time AVE and traffic flow stability STA are used to quantitatively evaluate the status of the traffic network:

Where _Ts represents the test time, and _Ei (t) represents the average waiting time of vehicles of agent i during the test time _Ts (see neuron E in Figure 2).

A specific embodiment is used here to illustrate the method. The multi-intersection traffic light coordinated control method of the present invention is applied to two different traffic networks for simulation verification, wherein the simulation platform is Simulation of Urban Mobility (SUMO), and the simulation scene of the multi-intersection traffic light is designed as follows.

For example, the traffic network is set as a 5×5 traffic grid, in which each intersection has three two-way lanes in the four directions of east, south, west and north. The lane length between adjacent intersections is 100 meters, the speed limit of each lane is 13.89m/s, and 930 simulated vehicles enter the network in each round.

For example, for an irregular traffic network, it is set as an irregular traffic network with 8 traffic intersections, including 2 three-way intersections and 6 crossroads. The lane length between adjacent intersections is between 75 meters and 150 meters, the speed limit of each lane is 13.89m/s, there are 10 external input lanes, and 250 simulated vehicles are input into the network each round.

In addition, to ensure the safety of traffic intersections and the effectiveness of signal light output, the yellow light time of the intersection signal light is set to 2 seconds, and the minimum and maximum green light times are 5 seconds and 50 seconds respectively. By calling the traci interface, the traffic flow information on each traffic network at time k can be obtained through the sensors on each lane of the simulation platform.

Specifically, by using python on the SUMO simulation platform, the phase of the traffic light at each intersection is controlled, and the traffic flow information of each time step k intersection is collected through sensors on the traffic road. Referring to Figures 5 to 8, the present invention is experimented on the simulated irregular traffic network of Figure 5. Figure 6 shows the intersection waiting time results of the present invention compared with timing control, IPPO algorithm (independent policy proximal optimization), IDQN algorithm (independent deep Q-Network), MADDPG algorithm (multi-agent deep deterministic policy gradient), and DGN algorithm (Graph Convolutional Reinforcement Learning). Figure 7 shows a comparison chart of the average reward per round between the present invention and the existing advanced algorithms. Figure 8 shows a comparison chart of the stability of the road network traffic flow per round between the present invention and the existing advanced algorithms. The algorithm proposed by the present invention has better control performance, can achieve lower traffic network delay and smoother traffic conditions, and effectively alleviates traffic congestion at multiple intersections. At the same time, the present invention completed two ablation experiments. Compared with the BR-EHH-waiting-time-reward experiment, the new reward function proposed in the present invention can obtain better control effect; compared with the ReLU-EHH experiment, the actor-critic framework based on the biased ReLU neural network used in the present invention has better control effect.

The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and/or implementation method may have various modifications and changes.

Claims (10)

1. A method for coordinated control of traffic lights at multiple intersections, characterized in that, each lane at each traffic intersection is provided with a road sensor for transmitting vehicle information to a traffic control center; The method comprises the following steps: S100, acquiring traffic flow information through the road sensor and inputting it into a traffic light control model based on a multi-agent reinforcement learning algorithm to obtain and store the state, action and reward function of each agent; S200, according to the traffic flow information of each control interval, using an interpretable influence mechanism based on an efficient link neural network, to find the importance coefficient of input data on different traffic road networks, and to find the weighted aggregated traffic flow latent variables; S300, using biased ReLU neural network to approximate the value function and policy function in the actor-critic reinforcement learning algorithm to construct a piecewise linear actor-critic framework; S400, using a centralized training distributed execution method, each intelligent agent actor obtains the local observed traffic flow information through step S100, and trains to obtain its own strategy function; the centralized critic trains to obtain a joint value function based on the weighted aggregated traffic flow hidden variables obtained in step S200; S500. According to the strategy function and joint value function obtained through training, the optimal solution of the multi-agent reinforcement learning algorithm is obtained for application in the coordinated control of traffic lights at multiple intersections. 2. The method according to claim 1, characterized in that in step S100, The traffic flow information includes the number of vehicles on lane _j , the density of vehicles on lane j, The waiting time _Wi (k) at intersection i at time k is expressed as follows: In the formula, u _j , d _j represent the upstream node and downstream node of lane j respectively. represents the traffic flow into lane j, represents the traffic flow leaving lane j and heading to the next intersection v, represents the adjacent intersection of the downstream node d _j , τ represents the interval coefficient between vehicles, L(e _j ) represents the lane length of lane j, Represents the waiting time of vehicles on the lane with upstream node u _j and downstream node _vi . 3. The method according to claim 2, characterized in that in step S100, based on the partially observable Markov decision mechanism, at time step k, the local observation o _i of each agent i used to represent the joint state S = o ₁ ×o ₂ ×…o _N of the agents is expressed as follows: In the formula, the observation value of each agent i is the vehicle queue length of the adjacent road of the traffic intersection _vi ρ _i = a _i is the signal light phase of the current intersection used to represent the action of the agent; Indicates the road density of the current intersection; wherein each traffic intersection is provided with a separate controller for providing a signal light phase; Among them, the reward function of the agent is expressed as follows: Where κ ₁ ,κ ₂ ,κ ₃ represent the hyperparameters of the reward function under different traffic intersection conditions, and ΔQ _i (k) represents the change of the vehicle queue between adjacent time steps; The global reward on the entire traffic network is set to the linear sum r(k) of each agent’s reward _ri , which is expressed as follows: In the formula, N represents the number of agents. 4. The method according to claim 1, characterized in that in step S200, All hidden layer nodes of the efficient link neural network are connected to the output layer, and the neurons in the hidden layer of the efficient link neural network include source nodes D and intermediate nodes C; Among them, the importance coefficients of input data on different traffic road networks are expressed as follows: In the formula, the variance _σm is used as the input data of different traffic networks. The importance coefficient of VAR(·) represents the input variable The corresponding mth output component The corresponding variance, where f(·) is the output function of the efficient link neural network; represents the input of the efficient link neural network at time step k obtained according to the traffic flow information; α _p,s is the weight coefficient of the efficient link neural network, and z _p,s is the output of the intermediate node C; Among them, the traffic flow latent variable after weighted aggregation is expressed as follows: Where H _i,k is the latent variable output by the efficient link neural network at the i-th intersection at time step k. 5. The method according to claim 1, characterized in that in step S300, The neuron z(x) of the biased ReLU neural network is expressed as follows: In the formula, q _i represents the number of segmentation biases in the i-th dimension of the input data, and the biased ReLU neural network uses different bias parameters in different dimensions. 6. The method according to claim 1, characterized in that in step S300, The optimal solution π* of the reinforcement learning algorithm is expressed as follows: u∈π(x) Where x represents the state in reinforcement learning, w represents a random noise with probability distribution P(·|x,u), π is the policy function, g(x,u,w) is the cost of each step, and γ is the discount factor; is the optimal value function, and f(x,u,w) represents the state transition equation of reinforcement learning. 7. The method according to claim 4, characterized in that in step S400, By summing up the loss functions of all critics, we get a joint value function L(φ), which is expressed as follows: In the formula, represents the advantage function, _Nb represents the training batch size, b′ is any sampling sequence after sample b, γ is the discount factor, λ is the regularization parameter, and W is the estimated joint value function The weight coefficients of the neural network; Each decentralized actor updates its policy function π _i,θ according to the state variables observed locally in the traffic network, and uses the clip function to limit the change ratio of the new and old policies _Li (θ): In the formula, ∈ represents the truncation coefficient in the clip function, The coding variable entered for the local observations. 8. The method according to claim 2, characterized in that, in step S500, the average waiting time AVE and traffic flow stability STA are used to quantitatively evaluate the status of the traffic network, which is expressed as follows: Where _Ts represents the test time, and _Ei (t) represents the average waiting time of the vehicle of agent i during the test time _Ts . 9 . A computer-readable storage medium having program instructions stored thereon, wherein the program instructions, when executed by a processor, implement the method according to any one of claims 1 to 8. 10. A multi-intersection traffic light coordinated control device, characterized in that it includes: A computer device comprising a computer readable storage medium according to claim 9.