CN114928394B - Low-orbit satellite edge computing resource allocation method with optimized energy consumption - Google Patents

Abstract

The invention provides an energy consumption optimized low-orbit satellite edge computing resource allocation method, which comprises the following steps: acquiring environment state information of a dynamic low-orbit satellite edge computing network; according to the environmental state information, an optimization problem model taking the minimum system energy consumption expense as an optimization target is constructed, wherein the system energy consumption expense is a weighted sum of the processing energy consumption of the ground mobile terminal and the low-orbit satellite; based on the optimization problem model, defining a core element of the reinforcement learning model, and designing a state evaluation function to optimize a state space; solving a deep reinforcement learning model by using a deep reinforcement learning algorithm based on the optimized DQN; and acquiring an energy consumption optimized computing resource allocation strategy based on the solving result, and distributing the energy consumption optimized computing resource allocation strategy to each ground mobile terminal, the low-orbit satellite and the ground cloud server. The design of the deep reinforcement learning algorithm based on the optimized DQN solves the problem of energy consumption optimization computing resource allocation in a low-orbit satellite edge computing network, improves the computing efficiency and reduces the energy consumption expenditure of the system.

Description

An energy-optimized low-orbit satellite edge computing resource allocation method

Technical Field

The present invention belongs to the technical field of wireless communications, and in particular relates to a low-orbit satellite edge computing resource allocation method with optimized energy consumption.

Background technique

In low-orbit satellite edge computing networks, a key challenge is how to deal with the contradiction between energy-intensive computing tasks and computing service providers with limited resources. However, in current research on low-orbit satellite edge computing networks, the task processing energy consumption of ground mobile terminals or low-orbit satellites is usually designed as the system optimization target, while ignoring the inclusion of both of them in the task processing energy consumption overhead. Combined with the low-orbit satellite edge computing network scenario, due to the characteristics of high-speed mobility, limited battery capacity and computing power of low-orbit satellites, the network environment information in the low-orbit satellite edge computing network is dynamically updated, resulting in a high dimensionality of the environmental state information. In addition, the dimensions of the environmental state space and the computing resource allocation solution space increase exponentially with the increase in the number of tasks, low-orbit satellites and ground cloud servers, which requires the computing resource allocation solution method to have a certain degree of generalization and extensibility.

At present, the research on low-orbit satellite edge computing networks mainly focuses on minimizing satellite energy consumption or ground mobile terminal energy consumption as a single optimization goal. Both of them have not been simultaneously included in the system energy consumption overhead for joint optimization. There is also a lack of further research on computing resource allocation methods when low-orbit satellites move at high speed and resources are limited.

In reference [1], researchers minimized the energy consumption of ground mobile terminals in the network as the optimization goal, split the resource allocation optimization problem into multiple convex optimization problems, and solved them one by one using methods based on traditional optimization theory. In reference [2], researchers minimized the energy consumption of ground mobile terminals in a dynamic network environment as the optimization goal, converted the non-convex problem into a linear programming problem, and used the alternating direction multiplier method to obtain the optimal computing resource allocation strategy. However, in the actual low-orbit satellite edge computing network scenario, considering the characteristics of high-speed movement and limited resources of low-orbit satellites, the above methods are difficult to customize according to the dynamic network environment state, are easily affected by system disturbances, have poor versatility and scalability, and have bottlenecks in computing efficiency.

Therefore, how to minimize the weighted system energy consumption overhead of ground mobile terminals and low-orbit satellites, and optimize the computing resource allocation of the dynamic low-orbit satellite edge computing network system while taking into account the high mobility and limited resources of low-orbit satellites is a key issue that needs to be considered in the low-orbit satellite edge computing network.

references:

[1]Z.Song,Y.Hao,Y.Liu,and –14 218, 2021.

[2] Q.Tang, Z.Fei, B.Li and Z.Han, "Computation Offloading in LEOSatellite Networks With Hybrid Cloud and Edge Computing," in IEEE Internet ofThings Journal, vol.8, no.11, pp.9164 -9176,1June1,2021.

Summary of the invention

The purpose of the present invention is to provide an energy-optimized low-orbit satellite edge computing resource allocation method to improve computing efficiency and reduce system energy consumption overhead when the low-orbit satellite moves rapidly and resources are limited.

Based on the above problems, the present invention provides a low-orbit satellite edge computing resource allocation method with energy consumption optimization, comprising:

S1: Using intelligent agents to obtain environmental status information of dynamic low-orbit satellite edge computing networks;

S2: Based on the acquired environmental status information, an optimization problem model is constructed with the goal of minimizing the system energy consumption overhead. The system energy consumption overhead is defined as the weighted sum of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite.

S3: Based on the optimization problem model, define the state space, action space and benefit function of the reinforcement learning model, and design a state evaluation function to optimize the state space;

S4: solving a deep reinforcement learning model using a deep reinforcement learning algorithm based on an optimized DQN, wherein a discrete state generated by mapping the environmental state information through a state evaluation function is input into a network of the deep reinforcement learning algorithm as input information;

S5: Based on the solved deep reinforcement learning model, obtain the energy-optimized computing resource allocation strategy and distribute it to various ground mobile terminals, low-orbit satellites and ground cloud servers to realize computing resource allocation.

Preferably, the environmental status information of the low-orbit satellite edge computing network includes: the status information vector W ^k of the kth batch of task sets generated by the ground mobile terminal, the geocentric angle vector β ^k between each ground mobile terminal and the low-orbit satellite when the kth batch of tasks starts to be executed, the visibility vector b ^k between each ground mobile terminal and the ground cloud server when the task starts to be executed, and the battery usage status information vector U ^k of each low-orbit satellite when the kth batch of tasks starts to be executed.

Preferably, the step S1 comprises:

Step S11: providing a low-orbit satellite edge computing network consisting of M ground mobile terminals and J ground cloud servers located on the ground, and N low-orbit satellites located in space; the set of ground mobile terminals, the set of low-orbit satellites and the set of ground cloud servers are respectively expressed as M'＝{1,...,m,...,M}, N'＝{1,...,n,...,N} and J'＝{1,...,j,...,J}, m, n, j respectively represent the ordinal number of the ground mobile terminal, the ordinal number of the low-orbit satellite and the ordinal number of the ground cloud server, and M, N, J are the number of ground mobile terminals, the number of low-orbit satellites and the number of ground cloud servers respectively; setting each ground mobile terminal to be able to connect to at most one low-orbit satellite at a time; and setting each ground mobile terminal to be able to establish a connection with at most one ground cloud server through a low-orbit satellite at a time;

Step S12: Set each ground mobile terminal to generate only one indivisible computing task in each batch; then, the set K of task batches to be executed by the entire low-orbit satellite edge computing network is represented as: K'={1,...,k,...,K}, k represents the kth task batch, and K is the total number of task batches; the task batch generated by the kth batch of the mth ground mobile terminal is described as in, Expressed as the data size of the task payload, It is expressed as the number of CPU processing cycles required for the task load. The state information vector ^Wk of the k-th batch of task sets generated by the ground mobile terminal is defined as M is the number of ground mobile terminals;

Step S13: Set the low-orbit satellites to run on circular orbits, and represent the orbit height as H, the radius of the earth as R, and the elevation angle between the ground mobile terminal m and the low-orbit satellite n when starting to execute the kth batch of tasks as Obtain the geocentric angle vector β ^k between each ground mobile terminal and the low-orbit satellite when the k-th batch of tasks starts to be executed, as well as the visibility duration of each low-orbit satellite in the entire low-orbit satellite edge computing network to each ground mobile terminal when executing the k-th batch of tasks;

Step S14: Initialize the visibility vector ^bk between each ground mobile terminal and the ground cloud server when the task starts to be executed and the battery usage status information vector ^Uk of each low-orbit satellite when the kth batch of tasks starts to be executed.

Preferably, the visible duration of the low-orbit satellite n to the ground mobile terminal m when executing the kth batch of tasks is for:

Wherein, T ^LEO is the operation period of the low-orbit satellite, is the geocentric angle between the ground mobile terminal m and the low-orbit satellite n;

The geocentric angle between the ground mobile terminal m and the low-orbit satellite n for:

Where R is the radius of the Earth, H is the orbital altitude, is the elevation angle between the ground mobile terminal m and the low-orbit satellite n at the beginning of the kth batch of missions;

The operating period of a low-orbit satellite, T ^LEO , is:

Among them, R is the radius of the earth, H is the orbital altitude, and μ represents the Kepler constant.

Preferably, step S2 comprises:

Step S21: The task scheduling mode vector corresponding to the state information vector ^Wk of the kth batch of task sets generated by the ground mobile terminal is defined as The tasks generated for the kth batch of the mth ground mobile terminal The decision vector for scheduling to each LEO satellite in the LEO satellite edge computing network, The task generated for the kth batch of the mth ground mobile terminal The decision vectors for scheduling to various ground cloud servers in the low-orbit satellite edge computing network. Multiple tasks in the same batch of task sets of all ground mobile terminals can choose different task scheduling methods; task scheduling methods include: local processing, transmission to low-orbit satellites for processing, and transmission to ground cloud servers via low-orbit satellites for processing;

Step S22: determining the processing delay of each task in the task set, the task processing energy consumption of the ground mobile terminal, and the task processing energy consumption of the low-orbit satellite according to the environmental state information and task scheduling mode vector of the task set of the kth batch obtained;

Step S23: The weighted sum of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite is defined as the system energy consumption overhead, and an optimization problem model with minimizing the system energy consumption overhead as the optimization goal is constructed.

Preferably, the task generated by the kth batch of the mth ground mobile terminal Decision vectors for scheduling to each LEO satellite in the LEO satellite edge computing network for:

in, represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to low-orbit satellite n for execution; represents the task generated by the kth batch of the mth ground mobile terminal Not dispatched to low-orbit satellite n for execution;

The task generated by the kth batch of the mth ground mobile terminal Decision and scheduling of each low-orbit satellite in the low-orbit satellite edge computing network for

The tasks generated by the kth batch of the mth ground mobile terminal Decision vector for scheduling to various ground cloud servers in the LEO satellite edge computing network for:

in, represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to the ground cloud server j for execution via low-orbit satellite n; represents the task generated by the kth batch of the mth ground mobile terminal It is not dispatched to the ground cloud server j for execution through the low-orbit satellite n;

The task generated by the kth batch of the mth ground mobile terminal The decision and for

Preferably, the optimization problem model is:

Wherein, C ₁ , C ₂ , C ₃ , C ₄ , and C ₅ represent the first, second, third, fourth, and fifth constraints, respectively; represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to low-orbit satellite n for execution; represents the task generated by the kth batch of the mth ground mobile terminal Not dispatched to low-orbit satellite n for execution; represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to the ground cloud server j for execution via low-orbit satellite n; represents the task generated by the kth batch of the mth ground mobile terminal It is not dispatched to the ground cloud server j for execution through the low-orbit satellite n; They are the tasks generated by the kth batch of the mth ground mobile terminal. The processing delay when the task scheduling method is to transmit to a low-orbit satellite for processing and transmit via a low-orbit satellite to a ground cloud server for processing; is the visibility duration of low-orbit satellite n to ground mobile terminal m when performing the kth batch of tasks; The tasks generated for the kth batch of the mth ground mobile terminal for the low-orbit satellite n Allocated computing resources; z ^LEO is the upper limit of computing resources owned by a single low-orbit satellite; It is the battery usage status of low-orbit satellite n when the kth batch of missions starts to be executed.

Preferably, each state _sk in the state space of the reinforcement learning model includes a state information vector ^{Wk of} a set of tasks of the kth batch generated by a ground mobile terminal, a geocentric angle vector ^βk between each ground mobile terminal and a low-orbit satellite when the kth batch of tasks starts to be executed, a visibility vector ^bk between each ground mobile terminal and a ground cloud server when the tasks start to be executed, and a battery usage state information vector ^Uk of each low-orbit satellite when the kth batch of tasks starts to be executed;

The state evaluation function g _k is:

g _k ={g ^k,1 ,g ^k,2 ,g ^k,3 },

in, Indicates that the state s _k cannot satisfy the task generated by the low-orbit satellite for the kth batch of the mth ground mobile terminal under the action a _k The corresponding third constraint C ₃ ; It means that the state s _k can satisfy the task generated by the low-orbit satellite for the kth batch of the mth ground mobile terminal under the action a _k . The corresponding third constraint C ₃ ; It means that the state s _k cannot satisfy the fourth constraint condition corresponding to the low-orbit satellite n under the action a _k . On the contrary, It means that the state s _k cannot satisfy the fifth constraint condition corresponding to the low-orbit satellite n under the action a _k . On the contrary,

The action a _k performed by the k-th batch of tasks in the action space of the reinforcement learning model includes:

a _k ={c ^k ,f ^k,GMT ,f ^{k ,LEO} ,f ^{k ,GCS} },

Wherein, c ^k represents the task scheduling method vector of the k-th batch of task set, f ^k,GMT represents the computing resource vector allocated by the ground mobile terminal to each task in the k-th batch of task set, f ^k,LEO represents the computing resource vector allocated by the low-orbit satellite to each task in the k-th batch of task set, and f ^k,GCS represents the computing resource vector allocated by the ground cloud server to each task in the k-th batch of task set;

The benefit function of the reinforcement learning model includes an instantaneous benefit function and a cumulative benefit function;

The instantaneous reward function r _k of the reinforcement learning model is:

in, The tasks generated for the kth batch of the mth ground mobile terminal Task processing energy consumption in ground mobile terminals, The tasks generated for the kth batch of the mth ground mobile terminal Mission processing energy consumption in low-orbit satellites;

The optimization objective is described as a computing resource allocation strategy π ^* that can maximize the cumulative benefit function. For the computing resource allocation strategy π:S→A of the system, the cumulative benefit function executed until the start of the k-th batch of tasks is expressed as:

Among them, γ∈[0,1] is used as the profit discount rate to map the importance of future profits, _Eπ [·] represents the expectation under the possible strategy π, K represents the total number of task batches to be processed, k' represents the task batch in the calculation process, and k represents the batch of the currently executed task.

In step S4, DNN is introduced into the reinforcement learning model, and the neural network parameter θ is iteratively updated by using the fitting Q function obtained by fitting the actual Q function Q(s _k , _ak ) using the neural network parameter θ of DNN. The optimal result of the fitting Q function finally obtained is the optimal strategy evaluation function Q ^* (s _k , _ak ), and the deep reinforcement learning model is solved at this time.

In step S5, the intelligent agent uses the kth batch of collected environmental state information as the state _sk input, and calculates to obtain the state evaluation function _gk ; then uses the optimization problem model established in step S3 and the deep reinforcement learning algorithm based on optimized DQN adopted in step S4 to solve, and outputs the computing resource allocation strategy _ak = { ^ck , ^{fk, GMT} , fk ^{, LEO} , fk ^{, GCS} }, obtains each task scheduling mode and the computing resource allocation status of each ground mobile terminal, low-orbit satellite and ground cloud server { ^{fk , GMT} , ^{fk, LEO} , ^{fk, GCS} }, and distributes them to each ground mobile terminal, low-orbit satellite and ground cloud server.

The method of the present invention constructs an optimization problem model with the goal of minimizing the weighted system energy consumption overhead of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite, so that the intelligent agent distributes the system's optimal computing resource allocation strategy under the condition of considering the high-speed movement of the low-orbit satellite, limited energy and computing resources, completes the task execution, realizes the computing resource allocation of the ground mobile terminal, low-orbit satellite and ground cloud server in the low-orbit satellite edge computing network and reduces the system energy consumption overhead; in addition, the core elements of the optimization problem under the reinforcement learning model are defined with MDP as the framework, and the state evaluation function is designed according to the system constraints to optimize the state space to obtain the system's computing resource allocation strategy, thereby realizing an efficient computing resource allocation strategy and improving computing efficiency. In addition, the present invention is based on the deep reinforcement learning algorithm that optimizes DQN, further efficiently calculates the resource allocation strategy and improves computing efficiency.

In summary, the deep reinforcement learning algorithm designed by the present invention based on the optimized DQN solves the problem of low-orbit satellite edge computing resource allocation with energy consumption optimization in the low-orbit satellite edge computing network, improves computing efficiency, and reduces system energy consumption overhead.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of the energy-optimized low-orbit satellite edge computing resource allocation method of the present invention.

FIG2 is a schematic diagram of the computing architecture of the intelligent agent of the energy-optimized low-orbit satellite edge computing resource allocation method of the present invention.

FIG3 is an example diagram of an experimental scenario of the energy-optimized low-orbit satellite edge computing resource allocation method of the present invention.

FIG. 4 is a diagram of a circular orbit model of a low-orbit satellite.

Detailed ways

The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

In view of the shortcomings of the prior art, the present invention proposes an energy-optimized low-orbit satellite edge computing resource allocation method. The energy-optimized low-orbit satellite edge computing resource allocation method of the present invention takes minimizing the weighted system energy consumption overhead of the ground mobile terminal and the low-orbit satellite as the optimization goal. The method uses the ground mobile terminal, low-orbit satellite and ground cloud server in the dynamic low-orbit satellite edge computing network to allocate computing resources, designs reasonable reinforcement learning model core elements and state evaluation functions to simplify the state space, obtains the optimized computing resource allocation strategy based on the deep reinforcement learning algorithm of the optimized DQN, and distributes the strategy.

As shown in FIG1 , the specific steps of the energy consumption optimized low-orbit satellite edge computing resource allocation method of the present invention are as follows:

Step S1: Using the intelligent agent to obtain the environmental status information of the dynamic low-orbit satellite edge computing network;

The intelligent agent can be located on the ground or on a satellite, and is usually located on the ground. In this embodiment, the intelligent agent is preferably a ground cloud server.

The system considered in the present invention, namely, the low-orbit satellite edge computing network is composed of M ground mobile terminals and J ground cloud servers located on the ground, and N low-orbit satellites located in space. The set of ground mobile terminals, the set of low-orbit satellites and the set of ground cloud servers can be expressed as M'={1,...,m,...,M}, N'={1,...,n,...,N} and J'={1,...,j,...,J}, respectively, where m, n, j represent the ordinal number of the ground mobile terminal, the ordinal number of the low-orbit satellite and the ordinal number of the ground cloud server, respectively, and M, N, J are the number of ground mobile terminals, the number of low-orbit satellites and the number of ground cloud servers, respectively.

The environmental status information of the low-orbit satellite edge computing network includes: a status information vector W ^k of a task set of the kth batch generated by a ground mobile terminal, which is used to determine the status information vector of the task generated by the ground mobile terminal; a geocentric angle vector β ^k between each ground mobile terminal and the low-orbit satellite when the kth batch of tasks starts to be executed, which is used to determine the coverage of the low-orbit satellite; a visibility vector b ^k between each ground mobile terminal and the ground cloud server when the task starts to be executed, which is used to reflect the visibility of the ground cloud server to the task; and a battery usage status information vector U ^k of each low-orbit satellite when the kth batch of tasks starts to be executed, which is used to reflect the battery usage status of the low-orbit satellite.

This is because the task generated by the kth batch of the mth ground mobile terminal The computing resource allocation strategy depends on the state information vector of the task generated by the ground mobile terminal (i.e., the state information vector W ^k of the task set of the kth batch generated by the ground mobile terminal), the coverage of the low-orbit satellite (i.e., the geocentric angle vector β ^k between each ground mobile terminal and the low-orbit satellite when the kth batch of tasks starts to be executed), the visibility of the ground cloud server to the task (i.e., the visibility vector b ^k between each ground mobile terminal and the ground cloud server when the task starts to be executed) and the battery usage status of the low-orbit satellite (i.e., the battery usage status information vector U ^k of each low-orbit satellite when the kth batch of tasks starts to be executed).

In the step S1, obtaining the environmental status information of the low-orbit satellite edge computing network includes:

Step S11: Provide a low-orbit satellite edge computing network consisting of M ground mobile terminals and J ground cloud servers located on the ground, and N low-orbit satellites located in space, wherein the ground mobile terminals and the low-orbit satellites have mobile edge computing capabilities for processing tasks, and the ground cloud servers have computing capabilities; each ground mobile terminal is configured to be able to connect to at most one low-orbit satellite at a time; and each ground mobile terminal is configured to be able to connect to at most one ground cloud server at a time through a low-orbit satellite to achieve visible satellite-to-ground transmission link transfer, and then establish a connection through the low-orbit satellite.

Step S12: Set each ground mobile terminal to generate only one indivisible computing task in each batch; then, the set K of task batches to be executed by the entire low-orbit satellite edge computing network is represented as: K'={1,...,k,...,K}, k represents the kth task batch, and K is the total number of task batches; the task batch generated by the kth batch of the mth ground mobile terminal is described as in, Expressed as the data size of the task payload, It is represented by the number of CPU processing cycles required for the task load. Then, the state information vector W ^k of the task set of the kth batch generated by the ground mobile terminal is defined as M is the number of ground mobile terminals.

Step S13: Considering the high-speed mobility of low-orbit satellites in actual scenarios, all low-orbit satellites are set to run on circular orbits, the orbit height is represented as H, the earth radius is represented as R, and the elevation angle between the ground mobile terminal m and the low-orbit satellite n at the beginning of the kth batch of tasks is represented as The geocentric angle vector ^βk between each ground mobile terminal and the low-orbit satellite when the k-th batch of tasks starts to be executed is obtained, as well as the corresponding visibility duration of each low-orbit satellite in the entire low-orbit satellite edge computing network to each ground mobile terminal when executing the k-th batch of tasks, so as to determine the coverage of the low-orbit satellite.

At this time, the geocentric angle between the ground mobile terminal m and the low-orbit satellite n is It can be expressed as:

Where R is the radius of the Earth, H is the orbital altitude, is the elevation angle between the ground mobile terminal m and the low-orbit satellite n when the kth batch of missions begins. m and n are the ordinal numbers of the ground mobile terminal and the low-orbit satellite, respectively.

When the kth batch of tasks begins to execute, the geocentric angle vector β ^k between each ground mobile terminal and the low-orbit satellite can be expressed as:

For a low-orbit satellite at an orbital height H, the operating period T ^LEO of the low-orbit satellite is:

Among them, R is the radius of the earth, H is the orbital altitude, and μ represents the Kepler constant.

Therefore, the visible time of low-orbit satellite n to ground mobile terminal m when performing the kth batch of tasks is It can be expressed as:

Wherein, T ^LEO is the operation period of the low-orbit satellite, is the geocentric angle between the ground mobile terminal m and the low-orbit satellite n.

Step S14: Initialize the visibility vector ^bk between each ground mobile terminal and the ground cloud server when the task starts to be executed and the battery usage status information vector ^Uk of each low-orbit satellite when the kth batch of tasks starts to be executed.

Low-orbit satellite n starts executing the kth batch of tasks generated by the mth ground mobile terminal for the ground cloud server j The visibility when can be expressed as in, Indicates that the ground cloud server j can be used to process the kth batch of tasks generated by the mth ground mobile terminal k represents the task batch, m, n, j represent the ordinal number of the ground mobile terminal, the ordinal number of the low-orbit satellite, and the ordinal number of the ground cloud server respectively. Accordingly, the task generated by the kth batch of the mth ground mobile terminal can be started for the ground cloud server j according to the low-orbit satellite n. Visibility and the geocentric angle vector β ^k between each ground mobile terminal and the low-orbit satellite when the k-th batch of tasks starts to be executed, to obtain the visibility vector b ^k between each ground mobile terminal and the ground cloud server when the k-th batch of tasks starts to be executed. Under the premise that the visibility between the low-orbit satellite and the ground cloud server is established (the low-orbit satellite and the ground cloud server are visible), when the k-th batch of tasks starts to be executed, if the ground mobile terminal is within the service coverage of the low-orbit satellite, then the visibility between the ground mobile terminal and the ground cloud server is determined to be 1 when the k-th batch of tasks starts to be executed, otherwise, the visibility between the ground mobile terminal and the ground cloud server is 0 when the k-th batch of tasks starts to be executed.

The battery usage status of low-orbit satellite n when the kth batch of missions begins to execute can be expressed as In the entire LEO satellite edge computing network, the battery usage status information vector U ^k of each LEO satellite when the kth batch of tasks starts to execute can be expressed as

Step S2: Based on the acquired environmental status information, a problem model is constructed with minimizing the system energy consumption overhead as the optimization goal. The system energy consumption overhead is defined as the weighted sum of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite.

The step S2 comprises:

Step S21: The task scheduling mode vector corresponding to the state information vector ^Wk of the task set of the kth batch of all ground mobile terminals is defined as The tasks generated for the kth batch of the mth ground mobile terminal The decision vector for scheduling to each LEO satellite in the LEO satellite edge computing network, The task generated for the kth batch of the mth ground mobile terminal The decision vectors for scheduling to various ground cloud servers in the low-orbit satellite edge computing network, multiple tasks in the same batch (for example, the kth batch) of task sets of all ground mobile terminals can choose different task scheduling methods.

According to different network environments and task requirements, the task scheduling methods include: local processing, transmission to low-orbit satellites for processing, and transmission to ground cloud servers via low-orbit satellites for processing. In other words, the task generated by the kth batch of the mth ground mobile terminal You can choose to process it locally, transmit it to a low-orbit satellite for processing, or transmit it to a ground cloud server via a low-orbit satellite for processing.

For the state information vector W ^k of the task set of the kth batch of all ground mobile terminals in the low-orbit satellite edge computing network, the corresponding task scheduling method vector is described It can be expressed as:

The tasks generated for the kth batch of the mth ground mobile terminal Decision vector for scheduling to each LEO satellite in the LEO satellite edge computing network.

Among them, the tasks generated by the kth batch of the mth ground mobile terminal Decision vectors for scheduling to each LEO satellite in the LEO satellite edge computing network It can be expressed as:

in, represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to low-orbit satellite n for execution; Indicates the task Not scheduled to execute on low-orbit satellite n.

Therefore, the task generated by the kth batch of the mth ground mobile terminal Decision and scheduling of each low-orbit satellite in the low-orbit satellite edge computing network It can be expressed as

Among them, the tasks generated by the kth batch of the mth ground mobile terminal Decision vector for scheduling to various ground cloud servers in the LEO satellite edge computing network It can be expressed as:

in, represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to the ground cloud server j for execution via low-orbit satellite n; represents the task generated by the kth batch of the mth ground mobile terminal The task that is not dispatched through the low-orbit satellite n is dispatched to the ground cloud server j for execution.

Therefore, the task generated by the kth batch of the mth ground mobile terminal The decision and It can be expressed as

Since for any m, k, the k-th batch of tasks generated by the m-th ground mobile terminal Only one task scheduling method can be selected at a time, so we can get:

Step S22: determining the processing delay of each task in the task set, the task processing energy consumption of the ground mobile terminal, and the task processing energy consumption of the low-orbit satellite according to the environmental state information and task scheduling mode vector of the task set of the kth batch obtained;

The following is the task generated by the kth batch of the mth ground mobile terminal Taking as an example, the processing delay corresponding to each task in the task set of the kth batch, the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite are explained.

(a) Specifically, when the kth batch of tasks generated by the mth ground mobile terminal When you choose to execute the strategy locally, you can get The computing resources allocated to the ground mobile terminal when executing local tasks are expressed as Then in the low-orbit satellite edge computing network, the computing resource vector allocated by the ground mobile terminal to each task in the k-th batch task set can be expressed as:

in, The tasks generated for the kth batch of the mth ground mobile terminal Computational resources allocated in executing local tasks.

It should be noted that if a part of the tasks adopts other non-local execution strategies, the terminal computing resources of the tasks adopting other strategies are still represented by this, but the corresponding terminal computing resources are 0.

At this time, the task generated by the kth batch of the mth ground mobile terminal Processing delay Equal to the task The computational delay It can be expressed as The task generated by the kth batch of the mth ground mobile terminal Task processing energy consumption Equal to the task processing energy consumption of ground mobile terminals It is also equal to the task computing energy consumption of the ground mobile terminal Right now Where ζ represents the chip energy consumption coefficient, which is used to calculate the task processing energy consumption.

(b) Specifically, when the kth batch of tasks generated by the mth ground mobile terminal When the strategy of being scheduled to a low-orbit satellite is selected, the tasks generated by the kth batch of the mth ground mobile terminal can be obtained. Decision and scheduling of each low-orbit satellite in the low-orbit satellite edge computing network The low-orbit satellite n is the kth batch of tasks generated by the mth ground mobile terminal The allocated computing resources are represented as Then the computing resource vector allocated by the low-orbit satellite to each task in the k-th batch of tasks can be expressed as Since the computing resources of each low-orbit satellite are limited, the computing resources allocated to each task cannot exceed the computing resources available to the low-orbit satellite. The propagation delay between the ground mobile terminal and the low-orbit satellite performing the mission Transmission delay of mission upload to low-orbit satellite And the mission calculation delay of the low-orbit satellite performing the mission Right now The task generated by the kth batch of the mth ground mobile terminal Task processing energy consumption Including the task processing energy consumption of ground mobile terminals and low-orbit satellite mission processing energy consumption Right now Among them, the task processing energy consumption of ground mobile terminals Equal to the transmission energy consumption of uploading the mission to the low-orbit satellite Right now Mission processing energy consumption of low-orbit satellites Including the transmission energy consumption of the receiving task and the computational energy consumption of the task Right now

(c) Specifically, when the kth batch of tasks generated by the mth ground mobile terminal When the strategy of dispatching to the ground cloud server for processing via the low-orbit satellite is selected, the tasks generated by the kth batch of the mth ground mobile terminal can be obtained. The decision and The task generated by the kth batch of the mth ground mobile terminal The computing resources dispatched to the ground cloud server j by the low-orbit satellite n are expressed as Then the computing resource vector allocated by the ground cloud server to each task in the k-th batch task set can be expressed as At this time, the task generated by the kth batch of the mth ground mobile terminal Processing delay Including the propagation delay between the ground mobile terminal via the low-orbit satellite and the ground cloud server that performs the mission Transmission delay of mission upload to transit low-orbit satellite The transmission delay of the task offloading from the low-orbit satellite to the ground cloud server And the task calculation delay of the ground cloud server that executes the task Right now The task generated by the kth batch of the mth ground mobile terminal Task processing energy consumption Including the task processing energy consumption of ground mobile terminals and low-orbit satellite mission processing energy consumption Right now Among them, the task processing energy consumption of the ground mobile terminal is equal to the transmission energy consumption of uploading the task to the low-orbit satellite. Right now Mission processing energy consumption of low-orbit satellites Including the transmission energy consumption of the receiving task and the transmission energy consumption of the download task Right now

(d) Combining the tasks generated by the kth batch of the mth ground mobile terminal Description of different scheduling methods, the tasks generated by the kth batch of the mth ground mobile terminal Processing delay It can be expressed as They are the tasks generated by the kth batch of the mth ground mobile terminal. The processing delay when the task scheduling mode is local processing, transmission to low-orbit satellite for processing, and transmission to ground cloud server for processing via low-orbit satellite ( The values of two of them are 0). Therefore, for the k-th batch of tasks consisting of the mobile ground terminal set M, the maximum processing delay can be expressed as Whenever the kth batch of tasks in set M is processed, set M starts processing the k+1th batch of tasks. Energy consumption of task processing in ground mobile terminals It can be expressed as in, They are the tasks generated by the kth batch of the mth ground mobile terminal. The energy consumption of task processing at the ground mobile terminal when the task scheduling mode is to process locally and transmit to the low-orbit satellite for processing. The task generated by the kth batch of the mth ground mobile terminal Energy consumption of mission processing in low-orbit satellites It can be expressed as in, They are the tasks generated by the kth batch of the mth ground mobile terminal. The energy consumption of task processing on low-orbit satellites when the task scheduling method is to transmit to low-orbit satellites for processing and transmit to ground cloud servers for processing via low-orbit satellites.

In addition, considering that the task scheduling method is affected by the limited battery capacity of low-orbit satellites, the kth batch of tasks must meet

Step S23: The weighted sum of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite is defined as the system energy consumption overhead, and an optimization problem model with minimizing the system energy consumption overhead as the optimization goal is constructed.

The system energy consumption overhead defined in the present invention is the weighted sum of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite. The weight reflects the relative importance of the energy consumption of the ground mobile terminal and the energy consumption of the low-orbit satellite in the system energy consumption overhead, where α∈[0,1] represents the weight of the mobile ground terminal energy consumption in the system energy consumption overhead, and (1-α) represents the weight of the low-orbit satellite energy consumption in the system energy consumption overhead.

Therefore, the specific description of the optimization problem model with minimizing the system energy consumption overhead as the optimization goal (i.e., the joint energy consumption optimization problem) is as follows:

Wherein, C ₁ , C ₂ , C ₃ , C ₄ , and C ₅ represent the first, second, third, fourth, and fifth constraints, respectively; represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to low-orbit satellite n for execution; represents the task generated by the kth batch of the mth ground mobile terminal Not dispatched to low-orbit satellite n for execution; represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to the ground cloud server j for execution via low-orbit satellite n; represents the task generated by the kth batch of the mth ground mobile terminal It is not dispatched to the ground cloud server j for execution through the low-orbit satellite n; They are the tasks generated by the kth batch of the mth ground mobile terminal. The processing delay when the task scheduling method is to transmit to a low-orbit satellite for processing and transmit via a low-orbit satellite to a ground cloud server for processing; is the visibility duration of low-orbit satellite n to ground mobile terminal m when performing the kth batch of tasks; The tasks generated for the kth batch of the mth ground mobile terminal for the low-orbit satellite n Allocated computing resources; z ^LEO is the upper limit of computing resources owned by a single low-orbit satellite; It is the battery usage status of low-orbit satellite n when the kth batch of missions starts to be executed.

That is, the first and second constraints _C1 and _C2 represent each task (i.e. ) can only select one scheduling method; the third constraint C ₃ means that if each task selects a task scheduling method involving low-orbit satellites, the task execution delay should not exceed the effective coverage time of the corresponding low-orbit satellite for the task; the fourth constraint C ₄ means that the sum of the computing resources allocated by each low-orbit satellite to process each task in the task set cannot exceed the upper limit of the available computing resources; the fifth constraint C ₅ means that each low-orbit satellite should keep the available energy state always greater than 0.

Step S3: Based on the optimization problem model, the core elements of the reinforcement learning model (i.e., state space, action space, and instantaneous benefit function) are defined, and a state evaluation function is designed to optimize the state space;

In step S3, the Markov Decision Process (MDP) framework is used to establish a solution method for the reinforcement learning model. Reinforcement learning is a computational method for understanding and automating goal-oriented learning and decision-making problems, and defines the process of interaction between the agent and the environment by using three core elements: state, action, and benefit.

Based on the optimization problem established in step 2, the state space, action space and benefit function of the reinforcement learning model constructed by the present invention are defined as follows:

State space: Each state in the state space of the reinforcement learning model corresponds to the environmental state information of the low-orbit satellite edge computing network, which includes the state information vector ^Wk of the kth batch of tasks generated by the ground mobile terminal, the geocentric angle vector ^βk between each ground mobile terminal and the low-orbit satellite when the kth batch of tasks starts to be executed, the visibility vector ^bk between each ground mobile terminal and the ground cloud server when the task starts to be executed, and the battery usage state information vector ^Uk of each low-orbit satellite when the kth batch of tasks starts to be executed.

Therefore, the state s _k ∈ S when the kth batch of tasks starts to execute is expressed as:

s _k ={W ^k ,β ^k ,b ^k ,U ^k },

Among them, ^Wk represents the state information vector of the k-th batch of task set generated by the ground mobile terminal; ^βk represents the geocentric angle vector between each ground mobile terminal and the low-orbit satellite when the k-th batch of tasks starts to be executed; ^bk represents the visibility information vector between each ground mobile terminal and the ground cloud server when the k-th batch of tasks starts to be executed; ^Uk represents the battery usage status information vector of each low-orbit satellite when the k-th batch of tasks starts to be executed.

However, since s _k has infinite state values and the spatial dimension grows exponentially with the increase in the number of tasks, this poses a great challenge to obtaining an efficient computing resource allocation strategy. Therefore, the present invention designs a state evaluation function under the constraints of the optimization problem to reflect the quality of the current state s _k under the action a _k , so as to achieve the purpose of simplifying the state space s _k with infinite values. The state evaluation function g _k can be expressed as a vector group composed of binary variables, and the state evaluation function g _k is expressed as:

g _k ={g ^k,1 ,g ^k,2 ,g ^k,3 },

in, Indicates that the state s _k cannot satisfy the task generated by the low-orbit satellite for the kth batch of the mth ground mobile terminal under the action a _k The corresponding third constraint C ₃ (i.e., coverage time constraint) is It means that the state s _k can satisfy the task generated by the low-orbit satellite for the kth batch of the mth ground mobile terminal under the action a _k . The corresponding third constraint C ₃ (i.e., coverage time constraint) is It means that the state s _k cannot satisfy the fourth constraint condition corresponding to the low-orbit satellite n under the action a _k (that is, the computing resources allocated to the low-orbit satellite n should not exceed the upper limit of the computing resources owned), that is, on the contrary, Right now It means that the state s _k cannot satisfy the fifth constraint condition corresponding to the low-orbit satellite n (i.e., the constraint that the battery state of the low-orbit satellite n is always greater than 0) under the action a _k ; otherwise,

Action space: Each action in the action space of the reinforcement learning model includes the task scheduling method and the computing resources allocated to each task by the ground mobile terminal, low-orbit satellite and ground cloud server. Specifically, the action a _k ∈ A executed by the k-th batch of tasks in the action space of the reinforcement learning model is expressed as:

a _k ={c ^k ,f ^k,GMT ,f ^{k ,LEO} ,f ^{k ,GCS} }

Among them, c ^k represents the task scheduling method vector of the k-th batch of task set, f ^k,GMT represents the computing resource vector allocated by the ground mobile terminal to each task in the k-th batch of task set, f ^k,LEO represents the computing resource vector allocated by the low-orbit satellite to each task in the k-th batch of task set, and f ^k,GCS represents the computing resource vector allocated by the ground cloud server to each task in the k-th batch of task set.

The value of the allocated computing resources is artificially determined, and the value is determined by discretizing the maximum allocable computing resources.

Reward function: The instantaneous reward function r _k is considered to be the feedback of the environment under the state s _k under the action a _k . In the computing resource allocation problem with the optimization goal of minimizing the weighted system energy consumption overhead consisting of the ground mobile terminal energy consumption and the low-orbit satellite energy consumption for task processing, the instantaneous reward function r _k of the reinforcement learning model can be expressed as:

in, The tasks generated for the kth batch of the mth ground mobile terminal The task processing energy consumption of ground mobile terminals, The tasks generated for the kth batch of the mth ground mobile terminal The mission processing energy consumption of low-orbit satellites.

The parameter means the weight of ground mobile terminal energy consumption in the system energy consumption overhead, and its value range is [0,1].

At this time, the optimization goal is described as a computing resource allocation strategy π ^* that can maximize the cumulative benefit function. For the computing resource allocation strategy π:S→A of the system, the cumulative benefit function executed until the start of the kth batch of tasks can be expressed as:

Among them, γ∈[0,1] is used as the profit discount rate to map the importance of future profits, _Eπ [·] represents the expectation under the possible strategy π, K represents the total number of task batches to be processed, k' represents the task batch in the calculation process, which is used for the profit summation calculation, and k represents the batch of the currently executed task. The difference between k' and k is that k' is a local variable introduced in the formula calculation, and k represents the kth batch of the task.

Step S4: solving a deep reinforcement learning model using a deep reinforcement learning algorithm based on an optimized DQN (deep Q network), wherein the discrete state generated by mapping the environmental state information through a state evaluation function is input into the reinforcement learning model as input information;

The reinforcement learning model constructed in step S3 above uses the state evaluation function to replace the original action space, thereby mapping the potentially infinite number of system states to a discrete and finite state evaluation function. However, this reinforcement learning model still has a discrete high-dimensional input and action space.

Therefore, in order to efficiently solve the high-performance computing resource allocation strategy, in step S4 of the present invention, the reinforcement learning model is a reinforcement learning model based on optimized DQN, DNN is introduced on the traditional reinforcement learning model, and the neural network parameter θ is used to fit the actual Q function Q( _sk , _ak ) to iteratively update the neural network parameter θ, and the optimal result of the fitting Q function finally obtained is the optimal strategy evaluation function Q ^* ( _sk , _ak ), that is, Q( _sk , _ak ;θ)≈Q ^* ( _sk , _ak ), Q( _sk , _ak ;θ) represents the fitting Q function of taking the action _ak in the _sk state obtained by fitting the neural network parameter θ. At this time, the corresponding neural network is the solved deep reinforcement learning model, and the deep reinforcement learning model is solved.

Among them, the Q function Q(s _k , _ak ) of the state-action pair (s _k , _ak )∈A×S is used to represent the quality of the selected state-action pair. Based on the Bellman equation, the calculation method of the optimal policy evaluation function Q ^* (s _k , _ak ) can be expressed as E represents the expectation under the uncertainty of s _k+1 , γ represents the discount rate of future benefits, _and Q ^* (s _k+1 , a _k+1 )|s _k , a _k represents the optimal strategy evaluation function Q ^* (s _k , a _k ) of taking action a _k+1 in state s _{k+1 under} the condition of s k, a k. Therefore, the method proposed in the present invention overcomes the bottlenecks of storage space and computing efficiency encountered by traditional reinforcement learning methods by adapting the deep reinforcement learning algorithm based on optimized DQN, reduces system energy consumption and improves network performance.

The computing architecture of the intelligent body of the energy-optimized low-orbit satellite edge computing resource allocation method designed by the present invention is shown in Figure 2.

In the low-orbit satellite edge computing network, the ground cloud server, as an intelligent agent, obtains the optimized computing resource allocation strategy by executing the energy-optimized low-orbit satellite edge computing resource allocation method of the present invention, and distributes the optimized strategy to each ground mobile terminal, low-orbit satellite and ground cloud server in the network. In step S1, the intelligent agent collects environmental state information (from the above definition, it can be seen that the environmental state information specifically includes the following information: task status information generated by each ground mobile terminal in the low-orbit satellite edge computing network, geocentric angle information between each ground mobile terminal and the low-orbit satellite, visibility information between each ground mobile terminal and the ground cloud server, and battery usage status information of each low-orbit satellite). Secondly, the intelligent agent maps the environmental state information through the state evaluation function to generate a discrete state reflecting the quality of the current state as input information, and inputs it into the network of the deep reinforcement learning algorithm based on the optimized DQN.

The network of this deep reinforcement learning algorithm consists of two parts, namely the online network and the target network, which are used to stabilize and optimize network performance. The online network updates the corresponding strategy by minimizing the loss function gradient update, and the target network is used to limit the update range of the online network strategy and stabilize network performance. Among them, the neural network parameters of the online network and the target network are defined as θ and θ ^- respectively. The online network and the target network have the same network structure. The target network copies the network parameters θ from the online network every certain number of iterations to update its own network parameters θ ^- .

The network parameters θ of the online network are updated by minimizing the corresponding loss function in each iteration. The loss function can be expressed as:

Where y represents the Q function value of the target network, Q(s _k , _ak ;θ) represents the fitted Q function of taking action a _k in state s _k obtained by fitting the network parameters θ of the online network, E[] represents the expectation under the uncertainty of experience (s _k , _ak , _rk ,s _k+1 ), and L _π (θ) represents the loss function under strategy π.

The calculation method of the Q function value y of the target network can be expressed as:

Wherein, Q(s _k+1 , _ak+1 ;θ ⁻ ) represents the fitted Q function of taking action a _k in state s _k obtained by fitting the network parameters θ ⁻ of the target network, γ is the benefit discount rate, and r _k is the instantaneous benefit function r _k of the reinforcement learning model.

In addition, as an offline strategy method, DQN uses the experience replay mechanism. When each task batch k is executed, DQN stores the experience (s _k , a _k , r _k , s _k+1 ) acquired by the agent in the experience replay pool, and then randomly samples small batches of samples from the experience replay pool for update each time the network parameters are updated. The present invention uses the state evaluation function g _k to replace the state s _k , replaces the experience of the agent with (g _k , a _k , r _k , g _k+1 ), simplifies the input state space, and performs parameter update.

After the network of the deep reinforcement learning algorithm has collected enough sample experience sets that reflect the interaction between the training environment and the agent, and obtained a stable and convergent computing resource allocation strategy by replaying the sample experience of a small batch, the training optimization ends and the iteration stops. Whether the network has collected enough sample experience sets that reflect the interaction between the training environment and the agent can be determined by observing whether the benefits of the computing resource allocation strategy obtained converge and stabilize, or by whether the loss function of the online network converges to zero.

Step S5: Based on the solved deep reinforcement learning model, obtain the energy-optimized computing resource allocation strategy and distribute it to various ground mobile terminals, low-orbit satellites and ground cloud servers in the system to realize computing resource allocation.

In step S5, the intelligent agent uses the kth batch of collected environmental state information (specifically including the task state information generated by each ground mobile terminal in the low-orbit satellite edge computing network, the geocentric angle information between each ground mobile terminal and the low-orbit satellite, the visibility information between each ground mobile terminal and the ground cloud server, and the battery usage status information of each low-orbit satellite) as the state _sk input, and calculates to obtain the state evaluation function _gk ; then the reinforcement learning model established in step S3 and the deep reinforcement learning algorithm based on the optimized DQN adopted in step S4 are used to solve, and the computing resource allocation strategy _ak = {ck, ^{fk , GMT} , fk ^{, LEO} , fk ^{, GCS} } is output to obtain the scheduling mode of each task and the computing resource allocation status of each ground mobile terminal, low-orbit satellite and ground cloud server in the system { ^{fk, GMT} , ^{fk, LEO} , fk ^{, GCS} }, and distribute them to the corresponding devices in the system.

Therefore, the energy consumption optimized low-orbit satellite edge computing resource allocation method of the present invention has the advantages of:

1) In a low-orbit satellite edge computing network that includes ground mobile terminals, low-orbit satellites, and ground cloud servers, the ground cloud server is used as an intelligent agent. Dynamic characteristics including the dynamic coverage of low-orbit satellites for tasks, the maximum computing resources that can be allocated by low-orbit satellites, and the battery usage status on low-orbit satellites are considered. The weighted system energy consumption overhead composed of the energy consumption of ground mobile terminals and low-orbit satellites is minimized as the optimization goal, and computing resources are allocated within the system for computing tasks on ground mobile terminals. Using intelligent agents to allocate computing resources in a dynamic low-orbit satellite edge computing network can reduce the energy consumption overhead of ground mobile terminals and satellites and improve the performance of the low-orbit satellite edge computing network.

2) Aiming at the dual energy consumption optimization goals of low-orbit satellites and ground mobile terminals, the weighted system energy consumption overhead is defined as the optimization target. The deep reinforcement learning method is introduced to solve the problem of computing resource allocation in the dynamic low-orbit satellite edge computing network. The core elements of the reinforcement learning model are defined based on the MDP framework, and the state evaluation function is defined for optimizing the state space. The algorithm solution and generation strategy distribution method based on the optimized DQN are proposed. Considering the characteristics of high-speed movement and limited resources of low-orbit satellites, the proposed method has obvious performance advantages in computing efficiency and system energy consumption overhead in the dynamic low-orbit satellite edge computing network.

The method of the present invention constructs an optimization problem model with the goal of minimizing the weighted system energy consumption overhead of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite, so that the intelligent agent distributes the system's optimal computing resource allocation strategy under the condition of considering the high-speed movement of the low-orbit satellite, limited energy and computing resources, completes the task execution, realizes the computing resource allocation of the ground mobile terminal, low-orbit satellite and ground cloud server in the low-orbit satellite edge computing network and reduces the system energy consumption overhead; in addition, the core elements of the optimization problem under the reinforcement learning model are defined with MDP as the framework, and the state evaluation function is designed according to the system constraints to optimize the state space to obtain the system's computing resource allocation strategy, thereby realizing an efficient computing resource allocation strategy and improving computing efficiency. In addition, the present invention is based on the deep reinforcement learning algorithm that optimizes DQN, further efficiently calculates the resource allocation strategy and improves computing efficiency.

In summary, the deep reinforcement learning algorithm designed by the present invention based on the optimized DQN solves the problem of low-orbit satellite edge computing resource allocation with energy consumption optimization in the low-orbit satellite edge computing network, improves computing efficiency, and reduces system energy consumption overhead.

Experimental results:

Below, taking a scenario of 5 ground mobile terminals, 3 low-orbit satellites and 2 ground cloud servers as an example, a specific example of the energy-optimized low-orbit satellite edge computing resource allocation method of the present invention is given.

According to step S1, the intelligent agent is used to obtain the environmental status information of the dynamic low-orbit satellite edge computing network.

In this experimental example, the computing resource allocation scenario of the low-orbit satellite edge computing network is shown in Figure 3. The low-orbit satellite edge computing network uses ground cloud servers as intelligent entities, including M ground mobile terminals, N low-orbit satellites and J ground cloud servers, specifically M = 5, N = 3, J = 2. Assuming that all low-orbit satellites are operating on circular orbits, the low-orbit satellite orbit model is shown in Figure 4. Among them, the orbit height is expressed as H = 800km, and the earth radius is expressed as R = 6370km.

According to step S2, based on the acquired environmental status information, an optimization problem model is constructed with minimizing the system energy consumption overhead as the optimization goal. The system energy consumption overhead is defined as the weighted sum of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite.

In order to solve the computing resource allocation problem with the optimization goal of minimizing the system energy consumption overhead, the intelligent agent (ground cloud server) uses the acquired network environment status information to mathematically model the optimization problem under the constraints of the actual dynamic low-orbit satellite edge computing network (the coverage time constraints of the low-orbit satellite for the task, the computing resource constraints allocated by the low-orbit satellite, and the battery usage status constraints of the low-orbit satellite).

Specifically, when the task When the local execution strategy is selected, the task processing delay and energy consumption are calculated as follows: Among them, ζ represents the energy consumption coefficient of the chip.

When the task When the strategy of scheduling to a low-orbit satellite is selected, the task processing delay can be obtained by the following calculation method: in, represents the distance from the ground mobile terminal m to the low-orbit satellite n, c represents the propagation speed of light, Indicates the task The upload rate of data uploaded to low-orbit satellite n. It can be expressed as The energy consumption of the ground mobile terminal for task processing can be expressed as in, represents the uplink transmission power of the ground mobile terminal m. In addition, the energy consumption of the low-orbit satellite can be expressed as in, Represents the energy consumption of acquiring each bit of mission data for a low-orbit satellite.

When the task When the strategy of dispatching from low-orbit satellites to ground cloud servers for processing is selected, the task processing delay can be obtained by the following calculation method: in, represents the distance from low-orbit satellite n to ground cloud server j, Indicates the task The download rate offloaded to the ground cloud server j through the low-orbit satellite n. The energy consumption of the ground mobile terminal for task processing can be expressed as The energy consumption of low-orbit satellites for mission processing can be expressed as in, represents the downlink transmission power of low-orbit satellite n.

The present invention takes the Iridium system as an example, and the constraint condition of the battery usage status of the low-orbit satellite n at the beginning of the k+1 batch mission can be expressed as: Among them, U _max , They represent the maximum energy used by the battery on low-orbit satellite n, the energy obtained by low-orbit satellite n using solar panels, and the energy consumed by low-orbit satellite n to process the kth batch of tasks. It can be obtained by the following calculation method. represents the energy obtained by low-orbit satellite n using solar panels in the execution of batch k, represents the energy consumed by low-orbit satellite n in performing the kth batch of missions, represents the maximum delay required to execute the kth batch of tasks, Indicates the efficiency of converting solar energy into energy per second. It can be obtained by the following calculation method, _Pn represents daily energy consumption.

According to step S3, based on the optimization problem, the core elements of the reinforcement learning model are defined, and the state evaluation function is designed to optimize the state space.

The core elements of the reinforcement learning model using MDP modeling mainly include state space, action space and benefit function. In order to optimize the state space, the present invention designs a state evaluation function to replace the state space. In the context of a dynamic low-orbit satellite edge computing network, the specific design of the core elements of the optimization problem model is as follows:

State space design: Taking the state s _k ∈S when the kth batch of tasks starts to execute as an example, it includes the state information vector generated by the task set; the geocentric angle vector between each ground mobile terminal and the low-orbit satellite when the task starts to execute, which is used to reflect the coverage of the low-orbit satellite to the task; the visibility information vector between each ground mobile terminal and the ground cloud server when the task starts to execute, which is used to reflect the visibility of the ground cloud server to the task; the battery usage status information vector of each low-orbit satellite when the task starts to execute, which is used to reflect the battery usage status of the low-orbit satellite at this time.

Design of state evaluation function: It includes a vector group composed of three types of binary variables, which represent the quality of the current state under action, namely, the coverage time constraint of the low-orbit satellite for the task, the upper limit constraint of the computing resources allocated by the low-orbit satellite, and the battery usage status constraint of the low-orbit satellite.

Action space design: Taking the action a _k ∈ A executed on the k-th batch of tasks as an example, it includes the task scheduling method and the computing resources allocated to each task by the ground mobile terminal, low-orbit satellite and ground cloud server.

Revenue function design: Taking the feedback r _k of state s _k under action a _k as an example, it is described as the system energy consumption overhead composed of the energy consumption of the ground mobile terminal caused by task processing and the energy consumption of the low-orbit satellite. The system optimization goal is to maximize the cumulative revenue function.

According to step S4, the deep reinforcement learning model is solved using a deep reinforcement learning algorithm based on optimized DQN.

Specifically, the DQN-based computing resource allocation algorithm provided in the present invention includes the following steps:

Step S41: Initialize the experience replay pool U and the online neural network parameters θ;

Initializing the experience replay pool means clearing the sample cache and randomly generating the initial values of the neural network parameters.

Step S42: Initialize the target neural network parameter θ ^- ←θ;

Step S43: Initialize the number of training rounds v to 1;

Step S44: Initialize the evaluation function g ₀ of the environment and network environment status;

The evaluation function is defined in binary form according to step S3 of the specific technical solution, and the initial value is set to a vector consisting of 1s.

Step S45: Initialize the task batch k in the current training round number v to 1;

Step S46: randomly select an action a _k according to the ε-greedy strategy, otherwise a _k =argmax _a∈A Q(g _k ,a;θ); wherein the ε-greedy strategy refers to randomly selecting an action with a probability of e(0<e<1), otherwise adopting the action with the maximum action value.

Step S47: Execute action a _k and obtain the evaluation function g _k+1 and benefit function r _k of the next network environment state;

Step S48: storing (g _k , a _k , r _k , g _k+1 ) experience data into the experience replay pool U;

Step S49: Randomly sample small batch samples ( _gi , _ai , _ri , gi ₊₁ ) from U; the small batch samples are used to update the network parameters θ and ^θ- of the online network and the target network.

Step S410: Calculate the loss function L(θ) of the Q function value of the online network and the target network using the small batch samples, and use the loss function to perform small batch gradient descent to update the network parameter θ of the online network;

Step S411: every τ ^-batch , the network parameters of the target network are updated by θ ^- =θ; τ ^- means the step length for periodically updating the target network, and the value range is greater than 0.

Step S412: determine whether k<K is satisfied, where K is the set threshold of the task execution batch. If so, k=k+1, proceed to step S46, thereby iteratively updating the network parameters θ and θ ^- of the online network and the target network; otherwise, proceed to step S413;

Step S413: Determine whether v<V, where V is a threshold for the number of training round iterations. If so, v=v+1, and go to step S44. Otherwise, the optimization ends and a trained deep reinforcement learning model is obtained.

Step S5: Based on the solved deep reinforcement learning model, obtain the energy-optimized computing resource allocation strategy and distribute it to various ground mobile terminals, low-orbit satellites and ground cloud servers in the system to realize computing resource allocation.

The optimal computing resource allocation strategy under the dynamic low-orbit satellite edge computing network can be obtained by using the deep reinforcement learning model obtained by the DQN algorithm training convergence solution. The k-th batch of collected environmental state information (specifically including the task state information generated by each ground mobile terminal in the low-orbit satellite edge computing network, the geocentric angle information between each ground mobile terminal and the low-orbit satellite, the visibility information between each ground mobile terminal and the ground cloud server, and the battery usage status information of each low-orbit satellite) is used as the state _sk input to calculate and obtain the state evaluation function _gk . The reinforcement learning model established in step S3 and the deep reinforcement learning algorithm based on DQN adopted in step S4 are used to solve and output the computing resource allocation strategy _ak = {ck, ^{fk , GMT} , ^{fk, LEO} , ^{fk, GCS} }, and the task scheduling mode and the computing resource allocation of each ground mobile terminal, low-orbit satellite and ground cloud server in the system { ^{fk, GMT} , ^{fk, LEO} , fk ^{, GCS} } are obtained, and distributed to the corresponding devices in the system.

The above is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. The above embodiments of the present invention can also be modified in various ways. All simple, equivalent changes and modifications made according to the claims and the contents of the specification of the present invention fall within the scope of protection of the claims of the present invention. The contents not described in detail in the present invention are all conventional technical contents.

Claims (5)

1. A method for allocating low-orbit satellite edge computing resources with energy consumption optimization, characterized by comprising: Step S1: Using the intelligent agent to obtain the environmental status information of the dynamic low-orbit satellite edge computing network; Step S2: Based on the acquired environmental status information, an optimization problem model is constructed with minimizing the system energy consumption overhead as the optimization goal. The system energy consumption overhead is defined as the weighted sum of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite; Step S3: Based on the optimization problem model, define the state space, action space and benefit function of the reinforcement learning model, and design a state evaluation function to optimize the state space; Step S4: solving the deep reinforcement learning model using a deep reinforcement learning algorithm based on optimized DQN, wherein the discrete state generated by mapping the environmental state information through the state evaluation function is input into the network of the deep reinforcement learning algorithm as input information; Step S5: Based on the solved deep reinforcement learning model, an energy-optimized computing resource allocation strategy is obtained and distributed to various ground mobile terminals, low-orbit satellites, and ground cloud servers to achieve computing resource allocation; The environmental status information of the low-orbit satellite edge computing network includes: the status information vector W ^k of the kth batch of task sets generated by the ground mobile terminal, the geocentric angle vector β ^k between each ground mobile terminal and the low-orbit satellite when the kth batch of tasks starts to be executed, the visibility vector b ^k between each ground mobile terminal and the ground cloud server when the task starts to be executed, and the battery usage status information vector U ^k of each low-orbit satellite when the kth batch of tasks starts to be executed; The step S1 comprises: Step S11: providing a low-orbit satellite edge computing network consisting of M ground mobile terminals and J ground cloud servers located on the ground, and N low-orbit satellites located in space; the set of ground mobile terminals, the set of low-orbit satellites and the set of ground cloud servers are respectively expressed as M'＝{1,...,m,...,M}, N'＝{1,...,n,...,N} and J'＝{1,...,j,...,J}, m, n, j respectively represent the ordinal number of the ground mobile terminal, the ordinal number of the low-orbit satellite and the ordinal number of the ground cloud server, and M, N, J are the number of ground mobile terminals, the number of low-orbit satellites and the number of ground cloud servers respectively; setting each ground mobile terminal to be able to connect to at most one low-orbit satellite at a time; and setting each ground mobile terminal to be able to establish a connection with at most one ground cloud server through a low-orbit satellite at a time; Step S12: Set each ground mobile terminal to generate only one indivisible computing task in each batch; then, the set K of task batches to be executed by the entire low-orbit satellite edge computing network is represented as: K'={1,...,k,...,K}, k represents the kth task batch, and K is the total number of task batches; the task batch generated by the kth batch of the mth ground mobile terminal is described as in, Expressed as the data size of the task payload, It is expressed as the number of CPU processing cycles required for the task load. The state information vector ^Wk of the k-th batch of task sets generated by the ground mobile terminal is defined as M is the number of ground mobile terminals; Step S13: Set the low-orbit satellites to run on circular orbits, and represent the orbit height as H, the radius of the earth as R, and the elevation angle between the ground mobile terminal m and the low-orbit satellite n when starting to execute the kth batch of tasks as Obtain the geocentric angle vector β ^k between each ground mobile terminal and the low-orbit satellite when the k-th batch of tasks starts to be executed, as well as the visibility duration of each low-orbit satellite in the entire low-orbit satellite edge computing network to each ground mobile terminal when executing the k-th batch of tasks; Step S14: Initialize the visibility vector ^bk between each ground mobile terminal and the ground cloud server when the task starts to be executed and the battery usage status information vector ^Uk of each low-orbit satellite when the kth batch of tasks starts to be executed; The step S2 comprises: Step S21: The task scheduling mode vector corresponding to the state information vector ^Wk of the kth batch of task sets generated by the ground mobile terminal is defined as The tasks generated for the kth batch of the mth ground mobile terminal The decision vector for scheduling to each LEO satellite in the LEO satellite edge computing network, The task generated for the kth batch of the mth ground mobile terminal The decision vectors for scheduling to various ground cloud servers in the low-orbit satellite edge computing network. Multiple tasks in the same batch of task sets of all ground mobile terminals can choose different task scheduling methods; task scheduling methods include: local processing, transmission to low-orbit satellites for processing, and transmission to ground cloud servers via low-orbit satellites for processing; Step S22: determining the processing delay of each task in the task set, the task processing energy consumption of the ground mobile terminal, and the task processing energy consumption of the low-orbit satellite according to the environmental state information and task scheduling mode vector of the task set of the kth batch obtained; Step S23: defining the weighted sum of the task processing energy consumption of the ground mobile terminal and the task processing energy consumption of the low-orbit satellite as the system energy consumption overhead, and constructing an optimization problem model with minimizing the system energy consumption overhead as the optimization goal; The task generated by the kth batch of the mth ground mobile terminal Decision vectors for scheduling to each LEO satellite in the LEO satellite edge computing network for: in, represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to low-orbit satellite n for execution; represents the task generated by the kth batch of the mth ground mobile terminal Not dispatched to low-orbit satellite n for execution; The task generated by the kth batch of the mth ground mobile terminal Decision and scheduling of each low-orbit satellite in the low-orbit satellite edge computing network for The tasks generated by the kth batch of the mth ground mobile terminal Decision vector for scheduling to various ground cloud servers in the LEO satellite edge computing network for: in, represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to the ground cloud server j for execution via low-orbit satellite n; represents the task generated by the kth batch of the mth ground mobile terminal It is not dispatched to the ground cloud server j for execution through the low-orbit satellite n; The task generated by the kth batch of the mth ground mobile terminal The decision and for The optimization problem model is: Wherein, C ₁ , C ₂ , C ₃ , C ₄ , and C ₅ represent the first, second, third, fourth, and fifth constraints, respectively; represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to low-orbit satellite n for execution; represents the task generated by the kth batch of the mth ground mobile terminal Not dispatched to low-orbit satellite n for execution; represents the task generated by the kth batch of the mth ground mobile terminal It is dispatched to the ground cloud server j for execution via low-orbit satellite n; represents the task generated by the kth batch of the mth ground mobile terminal It is not dispatched to the ground cloud server j for execution through the low-orbit satellite n; They are the tasks generated by the kth batch of the mth ground mobile terminal. The processing delay when the task scheduling method is to transmit to a low-orbit satellite for processing and transmit via a low-orbit satellite to a ground cloud server for processing; is the visibility duration of low-orbit satellite n to ground mobile terminal m when performing the kth batch of tasks; The tasks generated for the kth batch of the mth ground mobile terminal for the low-orbit satellite n Allocated computing resources; z ^LEO is the upper limit of computing resources owned by a single low-orbit satellite; is the battery usage status of low-orbit satellite n when the k-th batch of tasks begins to execute; f ^k,GMT represents the computing resource vector allocated by the ground mobile terminal to each task in the k-th batch of tasks, f ^k,LEO represents the computing resource vector allocated by the low-orbit satellite to each task in the k-th batch of tasks, and f ^k,GCS represents the computing resource vector allocated by the ground cloud server to each task in the k-th batch of tasks; The tasks generated for the kth batch of the mth ground mobile terminal Task processing energy consumption in ground mobile terminals, The tasks generated for the kth batch of the mth ground mobile terminal The energy consumption of mission processing in low-orbit satellites; α represents the weight of ground mobile terminal energy consumption in the system energy consumption overhead. 2. The energy-optimized low-orbit satellite edge computing resource allocation method according to claim 1 is characterized in that the visible time of the low-orbit satellite n to the ground mobile terminal m when executing the kth batch of tasks is for: Wherein, T ^LEO is the operation period of the low-orbit satellite, is the geocentric angle between the ground mobile terminal m and the low-orbit satellite n; The geocentric angle between the ground mobile terminal m and the low-orbit satellite n for: Where R is the radius of the Earth, H is the orbital altitude, is the elevation angle between the ground mobile terminal m and the low-orbit satellite n at the beginning of the kth batch of missions; The operating period of a low-orbit satellite, T ^LEO , is: Among them, R is the radius of the earth, H is the orbital altitude, and μ represents the Kepler constant. 3. The energy-optimized low-orbit satellite edge computing resource allocation method according to claim 1, characterized in that each state _sk in the state space of the reinforcement learning model includes a state information vector ^Wk of a set of tasks of the kth batch generated by a ground mobile terminal, a geocentric angle vector ^βk between each ground mobile terminal and the low-orbit satellite when the kth batch of tasks starts to be executed, a visibility vector ^bk between each ground mobile terminal and the ground cloud server when the task starts to be executed, and a battery usage state information vector ^Uk of each low-orbit satellite when the kth batch of tasks starts to be executed; The state evaluation function g _k is: g _k ={g ^k,1 ,g ^k,2 ,g ^k,3 }, in, Indicates that the state s _k cannot satisfy the task generated by the low-orbit satellite for the kth batch of the mth ground mobile terminal under the action a _k The corresponding third constraint C ₃ ; It means that the state s _k can satisfy the task generated by the low-orbit satellite for the kth batch of the mth ground mobile terminal under the action a _k . The corresponding third constraint C ₃ ; It means that the state s _k cannot satisfy the fourth constraint condition corresponding to the low-orbit satellite n under the action a _k . On the contrary, It means that the state s _k cannot satisfy the fifth constraint condition corresponding to the low-orbit satellite n under the action a _k . On the contrary, The action a _k performed by the k-th batch of tasks in the action space of the reinforcement learning model includes: a _k ={c ^k ,f ^k,GMT ,f ^{k ,LEO} ,f ^{k ,GCS} }, Wherein, c ^k represents the task scheduling method vector of the k-th batch of task set, f ^k,GMT represents the computing resource vector allocated by the ground mobile terminal to each task in the k-th batch of task set, f ^k,LEO represents the computing resource vector allocated by the low-orbit satellite to each task in the k-th batch of task set, and f ^k,GCS represents the computing resource vector allocated by the ground cloud server to each task in the k-th batch of task set; The benefit function of the reinforcement learning model includes an instantaneous benefit function and a cumulative benefit function; The instantaneous reward function r _k of the reinforcement learning model is: in, The tasks generated for the kth batch of the mth ground mobile terminal Task processing energy consumption in ground mobile terminals, The tasks generated for the kth batch of the mth ground mobile terminal The energy consumption of mission processing in low-orbit satellites; α represents the weight of ground mobile terminal energy consumption in the system energy consumption overhead; The optimization objective is described as a computing resource allocation strategy π ^* that can maximize the cumulative benefit function. For the computing resource allocation strategy π:S→A, the cumulative benefit function executed until the start of the k-th batch of tasks is expressed as: Among them, γ∈[0,1] is used as the profit discount rate to map the importance of future profits, _Eπ [·] represents the expectation under the possible strategy π, K represents the total number of task batches to be processed, k' represents the task batch in the calculation process, and k represents the batch of the currently executed task. 4. The energy-optimized low-orbit satellite edge computing resource allocation method according to claim 3 is characterized in that, in step S4, DNN is introduced into the reinforcement learning model, and the neural network parameter θ is iteratively updated by using the fitting Q function obtained by fitting the actual Q function Q(s _k , _ak ) using the neural network parameter θ of DNN, and the optimal result of the fitting Q function finally obtained is the optimal strategy evaluation function Q ^* (s _k , _ak ), and the deep reinforcement learning model is solved at this time. 5. The energy-optimized low-orbit satellite edge computing resource allocation method according to claim 1 is characterized in that, in step S5, the intelligent agent uses the collected environmental state information of the kth batch as the state _sk input, and calculates to obtain the state evaluation function _gk ; then the optimization problem model established in step S3 and the deep reinforcement learning algorithm based on optimized DQN adopted in step S4 are used to solve, and the computing resource allocation strategy _ak = { ^ck , ^{fk, GMT} , fk ^{, LEO} , fk ^{, GCS} } is output to obtain the computing resource allocation conditions {fk ^{, GMT} , ^{fk, LEO} , ^{fk , GCS} } of each task scheduling mode and each ground mobile terminal, low-orbit satellite and ground cloud server, and distribute them to each ground mobile terminal, low-orbit satellite and ground cloud server; Among them, c ^k represents the task scheduling method vector of the k-th batch of task set, f ^k,GMT represents the computing resource vector allocated by the ground mobile terminal to each task in the k-th batch of task set, f ^k,LEO represents the computing resource vector allocated by the low-orbit satellite to each task in the k-th batch of task set, and f ^k,GCS represents the computing resource vector allocated by the ground cloud server to each task in the k-th batch of task set.