CN113726858B - Self-adaptive AR task unloading and resource allocation method based on reinforcement learning - Google Patents

Abstract

The invention discloses an adaptive AR task offloading and resource allocation method based on reinforcement learning, including selecting AR tasks with different video frame resolutions at the user end, and obtaining corresponding accuracy rates at the same time; Based on the delay, user energy consumption and user cost, a user experience model is established, thereby forming a joint optimization problem with the goal of improving user experience; the optimization problem is transformed into a Markov decision process, and its state space, action space and Reward setting, the reinforcement learning network is designed according to the Markov decision process; the reinforcement learning network is trained until the network converges; after the network is trained, the state of the user terminal and the MEC server is input into the network to obtain the corresponding strategy. The present invention comprehensively considers the different requirements of the user terminal on the accuracy rate, user energy consumption and user cost, effectively improves the user experience within the task delay threshold, and achieves the Nash equilibrium of the user experience.

Description

An adaptive AR task offloading and resource allocation method based on reinforcement learning

technical field

The invention belongs to the technical field of wireless communication, and in particular relates to an adaptive AR task offloading and resource allocation method based on reinforcement learning.

Background technique

With the development of the fifth-generation mobile communication technology, more and more computing-intensive and delay-sensitive applications are emerging. Among them, Augmented Reality (AR) technology has been gradually applied in intelligent inspection, urban operation and maintenance, and industrial production, and can obtain a large amount of data in real time. Process large amounts of data in a short amount of time. Traditional cloud computing can provide abundant computing resources, but there are disadvantages such as backhaul link congestion and transmission delay. Mobile Edge Computing (MEC) can provide services to users at the edge of the mobile network. Users can offload tasks to the MEC server for processing, which can reduce local load, save user energy consumption, and bring low latency and low energy consumption. and other advantages, so as to improve the comprehensive experience of users.

In the existing technologies, there have been many research works on optimizing computing task offloading delay or energy consumption. In addition, some research works also focus on optimizing user cost. However, in the current work, the user's task size is fixed, and there is no in-depth study of the change of user task requirements. Even for the same task, different users may have different specific needs, and existing research rarely considers the quality of user experience. For AR computing-intensive tasks, local resources are limited and need to be offloaded to the MEC for processing, and different AR video frame resolutions will bring different accuracy rates and thus different user experiences.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the shortcomings of the above-mentioned existing work, and to provide an adaptive AR task offloading and resource allocation method based on reinforcement learning, which can effectively improve user experience and achieve Nash equilibrium of user experience.

The present invention adopts following technical scheme to realize:

An adaptive AR task offloading and resource allocation method based on reinforcement learning, comprising the following steps:

1) The user terminal selects AR tasks with different video frame resolutions, and simultaneously obtains the corresponding accuracy rates of the AR tasks with the video frame resolutions;

2) Partially offload AR tasks with different video frame resolutions selected by the client. First, the AR tasks are divided into different proportions, and then the tasks are partially offloaded to different MEC servers under the allocation of different computing resources and wireless resources. Latency, user energy consumption and user fees caused by local execution;

3) Under the constraint of task delay threshold, a user experience model is established by comprehensively considering the accuracy rate, user energy consumption and user cost; by jointly optimizing the AR video frame resolution selection strategy, task part offloading strategy, computing resource and wireless resource allocation strategy , forming an optimization problem with the goal of improving user experience;

4) Transform the optimization problem into a Markov decision process, initialize its state space, action space and reward settings; design a reinforcement learning network according to the Markov decision process;

5) Train the reinforcement learning network through the MADDPG algorithm until the network converges;

6) After the network is trained, input the state of the client and MEC server into the network to get the corresponding strategy.

表示视频帧分辨率的集合，其中

为用户集合；不同的帧分辨率对应有不同的任务大小和时延门限，用户i的计算任务表示为τ_i＝{d_i,c_i，thr_i}，分别包括任务的大小、计算量和时延门限，d_i＝δv_i ²为任务大小，其中δ定义为表示一个像素信息所需的比特数，任务计算量与任务大小之间的关系为c_i＝η_id_i，其中η_i为计算密度，即单位任务所需的计算量，根据AR视频帧分辨率与准确率的关系，用户i得到的准确率Q_i为：A further improvement of the present invention is that, in step 1), it is assumed that the AR video selected by user i is preprocessed into a video frame with a video frame resolution of v _i ×vi pixels, that is, the video frame resolution of user _i is

represents the set of video frame resolutions, where

is a set of users; different frame resolutions correspond to different task sizes and delay thresholds, and the computing task of user i is expressed as τ _i ={d _i , _{ci , thr i }} , including the size of the task, the amount of computation and Time delay threshold, d _i =δv _i ² is the task size, where δ is defined as the number of bits required to represent one pixel information, and the relationship between task computation and task size is c _i =η _i d _i , where η _i In order to calculate the density, that is, the amount of calculation required for a unit task, according to the relationship between the AR video frame resolution and the accuracy rate, the accuracy rate Qi obtained by user _i is:

A further improvement of the present invention is that, in step 2), since the partial unloading of tasks can be executed in parallel, for the task of user i, the time delay T _i required to complete the execution is:

Among them, TL _i is the local execution delay of some tasks of user i, and TM _i,k is the execution delay of unloading some tasks of user i to MEC server k.

A further improvement of the present invention is that the local execution time delay TL _i of some tasks of user i is:

Among them, ci represents the task calculation amount of user _i , fi represents the local computing resources allocated by user _i to process the current task, and b _i,k represents the proportion of tasks that user i unloads to MEC server k;

The time delay TM _{i, k} for offloading part of the tasks of user i to the MEC server k is:

Among them, d _i represents the task size of user _{i, r i,k} represents the uplink transmission rate between user i and MEC server k, and m _i,k represents the computing resource size of MEC server k allocated by user i;

The user energy consumption E _i of user i performing the current task is:

Among them, EL _i is the local processing energy consumption, EM _i,k is the transmission energy consumption when the user unloads some tasks to the MEC server k;

The local processing energy consumption EL _i is:

Among them, θ represents the energy density required to process the task, and f _i is the locally allocated computing resource;

The transmission energy consumption EM _{i when the user unloads some tasks to the MEC server k, k} is:

Among them, P _i represents the user transmission power.

A further improvement of the present invention is that, in step 2), when the user i unloads the task to the MEC server for execution, according to the size c _i of the task calculation amount, the unloaded task ratio b _i,k , and the unit price ε charged by the MEC server, need to pay the corresponding The cost _Wi :

A further improvement of the present invention is that, in step 3), in order to improve the comprehensive experience of each user, achieve the Nash equilibrium of each user's experience, reduce user energy consumption and user cost, and improve the accuracy rate of user AR tasks, therefore, combining these three Factors that form a user experience model:

U _i =ξ _Q Q _i -ξ _E E _i -ξ _W Wi _i

任务部分卸载策略

无线资源分配策略

本地和MEC计算资源分配策略

形成最大化各用户体验为目标的优化问题：Under the constraints of delay threshold, computing resources and wireless resources, jointly optimize the user's AR video frame resolution selection strategy

Task Section Uninstall Policy

Radio resource allocation strategy

Local and MEC computing resource allocation strategies

Form an optimization problem with the goal of maximizing the user experience:

A further improvement of the present invention is that in step 4), the state space includes the size of all user computing resources, the size of MEC computing resources, the idle wireless resources of the base station, the allocation scheme of initial computing resources and wireless resources, the AR video frame resolution selection scheme, The task part offloading scheme; the action space is the allocation scheme for initial computing resources and wireless resources, the AR video frame resolution selection scheme, and the change amount of the task partial offloading scheme; the reward is to set three layers according to the constraints (c1)-(c5) award.

A further improvement of the present invention is that in step 5), first obtain new states, actions and rewards according to the reinforcement learning network, store them in the experience playback pool, and then start training the designed reinforcement learning network when the data in the experience playback pool reaches the threshold , and continuously update the data and network parameters in the experience replay pool until the network training converges.

A further improvement of the present invention is that, in step 6), after the network is trained, in the specific application process, the state at the current moment is input into the network to obtain the AR video frame resolution selection strategy, the task part unloading strategy, Computing resources and radio resource allocation strategies.

The present invention has at least the following beneficial technical effects:

In the reinforcement learning-based adaptive AR task offloading and resource allocation method provided by the present invention, during specific operations, the user terminal first selects AR tasks with different frame resolutions to obtain corresponding accuracy rates. The user divides the tasks in different proportions, and under the allocation of different computing resources and wireless resources, some are offloaded to different MEC servers and some are executed locally, and the corresponding delay, user energy consumption and user fees are calculated respectively. Under the constraint of task delay threshold, a user experience model is established, and an optimization problem aiming at improving user experience is formed by jointly optimizing the video frame resolution selection strategy, task part offloading strategy, computing resource and wireless resource allocation strategy. Transform the optimization problem into a Markov decision process, initialize its state space and action space, and set the reward, start training the network according to the proposed MADDPG algorithm, and continuously update the data and network parameters in the experience playback pool until the network converges. Finally, the network obtains the frame resolution selection strategy, task offloading and resource allocation strategy according to the state of the client and the MEC server.

To sum up, the present invention comprehensively considers the different requirements of the user terminal for accuracy, user energy consumption and user cost, and jointly optimizes the AR video frame resolution selection strategy, the task part offloading strategy, the computing resource and the wireless resource allocation strategy, Within the task delay threshold, the user experience is effectively improved.

Description of drawings

Fig. 1: the schematic flow chart of the present invention;

Figure 2: Schematic diagram of user experience of each user (MEC computing capability is 5G Cycles/s);

Figure 3: Schematic diagram of user experience of each user (MEC computing capability is 7G Cycles/s);

Figure 4: Schematic diagram of user experience of each user (MEC computing capability is 9G Cycles/s);

Figure 5: Schematic diagram of user experience of each user (MEC computing capability is 11G Cycles/s);

Figure 6: Schematic diagram of average user experience when MEC computing power changes;

Figure 7: Schematic diagram of average energy consumption when MEC computing power changes;

Figure 8: Schematic diagram of average cost when MEC computing power changes;

Figure 9: Average Accuracy Rates as MEC Computational Power Changes

Figure 10: Schematic diagram of the user experience of each user (MEC unit price is 0.009);

Figure 11: Schematic diagram of user experience of each user (MEC unit price is 0.010);

Figure 12: Schematic diagram of user experience of each user (MEC unit price is 0.011);

Figure 13: Schematic diagram of user experience of each user (MEC unit price is 0.012);

Figure 14: Schematic diagram of the average user experience when the MEC unit price changes;

Figure 15: Schematic diagram of average energy consumption when the unit price of MEC changes;

Figure 16: Schematic diagram of the average cost when the unit price of MEC changes;

Figure 17: Schematic diagram of the average accuracy rate when the MEC unit price changes;

Figure 18: Schematic diagram of the average user experience when the number of users changes;

Figure 19: Schematic diagram of average energy consumption when the number of users changes;

Figure 20: Schematic diagram of the average cost when the number of users changes;

Figure 21: Schematic diagram of the average accuracy when the number of users varies.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art. It should be noted that the embodiments of the present invention and the features of the embodiments may be combined with each other under the condition of no conflict. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

The overall flow chart of the present invention is shown in accompanying drawing 1, and the present invention is described in detail below with reference to the accompanying drawings:

每个MEC服务器服务范围内的用户数为I，则系统内用户总数为KI，用户集合表示为

在每一决策周期，用户i需要处理视频帧分辨率为

的AR计算任务，对应的准确率Q_i为：The present invention is aimed at a multi-user multi-MEC server system. One MEC server is deployed next to each base station in the system, and there are K MEC servers in total, which are expressed as

The number of users in the service range of each MEC server is I, the total number of users in the system is KI, and the user set is expressed as

In each decision cycle, user i needs to process the video frame resolution of

The AR computing task of , the corresponding accuracy rate _Qi is:

The delay of some tasks offloaded to the MEC server k is defined as TM _i,k , and the delay of local execution is defined as TL _i . Since user tasks are executed in parallel, the total delay T _i required to execute the task is:

The energy consumed by user i to unload some tasks to the MEC server k for execution is defined as EM _i,k , and the energy consumed by local execution is defined as EL _i , then the total energy E _i consumed by user i is:

When the user i unloads the task to the MEC server for execution, according to the size of the task calculation c _i , the proportion of the unloaded task b _i,k , and the unit price ε charged by the MEC server, the corresponding fee W _i needs to be paid:

This results in a user experience model:

U _i =ξ _Q Q _i -ξ _E E _i -ξ _W Wi _i

The present invention aims at maximizing user experience, and forms the following optimization problem under the constraints of time delay threshold, computing resources and wireless resources:

s.t.

(c1)

(c2)

(c3)

(c4)

(c5

The reinforcement learning-based adaptive AR task offloading and resource allocation method of the present invention includes the following steps:

1) The user terminal selects AR tasks with different video frame resolutions, and simultaneously obtains the corresponding accuracy rates of the AR tasks with the video frame resolutions;

2) Partially offload AR tasks with different video frame resolutions selected by the client. First, the AR tasks are divided into different proportions, and then the tasks are partially offloaded to different MEC servers under the allocation of different computing resources and wireless resources. Latency, user energy consumption and user fees caused by local execution;

3) Under the constraint of task delay threshold, a user experience model is established by comprehensively considering the accuracy rate, user energy consumption and user cost; by jointly optimizing the AR video frame resolution selection strategy, task part offloading strategy, computing resource and wireless resource allocation strategy , forming an optimization problem with the goal of improving user experience;

4) Transform the optimization problem into a Markov decision process, initialize its state space, action space and reward settings; design a reinforcement learning network according to the Markov decision process;

5) Train the reinforcement learning network through the MADDPG algorithm until the network converges;

6) After the network is trained, input the state of the client and MEC server into the network to get the corresponding strategy.

The following is a detailed description with reference to Figure 1:

Step 11): The user terminal selects AR tasks with different video frame resolutions to obtain the corresponding accuracy rates. The specific process is:

的视频帧。不同的帧分辨率对应有不同的任务大小和时延门限，用户i的计算任务表示为τ_i＝{d_i,c_i,thr_i}，分别包括任务的大小、计算量和时延门限。d_i＝δv_i ²为任务大小，其中δ定义为表示一个像素信息所需的比特数。任务计算量与任务大小之间的关系为c_i＝η_id_i，其中η_i为单位任务所需的计算量。根据AR视频帧分辨率与准确率的关系，得到用户i的准确率Q_i为：Suppose that the AR video task selected by user i is preprocessed to a video frame resolution of

video frame. Different frame resolutions correspond to different task sizes and delay thresholds. The computing task of user i is expressed as τ _i ={d _i , _{ci ,thr i }} , including the task size, calculation amount and delay threshold, respectively. d _i =δv _i ² is the task size, where δ is defined as the number of bits required to represent one pixel information. The relationship between the task computation amount and the task size is c _i =n _i d _i , where n _i is the computation amount required for a unit task. According to the relationship between AR video frame resolution and accuracy rate, the accuracy rate Q _i of user i is obtained as:

Step 12): When the user i unloads some tasks to the MEC server k for execution, the consumed time delay TM _i,k is:

where b _i,k is the proportion of tasks unloaded by user i to MEC server k, d _i is the task size of user _i , ci is the task calculation amount of user i, and rv _i,k is the difference between user i and MEC server k Uplink transmission rate, m _i,k is the computing resource size of MEC server k allocated by user i;

At this time, the corresponding user transmission energy consumption EM _i,k is:

where P _i is the transmission power of user i;

When user i performs some tasks locally, according to the allocated local computing resources f _i , the local execution delay TL _i is obtained as:

At this time, the corresponding user processing energy consumption EL _i is:

where θ is the energy density required to process the task, and f _i is the locally allocated computing resource;

From this, the total delay for user i to perform this task can be obtained as:

The user energy consumption of user i to perform this task is:

When the user i unloads the task to the MEC server for execution, according to the size of the task calculation amount c _i , the proportion of the unloaded task b _i,k , and the unit price ε charged by the MEC server according to the task calculation amount, the corresponding user fee W _i needs to be paid:

Step 13): In order to improve user experience and achieve the Nash equilibrium of each user experience, it is necessary to reduce user energy consumption and user cost, and improve user accuracy. By calculating the accuracy, delay, energy consumption and cost brought by different frame resolution selection strategies, partial offloading strategies and resource allocation strategies, the user experience model is obtained:

U _i =ξ _Q Q _i -ξ _E E _i -ξ _W Wi _i

With the goal of maximizing user experience, under the constraints of delay threshold, computing resources and wireless resources, the optimization problem formed is:

s.t.

(c1)

(c2)

(c3)

(c4)

(c5)

Among them, z _i,k,n indicates whether the channel n between the user and the MEC server k is allocated to the user i, if it is allocated, it is 1, otherwise it is 0. M _k is the computing resource size of MEC server k, and F _i is the local computing resource of user i.

Step 14): Initialize the states, actions and rewards of the Markov decision process. A state is a description of the environment that will change after the agent generates an action. _{S t} =( _{s 1} , _.

s _i =[v _i ,b _i,1 ,...,b _i,K ,m _i,1 ,...,m _i,K ,z _i,1,1 ,...,z _i,1,N ,...,z _i,k,n ,…,z _i,K,N ,f _i ,rf _i ,rm ₁ ,…rm _k ]

Among them, vi is the frame resolution selected by user _i , bi _,1 ,...,bi _,K represents the proportion of the task size that user i unloads to each MEC server, _mi,1 ,..., _mi,K represents The computing power allocated by each MEC server to user i, zi _{, 1,1} ,…,zi _,1,N ,…,zi _,k,n ,…,zi _,K,N represents the base station where each MEC server is located Whether the uplink channel resource is idle, f _i represents the local computing capability allocated by user i, rf _i represents the current remaining computing capability of user i, rm ₁ ,...rm _k represents the current remaining computing capability of each MEC server;

The action of the agent represents the increase or decrease of the current resource allocation state, is the description of the agent's behavior, and is the result of the agent's decision. Let At = (a ₁ ,..., a _i ,..., a _I ) be the action space of the system at time _t , where a _i represents the action of user i, expressed as:

a _i = [cv _i , cb _i,1 ,...,cb _i,K ,cm _i,1 ,...,cm _i,K ,cz _i,1,1 ,...,cz _i,1,N ,...,cz _i,k,n ,…,cz _i,K,N ,cf _i ]

Among them, cv _i is the change amount of the frame resolution selected by user i, cb _i,1 ,…,cb _i,K indicates the change amount of the task ratio that user i unloads to each MEC server, cm _i,1 ,… ,cm _i,K represents the variation of computing power allocated to user i by each MEC server, cz _i,1,1 ,…,cz _i,1,N ,…,cz _i,k,n ,…,cz _{i, K, N} represents whether the uplink channel resources of the base station where each MEC server is located is allocated to the user, and cf _i represents the variation of the local computing capability allocated by user i;

When the state after the user generates an action does not satisfy the constraints (c1)-(c4), the reward function is set as:

Among them, Λ _(·) indicates that when the condition of (·) is not satisfied, the result is -1, otherwise it is 0; l ₁ , χ ₁ , χ ₂ , χ ₃ , χ ₄ are experimental parameters. When constraints (c1)-(c4) are satisfied but not (c5), the reward function is set as:

r _i =l ₂ +exp(thr _i -T _i )

Among them, exp(·) represents an exponential function; l ₂ is an experimental parameter. When all constraints are met, the reward function is:

r _i =l ₃ +exp(U _i )

Among them, _l3 is the experimental parameter.

Step 15): obtain new states, actions and rewards according to the proposed reinforcement learning-based method, and store them in the experience replay pool;

Step 16): judge whether the data in the experience playback pool reaches the threshold;

Step 17): start training the network until convergence;

The network training process is described in detail below.

表示所有智能体的动作策略集合。网络训练时，从经验回放池中随机抽取mini-batch大小的数据进行训练。经验回放池由一系列的状态、动作以及奖励对组成，即(S,A,R,S′)。Actor网络的Eval-net根据智能体当前的状态S产生相应的动作，接着Critic网络的Eval-net对当前产生的动作进行评判，得到对应的评判值Q。Actor网络的Target-net根据从经验回放池中得到的状态S′产生相应的动作，接着Critic网络的Target-net对动作进行评判，得到对应的评判值y。根据两个评判值得到Loss函数对Critic网络进行更新，用公式表示如下：The network is mainly composed of two parts, namely Actor network and Critic network. Each agent has an Actor network and Critic network. Actor network and Critic network each contain Eval-net and Target-net, Eval-net network parameters Periodically assigned to Target-net. Actor network parameters are θ={θ ₁ ,...,θ _I }, let

Represents the set of action policies for all agents. During network training, a mini-batch size of data is randomly selected from the experience replay pool for training. The experience replay pool consists of a series of state, action and reward pairs, namely (S, A, R, S′). The Eval-net of the Actor network generates corresponding actions according to the current state S of the agent, and then the Eval-net of the Critic network evaluates the currently generated actions to obtain the corresponding evaluation value Q. The Target-net of the Actor network generates corresponding actions according to the state S' obtained from the experience playback pool, and then the Target-net of the Critic network judges the actions and obtains the corresponding judgment value y. According to the two evaluation values, the Loss function is obtained to update the Critic network, which is expressed as follows:

Among them, γ is the discount factor, and the Actor network is updated by minimizing the policy gradient of the agent, which can be expressed as:

where X is the size of the mini-batch and j is the index of the sample.

Step 18): After the network is trained, you only need to input the state of the user and the MEC server into the Actor network each time to obtain the corresponding strategy;

Step 19): The algorithm execution is completed.

The simulation settings and experimental results analysis are given below.

The simulation platform is implemented in the environment of Python3.7 and Pytorch1.5.0. The detailed simulation parameter settings are shown in Table 1 and Table 2.

Table 1. Simulation parameter configuration

parameter set value User computing power [1.5,2.0]G Cycles/s User transmit power 1.0W MEC computing power 5,7,9,11G Cycles/s Upstream channel bandwidth 100MHz channel model Typical Urban MEC fee unit price 0.009,0.010,0.011,0.012per G Cycles Video frame resolution [200*200,600*600] delay threshold [20,25]ms Calculate density [10,50] Cycles/bit Energy Density 1.0×10^-26J/Cycle

Table 2. Neural network and training parameters

Experimental Results and Analysis

In the experimental results, the comparison algorithms are AL, AO, Random, and the corresponding algorithm of the present invention is MADDPG.

The first set of experiments: the effect of changes in MEC computing power on the experimental results. Figures 2 to 5 show a comparison of the user experience of each user using different algorithms under different computing capabilities of the MEC server. As can be seen from the figure, compared with the AL, AO and Random algorithms, the MADDPG algorithm proposed by the present invention can always provide a higher and stable user experience for each user, because the MADDPG algorithm proposed by the present invention improves the user experience As the objective function, the reward function is designed according to the objective function and constraints by the method of reinforcement learning, and the network is guided to learn by maximizing the reward function, so as to make a near-optimal frame resolution selection strategy and partial unloading for each user. and resource allocation strategies to maximize user experience. Figures 6 to 9 respectively show the changes of the average user experience, average energy consumption, average cost and average accuracy rate of all users under different algorithms as the computing power of the MEC server increases. It can be seen that the MADDPG algorithm proposed by the present invention can achieve a better average user experience by compromising energy consumption, cost and accuracy. Except for the AL algorithm, it can be seen that the average accuracy of other algorithms gradually increases and tends to be stable. This is because as the computing power of the MEC server increases, the computing resources that can be allocated to users increase. Accuracy for high frame resolution tasks. Higher frame resolutions generally allow better accuracy, but at the cost of longer task execution latency, energy consumption, and higher cost of offloading, as shown in the results in the figure. Therefore, when the computing power of MEC continues to increase, users will not choose tasks with higher frame resolution, but comprehensively consider the delay threshold, energy consumption, cost and accuracy rate, so that the user experience can be maintained at a better level.

The second set of experiments: the influence of the change of MEC unit price on the experimental results. FIG. 10 to FIG. 13 show the comparison of the user experience of each user using different algorithms in the case of different charging unit prices of the MEC server. As can be seen from the figure, compared with the AL, AO and Random algorithms, the MADDPG algorithm proposed by the present invention can always provide a higher and more stable user experience for each user, because the algorithm can and user status, etc., to more accurately select frame resolution, allocate computing resources and wireless resources for users, so as to achieve a better user experience. Figures 14 to 17 respectively show the changes of the average user experience, average energy consumption, average cost and average accuracy rate of all users under different algorithms as the unit price of the MEC server increases. It can be seen that when the unit price of the MEC server increases, the average user experience gradually declines. This is because when the unit price is higher, users will not choose large tasks with high frame resolution and offload more tasks to the MEC server for processing. The task of choosing the appropriate frame resolution to trade off energy, cost, and accuracy to get the best possible user experience. It can be seen that when the unit price charged by the MEC server increases, the MADDPG algorithm proposed by the present invention is relatively stable in terms of energy consumption, cost and accuracy, and can obtain better user experience than other algorithms. For better user experience, the AO algorithm will choose low-resolution tasks to reduce the average accuracy and average cost more.

The third group of experiments: the effect of changes in the number of users on the experimental results. This experiment mainly compares the average user experience, average energy consumption, average cost and average accuracy rate of all users under different algorithms when the number of users increases. It can be seen from Figure 18 to Figure 21 that with the increase of the number of users, the average user experience, average cost and average accuracy rate all gradually decrease, on the contrary, the average energy consumption increases gradually. Because the increase in the number of users will generate more computing tasks, which will compete for the limited computing resources and wireless resources of the MEC server, which means that a higher proportion of computing tasks will be processed locally by users, resulting in increased local processing energy consumption. Therefore, due to the limitation of MEC computing resources and wireless resources, users will choose tasks with lower frame resolutions, and user costs will be reduced accordingly, so as to obtain a better user experience as much as possible.

Although the present invention has been described in detail above with general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention fall within the scope of the claimed protection of the present invention.

Claims (1)

1. an adaptive AR task unloading and resource allocation method based on reinforcement learning, is characterized in that, comprises the following steps: 1) The user terminal selects AR tasks with different video frame resolutions, and simultaneously obtains the corresponding accuracy rate of the AR tasks with the video frame resolution; it is assumed that the AR video selected by user i is preprocessed to a video frame resolution of v _i ×v The video frame of _i pixel, that is, the video frame resolution of user i is

represents the set of video frame resolutions, where

is a set of users; different frame resolutions correspond to different task sizes and delay thresholds, and the computing task of user i is expressed as τ _i ={d _i , _{ci ,thr i }} , including the size of the task, the amount of computation and Time delay threshold, d _i =δv _i ² is the task size, where δ is defined as the number of bits required to represent one pixel information, and the relationship between task computation and task size is c _i =η _i d _i , where η _i In order to calculate the density, that is, the amount of calculation required for a unit task, according to the relationship between the AR video frame resolution and the accuracy rate, the accuracy rate Qi obtained by user _i is:

2) Partially offload AR tasks with different video frame resolutions selected by the client. First, the AR tasks are divided into different proportions, and then the tasks are partially offloaded to different MEC servers under the allocation of different computing resources and wireless resources. Time delay, user energy consumption and user cost brought by local execution; since partial offloading of tasks can be executed in parallel, for the task of user i, the time delay T _i required to complete the execution is:

Among them, TL _i is the local execution delay of some tasks of user i, and TM _i,k is the execution delay of unloading some tasks of user i to MEC server k; The local execution delay TL _i of some tasks of user i is:

Among them, ci represents the task calculation amount of user _i , fi represents the local computing resources allocated by user _i to process the current task, and b _i,k represents the proportion of tasks that user i unloads to MEC server k; The time delay TM _{i, k} for offloading part of the tasks of user i to the MEC server k is:

Among them, d _i represents the task size of user _{i, r i,k} represents the uplink transmission rate between user i and MEC server k, and m _i,k represents the computing resource size of MEC server k allocated by user i; The user energy consumption E _i of user i performing the current task is:

Among them, EL _i is the local processing energy consumption, EM _i,k is the transmission energy consumption when the user unloads some tasks to the MEC server k; The local processing energy consumption EL _i is:

Among them, θ represents the energy density required to process the task, and f _i is the locally allocated computing resource; When the user unloads some tasks to the MEC server k, the transmission energy consumption EM _i,k is:

Among them, P _i represents the user transmission power; When the user i unloads the task to the MEC server for execution, according to the size of the task calculation c _i , the proportion of the unloaded task b _i,k , and the unit price ε charged by the MEC server, the corresponding fee W _i needs to be paid:

3) Under the constraint of task delay threshold, a user experience model is established by comprehensively considering the accuracy rate, user energy consumption and user cost; by jointly optimizing the AR video frame resolution selection strategy, task part offloading strategy, computing resource and wireless resource allocation strategy , forming an optimization problem with the goal of improving user experience; in order to improve the comprehensive experience of each user, achieve the Nash equilibrium of each user experience, reduce user energy consumption and user costs, and improve the accuracy of user AR tasks, therefore, combining these three Factors that form a user experience model: U _i =ξ _Q Q _i -ξ _E E _i -ξ _W Wi _i Under the constraints of delay threshold, computing resources and wireless resources, jointly optimize the user's AR video frame resolution selection strategy

Task Section Uninstall Policy

Radio resource allocation strategy

Local and MEC computing resource allocation strategies

Form an optimization problem with the goal of maximizing the user experience:

s.t.

Among them, zi _,k,n indicate whether the channel n between the user and the MEC server k is allocated to the user i, if it is allocated, it is 1, otherwise it is 0, M _k is the computing resource size of the MEC server k, F _i user i of local computing resources; 4) Transform the optimization problem into a Markov decision process, initialize its state space, action space and reward settings; design a reinforcement learning network according to the Markov decision process; the state space includes the size of all user computing resources, the size of MEC computing resources, and the base station. Idle wireless resources, initial computing resources and wireless resource allocation scheme, AR video frame resolution selection scheme, task part offload scheme; action space is the allocation scheme for initial computing resources and wireless resources, AR video frame resolution selection scheme, The amount of change in the unloading scheme of the task part; the reward is to set three-tier rewards according to the constraints (c1)-(c5); 5) Train the reinforcement learning network through the MADDPG algorithm until the network converges; first obtain new states, actions and rewards according to the reinforcement learning network, and store them in the experience playback pool, and then start the training design reinforcement when the data in the experience playback pool reaches the threshold Learn the network and continuously update the data and network parameters in the experience playback pool until the network training converges; 6) After the network is trained, the state of the client and MEC server is input into the network, and the AR video frame resolution selection strategy, task part offloading strategy, computing resource and wireless resource allocation strategy are obtained.

Images