CN111444009A - A resource allocation method and device based on deep reinforcement learning - Google Patents

Abstract

Embodiments of the present invention provide a method and device for resource allocation based on deep reinforcement learning. The method includes: determining a variety of services of resources to be allocated included in a user's application request, and the allocation priority of each service; determining the current edge The state parameters of the micro cloud system, the state parameters include the resource balance evaluation parameter, the response delay evaluation parameter, and the remaining amount of resources of each computing node in each micro cloud; input the state parameters into the pre-trained resource balance optimization model, and get the first The first target computing node of a service; the resource balancing optimization model is completed based on deep reinforcement learning training, and the first service is deployed on the first target computing node; the state parameter is updated, and the parameter input step is returned until the application request contains Resource allocation is completed for each service to which resources are to be allocated. Compared with the traditional resource allocation method, it can not only meet the communication delay requirement, but also achieve a higher degree of resource utilization balance.

Description

A resource allocation method and device based on deep reinforcement learning

technical field

The present invention relates to the technical field of wireless communication, and in particular, to a method and device for resource allocation based on deep reinforcement learning.

Background technique

In recent years, with the continuous development of informatization and networking, information systems have played an increasingly important role in military, disaster relief and other fields. In this highly dynamic environment, mission plans and equipment composition may change frequently, and network connectivity may fluctuate. Service resources based on stand-alone devices are very limited and cannot cope with complex computing tasks. Cloud computing technology is an effective means to deal with this scenario. In cloud computing technology, resource configuration can be customized according to task requirements, thereby providing convenient and flexible management services for large-scale applications. However, traditional cloud platforms are usually deployed in areas far away from users, and the communication delay is high. And in an environment with an unstable network, it is difficult to provide continuous and reliable services.

In order to solve the above problems, the edge micro-cloud platform was created. The edge micro-cloud platform is an emerging cloud computing model. It consists of multiple distributed edge micro-clouds. Each edge micro-cloud contains several small servers. The scale of the edge micro-cloud platform can be adjusted according to the task requirements. Edge micro-clouds are mostly deployed on mobile vehicles and move according to mission requirements to provide higher-quality cloud services. With the development of microservice technology, an application is usually composed of multiple composite services that communicate with each other, and each composite service has different requirements for resources in different dimensions. Since the computing power of a single edge micro-cloud is limited and cannot meet all service requirements, the combined services can be distributed and deployed in different edge micro-clouds, and different edge micro-clouds cooperate with each other to provide computing power.

However, in the existing edge micro-cloud technology, when allocating resources for services, resources are often fragmented, that is, resource allocation is not balanced, resulting in a waste of resources in a certain dimension. In addition, resource allocation not only needs to consider the resource requirements of each service, but also needs to consider the communication requirements between services, which further increases the complexity of resource allocation, and the existing related technologies do not consider the communication requirements between services.

It can be seen that there is an urgent need for a resource allocation method that can meet the communication requirements between services and has a high degree of balanced resource utilization.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a resource allocation method and device based on deep reinforcement learning, so as to improve the balance of resource utilization. The specific technical solutions are as follows:

In order to achieve the above object, an embodiment of the present invention provides a resource allocation method based on deep reinforcement learning, which is applied to a control platform of an edge micro-cloud system. The edge micro-cloud system further includes a plurality of micro-clouds, and each micro-cloud includes A plurality of computing nodes, the method includes:

Determine the various services to be allocated resources contained in the user's application request, and the allocation priority of each service;

Determine the state parameters of the current edge micro cloud system, the state parameters include resource balance evaluation parameters, response delay evaluation parameters, and the remaining resources of each computing node in each micro cloud;

Input the state parameter into the pre-trained resource balancing optimization model to obtain the first target computing node of the first service; the first service is the service with the highest current allocation priority; the resource balancing optimization model is based on deep reinforcement The learning and training are completed, wherein the training set of deep reinforcement learning includes: the sample state parameters of the edge micro-cloud system;

deploying the first service on the first target computing node;

The state parameter is updated, and the step of inputting the state parameter into the pre-trained resource balancing optimization model is returned, until the resource allocation is completed for each service of the to-be-allocated resources included in the application request.

Optionally, the resource balance degree evaluation parameter is calculated based on the following formula:

表示第i个微云中第j个计算节点的资源利用方差，D表示资源的种类数，

表示第i个微云中第j个计算节点中第d类资源的资源利用率，

表示第i个微云中第j个计算节点中所有种类资源的资源利用率的平均值，X表示资源分配策略，

表示第i个微云中第j个计算节点的资源均衡率，RUBD_i表示第i个微云的资源利用均衡度，L_i表示第i个微云中计算节点的总数，RUBD_Total表示所述边缘微云系统的资源均衡度评估参数，K表示所述边缘微云系统中微云的总数；in,

represents the resource utilization variance of the jth computing node in the ith microcloud, D represents the number of resource types,

Represents the resource utilization rate of the d-th resource in the j-th computing node in the i-th microcloud,

represents the average resource utilization of all kinds of resources in the jth computing node in the ith microcloud, X represents the resource allocation strategy,

Represents the resource balance rate of the jth computing node in the ith microcloud, RUBD _i represents the resource utilization balance of the _ith microcloud, Li represents the total number of computing nodes in the ith microcloud, and RUBD _Total represents the The resource balance evaluation parameter of the edge micro-cloud system, K represents the total number of micro-clouds in the edge micro-cloud system;

The response delay evaluation parameter is calculated based on the following formula:

t _Total = T _Comp (X)+T _TR (X)

t _Total is the response delay evaluation parameter, T _Comp (X) is the computation delay, and T _TR (X) is the transmission delay.

Optionally, the resource balancing optimization model is trained according to the following steps:

Obtain a preset neural network model and the training set;

Inputting the sample state parameters into the neural network model to obtain a service placement action; the service placement action is represented as a sample service to determine the placed target computing node;

Based on the service placement action, the sample state parameter is updated to obtain the updated sample state parameter;

Based on the resource balance evaluation parameters and response delay evaluation parameters included in the sample state parameters, and the resource balance evaluation parameters and response delay evaluation parameters included in the updated sample state parameters, calculate the value of this service placement action. reward value;

Substitute the sample state parameter, the updated sample state parameter, the current service placement action, and the reward value of the current service placement action into a preset loss function to calculate the loss value of this service placement action ;

Determine whether the neural network model converges according to the loss value;

If not, then adjust the parameter values in the neural network model, and return to the step of inputting the updated sample state parameters into the neural network model to obtain the service placement action;

If so, the current neural network model is determined as the resource balance optimization model.

Optionally, the loss function is:

表示t时刻之后n组历史迭代数据的优先级权重，

表示t时刻后n次迭代的奖励值之和，

表示针对t时刻后n次迭代奖励值的衰减因子，Q_target表示目标网络，Q_eva表示估计网络，s_t表示时刻t的样本状态参数，a_t表示时刻t的服务放置动作，s_t+n表示迭代n次后的样本状态参数，a′表示使估计网络输出最大值的服务放置动作，k表示迭代序号，

表示针对t时刻后第k次迭代奖励值的衰减因子，r_t+k+1表示t时刻后第k次迭代的奖励值。Among them, L represents the loss function, E[] represents the mathematical expectation, n represents the number of groups of historical iteration data referenced in each iteration, t represents the time,

Represents the priority weight of n groups of historical iteration data after time t,

represents the sum of the reward values of n iterations after time t,

represents the decay factor of the reward value for n iterations after time t, Q _target represents the target network, Q _eva represents the estimation network, s _t represents the sample state parameter at time t, at _t represents the service placement action at time t, s _t+n Represents the sample state parameter after n iterations, a' represents the service placement action that maximizes the estimated network output, k represents the iteration number,

Represents the decay factor for the reward value of the k-th iteration after time t, and r _t+k+1 represents the reward value of the k-th iteration after time t.

In order to achieve the above purpose, the embodiment of the present invention also provides a resource allocation device based on deep reinforcement learning, which is applied to the control platform of an edge micro-cloud system. The edge micro-cloud system further includes a plurality of micro-clouds, each micro-cloud Including a plurality of computing nodes, the apparatus includes:

a first determining module, used for determining a variety of services of resources to be allocated included in the application request of the user, and the allocation priority of each service;

The second determination module is used to determine the state parameters of the current edge micro-cloud system, where the state parameters include the resource balance evaluation parameter, the response delay evaluation parameter, and the remaining amount of resources of each computing node in each micro-cloud;

The input module is used to input the state parameter into the pre-trained resource balancing optimization model to obtain the first target computing node of the first service; the first service is the service with the highest allocation priority currently; the resource balancing optimization The model is completed based on deep reinforcement learning training, wherein the training set of deep reinforcement learning includes: the sample state parameters of the edge micro-cloud system;

a deployment module, configured to deploy the first service on the first target computing node;

An update module, configured to update the state parameter and trigger the input module until the resource allocation is completed for each service of the to-be-allocated resources contained in the application request.

Optionally, the device further includes: a calculation module, configured to calculate the resource balance degree evaluation parameter based on the following formula:

表示第i个微云中第j个计算节点的资源利用方差，D表示资源的种类数，

表示第i个微云中第j个计算节点中第d类资源的资源利用率，

表示第i个微云中第j个计算节点中所有种类资源的资源利用率的平均值，X表示资源分配策略，

表示第i个微云中第j个计算节点的资源均衡率，RUBD_i表示第i个微云的资源利用均衡度，L_i表示第i个微云中计算节点的总数，RUBD_Total表示所述边缘微云系统的资源均衡度评估参数，K表示所述边缘微云系统中微云的总数；in,

represents the resource utilization variance of the jth computing node in the ith microcloud, D represents the number of resource types,

Represents the resource utilization rate of the d-th resource in the j-th computing node in the i-th microcloud,

represents the average resource utilization of all kinds of resources in the jth computing node in the ith microcloud, X represents the resource allocation strategy,

Represents the resource balance rate of the jth computing node in the ith microcloud, RUBD _i represents the resource utilization balance of the _ith microcloud, Li represents the total number of computing nodes in the ith microcloud, and RUBD _Total represents the The resource balance evaluation parameter of the edge micro-cloud system, K represents the total number of micro-clouds in the edge micro-cloud system;

The response delay evaluation parameter is calculated based on the following formula:

t _Total = T _Comp (X)+T _TR (X)

t _Total is the response delay evaluation parameter, T _Comp (X) is the computation delay, and T _TR (X) is the transmission delay.

Optionally, the device further includes: a training module, the training module is configured to train the resource balancing optimization model according to the following steps:

Obtain a preset neural network model and the training set;

Inputting the sample state parameters into the neural network model to obtain a service placement action; the service placement action is represented as a sample service to determine the placed target computing node;

Based on the service placement action, the sample state parameter is updated to obtain the updated sample state parameter;

Based on the resource balance evaluation parameters and response delay evaluation parameters included in the sample state parameters, and the resource balance evaluation parameters and response delay evaluation parameters included in the updated sample state parameters, calculate the value of this service placement action. reward value;

Substitute the sample state parameter, the updated sample state parameter, the current service placement action, and the reward value of the current service placement action into a preset loss function to calculate the loss value of this service placement action ;

Determine whether the neural network model converges according to the loss value;

If not, then adjust the parameter values in the neural network model, and return to the step of inputting the updated sample state parameters into the neural network model to obtain the service placement action;

If so, the current neural network model is determined as the resource balance optimization model.

Optionally, the loss function is:

表示t时刻之后n组历史迭代数据的优先级权重，

表示t时刻后n次迭代的奖励值之和，

表示针对t时刻后n次迭代奖励值的衰减因子，Q_target表示目标网络，Q_eva表示估计网络，s_t表示时刻t的样本状态参数，a_t表示时刻t的服务放置动作，s_t+n表示迭代n次后的样本状态参数，a′表示使估计网络输出最大值的服务放置动作，k表示迭代序号，

表示针对t时刻后第k次迭代奖励值的衰减因子，r_t+k+1表示t时刻后第k次迭代的奖励值。Among them, L represents the loss function, E[] represents the mathematical expectation, n represents the number of groups of historical iteration data referenced in each iteration, t represents the time,

Represents the priority weight of n groups of historical iteration data after time t,

represents the sum of the reward values of n iterations after time t,

represents the decay factor of the reward value for n iterations after time t, Q _target represents the target network, Q _eva represents the estimation network, s _t represents the sample state parameter at time t, at _t represents the service placement action at time t, s _t+n Represents the sample state parameter after n iterations, a' represents the service placement action that maximizes the estimated network output, k represents the iteration number,

Represents the decay factor for the reward value of the k-th iteration after time t, and r _t+k+1 represents the reward value of the k-th iteration after time t.

To achieve the above object, an embodiment of the present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

memory for storing computer programs;

The processor is configured to implement any of the above method steps when executing the program stored in the memory.

To achieve the above object, an embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any one of the above method steps is implemented.

It can be seen that the resource allocation method and device based on deep reinforcement learning provided by the embodiments of the present invention can determine a variety of services to be allocated resources included in the user's application request, and the allocation priority of each service; determine the current edge micro cloud The state parameters of the system, the state parameters include the resource balance evaluation parameter, the response delay evaluation parameter, and the remaining amount of resources of each computing node in each micro cloud; enter the state parameters into the pre-trained resource balance optimization model to get the first service The first target computing node of the first target computing node, deploy the first service on the first target computing node; update the state parameters, and return to the steps of inputting the state parameters into the pre-trained resource balancing optimization model, until the application request contains each type of to-be-allocated The services of the resource complete the resource allocation. Therefore, the response delay and resource allocation balance are comprehensively considered, and the deep reinforcement learning method is used to train the network model. Compared with the traditional resource allocation method, it can not only meet the communication delay requirements, but also achieve a higher resource utilization balance.

Of course, it is not necessary for any product or method to implement the present invention to simultaneously achieve all of the advantages described above.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a schematic flowchart of a resource allocation method based on deep reinforcement learning provided by an embodiment of the present invention;

2 is a schematic flowchart of a training resource balancing optimization model provided by an embodiment of the present invention;

3 is a schematic structural diagram of a resource allocation device based on deep reinforcement learning provided by an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In order to solve the technical problem of low resource utilization balance in the existing service resource allocation method in the edge micro-cloud field, the embodiments of the present invention provide a resource allocation method, device, electronic device, and computer-readable based on deep reinforcement learning storage medium.

For ease of understanding, an application scenario of the embodiment of the present invention is first described below.

The resource allocation method based on deep reinforcement learning provided by the embodiments of the present invention can be applied to highly dynamic scenarios such as the military field and disaster relief field, and these scenarios usually use an edge micro-cloud system to provide services for applications. The edge micro-cloud system can include a control platform, multiple micro-clouds, and each micro-cloud contains multiple computing nodes, where the computing nodes can represent electronic devices including processors, communication interfaces, memory, and communication buses, such as personal computers, etc., Typically it can be placed on a moving vehicle. A micro cloud represents a collection of multiple computing nodes, and a micro cloud can usually include multiple computing nodes on a mobile vehicle. The resource allocation method based on deep reinforcement learning provided by the embodiment of the present invention can be applied to the control platform, that is, the control platform decides which computing node to deploy each service included in the application program.

Specifically, referring to FIG. 1 , the resource allocation method based on deep reinforcement learning provided by the embodiment of the present invention may include the following steps:

S101: Determine a variety of services of resources to be allocated included in the application request of the user, and the allocation priority of each service.

In this embodiment of the present invention, the user's application request may include various services, such as positioning services, image processing services, etc. These services need to be deployed on computing nodes in the micro cloud, so that the computing nodes can provide corresponding services for these services. resource. In this embodiment of the present invention, the process of allocating resources for a service may also be understood as a process of allocating computing nodes for the service.

In addition, since there is a certain relationship between services, the service to which the resource is to be allocated has an allocation priority. For example, if service a needs to depend on service b, the allocation priority of service b is higher than that of service a, that is, service a can be allocated only when computing nodes are allocated to service b first.

S102: Determine state parameters of the current edge micro-cloud system, where the state parameters include a resource balance evaluation parameter, a response delay evaluation parameter, and the remaining amount of resources of each computing node in each micro-cloud.

In the embodiment of the present invention, the resource balance degree evaluation parameter represents the balance degree of resource allocation of computing nodes in the micro cloud.

In an embodiment of the present invention, the resource balance degree evaluation parameter can be calculated based on the following formula:

First define the resource utilization variance as follows:

表示第i个微云中第j个计算节点的资源利用方差，D表示资源的种类数，

表示第i个微云中第j个计算节点中第d类资源的资源利用率，

表示第i个微云中第j个计算节点中所有种类资源的资源利用率的平均值，X表示资源分配策略。in,

represents the resource utilization variance of the jth computing node in the ith microcloud, D represents the number of resource types,

Represents the resource utilization rate of the d-th resource in the j-th computing node in the i-th microcloud,

Represents the average resource utilization of all kinds of resources in the j-th computing node in the i-th microcloud, and X represents the resource allocation strategy.

Since the resource utilization variance cannot reflect the unbalanced utilization of resources in which the utilization rate of a specific resource is too high and the utilization rate of other types of resources is low, the resource balancing rate can be defined as follows:

表示第i个微云中第j个计算节点的资源均衡率。in,

Represents the resource balance rate of the jth computing node in the ith cloud.

Further, by normalizing the resource balance rate, the resource utilization balance degree can be defined according to the following formula:

表示第i个微云中第j个计算节点的资源利用均衡度，其值在0到1之间，资源利用均衡度值越大，则资源利用越均衡。in,

Represents the resource utilization balance degree of the jth computing node in the ith microcloud, and its value is between 0 and 1. The larger the resource utilization balance degree value is, the more balanced the resource utilization is.

Then, the resource utilization balance of the entire edge micro-cloud system can be determined:

Among them, RUBD _i represents the resource utilization balance of the _ith microcloud, Li represents the total number of computing nodes in the ith microcloud, and RUBD _Total represents the resource utilization balance of the edge microcloud system, that is, the resource balance evaluation parameter , K represents the total number of microclouds in the edge microcloud system.

In the embodiment of the present invention, the response delay evaluation parameter represents the sum of the computing delay and the communication delay of the edge micro-cloud system, namely:

t _Total = T _Comp (X)+T _TR (X)

t _Total is the response delay evaluation parameter, T _Comp (X) is the computation delay, and T _TR (X) is the transmission delay.

表示第i个微云中第j个计算节点的资源剩余量。In this embodiment of the present invention, the

Represents the resource remaining of the jth computing node in the ith microcloud.

Furthermore, in this embodiment of the present invention, the set s can be used to represent the state parameters of the current edge micro-cloud system, then:

That is to say, the state parameters of the edge micro-cloud system can be composed of the resource balance evaluation parameter, the response delay evaluation parameter, and the remaining amount of resources of each computing node in each micro-cloud.

S103: Input the state parameters into the pre-trained resource balancing optimization model to obtain the first target computing node of the first service; the first service is the service with the highest allocation priority currently, and the resource balancing optimization model is completed based on deep reinforcement learning training , where the training set of deep reinforcement learning includes: the sample state parameters of the edge micro-cloud system.

In the embodiment of the present invention, after obtaining the state parameters of the current edge micro-cloud system, the control platform can input it into the resource balance optimization model. Since the resource balance optimization model is trained based on the training set based on deep reinforcement learning, it can output the most Resource allocation strategy suitable for current state parameters.

则

表示该定位服务被分配至第i个微云中的第j个节点。Specifically, the resource balance optimization model outputs the first target computing node of the first service, where the first service is the service with the highest currently assigned priority. For example, if the service with the highest allocation priority is the location service, the resource balance optimization model outputs the target computing node of the location service, which can be recorded as

but

Indicates that the location service is allocated to the jth node in the ith microcloud.

The training process of the resource balancing optimization model can be referred to below, which will not be repeated here.

S104: Deploy the first service on the first target computing node.

In this embodiment of the present invention, after the first target computing node of the first service is determined, the first service can be deployed on the first target computing node, and the first service can be run based on the resources in the first target computing node .

S105: Update the state parameters, and return to the step of inputting the state parameters into the pre-trained resource balancing optimization model, until the resource allocation is completed for each service of the to-be-allocated resources included in the application request.

In the embodiment of the present invention, after allocating resources for the first service, the state parameters of the edge micro-cloud system will change. Therefore, the control platform re-counts the current state parameters of the edge micro-cloud system and continues to allocate resources for the next service.

The control platform inputs the current state parameters into the resource balancing optimization model, and then obtains the target computing node that currently allocates the service with the highest priority.

In this embodiment of the present invention, the above steps may be performed cyclically until resource allocation is completed for each service of to-be-allocated resources included in the application request.

It can be seen that the resource allocation method based on deep reinforcement learning provided by the embodiment of the present invention can determine a variety of services to be allocated resources included in the user's application request, and the allocation priority of each service; determine the current edge micro cloud system The state parameters include the resource balance evaluation parameters, the response delay evaluation parameters, and the remaining resources of each computing node in each microcloud; input the state parameters into the pre-trained resource balance optimization model to obtain the first service A target computing node, deploying the first service on the first target computing node; updating the state parameters, and returning to the step of inputting the state parameters into the pre-trained resource balancing optimization model, until each resource to be allocated contained in the application request is All services complete resource allocation. Therefore, the response delay and resource allocation balance are comprehensively considered, and the deep reinforcement learning method is used to train the network model. Compared with the traditional resource allocation method, it can not only meet the communication delay requirements, but also achieve a higher resource utilization balance.

In this embodiment of the present invention, a resource balancing optimization model can be trained based on deep reinforcement learning. Deep reinforcement learning is a combination of reinforcement learning and deep learning.

For ease of understanding, a brief introduction to reinforcement learning is given below.

Reinforcement learning is a type of machine learning. Its basic idea is to generate actions according to the scene state, and then obtain learning information and update the model parameters by receiving the reward of the environment for the action, and finally realize the optimal action in a specific scene state.

In this embodiment of the present invention, the resource allocation process can be modeled as a reinforcement learning model, where the scene state is a state parameter of the edge micro-cloud system, and the action is a resource allocation strategy for a certain service. Therefore, the trained resource balance optimization model can output the optimal resource allocation strategy for a certain service according to the state parameters of the edge micro-cloud system.

In an embodiment of the present invention, referring to FIG. 2 , the following steps can be used to train the resource balancing optimization model:

S201: Obtain a preset neural network model and a training set.

Those skilled in the art can understand that compared with traditional supervised learning, reinforcement learning does not have labels as samples during training, and in the training process of reinforcement learning, only the initial input state is required as a training set.

In the embodiment of the present invention, the training set may be the sample state parameters of the edge micro-cloud system.

S202: Input the sample state parameters into the neural network model to obtain a service placement action; the service placement action represents determining a placed target computing node for the sample service.

表示资源均衡度评估参数，响应延迟评估参数，以及每个微云中各个计算节点的资源剩余量的集合。服务放置动作可以用a表示，

表示服务被分配至第i个微云中的第j个节点。In the embodiment of the present invention, the input of the neural network model is the sample state parameter, the output is the service placement action, and the service placement action is represented as the target computing node where the sample service is determined to be placed. The sample state parameter can be represented by s, same as above,

Represents the resource balance evaluation parameter, the response delay evaluation parameter, and the set of resource remaining amount of each computing node in each microcloud. A service placement action can be represented by a,

Indicates that the service is assigned to the jth node in the ith microcloud.

S203: Based on the service placement action, update the sample state parameter to obtain the updated sample state parameter.

After the service placement action is generated, the sample state parameters will be updated, and the updated sample state parameters will be obtained as the input for the next round of training.

S204: Calculate the reward value of the current service placement action based on the resource balance degree evaluation parameter and the response delay evaluation parameter included in the sample state parameter, and the resource balance degree evaluation parameter and response delay evaluation parameter included in the updated sample state parameter .

In this embodiment of the present invention, after each round of iterations, the reward value of the service placement action in the current iteration can be calculated. It is easy to understand that if the resource balance evaluation parameter shows that the resource allocation is more balanced after the placement action in this round, the response delay is evaluated. The parameter shows that the lower the response delay, the higher the reward value of the placement action in this round.

Specifically, in an embodiment of the present invention, the reward value of the service placement action can be calculated based on the following formula:

表示第n-1轮迭代后的资源均衡度评估参数，

表示第n轮迭代后的资源均衡度评估参数，

表示第n轮迭代后的服务延迟，L_ave表示平均服务延迟约束。Among them, rn represents the reward value of the _nth iteration,

Represents the resource balance evaluation parameter after the n-1 round of iteration,

Represents the resource balance evaluation parameter after the nth round of iteration,

represents the service delay after the nth round of iteration, and L _ave represents the average service delay constraint.

The above formula is only one way to calculate the reward value, and the embodiment of the present invention is not limited to using the above formula to calculate the reward value.

S205: Substitute the sample state parameter, the updated sample state parameter, the current service placement action, and the reward value of this service placement action into a preset loss function to calculate the loss value of this service placement action.

For ease of understanding, the nth round of iteration is taken as an example for illustration. The sample state parameter after the n-1th round of iteration is _sn-1 , the sample state parameter after the _nth round of iteration is sn , and the output of the nth round of iteration is The service placement action is an _n , and the reward value of the nth iteration is r _n , then the service placement in the _nth iteration can be calculated based on s _{n -1} , s _n , an , rn , and the preset loss function The loss value of the action.

Those skilled in the art can understand that in the field of deep reinforcement learning, a new Q value can be obtained in each round of iteration. expected return. The Q value is usually determined by the target network and the estimation network. The target network is represented by Q _target and the estimation network is represented by Q _eva . The Q value output by the target network Q _target is iteratively updated, and the Q value output by the estimation network Q _eva is Before iterative update.

In the embodiment of the present invention, the output Q value of the target network and the estimated network can be determined based on the sample state parameters before and after the iteration and the corresponding service placement actions, and a loss function can be constructed in combination with the reward value.

Specifically, in an embodiment of the present invention, the preset loss function may be:

Loss=E[(r _n+1 +γQ _target (s _n+1 ,argmax _a′ (Q _eva (s _n+1 ,a′)))-Q _eva (s _n ,a _n )) ² ]

Among them, E[] represents the mathematical function, r _n+1 represents the reward value of the n+1 round of iteration, γ represents the decay factor, Q _target represents the target network, Q _eva represents the estimation network, and s _n represents the nth round of iterations. Sample state parameters, an _n represents the service placement action output by the nth iteration, and a′ represents the service placement action that maximizes the estimated network output.

In the embodiment of the present invention, the sample state parameters before and after each round of iteration, the service placement action and the reward value are used as variables in the loss function, and then the neural network model is trained. In order to speed up the convergence speed and improve the accuracy of the network model, in the subsequent iteration process , the data from previous steps can be combined for training. For example, for the third iteration, the sample state parameters, service placement actions, and reward values of the first iteration, and the sample state parameters, service placement actions, and reward values of the second iteration can be used as training data. That is, the data of the previous iterations can be comprehensively considered in the loss function.

In addition, because in each round of iteration, the difference between the Q values output by the target network and the estimated network can reflect the data in this round of iterations as a reference for training data, that is, the larger the difference, the more worthy the data in this round of iterations. training, so you can set larger sampling weights for it. For example, if the current iteration is the 5th round, if the data from the 2nd to 4th iterations are selected for training, for the 2nd to 4th iterations, if in the 3rd iteration, the difference between the Q value output by the target network and the estimated network is If the difference is the largest, a larger sampling weight can be set for the data of the third iteration.

In the embodiment of the present invention, the loss function can be improved based on two aspects of multi-step joint training and setting sampling weights. Specifically, the improved loss function based on the above two aspects can be:

表示t时刻之后n组历史迭代数据的优先级权重，

可以根据历史迭代数据中目标网络和估计网络输出的Q值的差值来确定，即Q值的差值越大，则该轮迭代中历史迭代数据的优先级权重越大，

表示t时刻后n次迭代的奖励值之和，

表示针对t时刻后n次迭代奖励值的衰减因子，

的值可以根据实际情况进行设定，Q_target表示目标网络，Q_eva表示估计网络，s_t表示时刻t的样本状态参数，a_t表示时刻t的服务放置动作，s_t+n表示迭代n次后的样本状态参数，a′表示使估计网络输出最大值的服务放置动作，k表示迭代序号，

表示针对t时刻后第k次迭代奖励值的衰减因子，r_t+k+1表示t时刻后第k次迭代的奖励值。Among them, L represents the improved loss function, E[] represents the mathematical expectation, n represents the number of groups of historical iteration data referenced in each iteration, t represents the time,

Represents the priority weight of n groups of historical iteration data after time t,

It can be determined according to the difference between the Q value output by the target network and the estimated network in the historical iteration data, that is, the greater the difference in the Q value, the greater the priority weight of the historical iteration data in this round of iteration.

represents the sum of the reward values of n iterations after time t,

represents the decay factor of the reward value for n iterations after time t,

The value of can be set according to the actual situation, Q _target represents the target network, Q _eva represents the estimated network, s _t represents the sample state parameter at time t, at _t represents the service placement action at time t, and s _t+n represents n iterations After the sample state parameters, a' represents the service placement action that maximizes the estimated network output, k represents the iteration number,

Represents the decay factor for the reward value of the k-th iteration after time t, and r _t+k+1 represents the reward value of the k-th iteration after time t.

It can be seen that the improved loss function considers multiple sets of historical iteration data generated by previous iterations, and sets the priority weight based on the difference between the Q values in each previous iteration, which can train the neural network in a more targeted manner. Accelerates convergence and improves the accuracy of network models.

S206: Determine whether the neural network model converges according to the loss value, otherwise, perform S207, and if yes, perform S208.

When the loss value does not exceed the preset loss threshold, the neural network model can be considered to have converged. In addition, the maximum number of iterations may also be preset, and when the maximum number of iterations is reached, the neural network model may also be considered to have converged, which is not limited.

S207: Adjust the parameter values in the neural network model, and return to step S202.

When the loss value shows that the neural network model has not converged, adjust the parameter value, return to step S202, and start a new round of iterative training.

S208: Determine the current neural network model as the resource balance optimization model.

Based on the same inventive concept, according to the above-mentioned embodiment of the resource allocation method based on deep reinforcement learning, the embodiment of the present invention also provides a resource allocation method based on deep reinforcement learning, referring to FIG. 3 , which may include the following modules:

A first determining module 301, configured to determine a variety of services of resources to be allocated included in the application request of the user, and the allocation priority of each service;

The second determination module 302 is configured to determine the state parameters of the current edge micro-cloud system, where the state parameters include a resource balance evaluation parameter, a response delay evaluation parameter, and the remaining amount of resources of each computing node in each micro-cloud;

The input module 303 is used to input the state parameters into the pre-trained resource balancing optimization model to obtain the first target computing node of the first service; the first service is the service with the highest allocation priority currently; the resource balancing optimization model is based on deep reinforcement The learning and training are completed, wherein the training set of deep reinforcement learning includes: the sample state parameters of the edge micro-cloud system;

a deployment module 304, configured to deploy the first service on the first target computing node;

The updating module 305 is configured to update the state parameters and trigger the input module until the resource allocation is completed for each service of the resources to be allocated included in the application request.

In an embodiment of the present invention, on the basis of the device shown in FIG. 3 , a calculation module may be further included, and the calculation module is configured to calculate the resource balance degree evaluation parameter based on the following formula:

表示第i个微云中第j个计算节点的资源利用方差，D表示资源的种类数，

表示第i个微云中第j个计算节点中第d类资源的资源利用率，

表示第i个微云中第j个计算节点中所有种类资源的资源利用率的平均值，X表示资源分配策略，

表示第i个微云中第j个计算节点的资源均衡率，RUBD_i表示第i个微云的资源利用均衡度，L_i表示第i个微云中计算节点的总数，RUBD_Total表示边缘微云系统的资源均衡度评估参数，K表示边缘微云系统中微云的总数；in,

represents the resource utilization variance of the jth computing node in the ith microcloud, D represents the number of resource types,

Represents the resource utilization rate of the d-th resource in the j-th computing node in the i-th microcloud,

represents the average resource utilization of all kinds of resources in the jth computing node in the ith microcloud, X represents the resource allocation strategy,

Represents the resource balance rate of the jth computing node in the ith microcloud, RUBD _i represents the resource utilization balance of the _ith microcloud, Li represents the total number of computing nodes in the ith microcloud, and RUBD _Total represents the edge microcloud The resource balance evaluation parameter of the cloud system, K represents the total number of micro-clouds in the edge micro-cloud system;

The response delay evaluation parameter is calculated based on the following formula:

t _Total = T _Comp (X)+T _TR (X)

t _Total is the response delay evaluation parameter, T _Comp (X) is the computation delay, and T _TR (X) is the transmission delay.

In an embodiment of the present invention, on the basis of the device shown in FIG. 3 , a training module may also be included, and the training module is used to train the resource balancing optimization model according to the following steps:

Obtain preset neural network models and training sets;

Input the sample state parameters into the neural network model to obtain the service placement action; the service placement action represents the target computing node to be placed for the sample service;

Based on the service placement action, update the sample state parameters to obtain the updated sample state parameters;

Based on the resource balance evaluation parameters and response delay evaluation parameters included in the sample state parameters, and the resource balance evaluation parameters and response delay evaluation parameters included in the updated sample state parameters, calculate the reward value of this service placement action;

Substitute the sample state parameters, the updated sample state parameters, the current service placement action, and the reward value of this service placement action into the preset loss function to calculate the loss value of this service placement action;

Determine whether the neural network model converges according to the loss value;

If not, adjust the parameter values in the neural network model, and return to the steps of inputting the updated sample state parameters into the neural network model to obtain the service placement action;

If so, the current neural network model is determined as the resource balance optimization model.

In an embodiment of the present invention, the loss function may be:

表示t时刻之后n组历史迭代数据的优先级权重，

表示t时刻后n次迭代的奖励值之和，

表示针对t时刻后n次迭代奖励值的衰减因子，Q_target表示目标网络，Q_eva表示估计网络，s_t表示时刻t的样本状态参数，a_t表示时刻t的服务放置动作，s_t+n表示迭代n次后的样本状态参数，a′表示使估计网络输出最大值的服务放置动作，k表示迭代序号，

表示针对t时刻后第k次迭代奖励值的衰减因子，r_t+k+1表示t时刻后第k次迭代的奖励值。Among them, L represents the loss function, E[] represents the mathematical expectation, n represents the number of groups of historical iteration data referenced in each iteration, t represents the time,

Represents the priority weight of n groups of historical iteration data after time t,

represents the sum of the reward values of n iterations after time t,

represents the decay factor of the reward value for n iterations after time t, Q _target represents the target network, Q _eva represents the estimation network, s _t represents the sample state parameter at time t, at _t represents the service placement action at time t, s _t+n Represents the sample state parameter after n iterations, a' represents the service placement action that maximizes the estimated network output, k represents the iteration number,

Represents the decay factor for the reward value of the k-th iteration after time t, and r _t+k+1 represents the reward value of the k-th iteration after time t.

By applying the resource allocation device based on deep reinforcement learning provided by the embodiment of the present invention, it is possible to determine a variety of services of resources to be allocated contained in the user's application request, and the allocation priority of each service; determine the current state of the edge micro cloud system Parameters, state parameters include resource balance evaluation parameters, response delay evaluation parameters, and the remaining resources of each computing node in each micro cloud; input the state parameters into the pre-trained resource balance optimization model to get the first service of the first service. The target computing node, deploying the first service on the first target computing node; updating the state parameter, and returning to the step of inputting the state parameter into the pre-trained resource balancing optimization model, until the service of each resource to be allocated included in the application request resource allocation has been completed. Therefore, the response delay and resource allocation balance are comprehensively considered, and the deep reinforcement learning method is used to train the network model. Compared with the traditional resource allocation method, it can not only meet the communication delay requirements, but also achieve a higher resource utilization balance.

Based on the same inventive concept, according to the implementation of the above-mentioned resource allocation method based on deep reinforcement learning, an embodiment of the present invention provides an electronic device, as shown in FIG. 4 , including a processor 401 , a communication interface 402 , a memory 403 and a communication bus 404 , wherein, the processor 401, the communication interface 402, and the memory 403 complete the communication with each other through the communication bus 404,

a memory 403 for storing computer programs;

When the processor 401 is used to execute the program stored in the memory 403, the following steps are implemented:

Determine the various services to be allocated resources contained in the user's application request, and the allocation priority of each service;

Determine the status parameters of the current edge micro-cloud system, the status parameters include resource balance evaluation parameters, response delay evaluation parameters, and the remaining resources of each computing node in each micro-cloud;

Input the state parameters into the pre-trained resource balancing optimization model to obtain the first target computing node of the first service; the first service is the service with the highest priority currently assigned; the resource balancing optimization model is based on deep reinforcement learning training, where , the training set of deep reinforcement learning includes: the sample state parameters of the edge micro-cloud system;

deploying the first service on the first target computing node;

Update the state parameter, and return to the step of inputting the state parameter into the pre-trained resource balancing optimization model, until the resource allocation is completed for each service of the to-be-allocated resources included in the application request.

The communication bus mentioned in the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in FIG. 4, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above electronic device and other devices.

The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; may also be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

By applying the electronic device provided by the embodiment of the present invention, it is possible to determine a variety of services to be allocated resources included in the user's application program request, and the allocation priority of each service; determine the state parameters of the current edge micro-cloud system, and the state parameters include resources The balance degree evaluation parameter, the response delay evaluation parameter, and the remaining amount of resources of each computing node in each micro cloud; input the status parameter into the pre-trained resource balance optimization model to obtain the first target computing node of the first service, and put the first target computing node of the first service. A service is deployed on the first target computing node; the state parameter is updated, and the step of inputting the state parameter into the pre-trained resource balancing optimization model is returned, until the resource allocation is completed for each service of the to-be-allocated resources included in the application request. Therefore, the response delay and resource allocation balance are comprehensively considered, and the deep reinforcement learning method is used to train the network model. Compared with the traditional resource allocation method, it can not only meet the communication delay requirements, but also achieve a higher resource utilization balance.

Based on the same inventive concept, according to the implementation of the above-mentioned resource allocation method based on deep reinforcement learning, in another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer-readable storage medium is stored A program, when the computer program is executed by the processor, implements any of the above-mentioned steps of the resource allocation method based on deep reinforcement learning.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the above-mentioned embodiments of the resource allocation apparatus based on deep reinforcement learning, the electronic device embodiments, and the computer-readable storage medium embodiments, since they are basically similar to the above-mentioned embodiments of the resource allocation method based on deep reinforcement learning, the description It is relatively simple, and for relevant details, please refer to the partial description of the above-mentioned embodiment of the resource allocation method based on deep reinforcement learning.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (10)

1. A resource allocation method based on deep reinforcement learning is applied to a control platform of an edge micro-cloud system, wherein the edge micro-cloud system further comprises a plurality of micro-clouds, each micro-cloud comprises a plurality of computing nodes, and the method comprises the following steps:

determining services of various resources to be allocated contained in an application program request of a user and allocation priority of each service;

determining state parameters of a current edge micro cloud system, wherein the state parameters comprise a resource balance degree evaluation parameter, a response delay evaluation parameter and the resource surplus of each computing node in each micro cloud;

inputting the state parameters into a resource balance optimization model which is trained in advance to obtain a first target computing node of a first service; the first service is the service with the highest priority currently allocated; the resource balance optimization model is completed based on deep reinforcement learning training, wherein a training set of the deep reinforcement learning comprises: sample state parameters of the edge micro-cloud system;

deploying the first service to the first target computing node;

and updating the state parameters, and returning to the step of inputting the state parameters into the resource balance optimization model which is trained in advance until the resource allocation of each service of the resources to be allocated contained in the application program request is completed.

2. The method of claim 1, wherein the resource balance evaluation parameter is calculated based on the following formula:

wherein, the RUV_i ^jRepresenting the resource utilization variance of the jth computing node in the ith cloudlet, D representing the number of classes of resources,

indicating the resource utilization rate of the d-th type resource in the j-th computing node in the ith micro cloud,

represents the average value of the resource utilization rate of all kinds of resources in the jth computing node in the ith micro cloud, X represents the resource allocation strategy,

representing the resource balance rate, RUBD, of the jth computing node in the ith clout_iIndicating resource utilization balance of the ith cloudlet, L_iRepresenting the total number of compute nodes, RUBD, in the ith clout_TotalRepresenting a resource balance evaluation parameter of the edge micro cloud system, wherein K represents the total number of micro clouds in the edge micro cloud system;

calculating the response delay evaluation parameter based on the following formula:

t_Total＝T_Comp(X)+T_TR(X)

t_Totalindicating a response delay evaluation parameter, T_Comp(X) represents a calculation delay, T_TR(X) represents a transmission delay.

3. The method of claim 1, wherein the resource balancing optimization model is trained by:

acquiring a preset neural network model and the training set;

inputting the sample state parameters into the neural network model to obtain service placement actions; the service placement action represents determining a placed target compute node for a sample service;

updating the sample state parameters based on the service placement action to obtain updated sample state parameters;

calculating an incentive value of the service placement action based on the resource balance degree evaluation parameter and the response delay evaluation parameter contained in the sample state parameter and the resource balance degree evaluation parameter contained in the updated sample state parameter and the response delay evaluation parameter;

substituting the sample state parameter, the updated sample state parameter, the service placing action and the reward value of the service placing action into a preset loss function, and calculating the loss value of the service placing action;

determining whether the neural network model converges according to the loss value;

if not, adjusting parameter values in the neural network model, and returning to the step of inputting the updated sample state parameters into the neural network model to obtain a service placing action;

and if so, determining the current neural network model as a resource balance optimization model.

4. The method of claim 3, wherein the loss function is:

wherein L represents a loss function, E [ [ alpha ] ]]Representing a mathematical expectation, n representing the number of sets of historical iteration data referenced per iteration, t representing the time of day,

representing the priority weight, r, of n sets of historical iterative data after time t_t ⁽ⁿ⁾Representing the sum of the prize values for n iterations after time t,

representing a decay factor, Q, for a reward value of n iterations after time t_targetRepresenting the target network, Q_evaRepresenting an estimated network, s_tA sample state parameter representing the time t, a_tService Placement action, s, representing time t_t+nRepresents the state parameters of the samples after iteration n times, a' represents the service placement action for making the estimated network output the maximum value, k represents the iteration number,

representing a decay factor, r, for the reward value of the kth iteration after time t_t+k+1Representing the prize value of the kth iteration after time t.

5. A resource allocation device based on deep reinforcement learning is applied to a control platform of an edge micro-cloud system, wherein the edge micro-cloud system further comprises a plurality of micro-clouds, each micro-cloud comprises a plurality of computing nodes, and the device comprises:

the first determining module is used for determining services of various resources to be allocated contained in an application program request of a user and the allocation priority of each service;

the second determining module is used for determining state parameters of the current edge micro cloud system, wherein the state parameters comprise a resource balance degree evaluation parameter, a response delay evaluation parameter and the resource surplus of each computing node in each micro cloud;

the input module is used for inputting the state parameters into a resource balance optimization model which is trained in advance to obtain a first target computing node of a first service; the first service is the service with the highest priority currently allocated; the resource balance optimization model is completed based on deep reinforcement learning training, wherein a training set of the deep reinforcement learning comprises: sample state parameters of the edge micro-cloud system;

a deployment module to deploy the first service to the first target computing node;

and the updating module is used for updating the state parameters and triggering the input module until each service of the resources to be allocated contained in the application program request completes resource allocation.

6. The apparatus of claim 5, further comprising: a calculating module, configured to calculate the resource balance evaluation parameter based on the following formula:

wherein, the RUV_i ^jRepresenting the resource utilization variance of the jth computing node in the ith cloudlet, D representing the number of classes of resources,

indicating the resource utilization rate of the d-th type resource in the j-th computing node in the ith micro cloud,

represents the average value of the resource utilization rate of all kinds of resources in the jth computing node in the ith micro cloud, X represents the resource allocation strategy,

representing the resource balance rate, RUBD, of the jth computing node in the ith clout_iIndicating resource utilization balance of the ith cloudlet, L_iRepresenting the total number of compute nodes, RUBD, in the ith clout_TotalRepresenting a resource balance evaluation parameter of the edge micro cloud system, wherein K represents the total number of micro clouds in the edge micro cloud system;

calculating the response delay evaluation parameter based on the following formula:

t_Total＝T_Comp(X)+T_TR(X)

t_Totalindicating a response delay evaluation parameter, T_Comp(X) represents a calculation delay, T_TR(X) represents a transmission delay.

7. The apparatus of claim 5, further comprising: a training module, configured to train the resource balancing optimization model according to the following steps:

acquiring a preset neural network model and the training set;

inputting the sample state parameters into the neural network model to obtain service placement actions; the service placement action represents determining a placed target compute node for a sample service;

updating the sample state parameters based on the service placement action to obtain updated sample state parameters;

calculating an incentive value of the service placement action based on the resource balance degree evaluation parameter and the response delay evaluation parameter contained in the sample state parameter and the resource balance degree evaluation parameter contained in the updated sample state parameter and the response delay evaluation parameter;

substituting the sample state parameter, the updated sample state parameter, the service placing action and the reward value of the service placing action into a preset loss function, and calculating the loss value of the service placing action;

determining whether the neural network model converges according to the loss value;

if not, adjusting parameter values in the neural network model, and returning to the step of inputting the updated sample state parameters into the neural network model to obtain a service placing action;

and if so, determining the current neural network model as a resource balance optimization model.

8. The apparatus of claim 7, wherein the loss function is:

wherein L represents a loss function, E [ [ alpha ] ]]Representing a mathematical expectation, n representing the number of sets of historical iteration data referenced per iteration, t representing the time of day,

representing the priority weight, r, of n sets of historical iterative data after time t_t ⁽ⁿ⁾Representing the sum of the prize values for n iterations after time t,

representing a decay factor, Q, for a reward value of n iterations after time t_targetRepresenting the target network, Q_evaRepresenting an estimated network, s_tA sample state parameter representing the time t, a_tService Placement action, s, representing time t_t+nRepresents the state parameters of the samples after iteration n times, a' represents the service placement action for making the estimated network output the maximum value, k represents the iteration number,

representing a decay factor, r, for the reward value of the kth iteration after time t_t+k+1Representing the prize value of the kth iteration after time t.

9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;

a memory for storing a computer program;

a processor for implementing the method steps of any of claims 1 to 4 when executing a program stored in the memory.

10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 4.

Images