KR102463146B1 - HEMS optimization method and apparatus using hierarchical deep reinforcement learning - Google Patents

Abstract

The present invention discloses a HEMS optimization method and apparatus using hierarchical deep reinforcement learning. According to this embodiment, a non-controllable device, a non-blockable shiftable device and a scalable device with a fixed load, at a first level based on an actor-critic network model, comprising a processor and a memory connected to the processor; Performs energy consumption scheduling for ESS (Energy Storage System) and EV (Electric Vehicle) optimal charge/discharge scheduling in consideration of the energy consumption scheduling in the first level in the second level based on the actor-critic network A DRL-based HEMS optimization apparatus for storing program instructions executable by the processor is provided.

Description

HEMS optimization method and apparatus using hierarchical deep reinforcement learning

The present invention relates to a method and apparatus for optimizing HEMS using hierarchical deep reinforcement learning.

As residential households account for one-third of total electricity consumption, Home Energy Management System (HEMS) has become an essential technology for efficient and economical energy management.

The main goal of HEMS is to reduce electricity costs while ensuring comfort by scheduling the optimal energy consumption of smart appliances (eg air conditioners and washing machines, etc.).

More recently, distributed energy resources (DERs, such as roof solar photovoltaic (PV) and Energy Storage Systems (ESS)), advanced metering infrastructure with smart meters and demand management Smart grid technologies that include

The core technology of HEMS is an optimization method used to perform economical load reduction and load shifting of smart home appliances in addition to DER operation scheduling (charge/discharge).

However, the conventional HEMS optimization algorithm is model-based, but there is a problem in that it cannot provide a rather appropriate solution in an environment including smart home appliances, which is increasing recently.

Korean Patent Publication No. 10-2015-0040894

In order to solve the problems of the prior art, the present invention intends to propose a HEMS optimization method and apparatus using hierarchical deep reinforcement learning that can provide an optimized solution in consideration of a rooftop PV system, ESS, and smart home appliances.

In order to achieve the above object, according to an embodiment of the present invention, a DRL (Deep Reinforce Learning)-based HEMS (Home Energy Management System) optimization apparatus, comprising: a processor; and a memory coupled to the processor, wherein at a first level based on an actor-critic network model, perform energy consumption scheduling for uncontrollable devices with fixed loads, non-blockable shiftable devices and reducible devices. and to perform optimal charge/discharge scheduling of ESS (Energy Storage System) and EV (Electric Vehicle) in consideration of the energy consumption scheduling in the first level in the second level based on the actor-critic network, executed by the processor A DRL-based HEMS optimizer for storing possible program instructions is provided.

The non-blockable shiftable device may include a washing machine, and the shrinkable device may include an air conditioner.

The program instructions, in each of the first level and the second level, select and execute an action corresponding to a current state of each device agent, calculate a Q-value according to the current state and the selected action, an actor network and The optimal policy having the maximum Q-value may be determined by calculating the loss function of each critic network and repeating the above process.

The actor-critic network model has one input layer, a first hidden layer with 2n neurons, a second hidden layer corresponding to an actor network with n neurons, and a third corresponding to a critic network with n neurons. It may include an output layer for outputting a hidden layer and an average, variance, and Q-value related to an operation schedule of a device.

According to another aspect of the present invention, there is provided a method of optimizing HEMS (Home Energy Management System) based on Deep Reinforce Learning (DRL) in a device including a processor and a memory, at a first level based on an actor-critic network model, fixed performing energy consumption scheduling for uncontrollable devices with load, non-blockable shiftable devices, and reducible devices; And DRL-based HEMS comprising the step of performing optimal charge/discharge scheduling of ESS (Energy Storage System) and EV (Electric Vehicle) in consideration of the energy consumption scheduling in the first level in the second level based on the actor-critic network An optimization method is provided.

According to another aspect of the present invention, a computer readable program for performing the above method.

According to the present invention, since energy consumption scheduling of smart home appliances and ESS/EV is performed at two levels based on hierarchical reinforcement learning, there is an advantage in that optimal policy determination is possible.

1 is a diagram conceptually illustrating a DRL-based HEMS optimization system using the actor-critic method according to the present embodiment.
2 is a diagram illustrating an actor-critic network model configuration according to the present embodiment.
3 is a diagram illustrating a DRL-based HEMS optimization algorithm according to the present embodiment.
4 is a diagram showing the configuration of an apparatus for optimizing HEMS based on reinforcement learning according to a preferred embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

In the present invention, a 2-level DRL (Deep Reinforce Learning) framework to which the actor-critic method is applied is proposed, and (home appliance) devices (washing machine (WM) and air conditioner (AC)) that can be controlled at the first level are scheduled by consumers. and scheduling at a preferred level.

ESS and EV (Electric Vehicle) are scheduled in the second level to accommodate the WM and AC loads calculated in the first level along with the fixed load of uncontrollable devices.

The two-level framework according to this embodiment is motivated by the interdependent operation between the device of the first level and the ESS/EV of the second level.

1 is a diagram conceptually illustrating a DRL-based HEMS optimization system using the actor-critic method according to the present embodiment.

1, the DRL-based HEMS according to the present embodiment considers the TOU rate policy of the power supply company, weather information (eg, external temperature) of the Meteorological Agency, and the convenience level and PV of consumers. The operation of the washing machine and the air conditioner is scheduled in the level, and then the operation of the ESS and the EV is scheduled in the second level.

In the present invention, a situation in which automatic energy management is performed by the HEMS is considered, and it is assumed that the HEMS schedules and controls the following devices according to a time-to-use (TOU) fee.

)는 TV, PC 또는 조명과 같이 HEMS에 의해 스케줄링되거나 동작할 수 없는 기기이다. 따라서,

는 고정된 에너지 소비 스케줄링을 유지한다. uncontrollable devices (

) is a device that cannot be scheduled or operated by HEMS, such as a TV, PC, or lighting. therefore,

maintains a fixed energy consumption scheduling.

)는 HEMS에 의해 동작이 스케줄링되고 제어되는 기기이다. controllable device (

) is a device whose operation is scheduled and controlled by the HEMS.

)와 시프트 가능 기기(shiftable appliance,

)로 분류된다. The operational characteristics of a controllable device are

) and shiftable appliance;

) is classified as

For example, devices that can be reduced are devices that can reduce energy consumption to reduce electricity costs, such as air conditioners.

However, according to the TOU pricing scheme, the energy consumption of shiftable devices can be changed to different time zones to minimize the total cost of electricity.

)이고 다른 하나는 차단 가능 부하(interruptible load,

)이다. Shiftable devices are divided into two load types, one being a non-interruptible load.

) and the other is an interruptible load (interruptible load,

)to be.

The operation of a shiftable device with a breakable load during the device's task period shall not be interrupted by HEMS control.

For example, a washing machine must finish washing on a drying gun.

A shiftable device with a blockable load can be stopped at any time.

For example, when the PV power generation is greater than the load demand, the HEMS must stop the discharging process and immediately start charging the ESS.

Hereinafter, a conventional HEMS optimization algorithm will be first described, and a DRL-based HEMS optimization algorithm according to the present embodiment will be described in detail.

를 포함한다. As shown in Equation 1, the objective function for the HEMS optimization problem consists of two parts, each with a different decision variable.

includes

는 TOU 가격

및 시간 t에서 순 에너지 소비량에서 전체 에너지 비용이다. here,

is the TOU price

and total energy cost in net energy consumption at time t.

은 제어 가능 기기/제어 불가능 기기의 에너지 소비 및 예측된 PV 생성 출력의 관점에서 기재된다. In addition,

is described in terms of the energy consumption of the controllable/non-controllable equipment and the predicted PV generated output.

는 소비자 불편 비용과 관련된 총 패널티 금액이다.

is the total penalty amount associated with consumer inconvenience costs.

로부터 원하는 소비자 온도

와의 편차를 의미한다. Discomfort is room temperature

desired consumer temperature from

means a deviation from

는 소비자 불편 비용에 대한 패널티 기간이다.

is the penalty period for consumer inconvenience costs.

는 소비자의 전기 요금을 희생하여 소비자의 선호 편의 수준을 만족시키기 위해 HEMS 관리자에 의해 결정될 수 있다

can be determined by the HEMS manager to satisfy the consumer's preferred level of convenience at the expense of the consumer's electricity bill.

Hereinafter, equality and inequality constraints for the HEMS optimization problem are described.

와 예측된 PV 생성 출력

의 차이이다. Equation 2 is the constraint on net energy consumption, i.e. the total consumption of all devices.

and predicted PV generation output

is the difference of

In Equation 3, the total consumption of all devices can be divided into four different types of devices, including sensible devices, shiftable devices with non-blockable loads, shiftable devices with cutable loads, and non-controllable devices .

에서의,

, 예측된 외부 온도(

), 기기의 에너지 소비량 및 내부 열적 조건을 나타내는 환경 파라미터

표현되는 시간 t

에서의 온도 역학에 대한 제약을 나타낸다. For devices that can be reduced, Equation 4 is

in,

, the predicted external temperature (

), an environmental parameter indicating the energy consumption of the appliance and its internal thermal conditions

time t expressed

represents the constraint on the temperature dynamics at

및

로 제한되는 것을 나타낸다. Equation 5 represents the upper and lower limits of the range of the indoor temperature desired by the consumer, and Equation 6 represents the energy consumption capacity of the device that can be reduced.

and

indicates that it is limited to

를 갖는 세탁기와 같이 차단 불가능 부하를 갖는 시프트 가능 기기의 바람직한 동작을 보장한다.Equations 7 to 9 are the following binary decision variables

Ensure desirable operation of shiftable appliances with non-blocking loads, such as washing machines with

와

는 소비자가 원하는 시작 시간 및 종료 시간인 정지 기간 동안, (ii) 수학식 8에서 하루 중 동작 기간

시간 동안, (iii) 수학식 9에서

시간의 연속 동작 기간 동안 (i) in Equation 7

Wow

is during the stop period, which is the start time and end time desired by the consumer, (ii) the operation period of the day in Equation (8)

For time, (iii) in Equation 9

for a period of continuous operation of time

이다. In Equation 10, the energy consumption capacity of a shiftable device with a non-breakable load is

to be.

에 대한 에너지 상태(SOE)의 동역학을 이전 시간(t-1)에서의 SOE로 나타낸 것이고, 충전 및 방전 효율은

,

, 충방전 에너지는

및

이다. Equation 11 is ESS and EV at the current time (t)

The kinetics of the energy state (SOE) for is expressed as SOE at the previous time (t-1), and the charging and discharging efficiencies are

,

, the charge and discharge energy is

and

to be.

및

를 갖는 SOE 용량 제약을 나타낸다. Equation 12 is

and

SOE capacity constraint with .

는 ESS 및 EV 온/오프를 결정하는 이진 변수이다. Equations 13 and 14 represent the constraints on the charge and discharge energy of the ESS and EV, respectively, where

is a binary variable that determines ESS and EV on/off.

에서 그대로 유지되는 반면,

및

는

에서 0이 된다. 또한,

에 집을 떠날 때, EV의 SOE는 소비자 선호

보다 커야 한다. In particular, Equations 11 to 14, which are constraints on EV, are

While it remains the same in

and

Is

becomes 0 in In addition,

When leaving home to, EV's SOE is a consumer preference

should be greater than

Hereinafter, reinforcement learning will be described in detail.

Reinforcement learning is one of the machine learning techniques for making optimal decisions in non-deterministic environments.

While the agent interacts with the environment, the agent learns an action type that depends on the state of the environment and sends the learned action to the environment.

The environment then returns the reward to the agent along with the new environment state.

This learning process continues until the agent maximizes the total cumulative reward received from the environment.

A policy is defined as how an agent behaves in a particular state, and the agent's main goal is to determine the optimal policy that maximizes the reward.

In this embodiment, it is assumed that the environment is described by a Markov decision-making process in which agent state transitions depend only on the current state with selected actions from the current state without considering all past states and actions.

를 결정하기 위한 대표적인 강화학습 기법 중 하나이다. Q-learning is the optimal policy for decision-making problems

It is one of the representative reinforcement learning techniques for determining

와 행동

의 Q-값(

)을 계산하고, 총 보상을 최대화하는 방향으로 Q-값을 업데이트한다. The general process of Q-learning is a pair of pairs at time t using the Bellman equation

and action

Q-value of (

) and update the Q-value in the direction that maximizes the total reward.

에 기초하여 최적 Q-값(

)은 현재 보상

및 최대 디스카운트된 미래 보상

의 합에 의해 얻어지며, 여기서,

는 현재 및 미래 보상의 상대적 중요도를 설명하는 디스카운트 팩터를 나타낸다. In Equation 16, the optimal policy

Based on the optimal Q-value (

) is the current reward

and maximum discounted future rewards

is obtained by the sum of , where

denotes a discount factor that describes the relative importance of present and future rewards.

가 감소함에 따라 에이전트는 현재 보상에 점점 더 집중하기 때문에 근시안적으로 된다. 그러나, 더 큰

를 사용하면, 에이전트를 향후 보상에 점점 더 집중할 수 있어 원시안적으로 된다. 현재와 미래의 보상의 균형을 맞추기 위해

값은 Q-러닝을 이용하여 시스템 운영자에 의해 조정될 수 있다. discount factor

As α decreases, the agent becomes myopic because it focuses more and more on the current reward. However, larger

Using , you can focus the agent more and more on future rewards, becoming primitive. To balance present and future rewards

The values can be adjusted by the system operator using Q-learning.

Hereinafter, the actor-critic method will be described in detail.

The actor-critic method is an extension of the policy gradient method that can improve stability and reduce gradient variance when the algorithm's optimal solution converges.

When the specific state value of the agent is known, the corresponding Q-value can be calculated and applied to the RL method to compute the gradient of the policy network parameters and update the policy network. Thereby, it is possible to obtain a better cumulative result by increasing the agent action probability.

In the actor-critic method, the agent can use an additional network to judge the superiority of the action chosen by the agent in a particular state.

The policy network that returns the agent's action probability is called the actor network, and the network that returns the evaluation value of the agent's behavior is called the critic network.

The policy gradient method is suitable for dealing with problems in a continuous action space.

However, this method may have poor convergence performance.

An additional critic network of the actor-critic method can solve the convergence problem of the policy gradient method.

The actor network returns the probabilities of actions chosen by the agent in a particular state, and the critic network returns the numerical future that the agent can achieve in the final state.

The critic network updates the function that distinguishes the behavior and the current state numerical value, while the policy network updates the parameters in the direction suggested by the critic network.

In this embodiment, parameters of the actor network are updated by reinforcement learning, and parameters of the critic network are updated by linear temporal difference (TD).

It is well known that this is useful for solving Markov decision process problems.

In this embodiment, we choose a linear value function approximation to apply TD to the critic network.

The actor-critic method minimizes TD errors by updating the parameters of the actor and critic networks.

It encodes the difference between the current state value function and the target value function.

Algorithm 1 below is about reinforcement learning, and Algorithm 2 shows an actor-critic approach using the TD method.

Below, we describe a hierarchical two-level DRL framework for scheduling optimal daily consumption of a single household using smart appliances and DERs within consumer preferred device scheduling and convenience levels using actor-critic methods.

In the framework according to the present embodiment, the energy scheduling problem is divided into i) a washing machine and an air conditioner in a first level, and ii) an energy consumption scheduling of an ESS and an EV in a second level.

The first level describes the energy management model for washing machines and air conditioners.

The DRL-based HEM optimization algorithm according to the present embodiment considers a situation where it is executed for 24 hours with a scheduling resolution of 1 hour.

에서, 제1 레벨에서 세탁기 및 에어컨의 상태 공간(state space)은 각각 다음과 같이 표현된다.

In the first level, the state spaces of the washing machine and the air conditioner are respectively expressed as follows.

와

는 시간 t에서 세탁기 및 에어컨의 스케줄링 시간, TOU 요금 및 실외/실내 온도를 나타낸다. here,

Wow

denotes the scheduling time, TOU rate, and outdoor/indoor temperature of the washing machine and air conditioner at time t.

At the first level, the optimal behavior for each device depends on the agent's environment, including its current state, as described above.

In the first level, the action space of the washing machine and the air conditioner is as follows.

와

는 시간 t에서 세탁기 및 에어컨의 에너지 소비량이다.

Wow

is the energy consumption of washing machines and air conditioners at time t.

는 연속적인 값이고,

는 불연속적인 값이다. here,

is a continuous value,

is a discrete value.

이고, 오프되면 0이다. WM is on

, and 0 when off.

The reward function for each device agent is formulated as the sum of negative electricity costs and negative dissatisfaction costs related to consumer preference convenience and device operating characteristics.

In the first level, the total rewards are as follows:

및

은 세탁기 및 에어컨의 비용 함수이다. here,

and

is a function of the cost of washing machines and air conditioners.

Each cost function includes the electrical cost of the appliance, along with the consumer's dissatisfied cost of undesired operation and indoor thermal discomfort.

First, the cost function of the washing machine is as follows.

및

는 각각 세탁기의 소비자 선호 시작 및 종료 시간이고,

및

는 선호 동작 시간 간격과 비교하여 이른 동작 및 늦은 동작에 대한 패널티이다. 세탁기 에이전트가

이전 또는

이후에 세탁기 에너지 소비를 스케줄링하면 불만족 비용이 비용 함수에 추가되며, 그렇지 않으면 비용 함수는 전기 비용만을 포함한다. here,

and

are the consumer preferred start and end times of the washing machine, respectively,

and

is the penalty for early and late operation compared to the preferred operation time interval. washing machine agent

before or

If the washing machine energy consumption is subsequently scheduled, the dissatisfaction cost is added to the cost function, otherwise the cost function includes only the electricity cost.

The cost function of the air conditioner is

와

는 소비자의 열적 불편함에 대한 패널티이다. here,

Wow

is a penalty for thermal discomfort of consumers.

와

및

의 차이로 정의된다.The cost of dissatisfaction is the consumer's preferred temperature

Wow

and

is defined as the difference between

및

에 대한 소비자의 불만족 비용이 트레이드 오프 관계를 가지는 점을 나타낸다. The two terms in Equations (20) and (21) are energy cost savings and penalty

and

It indicates that the cost of consumer dissatisfaction with respect to

According to the trade-off relationship, the HEMS system using the DRL algorithm according to the present embodiment may adaptively adjust the penalty according to the situation where the consumer wants to save more electricity cost or the consumer wants to maintain the desired comfort and preference.

The choice of these penalty values depends on the comfort level and environment desired by the consumer.

Optimal scheduling of the energy consumption of washing machines and air conditioners at the first level with a fixed load of uncontrollable appliances is embedded in the actor-critic module at the second level.

In the second level, the ESS and EV agents initiate a learning process to determine the optimal charging/discharging scheduling of the ESS and EV to minimize household electricity costs.

It is assumed that the energy generated by the PV system in the learning process at the second level is first charged to the ESS.

In the second level, the state space of the agent managing the operation of the ESS and EV is defined as follows.

및

는 각각 ESS 및 EV의 스케줄링 시간, 시간 t에서의 TOU 요금, ESS 및 EV의 SOE, 예측된 PV 생성 출력, 제1 레벨에서 계산된 전체 에너지 소비 스케줄링이다. here,

and

are the scheduling times of the ESS and EV, the TOU fee at time t, the SOE of the ESS and EV, the predicted PV generation output, and the total energy consumption scheduling calculated at the first level, respectively.

Similar to the washing machine and air conditioner described above, the action spaces of the ESS and EV in the second level are expressed as follows.

및

는 시간 t에서 ESS 및 EV의 연속적인 에너지 충전 및 방전을 나타낸다. here,

and

represents the continuous energy charging and discharging of ESS and EV at time t.

를 이용하여 최적 충방전 행동을 선택하고,

는 제1 레벨에서 고정 부하를 갖는 제어할 수 없는 기기와 함께 세탁기 및 에어컨의 에이전트의 행동을 포함한다. In the two-level DRL-based HEMS framework according to the present embodiment, the ESS and EV agents are

to select the optimal charging/discharging behavior,

contains the behavior of agents of washing machines and air conditioners with uncontrollable appliances with fixed loads at the first level.

If the DRL-based HEMS algorithm is modeled as a single-level framework, the agent for ESS and EV cannot find an optimal policy. This is because ESS and EV cannot know the energy consumption data of other devices in their state space.

Compensation at the second level is defined as the negative of the sum of the cost of electricity and the dissatisfaction of the ESS and EV related to the consumer's preferred comfort and operating characteristics of the appliance.

및

는 ESS와 EV의 비용 함수이다. here,

and

is the cost function of ESS and EV.

Each cost function includes the cost of electricity for the device, along with the dissatisfaction cost for overcharging and undercharging of ESS and EV.

These cost functions include the discharge energy from the ESS and EV, which supports the uncovered energy consumption of the full load of washing machines, air conditioners and uncontrollable appliances.

First, the cost function of ESS is expressed as follows.

와

는 ESS 과충전 및 부족충전에 대한 패널티이다. 이러한 경우, ESS의 에너지 부족 이용 및 에너지 소실은 SOE가

보다 낮거나(부족충전) 또는

(과충전)보다 커지면 발생하고, ESS의 에너지 부족 이용 중에 에너지 비용과 함께 보상에 반영된다.here,

Wow

is a penalty for ESS overcharging and undercharging. In this case, the energy shortage use and energy dissipation of the ESS is

lower (undercharge) or

It occurs when it becomes larger than (overcharge) and is reflected in the compensation along with the energy cost during the use of energy shortage of the ESS.

Next, the cost function of EV is expressed as

및

는 EV의 과충전 및 부족충전에 대한 패널티이다. ESS와 유사하게, 에너지 부족 이용 및 에너지 소실은 SOE가

보다 낮거나(부족충전) 또는

(과충전)보다 커지면 발생한다. ESS의 보상 함수와는 달리, EV의 보상 함수는 파라미터

를 포함하고, 이는 EV의 소비자의 선호도 패널티를 나타내고, 선호도 패널티는 EV가 출발할 때 소비자가 선호하는 SOE와 EV의 SOE와의 차이이다.

and

is a penalty for overcharging and undercharging of EVs. Similar to ESS, energy scarcity utilization and energy dissipation

lower (undercharge) or

Occurs when it becomes greater than (overcharge). Unlike the reward function of ESS, the reward function of EV is a parameter

, which represents the consumer's preference penalty of EV, and the preference penalty is the difference between the consumer's preferred SOE and the EV's SOE when the EV starts.

가 출발 시간

에서

보다 낮으면 불만족 비용의 부족한 SOE에 따라 증가한다. if

autumn departure time

at

If lower, the cost of dissatisfaction increases with insufficient SOE.

Compared with the RL method, the policy gradient approach is suitable for engineering problems with a continuous action space. In general, continuous policy gradient networks obtain state information from agents and return appropriate actions using a normal distribution.

These networks compute the mean and variance to achieve a normal distribution, and the agent randomly samples the behavior according to the resulting distribution.

In the actor-critic approach, an additional method of critique of Q-values for efficiency and convergence is added.

Therefore, the network provides the mean, variance and Q-value to find the optimal behavior.

As shown in Figure 2, the actor-critic network model for each level consists of one input layer for state elements, a first hidden layer for common body network with 512 neurons, and actor and critic networks with 256 neurons. 2nd and 3rd hidden layers for , and one output layer for average, variance, and Q-values related to the operation schedule of the device.

In this embodiment, a hyperbolic tangent function is used as a transfer function. In addition, an adaptive moment estimation algorithm is used for training the DRL model.

The algorithm of Fig. 3 shows the process of the actor-critic based HEMS learning energy management policy, and optimizes the electricity rate for the first level and the second level and the convenience level of the consumer.

Referring to FIG. 3 , in each of a first level and a second level, a series of actions for a current state is selected, one of the selected actions is executed, and a Q-value according to the current state and the next state is calculated.

After that, each loss function of the actor network and the critic network is calculated, and each loss is stored in a buffer.

Next, the parameters of the policy network are updated by applying the RL method, and the above method is repeatedly performed to determine the optimal policy having the maximum Q-value.

4 is a diagram showing the configuration of an apparatus for optimizing HEMS based on reinforcement learning according to a preferred embodiment of the present invention.

As shown in FIG. 4 , the apparatus for optimizing HEMS based on reinforcement learning according to the present embodiment may include a processor 400 and a memory 402 .

The processor 400 may include a central processing unit (CPU) capable of executing a computer program or other virtual machines.

Memory 402 may include a non-volatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, and the like. Memory 402 may also include volatile memory, such as various random access memories.

The memory 402 stores program instructions executable by the processor 400 .

The program instructions according to this embodiment perform, at the first level, based on the actor-critic network model, energy consumption scheduling for uncontrollable devices with fixed loads, non-blockable shiftable devices and reducible devices, Based on the actor-critic network, the optimal charging/discharging scheduling of ESS (Energy Storage System) and EV (Electric Vehicle) is performed in consideration of the energy consumption scheduling in the first level in the second level.

Here, the non-blockable shiftable device may include a washing machine, and the shrinkable device may include an air conditioner.

In addition, the program instructions according to this embodiment select and execute an action corresponding to the current state of each device agent at each of the first level and the second level, and calculate a Q-value according to the current state and the selected action and calculates the loss function of each of the actor network and the critic network, and repeats the above process to determine the optimal policy with the maximum Q-value.

The above-described embodiments of the present invention have been disclosed for the purpose of illustration, and various modifications, changes, and additions will be possible within the spirit and scope of the present invention by those skilled in the art having ordinary knowledge of the present invention, and such modifications, changes and additions should be regarded as belonging to the following claims.

Claims (7)

As a DRL (Deep Reinforce Learning) based HEMS (Home Energy Management System) optimization device,
processor; and
a memory coupled to the processor;
At the first level based on the actor-critic network, the TOU (time-to-use) rate of the power supply company, the weather information of the Meteorological Agency (eg outside temperature), and the convenience level of consumers and the distributed energy resource are fixed in consideration of the perform energy consumption scheduling for uncontrollable devices with load, non-blockable shiftable devices, and devices that can be reduced;
To perform optimal charging/discharging scheduling of ESS (Energy Storage System) and EV (Electric Vehicle) in consideration of the energy consumption scheduling in the first level in the second level based on the actor-critic network,
Stores program instructions executable by the processor,
The program instructions are
In each of the first level and the second level, select and execute an action corresponding to the current state of each device agent, calculate a Q-value according to the current state and the selected action, and lose each of the actor network and the critic network Calculate the function, repeat the above process to determine the optimal policy with the maximum Q-value,
The reward function for each device agent is formulated as the sum of negative electricity costs and negative dissatisfaction costs related to consumer preference convenience and device operating characteristics,
The state space of each agent of the ESS and EV includes the scheduling time of the ESS and EV, the TOU fee at time t, the energy state of the ESS and EV, the predicted PV generation output, and the total energy consumption calculated at the first level. Scheduling is defined as
The action space of each agent of the ESS and EV is defined as the continuous energy charging and discharging of the ESS and EV at time t,
The total energy consumption scheduling calculated in the first level is a DRL-based HEMS optimization device including the actions of agents of uncontrollable devices, non-blockable shiftable devices and reduceable devices with the fixed load in the first level. . According to claim 1,
The non-blockable shiftable device includes a washing machine, and the shrinkable device includes an air conditioner. delete According to claim 1,
The actor-critic network model has one input layer, a first hidden layer with 2n neurons, a second hidden layer corresponding to an actor network with n neurons, and a third corresponding to a critic network with n neurons. A DRL-based HEMS optimization apparatus including a hidden layer and an output layer for outputting mean, variance, and Q-values related to an operation schedule of a device. A method of optimizing a Home Energy Management System (HEMS) based on Deep Reinforce Learning (DRL) in a device comprising a processor and memory, the method comprising:
(a) At the first level based on the actor-critic network model, the time-to-use (TOU) rate of the power supply company, the weather information of the Meteorological Administration (eg outside temperature) and the level of convenience of consumers and distributed energy resources performing energy consumption scheduling for uncontrollable devices, non-blockable shiftable devices, and reducible devices with a fixed load taking into account; and
(b) performing optimal charge/discharge scheduling of ESS (Energy Storage System) and EV (Electric Vehicle) in consideration of the energy consumption scheduling in the first level in the second level based on the actor-critic network,
Each of the steps (a) and (b) selects and executes an action corresponding to the current state of each device agent, calculates a Q-value according to the current state and the selected action, respectively, an actor network and a critic network Calculate the loss function of , repeat the above process to determine the optimal policy with the maximum Q-value,
The reward function for each device agent is formulated as the sum of negative electricity costs and negative dissatisfaction costs related to consumer preference convenience and device operating characteristics,
The state space of each agent of the ESS and EV is the scheduling time of the ESS and EV, the TOU fee at time t, the energy state of the ESS and EV, the predicted PV generation output, and the total energy consumption calculated at the first level. Scheduling is defined as
The action space of each agent of the ESS and EV is defined as the continuous energy charging and discharging of the ESS and EV at time t,
The total energy consumption scheduling calculated in the first level is a DRL-based HEMS optimization method comprising the actions of agents of uncontrollable devices, non-blockable shiftable devices and reduceable devices with the fixed load in the first level . 6. The method of claim 5,
The non-blockable shiftable device includes a washing machine, and the shrinkable device includes an air conditioner. A computer readable program for performing the method according to claim 5 .

Images