CN107423813A - A kind of state space based on depth learning technology decomposes and sub-goal creation method - Google Patents

Abstract

The present invention relates to a kind of state space based on depth learning technology to decompose and sub-goal creation method.Specifying for task is completed in the reward mechanism specified according to field, typical intensified learning (RL) agency study.In order to solve this problem, a framework is developed, a depth RL agency can use the attention mechanism of a repetition, from smaller, simpler domain, to more complicated domain.Task is presented to agency with the instruction of image and specified target.This cell controller guides agency to realize its target by designing a less subtask sequence in state space, so that effectively decomposition goal.

Description

A State Space Decomposition and Subgoal Creation Method Based on Deep Learning Techniques

Technical field:

The invention relates to a state space decomposition and sub-goal creation method based on deep learning technology.

Background technique

Design a deep learning framework in which deep learning can use a repeated attention mechanism through a proxy mechanism to map from smaller, simpler domains to more complex domains. The learning task is presented to the agent with images and instructions specifying the goal, and the meta-controller guides the agent to achieve its goal by designing several subtask sequences in the state space, thereby effectively decomposing, the meta-controller will create within the scope of attention subgoal. With a meta-controller, it learns to decompose the state space and provide resolvable subgoals in a smaller space, since the meta-controller is working on a delayed reward problem while the underlying agent solves the original task, thus being positively reinforced. It proposes a series of subgoals that maximize the expectation of this reinforcement. In addition to creating subgoals, the meta-controller also fragments the state space so that the underlying agent presents a smaller state that can be easily It learns an optimal policy for subgoals, it does this by using an attention mechanism, similar to the repeated attention pattern, the meta-controller learns to control its attention and only transfers part of the state to acting.

The formula for the meta controller is:

(1) State: S, is the past and present state representation.

Actions: A, is the attention location L ^attn , and the distribution g of a series of sub-goals.

Reward: r, if the underlying agent can solve the task, the reward r is negative; otherwise the reward r is positive.

Transition: The underlying agent executes its policy based on the provided state and sub-targets. Since this policy is unknown to the meta-control, it is an additional factor in the environment.

The meta-controller chooses a value for L ^attn and distribution P(g). In this value state space and a subgoal g, is passed to the underlying agent. The agent then chooses an atomic action that moves it to realize g. The new agent location L ^agent changes the environment of the meta-controller, which chooses a new focus and subgoal. First, the present invention assumes that the underlying agent has access to the optimal policy for each subgoal. Such goal-dependent policies can be learned by techniques such as Universal Approximate Value Functions (abbreviated: UVFAs). UVFAs learn a value function close to V(S,g) or about the target, using a function approximation similar to deep neural networks. Learning the value function V(S, g) can be used to construct a policy to achieve the goal g. This value function can be trained independently or combined with meta to provide intrinsic rewards for achieving subgoals. Second, the invention assumes that the agent still maintains state unless its position and subgoals are state provided to it by the meta-controller. In general, the meta-controller is automatically motivated to focus and provide subgoals so that potential agents can solve a given task due to its reward structure. In this case, that means keeping agent positions and subgoals in mind. For example, in the game of Pacman, if the subgoal is to eat the closest pill, then a potential agent should have Pacman and at least one kind of pill. Otherwise, the agent may move randomly and it will miss the overall goal of getting a high score. The assumptions above simplify the training of the meta-controller, but the approach presented should work for basic settings, where the policies of the underlying agents are also learned.

Invention content:

The purpose of the present invention is to provide a road edge detection method based on heuristic probability Hough transform.

Above-mentioned purpose realizes by following technical scheme:

Provides a hierarchical framework in which agents learn from intrinsic rewards from higher-level agents setting subgoals and operating over longer time frames. Rewards for advanced agents are provided by the environment for completing tasks. Subobjects are in turn provided by entities and relationships in an object-oriented framework. The inventive method further decomposes the state space so that the base agent only sees a small portion of it at any one time. The advantage is that it yields better computational efficiency, since the underlying agent can now use a smaller network, and can allow the learned policy to be transferred to a different part of the state space without having to explore it explicitly. To achieve this, higher-level agents, or meta-controllers, must learn to integrate information from the state observed so far. Therefore, the present invention uses a recurrent model to represent the meta-controller through a long short-term memory (LSTM) network. To train the attention mechanism of the meta-controller of the present invention, the present invention adopts a policy gradient to train the attention mechanism for classification and simple control tasks. In the inventive method, instead of using complex sensors, a 5x5 input image is simply used. This can easily be incorporated into the setup. Furthermore, instead of directly using the continuous output L ^attn , discrete actions, up, down, and noop, are used to divert attention.

Reinforcement Learning and Decision Process (MDP)

Reinforcement learning addresses the problem of choosing actions that maximize the notion of long-term cumulative reward. It is usually formulated as a Markov decision process (MDP). MDPs are characterized by tuples <S, A, T, R>. S is the set of states the agent can enter. A(S) is the set of actions available to the agent in each state. Typically, the agent chooses an action to perform from A, according to the transition function , which can bring it into a new state. is a scalar value received when performing an action in a state. At last, is the weight factor. strategy, namely , which tells the agent which action to perform in each state. The goal of a reinforcement learning agent is to find the optimal policy , to maximize the long-run expected return or utility.

gradient strategy

Gradient policy methods directly maximize expected rewards by tuning policy parameters. The expected reward for a trajectory sampled from policy π, θ is parameterized by

(1)

When p(S1:T;θ) depends on the distribution by π, the return orbit of R. In this formula, no discounting is done. The policy gradient theorem gives the gradient of J

(2)

The present invention can sample a set of trajectories from the current policy and average gradients on them. A common modification to reduce variance is to subtract the baseline from the return. The baseline bt is calculated by taking the average return observed over the past N time steps. N = 100 is chosen in this paper.

environment

By using an environment containing a 10x5 grid. The grid consists of four "spaces", each of which is a horizontal kx5 bar, some k not exceeding 4. These spaces are layered on top of each other, and each space has a different color, namely red, green, blue and yellow. The environment also generates a directive, as a one-hot vector of length 4, specifying the target space. An event terminates when the agent reaches the goal space, receives a positive reward of +1, or it times out without being reached. Step cost −0.01.

method

Three frameworks are constructed for the meta-controller, which is responsible for providing subgoals to the underlying agent so that it can successfully navigate to the goal space. In the present invention, the meta-controller uses the optimizer with a learning rate of 1×e−5. Agents and attention (if used) always start at the top left corner of the grid.

There is no attention mechanism in the meta controller

Simplifies the problem by taking the entire state space as input to the meta-controller at each stage, thus not using any attention mechanism. Specifically, the meta-controller receives as input a 10x5 grid image containing the space and the location of the agent, Lagent. The output of the meta-controller is a distribution over space P(g). A space is extracted from P(g) and provided as an instruction to the base agent. Assume that the underlying agent can move optimally over the entire 10x5 grid given the instruction. In this setting, the optimal policy for the meta-controller is to directly output the target space.

Meta-controller with partial state decomposition

In this setup, there is a meta controller with an attention mechanism consisting of a 5x5 window into a 10x5 grid. In addition to subgoal instructions, it must also output an action to control the watchpoint. The attention mechanism only partially decomposes the state space, which means that the agent may optimally move to the provided subgoal even if the agent is not currently in the attention domain. However, subgoals must be inside the attention. The goal of the meta-controller is to use its attention mechanism to find the location of the target space, and then instruct the agent to enter the color of the space at each step. The architecture of this meta-controller consists of a network of state processors that takes the environment as input to provide a 5x5 attention window and target instruction at each time step. It processes these inputs using a feed-forward convolutional network and uses LSTM cells to output P(g) and P(a), where P(a) is a probability distribution over the actions of the observation points. The convolutional layers of the network use rectified linear unit activation functions. Attention operations affect the next attention location Lattn, while subgoal instructions affect the next agent location Lagent. Thus, the hidden state of the LSTM incorporates the knowledge gained from taking instructions and observing point-action sequences in an episode. To efficiently train this network using policy gradients, it is assumed that the probabilities of attention and instruction actions are independent of each other.

Meta-controller with limited attention mechanism

In this setting, the agent will not move unless it is within attention range. This means that the meta-controller must instruct the agent to move into a space before moving down, and then repeat the process until the agent reaches the target space. Therefore, if the agent or subgoal does not appear in the decomposed state space, the goal task will not be achieved. Partial State Decomposition and Constraint Mechanism Experiments Note LSTM units allow the meta-controller to use the memory space of agent and target locations to guide its behavior, choosing an attention window that does not exist within the agent or target space at a particular time. The constrained viewpoint mechanistic framework adds an extra step for the agent to reach an optimal sequence of subgoals in the goal space.

Beneficial effect:

The overall contribution of this invention is a framework that allows an agent to complete a task in a large environment and learn how to do so in a smaller environment. By using an attention mechanism, smaller networks are required and easier to train. After developing these three frameworks, the meta-controller understands how spatial colors are represented and how to transfer the representation to sub-instructions that guide the agent to the desired goal. Our results show that policies can be scaled up in smaller environments by using attention mechanisms to decompose a large state space. The ultimate goal of the present invention is to train the underlying agents together with the meta-controller and apply this framework to dynamic and complex environments.

Description of drawings:

Accompanying drawing 1 is the status processing network. Network Architectures for Partial State Decomposition and Constrained Attention Mechanism Experiments. The 5x5 attention window and target instructions are input at each pass into the meta-controller, which then outputs a probability distribution.

Accompanying drawing 2 is the architecture of the element controller with attention mechanism. The architecture of this meta-controller consists of a network of state processors that takes the environment as input to provide a 5x5 attention window and target instruction at each time step. It processes these inputs using a feed-forward convolutional network and uses LSTM cells to output P(g) and P(a), where P(a) is a probability distribution over the actions of the observation points.

detailed description:

Embodiment 1: A method for state space decomposition and sub-goal creation based on deep learning technology, characterized in that: a deep learning framework is designed, in which deep learning can use a repeated attention mechanism through an agent mechanism, so that through more Small, simpler domains map to more complex domains. The learning task is presented to the agent with images and instructions specifying the goal, and the meta-controller guides the agent to achieve its goal by designing several subtask sequences in the state space, thereby effectively decomposing, the meta-controller will create within the scope of attention subgoal.

Embodiment 2: The meta-controller according to embodiment 1, characterized by the use of a meta-controller that learns to decompose the state space and provide resolvable sub-goals in a smaller space, because when the underlying agent solves the original task, the meta-controller is processing a delayed reward problem, which is positively reinforced. It proposes a series of sub-goals that maximize the expectation of this reinforcement. In addition to creating sub-goals, the meta-controller also fragments the state space so that the underlying agent presents a smaller state that can be easily It learns an optimal policy for subgoals, it does this by using an attention mechanism, similar to the repeated attention pattern, the meta-controller learns to control its attention and only transfers part of the state to acting.

Claims (2)

1. a kind of state space based on deep learning technology decomposes and sub-goal creation method, it is characterized in that：Design a depth Learning framework is spent, in the framework, deep learning can use the attention mechanism of a repetition by agency mechanism, so as to logical Cross smaller, simpler domain mapping and agency be presented to the instruction of image and specified target to more complicated domain, learning tasks, Using cell controller by designing several subtask sequences in state space the target that guides agency to realize it, so as to have Effect ground decomposes, and cell controller can create sub-goal in the range of concern.

2. cell controller according to claim 1, it is characterized in that：Using cell controller, it learns decomposing state space, and Analysable sub-goal is provided in less space, because when bottom acts on behalf of solution ancestral task, cell controller is located The award problem of one delay of reason, it obtains positive reinforcement, and it proposes a series of sub-goal, so that the expectation of this reinforcing is most Bigization, in addition to sub-goal is created, cell controller can also carry out fragmentation to state space, bottom agency be presented one smaller State, so as to easily be sub-goal learn an optimal policy, it is completed by using a kind of notice mechanism This process, similar to the attention force mode repeated, cell controller association controls its notice, and only by one of state Divide and pass to agency.