JP2002133390A - Learning device and recording medium - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To provide a learning device capable of improving learning efficiency by effectively using both rewards and teacher signals when given from the environment. SOLUTION: A learning device 260 that generates an output value 17 according to an input value 15 executes a first learning based on a reward signal to thereby perform a first learning based on at least one first parameter. The first learning unit 12 that generates one output value 212 and the second learning based on the teacher signal are executed to generate a second output value 213 corresponding to the input value 15 based on at least one second parameter. A second learning unit 13, a first output value 212 and a second output value 2
13, an output unit 14 for selectively outputting one of the output values as an output value 17, whether to designate first learning by the first learning unit 12, and whether to designate second learning by the second learning unit 13. And a learning designating unit 11 for determining whether or not there is a difference.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a learning apparatus and a recording medium to which two types of information, a teacher signal and a reward, are externally provided and learning is performed based on both.

[0002]

2. Description of the Related Art In recent years, there has been a strong demand for industrially used systems that can be customized in response to individual user requirements. The requirements of individual users are becoming increasingly complex. In addition, information networks, which are used environments of systems used in industry, are rapidly developing, and the systems must be able to cope with such environmental changes. For these reasons, it is becoming difficult to design all the functions necessary for the system in advance when developing the system. for that reason,
Research is being conducted on learning techniques that allow the system itself to adapt to the user and the environment. Such a system that can adapt to a user or an environment by using a learning technique is referred to as a learning device in this specification.

[0003] When the conventional learning techniques are classified from the viewpoint of information used when the learning device performs learning, they are classified into learning based on teacher signals and learning based on rewards.

As learning based on teacher signals, a method using a neural network (document: DE Ru)
melhart et al., Parallel distri
butted processing: Explora
tensions in themicrostructure
e of cognition, The MITPr
ess, 1986) and a method using a decision tree (reference:
J. R. Quinlan, C4.5: Progr
ams for machine learning,
Morgan Kaufmann, 1993).

Reward-based learning includes reinforcement learning (RS Sutton and A. Barto:
Reinforcement Learning: A
n Induction, A Bradfor
d Book, The MIT Press, 19
98) are known.

[0006] As a conventional technique which employs the above-mentioned learning method in a learning apparatus, there are a technique disclosed in Japanese Patent No. 2856259 and a technique disclosed in Japanese Patent Application Laid-Open No. 2000-35956.

In the prior art disclosed in Japanese Patent No. 2856259, supervised learning is performed when learning a behavior method of a robot in a neural network for behavior determination. As the teacher data for the supervised learning, a data set having a good evaluation is selected from a data set appropriately generated by the noise adder. For the selection of the teacher data, the evaluation value learning neural network stores the quality of the reward up to now, and only when the reward is good, is used as the teacher data for learning the action determining neural network.

In the prior art disclosed in Japanese Patent Application Laid-Open No. 2000-35956, the action of the action generator is determined based on the reward from the environment, and the state of how the environment is changed by the action of the action generator is determined. A supervised learning is performed by a predictor, thereby predicting a change in state.

[0009]

When a system attempts to use a learning technique to adapt to a complex environment in the real world, a teacher signal and a reward are required to efficiently learn the system (learning apparatus). It is necessary to use both information effectively instead of either one. In a real environment, a learning device can often use both a teacher signal and a reward given from the outside. For example, in a learning device used for constructing a design support system, information of a design process by design specialization as a teacher signal, that is, what kind of design operation was performed in what situation, specifically, layout design In what kind of arrangement situation and how to perform the arrangement conversion operation,
Information is available. At the same time, information on the quality of the final design result is usually available as a reward. Further, in the learning device used for the transport robot, information of a transport example indicated by human operation can be used as a teacher signal, and information on whether or not the transport was successfully performed can be used as a reward. is there. Effective use of both the available teacher signals and rewards is necessary for effective learning.

However, a learning device that performs learning based on both a teacher signal obtained from the environment and a reward has not been realized conventionally.

In the prior art disclosed in Japanese Patent No. 2856259 and the prior art disclosed in Japanese Patent Application Laid-Open No. 2000-35956, the information given from the environment among the teacher signal and the reward is only the reward. Met.

As described above, in the learning device according to the prior art, learning is performed using only rewards or only teacher signals among available information given from the environment. For this reason, information given from the environment cannot be effectively used, and learning efficiency cannot be improved.

[0013] The present invention has been made in view of the above-mentioned problem, and when a reward and a teacher signal are given from the environment, the learning efficiency can be improved by effectively using both of them. It is intended to provide a device.

[0014]

A learning device according to the present invention is a learning device for generating an output value corresponding to an input value based on at least one parameter, wherein a reward indicating an evaluation value related to the output value is provided. A reward signal input unit for receiving a signal from outside the learning device; a teacher signal input unit for receiving a teacher signal indicating an expected value of the output value with respect to the input value from outside the learning device; the reward signal and the teacher signal And an adjustment unit that adjusts the value of the at least one parameter so that the evaluation value increases and the output value with respect to the input value approaches the expected value. The above object is achieved.

[0015] The apparatus further includes a pattern generation unit that generates a first learning pattern based on the reward signal and generates a second learning pattern based on the teacher signal.
The value of the at least one parameter is adjusted based on the first learning pattern so that the evaluation value is increased, and the output value with respect to the input value approaches the expected value based on the second learning pattern. Adjusting the value of the at least one parameter, wherein the first learning pattern adjusts the value of the at least one parameter such that the evaluation value is increased based on the first learning pattern. In this case, the output value with respect to the input value is configured to approach the expected value, and the second learning pattern is such that the adjustment unit outputs the output with respect to the input value based on the second learning pattern. When the value of the at least one parameter is adjusted so that the value approaches the expected value, the evaluation value may be configured to increase. .

[0016] The pattern generating section always generates the second learning pattern based on the teacher signal when the teacher signal is input to the pattern generating section, and the teacher signal is not input to the pattern generating section. In this case, the first learning pattern may be generated based on the reward signal.

The apparatus further comprises a determination unit for determining the quality of the teacher signal based on a predetermined criterion, wherein the pattern generation unit depends on the quality of the teacher signal and determines the second learning pattern based on the teacher signal. May be determined.

Another learning device of the present invention is a learning device that generates an output value according to an input value, and executes a first learning based on a reward signal to obtain at least one first parameter. A first learning unit that generates a first output value corresponding to the input value based on the input value, and a second learning unit that performs a second learning based on a teacher signal, based on at least one second parameter. A second learning unit that generates two output values, an output unit that selectively outputs one of the first output value and the second output value as the output value,
Whether to designate the first learning by the first learning unit,
And a learning designating unit that determines whether or not to designate the second learning by the second learning unit, wherein the first learning unit sends a reward signal indicating an evaluation value related to the output value to the learning unit. A reward signal input unit received from outside the device, and a first adjustment unit that adjusts a value of the at least one first parameter based on the reward signal so that the evaluation value increases. A learning unit configured to receive a teacher signal indicating an expected value of the output value with respect to the input value from outside the learning device; and a second output value corresponding to the input value based on the teacher signal. A second adjusting unit that adjusts a value of the at least one second parameter so as to approach an expected value, thereby achieving the above object.

The learning specifying section may determine whether or not to specify the first learning according to a value of an attention parameter that changes with time.

The output unit outputs the first output value as the output value or outputs the second output value as the output value according to a value of a storage status parameter that changes with time. May be determined.

[0021] The output unit, based on the reward signal,
It may be determined whether to output the first output value as the output value or to output the second output value as the output value.

[0022] The learning device is configured to perform
The image processing apparatus may further include a direct imitation unit that outputs the teacher signal as the second output value.

[0023] The learning designating section alternatively designates the first learning by the first learning section and the second learning by the second learning section, and the learning designating section designates the first learning. If the learning state has transitioned to a state in which the learning specifying unit specifies the second learning, the at least one
Two learning parameters are adjusted based on the at least one first parameter, and a state in which the learning designating section designates the second learning is a state in which the learning designating section designates the first learning. In the case where the transition has been made, the at least one first parameter may be adjusted based on the at least one second parameter.

[0024] The learning device further includes a voice recognition unit that recognizes voice input from outside the learning device, and a semantic interpretation unit that extracts semantic information based on a recognition result by the voice recognition unit. Is also good.

[0025] The learning device may further include an instruction input unit for receiving an operation instruction to the learning device, wherein the semantic information is information related to at least one of the operation instruction, the teacher signal, and the reward signal. You may.

Another learning device of the present invention is a learning device for generating an output value according to an input value, wherein the learning device comprises:
A plurality of learning modules that generate a module output value according to the input value by executing learning, and determine whether or not to specify the learning by each of the plurality of learning modules based on a predetermined first rule. A learning designation unit to be determined, an operation selection unit to selectively output one of a plurality of module output values output from the plurality of learning modules as a first output value based on a predetermined second rule, A learning section that generates a second output value according to the input value based on one parameter; and outputs one of the first output value and the second output value to the output value based on a predetermined third rule. And an output unit that selectively outputs the at least one parameter, such that the second output value approaches the first output value, and the learning unit adjusts the at least one parameter. When the two output values are selected, an operation of generating a module output value by the plurality of learning modules, an operation of determining whether or not to specify the learning by the learning specifying unit, and an operation of determining whether to specify the learning by the operation selecting unit. The output operation of selectively outputting one of the module output values as the first output value is stopped, thereby achieving the above object.

Another learning device of the present invention is a learning device for generating an output value according to an input value, wherein the learning device comprises:
Generating a first new classifier based on at least one classifier indicating a rule defining a first output value for the input value and a reward signal indicating an evaluation value related to the output value; A first generation unit that adds one new classifier to the at least one classifier; and generates a second new classifier based on a teacher signal indicating an expected value of the output value with respect to the input value. A second generating unit that adds the second new classifier to the at least one classifier, and the at least one of the first and the second new classifiers to which the first new classifier and the second new classifier are added. A setting unit for setting a reliability for each of the two classifiers, the first new classifier and the second
At least one of the first output values output from each of the at least one classifier to which the new classifier has been added, the first new classifier and the second new classifier, And an output unit that generates the output value based on the reliability set for each of the at least one classifier to which the above-mentioned is added, thereby achieving the above object.

A recording medium according to the present invention is a computer-readable recording medium recording a program for causing a learning device to execute a process of generating an output value corresponding to an input value based on at least one parameter, The process of generating an output value according to the input value includes: (a) receiving a reward signal indicating an evaluation value related to the output value from outside the learning device; and (b) outputting the output value with respect to the input value. (C) receiving, based on the teacher signal and the reward signal, the output value with respect to the input value approaching the expected value, and Adjusting the value of the at least one parameter so that the evaluation value becomes higher, thereby achieving the above object.

Another recording medium of the present invention is a computer-readable recording medium in which a program for causing a learning device to execute a process of generating an output value according to an input value is recorded. (A) performing a first learning based on a reward signal to generate a first output value corresponding to the input value based on at least one first parameter;
(B) generating a second output value corresponding to the input value based on at least one second parameter by executing second learning based on a teacher signal; and (c).
(D) selectively outputting one of the first output value and the second output value as the output value;
Determining whether to designate learning and whether to designate the second learning, wherein the step (a) comprises: (a1) a reward signal indicating an evaluation value related to the output value; (A2) adjusting the value of the at least one first parameter based on the reward signal so as to increase the evaluation value. b)
(B1) receiving a teacher signal indicating an expected value of the output value with respect to the input value from outside the learning device; and (b2) determining the first output value with respect to the input value based on the teacher signal. Adjusting the value of the at least one second parameter to approach the expected value, thereby achieving the above object.

Another recording medium of the present invention is a computer-readable recording medium recording a program for executing a process of generating an output value corresponding to an input value, wherein the output value corresponds to the input value. The process of generating
(A) generating a plurality of module output values corresponding to the input values by executing a plurality of learnings;
(B) determining whether to designate each of the plurality of learning, based on a predetermined first rule; and (c).
Selectively outputting one of the plurality of module output values as a first output value based on a predetermined second rule; and (d) performing a second output corresponding to the input value based on at least one parameter. Generating two output values;
(E) selectively outputting one of the first output value and the second output value as the output value based on a predetermined third rule, wherein the step (d) comprises: (d)
1) so that the second output value approaches the first output value,
Adjusting the at least one parameter, and when the second output value is output as the output value in the step (e), generating the plurality of module output values in the step (a). Determining whether or not to designate each of the plurality of learning in step (b); and selectively outputting one of the plurality of module output values as the first output value in step (c). Operation is stopped, thereby achieving the above object.

Another recording medium of the present invention is a computer-readable recording medium recording a program for executing a process of generating an output value corresponding to an input value using at least one classifier, Each of the at least one classifier indicates a rule that defines a first output value for the input value, and the processing of generating an output value according to the input value includes: (a) an evaluation value associated with the output value Generating a first new classifier based on the reward signal indicating: and adding the first new classifier to the at least one classifier; and (b) the output value for the input value Generating a second new classifier based on a teacher signal indicating an expected value of the second classifier, and adding the second new classifier to the at least one classifier; (c)
Setting a reliability for each of the at least one classifier to which the first new classifier and the second new classifier have been added; and (d) the first new classifier. At least one first output value output from each of the at least one classifier to which the new new classifier has been added, the first new classifier, and the second new classifier. Generating the output value based on the reliability set for each of the at least one classifier to which the new classifier has been added, thereby achieving the above object.

[0032]

DESCRIPTION OF THE PREFERRED EMBODIMENTS First, the principles of learning based on a teacher signal and learning based on a reward will be described.

FIG. 1A shows an input / output relationship of a learning device 100 that performs learning based on a teacher signal. An input value X is input to the learning device 100 from an environment (outside the learning device 100). Learning apparatus 100 includes a variable a ₁ as an internal parameter.

The learning device 100 outputs an output value Y corresponding to the input value X. Here, the relationship between the output value Y and the input value X is represented by (Equation 1).

[0035]

Y = f ₁ (a ₁ , X) where f ₁ is a predetermined function.

In general, there is an expected value of the output value Y with respect to the input value X. The expected value is, for example, information that directly specifies an output value (or an action to be taken) that the learning device 100 should output in a certain state. Such expected value, from the outside of the learning device 100, is input to the learning apparatus 100 as a teacher signal Y _0. Based on the teacher signal Y ₀ , the learning device 100 sets the variable a ₁ so that the output value Y = f ₁ (a ₁ , X) for a specific input value X approaches the teacher signal Y ₀ (expected value). Adjust the value of. This is called learning based on a teacher signal.

In the example shown in FIG. 1A, input values,
Although the output value and the internal parameter are each assumed to be a single scalar quantity, the input value, the output value and the internal parameter may each be a plurality of scalar quantities or may be vector quantities. When the learning device 100 is a learning device employing a neural network, the internal parameter may be, for example, a connection weight of the neural network.
When the learning device 100 describes the input / output relationship by a set of IF-THEN rules, and the output value is determined by the weight coefficient set for each of the IF-THEN rules, the internal parameter is, for example, the IF-THEN rule. It may be a weight coefficient set for each.

FIG. 1B shows an input / output relationship of a learning device 200 that performs learning based on rewards. An input value X is input to the learning device 200 from an environment (outside the learning device 200). The learning device 200 uses the variable a ₂ as an internal parameter.
have.

The learning device 200 outputs an output value Y corresponding to the input value X. Here, the relationship between the output value Y and the input value X is represented by (Equation 2).

[0040]

Y = f ₂ (a ₂ , X) where f ₂ is a predetermined function.

When the learning device 200 outputs the output value Y to the environment, a reward (reward signal) R _{0 is} output from the environment according to the output value Y.
Is input to the learning device 200. Reward is learning device 2
00 does not directly specify the output value to be output (or the action to be taken), and the learning device 200
Output to the environment, the learning device 200,
Information given to the learning device 200 when a state defined by the environment in which the learning device 200 exists and a specific state, and whether the state is desirable, that is, the output value Y Indicates whether or not was desirable.

Thus, the reward R₀Is related to the output value Y
The evaluation value is shown. The learning device 200 outputs the output value Y
By doing so, the learning device 200 and the learning device 200
The state defined by the existing environment and the desired state
If it becomes, the reward R₀Is a positive value (high rating
Value), and the reward R
₀Is a negative value (low evaluation value).

The value of the reward R ₀ may take two values, a positive predetermined value and a negative predetermined value, or may take a continuous value according to the degree of desirability of the state. Good.

The learning device 200, based on the reward R _0,
As the evaluation value is high, i.e. as positive reward is obtained much, adjusting the value of the variable a _2. This is called reward-based learning.

In the example shown in FIG. 1B, the input value,
Although the output value and the internal parameter are each assumed to be a single scalar quantity, the input value, the output value and the internal parameter may each be a plurality of scalar quantities or may be vector quantities. When the learning device 200 is a learning device employing a neural network, the internal parameter may be, for example, a connection weight of the neural network.
When the learning device 200 describes the input / output relationship by a set of IF-THEN rules, and the output value is determined by the weight coefficient set for each of the IF-THEN rules, the internal parameter is, for example, the IF-THEN rule. It may be a weight coefficient set for each.

The output value Y may be a value obtained by adding a component fluctuating at random to the value indicated by f ₂ (a ₂ , X). When the value of the output value Y includes such a component that fluctuates randomly, the value of the variable a ₂ is adjusted because the input value X does not change and the positive or negative reward P ₀ is not obtained. The output value Y can be fluctuated even in a situation where it is not possible. Thus, positive or negative compensation P ₀ be followed longer adjustment is not performed status value of the variable a ₂ not be obtained from the environment, that is, greater potential to avoid not proceed learning based on reward Become.

In the examples shown in FIGS. 1A and 1B, the learning device 100 and the learning device 200 receive values (input values) from the environment and output values (output values). However, the interface between each of the learning devices 100 and 200 and the environment may include any known components. For example, as an interface between the learning device 100 and the learning device 200 and the environment,
Any sensor can be used. The learning device 100 and the learning device 200 can receive an input value from the environment via a sensor. Alternatively, the learning device 100 and the learning device 200 may each receive a teacher signal and a reward via a sensor. In this case, the teacher signal is not explicitly given to the learning device 100, but may be obtained based on the human behavior by the learning device 100 observing the human behavior.

Any actuator can be used as an interface with the environment. The learning device 100 and the learning device 200 can perform an operation or an action corresponding to the output value on the environment by the actuator.

The fact that an arbitrary sensor and actuator can be interposed as an interface between the learning device and the environment, and that the learning device can receive a teacher signal and a reward via the sensor are the same as those in all embodiments described later. Applies commonly to devices.

An embodiment of the present invention will be described below with reference to FIGS.

(Embodiment 1) FIG. 2 shows a configuration of a learning device 250 according to Embodiment 1 of the present invention.

The learning device 250 generates an external output value 17 corresponding to the external input value 15. The learning device 250 receives a teacher signal and a reward from outside the learning device 250. The learning device 250 includes a learning unit 210 and a learning unit 1
9 and a learning switching unit 20.

The learning section 210 generates a learning section output value 211 corresponding to the external input value 15 as a whole. Learning part 2
10 is a learning designating unit 11 and a learning unit 12 based on a reward.
And a learning unit 13 based on a teacher signal, and an operation selecting unit 14.

The learning unit 19 outputs a learning unit output value 214 corresponding to the external input value 15 based on the internal parameters of the learning unit 19. The learning unit 19 outputs a learning unit output value 214.
Is set to be equal to the learning unit output value 211.
Adjust the values of the parameters inside 9. That is, the learning unit 19 learns the input / output relationship of the entire learning unit 210.

The learning switching unit 20 outputs the learning unit output value 211 as the external output value 17 when the learning by the learning unit 19 has not progressed, and outputs the learning unit output value 214 when the learning by the learning unit 19 progresses. Output as the external output value 17.

Hereinafter, the operation of the learning device 250 when solving a specific task will be described in detail taking a specific task as an example.

FIG. 3 shows a task used to explain the details of the operation of the learning device 250. The example shown in FIG. 3 is an example of a simplified ping-pong game. Reference numeral 25 indicates an environment in which a ping-pong game is executed, and is represented as an area in a two-dimensional xy plane. Environment 25 is 0
≤x≤1 and 0≤y≤1. Reference numeral 22 indicates a ball, and its coordinates are B (Bx, By). Reference numeral 23 indicates a controller controlled by the teacher, and reference numeral 24 indicates a controller controlled by the learning device 250. The coordinates of the controller 23 are P (P
x, Py), and the coordinates of the controller 24 are C (C
x, Cy). Here, the teacher is, for example, a person who knows the rules related to the task (simplified ping-pong game).

The ball 22 and the controllers 23 and 24
May each move within environment 25. However, the y-coordinate Py of the controller 23 and the y-coordinate Cy of the controller 24 are fixed to fixed values, and the controller 23 and the controller 24 are directed leftward or rightward (the direction in which the x coordinate decreases or the x coordinate increases). It is only possible to move.

When the ball 22 does not hit the wall (the boundary defining the range of the environment 25), the ball 22 keeps moving in a certain direction, and when hitting the wall, it reflects. When the ball 22 hits the lower wall (x-axis), it is determined whether or not the controller 24 is present in the vicinity of the point hit by the ball 22 within a predetermined distance. If the controller 24 is present in the vicinity within a predetermined distance, a positive reward “R +” is given to the learning device 250; otherwise, a negative reward “R−” is given to the learning device 250.

The controller 23 moves according to the operation signal Mp output by the teacher. The controller 24 moves according to the operation signal Mc output from the learning device 250.
Each of the operation signals Mp and Mc can take three values, “right”, “do not move”, and “left”. When the operation signal Mc is “left”, the controller 24 moves to the left (to the direction in which Cx decreases). When the operation signal Mc is “not move”, the controller 24 does not move.

The learning device 25 in the environment shown in FIG.
The task assigned to 0 is to learn how to operate the controller so that the cumulative value of the obtained rewards increases. The learning device 250 initially does not have knowledge of the rules of ball movement and the timing at which rewards are obtained. The learning device 250 determines an appropriate operation signal Mc (that is, the external output value 1) according to the coordinate B of the ball 22 and the coordinate C of the controller 24 (that is, the external input value 15).
7) is required. The teacher displays the controller 23 in the environment and performs a model operation,
It is assumed that the operation of the controller 24 by the learning device 250 can be watched without displaying the controller 23. The coordinates P of the controller 23 displayed in the environment are
It is input to the learning device 250.

Referring to FIG. 3 again, the operation of learning section 210 will be described in detail.

The reward-based learning unit 12 performs learning so that more positive rewards are given. The learning unit 12 based on a reward is, for example, an Elman-type neural network (document: JL Elman, Finding).
structure intime, Cognit
ave Science, 14, pp. 179-2
11). This neural network is a neural network that performs learning by the back propagation method.

The variables input to the neural network are the six variables of the coordinates B (Bx, By) of the ball 22, the coordinates P (Px, Py) of the controller 23, and the coordinates C (Cx, Cy) of the controller 24. The output variable is an operation signal Mc to the controller 24. Since the operation signal Mc can take three values, “right”, “do not move”, and “left”, the neural network has three output units (not shown) corresponding to the respective values of the operation signal. Is configured.

The neural network of the learning unit 12 based on the reward sets values corresponding to the input six variables to each of the three output units. The value set for each of the output units is calculated based on the coupling load. Of the values set for each output unit, the value of the operation signal corresponding to the output unit for which the largest value is set is adopted as the value of the operation signal Mc. For example,
When the values set in the output units corresponding to “right”, “do not move” and “left” are 0.2, 0.3 and 0.6, respectively, the value of the operation signal Mc is “left”. Adopted. This value “left” indicates the learning unit 12 based on the reward.
Is output to the operation selection unit 14 as the output value 212 of.

The connection weight of the neural network of the learning unit 12 based on the reward is initially set at random (in a state where learning has not progressed). For this reason, the output value 112 of the learning unit 12 based on the reward is also random at first. While the reward is not obtained, the neural network of the learning unit 12 based on the reward calculates the values set in the three output units according to the connection weight, and generates the output value 212. As described above, the generation of the output value 212 based on the randomly set connection weight is referred to as “trial and error”.
Call.

When performing such trial and error, a component that fluctuates randomly may be added to the output value 212 of the learning unit 12 based on the reward.

The learning unit 12 based on the reward operates the learning device 250
When a reward is received from outside of the network, the connection weight of the neural network is adjusted according to the back propagation method.

When a positive reward “R +” is given to the learning device 250 as a reward, learning is performed by a back propagation method in order to strengthen the operation at that time. For example, the values set for the units corresponding to “right”, “do not move” and “left” are 0.2, 0.
In the case of 3 and 0.6, “left” is used as the operation signal Mc.
Is generated as the output value 212. This output value 212
Is output to the outside of the learning device 250 as an external output value 17 when a certain condition is satisfied. That is, the controller 24 moves in the “left” direction. At this time, if a positive reward "R +" is given to the learning device 250, the neural network of the learning unit 12 based on the reward has 0.0 for "right", 0.0 for "not moving", and 1 for "left". The learning is performed based on the learning data of 0.0. Thereby, the connection weight of the neural network is adjusted so that “left” is easily output as the value of the operation signal for the same input as when the positive reward “R +” is given to the learning device 250.

The conditions for outputting the output value 212 to the outside of the learning device 250 as the external output value 17 are as follows.
The operation selection unit 14 selects the output value 212 of the learning unit based on the reward as the learning unit output value 211, and
Is that the learning unit output value 211 is selected as the external output value 17. Action selection unit 14 and learning switching unit 20
The operation of will be described later.

Contrary to the above example, if a negative reward “R−” is given to the learning device 250 when the controller 24 moves in the “left” direction, the learning device 250 Judge that there is. In the neural network of the learning unit 12 based on the reward, learning is performed based on learning data in which “right” is 0.8, “does not move” is 0.8, and “left” is 0.0. Thereby, the connection weight inside the neural network is set so that a value other than “left” is easily output as the value of the operation signal for the same input as when the negative reward “R−” is given to the learning device 250. Is adjusted.

By adjusting the connection weight as described above, when the learning device 250 outputs the output value 212 of the learning unit 12 based on the reward to the environment as the external output 17, the learning device 250 outputs the positive reward “ It is easier to obtain “R +” and it is difficult to obtain a negative reward “R−”. That is, the adjustment of the connection load is performed such that the evaluation value indicated by the reward increases.

As described above, the reward-based learning section 12 functions as a reward signal input section that receives a reward (reward signal) from outside the learning device 250, and the evaluation value indicated by the reward increases based on the reward. Functions as a first adjusting unit for adjusting the coupling load (at least one first parameter).

Further, the reward-based learning unit 12 executes learning based on the reward (first learning), so that the output value corresponding to the external input value 15 is determined based on the connection weight (at least one first parameter). It functions as a first learning unit that generates 212 (first output value).

FIG. 4 shows the configuration of the learning unit 13 based on the teacher signal. The learning unit 13 based on the teacher signal stores a teacher action storage unit 131 that stores the relationship between the teacher's action and the state of the environment.
And an imitation unit 132 based on storage for estimating what output value the teacher outputs in the current environment state based on the teacher's behavior stored in the teacher behavior storage unit; A direct imitation unit 133 that outputs the teacher's action itself as an output value regardless of the state.

Hereinafter, the components included in the learning unit 13 based on the teacher signal will be described.

The teacher action storage section 131 stores the relationship between the teacher action and the state of the environment. In the task of the ping-pong game shown in FIG.
The relationship between the coordinate B of No. 2 and the controller operation Mp of the teacher is stored. The teacher action storage unit 131 stores, for example, Elman
It is composed of a type of neural network. The variable input to this neural network is ball 2
There are four variables of a coordinate B (Bx, By) of No. 2 and a coordinate P (Px, Py) of the controller 23, and the output variable is an operation signal Mc for the controller 24. Since the operation signal Mc can take three values, “right”, “do not move”, and “left”, the neural network has three output units (not shown) corresponding to the respective values of the operation signal. ing. As learning data of the neural network in the teacher action storage unit 131, a set of four variables that are inputs of the neural network at a certain time and a teacher's operation signal Mp at that time is used.

The neural network of the teacher action storage unit 131 performs learning by adding new learning data at regular intervals when a learning specifying unit 11 described later specifies learning based on a teacher signal. . As a result, the Elman-type neural network in the teacher action storage unit 131 stores the ball coordinates B (Bx, By) and the operation Mp (“right”, “do not move”, or “left”) taken by the teacher at that time. ) Is stored. This corresponds to creating a model of the teacher's action inside the neural network, and the teacher's knowledge of the task is stored in the teacher action storage unit 131 in the form of an operation in a certain environment state.

The teacher action storage unit 131 executes learning by the back propagation method, so that the connection weight of the neural network is adjusted. The coordinates B of the ball 22 and the coordinates P of the controller 23 at a certain point in time
And an operation signal Mp indicating the action taken by the teacher accordingly
Of the ball 22 and the coordinate P of the controller 23 at that time, the operation signal Mc output from the neural network.
Is adjusted so as to approach the operation signal Mp indicating the action taken by the teacher.

At the stage where learning has progressed, the teacher action storage unit 1
In a case where the ball 22 and the controller 24 are at a certain position, the teacher 31 can calculate what operation the teacher performs.

The memory-based imitation unit 132 determines the value of the operation signal Mc based on the teacher's behavior stored by the teacher behavior storage unit 131. The variables input to the imitation unit 132 based on the memory are four variables of the coordinates B (Bx, By) of the ball and the coordinates C (Cx, Cy) of the controller 24,
The output variable is an operation signal M for the controller 24.
c (“right”, “does not move” or “left”).

The imitation unit 132 based on the memory stores the coordinates B (Bx, By) of the ball and the coordinates C (C
When (x, Cy) and (that is, the external input value 15) are input, the data is passed to the teacher action storage unit 131 as it is. In the teacher action storage unit 131, the received values of the coordinates B (Bx, By) of the ball and the received coordinates C (Cx, Cy) of the controller 24 are input to the neural network. At this time, the coordinates C (Cx, Cy) of the controller 24 received by the teacher action storage unit 131 are input to the neural network as the coordinates P (Px, Py) of the controller 23. An operation signal Mc which is a calculation result of the neural network of the teacher action storage unit 131
Is returned to the imitation unit 132 based on the memory. As the teacher action storage unit 131 adjusts the connection weight, the operation signal Mc, which is the calculation result of the neural network, comes closer to the teacher's operation signal Mp for the coordinates B of the ball and the coordinates C of the controller 24. That is, an appropriate operation signal Mc based on the teacher's action memory is generated as the output value 213 according to the state of the environment.

The memory-based imitation section 132 passes the coordinates B of the ball 22 and the coordinates C of the controller 24 (that is, the external input value 15) to the teacher action storage section 131, and receives the operation signal received from the teacher action storage section 131. Mc output value 2
13 is output.

The teacher operation signal Mp is the operation signal Mp.
Indicates the expected value of c (that is, the expected value of the external output value 17), and is input to the learning device 250 as a teacher signal.

The teacher action storage section 131 functions as a teacher signal input section that receives a teacher signal from outside the learning device 250, and also has an external input value of 15 based on the teacher signal.
The output value 213 corresponding to (the coordinate B of the ball and the coordinate C of the controller 24) functions as a second adjustment unit that adjusts the coupling load (at least one second parameter) so as to approach the expected value.

The learning section 13 based on the teacher signal stores the teacher action storage section 131 by executing the learning (second learning) based on the teacher signal by the internal teacher action storage section 131.
It functions as a second learning unit that generates an output value 213 (second output value) corresponding to the external input value 15 based on a connection weight (at least one second parameter) of the 31 neural networks.

The direct imitation unit 133 imitates the operation signal Mp by the teacher at a certain point in time, not based on the memory of the teacher as in the imitation unit 132 based on the memory. The direct imitation unit 133 receives the operation signal Mp by the teacher as an input and the value (“right”, “does not move” or “left”)
Is output as the operation signal Mc as it is. That is, the direct imitation unit 133 determines the position of the ball 22 and the controller 2.
Regardless of the position of 3, the operation signal Mp by the teacher is imitated as it is. For example, if the teacher operates the controller 23 to “left”, the direct imitation unit 133 outputs “left” as the operation signal Mc for the controller 24. The direct imitation unit 133 does not perform learning. Although the direct imitation unit 133 performs a simple imitation, when the environment is complicated or when the teacher operates the controller 23 while considering the position of the controller 24, the direct imitation unit 133 is performed.
Will also be appropriate for the environment. The operation of the direct imitation unit 133 is stored in the learning unit 19 described later. As a result, even when the teacher does not exist, the direct imitation unit 133 can be used.
(I.e., the teacher's operation) can be reproduced.

Note that the direct copying unit 133 may be omitted.

Next, the operation of the learning designating section 11 (FIG. 2) will be described. The learning unit 12 based on the reward and the learning unit 13 based on the teacher signal may not be able to learn at the same time. For this reason, the learning specifying unit 11 determines whether to specify learning based on a reward and whether to specify learning based on a teacher signal.

As a method of the learning designation by the learning designation unit 11, a method of determining the learning designation according to the state of the environment and a variable indicating the internal state of the learning device 250 is employed. In the first embodiment of the present invention, information on whether or not a teacher exists as an environmental state (whether or not a teacher signal is input) is used, and an attention parameter is used as a variable indicating an internal state. . The attention parameter is a parameter that controls how concentrated the learning device 250 performs learning. For example, the value of the attention parameter changes with time. In the following description, it is assumed that the attention parameter takes a value of 0 to 1, decreases over a certain period of time, and then increases over a certain period of time.

FIG. 5 shows a processing procedure for learning designation by the learning designating section 11.

Hereinafter, the processing procedure will be described for each of steps S11 to S15 shown in FIG.
The processing of steps S11 to S15 is executed by the learning specifying unit 11 at a predetermined timing, and the learning module (the learning unit 19,
Learning is performed in a learning unit 12 based on a reward and a learning unit 13) based on a teacher signal.

Step S11: Learning by the learning section 19 is designated. In the learning section 19, learning is always performed. The learning by the learning section 19 will be described later.

Step S12: It is determined whether or not a teacher operation is present, that is, whether or not a teacher signal is input. If this determination is "YES", the process proceeds to step S
Proceed to 13. If this determination is "NO", the process proceeds to step S14.

Step S13: Learning based on the teacher signal is specified. When this designation is made, the teacher action storage unit 131 executes learning based on the teacher signal.

Step S14: The attention parameter is higher than a predetermined value (for example, 0.5), and the output value of the learning unit 12 based on the reward is set as the external output value 17 as the learning device 2
It is determined whether or not the data is output outside the 50. If this determination is "YES", the process proceeds to step S15. If this determination is "NO", the process ends. Whether the output value of the learning unit 12 based on the reward is output to the outside of the learning device 250 as the external output value 17 is determined.
It can be performed by monitoring the selection operation by the operation selection unit 14 and the selection operation by the learning switching unit 20. Whether the condition that the operation selecting unit 14 selects the learning unit output value 212 based on the reward as the learning unit output value 211 and selects the learning unit output value 211 as the external output value 17 is satisfied. Whether the output value of the learning unit 12 based on the reward is an external output value 17 depends on whether
This is because it is determined whether or not the output is outside 0.

Step S15: Learning based on the reward is specified. When this designation is made, the reward-based learning unit 12 executes learning based on the reward.

As described above, the learning specifying unit 11 determines whether or not to specify learning (first learning) based on the reward signal by the learning unit 12 (first learning unit) based on the reward signal, and determines whether or not to specify the teacher signal. It is determined whether or not the learning (second learning) based on the teacher signal by the learning unit 13 (second learning unit) is designated.

The monitoring operation in step 14 is as follows.
It may be executed by the learning designating unit 11 or by the learning unit 12 based on the reward. Step 14
In Step 15, it is determined whether or not the learning in the learning unit 12 based on the reward is to be performed depending on whether the output value of the learning unit 12 based on the reward is output to the outside of the learning device 250 as the external output value 17. The reason for this is that the reward given to the learning device 250 is the learning unit 1 based on the reward.
This is to ensure that 2 is an evaluation value given for the generated output value.

The learning specifying unit 11 specifies learning according to the processing procedure described above, so that all data (teacher signals) from the teacher are used for learning, and when there is no teacher (when no teacher signal is input), By performing learning based on the reward, the learning device 250 can continue learning regardless of whether a teacher exists in the surrounding environment.

The operation selecting section 14 (FIG. 2) selects one of the output value 212 of the learning section 12 based on the reward and the output value 213 of the learning section 13 based on the teacher signal, and determines the selected one as the learning section. It is output to the learning switching unit 20 as an output value 211.
In Embodiment 1 of the present invention, as described with reference to FIG. 4, the learning unit 13 based on the teacher signal outputs the output value output from the imitation unit 132 based on the memory and the direct imitation unit 133
And two output values, ie, an output value output from the operation selection unit 14. Accordingly, the operation selecting unit 14 selects one of the three output values output from the learning unit 12 based on the reward, the imitation unit 132 based on the memory, and the direct imitation unit 133, and outputs the selected output value. Learning unit output value 211
Is output to the learning switching section 20. This selection is made based on whether or not a teacher exists and the value of the storage status parameter which is a parameter indicating the internal state. It is assumed that the storage status parameter takes a value of 0 to 1, is set to a low value at the beginning of learning, and increases as the learning time elapses. This is because it is considered that the learning device 250 cannot accurately reproduce the behavior of the teacher at the beginning of learning.

FIG. 6 shows a processing procedure of output selection by the operation selecting section 14.

Hereinafter, the processing procedure will be described for each of steps S21 to S25 shown in FIG.
The processing of steps S21 to S25 is executed by the operation selecting unit 14 at a predetermined timing.

Step S21: It is determined whether or not a teacher operation exists. Whether or not the operation of the teacher exists is equivalent to whether or not the controller 23 of the teacher exists in the environment. If this determination is "YES", the process proceeds to step S
22. If the determination is "NO", the process proceeds to step S
Proceed to 23.

Step S22: An output value output directly from the imitation unit 133 is selected.

Step S23: Whether the storage situation is bad or not
That is, the value of the storage status parameter is a predetermined value (for example,
0.5) is determined. If the determination is "YES", the process proceeds to step S24, and if the determination is "NO", the process proceeds to step S25.

Step S24: Learning unit 12 based on reward
The output value 212 output from is selected.

Step S25: The action based on the memory of the teacher, that is, the output value output from the imitation unit 132 based on the memory is selected.

Next, the operation of learning section 19 (FIG. 2) will be described. The learning unit 19 always learns the relationship between the positions of the ball 22 and the controller 24, which are inputs from the environment, and the operation signal Mc from the learning device 250. The learning section 19
For example, it is configured by an Elman-type neural network that executes learning according to the back propagation method. The input of the neural network is ball 2
2 coordinates B (Bx, By), coordinates P of the controller 23
(Px, Py) and the coordinates C (C
x, Cy), and the output variable is an operation signal Mc for the controller 24. Operation signal Mc
Can take three values, "right", "static" and "left", so that the neural network is configured to have three output units (not shown) corresponding to each value. The neural network of the learning unit 19 sets a value corresponding to the values of the six input variables to each of the three output units. The value set for each of the output units is calculated based on the coupling load.

When the learning unit 19 performs learning according to the back propagation method, a set of six variables which are inputs of the neural network at a certain time and a learning unit output value 211 at that time is used as learning data. . Thereby, the learning unit output value 214 output by the learning unit 19 is
The connection weight (at least one parameter) is adjusted so as to approach the learning unit output value 211. Since the learning unit output value 211 is an output value selected by the operation selection unit 14, the learning unit 19 outputs the external input value 15 (the ball 22
The relationship between the coordinate B of the controller 24 and the coordinate C) of the controller 24 and the output value output by any of the learning unit 12 based on reward, the imitation unit 132 based on storage, and the direct imitation unit 133 is learned. In the early stage of learning of the learning unit 19, since learning by each learning module has not progressed, the learning unit 1
9 mainly stores an action history of the learning device 250. At the stage where learning by each learning unit has advanced, an action appropriate for the environment is given to the learning unit 19 as learning data. At this stage, the learning unit 19 stores the learning level of the entire learning device 250 instead of the learning units.

The inputs of the neural network of the learning section 19 include the coordinates B (Bx, By) of the ball 22, the coordinates P (Px, Py) of the controller 23, and the controller 24.
Are the six variables of the coordinates C (Cx, Cy). These six variables include an external input value 15 (the coordinates of the ball 22 and the coordinates C of the controller 24). Therefore, the learning unit 19 generates the learning unit output value 214 according to the external input value 15 based on the connection weight (at least one parameter) of the neural network.

The learning switching section 20 (FIG. 2)
As the external output value 17 of 0, the output value of the learning unit 19 (learning unit output value 214) or the output value of the operation selection unit 14 (learning unit output value 211) is switched. When the output value of the learning unit 19 is selected as the external output value 17 of the learning device 250, each of the learning specifying unit 11, the learning unit 12 based on the reward, the learning unit 13 based on the teacher signal, and the operation selecting unit 14 Operation is stopped. This allows
When the learning section output value 214 is selected, the learning device 250
The necessary computation cost can be reduced as a whole.

The learning switching unit 20 selects an output value based on the attention parameter used by the learning designating unit 11 and the storage status parameter used by the action selecting unit 14. When the attention of the learning device 250 is low, the learning switching unit 20 sets the value of the attention parameter to 0.
2 or the storage status of the learning device 250 is good, for example, when the value of the storage status parameter is higher than 0.8, the learning unit output value 214 output from the learning unit 19 is changed to the external output value. 17; otherwise, the learning unit output value 21 output from the operation selection unit 14
1 is output as the external output value 17. When the memory condition is good, it is expected that the learning by the learning unit 19 is sufficiently advanced, and the learning device 25 is calculated only by the learning unit 19.
This is because it is determined that the external output value 17 of 0 can be determined.

Thus, learning switching section 20 selectively outputs one of the output value generated by operation selecting section 14 and the output value generated by learning section 19 as external output value 17 based on a predetermined rule. Output to

The amount of calculation required for the operation of the learning unit 19 is smaller than the total amount of calculation required for the operations of the learning specifying unit 11, the learning unit 12 based on the reward, the learning unit 13 based on the teacher signal, and the operation selecting unit 14. I'm done. Therefore, the learning device 250
Is equipped with the learning unit 19 and the learning switching unit 20, compared with the case where the learning unit 19 and the learning switching unit 20 are not provided, the output generation calculation based on the learning result requires less calculation cost (the space calculation amount and the time calculation time). Volume).

FIG. 7 shows the relationship between the input and output of the reward-based learning unit 12, the teacher action storage unit 131, the memory-based imitation unit 132, the direct imitation unit 133, and the learning unit 19. Each of the modules (each component included in the learning device) shown in the “module” column 701 of FIG. 7 generates an output value corresponding to a certain input value. Each module is
The variables shown in the “input” column 702 and the “output” column 703
In order to learn the correspondence relationship with the variables shown in (1), the variables shown in the “teacher” column 704 or the rewards (“R +” or “R−”) shown in the “reward” column 705 are used.

At the stage where the learning has progressed, even if all the variables shown in the “input” column 702 shown in FIG.
An appropriate value can be generated as a variable shown in the column 703. For example, the learning unit 12 and the learning unit 19 based on the reward can generate an appropriate value as the control signal Mc even when the coordinates P of the controller of the teacher are not input (there is no teacher).

A process in which the learning device 250 (FIG. 2) executes learning while executing the task of the ping-pong game (FIG. 3) will be described.

When there is no teacher, the learning device 250 uses positive and negative rewards to perform learning by the learning unit 12 based on rewards. At the initial stage of learning, the learning device 250 has no information on how to be rewarded. For this reason, when there is no teacher, the learning device 250 executes learning based on the reward, outputs a random operation signal Mc, and performs a trial-and-error positive reward “R
+ "And negative reward" R- ".

When the relationship between the timing at which a reward is given, the position of the ball 22, and the position of the controller 24 of the learning device 250 is learned by the reward-based learning section 12, a positive reward "R +" is obtained. , An operation signal Mc for controlling the controller 24 can be output.

However, it is known that such a reinforcement learning method using only rewards requires an enormous amount of time for learning. Therefore, the controller 23
The learning device 250 can observe the teacher's operation and store it in the learning unit 13 based on the teacher signal. Thus, the learning device 250 can directly associate the position of the ball 22 with the operation of the controller 23, and can execute effective learning.

As described above, in the learning device 250,
Learning unit 12 based on reward and learning unit 13 based on teacher signal
Thus, learning by each learning method proceeds.

The learning unit 19 learns the relationship between the external input value 15 and the learning unit output value 211 output by the learning unit 210. Thereby, the learning unit 19 stores both the learning result obtained by the learning device 250 by trial and error and the learning result taught by the teacher. At the stage where learning has progressed,
It is possible to generate correct behavior (appropriate output) for the environment only by the learning unit 19, and it is not necessary to operate the learning unit 12 based on rewards and the learning unit 13 based on teacher signals. Therefore, the calculation cost (the amount of space calculation and the amount of time calculation) can be reduced.

As described above, in an environment where a teacher (for example, a human) is present, when a reward and a teacher signal are given from the environment, the learning device 250 effectively uses both of them. The learning device 250 can execute learning based on the teacher signal while the teacher signal is obtained, and can autonomously execute learning based on the reward while the teacher signal is not obtained. The learning device 250 can execute learning regardless of whether or not a teacher signal is obtained, and can improve learning efficiency as compared with a case where learning is performed based only on the teacher signal.

Further, the learning device 250 can execute effective learning based on the teacher signal, and can improve learning efficiency as compared with the case where learning is performed based only on rewards.

Note that the learning unit 12, the teacher action storage unit 131, and the learning unit 19 based on the reward need to learn the input / output correspondence. In the above-described example, an Elman-type neural network is used as a configuration capable of learning the correspondence between input and output. However, the learning unit 12 based on the reward, the teacher action storage unit 131, and the learning unit 19
Is not limited to this. For example, for all or a part of the learning unit 12, the teacher action storage unit 131, and the learning unit 19 based on the reward, a configuration using another type of neural network is adopted, or a configuration not using a neural network (for example, IF-THEN format rule). Further, the invention is not limited to executing learning according to the back propagation method.

Further, in the above-described example, the action selecting unit 14 determines whether the learning unit 12 based on the reward is in accordance with the state of the environment (whether or not a teacher signal is input) or the internal state (the value of the storage state parameter). It has been determined whether to select the generated output value or to select the generated output value from the learning unit 13 based on the teacher signal. Instead of or in addition to such a criterion, the operation selecting unit 14 selects the output value generated by the learning unit 12 based on the reward based on the reward or the learning unit 13 based on the teacher signal. May determine whether to select the output value generated by. For example, the operation selecting unit 14 may be configured to determine whether the output value generated by the learning unit 12 based on the reward is output as the output value of the learning device 250 or the cumulative value of the reward from the environment or human, and the learning unit based on the teacher signal. The output value to be selected may be determined based on a comparison with the environment or the accumulated value of rewards from humans when the output value generated by the learning device 13 is output as the output value of the learning device 250. This allows the entire learning device 250 to perform an operation that maximizes the expected value of the reward for a certain task.

Note that the learning unit 19 and the learning switching unit 20 described above can be omitted.

FIG. 8 shows a configuration of a learning device 260 in which the learning unit 19 and the learning switching unit 20 are removed from the learning device 250. The learning device 260 includes the learning device 250 shown in FIG.
Has the same configuration as that of the learning unit 210. In FIG.
The same components as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted.

Since the learning device 260 does not have the learning switching unit 20, the output value output from the operation selecting unit 14 is directly output to the outside of the learning device 260. As described above, the operation selecting unit 14 of the learning device 260 sets the learning unit 12 based on the reward.
And an output value 213 of the learning unit 13 based on the teacher signal, and functions as an output unit that outputs the selected one as the external output value 17.

The learning unit 12 based on the reward of the learning device 260 monitors whether the operation selecting unit 14 is outputting the output value 212 or the output value 213 as the external output value 17, and 11, learning based on the reward is designated, and the operation selecting unit 14 outputs the output value 21.
2 is output, the learning unit 12 based on the reward
Performs learning based on rewards.

The learning device 260 does not have the advantage that the calculation cost can be reduced at the stage of the learning as compared with the learning device 250 (FIG. 2). However, the internal configuration of the learning device can be simplified, and There is an advantage that the production cost of the device can be reduced.

The learning device may have a voice recognition function to make it easier to obtain knowledge and teaching from a person having some knowledge about the environment.

FIG. 9 shows a learning apparatus 2 having a voice recognition function.
70 shows the configuration. The learning device 270 includes the learning device 250
Has a structure in which a voice interpreting unit 271 is added to FIG. 9, the same components as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted.

The speech interpreting section 271 includes a speech recognizing section 30 and a semantic interpreting section 31.

The speech recognition section 30 performs speech recognition processing on a speech signal input from outside of the learning apparatus 270 and uttered by a human, and indicates a phoneme string, a syllable string, or a word string as a result of the speech recognition processing. Output to the interpretation unit 31. Any configuration having a voice recognition processing function can be adopted for the voice recognition unit 30.

The semantic interpreting unit 31 receives the result of the speech recognition processing from the speech recognizing unit 30 and determines whether or not a human voice input indicates that the learning device imitates a human operation.
A determination is made as to whether the output value of the learning device is appropriate or not, or whether a desired output value of the learning device is specified, and the learning device is controlled according to each determination result. in this way,
The semantic interpretation unit 31 extracts semantic information based on the recognition result by the speech recognition unit 30.

As described above, according to the learning device 270, a person who has some knowledge about the environment can communicate his / her intention to the learning device with the voice 31.

FIG. 10 shows examples of words spoken by humans, meanings of the words interpreted by the meaning interpreting section 31, and operations to be performed by the learning device 270 at that time. Learning device 270
Is said by a human being to "mimic", the meaning of the word is an instruction for imitation, and the learning designation unit 11
Only the operation selection unit 14 directly selects the imitation unit 133 and outputs the same value as the value of the teacher signal. In this case, the semantic information extracted by the semantic interpretation unit 31 is information related to an operation instruction (command) to the learning device 270. Such semantic information is input to the learning device 270 via an instruction input unit (not shown) that receives an operation instruction for the learning device 270.

[0140] The learning device 270 receives "like" from a human.
In the case of saying “no”, the meaning of the word is evaluation, and is given as a reward after judging whether the reward is positive or negative.
In other words, “like” is determined to be a positive reward,
"No" is determined to be a negative reward. In this case, the semantic information extracted by the semantic interpretation unit 31 is information related to a reward signal for the learning device 270.

When the learning device 270 is said to be “right” or “left” by a human, the meaning of the word is a movement instruction, and the learning designating unit 11 makes the learning unit 13 learn based on the teacher signal. Learning is performed in the same manner as when a teacher signal is input. As the external output value 17 of the learning device 270, an operation signal as specified (“right” or “left”) is output. In this case, the semantic information extracted by the semantic interpretation unit 31 is information related to the teacher signal.

With such a configuration, the learning device 270 receives not only a human operation having some knowledge about the environment, but also changes the operation of the learning device 270 by a voice signal from a human, thereby further improving the learning device 270. It is possible to proceed with learning directly.

[0143] The learning device 270 makes a "like" by voice.
Since a reward such as “no” can be received, it is easy for a human to reward the learning device 270. For this reason, the learning device 270 can receive a reward on more occasions as compared with a case where a reward by voice cannot be received. As already mentioned, the reward is
Information given to the learning device when the state defined by the learning device and the environment in which the learning device exists is a specific state, and whether the output value output by the learning device to the environment is desirable. Represents. For this reason, the reward may be an evaluation value related to an output value output in the past by the learning device. If the opportunity to receive the reward is small, when the learning device receives the reward, the relationship between the reward and the output value output by the learning device tends to be unclear, and the learning may not directly proceed. Since the learning device 270 can receive a reward on more occasions,
It is easy to specify at which point the reward indicates the evaluation value regarding the output value output. This is because it is not necessary to go back so much to the previous output value. For this reason, the learning device 270 can proceed with the learning based on the reward more directly.

Further, since the learning device 270 can receive a teacher signal such as “right” or “left” by voice, it becomes easy for a human to supply the learning device 270 with the teacher signal. Therefore, the learning device 270 can receive the teacher signal on more occasions as compared with the case where the teacher signal cannot be received by voice, and can perform more appropriate learning.

In the example described above, the learning unit 12 based on the reward executes learning based on the reward, and the learning unit 13 based on the teacher signal executes learning based on the teacher signal. As a learning method for learning the input / output relationship, for example, self-organizing learning is also known. Learning designation section 1
1, operation selection unit 14, learning switching unit 20, and learning unit 19
According to the configuration using, the learning method is not limited to learning based on reward and learning based on a teacher signal,
It is possible to realize a learning device integrating different learning methods.

FIG. 11 shows a configuration of a learning device 280 in which three different learning methods are integrated. 11, the same components as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted. Learning device 28
0 is the learning unit 1 based on the reward of the learning device 250 (FIG. 2).
2 and learning modules 21A to 21C instead of the learning unit 13 based on the teacher signal. The learning device 28
0 includes a learning specifying unit 281 instead of the learning specifying unit 11 of the learning device 250, and includes an operation selecting unit 282 instead of the operation selecting unit 14 of the learning device 250.

The learning modules 21A to 21C execute the learning method 1, the learning method 2, and the learning method 3, respectively. Each of the learning methods 1 to 3 may be, for example, learning based on reward, learning based on a teacher signal, or self-organizing learning. Each of the learning modules 21A to 21C generates a module output value 283A to 283C corresponding to the external input value 15 by executing the learning.

The learning designating section 281 is provided with a plurality of learning modules 21A to 21C based on a predetermined rule (first rule).
It is determined whether or not learning by each of the above is designated.

[0149] The operation selecting section 282 also determines a plurality of learning modules 21A based on a predetermined rule (second rule).
-21C, and selectively outputs one of the plurality of module output values as the first output value.

According to the learning device 280 configured as described above, the learning unit 19 includes a plurality of learning modules 21A to 21A.
The input / output relationship of a part (learning unit 285) including the 21C, the learning designating unit 281 and the operation selecting unit 282 is learned. Therefore, after the learning of the learning unit 19 is sufficiently advanced, the external output value 17
It is possible to decide. The result of the learning by the learning unit 19 is a result obtained by removing the redundant calculations by the plurality of learning modules 21A to 21C and the operation selecting unit 282, and integrating the calculations in the respective learning modules. For this reason, the amount of calculation necessary for calculating the external output value 17 depends on the plurality of learning modules 21A to 21C and the operation selecting unit 28.
The number of calculations required for the operation 2 is smaller than the total. Therefore, the learning device 280 includes the learning unit 19 and the learning switching unit 20, and thus requires less calculation to generate the output value based on the learning result as compared with the case where the learning unit 19 and the learning switching unit 20 are not provided. It can be executed at a cost (a space calculation amount and a time calculation amount).

Each of the learning modules 21A to 21C can be constituted by, for example, an Elman type neural network, and can execute learning according to a back propagation method. However, the configuration of the learning modules 21A to 21C is not limited to this. For example, the learning module 21
A learning system using another type of neural network may be used for all or a part of A to 21C, or a learning system not using a neural network (for example, an IF-THEN type rule) may be used. . Further, the invention is not limited to executing learning according to the back propagation method.

In the example shown in FIG. 11, the number of the plurality of learning modules is 3 (21A to 21C).
The number of learning modules is not limited to this. The learning unit 19 and the learning switching unit 20 can be omitted.

The plurality of learning modules may include a plurality of modules having the same learning method.

(Embodiment 2) In Embodiment 1, the learning unit based on the reward and the learning unit based on the teacher signal perform learning separately. Each of the learning unit based on the reward and the learning unit based on the teacher signal expresses the knowledge acquired by the learning using internal parameters. For example, when a learning unit based on a reward and a learning unit based on a teacher signal are configured by a neural network, knowledge is expressed as a connection weight.

If the learning unit based on the reward is the same as the learning unit based on the teacher signal, the knowledge can be shared. The case where the method of expressing knowledge is the same means that, for example, the learning unit based on the reward and the learning unit based on the teacher signal are each configured by a neural network, have the same input / output relationship, and have the same number of intermediate layers and the number of connections. If they are the same, the knowledge expressed as connection weights can be interchanged. In this case, when the learning method is switched, the knowledge is copied between the learning unit based on the reward and the learning unit based on the teacher signal, so that the knowledge is inherited from one learning module to the other learning module. It becomes possible.

FIGS. 12 (a) to 12 (f) show the learning between the learning module executing the learning method A and the learning module executing the learning method B, respectively, in the case where the knowledge is inherited and in the case where the knowledge is not inherited. The time change of the degree is shown. Here, the learning level is a value indicating which range of input values the learning module can generate a desired output value, and is defined by, for example, a validity rate.

FIGS. 12A to 12C show that the learning module A that executes the learning method A and the learning module B that executes the learning method B have the same knowledge expression method, and that the knowledge inheritance ( (Exchange of learning results).

FIGS. 12 (d) to 12 (f) show the change over time of the learning degree between the learning module A and the learning module B when the knowledge is not inherited.

Each of the learning method A and the learning method B can be, for example, learning based on a reward and learning based on a teacher signal. The learning module A and the learning module B do not perform learning at the same time.

First, referring to FIGS. 12A to 12C,
A description will be given of the time change of the learning degree when the knowledge is inherited. FIGS. 12A to 12C respectively show times t1 to t3.
5 shows the learning degrees of the learning module A and the learning module B in FIG. Here, t1 <t2 <t3. Time t1
In the learning of the learning module A is a _1, the learning of the learning module B is assumed to be 0 (FIG. 12
(A)). The time between the time t1~ time t2, performed the learning by the learning module B, the learning module B in this time it is assumed that the learning degree b ₁ increases. Prior to learning by the learning module B, the knowledge of the learning module A is copied to the learning module B.
Has a learning degree a ₁ + b ₁ at time t2 (FIG. 12)
(B)). Further, the learning by the learning module A is performed during the time between the time t2 and the time t3, and the learning degree of the learning module A increases by a ₂ during this time. Since the knowledge of the learning module B is copied to the learning module A prior to the learning by the learning module B, the learning module A has a learning degree a ₁ + b ₁ + a ₂ at time t3 (FIG. 12C).

Next, referring to FIGS. 12D to 12F,
A description will be given of a time change of the learning degree when the knowledge is not inherited. FIGS. 12D to 12F respectively show times t1 to t
3 shows the learning degree of learning module A and learning module B. Here, the times t1 to t3 are respectively shown in FIG.
It is the same as the times t1 to t3 shown in 2 (a) to (c). At time t1, the learning degree of the learning module A is a
_1, the learning of the learning module B is assumed to be 0 (FIG. 12 (d)). As in the example described with reference to FIGS. 12A to 12C, at the time between time t1 and time t2,
Learning module B by the learning is performed, the learning module B in this time it is assumed that the learning degree b ₁ increases. Learning module B has a learning level b ₁ at time t2 (FIG. 12 (e)). Further, the learning by the learning module A is performed during the time between the time t2 and the time t3, and the learning degree of the learning module A increases by a ₂ during this time. The learning module A learns a ₁ + at time t3.
having a ₂ (FIG. 12 (f)).

The examples shown in FIGS. 12A to 12C (in the case where knowledge is inherited) and FIGS.
Comparing the example shown in (f) (when knowledge is not inherited), the learning degree of each learning module is higher in the case where knowledge is inherited than in the case where knowledge is not inherited. It turns out that it becomes.

The learning apparatus according to the second embodiment of the present invention has a configuration in which knowledge can be inherited between learning modules executing different learning methods.

In the learning apparatus 250 (FIG. 2) of the first embodiment, the knowledge acquired by the learning based on the reward and the knowledge acquired by the learning based on the teacher signal are integrated in one learning module. A part 19 was provided. On the other hand, in the learning device according to the second embodiment of the present invention, both the knowledge acquired by the learning based on the reward and the knowledge acquired by the learning based on the teacher signal are determined by the learning unit based on the reward and the teacher. It is stored in each of the learning units based on the signal. Therefore, there is no need to provide a learning section in the first embodiment.

FIG. 13 shows a learning apparatus 310 in which knowledge can be inherited between learning modules executing different learning methods.
Is shown. The learning device 310 includes an input unit 311 that receives an external input value 15 from the environment, and a learning designation unit 312.
A learning unit (learning unit based on a reward) 313 that performs learning based on a reward, a learning unit (a learning unit based on a teacher signal) 314 that performs learning based on a teacher signal, and an output unit 31
5 is provided.

The learning device 310 receives a teacher signal and a reward from outside the learning device 310.

The learning unit 313 based on the reward, for example,
It is an lman-type neural network that executes learning based on rewards according to a back propagation method. The reward-based learning unit 313 receives an external input value 15 input to the learning device 310 from outside the learning device 310 via the input unit 311 and outputs an output value 323 according to the external input value 15.

As described above, the learning unit 313 based on rewards
Performs learning (first learning) based on a reward (reward signal), and based on a connection weight (at least one first parameter) inside the neural network, outputs an output value 323 ( It functions as a first learning unit that outputs a (first output value).

The reward-based learning unit 313 functions as a reward signal input unit that receives a reward (reward signal) indicating an evaluation value related to the external output value 17 from the outside of the learning device 310, and also, based on the reward. , Functions as a first adjustment unit that adjusts the connection weight (at least one first parameter) inside the neural network so that the evaluation value becomes higher.

The learning unit 314 based on the teacher signal is, for example, an Elman-type neural network, and executes learning based on the teacher signal according to the back propagation method. A learning unit 314 based on a teacher signal (for example, an input / output pattern performed by a human on the environment) receives an external input value 15 input to the learning device 310 from outside the learning device 310 via the input unit 311, and Input value 1
An output value 323 corresponding to 5 is output.

As described above, the learning unit 31 based on the teacher signal
4 executes learning based on a teacher signal (second learning), and based on a connection weight (at least one second parameter) inside the neural network, an output value 324 (second learning) corresponding to the external input value 15 It functions as a second learning unit that outputs an output value.

The learning unit 314 based on the teacher signal
At the same time as functioning as a teacher signal input unit that receives a teacher signal indicating the expected value of the external output value 17 with respect to the external input value 15 from outside the learning device 310, based on the teacher signal,
It functions as a second adjustment unit that adjusts the connection weight (at least one first parameter) inside the neural network so that the output value 324 for the external input value 15 approaches the expected value.

The learning designating section 312 alternatively designates learning based on rewards and learning based on teacher signals. This determination operation is performed based on, for example, whether or not a teacher exists in the environment around the learning device 310 (a teacher signal is input to the learning device 310). When there is a teacher, the learning specifying unit 312 specifies learning based on the teacher signal, and when no teacher exists in the environment around the learning device 310, the learning specifying unit 31
2 specifies learning based on reward.

The output unit 315 is provided for the learning unit 31 based on the reward.
One of the output value 323 (first output value) output by the learning unit 3 and the output value 324 (second output value) output by the learning unit 314 based on the teacher signal is selectively output as the external output value 17. Which of the output value 323 and the output value 324 the output unit 315 outputs as the external output value 17 is determined, for example, by a parameter indicating the internal state of the learning device 310 (for example, a storage status parameter that changes according to time). ).

Alternatively, the output unit 315 outputs the learning designation unit 3
The output value 323 may be output when 12 specifies learning based on a reward, and the output value 324 may be output when the learning specifying unit 312 specifies learning based on a teacher signal. . In the learning device 310, the knowledge acquired by the learning based on the reward and the knowledge acquired by the learning based on the teacher signal are both inherited by the respective learning modules. This is because it is considered that the learning module (the learning unit 313 based on the reward or the learning unit 314 based on the teacher signal) can always generate an output value appropriate for the environment.

The learning unit 313 based on the reward outputs the output unit 31
5 monitors whether it is outputting the output value 323 or the output value 324. Learning based on reward (first learning) is specified by the learning specifying unit 312, and the output unit 315 outputs the output value 323 (that is, the output value output by the learning unit 313 based on reward: first output value). When outputting, the learning unit 313 (first learning unit) based on the reward executes learning (first learning) based on the reward. Thus, the reason that the learning unit 313 based on the reward monitors the output of the output unit 315 is that the reward given to the learning device 310 is an evaluation value given with respect to the output value generated by the learning unit 313 based on the reward. This is to ensure that there is.
However, if learning based on a reward (first learning) is specified by the learning specifying unit 312, the output unit 315 outputs the output value 323. If the output of the output unit 315 by the learning unit 313 based on the reward is specified. Monitoring can be omitted.

The learning section 314 (second learning section) based on the teacher signal, when learning based on reward (first learning) is specified by the learning specifying section 312, learning based on the teacher signal (first learning). Execute

The learning unit 313 based on the reward and the learning unit 314 based on the teacher signal share a method of expressing the learning result obtained by learning. Therefore, between the learning unit 313 based on the reward and the learning unit 314 based on the teacher signal, the learning result can be exchanged (knowledge inheritance). When the alternative learning designation by the learning designation unit 312 is switched, the learning result is exchanged between the learning unit 313 based on the reward and the learning unit 314 based on the teacher signal.

For example, the state in which the learning specifying unit 312 specifies learning based on a reward (first learning) corresponds to the learning specifying unit 312.
When the learning state transitions to a state in which the learning signal 12 designates learning (second learning) based on the teacher signal, the learning unit 3 based on the teacher signal
A learning unit 313 based on a reward is added to the connection weights (at least one second parameter) of the 14 neural networks.
Are copied (at least one first parameter).

Conversely, the state in which the learning specifying unit 312 specifies learning based on the teacher signal (second learning) is changed to the state in which the learning specifying unit 312 specifies learning based on reward (first learning). When the transition has occurred, the connection weight of the neural network of the learning unit 314 based on the teacher signal is copied to the connection weight of the neural network of the learning unit 313 based on the reward.

Alternatively, when learning is switched,
Not a complete copy is made between the connection weight of the neural network of the learning unit 314 based on the teacher signal and the connection weight of the neural network of the learning unit 313 based on the reward, but a part of the connection weight is copied. You may do so. The connection weight of the neural network on which learning is to be performed may be adjusted based on the connection weight of the neural network on which learning has been completed.

With the above configuration, the learning device 310 includes the learning unit 313 based on the reward and the learning unit 3 based on the teacher signal.
The learning can be performed while passing the knowledge to and from the P.14, and the learning efficiency can be improved.

The learning unit 313 based on the reward and the learning unit 314 based on the teacher signal used an Elman type neural network. However, the configurations of the learning unit 313 based on the reward and the learning unit 314 based on the teacher signal are not limited thereto. For example, learning unit 3 based on reward
13 and a learning system using a neural network of another format as the learning unit 314 based on the teacher signal, or a learning system not using a neural network (for example, an IF-THEN format rule). Is also good. Further, the invention is not limited to executing learning according to the back propagation method. Learning unit 313 based on reward and learning unit 31 based on teacher signal
It is only necessary that the exchange of the learning result is possible with the program No. 4. That is, the learning unit 313 based on the reward and the learning unit 314 based on the teacher signal can inherit knowledge as long as the configuration can adjust one internal parameter based on the other internal parameter.

The learning device 310 shown in FIG. 13 has a learning unit based on a reward and a learning unit based on a teacher signal, and inherits knowledge between them. by this,
The learning is performed such that the knowledge acquired by the learning based on the reward and the knowledge acquired by the learning based on the teacher signal are mutually accumulated. The configuration of the learning device for realizing learning in which the knowledge acquired by the learning based on the reward and the knowledge acquired by the learning based on the teacher signal are mutually accumulated includes a learning unit based on the reward and a teacher signal. A configuration having a single neural network without having a learning unit based on the network may be adopted.

FIG. 14 shows a learning method for realizing learning in which knowledge acquired by learning based on a reward and knowledge acquired by learning based on a teacher signal are mutually accumulated using a single neural network. Device 320
Is shown.

The learning device 320 executes learning and generates an external output value 17 corresponding to an external input value 15 input from outside the learning device 320;
2 includes a learning pattern generation unit 324 that generates a learning pattern used for learning by the learning unit 2 and a short-term storage unit 325 that temporarily stores input values and output values of the adjustment unit 322.

The adjusting unit 322 is constituted by an Elman type neural network, and performs learning according to the back propagation method.

The learning pattern generation unit 324 receives a reward and a teacher signal from outside the learning device 320.
The teacher signal is, for example, an input / output pattern indicating the relationship between input and output performed by a human on the environment. Here, the input / output pattern means a set of an input value and an output value. The input / output pattern to the human environment is a set of the state of the environment detected by the human (input value input from the environment to the human) and the output value that the human outputs to the environment according to the state of the environment. is there. Since humans have some knowledge of the environment, the output values that humans output to the environment are considered to be output values appropriate for the environment. Therefore, the learning device 320
Also, it is expected that the same value as the output value output by a human for the state of the environment is output to the environment. The output value output by the human to the environment can be used as a teacher signal input from outside the learning device 320. The input / output pattern to the human environment includes a teacher signal.

The input / output pattern to the human environment is
The information may be input to the learning device 320 by a human operating an arbitrary input device (not shown). Or
The learning device 320 may have a function of determining whether or not a human is present around the learning device 320 and, when a human is present, a function of detecting an input / output pattern to a human environment. Such a function is realized by a known configuration such as an image processing system that monitors the periphery of the learning device 320, for example.

FIG. 15 shows a processing procedure in which the learning device 320 executes learning. This processing procedure is performed by the learning device 320
However, this is a processing procedure when all human actions (input / output patterns to the environment) are used as teacher signals whenever a human is present around the learning device 320. Hereinafter, FIG.
Will be described.

Step S31: The external input value 15 is input to the learning device 320 from the environment (outside the learning device 320).

Step S32: The adjusting unit 322 performs
An external output value 17 corresponding to the external input value 15 is generated.

Step S33: The set of the external input value 15 input to the learning device 320 in step S31 and the external output value 17 generated in step S32 according to the external input value 15 is a short-term input / output relationship. It is stored in the storage unit 325.

Step S34: The external output value 17 generated in step S32 is output to the environment.

Step S35: It is determined whether or not a human exists around the learning device 320. If this determination is "YES", the process proceeds to step S36. If this determination is "NO", the process proceeds to step S38.

Step S36: A learning pattern is generated based on an input / output pattern (teacher signal) to / from a human environment. This learning pattern generation processing is executed by the learning pattern generation unit 324. Such a learning pattern is expressed as a set of a specific external input value 15 and an expected value of an external output value 17 for the specific external input value 15.

Step S37: Learning by the adjusting unit 322 is performed based on the learning pattern generated in step S36. This learning is performed by, for example, a back propagation method. In the back propagation method, the value of the coupling weight (at least one parameter) of the adjustment unit 322 is adjusted such that the external output value 17 for the specific external input value 15 approaches the expected value.

Thus, in steps S36 and S37, learning device 320 executes learning based on the teacher signal.

Step S38: It is determined whether or not a positive reward has been input from the environment to the learning device 320. If this determination is "YES", the process proceeds to step S39. If this determination is "NO", the process proceeds to step S40.
Proceed to.

Step S39: Based on the input / output relationship stored in the short-term storage unit 325 in step S33,
A learning pattern (learning data) is generated. This learning pattern generation processing is executed by the learning pattern generation unit 324. Such a learning pattern represents a set of a specific external input value 15 and a specific external output value 17 output according to the specific external input value 15. It is considered that the learning device 320 has obtained a positive reward in connection with generating the specific external output value 17 for the specific external input value 15.

Step S40: It is determined whether or not a negative reward has been input from the environment to the learning device 320. If this determination is "YES", the process proceeds to step S39. If this determination is "NO", the process proceeds to step S40.
Proceed to.

Step S41: A learning pattern is generated by changing the output portion of the input / output relation stored in the short-term storage section 325 in step S33 according to a predetermined rule. The process of generating the learning pattern
This is executed by the learning pattern generation unit 324. Such a learning pattern represents a set of a specific external input value 15 and a value obtained by changing a specific external output value 17 output according to the specific external input value 15 according to a predetermined rule. It is considered that the learning device 320 has obtained a negative reward in connection with generating the specific external output value 17 for the specific external input value 15.

Step S42: Based on the learning pattern generated in step S39 or S42, learning by the adjusting unit 322 is executed. This learning is performed by, for example, a back propagation method. In the back propagation method, a specific external input value 15 generated in response to a specific external input value 15 when a positive reward is obtained is obtained by adjusting the coupling weight of the adjusting unit 322. It becomes easy to generate, and it becomes difficult to generate a specific external output value 17 generated in response to a specific external input value 15 when a negative reward is obtained. That is, the connection weight is adjusted such that the evaluation value related to the external output value 17 indicated by the reward increases.

As described above, steps S38 to S38
At 42, the learning device 320 performs learning based on the reward.

The learning pattern generator 324 determines in step S3
9 and the learning pattern generated in step S41 are generated when a positive reward is obtained and when a negative reward is obtained. As described above, the learning pattern generation unit 324 performs steps S39 and S41.
, A learning pattern (first learning pattern) based on the reward is generated. In step S36, the learning pattern generation unit 324 generates a learning pattern (second learning pattern) based on the teacher signal.

In the processing procedure shown in FIG. 15, the learning device 320 determines whether or not a human is present around the learning device 320, and when a human is present around the learning device 320, the learning pattern generation is performed. The unit 324 always generates the second learning pattern based on the input / output pattern to the human environment,
If no human is present, a first learning pattern is generated based on the reward signal.

Alternatively, the learning device 320 is
The learning pattern generation unit 324 is configured not to have a function of determining whether or not a human exists around 0, and an input / output pattern (teacher signal) to a human environment is input to the learning pattern generation unit 324. In such a case, the second learning pattern may be always generated based on the teacher signal. If the teacher signal is not input to the learning pattern generation unit 324, the first learning pattern may be generated based on the reward.

As described above, the process of using all available teacher signals is particularly effective at a stage where learning of the learning device 320 has not progressed (early learning stage). This is because the learning speed can be increased by using all available teacher signals.

In step S37, the adjusting unit 322 adjusts the connection weight so that the evaluation value indicated by the reward becomes higher based on the first learning pattern. In step S42, the controller 322 adjusts the connection weight based on the second learning pattern. It functions as an adjustment unit that adjusts the connection weight so that the external output value 17 with respect to the input value 15 approaches the expected value indicated by the teacher signal.

[0210] In the learning device 320, both the learning result obtained by the learning based on the reward and the learning result obtained by the learning based on the teacher signal are accumulated in the adjusting unit 322. It is not desirable that the learning result obtained by the learning based on the reward and the learning result obtained by the learning based on the teacher signal give conflicting results. As a result of the learning device 320 learning based on the reward,
It is not desirable that the external output value 17 for the external input value 15 depart from the expected value indicated by the teacher signal.
Further, it is not desirable that the evaluation value indicated by the reward be low as a result of learning performed by the learning device 320 based on the teacher signal.

For this reason, the first learning pattern is generated by the adjusting unit 3
22 adjusts the connection weight based on the first learning pattern (when the learning based on the reward is executed) such that the external output value 17 with respect to the external input value 15 approaches the expected value indicated by the teacher signal. Preferably, it is configured. Similarly, when the adjustment unit 322 adjusts the connection weight based on the second learning pattern (when the learning based on the teacher signal is performed), the evaluation value indicated by the reward increases in the second learning pattern. It is preferable that it is comprised as follows.

In order to avoid an undesired state in which the learning result obtained by the learning based on the reward and the learning result obtained by the learning based on the teacher signal cause conflicting results, the learning device 320 must It is also effective to judge the quality of the learning and control the learning method based on the quality of the teacher signal. For example, when the quality of the teacher signal is low, learning based on the teacher signal is not performed.
The learning method is controlled.

FIG. 16 shows a processing procedure in which the learning device 320 determines the quality of the teacher signal and controls the learning method based on the quality of the teacher signal. 16, steps that are the same as the steps shown in FIG. 15 are given the same reference numerals, and a description thereof will be omitted. Hereinafter, each step of the processing procedure shown in FIG. 16 will be described.

Step S55: The input / output relationship to the human environment is obtained. The input / output relationship to the human environment can be acquired by a known configuration such as an image processing system that monitors the periphery of the learning device 320, for example.

Step S56: It is determined whether or not a reward has been input from the environment to the learning device 320. If this determination is "YES", the process proceeds to step S57.
If this determination is "NO", the process proceeds to step S58.

Step S57: The cumulative reward value of the learning device 320 is calculated by integrating the rewards input to the learning device 320. This accumulation is performed over a predetermined period, and rewards input to the learning device 320 before the predetermined period are excluded from the accumulated reward value.

Step S58: It is determined whether or not a human exists around the learning device 320. If this determination is "YES", the process proceeds to step S59. If this determination is "NO", the process proceeds to step S63.

Step S59: It is determined whether a human has been rewarded from the environment. The determination as to whether or not a human is being rewarded from the environment is performed, for example, by using an arbitrary sensor. This judgment is "YE
If "S", the process proceeds to step S60. If this determination is "NO", the process proceeds to step S61.

Step S60: The cumulative reward value of the human is calculated by integrating the reward given by the human from the environment. This accumulation is performed over a predetermined period, and the reward given to the human by the environment before the predetermined period is excluded from the accumulated reward value. This predetermined period is a period equal to the predetermined period in step S57.

Step S61: The cumulative reward value of the learning device 320 is compared with the cumulative reward value of the human, and it is determined whether or not the cumulative reward value of the human is higher than the cumulative reward value of the learning device 320. If this determination is "YES", the process proceeds to step S62. If this determination is "NO", the process proceeds to step S63. When the determination is “YES”, it means that the quality of the teacher signal given by a human is determined to be high, and when the determination is “NO”,
This means that the quality of the teacher signal given by a human is determined to be low. Such a determination process is performed by the learning pattern generation unit 324 of the learning device 320.
The learning pattern generation unit 324 functions as a determination unit that determines the quality of the teacher signal based on a predetermined criterion in step S61.

Step S62: Learning based on the teacher signal is executed. The content of the process in step S62 is the same as the content of the process in steps S36 and S37 described with reference to FIG. That is, when it is determined in step S61 that the quality of the teacher signal is high, the learning pattern generation unit 324 generates a learning pattern (second learning pattern) based on the teacher signal.

Step S63: Learning based on the reward is executed. The content of the processing in step S63 is shown in FIG.
Are the same as the contents of the processing in steps S38 to S42 described with reference to FIG. That is, when it is determined in step S61 that the quality of the teacher signal is low, the learning pattern generation unit 324 does not generate a learning pattern (second learning pattern) based on the teacher signal. Human behavior is ignored.

In the processing procedure shown in FIG. 16, the quality of the teacher signal is determined based on the reward of the person (teacher signal generation source) who gives the teacher signal. The criterion for determining the quality of the teacher signal is not limited to this. For example, if the cumulative value of rewards input from the environment to the learning device 320 decreases even though learning based on the teacher signal is continued for a certain period, the quality of the teacher signal is determined to be low. You may.

As described above, in the process in which the learning device 320 determines the quality of the teacher signal and controls the learning method based on the quality of the teacher signal, the learning of the learning device 320 progresses, and the action (external This is particularly effective at the stage when the output value 17 can be output. This is because unnecessary imitation of a human can be avoided.

In the example described above, the adjustment unit 322
El performing learning according to the back propagation method
It is assumed to be constituted by a man-type neural network. However, the configuration of the adjustment unit 322 is not limited to this. For example, as the adjustment unit 322, a learning system using another type of neural network may be employed, or a learning system not using a neural network (for example, an IF-THEN format rule) may be employed. Further, the invention is not limited to executing learning according to the back propagation method.

Further, the speech recognition function described with reference to FIG. 9 may be added to the learning device 310 or 320. When the speech recognition function is added to the learning device 310 or 320, the chance that the learning device 310 or 320 receives the teacher signal and the reward signal is increased. Therefore, the learning device 310 or 320 can efficiently perform learning.

(Embodiment 3) In Embodiment 3 of the present invention, integration of learning based on teacher signals and learning based on rewards is realized by a learning device employing a classifier system.

A classifier system (classifier system) is a learning system in which an output method for an environment is described by a set of classifiers as described in the following document.

Reference: David E. Goldber
g, Genetic Algorithm in Se
arch, Optimization, and M
Achievement Learning, Chapter 6,
Addison Wesley, 1989.

FIG. 17 shows the configuration of a learning device 400 using a classifier system. The learning device 400 includes a classifier system 40 and a classifier generation unit 41 based on a teacher signal.

The classifier system 40 performs a process of generating an external output value 44 according to the external input value 42. The classifier system 40 generates a new classifier 47 based on a reward and a set 47 of classifiers indicating rules defining a first output value with respect to the external input value 42, and classifies the new classifier into a set of classifiers. Classifier generating unit 46 based on the reward to be added to 47
(A first generation unit), a setting unit 45 that sets the reliability of each of the classifiers 401 included in the classifier set 47, a first output value output from each of the classifiers 401, and a classifier 401. And an output unit 48 for generating the external output value 44 based on the reliability set for each of the above.

The classifier system 40 uses the above David
E. FIG. As disclosed in the Goldberg document, by creating a new classifier (first new classifier) based on the reward or changing the reliability set for the classifier 401, This is a system that performs learning so that the indicated evaluation value increases. That is, the classifier system 40 executes learning based on a reward input from outside the learning device 400.

The classifier generation unit 41 based on the teacher signal creates a new classifier (second new classifier) based on the teacher signal input from outside the learning device 400, and generates a new classifier. Is added to the set 47 of classifiers.

When the classifier system 40 executes learning based on rewards, it mainly uses a genetic algorithm (Da
vid E. Chapter of Goldberg literature
r1) is adopted. However, according to the genetic algorithm, a large amount of calculation is required until an appropriate new classifier is generated. This is because the new classifier processing is mainly performed based on an operation between the classifiers and a randomly generated value. For this reason, according to the method of adding a classifier by the genetic algorithm, it is difficult to obtain an appropriate classifier in a short time when the environment changes.

As described above, there is a problem that the ability to follow the environment is insufficient only by the learning based on the reward by the classifier system 40.

Learning device 400 according to Embodiment 3 of the present invention
Is provided with a classifier system 40 and a classifier generation unit 41 based on a teacher signal, so that learning based on a teacher signal input from outside the learning device 400 can be performed. According to the learning device 400, when a human having some knowledge about the environment is present around the learning device 400, an appropriate new classifier is created based on a teacher signal given by the human. Classifiers can be generated easily. Alternatively, the learning device 400 may have a function of observing the behavior of the surrounding humans, and may generate a new classifier from the human behavior at a certain point in time and the state of the environment.

The setting unit 45 for setting the reliability for each of the classifiers 401 may set a high reliability for a new classifier generated based on the teacher signal.
By setting a high degree of reliability to a new classifier generated based on a teacher signal, the first output value generated by the new classifier greatly affects the external output value 44. Also, when performing learning using a genetic algorithm, it is possible to reduce the search space and calculation time by generating a new classifier by giving priority to the classifier generated based on the teacher signal. become.

As described above, a new classifier generated based on the reward signal and a new classifier generated based on the teacher signal are added to the set 47 of classifiers. The setting unit 45 sets the reliability for each of the classifiers 401 included in the classifier set 47, and the output unit 48 outputs the first output that is output from each of the classifiers 401 included in the classifier set 47. Value and classifier 4 included in classifier set 47
The external output value 44 is generated based on the reliability set for each of the external output values 01.

The learning apparatus 400 can reduce the calculation time required for learning and increase the ability to follow environmental changes, as compared with the case where the classifier system 40 performs learning based only on rewards. it can.

[0240] Note that the speech recognition function described with reference to Fig. 9 may be added to the learning device 400. Learning device 4
When the voice recognition function is added to the
0 have more opportunities to receive teacher signals and rewards. Therefore, the learning device 400 can efficiently perform learning.

According to the learning apparatus of the first to third embodiments of the present invention, learning is performed based on both the teacher signal and the reward. The learning device of the present invention is particularly suitable when used in an environment where a human exists (for example, at home). Therefore, it can be suitably applied to intelligent homes. The learning device of the present invention can observe and learn human behavior while a human is present, and can learn autonomously based on rewards while no human is present. Therefore, the learning efficiency is higher than in the case where learning is performed based on only one of the reward and the teacher signal. That is, it is possible to reduce the time required to adapt to the user and the environment. ADVANTAGE OF THE INVENTION According to the learning device of this invention, since efficient learning is performed, acquisition of acquired knowledge is easy. For this reason, it is not necessary to design all the functions necessary for the learning device in advance when developing the system, and it is possible to reduce the manufacturing cost.

The learning processing by the learning device according to the first to third embodiments of the present invention can be recorded on a recording medium in the form of a program. As the recording medium, a computer-readable recording medium of any type such as a floppy (registered trademark) disk or a CD-ROM can be used. By installing the learning processing program read from the recording medium on the computer,
This allows the computer to function as a learning device.

[0243]

According to the learning apparatus of the present invention, when a reward and a teacher signal are given from the environment, learning can be executed based on both of them. For this reason, the learning device can perform learning regardless of whether or not a teacher signal is obtained, and can improve learning efficiency as compared with the case where learning is performed based only on the teacher signal.

Further, the learning device of the present invention can execute effective learning based on a teacher signal, and can improve learning efficiency as compared with the case where learning is performed based only on rewards.

According to the learning apparatus of the present invention, the knowledge acquired by learning based on reward and the knowledge acquired by learning based on teacher signals are mutually accumulated.
Therefore, learning efficiency can be improved.

[Brief description of the drawings]

FIG. 1A is a learning device 100 that performs learning based on a teacher signal.
Diagram showing the input / output relationship of

FIG. 1B is a diagram showing an input / output relationship of a learning device 200 that performs learning based on a reward.

FIG. 2 is a block diagram showing a configuration of a learning device 250 according to the first embodiment of the present invention.

FIG. 3 is a diagram showing a task used to explain details of the operation of the learning device 250;

FIG. 4 is a block diagram showing a configuration of a learning unit 13 based on a teacher signal.

FIG. 5 is a flowchart showing a processing procedure of learning designation by a learning designation unit 11;

FIG. 6 is a flowchart showing a processing procedure of output selection by an operation selection unit 14;

FIG. 7 shows a learning unit 12 and a teacher action storage unit 13 based on a reward.
1. Diagram showing input and output relationships of memory-based imitation unit 132, direct imitation unit 133, and learning unit 19

FIG. 8 is a block diagram showing a configuration of a learning device 260 in which the learning unit 19 and the learning switching unit 20 are removed from the learning device 250.

FIG. 9 is a block diagram showing a configuration of a learning device 270 having a voice recognition function.

FIG. 10 shows an example of a word spoken by a human, the meaning interpreted by the meaning interpreting unit 31, and the learning device 270 at that time.
Diagram showing actions to be taken

FIG. 11 is a diagram showing a configuration of a learning device 280 in which three different learning methods are integrated.

12 (a) to 12 (c) show the time change of the learning degree when the knowledge is inherited between the learning module A executing the learning method A and the learning module B executing the learning method B. (D)-(f) are diagrams showing the time change of the learning degree when the knowledge is not inherited between the learning module A and the learning module B.

FIG. 13 is a block diagram showing a configuration of a learning apparatus 310 in which knowledge can be inherited between learning modules that execute different learning methods.

FIG. 14 illustrates a learning apparatus 320 that implements learning in which knowledge acquired by learning based on rewards and knowledge acquired by learning based on teacher signals are accumulated with each other using a single neural network. Block diagram showing configuration

FIG. 15 is a flowchart showing a processing procedure in which a learning device performs learning.

FIG. 16 shows a learning device 320 determining the quality of a teacher signal,
Flowchart showing a processing procedure for controlling a learning method based on the quality of a teacher signal

FIG. 17 is a block diagram showing a configuration of a learning device 400 using a classifier system.

[Explanation of symbols]

 11, 281, 312 Learning designation unit 12, 313 Reward-based learning unit 13, 314 Learning unit based on teacher signal 14 Action selection unit 19 Learning unit 20 Learning switching unit 21A-C Learning module 30 Voice recognition unit 31 Meaning interpretation unit 40 Classifier system 41 Classifier generation unit based on teacher signal 131 Teacher behavior storage unit 132 Imitation unit based on storage 133 Direct imitation unit 250, 260, 270, 310, 320 Learning device 271 Speech interpretation unit 311 Input unit 315 Output unit 322 Adjustment Unit 324 learning pattern generation unit 325 short term storage unit

Claims (18)

[Claims] 1. A learning device for generating an output value corresponding to an input value based on at least one parameter, wherein a reward signal indicating an evaluation value related to the output value is received from outside the learning device. An input unit, a teacher signal input unit that receives a teacher signal indicating an expected value of the output value with respect to the input value from outside the learning device, and the evaluation value increases based on the reward signal and the teacher signal. And a adjusting unit that adjusts the value of the at least one parameter such that the output value with respect to the input value approaches the expected value. 2. The apparatus according to claim 1, further comprising: a pattern generator configured to generate a first learning pattern based on the reward signal, and generate a second learning pattern based on the teacher signal. And adjusting the value of the at least one parameter based on the second learning pattern so that the output value for the input value approaches the expected value based on the second learning pattern. Adjusting the value of the parameter, wherein the first learning pattern is configured such that, when the adjustment unit adjusts the value of the at least one parameter such that the evaluation value increases based on the first learning pattern, The output value with respect to the value is configured to approach the expected value, and the second learning pattern is determined by the adjustment unit based on the second learning pattern. The learning device according to claim 1, wherein the evaluation value is increased when the value of the at least one parameter is adjusted such that the output value with respect to the input value approaches the expected value. . 3. The pattern generation unit generates the second learning pattern based on the teacher signal whenever the teacher signal is input to the pattern generation unit, and outputs the teacher signal to the pattern generation unit. The learning device according to claim 2, wherein when not input, the first learning pattern is generated based on the reward signal. 4. The apparatus according to claim 1, further comprising: a determination unit configured to determine a quality of the teacher signal based on a predetermined criterion, wherein the pattern generation unit determines the quality of the teacher signal based on the quality of the teacher signal. The learning device according to claim 2, which determines whether to generate a learning pattern. 5. A learning device for generating an output value according to an input value, wherein a first learning is performed based on a reward signal, and the first learning is performed based on at least one first parameter. A first learning unit that generates one output value, and a second learning that generates a second output value corresponding to the input value based on at least one second parameter by performing second learning based on a teacher signal. A unit, an output unit that selectively outputs one of the first output value and the second output value as the output value, and whether to designate the first learning by the first learning unit,
And a learning designating unit that determines whether or not to designate the second learning by the second learning unit, wherein the first learning unit is configured to perform a learning process on a reward signal indicating an evaluation value related to the output value. A reward signal input unit received from outside the device; and a first adjusting unit that adjusts a value of the at least one first parameter based on the reward signal so that the evaluation value increases. A learning unit configured to receive a teacher signal indicating an expected value of the output value with respect to the input value from outside the learning device;
The at least one value is selected so that an output value approaches the expected value.
A second adjustment unit that adjusts the values of the two second parameters. 6. The learning device according to claim 5, wherein the learning specifying unit determines whether to specify the first learning according to a value of an attention parameter that changes according to time. 7. The output unit outputs the first output value as the output value or outputs the second output value as the output value according to a value of a storage status parameter that changes according to time. The learning device according to claim 5, which determines whether to perform the learning. 8. The output unit determines, based on the reward signal, whether to output the first output value as the output value or to output the second output value as the output value. 6. The learning device according to 5. 9. The learning device according to claim 5, wherein the learning device further includes a direct imitation unit that outputs the teacher signal as the second output value regardless of the input value. 10. The learning designating section, wherein the first learning by the first learning section and the second learning by the second learning section are performed.
In the case where learning is alternatively specified and the state in which the learning specifying unit specifies the first learning transitions to a state in which the learning specifying unit specifies the second learning, the at least Adjusting one second parameter based on the at least one first parameter;
The state in which learning is specified is when the learning specifying unit is in the first state.
The learning device according to claim 5, wherein when transitioning to a state in which learning is specified, the at least one first parameter is adjusted based on the at least one second parameter. 11. A speech recognition unit for recognizing speech input from outside the learning device, and a semantic interpretation unit for extracting semantic information based on a recognition result by the speech recognition unit. 6. The learning device according to 5. 12. The learning device further includes an instruction input unit that receives an operation instruction for the learning device, wherein the semantic information is information related to at least one of the operation instruction, the teacher signal, and the reward signal. The learning device according to claim 11, wherein: 13. A learning device that generates an output value according to an input value, wherein the learning device performs learning to generate a module output value according to the input value, and a plurality of learning modules; A learning specifying unit that determines whether or not to specify the learning by each of the plurality of learning modules based on a predetermined first rule; and a learning output unit that is output from the plurality of learning modules based on a predetermined second rule. An operation selecting unit for selectively outputting one of a plurality of module output values as a first output value, a learning unit for generating a second output value corresponding to the input value based on at least one parameter, An output unit configured to selectively output one of the first output value and the second output value as the output value based on a predetermined third rule; The output unit adjusts the at least one parameter so as to approach one output value. When the second output value is selected, the output unit generates a module output value by the plurality of learning modules; Learning for stopping an operation of determining whether or not to specify the learning by a specifying unit and an output operation of selectively outputting one of the plurality of module output values as the first output value by the operation selecting unit; apparatus. 14. A learning device for generating an output value according to an input value, wherein the learning device comprises: at least one classifier indicating a rule defining a first output value for the input value; A first generating unit that generates a first new classifier based on a reward signal indicating an evaluation value related to the first classifier and adds the first new classifier to the at least one classifier; A second generation unit that generates a second new classifier based on a teacher signal indicating an expected value of the output value with respect to a value, and adds the second new classifier to the at least one classifier; A setting unit that sets a degree of reliability for each of the at least one classifier to which the first new classifier and the second new classifier have been added; and a first new classifier. The small classifier to which the second new classifier has been added. At least one first output value output from at least one of each of the classifiers; and the at least one classifier to which the first new classifier and the second new classifier are added. And an output unit that generates the output value based on the reliability set for each of the learning devices. 15. A computer-readable recording medium storing a program for causing a learning device to execute a process of generating an output value according to an input value based on at least one parameter, the computer-readable recording medium comprising: (A) receiving a reward signal indicating an evaluation value related to the output value from outside the learning device; and (b) indicating an expected value of the output value with respect to the input value. Receiving a teacher signal from outside the learning device; and (c) based on the teacher signal and the reward signal, the output value with respect to the input value approaches the expected value, and the evaluation value increases. Adjusting the value of the at least one parameter. 16. A computer-readable recording medium storing a program for causing a learning device to execute a process of generating an output value according to an input value, wherein the process of generating an output value according to the input value is performed. (A) performing first learning based on a reward signal to generate a first output value corresponding to the input value based on at least one first parameter; and (b) based on a teacher signal. Performing a second learning to generate a second output value corresponding to the input value based on at least one second parameter; and (c) one of the first output value and the second output value. Selectively outputting as the output value; and (d) determining whether to specify the first learning and whether to specify the second learning,
The step (a) includes: (a1) receiving a reward signal indicating an evaluation value related to the output value from outside the learning device; and (a2) increasing the evaluation value based on the reward signal. Adjusting the value of the at least one first parameter, wherein the step (b) comprises: (b1) outputting a teacher signal indicating an expected value of the output value with respect to the input value from outside the learning device; Receiving; and (b2) recording based on the teacher signal, adjusting the value of the at least one second parameter such that the first output value for the input value approaches the expected value. Medium. 17. A computer-readable recording medium storing a program for executing a process of generating an output value according to an input value, wherein the process of generating an output value according to the input value includes: a) generating a plurality of module output values corresponding to the input values by executing a plurality of learnings; and (b) specifying each of the plurality of learnings based on a predetermined first rule. (C) selectively outputting one of the plurality of module output values as a first output value based on a predetermined second rule; and (d) at least one parameter. Generating a second output value according to the input value based on the following: (e) one of the first output value and the second output value as the output value based on a predetermined third rule Selection And (d1) adjusting the at least one parameter such that the second output value approaches the first output value. When the second output value is output as the output value in e), the operation of generating the plurality of module output values in step (a) and the plurality of learning in step (b) are specified. A recording medium configured to stop the operation of determining whether or not to perform the operation and the operation of selectively outputting one of the plurality of module output values as the first output value in step (c). 18. A computer-readable recording medium recording a program for executing a process of generating an output value according to an input value using at least one classifier, wherein the computer-readable recording medium is a computer-readable recording medium. Each shows a rule that defines a first output value for the input value, and the process of generating an output value according to the input value includes: (a) based on a reward signal indicating an evaluation value related to the output value Generating a first new classifier and adding the first new classifier to the at least one classifier; and (b) a teacher signal indicating an expected value of the output value with respect to the input value. Generating a second new classifier based on and adding the second new classifier to the at least one classifier; and (c) generating the first new classifier and the second classifier. 2 new Setting a reliability for each of the at least one classifier to which a new classifier has been added; and (d) adding the first new classifier and the second new classifier. At least one said first output value output from each of said at least one classifier;
Generating the output value based on the reliability set for each of the at least one classifier to which the first new classifier and the second new classifier have been added; A recording medium, including: