JPH0283656A - Learning processor - Google Patents

Abstract

(57) [Abstract] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

[Detailed Description of the Invention] A. Industrial Field of Application The present invention uses a so-called neural network (neural network) composed of a plurality of units that each perform signal processing corresponding to a neuron. The present invention relates to a learning processing device that performs learning processing on a signal processing unit based on a hack propagation learning agent.

B. Summary of the Invention The present invention provides a learning processing device that performs learning processing in accordance with the backpropagation learning rule for a signal processing unit using a neural network, in which the strength of connections is increased while increasing the number of units in the intermediate layer. By performing coefficient learning processing, a local minimum value state in the learning processing process can be avoided.

C.Backpropagation learning rule which is a learning algorithm of conventional technology neural network” ParallelDis
TributedProcessingJVol,I
The MTT Press 1986 and Nikkei Electronics August 10, 1987 issue.

No, 427. ppH5-124, etc." is applied to a multilayer neural network that has an intermediate layer (32) between an input layer (31) and an output layer (33), as shown in Figure 8, and is used for high-speed image processing and Applications to various signal processing such as pattern recognition are being attempted.

That is, as shown in FIG. 8, each unit (U,) constituting this neural network is
, ) to the unit (uJ) with the coupling coefficient Wji. , is converted by a predetermined function f such as a sigmoid function, and a value o is output. So, when the value of pattern p is supplied as an input value to each unit (UJ) of the input layer, the output value 0++i of each unit F(u=) of the intermediate layer and the output layer is as follows. OpJ=f J(netpj) =f, (ΣWji·O9j+)... Expressed by the first equation.

Then, by sequentially calculating the output value of the unit (UJ) corresponding to each unit from the input layer (3I) to the output layer (33), the output value of the unit (U4) of the above output i (33) is calculated. Output (direct Op= is obtained.

In the bank blog game learning algorithm,
Actual output value 09j and desired output value Lp of each unit (u4) of the output layer (33) when pattern p is given
j, that is, the sum E of squared errors with respect to the teacher signal.

By sequentially performing a learning process that changes the coupling coefficient Wji from the output layer (33) to the input layer (31) so as to reduce the cross section, the output value Op= closest to the value LpJ of the teacher signal is The unit (u;) of the output layer (33) is now output.

Then, the coupling coefficient Wi that reduces the sum of squared errors E
(If the amount of change ΔVVjl is determined as ΔWji CC−aE, /gwJr −−−−− third formula, then the above third formula becomes ΔW1−η・δ9,・o21 ・・・・・・−・
7J44 fc (see the above-mentioned document for this process).

Here, η is a learning rate (constant), which is determined empirically from the number of units, the number of layers, the number of people, etc. Also, δ2. is the error value of the unit (uJ).

Therefore, in order to determine the amount of change ΔW j i,
The above error value δ2J may be obtained in the reverse direction from the output layer of the network toward the input layer. Error value δ2 of unit (u,) in the output layer. is δpj=(t,,' p=
) f '7 (net7) 0910. 5th formula
2, the error value δ2 of the unit (u,) of the intermediate layer is given by
, using the coupling coefficient WkJ and error value δ of each unit (um) (in this example, each unit of the output layer) to which the unit (uJ) is coupled, δpj ''' f ' j (nejj)Σδpkwk
, mountain...Calculated using the recursive function of the sixth formula (the fifth above)
For the formula and the process of determining No. 6, please refer to the above-mentioned literature).

Note that the above f' J (netJ) is the output function f J
It is the differential value of (netJ).

The amount of change Δw1 is determined by the above-mentioned equation 4 using the results of equations 5 and 6, but using the previous learning results, ΔWji, □l+=η・δpj'op juice α・Δwji(
A more stable result can be obtained by finding nl using the seventh equation. Note that α is a stabilizing constant for leaving error oscillations and speeding up convergence.

Then, this learning is repeated, and the learning is completed when the sum Ep of the squared errors between the output value □p4 and the value tIlj of the teacher signal becomes sufficiently small.

D Problems to be Solved by the Invention By the way, learning processing according to the hack propagation learning rule for multilayer neural networks as described above can be expected to have high functional ability, but in the learning processing process, the optimal minimum value (global mini
local minimum (local
In many cases, the sum Ep of the squared errors does not become sufficiently small.

Conventionally, when falling into the above local minimum value state, the optimal minimum value state was found by repeatedly performing the learning process by changing the initial value and learning rate η.
With conventional learning processing devices, the learning processing time is extremely long.
Moreover, there was a problem in that the fluctuation was large.

Therefore, in view of the conventional situation as described above, the present invention provides a learning processing device that performs learning processing in accordance with the backpropagation learning rule on a signal processing unit using a neural network, in which a local minimum value state in the learning processing process is In order to efficiently avoid this and converge to the optimal minimum value state stably and quickly, we have developed a learning processing device with a new configuration that performs learning processing while increasing the number of units in the middle layer. This is what we provide.

E. Means for Solving the Problems In order to achieve the above-mentioned objects, the present invention provides a signal processing system comprising an input layer, an intermediate layer, and an output layer, each of which is composed of a plurality of units that perform signal processing corresponding to neurons. and a coefficient of the strength of coupling between each unit based on error information between the output value of the output layer and the desired output value given as a teacher signal with respect to the input signal pattern input to the input layer. and a learning processing unit that repeatedly calculates the coefficients of the strength of the connection in order from the output layer side to the input layer side, and performs learning processing of the coefficient of the strength of the connection, wherein the coefficient of the strength of the connection is The learning processing section is provided with a control means for increasing the number of units in the intermediate layer in the learning processing process, and the learning processing section controls the coefficient of the connection strength while increasing the number of units in the intermediate layer. It is characterized by the fact that it performs learning processing.

F Function In the learning processing device according to the present invention, the learning processing unit performs learning processing of the coefficient of connection strength while increasing the number of units in the intermediate layer, thereby following the backpropagation learning rule. Learning processing is performed to avoid local minimum value states in the learning processing process and reliably converge to the optimal minimum value state.

G. Embodiments Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

As shown in FIG. 1, the learning processing device according to the present invention has at least an input layer (11) composed of a plurality of units that each perform signal processing corresponding to a neuron. Intermediate layer (12) and output layer (13
), and an output value of the output layer for the input signal pattern p input to the input layer (11) of the signal processing unit (10). The coefficient Wji of the strength of coupling between the units is calculated from the output layer (13) side to the input layer (11) based on the error information δ2 between the desired output value 0□ given as the teacher signal Lpj. and a learning processing section (20) that subjects the signal processing section (10) to a learning process of sequentially and repeatedly calculating the coupling coefficient Wji in accordance with the Basokupronovagation learning rule.

The learning processing unit (20) performs learning processing of the coupling coefficient Wji while increasing the number of units of the intermediate Ji (12) of the signal processing unit (10), and performs learning processing of the coupling coefficient Wji. In the process, the above intermediate layer (12)
For example, as shown in FIG. 2A, an arbitrary number of x, y, z units (u+ to u +++) corresponding to neurons, respectively,
Input layer (11), intermediate layer (12) and output layer (13) composed of (uHi-uHy)+(uo+=uog)
), as shown in FIG. 2B, while increasing the number of units in the intermediate layer (12) from y to (y+m), the coupling coefficient Performs Wji learning processing.

Here, the control to increase the number of units in the intermediate 11 (12) may be performed periodically in the process of learning the coupling coefficient W j i, and the occurrence of the local minimum value state may be detected. It may be done every time.

In the process of learning the coupling coefficient Wji, the learning processing unit (20) has a control function of increasing the number of units in the intermediate layer (12).
12) and an output layer (13), the number of units in the intermediate layer (12) of the signal processing unit (10) is By performing the learning process of the coupling coefficient Wji while increasing the coupling coefficient Wji, even if a local minimum value state occurs in the process of learning the coupling coefficient Wji, the unit of the intermediate layer (12) increases. It is possible to perform a learning process that quickly and reliably converges to the optimal minimum value state by starting from the minimum value state.

In this way, in the learning processing process of the coupling coefficient Wji, the learning processing unit (20) having a control function to increase the number of units in the intermediate layer, for example, as shown in FIG. any number X,
y, Z unit (u++~u+x)+(usr-u
sr)AuOt-uoz)
It is composed of a neural network with a three-layer structure: the middle layer (L,), the middle layer (L, ), and the output layer (Lo).
M) and output layer (Lo) each unit 7) (uMl~u)
Iy)+(uol"""uoi) is a loop (LP) which is provided with a delay means and whose output value oit is inputted to itself via the delay means, and a feed which is inputted to another unino I. Regarding the signal processing unit (100) that constitutes a recurrent circuit network including a hack (FB), the number of unins in the input layer (Ll) is 8 (x = 8), and the number of unins in the input layer (Ll) is 8 (x = 8),
o), the number of units is 3 (z=3), the number of delay means in each layer is 2, and the input signal pattern p during learning is /=8
Using 21 spatiotemporal patterns of X7, the middle layer (1-H
) We started learning from 3 units (y = 3), and repeated experiments in which we added units from the middle layer (LM) during the learning process.
) was added 3 to 5 times, in all learning processing experiments, experimental results were obtained that converged to the optimal minimum value state without falling into a local minimum value state.

FIG. 5 shows an example of the results of the above experiment, in which the middle layer (LJ) unit was added at the timing indicated by the arrow in the figure, and the middle layer (Lx) unit was added. The experimental results show that by increasing the number from 3 to 6, the learning process was able to converge to the optimal minimum value state.In Figure 5, the vertical axis represents the sum of the squared errors , and the horizontal axis indicates the number of learning processes.

Here, the processing algorithm shown in the flowchart of FIG. 4 above will be explained.

In this processing algorithm, first, in step 1, a variable K indicating the number of processes for detecting a local minimum value state is initially set to O, and a first variable Lms is set to O for determining the convergence condition of the learning process. 10000000
Initialize to 00.

In the next step 2, you will learn all the learning patterns! After initially setting a variable n indicating the number of learning times for P input signal patterns p to 0, the process moves to step 3 and learning processing for P input signal patterns p is performed.

In the next step 4, the variable n indicating the number of learning times is determined, and if n=3, the process moves to step 5 and n=
When n+1 is set, the process returns to step 3 and the learning process is repeated, and when n=3, step 6 is increased by one.

In step 6 above, the value of the first variable L+ms is held as the value of the second variable Lms(-1) for determining the convergence condition of the learning process, and then the teacher signal and output signal in each unit are The sum of the squared errors is calculated using equation 8, and this value is used as the new value of the first variable LI+ls.

Lms =ΣΣDot〜O□)2 ・・・・・・8th
In the next step 7, the first variable Lms and the second variable Lms are used to determine the convergence condition of the learning process.
(-1), and if the value of the first variable Lms is smaller than the value of the second variable Lms (-1), proceed to step 8 and detect a local minimum value state. It is determined whether the variable K indicating the number of times of processing is 0 or not.

In step 8, if the variable K is O, the process returns directly to step 2; if the variable K is not 0, in step 9, = is set to +1, and then returns to step 2, where n= 0, and the learning process for the two input signal patterns p described above is performed in step 3 above.

Further, in step 7, the first variable Lms
If the value of is larger than the value of the second variable L#15 (-1), the process moves to step 10 and the value of the variable indicating the number of processing times for detecting the local minimum state is set to = ni＋
After setting the value to 1, it is determined in step 11 whether the value of the variable is 2 or not.

In step 11, if the value of the variable is not 2, the process returns directly to step 2, and if the variable K is 2, it is determined that the state has fallen into a local minimum value state, and in step 12, Control is performed to add the unit of the intermediate layer (L), and further, in step 13, set = O, and then return to step 2, set n = 0,
The above-described learning process for one input signal pattern p is performed in step 3 of ``2''.

In the signal processing section (100) shown in FIG. 3, each unit (U.

~tJ+, ), each unit (u) of the intermediate layer (LH) II~u
Ny), the total human power netj is netj−Σ Σ Wjx*mha・O+a (t−
*)+θ, ・・・・・・Equation 9 is the 9th
For the total sum neJ of inputs, the output value 0□ftl is given by the sigmoid function of the 10th equation 10.

Furthermore, each unit (uoI to u
ox), the sum of its inputs netJ is +θJ
...Given in the 11th formula, 11th 2,
For this input total neJ, do the following.

Here, the above θ is a threshold value, Nl, NH, N.

indicate the number of delay means in each of the layers (L+), (L-i), and (LO).

H Comparative Example [Comparative Example 1] Regarding the signal processing unit (100) shown in FIG. When we performed this, we found that it was necessary to repeat the learning process extremely many times in order to converge to the optimal minimum value state, which not only took a lot of time, but also converged to the optimal minimum value state three times out of eight learning processing experiments. Experimental results were obtained that the system could fall into a local minimum state without any problem.

FIG. 6 shows an example of the experimental results when the learning processing experiment in Comparative Example 1 falls into a local minimum value state.

In FIG. 6, the reduced axis indicates the sum of squared errors LMS, and the horizontal axis indicates the number of learning processes.

(Comparative Example 2) Regarding the signal processing unit (100) shown in FIG. As shown in one example of the experimental results shown in FIG. 7, in all the learning processing experiments, the learning process fell into a local minimum value state without converging to the optimal minimum value state.

In addition, in Fig. 7 as well, the vertical axis is the sum of squared errors LMS
, and the horizontal axis indicates the number of learning processes.

■ Effects of the Invention In the learning processing device according to the present invention, the learning processing unit performs learning processing of the coefficient of connection strength while increasing the number of units in the intermediate layer. Accordingly, it is possible to perform stable learning processing that quickly and reliably converges to the optimal minimum value state by avoiding local minimum value states in the learning processing process.

[Brief explanation of drawings]

FIG. 1 is a block diagram conceptually showing the configuration of a learning processing device according to the present invention, and FIGS. 2A and 2B are signals at the start of learning processing and during the learning processing in the learning processing process by the learning processing device. FIG. 3 is a schematic diagram showing the state of the processing section; FIG. 3 is a schematic diagram showing the configuration of the neural network of the signal processing section subjected to learning processing by the learning processing device according to the present invention;
FIG. 4 is a flowchart showing an example of the learning processing process by the learning processing unit constituting the learning processing device; FIG. 5 is a characteristic diagram showing an example of the results of a learning processing experiment by the learning processing unit; Figure 6 is a characteristic diagram of Comparative Example 1 showing the results of a learning processing experiment with the number of units in the intermediate layer of the neural network in the signal processing section shown in Figure 3 fixed at 6; A characteristic diagram of Comparative Example 2 showing the results of a learning processing experiment with the number of units in the intermediate layer of the neural network of the signal processing section fixed at three shown in Fig. 3 above, and Fig. 8 is a graph of Banoku Block FIG. 1 is a schematic diagram showing a general configuration of a neural network to which a severation learning rule is applied. (10), (100)...Signal processing section (20
)・・・・・・・・・・Learning processing unit (L+)・・・・
・・・・・・Input layer (L,)・・・・・・・・・・・・
Middle layer (L,)... Output layer (u++
'u+x)+(uH+"1Jny)+(uo+""Ll
oj ・b meeting 0. -9. unit

Claims (1)

[Scope of Claims] A signal processing unit comprising an input layer, an intermediate layer, and an output layer each configured of a plurality of units that perform signal processing corresponding to a neuron; Based on the error information between the output value of the output layer and the desired output value given as a teacher signal, coefficients of the strength of coupling between the units are sequentially repeated from the output layer side to the input layer side. and a learning processing unit that performs learning processing of the coefficient of connection strength, the number of units of the intermediate layer being increased in the process of learning the coefficient of connection strength. A learning processing device characterized in that a control means is provided in the learning processing unit, and the learning processing unit performs learning processing of the coefficient of the strength of connection while increasing the number of units in the intermediate layer. .