JPH07113918B2 - Network learning processing device - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a network learning processing device and a learning processing method.

BACKGROUND ART In conventional sequential computers (von Neumann computers), it is difficult to adjust the data processing function of the computer according to changes in usage and environment. Therefore, a new parallel distributed processing method using a hierarchical network has been proposed as an adaptive data processing method. In particular, a processing method called the back propagation method (DERume
lhart, GE Hinton, and Williams, “LearningInternal R
epresentations by Error Propagation,"PARALLEL DIST
RIBUTED PROCESSING,Vol.1,pp.318−364,The MIT Pres.
s, 1986) has attracted attention due to its high practicality.

In the back-propagation method, a hierarchical network is constructed from a type of node called a basic unit and internal connections with weights. Figure 1 shows the basic configuration of a basic unit 1. This basic unit 1 is a multi-input-output system, and includes a multiplication processing unit 2 that multiplies multiple inputs by the weights of the respective internal connections, an accumulation processing unit 3 that adds up all the multiplication results, and a threshold processing unit 4 that performs nonlinear threshold processing on this accumulated value and outputs a single final output.
A large number of basic units 1 having such a configuration are connected hierarchically as shown in Figure 2 to form a hierarchical network, which performs the data processing function of converting an input signal pattern into a corresponding output signal pattern.

In this backpropagation method, the weights of the internal connections in the hierarchical network are determined according to a predetermined learning algorithm so that the output signal for a selected input signal becomes a teacher signal that indicates the signal value to be taken. Once the weights are determined through this process, even if an unexpected input signal is input, the hierarchical network will output a plausible output signal, thereby realizing a "flexible" parallel distributed data processing function.

To make such a network-configured data processing device practical, it is necessary to make it possible to achieve weight learning processing in a shorter time. This is one of the issues that must be resolved, given the need for hierarchical networks with more layers to process complex data.

If the h layer is the previous layer and the i layer is the next layer, the calculation performed by the accumulation processing unit 3 of the basic unit 1 is shown in the following equation (1), and the calculation performed by the threshold processing unit 4 is shown in the following equation (2).

y _pi = 1/(1 + exp(-x _pi + θ _i )) ... (2) where, h: unit number of the h layer i: unit number of the i layer p: pattern number of the input signal θ _i : threshold of the i-th unit of the i-th layer W _ih : weight of the internal connection between the h-i layers x _{pi :} sum of products of inputs from each unit of the h layer to the i-th unit of the i-th layer y ph: output of the h layer for the p-th pattern input signal y _pi : output of the i layer for the p-th pattern input signal In the back propagation method, the weight W _ih and threshold θ _i are adaptively and automatically adjusted by error feedback. As is clear from equations (1) and (2), the adjustment of the weight W _ih and threshold θ _i needs to be performed simultaneously, but this is a difficult task as they interfere with each other. Therefore, the applicant has applied for a method in accordance with the Japanese Patent Application No. 2004-20097444.
As disclosed in the publication No. 62-333484 (filed on December 28, 1987, "Network Configuration Data Processing Device"), the basic unit 1 always has "1" as an input signal in the input example layer.
By setting the threshold θ _i in the weight W _ih and incorporating the threshold θ i into the weight W ih, the threshold θ _i is not revealed. This can be expressed as y _pi = 1/(1-exp(-x _pi )) ... (4).

Next, a conventional technique for weight learning processing will be explained in accordance with equations (3) and (4). This explanation will be given for a hierarchical network with a structure of h-layer-i-layer-j-layer as shown in Figure 2.

By analogy with equations (3) and (4), the following equation can be obtained: y _pj = 1/(1 + exp(-x _pj )) ... (6) where, j: unit number of j layer W _ji : weight of internal connection between i-j layers x _pj : sum of products of inputs from each unit of i layer to the jth unit of j layer y _pj : output weight of j layer for pth pattern of input signal In the learning process, first the error vector E _P , which is the sum of squares of the error between the teacher signal and the output signal from the output layer, is calculated as the error of the hierarchical network according to the following formula. Here, the teacher signal is the signal that the output signal should take.

where E _p : error vector for the p-th pattern input signal E: sum of error vectors for all patterns of input signals d _pj : teacher signal to the j-th unit in the j-th layer for the p-th pattern input signal Here, to find the relationship between the error vector and the output signal, equation (7) is partially differentiated with respect to y _pj to obtain: Furthermore, to find the relationship between the error vector E _P and the input to the jth layer, the error vector E _P is partially differentiated with respect to x _pj , and we get Furthermore, to find the relationship between the error vector E _P and the weight between the i-j layers, the error vector E _P is partially differentiated with respect to W _ji , as follows: Obtain a solution expressed as a sum of products.

Next, the change in the error vector E _P relative to the output y _pi of the i layer is calculated as follows: Furthermore, if we calculate the change in the error vector relative to the change in the sum _xpi to the i-th layer input unit, we get Furthermore, the relationship between the change in the weight between the h-i layer and the change in the error vector is found as follows: Obtain the solution expressed as a sum of products.

From these, the error vectors for all input patterns and i
The relationship between the weights and the −j layer is as follows:

Furthermore, the relationship between the error vectors for all input patterns and the weights between the h-i layers can be calculated as follows:

Since equations (15) and (16) show the rate of change of the error vector with respect to the change in the weight between each layer, if the weights are changed so that this value is always negative, the sum E of the error vectors can be asymptotically reduced to 0 by the well-known gradient method. Therefore, in the conventional backpropagation method, the amount of weight update per time, ΔW _ji and ΔW _ih , is set as follows, and by repeating this weight update, the sum E of the error vectors is converged to a minimum value.

where ε is a control parameter for learning.

The biggest problem with the backpropagation method is the long number of learning iterations required to converge. This problem becomes more severe the more layers the network structure has. Therefore, in order to accelerate convergence, a data factor related to the weight update amount determined in the previous update cycle is added to equations (17) and (18), and ΔW _ih and ΔW' _ji are set as follows:

Here, α is also a control parameter for learning, and t represents the number of updates.

If the control parameters ε and α are made sufficiently small, the sum E of the error vectors will almost certainly converge.
This means that the number of learning iterations until convergence will be large. On the other hand, if both parameters are set large in an attempt to reduce the number of learning iterations, then there is a risk that the sum E of the error vectors will oscillate. Assuming a hierarchical network with 13 units in the input layer, 8 units in the intermediate layer, and 7 units in the output layer, Figure 5 shows the learning results when learning is performed using the 62 input pattern signals shown in Figure 4 and the corresponding teacher pattern signals. In Figure 5, the horizontal axis represents the number of learning iterations, and the vertical axis represents the sum of the error vectors at that time. Here, the control parameters are set to ε = 0.3 and α = 0.2. Furthermore, for the reasons mentioned above, the 13th basic unit is always assigned a value of 1.
I entered:

As is clear from FIG. 5, although differences in parameter settings result in differences in the results, it is clear that the prior art requires a considerable number of learning iterations to determine the weights.

The present invention has been made in view of the above circumstances, and aims to provide a learning processing method that enables weight determination to be achieved in a shorter number of learning iterations, i.e., in a shorter time, in the back propagation method of a network configuration data processing device.

DISCLOSURE OF THE INVENTION Figure 6 is a diagram illustrating the principle of the present invention.

In the figure, 1 is a basic unit that forms the basic unit of the hierarchical network, which receives multiple inputs and weights to be multiplied by these inputs to obtain an integral, and processes the obtained integral value by converting it using a threshold function to obtain the final output. 1-h is a plurality of basic units that form the input layer, 1-i is a plurality of basic units that form one or more intermediate layers, and 1-j is a plurality of basic units that form the output layer.
Basic unit 1-
Connections are made between h and basic unit 1-i, between basic units 1-i, and between basic unit 1-i and basic unit 1-j, and a hierarchical network with a weight of 10 is constructed by setting a weight corresponding to each connection.

Reference numeral 20 denotes a learning pattern storage means for storing learning patterns required for weight learning processing, and comprises an input signal storage area 21 for storing a plurality of predetermined input signals, and a teacher signal storage area 22 for storing teacher signals corresponding to the predetermined input signals; 30 denotes an output signal derivation means for supplying the input signals stored in the input signal storage area 21 to the hierarchical network 10, thereby processing to obtain an output signal corresponding to the input signal; and 40 denotes an error value calculation means for calculating an error value representing the degree of discrepancy between the output signal obtained by the output signal derivation means 30 and the teacher signal stored in the teacher signal storage area 22, and for processing to obtain error values for all input signals supplied.

Reference numeral 50 denotes a weight learning means, which includes a weight update amount calculation means 51, a first weight update amount holding means 52, and a second weight update amount holding means.
The weight update amount calculation means 51 calculates the weight update amount using the error value calculated by the error value calculation means 40. The first weight update amount storage means 52 stores the weight update amount for the previous update cycle calculated by the weight update amount calculation means 51. The second weight update amount storage means 53 stores the weight update amount for the update cycle before last calculated by the weight update amount calculation means 51. The weight update amount calculation means 51 calculates the weight update amount for the current update cycle from the partial differential value of the error value calculated by the error value calculation means 40, the weight update amount for the previous update cycle stored in the first update amount storage means 52, and the weight update amount for the update cycle before last stored in the second weight update amount storage means 53.
calculates a weight value based on the weight update amount calculated by the weight update amount calculation means 51, and inputs the weight value to the hierarchical network 10
Set to.

In the present invention, a second weight update amount holding means 53 is newly provided to hold the weight update amount from the update cycle two cycles ago, and when determining the weight value to be updated, the data factor related to the weight update amount held by this second weight update amount holding means 53 is added.

The weight update amount in the conventional backpropagation method is In other words, if we apply a difference approximation to this differential equation and solve it for ΔW, we get This is because the weight update amounts in the conventional back propagation method explained in equations (19) and (20) are derived.

In contrast, the amount of weight update in the present invention is In other words, when this differential equation is approximated by difference and solved for ΔW, In this way, the data factor relating to the amount of weight update in the update cycle before the last one is included.

This equation (23) expresses the following when there is a disturbance as shown on the right side:
It represents a forced vibration system with respect to W. Therefore, as is known from the theory of forced vibration systems, if J, M, and D are appropriately determined, W will converge quickly to "0" without vibrating.

From this, by using the present invention, the update amount ΔW corresponding to W can be made to approach "0" quickly as in a forced vibration system, and the error vector can be made to approach "0" quickly.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram showing the principle of the basic unit of the present invention; FIG. 2 is a diagram showing the principle of the hierarchical network; FIG. 3 is an explanatory diagram of the back propagation method; FIG. 4 is an explanatory diagram of data used in specific learning processing; FIG. 5 is an explanatory diagram of the number of learning times in the prior art; FIG. 6 is a diagram showing the principle of the present invention; FIG. 7 is a flowchart showing the learning algorithm of the present invention; FIG. 8 is an explanatory diagram of the number of learning times in the present invention; FIG. 9 is an explanatory diagram of the amount of weight update in the present invention; 16 is a diagram showing the principle of the analog neuron processor shown in FIG. 15A; FIG. 17 is a block diagram of one embodiment of the basic unit of the present invention; FIG. 18A is a conceptual diagram of a hierarchical neural network; FIG. 18B is a conceptual diagram of a hierarchical neural network according to the present invention; FIG. 19 is a block diagram of one embodiment in which a hierarchical network is constructed using a neurocomputer of the present invention; FIG. 20 is a specific block diagram of the embodiment shown in FIG. 19; FIG. 21 is a system configuration diagram of the main control circuit; FIGS. 22A and 22B are timing diagrams of signal processing in the embodiment shown in FIGS. 19 and 20; FIG. 27A is a specific circuit diagram of a master control block; FIG. 27B is a diagram showing the structure of a control pattern memory and a microcode memory; FIG. 28A is a diagram showing a method of filling data into a weight data memory; FIG. 28B is a specific configuration diagram of a weight data memory; FIG. 29 is a specific circuit diagram of a daisy chain circuit; FIG. 30 is a specific circuit diagram of a max value node circuit; FIG. 31A is a conceptual diagram explaining a feedback network; FIG. 31B is an explanatory diagram of a feedback network configured by a neurocomputer of the present invention; FIG. 32 is a block diagram of one embodiment of a feedback network; FIGS. 33A and 33B are timing diagrams showing signal processing of the embodiment shown in FIG. 32; and FIG. 34 is a diagram showing a process performed by a neurocomputer according to the present invention.
35A and 35B are timing diagrams showing signal processing in the embodiment shown in FIG. 34. FIG.

BEST MODE FOR CARRYING OUT THE INVENTION A flowchart of the weight update rules for realizing the present invention is shown in Fig. 7. The learning processing method of the present invention will be described in detail below with reference to this flowchart.

When a weight learning request is made, initial values are set to the weights of the internal connections as shown in step S1. This initialization process is determined according to random numbers as in the conventional method.
This is to avoid the fact that if the weight values are all the same or symmetrical with respect to the basic unit 1, no weight changes will occur in the back propagation method, and learning will not progress.

Next, in step S2, one pair of an input signal and a teacher signal registered as a learning pattern is selected.
In step S3, the selected input signal pattern is input to the basic unit 1 of the input layer of the hierarchical network 10, and the output signal from the basic unit 1 of the output layer, which will be output in accordance with the data conversion function of the hierarchical _network 10, is calculated.
Since "y _{pj " is defined as "the output from the j-th unit in the j-th layer (output layer) in response to an input signal of the j-th pattern," the processing of step S3 determines the output "y pj} " from each basic unit 1 constituting the output layer. Furthermore, in step S3, processing is also performed to simultaneously determine "y _ph ", the output of the h-th layer (input layer) and "y _pi ", the output of the i-th layer (intermediate layer) that will be output in response to the input of the selected input signal.

In this way, when the output signal "y _pj " corresponding to the input signal of the learning pattern is obtained, in the next step S4, the error "δ _pj " between this "y _pj " and the selected teacher signal is obtained. That is, j for the input signal of the pth pattern is
If the teacher signal to the jth unit of the layer is represented as "d _pj ", then
In step S4, y _pj - _{d pj} = δ _pj is calculated. Then, in the next step S5,
Using "δ _pj " calculated in S4 and "y _ph ", "y _pi " and "y _pj " calculated in step S3, according to equation (11), and, according to equation (14), _ypi (1- _ypj ) _yph is calculated. That is, the degree to which the error vector " _Ep " defined by equation (7) changes in response to the change in weight is found.

In step S2, if it is confirmed that the processing of steps S3 to S5 has been completed for the input signals of all the prepared learning patterns, the process proceeds to step S6, where the processing of steps S7 to S9 described below is executed to update the weight values from their initial values.

First, in step S7, using the value obtained in step S5,
(15) According to the formula, and, according to equation (16), In other words, the amount of change in "E", which is the sum of the error vectors "E _P " for all input signals, in response to the change in weight is calculated. Then, in the following step S8, the weight update amount ΔW _ji (t) that reduces the sum of the error vectors "E" is calculated according to the following formula:
and ΔW _ih (t).

Here, β is the same control parameter as ε and α, and t represents the number of updates as described above.

As is clear from this calculation formula, the weight update amount adopted in the present invention differs from the prior art described in equations (19) and (20) in that a new term, "-βΔW(t-2)", is added. This term means that the current update amount is determined not only based on the update amount in the previous update cycle, but also based on the update amount in the update cycle before that. The reason for this is, as mentioned above, that by configuring the back-propagation method using a forced vibration system model, it is possible to achieve weight convergence in a shorter time.

In step S8, the weight update amounts ΔW _ji (t) and ΔW
Once _ih (t') has been calculated, in the next step S9, new weight values are calculated in accordance with the following equations: W _ji (t) = W _ji (t-1) + ΔW _ji (t) W _ih (t) = W _ih (t-1) + ΔW _ih (t) These calculated weight values are set as the weights of the internal connections of the hierarchical network 10. Then, in the next step S10, it is determined whether or not the newly set weights have brought the error vector sum "E" to an acceptable level, and if it is determined that it is, the weight learning process is terminated, but if it is determined that it is still not within the acceptable range, the process returns to step S2 and continues the weight update process.

As described above, in the back propagation method, the present invention is characterized in that when learning weights so as to reduce the sum of error vectors "E", the update values of the weights are determined by taking into account the update amount in the update cycle before last in addition to the update amount in the previous update cycle.

Figure 8 shows the number of learning iterations of the present invention when using the learning pattern shown in Figure 4, compared with the number of learning iterations of the prior art shown in Figure 5. Here, β is set to 0.6, and other conditions are the same as those explained in Figure 5. As is clear from this figure, by using the present invention, the desired weights can be obtained with significantly fewer learning iterations than the prior art. Moreover, since the sum of the error vectors can be brought closer to "0" than the prior art, more accurate data processing can be achieved.

Of the changes in the weight update amount in the learning process shown in Figure 8, those related to the present invention are shown in Figure 9, and those related to the prior art are shown in Figure 10. From these two figures, it can be seen that in the present invention, after the weight update amount has changed significantly, it remains at "0" for a while.
The weight update amount becoming "0" indicates a convergence state, so
From these figures, it can be seen that the present invention attempts to achieve faster weight convergence.

The applicant has conducted various simulations and confirmed that when β is set to a positive value, the weight value converges more quickly than when it is set to a negative value. It has also been confirmed that even when β is set to a negative value, the convergence is much faster than in the prior art.

Next, an implementation of the weight learning means will be described.

11 is a diagram showing the configuration of an embodiment of the weight learning means of the present invention. The learning algorithm of the present invention is as shown in equation (24), where the weight update amount ΔW is calculated by multiplying the previous weight update amount ΔW by the (t-1) learning parameter α, and then partially differentiating the error E with respect to the weight, i.e., the partial differentiation of the error with respect to the weight, and multiplying it by the learning constant -
In the conventional method, weight update ΔW(t-2) is multiplied by ε, and the weight is expressed as the sum of the weight update amount ΔW(t-2) from the previous iteration multiplied by the learning parameter -β. This learning algorithm is executed by the weight learning means shown in Figure 11. In the figure, reference numeral 61 denotes a weight data memory for storing weight data, 62 denotes a weight data memory for storing the current weight update amount ΔW, 63 denotes a weight data memory for storing the previous weight update amount ΔW(t-1), and 64 denotes a weight data memory for storing the weight update amount ΔW(t-2) from the previous iteration. Reference numeral 65 denotes a parameter register for storing the learning constant -ε, 66 denotes a parameter register for storing the learning parameter α representing the learning speed coefficient, and 67 denotes a parameter register for storing the learning parameter -β.
The multiplier 68 calculates the weight update amount from the previous time, ΔW(t-2) and
β is multiplied by a multiplier, and 69 is the amount of update of the previous weight data ΔW
A multiplier 70 multiplies (t-1) by a learning parameter α, and a multiplier 70 multiplies the learning parameter ε by the current weight update amount ΔW.
The adder 71 adds the results of these multipliers. The output of the adder 71 is the weight update amount ΔW, so the weight difference W(t) - W(t-1)
Therefore, since the current weight data is stored in the weight data memory 61, the new weight data W(t) is obtained by reading out this content and adding the content W(t-1) and the output ΔW(t) of the adder 71. This addition is performed by the adder 72. The clear signal 81 is a signal for clearing the weight data memories 62, 63, and 64. The shift signal shifts the updated weight data update amount ΔW from the weight data memory 62 to 63 to make it the previous update amount for the next step, and further shifts the previous weight data update amount ΔW to make it the update amount before last ΔW(t-2).
This is a control signal for shifting W(t-1) from the weight data memory 63 to 64.
The weight selection signal 83 for accessing the weight data memory 61 is an address signal consisting of information such as the layer number of the hierarchical network, the unit number within the layer, and the connection number of the connection edge input to that unit. A weight read signal is given to the weight data memory 61, and it controls the reading of the specified weight data. The current weight W(t) is stored in the weight data memory 61.
is a signal with weight W(t) 85 read out from
In the learning algorithm, an initial value for a specific weight must be written into the weight data memory 61, and the control signal for this initial value setting signal is 86. In other words, it is the initial value write signal. 87 is the data for the initial value of the weight data at that time, and is the write data. The weight update signal 90 shown at the bottom of Figure 11 is a timing signal, and signals with different timings are generated for , , , and by the delay indicated by the symbol D. These signals are given pulses of the signal , , , to multipliers 70, 69, and 68, respectively, and multiplied, and after the multiplication is completed, adder 71 is executed at the timing of the pulse output after the delay time of delay circuit 73. Furthermore, a pulse , , is output after the delay time of delay circuit 24, and at this point adder
The addition operation of 72 is performed and the weight update amount is written to the weight data memory 62. Then, after the delay time of the delay circuit 75, a pulse signal is output, and this signal causes the result of adding the weight update amount ΔW(t) and the weight data W(t−1) to be written to the weight data memory 61 as the next weight data W(t).

The operations of the signals and components of the weight learning means shown in FIG. 11 explained above will be summarized below.

In the interface with the outside, the clear signal 81 is
The weight data memories 62, 63, and 64 are cleared to zero.

Shift signal 82 copies all of the contents of weight data memory 62 to weight data memory 63 and all of the contents of weight data memory 63 to weight data memory 64 .

The weight selection signal 63 specifies one of the weights based on the layer number, unit number, and connection number.

The weight read signal 84 reads out the data of the specified weight in the weight data memory 61 .

The weight 85 is weight data read out from the weight data memory 61 .

The initial value setting signal 86 is a signal for writing an initial value to the specified weight in the weight data memory 61 .

The initial value 87 is a data signal for writing an initial value into the specified weight in the weight data memory 61 .

The partial derivative 88 is the data ∂E/∂W calculated by the network and used to update the weights.

The learning parameters 89 are −ε, α, and −β, and are held in registers 65, 66, and 67, respectively.

The weight update signal 90 indicates the timing for calculating new values of the specified weights in the weight data memories 61 and 62.
and the timing of writing to the weight data memories 62 and 61.

In the internal module, weight data memory W(t)
61 is addressed by a weight selection signal 83, and outputs the value of the weight specified by a weight read signal 84 as a weight 85. It is also addressed by a weight selection signal 83, and stores an initial value 87 as a specified weight by an initial value setting signal 86. It is also addressed by a weight selection signal 83, and stores a delay 75.
A weight update signal from causes the output of adder 72 to be stored as the specified weight.

The weight data memory W(t) 62 has its contents all set to 0 by a clear signal 81 or the like, is addressed by a weight selection signal 83, and stores the output of the adder 71 as a specified weight by a weight update signal from a delay 74.

The weight data memory ΔW(t−1) 63 has all its contents set to 0 by the clear signal 81 .

It is addressed by weight select signal 83 and outputs the updated value of the specified weight to multiplier 69 .

The weight data memory ΔW(t−2) 64 has all its contents set to 0 by a clear signal 81 , is addressed by a weight selection signal 83 , and outputs the specified weight value to the multiplier 68 .

The adder 72 adds the output from the weight data memory 61 and the output from the adder 71 in response to a weight update signal from the delay 74, and outputs the result.

An adder 71 adds the outputs of the multipliers 68, 69, and 70 in response to a weight update signal from a delay 73 and outputs the result.

In response to the input of the weight update signal 90, the multiplier 68 multiplies the weight update amount ΔW(t−2) output from the weight data memory 64 by −β held in the parameter register 67, and outputs the result.

In response to the input of the weight update signal 90, the multiplier 69 multiplies the weight update amount (t-1) output from the weight data memory 63 by α held in the parameter register 66, and outputs the result.

Upon receiving the weight update signal 90, the multiplier 70 multiplies the partial differential 88 by -ε held in the parameter register 65 and outputs the result.

The parameter register 65 stores −ε, which is one of the inputs of the learning parameter 89 .

The parameter register 66 stores α, which is one of the inputs of the learning parameters 89 .

The parameter register 67 stores −β, which is one of the inputs of the learning parameters 89 .

The delay 73 receives the weight update signal 90 and generates a signal delayed by one timing.

The delay 74 receives the output of the delay 73 and generates a signal delayed by one timing.

The delay 75 receives the output of the delay 74 and generates a signal delayed by one timing.

The shift means 91 copies the entire contents of the weight data memory 62 to the weight data memory 63 in response to the shift signal 82 .

The shift means 92 copies the entire contents of the weight data memory 63 to the weight data memory 64 in response to the shift signal 82 .

The operation of the weight update means will be explained below. The operation of the weight update means is to initialize, call weights, and update weights. At the time of initialization, by turning on the clear signal 1, all contents of the weight data memories 62, 63, 64 are set to 0. Next, after setting the weight selection signal 83 and the initial value 87, by turning on the initial value setting signal 86, the weight contents specified by the weight selection signal 83 in the weight data memory 61 are set to the value given by the initial value 87. By performing this for all layers, units, and connections, all contents in the weight data memory 61 are initialized. Furthermore, -ε,
By giving α and −β, parameter registers 20 and 2
Store −ε, α, and −β in 1,22.

The weights are then updated as follows:

After setting the weight selection signal 83 and partial differential 88, turning on the weight update signal 90 causes the following actions to occur:
represents timing) A multiplier 68 obtains the product −βΔW(t−2) of the weight specified in the weight data memory 64 and the value of the parameter register 67.

A multiplier 69 calculates the product αΔW(t−1) of the weight specified in the weight data memory 63 and the value of the parameter register 66 .

Multiplier 70 multiplies the value of partial differential 88 by the value of parameter register 65. Ask for.

Adder 70 sums the outputs of multipliers 68, 69, and 70. This is the new ΔW(t).

At this timing, the weight read signal is sent externally.
Gives 84.

The adder 72 calculates the sum of the specified weight in the weight data memory 61 and the output of the adder 71, W(t-1)+ΔW(t), i.e., W
(t) is found.

The output of the adder 71, that is, ΔW(t), is written to a specified location in the weight data memory 62.

The output of the adder 72, that is, the new weight W(t), is written to a specified location in the weight data memory 61.

By performing the above operation for all layers, units, and connections, the contents of the weight data memories 61 and 62 are updated. By turning on the shift signal 82, the shift means
91 and 92 copy the entire contents of the weight data memories 62 and 63 to the weight data memories 63 and 64 .

12 is a diagram showing the configuration of an embodiment of the weight data memories 61, 62, 63, and 64 of the present invention. The weight data memory is composed of a random access memory (RAM) 93, and when a write signal is activated, write data is stored at a specified address, and when a read signal is activated, weight data already stored in RAM 93 is read from the specified address and output. In such RAM 93, an address signal is applied via a selector 94, i.e., a decoder.
An address register 95 is provided at the input of the selector 94, and as shown in the figure, information on the layer number of the hierarchical network, the unit number within the layer, and the connection number of the connection edge connected to that unit is set in the address register 95, whereby weight data specified by the layer number, unit number, and connection number is written to or read from the RAM 93. The contents of the weight data are the weight data W(t) itself in the case of 61, and the weight update amount Δ
W(t), in the case of 63, the previous weight update amount ΔW(t-
1), in the case of 64, the weight update amount ΔW(t−2)
In this case, the weight value, weight update amount, previous weight update amount, and weight update amount before previous weight update amount of the same connection edge are stored in the same address for the same connection edge, so as shown in the figure, the weight selection numbers such as layer number, unit number, and connection number are all common address information, and as shown in Figure 11, each signal is connected by a common address line. Such weight learning means uses a clear signal,
A shift signal, a weight selection signal, a weight read signal, an initial value setting signal, an initial value, a learning parameter, a weight update signal, and a partial differential ∂E/∂W of the error with respect to the weight are input from the outside,
The output is the weight W read from the weight data memory 61.
(t).

Fig. 13 shows an embodiment of the system configuration of the present invention. In the figure, 96 is the weight learning means described above (5 in Fig. 6).
10 and 11). 97 is a learning network, 98 is an input signal holding unit that holds an input signal to be given to the input layer of the network, and 99 is a teacher signal holding unit that gives a teacher signal to the error calculation unit. The input signal holding unit 98 and the teacher signal holding unit 99 constitute a learning pattern holding unit 100. The output of the network is given to an error calculation unit 101, which calculates the error with respect to the teacher signal from the teacher signal holding unit 99 and calculates the partial differential ∂E/
The output ∂W is given to the weight learning means 96. The entire system is controlled by a main control circuit 102.
The main control circuit outputs a control signal to the weight learning means 96, an error calculation signal to the error calculation means 101, an error clear signal, a partial differential selection signal, an execution signal to the network 97, and a pattern selection signal to the signal holding section.

In this system, the main control circuit 102 operates as follows. Fig. 14 is a flow chart explaining the operation of the main control circuit 101. The overall flow is an initial setting and learning process for updating weights. At the initial setting, first the clear signal 81 is turned on (S1) and all weight data memories 61, 62, 63, 64 are cleared.
4 is cleared (S12). Then, the weight selection signal 83
After setting the initial value 87, the initial value setting signal 86 is turned on, thereby setting the initial value 87 at the address specified by the weight selection signal in the weight data memory 61. This is done for all the connections of the layer units. If the weight read signal 84 is turned on at this time, it is possible to read out the specified weight information (S13 to S16).

In S17, the learning parameters are provided, and furthermore, the main control circuit turns on the error clear signal (S18) to set the current error in the error calculation means 100 to 0. Then, the learning process begins. In learning, the main control circuit 102 first provides an input signal to the network 97, and then sets the pattern selection signal and the execution signal to obtain the output of the network 97 (S20, S22).
1). In this case, initial weight data is provided to the network from the weight learning means 96. The network 97 then provides the output signal obtained through the input layer, hidden layer, and output layer to the error calculation means. The main control circuit 102 outputs the teacher signal from the teacher signal holding unit 99, which has been set via the pattern selection signal, and provides the teacher signal to the error calculation means 101. The error calculation means 101 then calculates the error and calculates the partial differential (S22). This is the value of ∂E/∂W, and is provided to the weight learning means 96 via the partial differential 88. The weight learning means 96 then performs the operation for updating the weights. The main control circuit 102 sets the weight selection signal, i.e., the weight selection signal and partial differential selection signal, to set the address signal for the weight memory and the partial differential value from the error calculation means to the weight learning means (S24, S25).

Next, a weight update signal 90 is applied and multiplied as shown in Figure 11, and addition for updating the weight data and control of writing of the weight data are performed (S16). Then, the weight read signal 84 is turned on and the newly updated weight data W(t) is stored in the weight data memory 61. This operation is performed for the weights of all connection edges. After the weight updating for all nets is completed (S23), the main control circuit 102 sets the shift signal 82 (S27) and the weight data update amounts, i.e., ΔW(t) and ΔW(t-1), are respectively set to ΔW(t)
A shift is performed so that the contents of the same address are written to the same address in the weight data memory 64 for ΔW(t-1) and the weight data memory 64 for ΔW(t-2).
When the weight data memory copy operation is completed in this way, the process proceeds to S28 to check whether learning is complete. If learning is complete, the process ends. However, if learning is not complete, the process returns to the start point of the repeated learning process via the loop line in the flowchart and the same operation is performed.

The new learning algorithm of the present invention explained above (7th
In the analog neuron processor network (Fig. 11), the feedforward propagation of signals is performed using an analog neuron processor network, which will be described later, and the backpropagation of signal errors is mainly performed by software in the host computer that controls the analog neuron processor network, and the weight update is performed using the hardware weight learning means shown in Fig. 11. In this case, the weight data memory 61 in Fig. 11 corresponds to the weight memories (150e, 185, 186, Fig. 28) which will be described later,
The read data is serial. The other weight data memories 62, 63, and 64 are also serial read/write type memories that use the same storage method as the weight data memory 61, storing data at consecutive addresses from each bit of the weight data.
Of course, 3,64 may be configured with parallel read/write type memory.

It should be noted that both the signal error backpropagation and weight update parts of the learning algorithm (FIG. 7) of the present invention may be implemented in software within a host computer that controls the neuron processor network described below.

The Analog Neuron Processor (ANP) and its hierarchical network used to implement the learning algorithm of the present invention will now be described in detail.

Figure 15A is a schematic diagram of a dual in-line package of a neuron chip in the case where the basic units 1-i, 1-h, and 1-j in Figure 6 are configured with an analog neuroprocessor (ANP) 105 according to the present invention. This is called B4442 and executes neuron model processing. The internal threshold processing unit is a model replaced with a sigmoid function.
The analog neurochip is called an ANP, and is a device that inputs and outputs analog data. Figure 15B shows the ANP of the present invention.
As shown in FIG. 15B, ANP105 is connected between analog bus B1 and analog bus B2.
ANP 105 consists of an analog multiplier 106 that multiplies the input analog signal by a weight, an analog adder 107 that calculates the sum of the products, a sample/hold unit 108 that holds the sum, and a nonlinear function unit 109 that outputs the value of a sigmoid function. The terminals of the ANP 105 in Figure 15A are explained below. The ANP 105 is internally composed of an analog circuit unit and a digital circuit unit. The +-6 volt terminal is the power supply terminal that supplies power to the operational amplifier in the analog circuit unit. _{D in} and D _out are terminals for the analog input and output signals. AGND is the ground terminal for the analog circuit unit. The Rt+ and Rt- terminals are terminals for the external resistor R of the integrator circuit in the analog circuit unit, and the Ct+ and Ct- terminals are terminals for the external capacitor C of the integrator circuit. DGND is the ground terminal for the digital circuit unit. +5 volts is the power supply terminal for the digital circuit unit. RST is a reset signal terminal that resets the charge of the capacitor in the integrator circuit. CSI and CSO are input/output terminals for daisy chain control signals, OC is the terminal for offset cancellation control signals, S/H is the terminal for sample/hold control signals, S
YNC is a synchronous signal terminal for processing each layer, DCLK is a basic clock signal terminal for processing analog input signals,
WCLK is a clock terminal for taking in digital weight data, and WD is a terminal for inputting digital weight data in bit serial format.

FIG. 16 shows the analog neuroprocessor (AN) of the present invention.
P) principle configuration diagram.

Analog input signals sent in a time-division manner from separate ANPs (not shown) are input from the analog bus B1 to the analog multiplier 106 in the ANP 105, and this analog multiplier 106 multiplies the analog input signals by the digital weight data WD that has been input bit-serial via the shift register 114 and then converted to serial-parallel, to obtain a product signal that indicates the product of the analog input signals and the digital weight data. The next analog adder 107 is a Miller integrator circuit consisting of an external resistor R and capacitor C, and is connected to the analog bus B1 and the multiple ANPs (A
The sum of the product signals obtained from the analog input signal sent in a time-division manner from the sample/hold unit 108 (the location where the NP exists is called a node) and the analog input signal for threshold sent from the dummy node is calculated.
After holding the product signal for the desired time,
The sampled/held output is then converted via a nonlinear function unit 109. The output control unit 113 delays the signal for a predetermined time under the control of a sequence generator 116, and then outputs an analog output signal D _out to an analog bus B2. The sequence generator 116 also generates control signals to be supplied internally. The transfer control unit 115 mainly controls the phase of the control signals so that each switch connecting the analog circuit unit and digital circuit unit within the ANP is turned on or off reliably. In particular, when a first switch is turned on and a second switch is turned off, the phase of the control signal is controlled so that the two switches are not turned on simultaneously.

The sequence generator 116 generates a reset signal RS
It inputs T, DCLK, WCLK, SYNC, S/H, OC, and CSI from the master control block (described later) and outputs CSO, generating the internal control signals for the ANP.

In neural networks, high-speed calculations are required through simultaneous processing. In this invention, time-shared data is used, and in a steady state, each ANP performs simultaneous processing in a pipelined manner. In an ideal neural network, neurons require interconnections with each other, but if we were to try to realize a system in this state, the number of wires would increase. Therefore, in this invention, time-shared data is used, so the processing time for product-sum calculations within each ANP increases, but
By arranging the chips in parallel vertically, i.e. in the same layer direction, the processing time is improved by simultaneous processing of the ANPs that make up the neurochips in the layer. In addition, pipeline processing is possible in each layer, which also reduces the processing time. For example, when an input comes into each of three neurochips connected to an analog bus, all three inputs come in at the same time, and each ANP generates a product of the analog voltage and a weight in parallel with all three, and stores it as a charge in the integrator capacitor. Then, in the next time section, each ANP generates a product of the analog input on the same analog bus.
The product of the weight and the analog input signal is added to the capacitor of the integrator in the previous time domain. After the sum of the products of the weights and the analog input signals from all the ANPs in the previous stage is generated, the sum is sampled and held. Then, it is output through a sigmoid function,
This is output when the CSI control signal is input. Then, when the output is completed, CSI falls, and after a certain time delay, C
The SO is activated to give the right to use the output bus to the ANP consisting of the adjacent neurochip in the same layer.

The present invention will be described in detail below with reference to examples.

17 is a diagram showing the configuration of an embodiment of a basic unit that is a neurochip. In the figure, a multiplication unit 106, an addition unit 107, a threshold processing unit
109 is an execution unit of the continuous neuron model, and in this embodiment, an output holding unit 117 is also included. Specifically, if the multiple inputs connected to the basic unit 105 are Yi and the weights set corresponding to each connection are Wi, then the multiplication unit 10
The adder 106 calculates Yi·Wi, and the adder 107 calculates X=ΣYi·Wi−θ, where θ is a threshold value. If the final output is Y, the threshold unit 109 calculates Y=1/(1+exp(−X)).

The value of "+1" input from the dummy node is "-θ"
The result "X-θ" is output by the adder 107. Therefore, the threshold unit 109 performs only the conversion using the S-curve.

The multiplication unit 106 is composed of a multiplication type D/A converter 106a, and receives an analog signal (input switch unit 118) from the basic unit 105 in the previous layer or from the circuit of a dummy node (to be described later).
107, an analog adder 107 configured by an adder 107, an integrator, and an analog adder 107
A holding circuit 1 holds the sum of a and the analog adder 107a.
The multiplying D/A converter 106a is configured by:
An analog input signal is input to the reference voltage terminal of the D/A converter, and each bit of the weight is input as a digital input signal to each digital input terminal. As a result,
The analog adder 107a generates the product of the analog input signal and the weight.
A new sum is calculated by adding the previously calculated sum and the sum held in the holding circuit 107b.
holds the sum calculated by the analog adder 107a and feeds back the held value to the analog adder 107a as the previous sum. These addition processes are performed in synchronization with the addition control signal output from the control circuit 103. The threshold unit 109 is composed of a nonlinear function generating circuit 109a, which is an analog function generating circuit,
It outputs a nonlinear signal such as a sigmoid function in response to an input. When the accumulation of the multiplication results, including the addition of the threshold value (-θ), is completed, the threshold value (-θ) is added to the sum value X held in the holding circuit 107b, and sigmoid function calculation processing is performed to obtain an analog output value Y. The output holding unit 117 is composed of a sample-and-hold circuit, and holds the output value Y of the analog signal from the nonlinear function generating circuit 109a, which is output to the basic unit 105 in the subsequent layer.

Also, 113 is an output switch unit, which receives an output control signal from the control circuit 103 and turns on for a certain period of time to process the final output held by the output holding unit 117 onto the analog bus B2, and 118 is an input switch unit, which receives an input control signal from the control circuit 103 and turns on when an analog output from the final output is sent from the basic unit 105 of the previous layer, thereby accepting input.
Numeral 9 denotes a weight holding unit, which is composed of a parallel-out shift register or the like, and when the gate of buffer 119a is opened (the weight input control signal from control circuit 103 is turned on) and the bit-serial weight signal sent from the weight memory, this weight signal is held as a bit-parallel weight required by multiplication unit 106. The bit-parallel weight is given in parallel to the multiplication unit when a multiplication control signal is given. Numeral 103 is a control circuit for the digital circuit unit, which generates an internal synchronization signal from an external synchronization signal and controls the internal analog processing functions.

With this configuration, the input and output of the basic unit 105 employing the signal processing configuration of FIG. 17 is realized by analog signals.

The multiplying D/A converter 106a may receive the weight information of a digital signal in parallel, or may receive the weight information in serial and then convert it into parallel. Alternatively, if the weight information is configured as an analog signal, the multiplying D/A converter 106a may receive the weight information in serial and then convert it into parallel.
Instead of a, an analog multiplier can be used.

The hierarchical structure shown in FIG. 18A is converted into the ANP shown in FIG.
When this is implemented using the above, as shown in Figure 18B, independent analog buses B1, B2, and B3 are provided between each layer, that is, between the input and hidden layer, between the hidden layer and the output layer, and at the output of the output layer. This results in a structure in which all vertical ANPs can be executed in parallel. The output of the output layer is provided with a sample-and-hold circuit.
Add SH.

FIG. 19 is a diagram showing the configuration of an embodiment of a hierarchical network using ANP.

In the figure, the input side circuit 120 corresponds to the input layer, 121 is an analog neuron processor ANP, i.e., a basic unit, which is a basic unit of the hierarchical network, 121-h is a plurality of basic units that constitute the intermediate layer, and in the case of three or more layers, 121-h is a plurality of basic units that constitute one or more intermediate layers, and 121-i is a single basic unit that constitutes the output layer.
One or more basic units 121-j are output layer circuits.
Electrical connections are made between the basic units 121-i and 121-j, and a hierarchical network indicated by 130 is constructed by weights set corresponding to each connection.

The basic unit 121 includes at least a multiplication unit 122 and an addition unit 123.
3, a threshold unit 124, and optionally an output holding unit 125. The multiplication unit 122 receives a plurality of inputs and weights for these inputs, performs multiplication, and outputs the result to an addition unit 123.
The multiplication unit 122 adds up the multiplication results for all basic units 121 in the previous layer, and the threshold unit 124 converts the summation result obtained by the addition unit 123 using a non-linear threshold function to calculate the final output. If there is an output holding unit 125, it holds the final output obtained by the threshold unit 124.
The input and output to the device are configured to be realized using analog signals.

Reference numeral 140 denotes an analog bus, which is a common line provided for electrical connection between the input layer and the first hidden layer (140a), between the hidden layers (140b), and between the last hidden layer and the output layer (140c). Reference numeral 150 denotes a main control circuit, which controls data transfer within the hierarchical network 130. This main control circuit 150 includes a drive unit selection means 151, a weight setting means 152, a threshold processing activation means 153, and an output value transmission means 154.

In this embodiment, the drive unit selection means 151 processes the basic units 121 of the previous layer to be selected in time series order. Then, the output value transmission means 154 synchronizes with this selection process and transmits the final output of the analog signal held by the selected basic unit 121 via the analog bus 140.
It processes the data so that it is output to the basic unit 121 in the subsequent layer in accordance with the time division transmission format.
The multiplication unit 122 of the basic unit 121 in the subsequent layer is a weight setting means
The basic unit 121 of the previous layer set by 152
The weight corresponding to the connection with the input is selected in order, and the weight is multiplied by the input, and the adder 123 sequentially adds the multiplication results obtained by the multiplier 122. When it is confirmed that all product-sum processing for the basic unit 121 of the previous layer has been completed, the threshold processing start means 153 performs processing to convert the basic unit 121 with a nonlinear threshold function f.
1 outputs f(X-θ). This latter layer becomes the new former layer, and the same process is repeated for the next latter layer. With this data transfer method, the output pattern corresponding to the input pattern is
It will be output from the output layer of the hierarchical network.

FIG. 20 shows an embodiment of a network configuration data processing device configured by connecting a plurality of basic units 121 in a hierarchical manner. In this embodiment, each basic unit 12
The weight given to 1 is temporarily stored outside the unit, and the CSI
This is an embodiment in which control of is given from the main control circuit. In this embodiment, electrical connections between the hierarchical structures of the hierarchical network are realized by a single common analog bus 140 (which may have identifiers a to c). Therefore, the final output value output from the output switch section 113 of the basic unit 121 (which becomes input to the basic unit 121 located in the subsequent layer) is configured to be output in analog signal output mode. In this embodiment, layers h and i are shown as intermediate layers, and the output layer is shown as layer j.

In the figure, 161 is a weight output circuit provided for each basic unit 121, which outputs weights for the weight holding section 119 in the basic unit 121, 162 is a weight signal line which connects the output of the weight output circuit 161 to the weight holding circuit 119,
Reference numeral 120 (which may be given an identifier a to n) denotes an input circuit provided according to the number of input signals, which outputs an initial signal that becomes an input pattern corresponding to the input layer of the hierarchical network. Reference numeral 164 (which may be given an identifier a to d) denotes a synchronization control signal line including CSI, which transmits a synchronization control signal from the main control circuit 150 that controls data transfer to the weight output circuit 161, the input circuit 120, and the basic unit.
This is a signal line for transmitting the signal to the control circuit 103 of 121. This synchronization control signal line 164 is shown as a common line in the figure,
More specifically, each circuit is connected to the main control circuit 150 by a separate signal line.

21 shows a detailed system configuration of the main control circuit 150. In the figure, the main control circuit 150 includes at least an external bus interface 150a, a microcode memory 150b, a program sequencer 150c, a control pattern memory 150d, and a weight data memory 150e. The external bus interface circuit 150a is connected to a host computer 150y and an external storage device 150z via a main bus 150x, and receives operation instructions from the host computer 150y. The microcode memory 150b stores microcode that defines the operation of the program sequencer 150c. The program sequencer 150c controls addresses of a control pattern memory 150d and a weight data memory 150e in accordance with the microcode in the microcode memory 150b. The control pattern memory 150d has output signal lines that are individually connected to the input side circuits 120, the intermediate layer, and the basic units 121 in the output layer, and in accordance with instructions from the program sequencer 150c, it controls one circuit of each group, or one basic unit 121 in each group, i.e., for each group of the input side circuits 120, the intermediate layer group, and the output layer group.
The weight data memory 150e outputs weights (digital data) to each weight output circuit 161 in accordance with instructions from the program sequencer 150c so that a weight is assigned to each basic unit 121 in synchronization with the time-division input signal. The host computer 150y includes an MPU and a main memory, and performs control to determine weights using a learning algorithm such as backpropagation, and to assign input pattern _Yi .
The external storage device stores data for constructing a neurocomputer.

Next, the operation process of the embodiment of FIG. 20 configured as above will be explained in accordance with the timing chart shown in FIG.

When a request for conversion into an output pattern is given from the host computer 150y via the main bus 150x, the main control circuit 15
The main control circuit 150 sequentially selects a plurality of input circuits 120 in a cyclical manner in time series by sending an output control signal to the input circuits 120 in a cyclical manner in time series. That is, the main control circuit 150 sequentially turns on the synchronous control signal line 164a for each input circuit 120 in order to sequentially select the input circuits 120 from the control pattern memory 150d in response to an instruction from the program sequencer 150c. That is, first, the input pattern given to the input circuit 120a is
In order to output the input pattern _Y1 to the analog bus 140a, only one of the n synchronous control signal lines 164, which is designated as 164a- ₁ in the figure, is turned on to open the gate of the input side circuit 120a, and the other synchronous control signal lines 164a are turned off.
(indicated as 164a-2 in the figure) is turned on, and the other synchronous control signal lines 164a are turned off.
The synchronous control signal line 164a is turned on and off until the input pattern _Yn of the input side circuit 120n is output to the analog bus 140a.
In order to provide a weight to each weight output circuit 161, the output of the weight data memory 150e is set for each weight output circuit 161 via the synchronous control signal line 164b in synchronization with the ON operation of each synchronous control signal line 164a.

In FIG. 22A, the synchronous control signal Y
i output control signal (i=1 to n), the input side circuit 120
The figure shows the process of cyclically selecting input patterns Yi in a time-series manner. Here, n is the number of input circuits 120. The input circuits 120 selected in this manner process analog signals Yi provided as input patterns onto the analog bus 140 (referred to as input layer analog bus 140a in the figure) provided between the input circuits 120 and the h layer. This input pattern is provided via a host computer 150y. Therefore, as shown in Figure 20A, analog signals _Yi are sequentially transmitted onto the input layer analog bus 140a in the same order as the number of input circuits 120, and the first input pattern Yi, followed by the next input pattern _Yi , and so on, are transmitted repeatedly.

When the multiplier 122 of each basic unit 121 of the h-th layer receives this analog signal Yi, it uses the weight Wi of the weight holding unit 119 set by the main control circuit 150 to execute the above-mentioned multiplication process (Yi·Wi). This weight Wi is calculated by the back propagation method and the seventh
The weights are already determined by the MPU according to the learning algorithm shown in the figure, but the weights are already determined by the MPU according to the back propagation method using the weight learning means shown in Figure 11.
It shall be determined in the

Therefore, as shown in FIG. 22B, the main control circuit 150 outputs the weight Wi according to the selected input side circuit 120 via the weight output circuit 161 in synchronization with the selection process of the input side circuit 120.
are set in the weight storage unit ₁₁₉ of each basic unit 121 in the h-layer.
and W _i are multiplied and summed with the previous sum of products.

The process of setting the weights to the basic units 121 can be realized in either analog or digital signal mode.

Since the weight is specified for each connection, it should be expressed as Wij (j is the basic unit number of the h layer) to be precise, but for ease of explanation it is expressed as Wi.

Here, the processing operation of the basic unit 121 will be explained according to the timing chart of signal processing of the basic unit 121 shown in Fig. 23. Note that here, the basic unit 121 (121a in the figure) in the hidden layer will be explained.

First, when the control circuit 103 receives a synchronization control signal given from the control pattern memory 150d of the main control circuit 150 via the synchronization control signal line 164 (164b-1 in the drawing), it turns on the input control signal (c) to make the input switch unit 118 conductive, and at the same time, it turns on the weight input control signal (d) to open the gate of the buffer 119a, and turns on the output switch unit 113.
At this time, the main control circuit 150 sequentially turns on the CSIs on the synchronous control signal line 164a in synchronization with the clock ( _a ).
In synchronization with this clock (a), the input side circuits 120a, 120, .
The input pattern signal _Yi held in the input signal generator 120n is applied to the multiplying D/A converter 106a via the analog bus 140 and the input switch section 118.

On the other hand, the main control circuit 150 also stores the weight data memory 150
The weight of e is given to the weight output circuit 161 via the synchronous control signal line 164b (164b-2 in the figure), and this weight (digital data) _Wi is sent to the weight holding unit 161 via the buffer 119a.
At this time, the output control signal (h ₁ ) of the CSI is turned on for one cycle of the clock (a), so the analog gate of the sample-and-hold circuit of the basic unit 121 is in an open state during this time, and the held analog value is output onto the analog bus 140b of the subsequent i-layer via the output switch unit 113.
When the weight _W1 of the digital value is stored in the multiplication D/A converter 106a, the multiplication control signal (e) is turned on.
multiplies the analog signal _Y1 given via the input switch unit 118 by the weight _W1 and outputs the multiplication result as an analog signal. Subsequently, the addition control signal (f) is turned on, so that the analog adder 107a, which is made up of an integrator, operates and multiplies the analog value held just before (which is initially cleared and is zero) by the multiplication type D/A converter 106a.
The result of the multiplication is added to the result of the multiplication, and the result of the addition is stored again in the sample-and-hold circuit 107b.

With the above operation, one bus cycle is completed, and in synchronization with the next clock (a), the switch unit 118 provides the input pattern _Y2 of the input side circuit 120b, and the weight output circuit 161
Since a weight _W2 corresponding to this input pattern _Y2 is given from the input side circuit 120n, the input pattern _Y2 is multiplied by the weight _W2 , and the result of this multiplication is added to the hold value of the sample-and-hold circuit 107b. This operation is repeated from this point onwards until the processing for the input pattern _Yn of the input side circuit 120n is completed. When the multiplication of the input patterns _Yn and _Wn is completed, the conversion control signal (g) is turned on, so that the value obtained by accumulating this multiplication result is input to the nonlinear function generating circuit 109a of the thresholding unit 124, and the corresponding Y value is held. That is, the thresholding unit 124 performs the above-mentioned calculation Y=1/(1-exp(-X-θ)), thereby obtaining the basic unit 121
The final output value Y, which is the final calculation output of the above, is calculated and held, and this result is output to the subsequent analog bus (140b) at the rising edge of the next output control signal ( _h1 ). Once this value Y is calculated, the accumulated value of the adder 107 is cleared by the input clear signal in synchronization with the next selection cycle of the input side circuit 120.

By performing the operations described above, each basic unit 121 obtains a final output value Y from the input pattern Y _i and the weight W _i .

Hereafter, the explanation will return to the configuration of the embodiment shown in Fig. 20. As described in detail with reference to Fig. 23, when the processing for the input patterns from all the input side circuits 120 is completed,
The main control circuit 150 again provides a synchronization control signal to each basic unit 121 (which may be identified by a to n), which then performs the same operation according to the input pattern Y _i and new weight W _i newly provided to the input side circuit 120.

On the other hand, the basic unit 12 of the h layer obtained in this way
The final output value Y of the input signal is held and is transmitted to the basic unit 12 of the i-th layer located in the next stage via the analog bus 140b by the same processing as that performed on the input side circuit 120.
That is, the main control circuit 150 transmits the synchronous control signal to the control circuit 103 of each of the basic units 121a to 121n in the h layer via the synchronous control signal line 1
64b (164b-1 in the figure) to output control signals h ₁ to h
₂ (FIG. 23) in a time series manner, the output switch section 3 of each basic unit 121a, 121b, etc.
6 are sequentially and cyclically turned on in a time series manner. As a result, the analog signals of the final output values held in each of the basic units 121a to 121n are output to the multiplication units of the basic units of the i-th layer.
Each basic unit in the i-layer executes the same processing operation as described above, and the basic unit 121 in the i-layer obtained by this processing is
The final output value Y of the output layer basic unit 121 is obtained by performing a similar time-division transmission process of the final output value Y of the output layer basic unit 121. That is, the main control circuit 150 controls each basic unit in the same manner via synchronization control signal lines 164c and 164d individually connected to each basic unit 121 of the intermediate layer and the output layer.

Fig. 24 is a block diagram of a neurocomputer of the present invention that realizes a hierarchical neural network. Analog neuron processors ANP1 to ANP5 are arranged in parallel in each layer from a neurochip, and analog buses (B1, B
In the figure, ANP1, 2, and 3 form the hidden layer, and ANP4 and 5 form the output layer.
On the input side, there are dizzy circuits 171 and 172 for inputting analog input signals in a timely manner. The circuits indicated by S/H are sample/hold circuits 173 and 174. ANP
Since each of 1 to 5 requires a control logic signal, many control signal lines are sent from the master control block (MCB) 181 to each layer.
DCLK is given to the daisy-chain circuits 171 and 172 on the input side of all ANPs, and serves as the basic clock for analog processing. The weighted clock WLCK is also given to all ANPs and the daisy-chain circuits 171 and 172 on the input side.
72, which is a high-speed clock for weight data. Weight data is input from weight memory blocks 185 and 186 to ANP4 and 5 and ANP1, 2 and 3 in synchronization with the weight clock WCLK. Also, the synchronization signal SYNC1 is a layer synchronization clock given to the ANPs of the hidden layer, and the synchronization signal SYNC2 is a layer synchronization clock given to the ANPs of the output layer. SH1 and OC1 are the synchronization clocks given to the ANPs of the hidden layer.
SH2 and OC2 are sample/hold control signals and offset control signals for the ANP of the output layer.

The daisy chain circuits 171 and 172 on the left side are input side circuits that correspond to the input layer. To realize the input nodes, that is, the neurons in the input layer, analog input signals given from analog input ports 0 and 1 must be input into the circuit at the same timing as the ANP outputs the analog signals in a time-division manner. In other words, from the perspective of the output layer,
The basic operation of ANPs 4 and 5 in the output layer is to receive analog signals from ANPs 1, 2, and 3 in the previous hidden layer via analog bus B2 in a time-sharing manner. The same relationship must exist between the hidden layer and the input layer. From the perspective of the ANPs in the hidden layer, the ANP in the input layer must appear to precede it. This imposes a constraint that the ANPs in the hidden layer must output analog signals to analog bus B2 in the same manner as they output analog input signals from analog input ports 0 and 1 to analog bus B1 according to a set rule. That is, input signals from analog input ports 0 and 1 are transmitted to analog bus B1 in a time-sharing manner. The analog signal from analog input port 0 is transmitted to analog bus B1 at an appropriate timing, and the next analog input signal from analog input port 1 is transmitted to the same analog bus B1 at the same time. To synchronize this, a daisy-chain circuit 171 receives an input control signal CSI, which is output at a fixed timing, and after a fixed time, the circuit outputs an output control signal CSO. This CSI is output from CSO1 of the master control circuit 181. The daisy chain circuits 171 and 172 are a type of delay circuit. When each daisy chain circuit 171 receives an input control signal CSI from the master control 181, it passes the CSO signal to the next daisy chain circuit 172 adjacent to it in the vertical direction so that it outputs the analog output signal of analog input port 1. This operation is called daisy chain control.

When CSO1 of the master control circuit 181 rises, the switch 175 turns on, and the analog input signal of analog input port 0 held in the sample/hold circuit 173 is placed on the analog bus B1. Since CSO1 is the CSI of the daisy chain circuit 171, CSO rises a certain time after it falls. This is the CSI of the daisy chain circuit 172, and at the same time, it controls the switch 176 to turn it on, so the analog input signal of analog input port 1 held in the sample/hold circuit 174 is placed on the bus B1. This daisy chain control is necessary in this hierarchical system. In other words, if the analog input signal is output from analog input port 0 to the analog bus B1 via the sample/hold circuit 173, then the analog input signal is output from analog input port 1 to the sample/hold circuit 174.
The analog signal is then output to the same analog bus B1 via the
Looking at each neuron in the hidden layer, the analog input signal from analog input port 0 and the next analog input signal from analog input port 1 are input sequentially in a time-division manner.

Each of the daisy chain circuits 171 and 172 delays the input control signal CSI by a specific time to prevent bus contention on the analog bus B1, and then outputs the output control signal CSO.

In the middle layer, ANP1, which receives the output control signal CSO2 from the master control block 181 as CSI, outputs an analog signal, then passes CSO to ANP2 as CSI, and ANP2 then outputs. ANP3, which receives CSO from ANP2 as CSI, then outputs an analog signal. In short, ANP1, 2, and 3 output in this order, and the daisy-chain operation of the middle layer is completed. In parallel with this, the master control block 181, which manages all operations, gives CSO3 to ANP4 in the output layer, which then outputs, and after the output is complete, ANP4 passes CSO to ANP5.
ANP5 will output when given.

The outputs from ANP4 and ANP5 of the output layer are sampled and held in sample/hold circuits 177 and 178, respectively, by a CSO3 signal from a master control block 181 and a daisy-chain output control signal CSO from ANP4.
This output voltage is output as an analog output signal from analog output ports 0 and 1, and is also used as an analog multiplexer.
After being selected by 179, it is A/D converted by A/D converter 180,
The data is input to a digital control means that is composed of an MPU 182, a memory 183, and a communication interface 184.
For example, the output signal is compared with the teacher signal stored in the MPU during learning to check whether it is the desired output signal, and the weight data in the weight memory (described later) is changed based on the result. The max value node circuit 187 receives the dummy node control signals DCS1, DCS2 from the master control block 181.
S2 is applied to output enables 1 and 2, and the output terminals are connected to analog buses B1 and B2.

Fig. 25 is a timing diagram of the hierarchical neuron computer according to the embodiment shown in Fig. 24. The control signal lines are extracted and written for each layer. First, the data clock DCLK and the weight clock WCLK are the basic operating clocks.
All ANPs on the same layer and input side daisy-chain circuits 171, 172
Enter at the same time.

The weight clock WCLK is a serial synchronous pulse for sending the weight digital data serially and is a synchronous clock for reading the weight from the weight memory block. The timing at which the input data is taken in is determined by the respective control signals. First, in the timing chart of Figure 25, CSO1 is the master control block 18.
1, that is, the daisy chain control signal CSO1 output from the daisy chain circuit 171, that is, the daisy chain control signal CSI to the daisy chain circuit 171. In the daisy chain circuit 171, CSI causes the first analog input signal to be output from the analog input port 0 to the analog bus B1 via the sample/hold circuit SH173. That is, in the timing chart, the analog signal is output to the analog bus B1. At this moment,
A voltage is applied to the analog bus B1, and ANP1, ANP2, and ANP3 perform multiplication and accumulation operations on this analog signal in parallel.
passes through the daisy chain circuit 171, and the next CSI rises as shown in a predetermined time after CSO falls, and that CSI enters the daisy chain circuit 172. The next CSI is a control signal that enters the second daisy chain circuit 172 in the input layer. Then, CSI
While this is high, the analog input signal from analog input port 1 is sent to ANP1, ANP2, and ANP3 via the sample/hold circuit SH174.
The input is input to NP3, where a product-sum operation is performed. DCS1 from the master control block 181 is a control signal to the dummy node. In addition to the input, each layer also has a signal from the dummy node, so it has a form of (number of neuron nodes + 1) nodes, and there are two inputs in the input layer, but the AN of each hidden layer
From the perspective of P, it appears to be three inputs. Explaining this in terms of time, two CSIs and one DCS1 are control signals that make up one block. The input cycle is
It starts from I and ends at the input to the dummy of DCSI. The dummy node is a max value node circuit 187, which outputs a fixed threshold voltage to the analog bus while DSC1 is input. That is, as shown by DC
While this voltage is being output after SI rises, each ANP in the middle layer performs a multiplication and accumulation operation in the same way as with normal input,
That fixed voltage is added to the result of the multiplication and accumulation of the previous two analog input signals, i.e.
After the multiplication, an addition is performed. SYNC1 goes high on the falling edge of DCLK before CSO1 rises, and goes low on the next falling edge of DCLK after DCS1 rises. This is a signal that synchronizes the input layer. The analog input and weight data are multiplied while WCLK is being input. The sample/hold signal SH1 input to the ANP in the hidden layer has two peaks, M1 and M2. The product is taken just before the first peak, M1, and the sum is generated and held at the peak. Then, at the next peak, M2, the offset voltage _Vb is subtracted and the result is sampled and held. This process is repeated sequentially for all input analog signals until the product-sum calculation is complete. In this case, each ANP in the hidden layer, including the dummy, performs the product-sum operation three times. This completes one process for each ANP in the hidden layer, completing the addition of the products for the three inputs.

Also, in the timing chart, when DCLK is high immediately after DCS1 falls, the results of the multiplication and accumulation operations on the three signals from analog 2 output ports 0 and 1 and the dummy node are held in the capacitors of ANP1, 2, and 3. This operation is basically repeated, but the master control block determines when the output signal of ANP1 is output to analog bus B2 between the hidden layer and the output layer.
It is determined by the rising edge of the CSO2 signal sent from 181.

The offset control signal OC1 shown below SH1 is
Offset cancellation is performed inside the ANP. That is, each ANP is an analog circuit that includes an operational amplifier internally, and the circuit itself has an offset, so the control signal for canceling this offset is the OC signal. As shown in OC1, one pulse is output each time a multiply-and-accumulate operation is performed, and offset cancellation is performed internally. As shown in the timing chart, when CSO2 rises, the signal that was held in ANP1 is output from ANP1 to analog bus B2,
While CSO2 is high, ANP4 in the output layer performs a multiply-and-accumulate operation. The rising edge of CSO2 indicated by is the timing to output the result of the multiply-and-accumulate of the previous input result.

Next, the timing between the hidden layer and the output layer will be explained using Figures 25A and 25B.

In the figure, the daisy-chain control signal output from the intermediate layer, , , and the output from the output layer,
The analog signal that appears on the analog bus in synchronization with the output of the daisy chain control signal from the input layer described above, and the result of one processing cycle before appears for the analog signal that is input on the analog bus in synchronization with the output of the daisy chain control signal from the input layer described above. The execution of pipeline processing will be explained later, but
At the rising edge of CSO2 shown in the timing chart, the output of ANP1 is output. Looking at the signal of SH2 in the timing chart at the rising edge of CSO2 shown in, two pulses are output. The SH2 signal is input to the first ANP4 in the output layer in the block diagram of Figure 24. That is, at the two peak pulses of the SH2 signal, ANP
One sum operation is performed within 4. As shown in the figure, the hidden layer has three hidden layer neurons, ANP1, 2, and 3, but one dummy node is added by the max value node circuit 187, so it is assumed that there are four neurons in total. Therefore, the two peak pulses of the SH2 signal are output four times as seen from the part , and the analog signal of the hidden layer is input to ANP4 with these four sets of peak pulses of the SH2 signal, and the product sum is calculated. Naturally, this operation is performed at the same time as the ANP in the hidden layer is performing the product sum operation on the input signal, and this is pipeline processing. The signal below CSO2 is the CSO signal of ANP1 in the hidden layer, and this is the CSI for ANP2 in the same hidden layer. This is the part indicated by . Below that is the CSO of ANP2, and below that is the CSI of ANP3, which is . Below that is the C of ANP3
SO is shown, and below it is the dummy node CSI, which is a signal output from DCS2, the master control block. Looking at CSI, it is input in the order of , , , to ANP1, ANP2, ANP3 in the hidden layer, and then to the max value node circuit 187 of the dummy node. During this time, the SH2 signal outputs four pulse signals with two peaks. In other words, the neuron in the output layer of ANP4 adds four products of the input analog signal and the weight. When CSI is input to ANP1 at part , an analog signal is output from ANP1 to the analog bus between the hidden layer and the output layer, and this is input to ANP4. At this time, the corresponding weight data is input to ANP4, and the product is executed, added at the first peak of the SH2 signal, and sampled and held at the second peak. When this calculation is complete, ANP1
The CSO signal rises from this point, and this becomes the CSI for ANP2.
This is the state where the weight data and the data on the analog bus are multiplied and the sum is calculated. After a predetermined time has passed since the falling edge of , CSI to ANP3 goes high and a sum-of-products operation is performed in ANP4 as shown by . This sum-of-products operation is performed within ANP4, and at , a fixed voltage output from the max value node circuit 187 is
4, which is then added to the accumulated product sum stored inside.

The above operations are also performed in parallel for ANP5 in the output layer. This is simultaneous processing. The timing at which the result of the product-sum operation calculated by ANP4 is output to the analog bus B3 connected to the output layer is the rising edge of CSO3 output from the master control block 181. The control signal that the max value node circuit 187 uses to output to the analog bus B2 is DCS2, which corresponds to this. Up to this DCS2, the operation is up to the output of the calculation result in the intermediate layer.
The signals written below this one on the timing chart operate in a similar manner, and are signal pulses that dictate the operation of the output layer, which is connected in cascade to the hidden layer. When CSO3 rises, the result of the multiplication and accumulation operation calculated by ANP4 is output. Two signals, ANP4 and ANP5, are output from the output layer. For example, the rising edge of CSO2 is the signal that enters ANP1, and this rising edge is delayed from DCLK. This is because, when performing a multiplication operation between an analog input signal and digital weight data, the digital data is read in serially with WCLK, and the rising edge of CSO2 is delayed to take into account the time it takes to read the digital data, which is converted internally to parallel, and the time it takes for the analog input signal to reach the D/A converter, i.e., the multiplication processing unit. In other words, the delay at the beginning includes the time it takes to call the data, that is, to read the serial data. The data is set up until
The time that has passed since the rising edge of DCLK, that is, WC
The analog multiplication starts 16 cycles later with CS
This is eight WCLK cycles after O2 has started up.

Figure 26 is a timing chart showing the timing of reading in the digital weight data. In this figure, the master clock MCLK, the synchronization signal SYNC, the weight clock WCLK, the data clock DCLK, and the actual weight data WDATA are shown. The weight data WDATA is read out bit-serial from the weight memory, and 16 bits are input serially. S
is the sign bit, and B14 to B0 are the numeric bits.
In the figure, the B8, B7, and B6 portions of the weight data WDATA are shown enlarged at the bottom of the figure as they correspond to the weight clock WCLK. The weight clock WCLK has a period of 250 nsec and a duty ratio of 50%. An address is given to the weight memory after the propagation delay time of the address update counter inside the sequencer from the falling edge of WCLK. In other words, the address of bit n of the weight memory (RAM) is the weight data W
This is the address of the weight memory where bit 7 of DATA is stored. After this address is determined, bit 7 is read out after time tAA. The transition from bit 7 to bit 6 is determined by the transition to the next cycle of the weight clock.
Bit 6 is read out in the next cycle.
The 16 bits are input to the ANP, and the product of this with the analog voltage input to the ANP is calculated by the internal D/A converter, so the start of the analog voltage input is input long after the rising edge of the data clock DCLK. In other words, there is a time lag between the analog input voltage being input and reaching the D/A converter, so it is necessary to control this time and the time at which the digital weight data is set internally, and input the analog voltage so that the arrival time of the weight data exactly matches the arrival time of the analog.

For example, the analog input voltage rises from around weight data B7, and weight data B0 is input.
After that, it is necessary to control the time so that the multiplication with the analog value starts when all the weight data has been determined internally. Then, the addition is performed during the period when DCLK next goes low.

The operation time of the ANP is determined by the SYNC signal, WCLK, and data DCLK. The analog input voltage has a significant time error in the voltage arrival time from the input terminal of the ANP to the D/A converter that performs the multiplication with the digital weight data.
To allow for a margin, the rising edge of CSI begins later than the rising edge of DCLK.

27A is a block diagram of the master control block 181. The master control block 181 is a part that controls all control signals. The main components are an external bus interface circuit 200, a control pattern memory 201, a microprogram sequencer 202, and a microcode memory 203.
03, and an address creation unit 204. The external bus interface circuit 200 is an interface for connecting to an MPU or the like, and is connected to address lines 205, data lines 206, and control signal lines 207. An upper address comparison circuit 208 and a D-FF 209, which is a register, of the external bus interface circuit 200 decode an upper address given from an MPU or the like, and when the upper address is a predetermined address, set the lower address and data in the D-FFs 209 and 211, respectively, using a latch signal from a timing circuit 214 as a trigger.
The address and data are transmitted to bus drivers 210 and 212, respectively.
The address is input internally via the internal address bus and internal data bus. The address is used when referring to the microcode memory 203 and writing the microcode from the MPU side via the data bus. The lower address is also passed to the microprogram sequencer 202 via the bus driver 210 as a microcode address, and the lower address is used when writing the microcode from the MPU side via the data bus.
The control pattern memory 201 can be referenced at a specific address from the PU portion.

Data from the MPU or main memory is latched into the D-FF 211 via the data line 206, and then sent to the separate I/O RAM 213 in the microcode memory or the separate I/O RAM 215, 2 in the control pattern memory 201 via the bus driver 212.
16. When a data strobe signal from the MPU or memory is applied to the timing circuit 214 via the control signal line 207, an acknowledge signal is returned.
A timing circuit 214 controls the latch timing to the D-FFs 211 and 209 and the write timing to the microcode memory 203 and the control pattern memory 201 via the WR signal.

The complex control signal "1" and "0" patterns given to the neurochip as shown in the timing charts of Figs. 25A and 25B are stored for one cycle in the control pattern memory 201, and the pattern for one cycle is sent to the microprogram sequencer.
The control pattern is generated by reading it from the control pattern memory 201 under the control of 202. For example, the reset signal Reset, the data clock DCLK, the weight clock WCLK, CSO1, CSO2, and CSO3
Control signals such as SYNC1, SYNC2, SH1, SH2, OC1, and OC2 are read from the separate I/ORAM 215, and control information associated with the patterns, i.e., sequence control flags, are read from the second separate I/ORAM 216. For example, if the control pattern memory 20 stores a pattern of 1000110001, this is a "1,0" bit pattern.
If the address of the control pattern memory 201 is controlled so that a 0 bit pattern is repeated, this repetition of the pattern is read out from the control pattern memory 201. In other words, since the control signal patterns are very complicated, it is necessary to prepare these patterns in advance in this separate memory.
The structure is such that the bit patterns are stored in the I/ORAM 215, and the addresses of the separate I/ORAM 215 are specified under the control of the microprogram sequencer 202, thereby outputting the bit patterns in sequence. Therefore, several identical patterns are repeated, and how this repetition is realized is determined by address control. This pattern for one cycle is called the original pattern. In order to repeat the original pattern, it is necessary to feed back specific information from the control pattern memory 201 to the microprogram sequencer 202. That is, the sequencer control flag in the second separate I/ORAM 216 is used as a condition input to the microprogram sequencer 202.
By inputting into the microprogram sequencer
The microprogram sequencer 202 controls the first separate I/ORAM 215 to return to the first address where the original pattern is stored. This causes the original pattern to be repeated. That is, the microprogram sequencer 202 sequentially generates address signals to the separate I/ORAM 215 via the general-purpose port output line 202-1 until the condition is met. Normally, this address is incremented, but when the condition that the address has reached the end of the original pattern is met, the address is returned to the first address where the original pattern is stored. As a result, the specific pattern is repeated and a control pattern is output from the separate I/ORAM 215.

27B shows the relationship between the information in the memories 201 and 203 that control the master control block 181. In the figure, the control pattern memory 1 is a first separate I/ORAM.
The control pattern memory 2 corresponds to the second separate I/O RAM 215, and the control pattern memory 2 corresponds to the second separate I/O RAM 216. The microcode memory 203 stores the control code of the sequencer 202, and mainly stores the Jump
The p instruction and the Repeat instruction are stored in the control pattern memory 2. Looking in the direction of address increment, there is a Repeat instruction at a specific address, and the number of repetitions of pattern 1 in the control pattern memory according to this repeat instruction is stored in the corresponding address in the control pattern memory 2. For example, if it is "10", 10 repetitions will be executed. In this way, when the address increments and the Jump instruction in the microcode memory is reached, the second Jump in the microcode memory 203 jumps to 500H, and the P
When Pattern2 is repeated five times, the third Jump in the microcode memory 203 again outputs "100H".
and outputs Pattern 1. In this way, the original pattern is repeated and read out from the control pattern memory 1.

WCLK is generated in synchronization with the read clock of the address referencing this control pattern memory 201, and information is read from the weight memories 185 and 186 in synchronization with WCLK. Addresses for the weight memories 185 and 186 are accessed by address signals output from address 1 and address 2 of the address generation unit 204. Address 1 and address 2 are separated to correspond to the hidden layer and the output layer, respectively. The weight data to be provided to the ANP in the hidden layer is read from weight memory 185 specified by address 1, and the weight data for the ANP in the output layer is the content read from weight memory 186 specified by address 2. Since the contents of weight memories 185 and 186 are stored at each address, one bit at a time in the direction in which the address increases, a count control signal must be provided to address counters 217 and 218 from the microprogram sequencer 202. These addresses are incremented by the address counters 217 and 218 and provided as address signals to the weight memories 185 and 186 one after another via bus drivers 219 and 220. Then, a plurality of weight data are read out from the weight memories 185 and 186 .

The WCLK from the first separate I/ORAM 215 and the counter control signal from the microprogram sequence 202 are applied to AND circuits 221 and 222 in the address creation unit 204 .
When the counter control signal is high, the address counter is updated by WCLK, and bits 1 to 16 of WCLK increment the address counters 217 and 218. For the remaining bits 17 to 26 of WCLK, the counter control signal is set low to inhibit WCLK, stopping the increment of the address counters 217 and 218. Then, in synchronization with SYNC1 and SYNC2, counter reset signals are sent from the microprogram sequence 202 to AND circuits 221 and 222, respectively, to reset the address counters 217 and 218. This returns the addresses of the weight memories 185 and 186 to the starting addresses. The mode signal output from the master control block 181 is used to select between normal use of the weight memory, i.e., a mode in which the weight memory is disconnected from the MPU data bus and weight data is provided to the ANP, and a mode in which the weight memory is connected to the MPU data bus and the weight memory is referenced by the MPU.

The mode signal is generated by setting the lower bits of the data from the MPU in a flip-flop 224, triggered by an address signal generated by an AND circuit 223 that generates WR from one bit of the lower address and the write signal from the timing circuit 214. When this mode signal is 0, the weight memory is in normal use.

The write signal WR and one bit of the internal address bus are input to the clock terminal of a flip-flop 224 via an AND circuit 223, and the LSB of the internal data bus is input to the data terminal of the flip-flop 224.
The master control block 181 is checked to see if it is selected, and if it is, the lower address and data are taken into DFFs 209 and 211. This kind of interface operation is also performed for other devices connected to the MPU, but since the weight memory normally supplies weight data to the ANP, if it is connected directly to the MPU data bus, bus contention will occur. To prevent this,
When the LSB is captured by the flip-flop 224, the mode is set to 1, the weight memory is not chip selected as described below, and data is not generated on the data bus from the weight memory. At a predetermined timing, the internal address bus specifies an address of either the microcode memory 203 or the control pattern memory 201, and the desired data is written from the internal data bus to the accessed address. As a result,
The programs stored in the microprogram sequencer 202, the microcode memory 203, and the separate I/O RAM 216 are changed, or the control pattern stored in the separate I/O RAM 215 is changed.

Figure 28A shows the data storage configuration of the weight data memory 230. In this diagram, eight bits in the column direction represent 8-bit data information stored at the same address, and each bit is provided to ANP1, ANP2, ..., ANP8 from the bottom up. The addresses in the row direction are different, and as shown in the diagram, the addresses increase as you move to the left. Weight data consists of 16 bits, including the sign bit, and is stored from lowest to highest addresses. The MSB is the sign bit, and the other 15 bits are numeric bits. When the address from the microprogram sequencer 202 is incremented in synchronization with WCLK, one word of weight data, i.e., 16 bits, is read out sequentially from the MSB to the LSB. These weight data are simultaneously passed to eight ANPs. Because data is stored in this manner in the order of increasing addresses, an address counter for this weight data is required. That is, a counting control is required so that when one word of weight data is counted from the MSB to the LSB, it becomes one weight data. This control is also performed by the microprogram sequencer 202.

Figure 28B shows a concrete circuit of the weight memory blocks 185 and 186. The memory 230 is a RAM called MB8464A-70. The output is A
It is 8 bits corresponding to NP1 to ANP8. Basically, it is the bus signal line seen from the MPU bus and the master control block 18
1 uses either address 1 or 2 visible from 1. Addresses 1 and 2 are addresses 1 and 2 in Figure 19A mentioned above.
Addresses 1 and 2 are input in an incremented manner in synchronization with WCLK. Eight bits of data are read out simultaneously, and each bit is simultaneously applied to ANP1 to ANP8.

When the mode signal is 0, the weight memory 230 is chip-selected via the AND gate 233, and at this time, addresses 1 and 2 from the microprogram sequencer 202 become valid in the multiplexer 234. Then, weight data is sent from the weight memory 230 to ANP1 to ANP8. On the other hand, the inverter circuit 231
Since the output of the tristate bus transceiver 232 is high, the tristate bus transceiver 232 is disabled and the weight memory 230
The output of is not sent to the MPU.

When outputting to the MPU, the mode signal is set to 1, and the address decoder 23 receives appropriate address information from the MPU.
5 to chip select the memory 230, and
Address is given from PU. When mode signal is 1, MPU
The read/write direction is controlled by the read signal Read from the data line sent from the MPU via an AND gate 236.
Determined by Signal.

Next, the learning algorithm will be explained. Figure 28C is a flowchart of the learning algorithm called backpropagation used in the present invention. Learning proceeds as follows: Complete information to be learned is input from the MPU via an input control circuit (not shown) to the input of the neural network of the present invention, i.e., a hierarchical network composed of a set of ANPs. Then, the input signal passes through the input side circuit, intermediate layer and output layer, and is sent to the network output via an A/D converter before being given to the MPU. The learning algorithm exists in the main memory on the MPU side. Inside the MPU,
The teacher signal is taken from the main memory and the error between the network output and the teacher signal is checked. If the error is large, the MPU will move the network in the direction of producing the correct output.
The weight data, which represents the strength of the network connections, is changed. This weight data is applied to the ANP of each layer via the weight memory 230.

Figure 29 is a diagram of the input-side daisy chain circuits 173 and 174. In the figure, 240, 241, and 242 are D-type flip-flops. At the rising edge of the DCLK signal, data input to the D terminal is set, and output Q is set to 1. The first flip-flop 240 sets the CSI signal at the falling edge of DCLK. Then, at the next rising edge, the output signal is set to the second flip-flop 241.

The output is input to the D terminal of the third flip-flop 242. The clock signal that sets the input is the output of a 4-bit counter 243. The counter 243 is WCLK
It is triggered by the falling edge of DCLK. It is cleared by the falling edge of DCLK. Therefore, the counter 243 becomes all 0 at the falling edge of DCLK, and after eight falling edges of WCLK are input, the upper bit QD signal becomes high, which triggers the flip-flop 242 to output a high signal to CSO. When the output of the flip-flop 241 becomes 0, C
By this operation, CSI falls, and after a predetermined time equivalent to eight WCLK pulses has passed, CSO is output, resulting in a daisy-chain operation.

30 is a specific circuit diagram of the max value node circuit 187 which forms the neuron of the dummy node. In the figure, a resistor 250, Zener diodes 251 and 252, resistor 25
3. The voltage followers 254 and 255 are circuits that form a constant voltage. When a current flows from +12 volts to -12 volts through resistors 250 and 253 and Zener diodes 251 and 252, the inputs of the voltage followers 254 and 255 become +7 volts and -7 volts, respectively.
These voltages are fed to the voltage follower 254,
The two constant voltages are output through an output resistor 256 of 255. The analog switches 257-258 are connected to each other so as to extract these two constant voltages in a time-division manner.
64. When the T mode signal is 0, the constant voltage is given to the next voltage follower 265 via the analog switch 257. When T mode is 1, that is, in the test mode, the output is held to the analog ground by the analog switch 258, so that 0 volts is input to the voltage follower 265. In the test mode, the offset on the bus is notified to the MPU. The voltage follower 265 is enabled by the switch control of the output section. When the output enable is 1, the analog switch 26
When 0 is on, it acts as a voltage follower and its output is given, but at this time, no output is given to the dummy node output.
Conversely, when Output Enable is 0, a signal is output to the dummy node output. The analog switch 260 and its output are controlled by Output Enable 1 or 2, which is 0 enabled. That is, when Output Enable 1 or 2 is 0, a constant voltage is output to the dummy node output. The upper dummy node output is for the dummy node in the input layer, and the second is for the dummy node in the middle layer. The output voltage of this dummy node is fixed to an appropriate value, so it can be used as a threshold voltage. Zener diodes 251 and 252 output a constant voltage in a reverse bias state, and this fixed voltage can be varied within a range from +7 volts to -7 volts. The enable states of Output Enables 1 and 2 are determined by the output voltages from other ANPs connected to the analog bus and the dummy node control signal DCS from the master control block 181 to avoid conflicts on the analog bus.

FIG. 31A is a conceptual diagram of a feedback network.

In the case of a feedback network, there is basically an input, but the network also has a feedback path in which the signal it outputs also returns. This feedback method is used as a type of multi-layered neural network that uses one layer in time division multiplexing.

Since the input and output signals of the ANP are time-shared, the output data of the same ANP is output sequentially at the output point of each ANP for a certain sequence cycle, and each ANP operates sequentially as the input layer, intermediate layer, and output layer of the hierarchical neural network for each sequence cycle.

According to an embodiment of the present invention, as shown in FIG. 31B:
The feedback path is realized by the common analog bus CB, and there is a U-shaped feedback section. Then, the result of each calculation is output, and the output from each ANP is fed back through the feedback path. This feedback operation is repeated.

Fig. 32 shows the configuration of a first embodiment of a feedback network. In the figure, 121 denotes a plurality of basic units 1 that constitute the processing layer. These basic units 12
Similarly, the basic unit 121 has a multiplication unit 122, an accumulation unit 123, a threshold unit 124, and may also have an output holding unit 125. Similarly, the input and output of this basic unit 121 are also configured to be realized by analog signals.
Electrical connections are made between the output of the basic unit 121 and the input of all basic units 121 via analog buses 141, 142, and 143, and a weight corresponding to each connection is set to form an equivalent hierarchical network or a hop-feed network.

Reference numeral 142 denotes an analog bus, which is a common line provided for electrical connection between the output section and the input section that constitute the feedback line.
51, weight setting means 152, threshold processing starting means 153 and output value sending means 154, and controls data transfer.

The drive unit selection means 151 processes to select the basic units 121 in time series order. Then, in synchronization with this selection process, the output value transmission means 154 transmits the final output of the analog signal held in the output holding unit of the selected basic unit 121 to the multiplication processing units 122 of all the basic units 121 via the analog bus 142 in accordance with a time-division transmission format.
Upon receiving this input, the multiplier 122 of each basic unit 121 processes the weight setting means 15
2, the weights corresponding to the respective connections are sequentially selected, and the input is multiplied by the weights.
3 sequentially accumulates the multiplication results obtained by the multiplication unit 122. When it is confirmed that all accumulation processes for the basic units 121 of the processing layer have been completed, the threshold processing activation means 153 activates the threshold processing unit 124 of the basic unit 121 and controls it to apply a constant threshold voltage and perform calculation of the sigmoid function.

This process is then repeated cyclically a predetermined number of times. With this data transfer method, for example, an output pattern corresponding to an input pattern is obtained as the output of a hierarchical network by time-division multiplexing the layers.

That is, this embodiment discloses a network configuration data processing device of the present invention configured as a single layer. In this embodiment, in order to make the multi-layered network configuration data processing device into a single layer, a common line analog bus 142 is provided, and this analog bus 142 is configured to feed back the output of the basic units 121 forming a single layer to the input, so that the output of each basic unit 121 is connected to the input parts of all basic units 121.

Next, the operation process of the embodiment of FIG. 32 configured as above will be described with reference to the timing chart shown in FIG. 33A.

When a request for conversion to an output pattern is made, the main control circuit 150
The input side circuit 120 is selected in order in time series by sending a CSI control signal to the input side circuit 120. This selection process is illustrated in FIG. 33A.
The input circuits 120 process the analog signals Yi given as an input pattern so as to transmit them in time series onto the analog bus 143. Therefore, as shown in Fig. 33A, the analog signals _Yi are transmitted in order onto the analog bus 143, the number of which is equal to the number of input circuits 120.

As shown in the timing chart of Fig. 33B, when the multiplier 122 of each basic unit 121 receives the analog signal _Yi , it executes multiplication processing using the weight _Wi of the weight holding unit 8 that is set by the main control circuit 150. Then, the accumulator 122 of each basic unit 121
23 sequentially accumulates the multiplication results that are obtained for each selection of the input side circuit 120, and when all the accumulation processes are completed by completing the selection of all the input side circuits 120,
The threshold section 124 of each basic unit 121 performs processing to obtain the final output value Y, which is the final calculation output of the basic unit 121.

The final output value Y thus obtained is held internally. Then, when the output switch section 113 of each basic unit 121 receives an output control signal as CSI from the main control circuit 150, it is turned on in sequence in a time series manner, and the analog signal of this held final output value is fed back to the multiplication section 122 of the basic unit 121 via the analog bus 142. By this process, the final output value of the basic unit 121 of the input layer is equivalently transmitted to the basic unit 121 of the hidden layer in a time-division transmission format. By repeating the same process for the basic unit 121, the final output value Y corresponding to the output layer is
33A shows a timing chart of the output control signal to the basic unit 121 in correspondence with the _Y1 output control signal to the input side circuit 120, and also shows a timing chart of the analog signal of the final output value Y on the analog bus 142.

Although the embodiment of FIG. 34 has the drawback that it is not possible to input continuous input patterns to the hierarchical network, it has the advantage that the hierarchical network can be simplified and therefore can be made extremely small when fabricated into a chip.

Figure 34 shows an embodiment in which the neurocomputer of the present invention is realized by a feedback network operating as a hierarchical network. ANPs 1, 2, and 3 perform product-sum operations on time-division analog input signals from analog input ports 1 and 2. ANPs 1, 2, and 3 then operate as an intermediate layer, outputting the time-division analog signals to analog bus B2. This output signal is fed back to analog bus B1 via analog common bus CB, which serves as a feedback path. ANPs 1, 2, and 3 then perform product-sum operations on this feedback signal again, operating ANPs 1, 2, and 3 as an output layer, thereby realizing a hierarchical network using one layer of ANPs 1, 2, and 3. Max Value Node Circuit 187
receives the DCS output of the master control block and generates a dummy signal on the analog bus B2. DCLK and WCLK are input from the master control block to a daisy-chain circuit 171, which determines the timing of the rise and fall of the CSI signal.

FIG. 35A is a timing chart of the feedback hierarchical network shown in FIG.

WCLK is generated only while DCLK is rising, and after DCLK rises, the analog signal becomes steady, and after the weight data is input serially, the SCO from the master control block 181 is generated at the timing before the parallel data is aligned.
1 is input to the daisy chain circuit 171 and rises as shown. At this time, the analog signal held in the sample/hold S/H from the analog input port 1 appears on the analog bus B1 via the analog switch 175, and a multiplication and accumulation operation is performed by ANP1, 2, and 3. At the next input of DCLK, the daisy chain circuit 172
When CSI to the analog input port rises as shown, the signal from the sample/hold circuit S/H holding the input signal from the analog input port appears on the analog bus B1 via the analog switch, and the second multiplication and accumulation operation is performed by ANP1, 2, and 3. After DCLK is input at the next timing, the master control block outputs a dummy signal DC
S occurs, and in ANP1, 2, and 3, the
The sum of products is then executed. While the next SYNC signal is rising, the sum of products of the output layers of ANP1, 2, and 3 are calculated. The WCLK that counts the address counter is enabled only while the address count inhibit signal for the address 1 signal to the weight memory is rising; otherwise, the count is inhibited. Next, when CSO2 is given to ANP1 from the master control block, ANP1 outputs the result of the previous sum of products to analog bus B2 and returns it to analog bus B1 via analog common bus CB, and ANP1, 2, and 3 perform the sum of products again as shown. After a predetermined delay is added in the daisy-chain circuit inside ANP1, CSO2 adds the input signal CSI to ANP2 as shown, and at this time, the output signal of ANP is added to analog bus B2 via common bus CB and analog buses A1 and B1 again to ANP1, where the sum of products is calculated. Similarly, the CSO signal from ANP2 becomes the CSI signal of ANP3 after a predetermined delay, and when this CSI signal rises as shown in Figure 1, the output signal of ANP3 is transmitted again via the analog bus B2, common bus CB, and analog bus B1 to A
The signal is fed back to NP1, 2, and 3, where the sum of products is calculated. Similarly, when the signal DCS from the dummy node rises, the sum of fixed voltages is calculated again by ANP1, 2, and 3. Then, when the signal CSO2 rises again,
The output is generated from ANP1, 2 via S/H as shown in Fig. 1. Note that there is no output from analog input port 2.

Here, ANP1, 2, and 3 act as intermediate layers,
, , ANP1, 2, 3 operate as the output layer. Therefore, according to this embodiment, a hierarchical network can be configured with only one layer of ANP1, 2, 3.

INDUSTRIAL APPLICABILITY According to the embodiment described above, data processing of the network configuration data processing device whose basic configuration is shown in FIG. 7 can be realized using weight information determined by the learning processing method of the present invention.

The learning processing method of the present invention can be implemented in various ways. One implementation method is to build a network configuration data processing device (Figure 11) dedicated to weight setting on a computer,
In this constructed weight setting device, it is conceivable to realize the system by determining weights according to the learning algorithm of the present invention, and then outputting the determined weights from the weight data memory 150e of the actual network configuration data processing device explained in Fig. 21. Another implementation method is to implement a program of the learning processing algorithm of the present invention in the host computer 150y for controlling the main control circuit 150 explained in Fig. 21,
The host computer 150y calculates the amount of weight update using the network configuration data processing device installed, obtains new weights, and stores the obtained weights in the weight data memory.
It is conceivable that this can be realized by a method in which learning is performed by setting the weights in the weight data memory 150e, and the weights finally obtained are set as weights when data processing is performed in the weight data memory 150e.
The weight update amounts for the previous and previous times required for the learning process are stored, for example, in the main memory of the host computer 150y.

Although the illustrated embodiment has been described above, the present invention is not limited to this. For example, the specific values of the hierarchical network and control parameters used in the description are used solely for the convenience of explanation.

In this way, according to the present invention, the weight values of a hierarchical network can be learned in fewer iterations than in the prior art, making it possible to realize a network configuration data processing device. Moreover, since the discrepancy with the teacher signal can be reduced compared to the prior art, the accuracy of the data processing function of the network configuration data processing device can be further improved.

Claims (14)

[Claims] [Claim 1] A basic unit is a unit that receives one or more inputs from a previous layer and weights to be multiplied by the inputs, obtains a sum of products, and converts the obtained sum of products using a threshold function to obtain a final output. The basic unit is an input layer and a plurality of such basic units are used as an intermediate layer.
1. A network learning processing device having one or more intermediate layers and an output layer consisting of one or more of said basic units, wherein internal connections are formed between the input layer and the intermediate layer at the front stage, between the intermediate layers themselves, and between the intermediate layer at the final stage and the output layer, and wherein the weights are set corresponding to the internal connections, the network learning processing device comprising: output signal derivation means for supplying a plurality of predetermined input signals to the basic units of the input layer to obtain output signals corresponding to the input signals from the basic units of the output layer; error value calculation means for calculating an error value representing the magnitude of discrepancy between the two signals, using the output signals obtained by the output signal derivation means and a teacher signal indicating the value that the output signal should take; and weight learning means for determining a weight update amount for a current update cycle based on a value obtained from the error value calculated by the error calculation means, the weight update amount for a previous update cycle, and the weight update amount for a cycle before last, and for calculating a weight from the weight update amount. 2. The weight learning means
a register group for temporarily storing the second parameter α and the third parameter −β; a storage means for respectively storing the current weight update amount, the previous weight update amount, and the weight update amount before the previous weight update amount; a first multiplication means for multiplying −ε by the value of the partial differential value ∂E/∂W of the error and a first multiplication means for multiplying α by the previous weight update amount ΔW
2. The network learning processing device according to claim 1, further comprising: multiplication means comprising a second multiplier for multiplying -β by (t-1) and a third multiplier for multiplying -β by the weight update value ΔW(t-2) from the second previous time; addition means for adding the results of the three multiplications; second addition means for adding a weight W(t) read from a weight data memory to the result of the addition means; and a group of weight data memories for storing information on the obtained weights and weight update amounts. 3. The weight update amount ΔW(t), the previous weight update amount ΔW(t-1), the weight update amount before last ΔW(t-
The memory for storing 2) has a common weight selection signal as an address signal, and by turning on the shift signal, the weight update amount ΔW(t) memory (62) is shifted to the previous weight update amount ΔW(t-1) memory (63), and the previous weight update amount ΔW
(t-1) The weight update amount ΔW from the memory (63)
3. The network learning processing device according to claim 2, wherein the weight update amounts are copied by sequentially shifting the weight update amounts so that the contents of the same addresses are written to the same addresses in the (t-2) memory (64). 4. A weight data memory for weight W(t), said Δ
3. The network learning processing device according to claim 2, wherein the weight data memory for W(t) and the weight data memories for storing the previous and previous weight update amounts ΔW(t-1) and ΔW(t-2) are addressed by a weight selection signal consisting of a layer number, a unit number, and a connection number. Claim 5: In the learning process of a hierarchical neural network with at least three layers, an input layer, an intermediate layer, and an output layer, if the error is E(t) and the weight of each connection between each layer is W(t), a differential equation is defined in which the update amount of the weight during learning is an external force. Based on this, the differential equation is approximated by difference and the weight update amount Δ
Regarding W The learning processing method for processing network configuration data is characterized in that the weight update amount is calculated using a value obtained by multiplying the weight update amount ΔW(t−2) in the update cycle before last by −β. 6. A hierarchical network comprising at least three layers, an input layer, an intermediate layer, and an output layer, an input signal holding unit for holding an input signal to be input to said network, a teacher signal holding unit for holding a teacher signal, and a method for calculating an error between an output signal output when an input signal from said input signal holding unit is given to said network and the teacher signal from said teacher signal holding unit, and calculating a partial differential ∂E/
a weight learning means for receiving the value of the partial differential and using a weight update amount ΔW(t) and weight update amounts ΔW(t-1) and ΔW(t-2) from the previous and previous previous times; and a main control circuit means for providing a weight value W(t) determined as a result of learning by said weight learning means from a weight data memory 61 to said hierarchical network, performing initial settings for said weight learning means, controlling the updating of the weights of said weight learning means, providing the obtained weight value W(t) to said network, controlling the operation of the network, providing a pattern selection signal to said input signal holding means, and also performing appropriate control over said error calculation means. 7. The control signal from the main control circuit to the weight learning means is transmitted to an internal weight data storage memory (62, 63, 64).
A clear signal to clear the weight update amount is the current ΔW
7. A network learning processing device according to claim 6, further comprising: a shift signal for controlling copying of data from memory 62 storing (t) to the previous update amount memory and to the update amount memory from the previous to the previous update amount memory; a weight selection signal which is an address signal for each weight memory; a weight read signal for controlling reading of each weight memory (61-64); an initial value setting signal for setting an initial value in weight data memory 61; signals for the initial value and learning parameters -ε, α and -β; and a weight update signal (90) for synchronizing the operations of the weight learning means. 8. The weight update signal is a multiplication signal during a first period, a first addition signal during a second period to output ΔW(t), and a weight data signal W(t) during a third period.
8. A network learning processing device according to claim 7, wherein the second addition for updating the weight data is a synchronization pulse for controlling the storage of the weight data memory (61) in the fourth period. 9. The network learning processing device according to claim 1, wherein said basic unit comprises an analog neuron chip. 10. The network learning processing device according to claim 1, wherein said hierarchical network is controlled by an analog time-division signal. [Claim 11] A network learning processing device as described in Claim 6, characterized in that all weight data memories 61, 62, 63, 64 of the weight learning means stored in the network consisting of the analog neuron chips are serial read/write type memories in which each bit of the weight and its corresponding update amount is stored at the same address. [Claim 12] A system using a basic unit as a basic unit that receives one or more inputs from a previous layer and weights to be multiplied by the inputs, obtains a sum of products, and converts the obtained sum of products using a threshold function to obtain a final output, the system having a feedback path that feeds the output back to the input, the input layer, intermediate layer, and output layer being constructed with a single-layer structure consisting of the plurality of basic units, and the output of the single-layer structure being fed back to the input via the feedback path, thereby operating the single-layer structure successively as an input layer, intermediate layer, and output layer, output signal derivation means that supplies a plurality of predetermined input signals to the basic unit of the input layer, thereby obtaining output signals from the basic unit of the output layer corresponding to the input signals, and error value calculation means that calculates an error value that represents the magnitude of discrepancy between the two signals using the output signal obtained by the output signal derivation means and a teacher signal that indicates the value that the output signal should take. a weight learning means for determining a weight update amount for the current update cycle based on a value obtained from the error value calculated by the error calculation means, the weight update amount for the previous update cycle, and the weight update amount for the update cycle before the previous update cycle, and for calculating a weight from the weight update amount. 13. The weight learning means calculates a partial differential value of the weight with respect to the error value calculated by the error calculation means, a value obtained by multiplying the weight update amount in the previous update cycle by a predetermined value α, and a value obtained by multiplying the weight update amount in the update cycle before last by a predetermined value −β.
2. The network learning processing device according to claim 1, wherein the weight update amount for the current update cycle is calculated from the value obtained by multiplying the weight by . 14. A basic unit is a basic unit that receives one or more inputs from a previous layer and weights to be multiplied by the inputs, obtains a sum of products, and converts the obtained sum of products by a threshold function to obtain a final output,
The input layer and the intermediate layer are composed of multiple basic units.
a learning method for a network configuration data processing device having one or more intermediate layers and an output layer consisting of one or more of said basic units, wherein internal connections are formed between the input layer and the intermediate layer at the front stage, between the intermediate layers, and between the intermediate layer at the last stage and the output layer, and wherein the weights are set corresponding to the internal connections, the learning method for a network configuration data processing device comprising: supplying a plurality of predetermined input signals to the basic units of said input layer to obtain output signals corresponding to the input signals from the basic units of said output layer; calculating an error value representing the magnitude of discrepancy between the two signals using the obtained output signals and a teacher signal indicating the value that the output signal should take; and determining a weight update amount for the current update cycle based on the value obtained from the error value, the weight update amount determined in the previous update cycle, and the weight update amount determined in the update cycle before last.