WO2019144311A1 - Rule embedded artificial neural network system and training method thereof - Google Patents

Abstract

Disclosed in the present invention are a rule embedded artificial neural network system and a training method thereof. The system comprises: an input layer, a first intermediate layer, a regularization modulation layer, a second intermediate layer, and an output layer. An output end of the input layer is respectively connected to input ends of the first intermediate layer and the second intermediate layer. A neuron output end of the first intermediate layer is connected to an input end of the regularization modulation layer. The second intermediate layer and a neuron output end of the regularization modulation layer are connected to an input end of the output layer. The invention solves the technical problem that the artificial neural network provided in the related art has high complexity and large demand for computing resources.

Description

Rule embedded artificial neural network system and training method thereof Technical field

The present invention relates to the field of neural networks, and in particular to a rule embedded artificial neural network system and a training method thereof.

Background technique

At present, artificial neural network technology has been widely used in speech recognition, image recognition, video analysis, natural language processing and other issues with its superior inferential performance, which greatly improves people's production efficiency and quality of life.

However, artificial neural networks mainly learn statistical laws from a large amount of data, so their performance is easily restricted by the amount of data. In order to achieve accurate identification of complex patterns, a wider and deeper neural network is commonly used in the related art, and more computing resources are needed for this. This makes computing devices with weak computing power (especially embedded processing platforms) difficult to meet the accuracy and real-time requirements of a given application scenario.

On the other hand, existing artificial neural networks usually use end-to-end inference methods, which are difficult to interpret through human knowledge logic. At the same time, this also causes the inference of the artificial neural network to be affected by some noise and errors. The more erroneous inferences made by artificial neural networks, the smaller the scope of application and the lower the value of production and life. This makes it necessary to use human knowledge logic for automatic error correction.

In fact, human knowledge logic can be described as computer-processable rules, which in turn automatically correct errors in partial error inference. Automatic correction using rules can be divided into pre-processing and post-processing.

The pre-processing error correction method uses rules to process the input data. For example, the following technical solution is provided in the Chinese application with the patent application number CN201710239556.X: First, the input data is preprocessed by using the existing knowledge base to generate a triplet (regularized data) embedded with the first-order logic rule. Each triple contains two words in the input data and their logical rules. Then, using the triples as input data, the neural network is used for model training and inference.

Although this method combines the correlation logic between the input signals, it can reduce the erroneous inference to some extent. However, the drawback of this approach is that since the knowledge base used is independent of the whole of the input data, the generated regularized data usually only achieves the understanding of local data (words), but it does not reach the global data. Modeling and understanding, resulting in limited error correction capabilities.

In the post-processing error correction mode, the output of the neural network is not the final inference result, but is further refined as an input feature, and finally inferred. For example, the Chinese application with the patent application number CN 201710083292.3 provides an implementation scheme for identifying a device area by an artificial neural network and designing a temperature warning rule in combination with device characteristics. An alarm is issued when the temperature warning rule is met.

Although this method considers the device characteristics and temperature at the same time, thus reducing the possibility of false alarms, the problem is that the output results are sensitive to rules and are not suitable for scenarios requiring end-to-end output of artificial neural networks.

In addition, a data conversion interface technique for connecting artificial neural networks of different functions is proposed in the U.S. Patent Application Serial No. US20170300813. The conversion rules mentioned in this patent application are data driven and do not contain automatic error correction rules based on human knowledge logic. Although the US application filed in US Patent Application No. US 20090324010 proposes regular modulation of the output data of the artificial neural network, the output result is sensitive to the rules and is difficult to apply to scenarios requiring direct output by the neural network. The fuzzy rules mentioned in the U.S. application Serial No. 5,740,322 are based on training set optimization, but do not include automatic error correction rules based on human knowledge logic.

In response to the above problems, no effective solution has been proposed yet.

Summary of the invention

At least some embodiments of the present invention provide a rule embedded artificial neural network system and a training method thereof, so as to at least solve the technical problem that the artificial neural network provided in the related art has high complexity and large computational resource requirement.

According to one embodiment of the present invention, a rule embedded artificial neural network system is provided, including:

An input layer, a first intermediate layer, a regularization modulation layer, a second intermediate layer, and an output layer, wherein an output end of the input layer is respectively connected to an input end of the first intermediate layer and the second intermediate layer, the first intermediate layer The neuron output is coupled to the input of the regularized modulation layer, and the second intermediate layer is coupled to the input of the output layer of the neuron output of the regularized modulation layer.

Optionally, the first intermediate layer and the second intermediate layer respectively comprise one or more layers of neural networks.

Optionally, the system further includes: a third intermediate layer, wherein the neuron output ends of the second intermediate layer and the regularization modulation layer are combined and connected to the input end of the output layer via the third intermediate layer; or, cancel The second intermediate layer, the input layer and the neuron output of the regularization modulation layer are combined and connected to the input end of the output layer via the third intermediate layer.

Optionally, the regularized modulation layer comprises: structured modeling and data conversion, wherein the structured modeling refers to extracting global structured feature rules in the input data based on human professional knowledge logic, and the data conversion refers to utilizing the global structure The feature rule analyzes the samples in the input data and converts them into neuron outputs, so that the neurons corresponding to the samples in the input data that conform to the global structured feature law generate an activation output and the remaining neurons generate an inactive output.

According to an embodiment of the present invention, a training method for a rule-embedded artificial neural network is further provided, which is applied to the above-mentioned rule embedded artificial neural network system, including:

Constructing training data and corresponding labels; constructing a first auxiliary layer, wherein a neuron structure of the first auxiliary layer is the same as a neuron structure of the output layer, and an input end of the first auxiliary layer is opposite to a neuron output end of the first intermediate layer Connecting; constructing a first optimizer and iterating and optimizing a connection weight of the network formed by the input layer, the first intermediate layer, and the first auxiliary layer using the first optimizer, wherein the loss function of the first optimizer is given by a given training In the data, the neuron output value of the first auxiliary layer is obtained by comparing with the corresponding label; and a second optimizer is constructed, wherein when the loss function of the second optimizer is given training data, the output value of the output layer is corresponding to the output value of the neuron The label is obtained by comparison; the connection weight of the network formed by the input layer and the first intermediate layer is fixed, and the second optimizer is used to iteratively optimize the connection right of the network formed by the input layer, the second intermediate layer, the regularization modulation layer and the output layer value.

Optionally, when there is a third intermediate layer, the connection weight of the network formed by the input layer and the first intermediate layer is fixed, and the input layer, the second intermediate layer, the regularization modulation layer, and the iterative optimization are performed by using the second optimizer. The third intermediate layer and the output layer constitute a connection weight of the network.

Optionally, when the third intermediate layer is present, the regularized modulation layer, the second intermediate layer, and the third intermediate layer are set as a neural network component unit, and between the input layer and the output layer and the first intermediate layer The structure between the output layers is larger than one neural network component, wherein more than one neural network component is connected end to end.

Optionally, when there is more than one neural network component unit, when optimizing parameters of each neural network component unit by training, constructing a second auxiliary layer, wherein the input end of the second auxiliary layer and the current neural network component unit The output ends of the third intermediate layer are connected, and the output shape of the second auxiliary layer is the same as that of the output layer.

Optionally, when there is more than one neural network component unit, a separate optimizer is separately constructed for each neural network component unit, wherein the loss function of each constructed optimizer is given by the first auxiliary layer when the training data is given. The neuron output values are compared to the corresponding labels.

Optionally, when there is more than one neural network component unit, the connection weights of each layer of the neural network in each neural network component unit are optimized step by layer using a plurality of separate optimizers constructed.

Optionally, the plurality of separate optimizers constructed adopt a gradient-based layer-by-layer greedy optimization algorithm; the loss function of each constructed optimizer is a derivable function that characterizes the difference between the output value and the target value, wherein The derivative functions include: a cross entropy distance metric function, and a mean squared distance metric function.

In at least some embodiments of the present invention, the input layer, the first intermediate layer, the regularization modulation layer, the second intermediate layer, and the output layer are used to construct a rule embedded artificial neural network system, and the output ends of the input layer are respectively An intermediate layer is connected to an input end of the second intermediate layer, a neuron output end of the first intermediate layer is connected to an input end of the regularization modulation layer, and a neuron output end and output of the second intermediate layer and the regularization modulation layer are The input ends of the layers are connected to achieve the purpose of simplifying the artificial neural network architecture, thereby realizing the technical effect of reducing the complexity of the artificial neural network and the demand for computing resources, thereby solving the artificial neural network provided by the related art. Technical problems with high complexity and large demand for computing resources.

DRAWINGS

The drawings described herein are intended to provide a further understanding of the invention, and are intended to be a part of the invention. In the drawing:

1 is a schematic diagram showing the basic structure of a rule-embedded artificial neural network system according to an embodiment of the present invention;

2 is a schematic diagram showing an expanded structure of a rule embedded artificial neural network system according to a preferred embodiment of the present invention;

3 is a flow chart of a training method of a rule embedded artificial neural network according to an embodiment of the present invention;

4 is a schematic structural diagram of a full convolution network according to a preferred embodiment of the present invention;

5 is a block diagram showing the structure of a regularly stacked artificial neural network of a secondary stack in accordance with one of the preferred embodiments of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is an embodiment of the invention, but not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of the present invention.

It is to be understood that the terms "first", "second" and the like in the specification and claims of the present invention are used to distinguish similar objects, and are not necessarily used to describe a particular order or order. It is to be understood that the data so used may be interchanged where appropriate, so that the embodiments of the invention described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "comprises" and "comprises" and "the" and "the" are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to Those steps or units may include other steps or units not explicitly listed or inherent to such processes, methods, products or devices.

In accordance with an embodiment of the present invention, an embodiment of a rule-embedded artificial neural network system is provided, it being noted that the steps illustrated in the flowchart of the figures may be performed in a computer system such as a set of computer executable instructions And, although the logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than the ones described herein.

1 is a schematic diagram showing the basic structure of a rule-embedded artificial neural network system according to an embodiment of the present invention. As shown in FIG. 1, the rule embedded artificial neural network system includes: an input layer 1, a first intermediate layer 2, and a rule. a modulation layer 3, a second intermediate layer 4, and an output layer 5, wherein the output ends of the input layers are respectively connected to the input ends of the first intermediate layer and the second intermediate layer, and the neuron output ends and rules of the first intermediate layer The input ends of the modulation layer are connected, and the second intermediate layer and the neuron output of the regular modulation layer are connected to the input end of the output layer.

The above regularized modulation layer may include: structured modeling and data conversion, wherein structured modeling refers to extracting global structured feature rules in input data based on human professional knowledge logic, and data conversion refers to utilizing global structured features The samples in the input data are analyzed and converted into neuron outputs such that the neurons corresponding to the samples in the input data that conform to the global structured feature law produce an activation output and the remaining neurons produce an inactive output.

Optionally, the first intermediate layer and the second intermediate layer respectively comprise one or more layers of neural networks.

2 is a schematic diagram of an extended structure of a rule-embedded artificial neural network system according to a preferred embodiment of the present invention. As shown in FIG. 2, the rule embedded artificial neural network system further includes: a third intermediate layer 6 And wherein the neuron output ends of the second intermediate layer and the regularization modulation layer are combined and connected to the input end of the output layer via the third intermediate layer.

Optionally, when the third intermediate layer is present, the second intermediate layer may also be cancelled, and the input ends and the neuron output ends of the regularized modulation layer are combined and connected to the input end of the output layer via the third intermediate layer. .

In the related art, the rule is usually not placed in the middle layer of the artificial neural network because the result of the rule is usually accompanied by a slight change in the data to cause a mutation. In other words, regularized modulation is usually a process that cannot be differentiated, and the process of non-differentiation destroys the parameter optimization process of the artificial neural network, so that it cannot converge to a very good value, and the parameters cannot be solved. To this end, at least some embodiments of the present invention place the error correction process in the middle layer of the neural network, and structurally model the intermediate layer (also called the feature layer) of the neural network in the form of regularized modulation of human knowledge logic. Data conversion, thus achieving the following technical effects:

First, structured modeling is the modeling and understanding of data globally, overcoming the technical flaws of preprocessing.

Secondly, the network uses the two-step training method to train the neural network separately, which overcomes the indivisible problem caused by regularized modulation, and overcomes the problem that the post-processing error correction method is sensitive to rules.

Thirdly, due to the introduction of human knowledge logic, it not only reduces the possibility of artificial neural networks being affected by noise, but also reduces the number of samples required to train artificial neural networks.

Finally, because the rules can constrain the global logical structure, the artificial neural network only needs to focus on the statistical analysis of the local information structure, which can reduce the complexity of the artificial neural network and the demand for computing resources.

According to an embodiment of the present invention, a training method for a rule embedded artificial neural network is further provided, which is applied to the above rule embedded artificial neural network system. FIG. 3 is a flowchart of a training method of a rule embedded artificial neural network according to an embodiment of the present invention. As shown in FIG. 3, the method includes the following steps:

Step S30, constructing training data and corresponding labels;

Step S31, constructing a first auxiliary layer, wherein a neuron structure of the first auxiliary layer is the same as a neuron structure of the output layer, and an input end of the first auxiliary layer is connected to a neuron output end of the first intermediate layer;

Step S32, constructing a first optimizer, and using a first optimizer to iterate and optimize a connection weight of the network formed by the input layer, the first intermediate layer, and the first auxiliary layer, wherein the loss function of the first optimizer is given by When training data, the neuron output value of the first auxiliary layer is obtained by comparing with the corresponding label;

Step S33, constructing a second optimizer, wherein the loss function of the second optimizer is obtained by comparing the neuron output value of the output layer with the corresponding tag when the training data is given;

Step S34, the connection weights of the network formed by the input layer and the first intermediate layer are fixed, and the connection weights of the network formed by the input layer, the second intermediate layer, the regularization modulation layer and the output layer are iteratively optimized using the second optimizer.

Optionally, in step S34, when there is a third intermediate layer, the connection weight of the network formed by the input layer and the first intermediate layer is fixed, and the input layer and the second intermediate layer are iteratively optimized by using the second optimizer. The regularization modulation layer, the third intermediate layer, and the output layer constitute a connection weight of the network.

Optionally, when the third intermediate layer is present, the regularized modulation layer, the second intermediate layer, and the third intermediate layer are set as a neural network component unit, and between the input layer and the output layer and the first intermediate layer The structure between the output layers is larger than one neural network component, wherein more than one neural network component is connected end to end.

Optionally, when there is more than one neural network component unit, when optimizing parameters of each neural network component unit by training, constructing a second auxiliary layer, wherein the input end of the second auxiliary layer and the current neural network component unit The output ends of the third intermediate layer are connected, and the output shape of the second auxiliary layer is the same as that of the output layer.

Optionally, when there is more than one neural network component unit, a separate optimizer is separately constructed for each neural network component unit, wherein the loss function of each constructed optimizer is given by the first auxiliary layer when the training data is given. The neuron output values are compared to the corresponding labels.

Optionally, when there is more than one neural network component unit, the connection weights of each layer of the neural network in each neural network component unit are optimized step by layer using a plurality of separate optimizers constructed.

Optionally, the plurality of separate optimizers constructed adopt a gradient-based layer-by-layer greedy optimization algorithm; the loss function of each constructed optimizer is a derivable function that characterizes the difference between the output value and the target value, wherein The derivative functions include: a cross entropy distance metric function, and a mean squared distance metric function.

The above preferred implementation process will be further described in detail below in conjunction with the following preferred embodiments.

Preferred embodiment 1

The preferred embodiment provides a rule embedded artificial neural network and its inference process.

In the rule-embedded artificial neural network, given input data is ECG sample data with a time length of N seconds, the sampling rate is 128 Hz, and the number of ECG leads is C. The goal of a given artificial neural network is to detect the peak position of the R wave in the input data.

The ruled embedded artificial neural network provided by the preferred embodiment includes: an input layer, an intermediate layer A (corresponding to the first intermediate layer), an intermediate layer B (corresponding to the second intermediate layer), a regularized modulation layer A, and an output. Floor.

Specifically, the input layer converts the input data into three-dimensional spatial data whose data shape is [1, 128*N, C]. Both the intermediate layer A and the intermediate layer B are composed of a full convolution network. For the full convolutional network, please refer to the following documents: http://www.cnblogs.com/gujianhan/p/6030639.html or Long J, Shelhamer E, Darrell T, et al. Fully convolutional networks for semantic segmentation [J]. Computer vision and pattern recognition, 2015: 3431-3440). 4 is a schematic structural diagram of a full convolutional network according to a preferred embodiment of the present invention. As shown in FIG. 4, in the preferred embodiment, the structure of the full convolutional network includes 12 convolutional layers and 6 batch normalizations. (batch normalization) layer, 4 maximum pooling layers, 4 deconvolution layers, and 4 stitching layers, the order between the layers is as shown by the direction of the arrow in the figure.

The convolutional layer described above is used to capture local patterns in the data. The maximum pooling layer described above is used to summarize the local patterns and to expand the field of view of subsequent neurons. The above batch normalization layer is used to overcome the gradient disappearance problem of the neural network and speed up the model training. The deconvolution layer described above is used to upsample the neural network such that each point of the output data corresponds exactly to each point of the input data. The above-mentioned stitching layer is used to introduce high-resolution features of the shallow layer of the neural network into the deep layer, thereby improving the resolution of the deep layer. The down sampling rate of the above pooled layer and the upsampling rate of the deconvolution layer are both set to 2. The template length of the last convolutional layer is 1, and the length of the template of other convolutional layers is 3.

Further, after the above structure, a convolution layer is added, and the input includes an output of the first convolution layer of the five convolution layers and an output of the last deconvolution layer of the five deconvolution layers, As described in the references, by introducing features of the front layer network (the output of the first convolutional layer) at the back end of the network (the last added convolutional layer), it is possible to combine low-level features and high-level features in the signal such that A full convolutional network produces accurate output.

In addition, each convolutional layer neuron uses a ReLU activation function.

The output data shapes of the intermediate layer A and the intermediate layer B are [1, 128*N, C1] and [1, 128*N, C2], respectively.

The regularized modulation layer A includes: structured modeling and data conversion of electrocardiogram data.

Based on human expertise in electrocardiogram, the global structural characteristics of R wave of ECG include: R wave can only occur at most once during the refractory period (for example: 0.3 seconds); there is a certain rhythm in the process of R wave generated by the heart. , co-regulated by autonomic and non-autonomic nerves; the standard deviation of the interval between R waves in a short period of time (eg, 60 seconds) usually does not change drastically (less fluctuations in healthy state, atrial fibrillation, premature beats, etc.) Fluctuations in the disease state).

The structured modeling process includes: extracting a maximum value set for each channel of the intermediate layer A output data according to the global structural feature rule, wherein the maximum value satisfies a value greater than 0.3 seconds before the channel and is not less than the value The value in the range of 0.3 seconds after the channel; then for each channel, the selected sample points are obtained iteratively.

Obtaining selected sample points in an iterative manner can be divided into the following steps:

In the first step, statistical analysis is performed on the amplitude of the data corresponding to the maximum value, and the abnormal value is excluded, and the element corresponding to the abnormal value is deleted from the maximum value set, and the excluded quantity is recorded as e1. The way to exclude outliers can be based on the 3*sigma criterion, that is, the data is assumed to follow a normal distribution, and the sample mean and sample standard deviation are calculated. Samples with a sample mean distance other than 3 times the sample standard deviation can be considered as outliers.

In the second step, after deleting the element corresponding to the outlier from the set of maxima, sequentially calculating the time difference dT _j between adjacent maxima in the maxima of the maxima set, and recording the triad< dT _j , TS _j , TE _j >, where TS _j , TE _j represent the start and end points of the time difference respectively (ie, the two adjacent maxima above), and j represents the number of each time difference.

The time difference between adjacent maxima is calculated as: dT _j = L _j /r, where L _j represents the data length between the two maxima and r represents the sampling rate. For a full convolutional network, the sample rate of the output data is usually the same as the sample rate of the original input signal.

In the third step, statistical analysis is performed on the time difference, and the abnormal value is excluded, and the excluded quantity is recorded as e2.

The way to exclude outliers can be based on the 3*sigma criterion, that is, the data is assumed to follow a normal distribution, and the sample mean and sample standard deviation are calculated. Samples with a sample mean distance other than 3 times the sample standard deviation can be considered as outliers.

In the fourth step, the maximum value set is updated using the start point and the end point in the above-mentioned triples corresponding to the remaining time difference. If e1>0 or e2>0, go to the first step to continue the iteration.

In the fifth step, the elements of the maximum value set are marked as selected sample points.

The data conversion process includes: generating output data of the same shape as the input data, and setting the position corresponding to the selected maximum value to 1 (indicating the activation output), and setting the other position to 0 (indicating the inactive output).

The output data of the intermediate layer B and the regularization modulation layer A are merged and connected as an input to the output layer, and the output layer processes the input data by a one-dimensional convolution operation to generate data of a shape of [1, 128*N, 2]. Then, the data of [128*N, 2] is reconstructed by Reshape. Finally, the output data is generated using the Softmax activation function, and the last column of the output data indicates the probability that the corresponding position is an R wave.

Further, an intermediate layer C (corresponding to the third intermediate layer described above) may be added before the output layer of the rule-embedded artificial neural network to increase the accuracy of the neural network. The intermediate layer C can also adopt a full convolutional network, and the network structure is similar to the intermediate layer A and the intermediate layer B; the structural parameters such as the network depth (layer number) and the template length of the convolution layer of each intermediate layer can be based on actual application scenarios. Make adjustments.

For convenience, a full convolutional network is used instead of the inferred single sample artificial neural network provided in the related art. The full convolutional network can take all the samples in the entire data (sample points of the ECG) as input, and provide the inference results of all the samples (whether each time point belongs to the R wave peak), thereby reducing the amount of calculation on the one hand. On the other hand, it is convenient to carry out structured modeling of the entire data. It is of course also possible to use an artificial neural network model that can only infer a single sample, but only requires multiple inferences before structural modeling. The output data of the regularized modulation also needs to input subsequent neurons one by one in the form of a single sampling point. .

Further, the training process of the above rule embedded artificial neural network may include the following steps:

The first step is to build a data set. The data set includes: electrocardiogram data and corresponding labels of M measurements. The electrocardiogram data for each measurement contains L _i sample points (i = 1, ..., M) that are continuously sampled. The corresponding label of each measurement data is a matrix of 2 columns of L _i rows. The values of each row of the matrix are: [1,0] and [0,1], where [1,0] indicates that the corresponding position is not the R wave peak, and [0,1] indicates that the corresponding position is the R wave peak.

In the second step, the electrocardiogram (eg, 70%) and the corresponding label in the data set are divided into training data sets, and the remaining electrocardiograms and their corresponding labels are divided into verification data sets.

The third step is to construct an auxiliary layer whose structure is consistent with the output layer structure, and the neuron output of the intermediate layer A is connected as an input to the auxiliary layer.

The fourth step is to construct a first optimizer whose loss function is obtained by comparing the neuron output value of the auxiliary layer with the corresponding tag when the training data is given. For example, when inputting a measured ECG data (the corresponding label is z), the neuron output vector of the auxiliary layer is x, then the loss function can be designed as: xx*z+log(1+exp(-x)) Where log() and exp() represent the logarithm (e is the base) and the exponent (e is the base), respectively.

In the fifth step, the first optimizer is used to iteratively optimize the connection weights of the network formed by the input layer, the intermediate layer A, and the auxiliary layer.

In the sixth step, the second optimizer is constructed, and the loss function is obtained by comparing the neuron output value of the output layer with the corresponding label when the training data is given.

The seventh step, the connection weight of the network formed by the fixed input layer and the intermediate layer A, uses the second optimizer to iteratively optimize the input layer, the intermediate layer B, the regularized modulation layer A, the intermediate layer C (if present), and the output layer The connection weights that make up the network.

It should be noted that the above optimizer adopts a layer-by-layer greedy training method based on gradient descent. The above loss function may be a cross-entropy, mean square error equal distance metric function.

Preferred embodiment two

The difference between this preferred embodiment and the preferred embodiment is that the structural modeling is modified as follows:

According to the global characterization feature rule, a maximum value set is extracted for each channel of the output data of the intermediate layer A, and the maximum value satisfies a value within a range of 0.3 seconds before the channel and is not less than 0.3 seconds after the channel Value; then for each channel, the selected sample points are obtained by:

In the first step, the data amplitude corresponding to the maximum value is clustered, three clusters are generated, and the cluster update maximum value set with the most elements is selected.

In the second step, starting with each maximal position (denoted as Ps), searching for the maximum value as the matching end point (denoted as Pe), and calculating the time difference between Ps and Pe (denoted as dT), which matches The condition is: dT < 1.5 seconds.

In the third step, a matching set is generated using all the triples <Ps, Pe, dT> satisfying the condition.

In the fourth step, the clustering technique is used to analyze the dT in the matching set, and three matching clusters are generated, and the matching cluster with the most elements is selected to update the matching set, and the maximum value updating maximum value set in the matching set is extracted.

In the fifth step, the elements of the maximum value set are marked as selected sample points.

The preferred embodiment replaces the abnormal point exclusion method based on the 3*sigma criterion by using a clustering technique, and is applicable to a scene in which an abnormality is not a Gaussian distribution and thus is not suitable for the 3*sigma criterion.

Preferred embodiment three

The difference between this preferred embodiment and the preferred embodiment is that the structured modeling process is modified as follows:

According to the global characterization feature rule, a maximum value set is extracted for each channel of the output data of the intermediate layer A, and the maximum value satisfies a value within a range of 0.3 seconds before the channel and is not less than 0.3 seconds after the channel Value; then for each channel, the selected sample points and their corresponding classification scores are obtained by:

In the first step, the data amplitude corresponding to the maximum value is clustered, and three clusters are generated, and the cluster with the most elements is selected, and the maximum value corresponding to the cluster element is marked as the selected sample point.

In the second step, starting with each maximal position (denoted as Ps), searching for the maximum value as the matching end point (denoted as Pe), calculating the time difference between Ps and Pe (denoted as dT), matching condition Is: dT < 1.5 seconds.

In the third step, a matching set is generated using all the triples <Ps, Pe, dT> satisfying the condition.

The fourth step is to calculate the global feature: the sample mean, sample standard deviation, and median of the dT in the matching set are recorded as dTavg, dTstd, and dTmid, respectively.

The fifth step is to calculate the local feature of each maximum value: the number of triplets including the maximum value (denoted as nt), the sample mean value of the dT in the triplet containing the maximum value, and the sample standard deviation ( Respectively denoted as dTiavg, dTistd), the time difference between the maximum value and the previous maximum value (denoted as dTif), the time difference between the maximum value and the latter maximum value (denoted as dTil).

In the sixth step, using a machine learning model (for example, SVM, neural network), the above global features and local features are input, and the classification score of each selected sample point is output.

The machine learning model adds a third optimizer in the process of regular embedded artificial neural network training. The loss function is obtained by comparing the above eigenvalues with corresponding position labels when given training data; and then corresponding to the machine learning model. The training method optimizes the parameters of the machine learning model.

Moreover, the difference between the preferred embodiment and the preferred embodiment 1 is that the data conversion process is modified as follows:

Generates output data of the same shape as the input data, and sets the position corresponding to the selected sample point to the corresponding classification output (indicating the activation output), and the other positions to 0 (indicating the inactive output).

In the preferred embodiment, the global structured feature law is expressed by a feature extraction algorithm, and the extracted features are further transformed into activation values of corresponding locations by the machine learning model. Compared with the purely regular algorithm, this method does not require empirically setting boundary conditions (for example: 3*sigma), and does not cause indivisible problems due to boundary conditions. Therefore, the artificial neural network model is more stable and has a wider range of applications.

Preferred embodiment four

The preferred embodiment differs from the preferred embodiment 1 in that the intermediate layer A and the intermediate layer B are partially shared. For example: share the parameters of the first five convolutional layers. This method can simplify the complexity of the artificial neural network model and improve the computational efficiency of artificial neural network model training and inference.

Preferred embodiment five

The preferred embodiment differs from the preferred embodiment 1 in that the regularized modulation layer A, the intermediate layer B, and the intermediate layer C are stacked two or more times. 5 is a schematic structural diagram of a regularly stacked artificial neural network of a second stack according to a preferred embodiment of the present invention. As shown in FIG. 5, the output of the input layer is connected as an input to the intermediate layer D, and the intermediate layer C is The output is connected as an input to the regularized modulation layer B, the outputs of the intermediate layer D and the regularized modulation layer B are connected as an input to the intermediate layer E and the output of the intermediate layer E is connected as an input to the output layer, wherein the rules are The modulation layer A, the intermediate layer B, and the intermediate layer C are disposed as a first neural network component unit, and the regularization modulation layer B, the intermediate layer D, and the intermediate layer E are set as second neural network constituent units. Therefore, in the above-mentioned secondary stacked regular embedded artificial neural network, there are two neural network constituent units (that is, the above is greater than one neural network constituent unit), and the first neural network constituent unit and the second neural network constituent unit are connected end to end. .

It should be noted that the regularized modulation layer B and the regularized modulation layer A respectively adopt different structural modeling or data conversion methods.

Further, the training process of the above rule embedded artificial neural network may include the following steps:

In the first step, the training data and the corresponding label are constructed, wherein the training data includes the electrocardiogram data of the M measurements, and the electrocardiogram data of each measurement includes the Li sampling points (i=1, . . . , M) that are continuously sampled. The corresponding label of each measurement data is a matrix of 2 rows of Li rows, wherein each row of the matrix is divided into [1, 0] and [0, 1], wherein [1, 0] indicates that the corresponding position is not the peak of the R wave. , [0, 1] indicates that the corresponding position is an R wave peak.

In the second step, an auxiliary layer A (corresponding to the first auxiliary layer described above) is constructed, the structure of which is identical to that of the output layer, and the neuron output of the intermediate layer A is connected as an input to the auxiliary layer A.

The third step is to construct a first optimizer whose loss function is obtained by comparing the neuron output value of the auxiliary layer with the corresponding tag when the training data is given.

The fourth step is to iteratively optimize the connection weights of the network formed by the input layer, the intermediate layer A, and the auxiliary layer A.

In the fifth step, an auxiliary layer B (corresponding to the second auxiliary layer described above) is constructed, the structure of which is identical to that of the output layer, and the neuron output of the intermediate layer C is connected as an input to the auxiliary layer B.

In the sixth step, a second optimizer is constructed, and when the loss function is given by the training data, the neuron output value of the auxiliary layer B is compared with the corresponding label.

In the seventh step, the connection weight of the network formed by the fixed input layer and the intermediate layer A is iteratively optimized the connection weights of the network formed by the input layer, the intermediate layer B, the regularized modulation layer A, the intermediate layer C, and the auxiliary layer B.

In the eighth step, the third optimizer is constructed, and the loss function is obtained by comparing the neuron output value of the output layer with the corresponding label when the training data is given.

The ninth step, the fixed input layer, the intermediate layer A, the intermediate layer B, the regularized modulation layer A, and the intermediate layer C constitute a connection weight of the network. Iteratively optimizes the connection weights of the network formed by the input layer, the intermediate layer D, the regularized modulation layer B, the intermediate layer E, and the output layer.

In the preferred embodiment, the regularized modulation layer is stacked multiple times and is suitable for scenarios with a variety of different structured modeling requirements.

Preferred embodiment six

In the structural modeling of the above preferred embodiment 1, the preferred embodiment 2 and the preferred embodiment 3, the empirically set hyperparameters are included, for example:

In a preferred embodiment 1, the abnormal value exclusion method is determined as an abnormal threshold (3 times sample standard deviation);

In a preferred embodiment 2, the clustering parameter (the number of output clusters is 3);

In a preferred embodiment 3, the hyperparameters of the machine learning model (C, γ of the SVM);

These hyperparameters usually need to be set by experience, and of course, they can be optimized by grid search during the training process.

In addition, after the training process is completed, the output results of the auxiliary layer and the output layer respectively represent:

(1) Inference based on "local signals";

(2) Inference based on "global signal + local signal".

Comparing the differences in the training data sets of the above two output results can be used to analyze the complementary effects of specific rules in the regularized modulation layer to the artificial neural network:

(1) When a specific rule does not have a beneficial effect on the model result, the rule is deleted;

(2) When the specific rules have obvious improvement on the model results, the advantages and defects of the rules are analyzed according to the professional knowledge, and then the experts are inspired to design new rules.

Accordingly, the content of the rule modulation layer is updated, and the parameters of the rule embedded artificial neural network are retrained. In the preferred embodiment, the artificial neural network model can be made more accurate by preferentially setting hyperparameters.

Preferred embodiment seven

The difference between the preferred embodiment and the preferred embodiment 1 is that the accuracy of the inference result of the auxiliary layer and the output layer is compared from the verification data set after the training process of the rule embedded artificial neural network is completed. If the output layer cannot compare the performance of the auxiliary layer, it indicates that the regularized modulation layer excessively suppresses the local analysis function of the artificial neural network.

In this case, the training method needs to be adjusted to force the optimizer to better optimize the parameters of the intermediate layer B, thereby improving the role of the intermediate layer B in the output layer. The training method is adjusted as follows:

In the structured modeling process, the original "mark the elements of the maximal set as selected sample points" is adjusted to the elements of the set of maxima and its adjacent (second range of 0.1 second) poles Large values are marked as selected sample points.

The above adjustments are used during the training process and should be eliminated when applied to data inference to avoid false inferences caused by noise.

According to an embodiment of the present invention, a storage medium is provided. The storage medium includes a stored program, wherein the device in which the storage medium is located controls the training method of the rule embedded artificial neural network. The above storage medium may include, but is not limited to, a U disk, a read only memory (ROM), a random access memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and the like, which can store program codes.

According to an embodiment of the present invention, there is further provided a processor, wherein the processor is configured to execute a program, wherein the program runs the training method of the rule embedded artificial neural network. The above processor may include, but is not limited to, a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA).

The serial numbers of the embodiments of the present invention are merely for the description, and do not represent the advantages and disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the descriptions of the various embodiments are different, and the parts that are not detailed in a certain embodiment can be referred to the related descriptions of other embodiments.

In the several embodiments provided by the present application, it should be understood that the disclosed technical contents may be implemented in other manners. The device embodiments described above are only schematic. For example, the division of the unit may be a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, unit or module, and may be electrical or otherwise.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention, which is essential or contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product stored in a storage medium. A number of instructions are included to cause a computer device (which may be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.

The above description is only a preferred embodiment of the present invention, and it should be noted that those skilled in the art can also make several improvements and retouchings without departing from the principles of the present invention. It should be considered as the scope of protection of the present invention.

Industrial applicability

As described above, a rule embedded artificial neural network system and a training method thereof provided by the embodiments of the present invention have the following beneficial effects: structured modeling is a global modeling and understanding of data, and overcomes technical defects of preprocessing methods. . The two-step training method is used to train the neural network separately, which overcomes the indivisible problem caused by regularized modulation and overcomes the problem that the post-processing error correction method is sensitive to rules. The introduction of human knowledge logic not only reduces the possibility of artificial neural networks being erroneously affected by noise, but also reduces the number of samples required to train artificial neural networks. Since the rules can constrain the global logical structure, the artificial neural network only needs to focus on the statistical analysis of the local information structure, which can reduce the complexity of the artificial neural network and the demand for computing resources.

Claims (11)

A rule embedded artificial neural network system, comprising: An input layer, a first intermediate layer, a regularization modulation layer, a second intermediate layer, and an output layer, wherein an output end of the input layer is respectively connected to an input end of the first intermediate layer and the second intermediate layer a neuron output end of the first intermediate layer is coupled to an input end of the regularization modulation layer, and an input of the second intermediate layer and the neuron output end of the regularization modulation layer and the output layer The ends are connected. The system of claim 1 wherein said first intermediate layer and said second intermediate layer comprise one or more layers of neural networks, respectively. The system of claim 1 further comprising: a third intermediate layer, wherein said second intermediate layer and said neuron output of said regularized modulation layer are merged via said third intermediate layer The input ends of the output layers are connected; or the second intermediate layer is eliminated, and the input layers and the neuron output ends of the regularization modulation layer are merged and then passed through the third intermediate layer and the output layer The inputs are connected. The system of claim 1 wherein said regularized modulation layer comprises: structured modeling and data transformation, wherein said structured modeling refers to extracting a global structure in the input data based on human expertise logic Characterizing the law of the feature, the data conversion refers to analyzing the sample in the input data by using the global structured feature rule and converting the sample into a neuron output, so that the input data conforms to the global structural characteristic rule The corresponding neurons of the sample produce an activation output and the remaining neurons produce an inactive output. A method of training a ruled embedded artificial neural network, the method being applied to the rule embedded artificial neural network system according to any one of claims 1 to 4, the method comprising: Construct training data and corresponding labels; Constructing a first auxiliary layer, wherein a neuron structure of the first auxiliary layer is the same as a neuron structure of the output layer, and an input end of the first auxiliary layer is connected to a neuron output end of the first intermediate layer; Constructing a first optimizer and iterating and optimizing a connection weight of the network formed by the input layer, the first intermediate layer and the first auxiliary layer using a first optimizer, wherein the loss function of the first optimizer Obtaining, by the training data, the neuron output value of the first auxiliary layer is compared with a corresponding label; Constructing a second optimizer, wherein the loss function of the second optimizer is obtained by comparing a neuron output value of the output layer with a corresponding tag when the training data is given; Securing a connection weight of the input layer and the network formed by the first intermediate layer, and using the second optimizer to iteratively optimize the input layer, the second intermediate layer, the regularization modulation layer, and the output layer The connection weights that make up the network. The method according to claim 5, wherein when there is a third intermediate layer, a connection weight of the network formed by the input layer and the first intermediate layer is fixed, and iteratively optimized by using the second optimizer The input layer, the second intermediate layer, the regularization modulation layer, the third intermediate layer, and the output layer constitute a connection weight of the network. The method according to claim 5, wherein when the third intermediate layer is present, the regularization modulation layer, the second intermediate layer, and the third intermediate layer are set as a neural network component unit, and Between the input layer and the output layer and between the first intermediate layer and the output layer, more than one neural network component unit is constructed, wherein the more than one neural network component unit is connected end to end. The method according to claim 7, wherein when there is more than one neural network constituent unit, the second auxiliary layer is constructed when the parameters of each neural network constituent unit are optimized by training, wherein the The input end of the second auxiliary layer is connected to the output end of the third intermediate layer of the current neural network component unit, and the output shape of the second auxiliary layer is the same as the output layer. The method according to claim 7, wherein when there is more than one neural network constituent unit, a separate optimizer is separately constructed for each neural network constituent unit, wherein the loss function of each constructed optimizer is The neuron output value of the first auxiliary layer is obtained by comparison with the corresponding tag when the training data is given. The method according to claim 9, wherein when there is more than one neural network component unit, the plurality of built-in optimizers are used to iteratively optimize the connection rights of each layer of neural networks in each neural network component unit. value. The method of claim 10, wherein said plurality of separate optimizers are constructed using a gradient-based layer-by-layer greedy optimization algorithm; each constructed optimizer's loss function is to characterize the difference between the output value and the target value a degree of derivable function, wherein the derivable function comprises: a cross entropy distance metric function, a mean squared distance metric function.

Images