CN111950810A - A multivariate time series prediction method and equipment based on self-evolution pre-training - Google Patents

Abstract

The invention discloses a multivariate time series prediction method and equipment based on self-evolution pre-training. The method is based on a pre-training strategy and a deep sequence model of a convolution network and a long-short memory network, combined with univariate self-evolution information and multi-variable dependence. The relationship information is modeled, and the optimization algorithm of multivariate time series forecasting is realized, taking into account the overall forecasting accuracy and the local univariate forecasting accuracy. The invention has better overall prediction accuracy, and is superior to the existing multivariate time series prediction method in terms of the guarantee of prediction accuracy for local single variables.

Description

A multivariate time series prediction method and equipment based on self-evolution pre-training

technical field

The invention relates to time series prediction, in particular to a multivariate time series prediction method.

Background technique

The use of time series forecasting methods to obtain forecast values in the future is of fundamental and important significance for assisting decision-making, optimizing resource allocation, and taking early intervention measures. It has a wide range of applications in real-time information systems. Many systems generate a large amount of time series data, mainly including system performance monitoring data, and many application values are generated in the performance prediction of these monitoring data, such as predicting network traffic for resource planning and anomaly detection, and predicting disks in data centers. Failure to replace in advance, etc. Therefore, future data to predict system performance provides important help for the daily operation, operation and maintenance of modern information systems. In recent years, a large amount of research has focused on the time series forecasting problem of real-time systems. Linear statistical algorithms (eg AR, ARIMA) have made an impact on the field of time series forecasting. However, due to the lack of applicability of linear models in many practical applications, nonlinear time series analysis and forecasting have been proposed and gradually applied. In addition to the classical statistical method models, machine learning models have become important algorithms in the field of time series forecasting. Researchers apply machine learning models such as linear regression models, support vector machines, and decision trees to time series analysis. The time series prediction algorithm based on machine learning regards time series prediction as a supervised learning task, in which the input attribute is the time series data of historical observations, and the output label is the future data. However, the current multivariate time series forecasting algorithms that focus on the optimization of the overall forecasting objective will bring about the problem of the loss of local variable forecasting accuracy. For example, in the regression task of multivariate prediction, existing algorithms take the overall prediction error (weighted) sum of all variables as the minimization objective, and improve the overall prediction accuracy by optimizing such a minimization objective.

SUMMARY OF THE INVENTION

Purpose of the invention: In order to ensure the prediction accuracy of local variables, a multivariate time series prediction algorithm based on self-evolution pre-training is proposed. It can not only realize the optimization algorithm of multivariate time series prediction, but also take into account the overall prediction accuracy and local univariate prediction accuracy.

Technical solution: In order to be able to predict the performance of the system running in real time, so as to carry out reasonable operation and maintenance of the predicted value, in the first aspect, the present invention proposes a multivariate time series prediction method based on self-evolution pre-training, including the following steps :

S1. Obtain N-dimension index data representing the performance of the real-time operating system, form N univariate time series, take it as a historical input sequence, and perform preprocessing;

S2. Input N univariate time series into the self-evolution pre-training model SE constructed based on difference features, establish a univariate time series linear regression model, and explicitly integrate the multi-order difference information into the linear autoregression model, form the corresponding prediction output;

S3. First perform one-dimensional convolution operations of different sizes on the two-dimensional tensors composed of N univariate time series, and extract the dependency information between variables to obtain multiple feature maps;

S4. Input the feature map into the long short-term memory network (LSTM) to perform time series modeling on the multi-scale feature map to capture the time series information of multivariate dependencies, and the corresponding multivariate modeling network components form corresponding outputs;

S5, perform the one-to-one correspondence addition and fusion of the vector elements to the outputs obtained in steps S2 and S4 to construct a multivariate prediction output, and train the network to obtain a trained model;

S6. Input the time series to be predicted into the trained model obtained in step S5, so as to obtain the predicted value of the time series.

In a second aspect, the present invention provides a computer device, the device includes one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, One or more processors are caused to implement the method according to the first aspect of the present invention.

Beneficial effects: The present invention proposes a multivariate time series prediction method based on self-evolution pre-training, which is based on pre-training strategies and deep sequence models of convolutional networks and long-short memory networks, combined with univariate self-evolution information and multi-variable dependency information. Modeling realizes the optimization algorithm of multivariate time series prediction, taking into account the overall prediction accuracy and the local univariate prediction accuracy. The invention has better overall prediction accuracy, and is superior to the existing multivariate time series prediction method in terms of the guarantee of prediction accuracy for local single variables.

Description of drawings

1 is an overall flow chart of a multivariate time series prediction method according to an embodiment of the present invention;

2 is a schematic diagram of the main process of the multivariate time series prediction method according to an embodiment of the present invention;

FIG. 3 is a process flow chart of a model training phase according to an embodiment of the present invention;

FIG. 4 is a process flow chart of a model running phase according to an embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be further described below with reference to the accompanying drawings. It should be understood that the embodiments provided below are only to disclose the present invention in detail and completely, and to fully convey the technical idea of the present invention to those skilled in the art, and the present invention can also be implemented in many different forms, and does not Limited to the embodiments described here. The terms used in the exemplary embodiments shown in the drawings are not intended to limit the invention.

In the embodiment of the present invention, a multivariate time series prediction method based on self-evolution pre-training is used to predict the time series. Based on the pre-training strategy, convolutional network, and long-short-term memory network deep sequence model, combined with univariate self-evolution information and multivariate dependency information for modeling, an optimization algorithm for multivariate time series prediction is realized, while taking into account the overall prediction accuracy and Prediction accuracy for local univariate. 1 and 2, the method of the present invention comprises the following steps:

Step S1, collect corresponding multivariate time series data from the real-time operating system through the monitoring system, which may include multiple key factors such as CPU load, bandwidth utilization, bandwidth speed, memory utilization, response delay, IO read and write, etc. gender indicators. In reality, data often have some problems such as missing data, irregular data, etc., so the data is preprocessed first, and then the preprocessed data is used as the input of the model. In addition, in order to have a better modeling effect on the time series, it is also necessary to evaluate the periodicity of the time series to obtain the historical window T.

Preprocessing the data, that is, dividing the data set in the time dimension, it is necessary to set aside a training set for the model to train, and standardize the data so that the data values can be mapped to the same interval, which is conducive to the learning of the model. Such as min-max normalization:

Among them, min _x is the minimum value of x, and max _x is the maximum value of x. The periodicity of the time series is evaluated by the autocorrelation function ACF to obtain the historical input sequence length T. Multiple key indicators such as CPU load, bandwidth utilization, bandwidth speed, memory utilization, response delay, IO read and write respectively constitute a univariate time series, the length of each series is T, and x represents the vector value of the series .

Step S2, input the N univariate time series distributions of key indicators including CPU load, bandwidth utilization, memory utilization, response delay, etc. preprocessed in step S1 into the self-evolution pre-training model SE ( In Self-Evolution), a univariate time series linear regression model is established. The function of this model is to ensure the prediction accuracy of the univariate time series generated by the real-time system. In addition, considering that the multi-order difference information can improve the applicability and robustness in the prediction of stationary and non-stationary sequences, the multi-order difference information is explicitly integrated into the linear autoregressive model to form the corresponding prediction output. Specifically include the following steps:

Step S2-1, input the N univariate time series into the self-evolution pre-training model SE constructed based on the difference feature, and establish a univariate time series linear regression model. For the i-th sequence we have:

表示第i个序列中第t-j时刻的时序值，

为对应的线性权值，T为通过自相关函数ACF来评估时间序列的周期性而得到的历史输入序列长度。in

represents the time series value at time tj in the i-th sequence,

is the corresponding linear weight, and T is the historical input sequence length obtained by evaluating the periodicity of the time series through the autocorrelation function ACF.

即

可见一阶差分下预测值含有

把

等差分特征显式地融入到线性自回归模型中，形成对应预测输庄

Step S2-2, when considering the difference order q=1, there are

which is

It can be seen that the predicted value under the first order difference contains

Bundle

The equal difference feature is explicitly integrated into the linear autoregressive model to form the corresponding prediction loss.

为差分特征的融合权值用于训练，限制为非负实数，softmax为对应的归一化操作，Q为最大差分阶数；

为标准的线性自回归项，x_t-T：t-1表示从t-1时刻到t-T时刻的时间序列，T为历史输入序列长度。

The fusion weight of the differential feature is used for training, limited to non-negative real numbers, softmax is the corresponding normalization operation, and Q is the maximum differential order;

is a standard linear autoregressive term, x _{tT: t-1} represents the time series from time t-1 to time tT, and T is the length of the historical input sequence.

Step S3, first perform one-dimensional convolution operations of different sizes on the two-dimensional tensors composed of N univariate time series, and extract the dependency information between variables to obtain multiple feature maps.

进行变量维度上的一维卷积操作，表示如下：The present invention uses time series convolution of one-dimensional CNN layers with multiple convolution kernels to represent complex dependencies between multivariate time series, in order to obtain a feature map containing multi-level information of local dependencies of multivariate sequences. ). for the input sequence

Perform a one-dimensional convolution operation on the variable dimension, which is expressed as follows:

C ^(k) = W ^(k) *X _{tT: t-1}

W ^(k) is the kth convolution kernel, and C ^(k) is the convolution result, that is, the feature map extracted from the dependencies between variables. In a convolutional network, k convolution kernels W ⁽¹⁾ , W ⁽²⁾ , ..., W ^(k) are usually used to form feature maps C ⁽¹⁾ , C ⁽² ), ..., W(k) of multiple channels. ., C ^(k) . Different from existing algorithms which set all convolution kernels to the same size, the present invention adopts variable size convolution kernels. Compared with the same-scale convolution kernels, the multi-scale convolution kernels explicitly describe the associated time period features at different scales. C ⁽¹⁾ , C ⁽²⁾ , ..., C ^(k) contain multi-level information of local dependencies of multivariate time series.

Step S4, input the feature map into the long short-term memory network (Long Short-Term Memory, LSTM) to perform time-series modeling on the multi-scale feature map to capture the time-series information of multi-variable dependencies, and input each feature map into the corresponding LSTM and thus get the corresponding output.

LSTM is a classic recurrent neural network that can solve long-term dependencies of sequences. The network component based on multivariate modeling in the present invention is a structure composed of multiple LSTM basic memory units, and each LSTM correspondingly inputs a feature map. Specifically, the feature maps C ^(k) of all channels of the kth convolution kernel obtained in step S3 are spliced according to the time dimension, wherein the number of channels is determined by the dimension of the time series data itself, and the spliced feature maps are then used as the corresponding th Inputs of k long-short memory networks; the hidden outputs of the corresponding network units are input to the fully connected layer, forming an output vector h ^(k) ∈ R ^N×1 . A total of K identical LSTM outputs h ⁽ 1 ⁾ , h ⁽²⁾ , ..., h ^(K) , these outputs contain sequence dependency information. Accordingly, all vectors h ⁽¹⁾ , h ⁽²⁾ , ..., h ^(K) are added together to form the multivariate dependency modeling output

In step S5, the outputs obtained in steps S2 and S4 are added and fused in one-to-one correspondence of vector elements to construct a multivariate prediction output, and train the network to obtain a trained model. The steps are as follows:

The outputs obtained in steps S2 and S4 are added and fused in one-to-one correspondence of vector elements, and the fusion method is as follows:

表示元素乘法。这样的两部分加性融合具有明确实际含义，增加了算法的可解释性，通过让损失函数达到最小，这样既可以控制局部单变量的精度损失不会变大，同时也可以控制整体预测精度损失，从而达到算法的目的。例如，第i个变量的预测值

那么对于该变量的预测，两项取值的相对大小便直接反映了两部分信息在预测中的占比。完成模型构建后，为训练模型，构造优化目标对应的损失函数如下：W ^(se) , W ^(id) are the fusion weights, which are manually set as the hyperparameters of the model,

Represents element-wise multiplication. Such two-part additive fusion has a clear practical meaning and increases the interpretability of the algorithm. By minimizing the loss function, it can not only control the accuracy loss of the local single variable, but also control the overall prediction accuracy loss. , so as to achieve the purpose of the algorithm. For example, the predicted value of the ith variable

Then, for the prediction of this variable, the relative size of the two values directly reflects the proportion of the two pieces of information in the prediction. After the model construction is completed, for the training model, the loss function corresponding to the optimization target is constructed as follows:

为一个多变量向量值。The overall model can be optimized based on gradient descent. The weights for model training are the weights of Self-Evolution, CNN and LSTM models. This model is a single-step prediction model, then the final output

is a multivariate vector of values.

The above steps S1-S5 are the model training stage. As shown in FIG. 3 , the multivariate time series prediction model based on self-evolution pre-training is trained through historical time series data.

Step S6, using the trained model to predict the time series to be predicted.

As shown in Figure 4, in the model running stage, the time series to be predicted is preprocessed and input into the trained model obtained in step S5 to obtain the predicted value of the time series, that is, the model finally obtains information about the real-time system The predicted output of , in this embodiment, is the predicted value of key indicators such as CPU load, bandwidth utilization, memory utilization, and response delay, which can provide a powerful reference for real-time monitoring and operation and maintenance of the system.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. In the context of the present invention, the computer-readable medium may be considered tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media include non-volatile memory circuits (eg, flash memory circuits, erasable programmable read-only memory circuits, or masked read-only memory circuits), volatile memory circuits (eg, static random access memory circuits or dynamic random access memory circuits), magnetic storage media such as analog or digital magnetic tapes or hard drives, and optical storage media such as CD, DVD or Blu-ray discs, among others.

Program code for implementing the methods of the present invention may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

Additionally, although operations are depicted in a particular order, this should be understood to require that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations should be performed to achieve desirable results. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, although the above discussion contains several implementation-specific details, these should not be construed as limitations on the scope of the invention. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination.

The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details of the above-mentioned embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention. These equivalent transformations All belong to the protection scope of the present invention.

Claims (7)

1. a multivariate time series prediction method based on self-evolution pre-training, is characterized in that, this method comprises the following steps: S1. Obtain N-dimension index data representing the performance of the real-time operating system, form N univariate time series, take it as a historical input sequence, and perform preprocessing; S2. Input N univariate time series into the self-evolution pre-training model constructed based on difference features, establish a univariate time series linear regression model, and explicitly integrate multi-order difference information into the linear regression model to form a corresponding prediction output

S3. Perform one-dimensional convolution operations of different sizes on the two-dimensional tensors composed of N univariate time series, and extract the dependency information between variables to obtain multiple feature maps; S4. Input the feature map into the long-short memory network to perform time-series modeling on the multi-scale feature map to capture the time-series information of multi-variable dependencies, and the corresponding multi-variable modeling network components form corresponding outputs

S5, the output obtained in steps S2 and S4

and

One-to-one correspondence addition and fusion of vector elements is performed to construct a multivariate prediction output model, and training is performed to obtain a trained model; S6. The time series to be predicted is preprocessed and then input into the trained model obtained in step S5, so as to obtain the predicted value of the time series. 2. The detection method according to claim 1, wherein the step S2 comprises: S2-1. Input N univariate time series into the self-evolution pre-training model constructed based on differential features, and establish a univariate time series linear regression model. For the i-th series, there are:

in

represents the time series value of time tj in the i-th sequence,

is the corresponding linear weight, T is the length of the historical input sequence; S2-2. Explicitly integrate the multi-order difference information into the linear regression model to form the corresponding prediction output

is the fusion weight of the differential feature, softmax is the corresponding normalization operation, and Q is the maximum differential order;

is a standard linear autoregressive term, x _{tT: t-1} represents the time series from time t-1 to time tT. 3. prediction method according to claim 1, is characterized in that, in described step S3, one-dimensional CNN convolution operation is as follows: C ^(k) = W ^(k) *X _{tT: t-1} W ^(k) is the kth convolution kernel, C ^(k) is the convolution result, that is, the feature map extracted from the dependencies between variables, X _t-Tt-1 represents the two-dimensional composition of N univariate time series tensor,

x _{tT: t-1} represents the time series from time t-1 to time tT, and T is the length of the historical input sequence. 4. prediction method according to claim 1, is characterized in that, described step S4 comprises: the feature map C (k) of all channels of k-th convolution kernel obtained in step S3 is spliced according to time dimension, then as corresponding The input of the kth long short-term memory network; the hidden output of the corresponding network unit is input to the fully connected layer, forming an output vector h ^(k) ∈ R ^N×1 , K identical LSTM total output h ⁽¹⁾ , h ^{(2 )} , ..., h ^(K) , add all vectors h ⁽¹⁾ , h ⁽²⁾ , ..., h ^(K) to form the multivariate dependency modeling output

5. detection method according to claim 1 is characterized in that, in described step S5, the result of step S2 and S4 output is weighted and fused, and fusion method is as follows:

W ^(se) , W ^(id) is the fusion weight,

Represents element-wise multiplication. 6. detection method according to claim 5, is characterized in that, in described step S5, training model, the loss function corresponding to construct optimization target is as follows:

N is the dimension of the multivariate time series. 7. A computer device, wherein the device comprises: one or more processors; memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one processor, the programs when executed by the processor implement as in claims 1-6 The steps of any one of the methods.

Images