CN105183573B - The ONLINE RECOGNITION method and system of the continuous failure task of cloud computing system medium-high frequency time - Google Patents

Abstract

The invention discloses an online identification method and system for high-frequency continuous failure tasks in a cloud computing system. According to offline monitoring data, offline analysis and learning based on time series are performed to obtain a certain confidence level that can represent all non-high-frequency continuous failure tasks. The failure frequency threshold of the failure frequency feature of the failure task is then identified to obtain high-frequency continuous failure tasks in the online data. The present invention analyzes the tasks in the cloud computing system from the perspective of events and resources, obtains the frequency of failure events within a time period and the system resources consumed by the tasks, and identifies cloud computing in real time by analyzing the failure frequency characteristics of tasks and the time series pattern of resource usage. For high-frequency continuous failure tasks that fail repeatedly and are difficult to repair in the system, notify the cloud computing system in advance to take proactive failure recovery measures to save system resources for the cloud computing system and improve the reliability and availability of the cloud computing system.

Description

Online identification method and system for high-frequency continuous failure tasks in cloud computing system

technical field

The invention belongs to the technical field of cloud computing, and in particular relates to an online identification method and system for high-frequency continuous failure tasks in a cloud computing system.

Background technique

With its on-demand consumption model, cloud computing is gradually widely used in various fields such as finance and business. The high availability of the system in the cloud computing environment is increasingly becoming the key to the maturity of cloud computing technology. However, due to the gradual expansion of the scale and heterogeneity of the cloud computing system, various types of failures frequently occur in the cloud computing system, which has become one of the key factors that threaten the availability and reliability of the cloud computing system. In a cloud computing system, a task, as the smallest scheduling unit running on a single node, is the basic guarantee for the normal execution of user applications, and it is also the level where failures commonly occur. In order to improve system availability, improve fault tolerance and automatic recovery capabilities, the current common method for cloud computing systems to deal with task failures and make them recover quickly is to resubmit the schedule.

However, there are often certain tasks in cloud computing systems, which usually have the following characteristics: 1) High-frequency failures: due to repeated failure events in their life cycle, their total failure frequency is much higher than other tasks; 2) failure Continuity: They cannot be quickly recovered by restarting after failure, so there is almost no normal operating state between any two failures, that is, the failure state is continuous. Therefore, we refer to such tasks as "high frequency continuous failure tasks".

After analysis, it is found that the high-frequency continuous failure tasks show the following specific patterns: 1) Resource usage pattern: there is a significant difference in resource usage between the failure time and the running time of the high-frequency continuous failure tasks, and because the failures are continuous, Therefore, their resource consumption has an obvious segmentation trend in the time series; 2) Failure frequency characteristics: the number of failures per unit time (failure frequency) of high-frequency continuous failure tasks is much larger than that of other tasks. These patterns are the important features that distinguish high-frequency continuous failure tasks from other tasks. However, the prior art does not have an effective method for identifying high-frequency continuous failure tasks. Although high-frequency continuous failure tasks will be rescheduled by the system immediately after each failure, they cannot be quickly recovered by restarting, but will fail repeatedly after repeated scheduling. Repeated failures not only cause a lot of waste of system resources, but also increase the load of the cluster scheduler, bringing potential harm to the cloud computing system, making it difficult to meet the high availability requirements of the cloud computing system.

Contents of the invention

In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an online identification system and method for high-frequency continuous failure tasks in the cloud computing system, which realizes real-time and accurate identification of tasks that fail frequently and cannot be quickly repaired in the cloud computing system, and notify in advance The cloud computing system adopts proactive failure recovery measures to save system resources for the cloud computing system, reduce the load of the system scheduler, and improve the reliability and availability of the cloud computing system.

The principle of the present invention is: analyze the task in the cloud computing system from two angles of event and resource, wherein the event angle refers to the failure event occurrence frequency in the time period, and the resource angle is the system resource (such as CPU, CPU, etc.) consumed by the task in the time period Memory and disk, etc.), by analyzing the failure frequency characteristics and resource usage time series patterns of high-frequency continuous failure tasks and non-high-frequency continuous failure tasks, real-time identification of high-frequency continuous failures that fail repeatedly and are difficult to repair in cloud computing systems Tasks can avoid unnecessary waste of resources and scheduling loads, and improve the reliability and availability of cloud computing systems.

The technical scheme provided by the invention is:

An online identification method for high-frequency continuous failure tasks in a cloud computing system. According to offline monitoring data, offline analysis and learning based on time series is carried out, and the failure frequency characteristics that can represent all non-high-frequency continuous failure tasks at a certain level of confidence are obtained. The failure frequency threshold, and then identify the high-frequency continuous failure tasks in the online data; including the following steps:

1) Extract event and resource time series data from offline monitoring data, and obtain offline data format conversion results by converting to a specific format, including task failure frequency and resource usage;

2) Configure parameter values, the parameters include a failure frequency threshold, a failure continuation index threshold, a resource change threshold and a confidence threshold; the failure frequency threshold and the failure continuation index threshold are used to define high-frequency continuous failure tasks;

3) According to the offline data format conversion result in step 1) and the high-frequency continuous failure task definition parameters (failure frequency threshold and failure continuous index threshold) in step 2), the tasks in the offline data are marked as high-frequency continuous Failure tasks or non-high frequency continuous failure tasks;

4) Using the resource change threshold and confidence threshold in step 2), analyze the resource usage pattern and failure frequency characteristics of non-high frequency continuous failure tasks, and learn to represent all non-high frequency continuous failure tasks at a certain confidence level The failure frequency threshold of the task failure frequency feature;

5) Input the online monitoring data in real time, extract the event and resource time series data of the task, convert it into a specific format, and obtain the online data format conversion result, including the task failure frequency and resource usage;

6) According to the online data format conversion result obtained in step 5) and the failure frequency threshold obtained in step 4), the high-frequency continuous failure tasks in the online data are identified in real time.

For the online identification method of high-frequency continuous failure tasks in the above-mentioned cloud computing system, further, step 1) described event and resource time series data are converted into a specific format, specifically:

It is set that there are n tasks in the event and resource time series data, task i (1≤i≤n) is submitted to enter the system at time t _i0 , and the residence time in the system is T _i time periods; for each task i extracts T _i triples {t, F _{i, t} , R _{i, t} };

For the time series data of events and resources, a specific format is obtained as a list of four-tuple {i,t,F _i,t ,R _i,t }; where, t∈{t _i0 ,T _i0 +1,...,t _i0 +T _i -1} characterizes each time period that task i (1≤i≤n) resides in the system; F _i,t is the failure frequency of task i in the tth time period, by statistical offline data The failure event in is obtained, if no failure event occurs in the task i in the tth time period, then F _i,t is 0; R _i,t is the resource usage of the task i in the tth time period, given by The monitoring system acquires and imports through offline data sources.

For the online identification method of high-frequency continuous failure tasks in the above-mentioned cloud computing system, further, in step 3), the tasks in the offline data are marked as high-frequency continuous failure tasks or non-high-frequency continuous failure tasks, specifically:

According to the failure frequency of the task in the offline data described in step 1), calculate the failure total frequency of the task; obtain the failure continuity index according to the percentage of the continuous failure times accounting for the total failure frequency of the task; make the failure total frequency greater than the step 2) If the failure frequency threshold and the task failure continuity index is greater than the failure continuity index threshold in step 2), it is marked as a high-frequency continuous failure task; otherwise, it is marked as a non-high frequency continuous failure task.

For the online identification method of high-frequency continuous failure tasks in the above-mentioned cloud computing system, further, step 4) obtains the failure frequency threshold value that can represent the failure frequency characteristics of all non-high-frequency continuous failure tasks on a certain confidence level through learning, Specifically include the following steps:

41) dividing the resource exception window;

According to the resource usage, the resource abnormality degree is obtained, and the resource abnormality threshold set in step 2) is used to divide the residence time of non-high frequency continuous failure tasks in the system into multiple resource abnormality windows according to the resource abnormality degree, and each The division result of the resource exception window of a task;

42) Calculate the failure frequency;

According to the division result of the resource exception window of each task obtained in step 41), the failure frequency of each task in each resource exception window is calculated; then the maximum value of each task's failure frequency in all windows is set as the failure frequency of the task;

43) Obtaining the failure frequency threshold through learning;

According to the failure frequency of all non-high-frequency continuous failure tasks obtained in step 42), the cumulative distribution function of task failure frequency is fitted, and the confidence threshold set in step 2) is calculated to represent Failure frequency threshold for failure frequency of all non-high frequency consecutive failure tasks.

For the online identification method of high-frequency continuous failure tasks in the above-mentioned cloud computing system, further, the division method of the resource exception window described in step 41) is specifically as follows:

If there is a significant difference between the resource usage of the task in the tth time period and t-1, set the tth time period to belong to a new resource exception window, otherwise t and t-1 belong to the same window;

Whether there is a significant difference in resource usage is measured by resource usage variation. The resource usage variation V _i,i is calculated by the resource usage R _{i,t of task i at time t} through formula 1:

(Formula 1)

In formula 1, V _i,t is the resource usage change; R _i,t-1 is the resource usage of task i at time t-1; R _i,t is the resource usage of task i at time t; p( R _i ) indicates the magnitude of resource usage of task i, which is used to standardize resource usage changes and avoid negative impacts on resource usage changes due to different resource usage magnitudes of different tasks; if V _i,t is greater than the resource change threshold, a new window, otherwise keep the previous window.

For the online identification method of high-frequency continuous failure tasks in the above-mentioned cloud computing system, further, the method for calculating the failure frequency of each task as described in step 42) is as follows:

First calculate the failure frequency of each task in each resource exception window; suppose the start time of the jth resource exception window of task i is S _ij , and the end time is E _ij ; F _i,t is the The failure frequency within a time period; then the failure frequency Fre _ij of task i in the jth window can be obtained by formula 2:

(Formula 2)

The overall failure frequency Fre _i of task i is set as the maximum failure frequency of task i in all windows by Equation 3, which is used as the judgment value to distinguish between high-frequency continuous failure tasks and non-high-frequency continuous failure tasks:

Fre _i ＝max(Fre _ij ) (Formula 3)

In formula 3, Fre _ij is the failure frequency of task i in the jth window.

For the online identification method of high-frequency continuous failure tasks in the above-mentioned cloud computing system, further, the calculation method of the failure frequency threshold in step 43) is as follows:

First, according to the failure frequency Fre _i of all non-high frequency continuous failure tasks, the cumulative distribution function F ₁ (fre) of the task failure frequency is fitted, and the formula 4 will satisfy that F ₁ (fre) is equal to the fre of the confidence threshold V _conf As the failure frequency threshold f _thre :

f _thre ＝F ₁ ^-1 (V _conf ) (Formula 4)

In Formula 4, F ₁ ^-1 (fre) is the inverse function of F ₁ (fre); V _conf is the confidence threshold.

For the online identification method of high-frequency continuous failure tasks in the above-mentioned cloud computing system, further, the online identification of high-frequency continuous failure tasks in step 6) specifically includes the following steps:

51) According to the converted results of online events and resource usage data, divide and obtain resource exception window results;

The division method of the resource exception window is as follows:

If there is a significant difference between the resource usage of the task in the tth time period and t-1, set the tth time period to belong to a new resource exception window, otherwise t and t-1 belong to the same window;

Whether there is a significant difference in resource usage can be measured by resource usage variation. The resource usage variation V _i,t is calculated by the resource usage R _{i,t of task i at time t} through formula 1:

(Formula 1)

In formula 1, V _i,t is the resource usage change; R _i,t-1 is the resource usage of task i at time t-1; R _i,t is the resource usage of task i at time t; p( R _i ) indicates the magnitude of resource usage of task i, which is used to standardize resource usage changes and avoid negative impacts on resource usage changes due to different resource usage magnitudes of different tasks; if V _i,t is greater than the resource change threshold, a A new window, otherwise it will remain in the previous window;

52) Calculate the failure frequency of each task within the current resource exception window, and calculate the failure frequency;

Set N _i as the current cumulative failure frequency of task i, F _i represents the failure frequency of task i in the current window, S _i is the start time of task i in the current window; if the current time t and time t-1 belong to For the same resource exception window, add time t to the resource exception window at time t-1, and calculate the failure frequency in this window; otherwise, open a new resource exception window and calculate the failure frequency in this window;

53) Identify high-frequency continuous failure tasks:

According to the failure frequency of tasks obtained in step 52) in the current window and the failure frequency threshold obtained in step 4), the high-frequency continuous failure tasks in the cloud computing system are identified by comparison; if task i is in the current failure window If the failure frequency is greater than the failure frequency threshold, task i is a high-frequency continuous failure task.

Aiming at the above-mentioned online identification method for high-frequency continuous failure tasks in the cloud computing system, further, the identification of high-frequency continuous failure tasks in online data is visualized, and an effective proactive failure recovery mechanism is adopted to avoid unnecessary scheduling Waste of load and system resources.

The present invention also provides an online identification system for high-frequency continuous failure tasks in a cloud computing system, including an ETL module, a threshold configuration module, a high-frequency continuous failure task marker, an offline analysis and learning module based on time series, and a high-frequency continuous failure task marker. Continuous failure task online identification module and high-frequency continuous failure task early warning module; it is characterized in that,

The ETL module is used to process the input offline monitoring data and online monitoring data, extract event and resource time series data from the data, perform conversion, and transfer the conversion results including task failure frequency and resource usage to high frequency Continuous failure task marker and high-frequency continuous failure task online identification module;

The threshold configuration module is used to configure the failure frequency threshold, the failure continuity index threshold, the resource change threshold and the confidence threshold, and pass the configuration parameter value to the high-frequency continuous failure task marker; the configuration failure frequency threshold and the failure continuity index threshold It is used to define high-frequency continuous failure tasks; the resource change threshold is used to judge whether the resource usage is abnormal; the confidence threshold is used to limit the confidence level that the learning results need to meet;

According to the failure frequency threshold and the failure continuation index threshold, the high-frequency continuous failure task marker marks the tasks in the offline data as high-frequency continuous failure tasks or non-high-frequency continuous failure tasks, and marks the non-high-frequency continuous failure tasks The event data and resource time series data of the failure task, resource change threshold and confidence threshold are passed to the offline analysis and learning module based on time series;

The off-line analysis and learning module based on time series is used to learn the failure frequency threshold and pass it to the high-frequency continuous failure task online identification module; the failure frequency threshold is used to characterize the failure frequency characteristics of non-high-frequency continuous failure tasks;

The online recognition module for high-frequency continuous failure tasks identifies high-frequency continuous failure tasks in the online data according to the failure frequency threshold, obtains high-frequency continuous failure tasks, and passes the identification results to the early warning of high-frequency continuous failure tasks module;

Early warning of high-frequency continuous failure tasks is used to visualize the identified high-frequency continuous failure tasks.

Compared with prior art, the beneficial effect of the present invention is:

Based on the analysis of task failure frequency characteristics and resource usage time series patterns, the present invention uses statistics and pattern matching methods to identify high-frequency continuous failure tasks in the cloud computing system, and mainly has the following characteristics:

1. The present invention analyzes the high-frequency and continuous failure tasks in the cloud computing system from the perspective of the time sequence mode of the tasks.

2. The present invention describes the mode of the task in the time sequence from two aspects of events and resources, the event describes the failure frequency of the task within the time period, and the resource usage describes the resource consumption mode of the task within the time period.

3. By analyzing and comparing the resource usage and failure frequency characteristics of high-frequency continuous failure tasks and non-high-frequency continuous failure tasks, the present invention can accurately identify high-frequency continuous failure tasks from a large number of tasks in the early stage of failure.

Therefore, the present invention can realize real-time and accurate identification of tasks that fail frequently and cannot be repaired quickly in the cloud computing system. Identify high-frequency continuous failure tasks in advance, and notify the cloud computing system in advance to take proactive failure recovery measures to avoid repeated scheduling of these tasks, so as to save system resources and reduce scheduling load. Save system resources for the cloud computing system and reduce system scheduler load, and improve the reliability and availability of cloud computing systems.

Description of drawings

Fig. 1 is a flowchart of an online identification method for high-frequency continuous failure tasks in a cloud computing system provided by an embodiment of the present invention.

Fig. 2 is a block diagram of the data processing flow in the online identification system for high-frequency continuous failure tasks in the cloud computing system provided by the embodiment of the present invention.

detailed description

Below in conjunction with accompanying drawing, further describe the present invention through embodiment, but do not limit the scope of the present invention in any way.

Fig. 1 is a flow chart of an online identification method for high-frequency continuous failure tasks in a cloud computing system provided by an embodiment of the present invention, and Fig. 2 is a structure and system of an online identification system for high-frequency continuous failure tasks in a cloud computing system provided by an embodiment of the present invention Flow chart of data processing. The following is a specific example to illustrate the process of performing the operation of the method provided by the present invention:

1) First, the ETL module reads the offline monitoring data from the offline data source, and converts the data into a specific data structure;

Data can be read through the API provided by the system. For file systems, commonly used operating systems such as WINDOWS and LINUX have available APIs for file read and write operations; for database systems, taking MYSQL as an example, you can connect to the database system and read data through JDBC.

Assuming that there are n tasks in the offline data, task i (1≤i≤n) is submitted to the system at time t _i0 , and the residence time in the system is T _i time periods. The function of the ETL module is to extract Generate T _i triples {t, F _{i, t} , R _{i, t} }. Among them, t∈{t _i0 ,t _i0 +1,...,t _i0 +T _i -1} represents each time period that task i resides in the system; F _i,t is the task i in the tth time period The failure frequency within is calculated by counting the failure events in the offline data. If no failure event occurs in the task i in the tth time period, then F _i,t is 0; R _i,t is the task i in the tth time period Resource usage within a time period. Resource usage may include multiple indicators such as CPU, memory, and disk, which are obtained by the monitoring system and imported through offline data sources. Since the monitoring granularity of event data and resource usage data may be different, in order to unify the monitoring granularity, the time period is equal to the coarser monitoring period of the two. Combining the extraction results of all tasks, the offline monitoring data can be transformed into the structure shown in Table 1, and passed to the high-frequency continuous failure task marker.

Table 1 Task, time period, event failure frequency and resource usage data

2) The user sets the failure frequency threshold, failure continuity index threshold, resource change threshold and confidence threshold through the threshold configuration module;

Among them, the failure frequency threshold is used to limit the minimum total failure frequency of high-frequency continuous failure tasks. Only tasks with a total failure frequency greater than the failure frequency threshold may be high-frequency continuous failure tasks. The failure frequency threshold is determined by the user according to the tasks in the system. The failure frequency distribution is determined. If F(x) is the cumulative distribution function of the failure frequency, the recommended value range of the failure frequency is [F ^-1 (0.5%), F ^-1 (10%)], where F ^-1 ( x) is the inverse function of F(x); the failure continuity index is equal to the percentage of the number of continuous failures in the total failure frequency of the task. Continuous failure means that the task is in the failure state in the current cycle and the previous cycle, and the high-frequency continuous failure The failure continuity index of the task must be greater than the failure continuity index threshold, and the value range of the failure continuity index threshold is the interval (0,1). The larger the threshold, the higher the failure continuity requirements for high-frequency continuous failure tasks; the resource change threshold It is used to limit the minimum value of the resource variation within the resource exception window, if the resource usage variation of two adjacent periods (for the calculation method, please refer to the method of dividing the resource exception window based on the offline analysis and learning module of time series in step 41) If it is greater than the resource change threshold, the two cycles are in different resource exception windows. The resource change threshold is determined by the user according to the resource change of the task in the system; the confidence threshold is used to limit how much confidence should be satisfied when learning the failure frequency characteristics of the task , that is, how high the confidence level of the learning results can represent all training tasks. In order to ensure that the learning results are sufficiently credible, the recommended value range of the confidence threshold is the interval [80%, 100%];

3) Based on the above threshold, the high-frequency continuous failure task marker marks the high-frequency continuous failure tasks and non-high-frequency continuous failure tasks in the offline data (tasks whose total failure frequency is greater than the failure frequency threshold may be high-frequency tasks Continuous failure tasks; the failure continuation index of high-frequency continuous failure tasks must be greater than the failure continuation index threshold), and then pass the marking results, received offline data and configuration parameters to the time series-based offline analysis and learning module;

In the present invention, the high-frequency continuous failure tasks must meet the characteristics of high-frequency continuous failure and continuous failure. Therefore, the high-frequency continuous failure tasks and non-high-frequency continuous failure tasks in the offline data are specifically divided according to the following method: If the total failure frequency of the task is greater than the failure frequency threshold, and the continuous failure index is greater than the continuous failure index threshold, the task is a high-frequency continuous failure task; if the total failure frequency of the task is greater than the failure frequency threshold, but the continuous failure index is less than the continuous failure index index threshold, the task is a non-high-frequency continuous failure task; if the task does not meet the high-frequency failure condition (the total failure frequency is less than the failure frequency threshold), obviously the task is not a high-frequency continuous failure task, but due to its failure frequency If it is too low, there is a large difference between its characteristics and the characteristics of high-frequency continuous failure tasks, and the present invention does not analyze this situation.

4) According to the obtained data and parameters, the offline analysis and learning module based on time series uses the resource change threshold and confidence threshold to analyze the resource usage pattern and failure frequency characteristics of non-high-frequency continuous failure tasks, and learns that at a certain confidence The failure frequency threshold that can represent the failure frequency characteristics of all non-high frequency continuous failure tasks horizontally, and pass the results to the online learning module of high frequency continuous failure tasks;

Learning to obtain the failure frequency threshold that can represent the failure frequency characteristics of all non-high frequency continuous failure tasks at a certain level of confidence, specifically includes the following steps:

41) dividing the resource exception window;

Since the resource consumption of high-frequency continuous failure tasks is different between normal operation time and failure time, and the high-frequency continuous failure tasks have continuous failure, the resource usage of high-frequency continuous failure tasks has a significant segmentation trend. Therefore, the main function of the resource exception window division module is to divide the residence time of non-high-frequency continuous failure tasks in the system into several resource exception windows according to the degree of resource exception, and pass the division results to the failure frequency calculation module; The exception window division method is as follows:

If there is a significant difference between the resource usage of the task in the t-th time period and t-1, it is considered that the t-th time period belongs to a new resource exception window, otherwise t and t-1 belong to the same window. Whether there is a significant difference in resource usage can be measured by the resource usage variation, that is, for the resource usage R _i,t of task i at time t (obtained from the high-frequency continuous failure task marker), the resource usage variation is calculated by formula 1 Quantity V _i,t :

(Formula 1)

In formula 1, V _i,t is the resource usage change; R _i,t-1 is the resource usage of task i at time t-1; R _i,t is the resource usage of task i at time t; p( R _i ) indicates the magnitude of resource usage of task i, which is used to standardize resource usage changes and avoid negative impacts on resource usage changes due to different resource usage magnitudes of different tasks; if V _i,i is greater than the resource change threshold (configured by the threshold Module settings, passed through the high-frequency continuous failure task marker), a new window will be opened, otherwise it will remain in the previous window.

42) Calculate the failure frequency;

After receiving the resource exception window division result of each task, the main function of the failure frequency calculation module is to calculate the failure frequency of each task and pass it to the failure frequency threshold learning module. The calculation method of failure frequency is as follows:

First calculate the failure frequency of each task within each resource exception window. Assuming that the start time of the jth resource exception window of task i is S _ij , and the end time is E _ij , F _i,t is the failure frequency of task i in the tth time period, then task i in the jth window The failure frequency Fre _ij in is:

(Formula 2)

Since the failure frequency of the same task in different resource exception windows may be different, we hope to obtain the maximum failure frequency of non-high frequency continuous failure tasks through learning as an indicator to distinguish between high frequency continuous failure tasks and non-high frequency continuous failure tasks. Judgment value, therefore, set the overall failure frequency Fre _i of task i to be the maximum value of the failure frequency in all windows:

Fre _i ＝max(Fre _ij ) (Formula 3)

43) Obtaining the failure frequency threshold through learning;

The main function of the failure frequency threshold learning module is to receive the failure frequency of all non-high-frequency continuous failure tasks from the failure frequency calculation module, and calculate the failure frequency that can represent all non-high-frequency continuous failure tasks at a certain confidence level according to the confidence threshold. The failure frequency threshold value of the failure frequency, and the failure frequency threshold value is passed to the high-frequency continuous failure task online identification module. The calculation method of the failure frequency threshold is as follows:

Firstly, according to the failure frequency Fre _i of all non-high frequency continuous failure tasks, the cumulative distribution function F ₁ (fre) of the task failure frequency is fitted, and then the fre satisfying that F ₁ (fre) is equal to the confidence threshold V _conf is taken as the failure The frequency threshold f _thre is passed to the online recognition module for high-frequency continuous failure tasks:

f _thre ＝F ₁ ^-1 (V _conf ) (Formula 4)

where F ₁ ^-1 (fre) is the inverse function of F ₁ (fre).

5) Input online data and identify high-frequency continuous failure tasks in the online data in real time;

Since online monitoring data is input into event data and resource usage of tasks in the system at a certain moment in real time, the input and conversion of online data is a cyclic process, and the number of tasks input each time may be different. When the online data is input, the ETL module converts the input data into a structure as shown in Table 1, which is a list of quadruples {j,t,F _j,t ,R _j,t } according to a method similar to processing offline data . Among them, t∈{t _j0 ,t _j0 +1,,t _j0 +T _j -1}, _{t j0} and T _j are the first submission time and residence time of task j respectively; The failure frequency in t time period is obtained by counting the failure events in the input data; R _j,t is the resource usage of task j in the t time period, which is obtained by the monitoring system and input through the online data source; Finally, the converted quadruple list is passed to the high-frequency continuous failure task online identification module;

The online recognition process is triggered by an online data input event. When monitoring data input online, the ETL module reads the data and transforms the data structure in real time. The online recognition module for high-frequency continuous failure tasks uses algorithms to identify high-frequency continuous failure tasks in the online data in real time based on the acquired data and parameters such as the failure frequency threshold, and transmits the results to the early warning module for high-frequency continuous failure tasks . The high-frequency continuous failure task early warning module displays the identification results, thereby prompting the cloud computing system management personnel to take corresponding measures. The online recognition process will be carried out cyclically with the online data input until the online data input is completed.

The high-frequency continuous failure task online identification module is the core of the system. Its main function is to identify high-frequency continuous failure tasks in the cloud computing system in real time based on the acquired failure frequency threshold and resource change threshold parameters, using the received online monitoring data. The failure task is passed to the high-frequency continuous failure task early warning module. The high-frequency continuous failure task online identification module includes three sub-modules:

51> Resource exception judgment

The main function of the resource exception judgment module is to receive the converted online events and resource usage data, judge whether the resource usage of each task i in each cycle t is in the same resource exception window as the previous cycle, and pass the judgment result to the failure frequency calculation module. The judgment method is similar to the method used in the resource exception window division sub-module in the time series-based offline analysis and learning module. If the resource usage of the task in the t-th cycle is relative to the resource usage change V _i of the t-1 cycle _,t is greater than the resource change threshold, it is considered that the resource usage of period t is abnormal relative to period t-1, that is, it is located in a different resource exception window

52>failure frequency calculation

After receiving the resource abnormal judgment result of the task at the current moment, the failure frequency calculation module needs to calculate the failure frequency of each task within the current resource abnormal window, and pass it to the high-frequency continuous failure task identification module. The calculation method is shown in the following pseudo code, where N _i is the current cumulative failure frequency of task i, F _i represents the failure frequency of task i in the current window, and S _i is the start time of task i in the current window. That is, if the current time t and time t-1 belong to the same resource exception window, add time t to the resource exception window at time t-1, and calculate the failure frequency in this window; otherwise, open a new resource exception window and Calculate the failure frequency within the window.

53>High-frequency continuous failure task identification

The main function of the high-frequency continuous failure task identification module is to identify high-frequency continuous failure tasks in the cloud computing system according to the failure frequency in the current window and the failure frequency threshold learned by the offline analysis and learning module based on time series. Pass the recognition result to the high-frequency continuous failure task early warning module. The core method of identification is to compare the failure frequency of the task in the current window with the failure frequency threshold. If the failure frequency of task i in the current failure window is greater than the failure frequency threshold, task i is a high-frequency continuous failure task.

6. Early warning of high-frequency continuous failure tasks

The main function of the high-frequency continuous failure task early warning module is to receive and visualize the high-frequency continuous failure task identified by the high-frequency continuous failure task online identification module. Users can view high-frequency continuous failure tasks in the cloud computing system through the high-frequency continuous failure task early warning module, so as to adopt an effective proactive failure recovery mechanism (for example, notify the scheduler to stop scheduling the task and request fault diagnosis for the task or Bug fixes), to avoid unnecessary scheduling load and waste of system resources.

The data processing flow of the whole system is shown in Figure 2. First, the ETL module reads the offline monitoring data from the offline data source and converts the data into a specific data structure. At the same time, the user sets the failure frequency threshold and failure frequency threshold through the threshold configuration module. Continuity Index Threshold, Resource Change Threshold, and Confidence Threshold. Based on the above thresholds, the high-frequency continuous failure task marker marks the high-frequency continuous failure tasks and non-high-frequency continuous failure tasks in the offline data, and then passes the marking results and the received offline data and configuration parameters to the time-based Sequence offline analysis and learning module. According to the acquired data and parameters, the offline analysis and learning module based on time series uses the resource change threshold and confidence threshold to analyze the resource usage pattern and failure frequency characteristics of non-high frequency continuous failure tasks, and learn to a certain level of confidence The failure frequency threshold that can represent the failure frequency characteristics of all non-high frequency continuous failure tasks, and pass the result to the online learning module of high frequency continuous failure tasks.

The online recognition process is triggered by an online data input event. When monitoring data input online, the ETL module reads the data and transforms the data structure in real time. The online recognition module for high-frequency continuous failure tasks uses algorithms to identify high-frequency continuous failure tasks in the online data in real time based on the acquired data and parameters such as the failure frequency threshold, and transmits the results to the high-frequency continuous failure task early warning module . The high-frequency continuous failure task early warning module displays the identification results, thereby prompting the cloud computing system management personnel to take corresponding measures. The online recognition process will be carried out cyclically with the online data input until the online data input is completed.

It should be noted that the purpose of the disclosed embodiments is to help further understand the present invention, but those skilled in the art can understand that various replacements and modifications are possible without departing from the spirit and scope of the present invention and the appended claims of. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the protection scope of the present invention is subject to the scope defined in the claims.

Claims (10)

1. An online identification method for high-frequency continuous failure tasks in a cloud computing system. According to offline monitoring data, offline analysis and learning based on time series is performed, and the failure frequency that can represent all non-high-frequency continuous failure tasks at a certain level of confidence is obtained. The failure frequency threshold of the feature, and then identify the high-frequency continuous failure tasks in the online data; including the following steps: 1) Extract event and resource time series data from offline monitoring data, and obtain offline data format conversion results by converting to a specific format, including task failure frequency and resource usage; 2) Configure parameter values, the parameters include a failure frequency threshold, a failure continuation index threshold, a resource change threshold and a confidence threshold; the failure frequency threshold and the failure continuation index threshold are used to define high-frequency continuous failure tasks; 3) According to the offline data format conversion result described in step 1) and the configuration parameter value described in step 2), the tasks in the offline data are marked as high-frequency continuous failure tasks or non-high-frequency continuous failure tasks; 4) Using the resource change threshold and confidence threshold in step 2), analyze the resource usage pattern and failure frequency characteristics of non-high frequency continuous failure tasks, and learn to represent all non-high frequency continuous failure tasks at a certain confidence level The failure frequency threshold of the task failure frequency feature; 5) Input the online monitoring data in real time, extract the event and resource time series data of the task, convert it into a specific format, and obtain the online data format conversion result, including the task failure frequency and resource usage; 6) According to the online data format conversion result obtained in step 5) and the failure frequency threshold obtained in step 4), the high-frequency continuous failure tasks in the online data are identified in real time. 2. the online identification method of high-frequency continuous failure task in cloud computing system as claimed in claim 1, is characterized in that, step 1) event and resource time series data are converted into specific format, specifically: It is set that there are n tasks in the event and resource time series data, task i (1≤i≤n) is submitted to enter the system at time t _i0 , and the residence time in the system is T _i time periods; for each task i extracts T _i triples {t, F _{i, t} , R _{i, t} }; For the time series data of events and resources, a specific format is obtained as a list of four-tuples {i,t,F _i,t ,R _i,t }; where, t∈{t _i0 ,t _i0 +1,...,t _i0 +T _i -1} represents each time period that task i resides in the system; F _i,t is the failure frequency of task i in the tth time period, which is obtained by counting failure events in offline data, If task i does not have a failure event in the tth time period, F _i,t is 0; R _i,t is the resource usage of task i in the tth time period, which is obtained by the monitoring system and passed offline data source incoming. 3. as claimed in claim 1, the online identification method of high-frequency continuous failure tasks in the cloud computing system is characterized in that, step 3) tasks in the offline data are marked as high-frequency continuous failure tasks or non-high-frequency continuous failure tasks tasks, specifically: According to the failure frequency of the task in the offline data described in step 1), calculate the failure total frequency of the task; obtain the failure continuity index according to the percentage of the continuous failure times accounting for the total failure frequency of the task; make the failure total frequency greater than the step 2) If the failure frequency threshold and the failure continuation index is greater than the failure continuation index threshold in step 2), it is marked as a high-frequency continuous failure task; otherwise, it is marked as a non-high-frequency continuous failure task. 4. as claimed in claim 1, the online identification method of high-frequency continuous failure tasks in the cloud computing system is characterized in that, step 4) obtains the failure frequency characteristics that can represent all non-high-frequency continuous failure tasks on a certain confidence level by learning The failure frequency threshold of , specifically includes the following steps: 41) dividing the resource exception window; The degree of resource abnormality is obtained according to the amount of resource usage, and the resource change threshold set in step 2) is used to divide the residence time of non-high-frequency continuous failure tasks in the system into multiple resource abnormal windows according to the degree of resource abnormality. The division result of the resource exception window of a task; 42) Calculate the failure frequency; According to the division result of the resource exception window of each task obtained in step 41), the failure frequency of each task in each resource exception window is calculated; then the maximum value of each task's failure frequency in all windows is set as the failure frequency of the task; 43) Obtaining the failure frequency threshold through learning; According to the failure frequency of all non-high-frequency continuous failure tasks obtained in step 42), the cumulative distribution function of task failure frequency is fitted, and the confidence threshold set in step 2) is calculated to represent Failure frequency threshold for failure frequency of all non-high frequency consecutive failure tasks. 5. the online identification method of high-frequency continuous failure task in cloud computing system as claimed in claim 4, it is characterized in that, step 41) the division method of described resource abnormality window is specifically as follows: If there is a significant difference between the resource usage of the task in the t-th time period and the resource usage of the task in the t-1 time period, set the t-th time period to belong to a new resource exception window, otherwise t belongs to the same window as t-1; Whether there is a significant difference in resource usage can be measured by resource usage variation. The resource usage variation V _i,t is calculated by the resource usage R _{i,t of task i at time t} through formula 1: In formula 1, V _i,t is the resource usage change; R _i,t-1 is the resource usage of task i at time t-1; R _i,t is the resource usage of task i at time t; p( R _i ) indicates the magnitude of resource usage of task i, which is used to standardize resource usage changes and avoid negative impacts on resource usage changes due to different resource usage magnitudes of different tasks; if V _i,t is greater than the resource change threshold, a new window, otherwise keep the previous window. 6. as claimed in claim 4, the online identification method of high-frequency continuous failure tasks in the cloud computing system is characterized in that, step 42) calculates the method for obtaining the failure frequency of each task as follows: First calculate the failure frequency of each task in each resource exception window; suppose the start time of the jth resource exception window of task i is S _ij , and the end time is E _ij ; F _i,t is the The failure frequency within a time period; then the failure frequency Fre _ij of task i in the jth window can be obtained by formula 2: The overall failure frequency Fre _i of task i is set as the maximum failure frequency of task i in all windows by Equation 3, which is used as the judgment value to distinguish between high-frequency continuous failure tasks and non-high-frequency continuous failure tasks: Fre _i ＝max(Fre _ij ) Formula 3 In formula 3, Fre _ij is the failure frequency of task i in the jth window. 7. the online identification method of high-frequency continuous failure task in cloud computing system as claimed in claim 4, it is characterized in that, step 43) the computing method of described failure frequency threshold is as follows: First, according to the failure frequency Fre _i of all non-high frequency continuous failure tasks, the cumulative distribution function F ₁ (fre) of the task failure frequency is fitted, and the formula 4 will satisfy that F ₁ (fre) is equal to the fre of the confidence threshold V _conf As the failure frequency threshold f _thre : f _thre ＝F ₁ ^-1 (V _conf ) Formula 4 In Formula 4, F ₁ ^-1 (fre) is the inverse function of F ₁ (fre); V _conf is the confidence threshold. 8. the online identification method of high-frequency continuous failure task in cloud computing system as claimed in claim 1, is characterized in that, step 6) online identification of high-frequency continuous failure task specifically comprises the following steps: 51) According to the converted results of online events and resource usage data, divide and obtain resource exception window results; The division method of the resource exception window is as follows: If there is a significant difference between the resource usage of the task in the t-th time period and the resource usage of the task in the t-1 time period, set the t-th time period to belong to a new resource exception window, otherwise t belongs to the same window as t-1; Whether there is a significant difference in resource usage can be measured by resource usage variation. The resource usage variation V _i,t is calculated by the resource usage R _{i,t of task i at time t} through formula 1: In formula 1, V _i,t is the resource usage change; R _i,t-1 is the resource usage of task i at time t-1; R _i,t is the resource usage of task i at time t; p( R _i ) indicates the magnitude of resource usage of task i, which is used to standardize resource usage changes and avoid negative impacts on resource usage changes due to different resource usage magnitudes of different tasks; if V _i,t is greater than the resource change threshold, a A new window, otherwise it will remain in the previous window; 52) Calculate the failure frequency of each task within the current resource exception window; Set N _i as the current cumulative failure frequency of task i, F _i represents the failure frequency of task i in the current window, S _i is the start time of task i in the current window; if the current time t and time t-1 belong to For the same resource exception window, add time t to the resource exception window at time t-1, and calculate the failure frequency in this window; otherwise, open a new resource exception window and calculate the failure frequency in this window; 53) Identify high-frequency continuous failure tasks; According to the failure frequency of tasks obtained in step 52) in the current window and the failure frequency threshold obtained in step 4), the high-frequency continuous failure tasks in the cloud computing system are identified by comparison; if task i is in the current failure window If the failure frequency is greater than the failure frequency threshold, task i is a high-frequency continuous failure task. 9. The online identification method of high-frequency continuous failure tasks in the cloud computing system as claimed in claim 1, wherein the high-frequency continuous failure tasks in the online data are identified to be visualized, and effective proactive failure recovery is adopted Mechanism to avoid unnecessary scheduling load and waste of system resources. 10. An online identification system for high-frequency continuous failure tasks in a cloud computing system, including an ETL module, a threshold configuration module, a high-frequency continuous failure task marker, an offline analysis and learning module based on time series, and a high-frequency continuous failure task Task online identification module and high-frequency continuous failure task early warning module; it is characterized in that, The ETL module is used to process the input offline monitoring data and online monitoring data, extract event and resource time series data from the data, perform conversion, and transfer the conversion results including task failure frequency and resource usage to high frequency Continuous failure task marker and high-frequency continuous failure task online identification module; The threshold configuration module is used to configure the failure frequency threshold, the failure continuity index threshold, the resource change threshold and the confidence threshold, and pass the configuration parameter value to the high-frequency continuous failure task marker; the configuration failure frequency threshold and the failure continuity index threshold It is used to define high-frequency continuous failure tasks; the resource change threshold is used to judge whether the resource usage is abnormal; the confidence threshold is used to limit the confidence level that the learning results need to meet; According to the failure frequency threshold and the failure continuation index threshold, the high-frequency continuous failure task marker marks the tasks in the offline data as high-frequency continuous failure tasks or non-high-frequency continuous failure tasks, and marks the non-high-frequency continuous failure tasks The event data and resource time series data of the failure task, resource change threshold and confidence threshold are passed to the offline analysis and learning module based on time series; The off-line analysis and learning module based on time series uses the resource change threshold and confidence threshold to analyze the resource usage pattern and failure frequency characteristics of non-high frequency continuous failure tasks, and learns that it can represent all non-high frequency tasks at a certain confidence level The failure frequency threshold value of the failure frequency feature of the second consecutive failure task, and passed to the online identification module of the high frequency continuous failure task; the failure frequency threshold is used to characterize the failure frequency characteristic of the non-high frequency continuous failure task; The online recognition module for high-frequency continuous failure tasks identifies high-frequency continuous failure tasks in the online data according to the failure frequency threshold, obtains high-frequency continuous failure tasks, and passes the identification results to the early warning of high-frequency continuous failure tasks module; The high-frequency continuous failure task early warning module is used to visualize the identified high-frequency continuous failure tasks.