CN107238407A - Project of South-to-North water diversion secure data abnormal patterns find method and system - Google Patents

Abstract

The present invention relates to the technical field of safety monitoring, in particular to a method and system for discovering abnormal patterns of safety data in the South-to-North Water Diversion Project. It focuses on sensor safety threshold determination, monitoring data prediction, and abnormal pattern discovery mechanism construction, and develops sensors based on the "self-help method". The research on safety threshold calculation, monitoring data prediction based on machine learning and multi-model methods, and abnormal pattern discovery mechanism based on intelligent information processing provides scientific theoretical basis and auxiliary decision-making for the South-to-North Water Diversion Project safety abnormal pattern discovery, including: Grouping; determining the safety threshold of each sensor; determining the established threshold of each group of sensors; judging whether the data received by any sensor exceeds the safety threshold of the sensor; judging whether the ratio of abnormal sensors in the group where the sensor is located exceeds the established threshold value.

Description

Method and system for discovering abnormal patterns in safety data of South-to-North Water Diversion Project

technical field

The invention relates to the technical field of safety monitoring, in particular to a method and system for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project.

Background technique

The South-to-North Water Diversion Project is a strategic infrastructure project to alleviate the shortage of water resources in Beijing, Tianjin, Hebei, Henan and other northern regions and to optimize the allocation of water resources in my country. The project starts from the Danjiangkou Reservoir in Hubei in the south and goes directly to Tuancheng Lake in Beijing and the Outer Ring River in Tianjin. The project has a total length of 1432km and is the largest cross-basin water diversion project in the world. The complexity of geological and meteorological conditions along the main line of the South-to-North Water Diversion Project poses severe challenges to the safety of the South-to-North Water Diversion Project. In order to ensure the safety of the project, more than 60,000 sensors have been deployed along the route of the South-to-North Water Diversion Project to collect project safety data in real time, so as to achieve full-time and comprehensive understanding of project safety. However, the mass data collected by sensors also has the problem of large amount of data but small amount of information. How to analyze and mine valuable information from the massive data and predict the trend of data changes is of great significance to ensure the safety of the South-to-North Water Diversion Central Route Project.

At present, there are still some problems in the work of using multi-sensors for engineering safety data collection, analysis, early warning and decision-making on the main line of the South-to-North Water Diversion Middle Line, including:

1) In the sensing and communication system, the complexity of the working environment leads to a large number of missing and abnormal phenomena in the monitoring data;

2) The traditional model based on hydraulic engineering mechanism analysis can analyze the mechanism of engineering entities under simple conditions, but it has certain limitations when analyzing and dealing with complex problems. At the same time, the use of the mechanism model usually requires certain requirements or assumptions on the data, and the model itself must have a clear mathematical form. But the distribution of real-world data is often complex and it is difficult to make any assumptions.

3) The traditional mathematical statistical method can obtain the generation mechanism of engineering safety data through the analysis of engineering safety data, but it has no predictive ability for monitoring data itself.

Proceeding from the above practical problems, the present invention selects typical buildings in the main line of the South-to-North Water Diversion Middle Line as the research object, and performs data preprocessing for multi-type sensor detection data of the same building, and multi-type, multi-dimensional South-to-North Water Diversion Project safety monitoring data , data modeling, data prediction and engineering safety data early warning and other issues.

Contents of the invention

In view of this, the present invention provides a method and system for discovering abnormal patterns of safety data in the South-to-North Water Diversion Project, focusing on the three aspects of sensor safety threshold determination, monitoring data prediction and abnormal pattern discovery mechanism construction, and launching sensor safety threshold calculation based on "self-help method", The research on monitoring data prediction based on machine learning and multi-model methods, and the construction of anomaly pattern discovery mechanism based on intelligent information processing provides a scientific theoretical basis and auxiliary decision-making for the discovery of safety anomalies in the South-to-North Water Diversion Project.

In order to achieve the above object, the present invention is achieved through the following technical solutions:

The present invention provides a method for discovering abnormal patterns of safety data in the South-to-North Water Diversion Project, including: grouping sensors; determining the safety threshold of each sensor; determining the predetermined threshold of each group of sensors; judging whether the data received by any sensor exceeds the Safety threshold: If it does not exceed the safety threshold, record the data; if it exceeds the safety threshold, judge that the sensor is abnormal, and proceed to the next step; judge whether the ratio of abnormal sensors in the group where the sensor is located exceeds the predetermined threshold: If it exceeds the predetermined threshold, an alarm will be issued; if it does not exceed the predetermined threshold, the sensor data will be predicted, and the safety threshold of the sensor will be re-determined.

Further, grouping the sensors includes: automatically clustering and grouping the sensors according to the time-space sequence of the sensors.

Further, determining the safety threshold of each sensor includes: sampling the data samples of each sensor; and calculating the sampling samples to obtain the confidence interval of each sensor, that is, the safety threshold of the sensor.

Further, determining the predetermined threshold of each group of sensors includes: presetting the upper limit of the proportion of abnormal sensors in each group of sensors, exceeding the upper limit means exceeding the predetermined threshold.

Further, re-determining the safety threshold of the sensor includes: establishing a nonlinear regression model of the sensor and the sensor data of the sensor group; using the nonlinear regression model to predict the data of the sensor; re-determining the sensor according to the prediction result safety threshold.

Further, after re-determining the safety threshold of the sensor, it also includes: judging the data received by the sensor according to the re-determined sensor safety threshold: if it does not exceed the new safety threshold, record the data; if it exceeds the new If the safety threshold is determined, it is judged that the sensor is abnormal, and it is judged in the group where the sensor is located whether it exceeds the predetermined threshold.

The present invention also provides a system for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project, including: a plurality of sensors for collecting engineering safety data; a sensor grouping module for grouping sensors; a safety threshold determination module for determining each sensor The safety threshold of the sensor; The established threshold determination module is used to determine the predetermined threshold of each group of sensors; The first judgment module is used to judge whether the data received by the sensor exceeds the safety threshold of the sensor; The second judgment module is used to judge whether each Whether the ratio of abnormal sensors in the sensor group exceeds the predetermined threshold.

Further, the sensor grouping module automatically clusters and groups the sensors according to the time-space sequence of the sensors.

Further, it also includes: a module for re-determining the safety threshold, configured to use a nonlinear regression model to predict the data of the sensor, and re-determine the safety threshold of the sensor according to the prediction result.

The invention provides a method for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project, which has the following beneficial effects:

The method of the present invention integrates machine learning, complex system parameter estimation, and time-space sequence mining methods into complex frontier scientific problems such as "intelligent clustering of time-space sequences", "prediction of time-space sequence parameters", and "discovery of abnormal patterns of time-space sequences". The early warning of the abnormal safety mode of the South-to-North Water Diversion Project was completed. It not only embodies the deep intersection and integration of modern machine learning methods and complex system parameter estimation, but also a bold attempt to solve the precise prediction of time-space sequences and the discovery of sequence abnormal patterns through modern computer intelligence methods. The advantages of the method of the present invention are mainly reflected in the following several aspects:

1. Based on the theory of time-space sequence mining, research the adaptive segmentation method and sequence clustering method of time-space sequence, and complete the automatic grouping of monitoring sensors;

2. Get rid of the traditional mechanism model that needs to establish multiple assumptions and precise data models before data processing, and broaden the data processing methods of water conservancy information. Only focus on the data characteristics of the original monitoring data itself, and use high-dimensional data regression analysis theory and modern machine learning methods to build a correlation model between time-space sequence parameters to achieve the goal of accurate prediction of time-space sequence parameters;

3. Deepen modern intelligent information processing methods and integrate complex system parameter estimation technology, construct a method for discovering abnormal patterns of time-space series, and complete the goal of timely early warning of the safety of the South-to-North Water Diversion Project.

The beneficial effects of the system for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project are similar to those of the method for discovering the abnormal patterns of safety data of the South-to-North Water Diversion Project, and will not be repeated here.

Description of drawings

Fig. 1 is a schematic flow diagram of a method for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project provided by an embodiment of the present invention;

Fig. 2 is a structural frame diagram of a system for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project provided by an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Exemplary embodiments of the present invention are described below in conjunction with specific circumstances:

The present invention provides a method for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project, which mainly includes the following steps:

Group sensors into groups;

Determine safety thresholds for each sensor;

Determine the established threshold for each group of sensors;

Judging whether the data received by any sensor exceeds the safety threshold of the sensor: if it does not exceed the safety threshold, record the data; if it exceeds the safety threshold, judge that the sensor is abnormal and proceed to the next step;

Judging whether the ratio of abnormal sensors in the group where the sensor is located exceeds the predetermined threshold: if it exceeds the predetermined threshold, an alarm will be issued; if it does not exceed the predetermined threshold, the sensor data will be predicted and the safety of the sensor will be re-determined threshold.

Please refer to Fig. 1, Fig. 1 is a schematic flow chart of the method for discovering the abnormal mode of safety data of the South-to-North Water Diversion Project provided by the embodiment of the present invention; this embodiment provides a method for discovering the abnormal mode of safety data of the South-to-North Water Diversion Project, which specifically includes the following steps:

Step S101, performing automatic clustering and grouping of sensors according to their spatio-temporal sequences. Based on the theory of spatio-temporal sequence mining, the self-adaptive segmentation method and sequence clustering method of time-space sequence are studied, and the monitoring sensors are automatically grouped.

Step S102, determining the safety threshold of each sensor. The data samples of each sensor are sampled; the sampled samples are calculated to obtain the confidence interval of each sensor, that is, the safety threshold of the sensor.

In this embodiment, for the safety monitoring data of the South-to-North Water Diversion Project, the "self-service method" is used to randomly sample the data samples 100 times with replacement, and the sample size of each sampling is 60% of the total data volume monitored by a single sensor, and then 100 The confidence interval of each sensor is calculated by sampling samples, and the sensor safety threshold interval is established to provide a judgment basis for the next step of data anomaly pattern discovery.

As an implementable manner, when calculating the confidence interval, the confidence degree is selected as 95%.

Step S103, determining a predetermined threshold value of each group of sensors. The upper limit of the proportion of abnormal sensors in each group of sensors is set in advance, exceeding the upper limit means exceeding the established threshold.

Step S104, judging whether the data received by any sensor exceeds the safety threshold of the sensor: if it does not exceed the safety threshold, record the data, that is, process it as normal data; if it exceeds the safety threshold, judge that the sensor is abnormal, and perform Next step;

Step S105 , judging whether the ratio of abnormal sensors in the sensor group exceeds a predetermined threshold: if it exceeds the predetermined threshold, an alarm is issued; if it does not exceed the predetermined threshold, the next step is performed.

Step S106, establishing a nonlinear regression model of the sensor and the sensor data of the sensor group;

Step S107, using a nonlinear regression model to predict the data of the sensor;

Step S108, redetermine the safety threshold of the sensor according to the prediction result.

Step S109, judge the data received by the sensor according to the re-determined sensor safety threshold: if it does not exceed the new safety threshold, record the data; if it exceeds the new safety threshold, judge that the sensor is abnormal, and In the group where the sensor is located, it is judged whether the predetermined threshold is exceeded.

Use the "self-help method" to get the safety threshold of each sensor; when new data comes, first judge whether it exceeds the sensor safety threshold, if not, then process it according to the normal data, if it exceeds the safety threshold, choose to cluster with the current sensor Investigate the sensor groups of the same group; for the relevant sensor group, if the ratio of abnormal sensors in the group is greater than the predetermined threshold within this time period, an engineering-level alarm will be issued; if an abnormality occurs in the relevant sensor group within the event period If the sensor ratio is less than the predetermined threshold, the machine learning method is used to establish a nonlinear regression model between the current sensor and the relevant sensor group, and the monitoring data of the sensor group is used to predict the current sensor data and generate a data security at that moment. Space; under the new safety interval, if the current monitoring value of the sensor exists in the new safety interval, it will be treated as normal data, otherwise, a sensor-level alarm will be issued.

The embodiment of the present invention also provides a system for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project, including:

Multiple sensors to collect engineering safety data;

The sensor grouping module is used to group the sensors;

a safety threshold determination module, configured to determine a safety threshold for each sensor;

An established threshold determination module, configured to determine an established threshold for each group of sensors;

The first judging module is used to judge whether the data received by the sensor exceeds the safety threshold of the sensor;

The second judging module is used to judge whether the ratio of abnormal sensors in each group of sensors exceeds a predetermined threshold.

Further, the sensor grouping module automatically clusters and groups the sensors according to the time-space sequence of the sensors.

Further, it also includes:

The safety threshold re-determining module is used for predicting the data of the sensor by using the nonlinear regression model, and re-determining the safety threshold of the sensor according to the prediction result.

Finally, it should also be noted that in this text, relational terms such as first and second etc. are only used to distinguish one entity or operation from another, and do not necessarily require or imply that these entities or operations, any such actual relationship or order exists. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The specific implementation methods provided by the present invention have been introduced in detail above, and the principles and implementation modes of the present invention have been explained by using specific examples in this paper. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present invention At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as limiting the present invention.

Claims (9)

1. A method for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project, characterized in that it comprises: Group sensors into groups; Determine safety thresholds for each sensor; Determine the established threshold for each group of sensors; Judging whether the data received by any sensor exceeds the safety threshold of the sensor: if it does not exceed the safety threshold, record the data; if it exceeds the safety threshold, judge that the sensor is abnormal and proceed to the next step; Judging whether the ratio of abnormal sensors in the group where the sensor is located exceeds the predetermined threshold: if it exceeds the predetermined threshold, an alarm will be issued; if it does not exceed the predetermined threshold, the sensor data will be predicted and the safety of the sensor will be re-determined threshold. 2. The South-to-North Water Diversion Project safety data anomaly pattern discovery method according to claim 1, is characterized in that, grouping sensors includes: Automatically cluster and group sensors based on their spatiotemporal sequences. 3. The South-to-North Water Diversion Project safety data anomaly pattern discovery method according to claim 1, is characterized in that, determining the safety threshold of each sensor comprises: Sample data samples from each sensor; The confidence interval of each sensor is obtained by calculating the sampling samples, that is, the safety threshold of the sensor. 4. The South-to-North Water Diversion Project safety data abnormal pattern discovery method according to claim 1, characterized in that determining the predetermined threshold of each group of sensors includes: The upper limit of the proportion of abnormal sensors in each group of sensors is set in advance, exceeding the upper limit means exceeding the established threshold. 5. The South-to-North Water Diversion Project safety data abnormal pattern discovery method according to claim 1, characterized in that re-determining the safety threshold of the sensor comprises: Establish a non-linear regression model of the data of the sensor and the sensor group where the sensor is located; Using a nonlinear regression model to predict the data of the sensor; Re-determine the sensor's safety threshold based on the predicted results. 6. The South-to-North Water Diversion Project safety data abnormal pattern discovery method according to claim 1, characterized in that after re-determining the safety threshold of the sensor, it also includes: Judgment is made on the data received by the sensor according to the re-determined sensor safety threshold: if it does not exceed the new safety threshold, record the data; if it exceeds the new safety threshold, it is judged that the sensor is abnormal, and It is judged within the group whether it exceeds the established threshold. 7. A system for discovering abnormal patterns in safety data of the South-to-North Water Diversion Project, characterized in that it includes: Multiple sensors to collect engineering safety data; The sensor grouping module is used to group the sensors; a safety threshold determination module, configured to determine a safety threshold for each sensor; An established threshold determination module, configured to determine an established threshold for each group of sensors; The first judging module is used to judge whether the data received by the sensor exceeds the safety threshold of the sensor; The second judging module is used to judge whether the ratio of abnormal sensors in each group of sensors exceeds a predetermined threshold. 8. The system for discovering abnormal patterns of safety data of the South-to-North Water Diversion Project according to claim 7, wherein the sensor grouping module performs automatic clustering and grouping of the sensors according to the time-space sequence of the sensors. 9. The South-to-North Water Diversion Project safety data abnormal pattern discovery system according to claim 7, is characterized in that, also comprises: The safety threshold re-determining module is used for predicting the data of the sensor by using the nonlinear regression model, and re-determining the safety threshold of the sensor according to the prediction result.