CN116109002A - Gas equipment early warning system and method based on big data analysis - Google Patents

Abstract

The invention relates to the technical field of gas early warning, in particular to a gas equipment early warning system and method based on big data analysis, wherein the method comprises the following steps: data acquisition and reading: firstly, historical pressure data are collected to an HDFS through Hadoop, and then a data set of the HDFS is read through Spark; data preprocessing: firstly converting character type features into numerical value types, filtering redundant features, and automatically identifying category features and continuity features by setting the maximum value of category numbers; dividing data: dividing the data set into a training set and a testing set; parameter setting: setting a Spark MLlib decision tree regression model parameter; conversion and assembly: feature conversion of the decision tree regression model is carried out, and the decision tree regression model training assembly is assembled on a production line; training and predicting: and training a decision tree regression model, and carrying out early warning through a prediction result of the decision tree regression model. The invention solves the technical problems that the prior art cannot accurately detect and early warn in time.

Description

Gas equipment early warning system and method based on big data analysis

technical field

The invention relates to the technical field of gas early warning, in particular to a gas equipment early warning system and method based on big data analysis.

Background technique

With the rapid development of urban construction and economic construction, as well as the general improvement of people's living standards and the development of the petrochemical industry, natural gas, as a high-quality and efficient clean energy, has gradually become the dominant source of urban gas, promoting social and economic development, reducing pollution to the environment. Natural gas is composed of a mixture of gaseous low molecular weight hydrocarbons and non-hydrocarbon gases, including methane (85%) and a small amount of ethane (9%), propane (3%), nitrogen (2%) and butane (1%). As fuel, it can produce carbon black, chemicals and liquefied petroleum gas. Propane and butane produced from natural gas are also important raw materials for modern industry. There are more and more residential buildings using natural gas in cities, especially in densely populated residential buildings. The peak hours of using natural gas in daily life are relatively concentrated. If natural gas leaks, it is easy to cause serious safety accidents. At present, it is only possible to detect abnormalities in the process of gas use in general, and it is impossible to accurately detect and give timely warnings, which makes the rectification efficiency of the staff low.

Contents of the invention

The invention provides a gas equipment early warning system and method based on big data analysis, which solves the technical problem that the prior art cannot accurately detect and timely early warning.

The basic solution provided by the present invention is: a gas equipment early warning method based on big data analysis, including:

S1. Data collection and reading: first collect the historical pressure data of gas equipment to HDFS through Hadoop, and then read the HDFS data set through Spark;

S2. Data preprocessing: first convert character-type features into numeric types and filter redundant features, and then automatically identify category features and continuity features by setting the maximum number of categories;

S3. Data division: divide the data set into training set and test set;

S4, parameter setting: set Spark MLlib decision tree regression model parameters;

S5. Transformation and assembly: perform feature transformation of the decision tree regression model, and assemble the decision tree regression model training on the assembly line;

S6. Training and prediction: conduct decision tree regression model training, and give early warning through the prediction results of the decision tree regression model.

The working principle and advantages of the present invention are: collect historical pressure data to HDFS through Hadoop, then read the data set of HDFS through Spark, convert character-type features into numerical types and filter redundant features, and then set the number of categories Automatically identify category features and continuous features, divide the data set into training set and test set, set the parameters of the decision tree regression model, perform feature conversion involved in the decision tree regression model, and assemble the model training on the pipeline. Model training and early warning of the results of the prediction model. Compared with the current existing technology, this solution collects historical pressure data to HDFS through Hadoop, and then reads the HDFS data set through Spark and uses Spark MLlib decision number tree regression algorithm The training data set can use the trained results for formal data comparison, and can accurately detect and give timely warnings.

The present invention collects historical pressure data to HDFS through Hadoop, and then reads the data set of HDFS through Spark and uses the Spark MLlib decision tree regression algorithm to train the data set. The trained results can be used for formal data comparison, which solves the existing problem Technical problems that technology cannot accurately detect and give timely warnings.

Further, in S1, the fields used for the pressure data include the temperature, pressure, collection time, and whether the gas consumption is peak or not during collection.

The beneficial effect is that temperature, pressure, gas consumption peaks are highly correlated with gas hazards, and at the same time, it is easy to detect, which is conducive to accurate detection and timely early warning.

Further, in S2, the pressure feature of the pressure data is used as a mark, and the features with different values less than or equal to 24 are identified as category features, and the features with different values of pressure data greater than 24 are identified as continuous features, and the classification features are indexed or Numerical.

The beneficial effect is that different values mean that the same formula can give different results. For example, one result is a positive number and the other is a negative number. Since most of the data sets are categorical features and some are continuous features, when using the decision tree model regression, pass Set the maximum number of categories to automatically identify category features and continuous features, and index or digitize the classification features, which is conducive to the division of data sets and the training of decision tree regression models; classification features are used to represent classification Features, unlike numerical features, which are continuous, classification features are discrete, such as IP addresses, user account IDs, etc.; in a relational database, an index is a separate, physical pair of one or more columns in a database table It is a storage structure for sorting the values of columns, which is a collection of one or several column values in a certain table and the corresponding list of logical pointers to the data pages that physically identify these values in the table. The function of indexing is equivalent to producing similar In the catalog of books, you can quickly find the required content according to the page number in the catalog; digitization refers to transforming many complex and changeable information into measurable numbers and data, which is convenient for storage, analysis and processing.

Further, it also includes S7 and data storage: the prediction results of the decision tree regression model in S6 are stored in Redis for early warning.

The beneficial effect is that the prediction result is stored in Redis, which is convenient for comparison and can be used for early warning at any time.

Further, in S3, when the data set is divided into a training set and a test set, the division is performed randomly.

The beneficial effect is that the random division has more statistical significance, and is beneficial to improving the accuracy of the decision tree regression model training.

Based on the above gas equipment early warning method based on big data analysis, the present invention also provides a gas equipment early warning system based on big data analysis, including:

The data acquisition and reading module is used to first collect historical pressure data to HDFS through Hadoop, and then read the HDFS data set through Spark;

The data acquisition and reading module is connected with a data preprocessing module, and the data preprocessing module is used to first convert character-type features into numeric types and filter redundant features, and then automatically set the maximum number of categories Identify categorical features and continuous features;

The data preprocessing module is connected with a data division module, and the data division module is used to divide the data set into a training set and a test set;

Described data division module is connected with parameter setting module, and described parameter setting module is used for setting SparkMLlib decision tree regression model parameter;

The parameter setting module is connected with a conversion and assembly module, and the conversion and assembly module is used for performing the feature conversion involved in the decision tree regression model, and assembling the decision tree regression model training on the assembly line;

The conversion and assembly module is connected with a training and prediction module, and the training and prediction module is used for decision tree regression model training, and carries out early warning through the prediction result of the decision tree regression model;

The training and prediction module is connected with a data storage module, and the data storage module is used to store the prediction results of the decision tree regression model into Redis for early warning.

The working principle and advantages of the present invention are: collect historical pressure data to HDFS through Hadoop, then read the data set of HDFS through Spark, convert character-type features into numerical types and filter redundant features, and then set the number of categories Automatically identify category features and continuous features, divide the data set into training set and test set, set the parameters of the decision tree regression model, perform feature conversion involved in the decision tree regression model, and assemble the model training on the pipeline. Model training and early warning of the results of the prediction model. Compared with the current existing technology, this solution collects historical pressure data to HDFS through Hadoop, then reads the HDFS data set through Spark, and uses Spark MLlib's decision tree regression Algorithm training data sets, put the trained results into Redis for formal data comparison, and can accurately detect and give timely warnings.

Description of drawings

Fig. 1 is a flow chart of an embodiment of the early warning method for gas equipment based on big data analysis in the present invention.

Detailed ways

Further detailed explanation through specific implementation mode below:

Example 1

The embodiment is basically shown in Figure 1, the early warning method for gas equipment based on big data analysis, including:

S1. Data collection and reading: first collect historical pressure data to HDFS through Hadoop, and then read the HDFS data set through Spark; the fields used include temperature, pressure, collection time, peak gas consumption, temperature, Pressure, gas consumption peaks are highly correlated with gas hazards, and are easy to detect, which is conducive to accurate detection and timely warning, for example, temperature (tempreture), pressure (press), specific time (accurate to seconds) ( date), month (month), whether the peak gas consumption (is_peak), etc.;

In this embodiment, Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without knowing the underlying details of the distribution, and make full use of the power of the cluster for high-speed computing and storage. Hadoop Implemented a distributed file system (Distributed File System), that is, HDFS (Hadoop Distributed File System), HDFS has high fault tolerance, can provide high throughput (high throughput) to access application data, suitable for those with large data The core design of the Hadoop framework is HDFS and MapReduce. HDFS provides storage for massive data, and MapReduce provides calculation for massive data. Among them, MapReduce is a programming model for large-scale Parallel computing of data sets (greater than 1TB), "Map (mapping)" and "Reduce (reduction)" are their main ideas, both borrowed from functional programming languages, with features borrowed from vector programming languages feature, which greatly facilitates programmers to run their programs on distributed systems without distributed parallel programming. The current software implementation is to specify a Map (mapping) function, which is used to convert a set of key values To map a new set of key-value pairs, specify a concurrent Reduce (reduction) function to ensure that each of all mapped key-value pairs shares the same key group; Spark, also known as Apache Spark, is a dedicated A fast and general-purpose computing engine designed for large-scale data processing. Spark is a general-purpose Hadoop MapReduce-like parallel framework open sourced by UC Berkeley AMP lab (AMP Lab of the University of California, Berkeley). Spark has the advantages of Hadoop MapReduce, but different The advantage of MapReduce is that the intermediate output results of jobs can be stored in memory, and there is no need to read and write HDFS. Therefore, Spark is more suitable for MapReduce algorithms that require iteration, such as data mining and machine learning.

S2. Data preprocessing: first convert character-type features into numeric types and filter redundant features, and then automatically identify category features and continuity features by setting the maximum number of categories; in this embodiment, pressure features As a sign, the specific operation is as follows, first:

1. Convert the characteristics of temperature to floating point type

val datal = rawdata.select(

rawdata("tempreture").cast("Double"),

2. Convert the feature of whether the peak gas consumption is into a floating point type

rawdata("is_peak").cast("Double"),

3. Convert the characteristics of specific time to floating point type

rawdata("date").cast("String"),

4. Convert the feature of the month to a floating point type

rawdata("month").cast("Double"),

5. Convert the characteristics of the pressure to a floating point type and modify the column name

rawdata("press").cast("Double").alias("label"))

Secondly, most of the data sets are categorical features, and some values are continuous features, such as temperature. When using the decision tree regression model, you can automatically identify which features are categorical features and which are continuous by setting the maximum number of categories. Features, here the features with different values less than or equal to 24 are regarded as categorical features, and the features greater than 24 are regarded as continuous features, and the categorical features are indexed or numericalized; for example:

Val featureIndexer = new

VectorIndexer().setInputCol("features").setOutputCol("indexedFeatures").setMaxCategories(24)

Before using the linear regression algorithm, you need to use One Hot Encoder to convert the category features to binary vectors (or one-hot encoding) to convert the first 4 category fields or features into binary vectors, as follows:

val data2 = new One Hot Encoder().setInputCol("").setoutputCol("")

Since the One Hot Encoder is not an Estimator, the data using the regression algorithm needs to be processed separately. First, a pipeline is established, and the above conversions are assembled into this pipeline, as follows:

val pipeline en = new Pipeline().setStages(Array(data2,data3,data4))pipeline

Splice the original 1 and converted 3 binary feature vectors into a feature vector, named features_lr:

VectorAssembler().setInputCols(Array("is_peakVec","dateVec","monthVec","tempreture",)).setOutputCol("features_lr")

S3. Data division: divide the data set into training set and test set; when dividing the data set into training set and test set, randomly divide it into decision tree regression model, which is more statistically significant and conducive to improving decision-making The accuracy of tree regression model training; as follows:

val Array(trainingData,testData)＝data1.randomSplit(Array(0.7,0.3),12)

Randomly partition the data_lr dataset for linear regression models:

val Array(trainingData_lr,testData_lr)=data_lr.randomSplit(Array(0.7,0.3),12)

S4, parameter setting: set Spark MLlib decision tree regression model parameters; for example:

featuresCol: feature column name, the default value is features;

labelCol: label column name, the default is label;

predictionCol: prediction result column name, the default is prediction;

maxDepth: the maximum depth of the tree, the default value is 5,

maxBins: The maximum number of discretization of continuous features, and the way to select the split features of each node, the default value is 32;

minInstancesPerNode: the minimum number of instances contained in each node after splitting, the default value is 1;

minInfoGain: The minimum information gain required when splitting a node, the default value is 0.0;

maxMemoryInMB: The maximum memory allocated for histogram aggregation, the default is 256MB,

cacheNodeIds=False,

checkpointInterval: set checkpoint interval (>=1), or not set checkpoint (-1) default value is 10

impurity: the criterion for calculating information gain, the default value is variance;

seed: random seed, default is None,

varianceCol: the column name of the predicted biased sample deviation, the default is none;

val dt = new

DecisionTreeRegressor().setLabelCol("label").setFeaturesCol("indexedFeatures").setMaxBins(64).setMaxDepth(15)

MLlib is functionally similar to machine learning libraries such as Scikit-Learn, but the calculation engine uses Spark, that is, all calculation processes are distributed;

S5. Transformation and assembly: Perform feature transformation involved in the decision tree regression model, and assemble the decision tree regression model training on the pipeline; as follows:

val pipeline_lr = new Pipeline().setStages(Array(assembler_lr,lr))

S6. Training and prediction: conduct decision tree regression model training, and give early warning through the prediction results of the decision tree regression model; as follows:

Training model: val model = pipeline.fit(trainingData)

Prediction model: val predictions = model.transform(testData)

S7, data storage: store the prediction results of the decision tree regression model in Redis for early warning;

Store the prediction results in Redis, which is convenient for comparison and early warning at any time. Redis (Remote DictionaryServer), that is, remote dictionary service, is an open source log type, Key -Value database, and provide APIs in multiple languages.

In this embodiment, collect historical pressure data to HDFS through Hadoop, then read the HDFS data set through Spark, convert character-type features into numeric types and filter redundant features, and then set the maximum number of categories Automatically identify category features and continuous features, divide the data set into training set and test set, set the parameters of the decision tree regression model, perform feature conversion involved in the decision tree regression model, and assemble the model training on the pipeline for model training. And the results of the prediction model will be warned. Compared with the current existing technology, this solution collects historical pressure data to HDFS through Hadoop, and then reads the HDFS data set through Spark and uses Spark MLlib decision number tree regression algorithm to train the data set , the trained results can be used for formal data comparison, which can accurately detect and give timely warnings; in addition, the data collected by pressure equipment can also be synchronized to Kafka in real time through Flink CDC, and the data can be processed in real time with FlinkSQL. Then use the processed data to match the data stored in Redis, and once matched, send an alarm by text message or phone.

Example 2

The only difference from Embodiment 1 is that, based on the above gas equipment early warning method based on big data analysis, the present invention also provides a gas equipment early warning system based on big data analysis, including:

The data acquisition and reading module is used to first collect historical pressure data to HDFS through Hadoop, and then read the HDFS data set through Spark;

The data preprocessing module is used to convert character-type features into numerical types and filter redundant features, and then automatically identify category features and continuity features by setting the maximum number of categories;

The data division module is used to divide the data set into a training set and a test set;

Parameter setting module, used to set Spark MLlib decision tree regression model parameters;

The conversion and assembly module is used to perform feature conversion involved in the decision tree regression model, and to assemble the decision tree regression model training on the pipeline;

The training and forecasting module is used for training the decision tree regression model and giving early warning through the prediction results of the decision tree regression model;

The data storage module is used to store the prediction results of the decision tree regression model in Redis for early warning.

In this embodiment, collect historical pressure data to HDFS through Hadoop, then read the HDFS data set through Spark, convert character-type features into numeric types and filter redundant features, and then set the maximum number of categories Automatically identify category features and continuous features, divide the data set into training set and test set, set the parameters of the decision tree regression model, perform feature conversion involved in the decision tree regression model, and assemble the model training on the pipeline for model training. And the results of the prediction model will be warned. Compared with the current existing technology, this solution collects historical pressure data to HDFS through Hadoop, then reads the HDFS data set through Spark, and uses Spark MLlib's decision tree regression algorithm to train data. Set, put the trained results into Redis for formal data comparison, which can accurately detect and give timely warnings.

What is described above is only an embodiment of the present invention, and the common knowledge such as the specific structure and characteristics known in the scheme is not described too much here, and those of ordinary skill in the art know all the common knowledge in the technical field to which the invention belongs before the filing date or the priority date Technical knowledge, being able to know all the existing technologies in this field, and having the ability to apply conventional experimental methods before this date, those of ordinary skill in the art can improve and implement this plan based on their own abilities under the inspiration given by this application, Some typical known structures or known methods should not be obstacles for those of ordinary skill in the art to implement the present application. It should be pointed out that for those skilled in the art, under the premise of not departing from the structure of the present invention, some modifications and improvements can also be made, which should also be regarded as the protection scope of the present invention, and these will not affect the implementation of the present invention. Effects and utility of patents. The scope of protection required by this application shall be based on the content of the claims, and the specific implementation methods and other records in the specification may be used to interpret the content of the claims.

Claims (6)

1. The gas equipment early warning method based on big data analysis is characterized by comprising the following steps:

s1, data acquisition and reading: firstly, acquiring historical pressure data of gas equipment to an HDFS through Hadoop, and then reading a data set of the HDFS through Spark;

s2, data preprocessing: firstly converting character type features into numerical value types, filtering redundant features, and automatically identifying category features and continuity features by setting the maximum value of category numbers;

s3, data division: dividing the data set into a training set and a testing set;

s4, parameter setting: setting a Spark MLlib decision tree regression model parameter;

s5, conversion and assembly: feature conversion of the decision tree regression model is carried out, and the decision tree regression model training assembly is assembled on a production line;

s6, training and predicting: and training a decision tree regression model, and carrying out early warning through a prediction result of the decision tree regression model.

2. The gas equipment early warning method based on big data analysis according to claim 1, wherein the fields adopted by the pressure data in S1 include temperature, pressure, acquisition time, and whether gas peak is used or not at the time of acquisition.

3. The gas equipment pre-warning method based on big data analysis according to claim 2, wherein the pressure characteristics of the pressure data are used as a sign in S2, the characteristics with different values smaller than or equal to 24 of the pressure data are identified as category characteristics, the characteristics with different values larger than 24 are identified as continuity characteristics, and the category characteristics are indexed or digitized.

4. The gas equipment early warning method based on big data analysis according to claim 3, wherein the data set is divided into a training set and a test set in S3 randomly.

5. The gas equipment early warning method based on big data analysis according to claim 4, further comprising S7, data storage: and (3) storing the prediction result of the decision tree regression model in the S6 into Redis for early warning.

6. Gas equipment early warning system based on big data analysis, its characterized in that includes:

the data acquisition and reading module is used for acquiring historical pressure data to the HDFS through Hadoop and then reading a data set of the HDFS through Spark;

the data acquisition and reading module is connected with a data preprocessing module, and the data preprocessing module is used for converting character type features into numerical value types and filtering redundant features, and automatically identifying category features and continuity features by setting the maximum value of category numbers;

the data preprocessing module is connected with a data dividing module, and the data dividing module is used for dividing a data set into a training set and a testing set;

the data dividing module is connected with a parameter setting module, and the parameter setting module is used for setting parameters of a Spark MLlib decision tree regression model;

the parameter setting module is connected with a conversion and assembly module, and the conversion and assembly module is used for carrying out feature conversion related to the decision tree regression model and assembling the decision tree regression model training on the assembly line;

the conversion and assembly module is connected with a training and predicting module, and the training and predicting module is used for training the decision tree regression model and performing early warning through the predicting result of the decision tree regression model;

the training and predicting module is connected with a data storage module, and the data storage module is used for storing the predicting result of the decision tree regression model into the Redis for early warning.

Images