CN105303456A - Method for processing monitoring data of electric power transmission equipment - Google Patents

Abstract

The invention provides a method for processing monitoring data of power transmission equipment, including: performing consistent hash storage for multiple backups according to the relevance and time and space attributes of monitoring data, and using a parallel computing framework to perform combined retrieval of multiple monitoring data sources and parallel retrieval and feature analysis. The invention proposes a method for processing monitoring data of power transmission equipment, which efficiently and reliably stores the monitoring data based on cloud computing technology, and realizes quick access and analysis.

Description

Monitoring data processing method for power transmission equipment

technical field

The invention relates to power grid data processing, in particular to a method for processing monitoring data of power transmission equipment.

Background technique

With the rapid growth of the grid scale and the increasingly complex grid structure, power companies have increased the promotion and application of power transmission equipment monitoring, and various types of data acquired and transmitted are also undergoing geometric growth. These data not only include various signals when the equipment is abnormal, status information of various equipment in operation, but also a large amount of related data, such as geographic information, weather, on-site temperature and humidity, and detection videos, images and related data. Documents, etc., gradually constitute the monitoring data of power transmission equipment. A large number of monitoring nodes continuously transmit collected data to the data platform, forming massive heterogeneous data streams. The data platform not only needs to store these data reliably, but also needs to analyze and process these data in a timely manner. Although the existing technology is based on the cloud computing platform to process massive monitoring data, compared with cloud computing applications in the Internet field, the monitoring of power transmission equipment is very different in terms of data storage, communication and calculation. How to efficiently and reliably store the above data, and quickly access and analyze them is an urgent problem to be solved.

Contents of the invention

In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a method for processing monitoring data of power transmission equipment, including:

Consistent hash storage of multiple backups is carried out according to the relevance and time and space attributes of the monitoring data, and the combined retrieval and parallel retrieval and feature analysis of multiple monitoring data sources are performed using the parallel computing framework.

Preferably, the consistent hash storage of multiple backups according to the relevance and time and space attributes of the monitoring data further includes:

Obtain the time and space characteristics of the data collected by each monitoring device, that is, the corresponding collection time and location of the data and the custom correlation coefficient as keywords for data retrieval and analysis; store the data in three backup versions on the cloud platform; use Consistent hashing performs hash mapping of the first backup of data according to the monitoring device number; hash mapping of the second backup of data according to the collection time data; hash mapping of the third backup of data according to the custom correlation coefficient, The correlation coefficient is a specific attribute of the monitoring data, which is assigned according to the needs of the upper application program; the consistent hash storage further includes the following process:

1) Predefining the correlation coefficient of the monitoring data and the number of redundant backups through the configuration file;

2) Calculate the hash value of each storage node in the cloud platform, and configure it to the pre-established circular hash queue interval;

3) Calculate the hash value of the data according to the time and space attributes of the monitoring data and the correlation coefficient, and calculate the first hash according to the source of the data, that is, the monitoring device number, for the first backup of multiple backups of the data existing under the cloud platform value, and map it to the circular hash queue; for the second backup, calculate the second hash value according to the time attribute of the monitoring data, that is, collect time data, and map it to the circular hash queue; for the third backup , calculate the third hash value according to the correlation coefficient of the data, and map it to the circular hash queue; if the cloud platform is configured with more than 3 backups, then alternately calculate the hash value according to the above-mentioned first to third backup methods The column values are mapped to the circular hash queue in turn;

4) Determine the storage location of the data according to the data hash value and the storage node hash value, and map the data to the nearest storage node clockwise;

5) If there is insufficient space in the node where the data will be stored, skip the current node to find the next storage node;

In addition, when reading data, the NameNode sorts multiple storage nodes according to the distance between the storage node and the client and returns them to the client to read data from the nearest node. The distance of is defined as the number of nodes passed by one node to reach another node.

Preferably, the combined retrieval of multiple monitoring data sources further includes:

Retrieve according to the following conditions: equipment attribute data, that is, name, running time, installation location, body parameters, monitoring data, that is, wire temperature, ampacity, tension, environmental data, that is, ambient temperature, humidity and air pressure, geographic information data, that is, altitude, latitude and longitude ; Connecting different data sources, the different data sources come from multiple files; the monitoring equipment collects and uploads unified data on insulation terminal leakage current, wire tension, wire current, wire temperature, and micro-meteorological data, In the case of abnormal insulation terminals, overheating or unbalanced wire joints, relevant information alarms are carried out; in the process of monitoring leakage current, three data files are used: equipment attribute data files, insulation terminal leakage current data files and environmental data files. Retrieve, generate the monitoring data of the monitoring equipment within a predetermined time, and connect the three data files for combined retrieval;

After the monitoring data of the power transmission equipment is stored, the method of retrieving the data is a parallel query method executed on the map side. The data filtering and connection process is completed in the map stage to avoid the reduce stage. The retrieval includes the following steps:

1) According to the search conditions proposed by the user, filter the data and remove the data that does not meet the conditions;

2) According to the retrieval requirement, the primary key is set; the primary key is the monitoring equipment number, time data or correlation coefficient;

3) For each record of each data source, use the data file name as a label to mark;

4) Segment records with the same attribute value into a group according to the primary key, and perform data connection;

The filtering, tag setting, group sorting, and connection operations in the map process of combined retrieval are performed on the local node, and then the result of combined retrieval is output to the distributed file system;

And, the parallel retrieval and feature analysis of multiple monitoring data sources further includes:

Based on the dynamic interrelationship of multi-channel time series, the integrated feature extraction is performed on the multi-channel synchronously collected signal data. First, the data is uploaded to the distributed file system, and the distributed file system divides the data into blocks and randomly distributes them to multiple storages. On the node, the calculation of the dynamic interrelationship of the multi-channel time series is completed in the reduce phase, and the calculation results are output to the distributed file system for storage. Using the time correlation of the data, the collection time data is used as the key to calculate the hash storage location. The feature extraction process further includes:

1) Calculate the task time, filter the data, and remove the data that does not meet the time condition; 2) Use the time data as the primary key to mark each record; 3) Divide the records with the same attribute value into a group according to the primary key, And call the multi-variable sample entropy calculation process, and output the calculation results to the distributed file system.

Compared with the prior art, the present invention has the following advantages:

The invention proposes a method for processing monitoring data of power transmission equipment, which efficiently and reliably stores the monitoring data based on cloud computing technology, and realizes quick access and analysis.

detailed description

A detailed description of one or more embodiments of the invention is provided below. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details.

The present invention conducts research on power transmission equipment monitoring data storage and parallel analysis and processing based on the cloud platform; considers data relevance and time and space attributes, proposes a multiple backup consistent hash storage method for data relevance, and analyzes the data of the cloud platform Segmentation strategy and cloud platform network architecture planning are optimized. On this basis, based on the parallel framework, the data source parallel retrieval of monitoring data and the multi-channel data integration feature extraction parallel computing are realized.

From the perspective of the upper application program of the power transmission equipment monitoring data platform, the distribution of data is mainly affected by the following factors: 1) The data needs to be distributed to each node in the cloud platform as evenly as possible to maintain load balance; 2) The cloud platform cloud platform Node failure is regarded as a normal state, and the problem of node failure needs to be considered when optimizing data distribution; 3) In order to ensure data reliability and retrieval processing efficiency, multiple backup schemes are required; 4) In the operating environment of the cloud platform, network transmission and Disk I/O operations are an important factor affecting the overall performance. If the data communication volume can be reduced, the data processing time will be effectively reduced. Taking the commonly used data association retrieval in the monitoring system as an example, when performing parallel computing association retrieval, the standard cloud platform data layout scheme is adopted (without considering data association), and the connection operation needs to be completed in the Reduce phase. In the Map stage, all data is grouped and sorted on multiple nodes, and then the nodes of the reduce task download the data through remote access. During this process, there may be a large amount of data irrelevant to the last connection operation that is also copied and transmitted in the network. If the data of the same device is stored on the same node as much as possible according to the device attributes of the data when uploading data, the connection operation can be completed in the map stage, eliminating the data communication in the reduce stage, and improving the overall execution efficiency.

According to the above analysis, optimize the data layout of the cloud platform, and use the following data storage methods: centrally store relevant data, and place the main work on the map side during data retrieval and analysis, so as to reduce the intermediate process from mapping to reduce Network communication load, thereby improving overall retrieval and analysis performance. Each type of monitoring data may have different data types and formats, but their common feature is that they all have time and space characteristics, that is, the data collected by each monitoring device corresponds to a specific collection time and a specific collection location. These constitute the most commonly used keywords for data retrieval and analysis. Since the cloud platform stores data in three backup versions by default, the method considers three aspects of correlation: monitoring device location, data collection time, and custom correlation. Using the consistent hash method, the first backup version of the data is hash-mapped according to the monitoring device number; the second backup version of the data is hash-mapped according to the collection time data; The coefficients are hash-mapped to meet different retrieval and data analysis needs. The correlation coefficient can be used as an attribute of the monitoring data, and it can be assigned according to the needs of the upper-layer application to achieve custom correlation. The method needs to build a circular hash queue. The specific process is described as follows:

1) The correlation coefficient of monitoring data and the number of redundant backups are predefined through the configuration file, where the number of redundant backup versions is defined as 3;

2) Calculate the hash value of each storage node in the cloud platform, and configure it on the interval of the circular hash queue;

3) Calculate the hash value of the data according to the time and space attributes and the correlation coefficient of the monitoring data. There are multiple backups of data under the cloud platform. For the first backup version, according to the source of the data, that is, the monitoring device number, calculate the hash value 1, and map it to the circular hash queue; for the second backup version, according to the time attribute of the monitoring data, that is, the collection time data, Calculate its hash value 2 and map it to the circular hash queue. For the third backup version, calculate its hash value 3 according to the correlation coefficient of the data, and map it to the circular hash queue. If higher storage reliability is required and the number of backup versions greater than 3 is configured, the hash value i is calculated alternately according to the above three methods, and mapped to the circular hash queue in turn;

4) Determine the storage location of the data according to the hash value of the data and the hash value of the storage node. Map the data to the nearest storage node clockwise;

5) If there is an abnormal situation such as insufficient space in the node where the data will be stored, skip this node to find the next storage node.

When reading data, the NameNode sorts multiple storage nodes according to the distance between the storage node and the client and returns them to the client, so as to read data from the nearest node. The network nodes in the cloud platform are in a tree structure. The root node of each subtree in the tree is usually a switching node connected to the computer. The distance between two nodes is defined as the number of nodes passed by one node to another node.

The default configuration of the cloud platform considers that all nodes are in one rack, so it is necessary to pass the network architecture of the cloud platform nodes to the cloud platform according to the actual configuration of the cloud platform, so that the cloud platform scheduler can select a reasonable storage node for storage. Data read and write. The network architecture structure can be transmitted to the cloud platform in the form of script code.

The monitoring of power transmission equipment requires a combined retrieval of various online monitoring equipment and line parameters according to the monitoring equipment number, collection time and other conditions. Combined retrieval involves equipment attribute data (name, running time, installation location, etc.), body parameters, monitoring data (conductor temperature, ampacity, tension, etc.), environmental data (environmental temperature and humidity, air pressure, etc.), geographic information data (elevation, Longitude, latitude, etc.) and other data sources, which require data connection of different data sources. Multi-source data usually comes from different files. The monitoring equipment collects and uploads unified data on insulation terminal leakage current, wire tension, wire current, wire temperature, micro-climate and other data. In the case of abnormal insulation terminals, overheating or unbalanced wire joints, relevant information alarms can be carried out. Taking leakage current retrieval as an example, the retrieval involves three data files: equipment attribute data file; insulation terminal leakage current data file; environment data file. Retrieval needs to generate the monitoring data of the monitoring equipment for a period of time, that is, to obtain the monitoring data list with equipment information and environmental information. This requires the connection processing of three data files to obtain a list that meets the requirements.

After the monitoring data of the power transmission equipment is stored, the method of retrieving the data is a parallel query method executed on the map side. The method mainly includes completing the data filtering and connection process in the map stage, avoiding the reduce stage, thereby saving network transmission overhead. The premise of the execution of the method is that the data has been distributed according to the multiple backup consistent hashing method based on data correlation described above, so that the data required for the connection is gathered to the same storage node. The retrieval process can be described as follows:

1) According to the search conditions proposed by the user, filter the data and remove the data that does not meet the conditions;

2) According to the retrieval requirement, set the primary key; the primary key can be monitoring equipment number, time data or correlation coefficient;

3) Each record of each data source is marked, and the data file name can be used as a label for marking;

4) Segment records with the same attribute value into a group according to the primary key, and perform data connection.

After the data distribution is optimized, operations such as filtering, label setting, group sorting, and connection in the map process of combined retrieval are performed on the local node, and the result of combined retrieval is output to the distributed file system.

As the multi-sensing measurement technology is widely used in various power equipment monitoring, the synchronously monitored multi-channel data sequences are collected and saved. The dynamic interrelationships within or between these synchronous multi-channel data sequences contain rich feature information, which can more comprehensively reflect the operating status of power equipment. Based on the dynamic interrelationship of the multi-channel time series, the present invention performs integrated feature extraction on the vibration signal data of vibration monitoring equipment collected synchronously by 6 channels. Under the cloud platform, a parallel feature extraction method is designed based on the consistent hash method to speed up feature extraction.

The synchronously collected 6-channel vibration monitoring signals are independently stored in 6 files, and the signals are stored in segments, and each segment of the signal has time data. In order to complete the parallel analysis of the signal, the data is first uploaded to the distributed file system. The distributed file system divides data into blocks and randomly distributes them to multiple storage nodes. Since the data correlation is not considered, the parallelized data relationship evaluation method can only use the calculation mode of filtering the data on the map side, and sending each segment of the signal to the reduce side through the network for solution. Each channel file is divided into multiple segments and distributed and stored on multiple storage nodes. The calculation of the dynamic interrelationship of the multi-channel time series is completed in the reduce phase, and the calculation results are output to the distributed file system for storage. Apply the above-mentioned data optimization distribution method to redistribute the synchronously collected multi-channel data, and use the time correlation of the data to calculate the hash storage location using the collected time data as a key.

Optimizing distribution enables synchronous data aggregation, and completes computing tasks in map tasks.

The feature extraction process based on the consistent hash algorithm can be described as follows:

1) Calculate the task time, filter the data, and remove the data that does not meet the time condition;

2) Use the time data as the primary key to mark each record;

3) Segment records with the same attribute value into a group according to the primary key, and call the calculation process of multivariate sample entropy. The calculation results are output to the distributed file system.

The multivariate sample entropy calculation process can be described as follows:

1) Suppose the original p-dimensional (channel) time series is {x _{k, i} } _i=1 ^N , k=1, 2, ..., p, where each dimensional sequence has N points. Firstly, for the pre-specified scale factor β, construct a multivariate time series {y _{k, j} ^β }, namely ${{\mathrm{the\; y}}_{k,j}}^{\mathrm{\&beta;}}=\frac{1}{\mathrm{\&beta;}}\underset{i=\left(j-1\right)\mathrm{\&beta;}+1}{\overset{j\mathrm{\&beta;}}{\mathrm{\&Sigma;}}}{x}_{k,i},k=1,2,\mathrm{...},p,$ in $1<j<\frac{N}{\mathrm{\&beta;}}.$

2) Preset p-dimensional parameter embedding vector M[m ₁ , m ₂ ,, m _p ], p-dimensional time delay vector

[T ₁ , T ₂ ,..., T _p ], using the multivariate time series {y _{k, j} ^β }, construct (Nn) composite delay vectors Y _m (i), namely:

3) Define the distance between Y _m (i) and Y _m (j) as d[Y _m (i), Y _m (j)], namely: d[Y _m (i), Y _m (j)] =max _{l=1,..., m} {|x(i+l-1)-x(j+l+1|)}

4) For a given threshold r, calculate event P _i for each i value: d[Y _m (i), Y _m (j)]<r(j≠i) occurrence probability B _i ^m (r)= P _i /(Nn-1), represents the degree of correlation between all Y _m (j) and Y _m (i).

5) Find the average value of B _i ^m (r) for all i, namely:

6) Extend m in step 2) to m+1, and repeat steps 3)-5) to obtain B ^m+1 (r).

7) Calculate the multivariate sample entropy as $mS\mathrm{E.}(m,T,r,N)=-l\mathrm{no}\left(\frac{B^{m+1}\left(r\right)}{B^m\left(r\right)}\right).$

Preferably, the cloud platform of the present invention proposes a merging strategy for small files, a certain number of small files are merged to generate a new storage file, and generally small files belonging to the same attribute are merged. Index files are updated at the same time as new stash files are written to the system. Indexes in the cloud platform include: the primary index is the resource collection to which the file belongs, such as the type; the secondary index is the specific resource entry. When a file needs to be read, the query is performed in the primary index and the secondary index in sequence, which narrows down the scope of the query and ensures a high read response. The core of the storage layer design of the cloud platform of the present invention includes: first merging small files to generate storage files, and then establishing a secondary index for the merged files based on the storage characteristics of the database, and improving the response speed of file reading through index prefetching. The specific details of the storage layer are described in detail below.

Divide the file into blocks or blocks. The default size of the block is 64M. The namespace of the distributed file system is persisted in an image file, which is loaded into memory by the name node at startup. A large number of small files will cause insufficient memory in the name node, and generating too large image files will reduce the efficiency of file search when reading files. For the read and write operations of each file, first query in the namespace to find information such as the block address and file size of the file, and then search in the data node space. When the read file is small, the main time in the read and write process is spent on retrieval and query instead of file data transmission, which affects the processing efficiency of the server cluster.

The cloud platform utilizes small file merging to generate storage files. First implement a filter to filter files by type and size, and select document files that can be searched in full text. In this paper, the file size threshold is set to 10M. When a file is larger than 10M, it is considered a large file and does not need to be merged. After filtering, the cloud platform merges the filtered small files into file blocks according to the resource set to which the file entry belongs. A resource collection is a collection of resource entries with certain dependencies, and a resource entry belongs to only one resource collection. Usually collections are divided according to attribute range, time, etc., and files can be divided according to attribute domains. The resource entries in the new file block have a great correlation, and the file block can be assigned to a MapReduce task in the future data processing, avoiding the waste of task allocation and switching time due to the small amount of calculation of the task, reducing Movement of data within the cluster.

After merging small files, the name node memory is the performance bottleneck of the entire file system, because all file metadata information needs to be stored in its memory. Merging small files can reduce the number of files and save a lot of memory space, but the merged files Reading efficiency will be very low. A preferred embodiment of the present invention adopts a hierarchical index to establish a small file metadata index, and divides a large index file into small index files according to reasonable rules. The resource collection is used as the main index, and the resource entry content under each resource collection is used as the secondary index. In this way, when searching, first search according to the collection where the resource entry is located, and then search in the corresponding secondary index file. Although there is an additional process of searching in the main index, since the number of resource collections is not too large, the search time is very small, and the divided secondary index file is much smaller than the global index file, so the overall search will be improved efficiency. At the same time, not all the secondary index files are loaded into the memory, and can be flexibly scheduled according to the memory usage and combined with the cache strategy to solve the problem of insufficient memory.

The index prefetching proposed here refers to predicting the data that the user will access next based on the data currently accessed by the user, and loading its index into the cache. If the prediction can be made accurately, the data to be accessed by the user can be loaded into the cache in advance, and a faster system response can be obtained when the user accesses it.

Before downloading or browsing resource items, users usually have to obtain an "intermediate result set" through retrieval or directory search, and then they can select the required resource items for further access. There is an interval of several seconds between the user seeing the result set page and executing the download or browsing. During this period, by caching the index of the resource entry in the intermediate result set in advance, it is unnecessary to execute a series of files when the user clicks to download or browse. For metadata query, you can directly transfer files, which can greatly improve the request response of these files. This responsiveness boost doesn't require much memory.

The storage layer architecture of the cloud platform of the present invention is described in detail below. In addition to using the above strategies, the cloud platform has its storage layer architecture as the basis of the system when it is implemented. The cloud platform storage layer is built on the distributed storage system on the Hadoop cluster to provide basic file storage and reading services.

The architecture of the storage layer of the cloud platform adopts a three-layer structure design: user interface layer, business logic layer and storage layer, and in order to improve performance, the method of separating the Web server and the server cluster is adopted. The user interface layer is the user interface provided, and the user sends requests and receives feedback information through the functions provided by this layer. The business logic layer is the functional implementation layer for reading and writing small files, including file merging, index building, and cache building.

The business logic layer includes functional modules such as file merging, retrieval system, small file index, cache and distributed system client. The specific implementation of each module is as follows:

(1) File merging: The file merging function includes two stages: creating a SequenceFile object to merge small files. Merge files that meet the merging requirements through filter filtering. First, search in the main index according to the resource collection where the resource entry is located. After finding the file path corresponding to the resource collection, create a SequenceFile object, obtain the Writer object of the SequenceFile, and It is configured and ready to be written to a file. Start a new thread while executing file writing, and write metadata information such as the file bit value and length corresponding to the resource entry into the sub-index of the resource entry. After the resource entry is successfully written, the output stream is closed and the submission success is returned, otherwise the submission failure is returned.

(2) Retrieval: Provide file retrieval function, and rely on this module to optimize reading of distributed file system based on "intermediate result set".

(3) Small file index: build a small file index, including resource collection primary index and resource entry secondary index, and provide index file creation, append and delete records and other functions.

The primary index data is stored in the relational database, accessed through the relational database access interface, and saved using the Map data structure in Java. Because the resource collection has been stored in the database, according to this index, it is only necessary to add the field with the value generated by the system when the resource entry is added, so it can be stored in the relational database without affecting the processing efficiency. The data in the main index adopts the Key/Value structure, and the Map data structure in Java can be used to improve query efficiency. In addition, in order to ensure retrieval efficiency, the Map object must be initialized according to the content in the database when the service starts and exists all the time. Since there are not many main index files, the Map object occupies very little memory, so the system overhead is limited. When there are new resources When the collection is added or some are deleted, the Map object needs to be updated.

The secondary index is created through the open source project Lucene, which supports small file metadata retrieval. Lucene has a complete set of index construction, update and search solutions, and the query efficiency is very high when the index file is less than 1G, which can be used to build a commercial search engine. The index to be created on the cloud platform requires some special functions, such as real-time update of the index file whenever a user adds a resource entry; concurrency control of file writing when multiple users add resource entries under a resource collection at the same time; compressed index files to reduce memory usage, etc.

(4) Prefetching: In order to better improve the response speed, cache management of "intermediate result sets" that are of interest to users is provided here, including cache space maintenance, cache update, update algorithm maintenance and other functions.

After the user sends a retrieval request, the Web service queries the resource item result set that meets the user's needs according to the user's retrieval condition, returns it to the user, and creates an asynchronous thread to update the cache at the same time. The cache content is updated in the interval between browsing operations. When the cache module receives a request to update the cache content, it calls the index module to retrieve and load the metadata of the current result set entry into the cache. When a user sends a download or browse request, the Web service calls the distributed system client to look up metadata in the cache to start reading the data and transmit it to the client.

The system maintains a thread pool with a fixed number of threads and calls a thread to process each time it receives a cache update request. If there are no idle threads in the thread pool, the cache task is made to wait. In this way, the system resources occupied by the cache update task can be maintained within a reasonable range without affecting the overall system performance. The present invention selects the FIFO algorithm to realize the scheduling function of the cache module, and eliminates the oldest cache entries in the most efficient manner. The specific implementation is: establish a cache pool, configure the size of the cache pool, the default is 32M, and can save 200,000 pieces of file metadata information. The cache pool stores key-value pairs key/value, the file name is used as the key, the data node ID of the file, the combination of the starting position and length is used as the value. The buffer pool provides two operations put and get. Put puts data into the cache pool. If the existing data in the cache pool reaches the upper limit, replace the corresponding data according to the cache replacement algorithm. If there is still space, just put it directly. The Get operation obtains the corresponding value value according to the key value, and returns empty if there is no value.

The distributed system client encapsulates the API for operating the file system to interact with the outside world, including reading and writing files and querying file locations. When the file system receives a file read request, it first judges through the file filter. If the file belongs to the merged file, it first searches the metadata information of the file in the cache. If it does not exist, it searches in the index file. If it still cannot be found, communicate with the name node. After finding the file metadata, construct the SequenceFile object and then obtain the Reader object of the SequenceFile to send a read request to the data node, close the input stream after transmitting the data to the user, and return to complete.

Users have two request methods, one is a write request to submit a file, and the other is a read request to query, browse or obtain resources. When the web server receives the resource request submitted by the user, it first judges whether it is necessary to merge small files. If necessary, it will merge the files. If not, it can directly use the distributed file system writing interface to write. After the file is merged, the distributed file system client prepares to write the file into the distributed file system. When the distributed system client writes the file, the small file index update module is called to perform the small file index and update, because the web server host and The server cluster is separated, and writes and updates can be performed simultaneously by different threads without affecting each other. After the distributed file system is successfully written, the web service returns the submission success message to the client.

Send a file read request when the user needs to browse the detailed content of the file or download the file. The frequency of this request is high and consumes the most system resources. When the web server receives the user's read request, it first searches through the retrieval system according to the conditions submitted by the user, obtains the resource item result set required by the user and returns it to the user for browsing, and at the same time displays the results in a concentrated manner on the first page of the user interface. The entry set (default 20) is sent to the cache module, and a separate thread is opened to update the cache. When the user browses the returned result set page and requests to download or browse details, the web service calls the distributed file system client to prepare to read the file Content, the distributed file system client first searches the file location information in the cache, if not found, it then searches in the small file index, and after finding the location information, it directly reads the data from the data node and returns it to the user.

It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principle of the present invention, and not to limit the present invention. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present invention shall fall within the protection scope of the present invention. Furthermore, it is intended that the appended claims of the present invention embrace all changes and modifications that come within the scope and metesques of the appended claims, or equivalents of such scope and metes and bounds.

Claims (3)

1. a power transmitting device monitor data disposal route, is characterized in that:

The consistance hash of carrying out multiple duplication according to the relevance of monitor data and Time and place attribute stores, and utilizes parallel computation frame to carry out combined retrieval and parallel search and signature analysis to multiple monitor data source.

2. method according to claim 1, is characterized in that, the consistance hash that the described relevance according to monitor data and Time and place attribute carry out multiple duplication stores, and comprises further:

Obtain the Time and place characteristic of each watch-dog image data, the acquisition time that namely data are corresponding and collecting location and self-defined related coefficient are as the key word of data retrieval and analysis; In cloud platform, data are stored as 3 backup versions; Utilize consistance hash that the 1st of data the backup is carried out Hash maps according to watch-dog numbering; The 2nd of data backup is carried out Hash maps according to acquisition time data; The 3rd of data backup is carried out Hash maps according to self-defined related coefficient, and described related coefficient is the particular community of monitor data, and it needs to come assignment according to upper level applications; Described consistance hash stores and comprises following process further:

1) by the described related coefficient of configuration file predefine monitor data and the quantity of redundancy backup;

2) calculate the hashed value of each memory node in cloud platform, and be configured between the circulation hash queue region set up in advance;

3) according to the Time and place attribute of monitor data and the hashed value of Calculation of correlation factor data, to the 1st backup of the multiple backup of the data existed under cloud platform, according to the source of data, i.e. watch-dog numbering, calculate the first hashed value, be mapped in the queue of circulation hash; To the 2nd backup, according to time attribute and the acquisition time data of monitor data, calculate the second hashed value, and be mapped in the queue of circulation hash; To the 3rd backup, according to Calculation of correlation factor the 3rd hashed value of data, and be mapped in the queue of circulation hash; If cloud platform configuration has the backup of more than 3, then alternately calculate its hashed value according to the mode of the above-mentioned first to the 3rd backup and be mapped in the queue of circulation hash successively;

4) according to the memory location of data hash value and memory node hashed value determination data, by clockwise by data-mapping on the memory node nearest apart from it;

5) if the node of storage is occurred insufficient space situation by data, then present node is skipped to find next memory node;

In addition, when carrying out digital independent, namenode returns to client after sorting to multiple memory node according to the distance between memory node and client, to read data from nearest node, wherein, the distance definition between two nodes by a node arrive another node the nodes of process.

3. method according to claim 2, is characterized in that, describedly carries out combined retrieval to multiple monitor data source, comprises further:

Retrieve according to following condition: device attribute data, i.e. title, working time, infield, body parameter, monitor data and conductor temperature, current-carrying capacity, pulling force, environmental data and environment temperature, humidity and air pressure, geographic information data and height above sea level, longitude and latitude; Different data sources is carried out data cube computation, and described different data source comes from multiple file; Watch-dog carries out unified data acquisition to insulated terminal leakage current, wire tension, current in wire, conductor temperature, microclimate data and uploads, abnormal at insulated terminal, terminal is overheated or unbalance the information of carrying out being correlated with report to the police; Wherein in the process of monitoring leakage current, these 3 data files of device attribute data file, insulated terminal leakage current data file and environmental data file are utilized to retrieve, generate the monitor data in the watch-dog schedule time, and 3 data files are carried out connection handling to carry out combined retrieval;

After power transmitting device monitor data completes storage, the method retrieved data is the parallel query method performed at map end, and complete the filtration of data and connection procedure in the map stage and avoid carrying out the reduce stage, retrieval comprises the following steps:

1) according to the search condition that user proposes, data are filtered, remove the data do not satisfied condition;

2) according to Search Requirement, setting major key; Described major key is watch-dog numbering, time data or related coefficient;

3) to every bar record of each data source, Data Filename is adopted to mark as label;

4) according to major key by the record cutting of same alike result value to one group, and carry out data cube computation;

Filtration in the map process of combined retrieval, flag settings, packet sequencing, attended operation are carried out at local node, and then the result of combined retrieval outputs to distributed file system;

Further, described parallel search and signature analysis are carried out to multiple monitor data source, comprise further:

Based on hyperchannel seasonal effect in time series dynamic interrelationships, integration characteristics extraction is carried out to the signal data of multichannel synchronousing collection, first by data upload to distributed file system, by distributed file system by deblocking, and stochastic distribution is on multiple memory node, the calculating of hyperchannel seasonal effect in time series dynamic interrelationships completed in the reduce stage, result of calculation outputs in distributed file system and preserves, utilize the temporal associativity of data, acquisition time data are calculated hash memory location as key word, and described characteristic extraction procedure comprises further:

1) the calculation task time, data are filtered, remove the data not meeting time conditions; 2) using time data as major key, every bar record is marked; 3) according to major key by the record cutting of same alike result value to one group, and call multivariate sample entropy computation process, result of calculation outputted to distributed file system.