CN105405070A - A method for constructing a distributed memory grid system - Google Patents

Abstract

The invention discloses a method for constructing a distributed memory grid system, which specifically includes the following steps: (1) Building a memory grid resource model: the memory grid resource modeling follows the public information model specification in the IEC61970 standard, and uses object-oriented packaging , inheritance and object application, establish a tree-type memory grid resource model; (2) Determine the memory storage structure: use an array to store all attribute data of the resource, and determine the attribute data by defining the mapping relationship between the attribute name identifier and the array subscript Position in the array to realize data access; (3) horizontally split grid resource data; (4) load and cache grid resource data; (5) synchronize memory grid data; (6) determine memory grid section; (7) Determine the memory grid access interface. The present invention can minimize data transmission between distributed nodes when realizing resource access and calculation, and greatly improve efficiency.

Description

A method for constructing a distributed memory grid system

technical field

The invention relates to a method for constructing a distributed memory power grid system, belonging to the technical field of power grids.

Background technique

In order to solve the storage, calculation and analysis problems of big data, Google proposed a distributed file system, a column-oriented distributed database, and a MapReduce distributed programming model. Google's distributed storage and distributed computing technologies provide solutions for the storage and analysis of large amounts of data in large-scale Internet site systems, and improve performance. In terms of big data real-time analysis, SAP has launched the HANA memory computing platform, which uses memory computing technology and an all-in-one computer technology combining software and hardware to achieve high-performance data query and analysis to meet users' real-time demand for big data processing. In terms of large-capacity and high-speed storage, Oracle launched the Exadata database all-in-one machine, which uses high-performance hardware, high-speed network interfaces, and technologies such as intelligent scanning, intelligent storage, intelligent indexing, and hybrid column compression to improve the performance of big data and multi-concurrency scenarios. Under the system processing efficiency. The Exadata database all-in-one machine can improve the performance of the existing system based on the oracle database by about 10 times. In addition, there are also some in-memory database technologies, such as: OracleTimesTen products that provide instant response and high throughput for applications; IBMSolidDB products that closely integrate memory- and disk-based full transaction processing database engines and highly available data replication functions ; Used for various eXtremeDBs that require high performance, small size, compact storage, and zero memory allocation; a lightweight SQLite in-memory database that occupies low resources and can be combined with many programming languages; uses high-performance key-value storage, Open-source Redis memory databases in the form of memory datasets have proposed real-time data storage solutions to avoid frequent disk I/O operations and improve data access efficiency.

In the field of distributed computing, common analysis methods use the Hadoop framework. Due to a large number of disk I/O operations and complex MapReduce processes, the system performance and operating efficiency are low, which cannot meet the real-time requirements and can only be used for non-real-time applications. Data analysis application scenarios. In the field of distributed caching, the storage method of Key-Value and the data division method based on Key hash hash are usually used. This method makes the cache nodes need to exchange a large amount of data when multiple data connections are queried, and the performance is low. It is generally used for Data with a simple structure is difficult to deal with power grid resource data with complex models. Due to their universal design, the current mainstream distributed and memory computing products cannot give full play to their advantages in the face of complex power grid resource models and massive resource data, and even have unsolvable engineering application problems.

Contents of the invention

In view of the deficiencies in the existing technology, the purpose of the present invention is to provide a method for constructing a distributed memory grid system, which can minimize data transmission between distributed nodes and greatly improve efficiency when realizing resource access and calculation.

In order to achieve the above object, the present invention is achieved through the following technical solutions:

A method for constructing a distributed memory grid system of the present invention specifically includes the following steps:

(1) Construct the memory grid resource model: the memory grid resource modeling follows the public information model specification in the IEC61970 standard, and uses object-oriented encapsulation, inheritance and object application to establish a tree-type memory grid resource model;

(2) Determining the memory storage structure: store all attribute data of the resource in the form of an array, and determine the position of the attribute data in the array by defining the mapping relationship between the attribute name identifier and the array subscript to realize data access;

(3) Split power grid resource data horizontally: map power grid resource data with different business attribute values to different data blocks, and map power grid resource data with the same business attribute value to the same data block;

(4) Loading and caching power grid resource data: When the system starts for the first time, according to the horizontal division method of data, the data in the power grid resource database is loaded according to the memory grid resource model, and then distributed and cached to the memory of multiple servers in the cluster Medium; after the data loading is complete, serialize the grid resource data in the cache to form a binary file and save it to the disk. When the system is restarted, the grid data is directly read from the file and read from the grid resource database from the last sequence After the grid resources are distributed and cached, the IP address of the cache server, the regional attribute value of the grid resource, the voltage level value of the grid resource, the occupied space, and the time used for the cache are sent to the grid resource management server for unification manage;

(5) Synchronize memory grid data: Synchronize the resource data in the grid database to the memory grid, and the resource data of the provincial memory grid to the headquarters memory grid, so that the data of the database and the memory grid, the headquarter's inland power grid and the provincial memory grid are consistent;

(6) Determine the section of the memory grid: the section of the memory grid is supported by the persistence function of the object-oriented parallel computing framework. All the section data of each province and headquarters are stored in their respective memory grid servers, and are loaded locally by sharing the memory with the memory grid server Historical section data;

(7) Determine the memory grid access interface: the memory grid interface is used to provide the grid business information system with a grid resource access mode independent of the physical location of the data, including the grid section interface, the grid data statistical analysis interface, and the data query interface.

In step (1), when performing tree display, the hierarchical relationship of grid resources in the memory grid resource model must first be determined, requiring that there must be a root node, and the parent node of each child node is unique.

In step (2), the specific method for determining the memory storage structure is as follows:

Suppose the resource data is A, which contains 4 attributes, and the number of records is 3, then an array of Object objects with a length of 4 is assigned to each record.

attribute 1 attribute 2 attribute 3 attribute 4

array 1 value 11 value 12 value 13 value 14 array 2 value 21 value 22 value 23 value 24 array 3 value 31 value 32 value 33 value 34

Use the Key-Value method to establish the mapping relationship between attribute identifiers and array subscripts as follows:

The attribute (key) corresponds to the array subscript (value)

attribute 1 1 attribute 2 2 attribute 3 3 attribute 4 4

When accessing an attribute value of a record, first find the array corresponding to the record, then find the array subscript corresponding to the attribute according to the relationship between the attribute identifier and the array subscript, and access the corresponding attribute value according to the array subscript.

In step (3), the grid resource data horizontal split method is as follows:

(3-1) Analyze the optimal N data segmentation attribute fields according to the business logic and the data used by the computing task object model;

(3-2) The types and attribute value ranges of the attributes of the N business data objects to be segmented are used as the original input for data segmentation;

(3-3) The N attributes of the object are regarded as the axes of the N-dimensional space, and are mapped to the multidimensional space region according to the object attribute value range to form a data hyperplane of the multidimensional space;

(3-4) Each data hyperplane is mapped to the memory of different computing nodes in the distributed computing cluster to form a distributed memory storage of data.

In step (6), the memory grid section uses the HDFS distributed file system for permanent and reliable storage, and the serialization of data persistence is implemented using the open source Hession component.

In step (6), the memory grid section provides section management, section generation, and multi-section loading; the section management provides section data query, section generation planning task formulation, and section loading instruction issuance; the section According to the scheduled task, the data of the memory grid is periodically persisted to generate a grid data section; the multi-section loading loads multiple historical sections into the memory according to the section instruction; the memory grid provides a section management tool for querying all historical sections At the same time, it can initiate a section generation request and a section loading request to the data section processor; after receiving the relevant request, the data section processor executes the section generation and section loading tasks.

In step (7), the memory grid access interface adopts the PRC remote call protocol, and the transmitted data is a binary stream after resource object serialization.

In step (7), query the grid section information through the grid section interface, and issue a section loading instruction, and load the specified grid section data into the computer memory; obtain the headquarters and provincial grid resource multi-conditions, Multi-dimensional statistical analysis results; through the data query interface, the business system is provided with the query of the power grid resources of the whole network, and the business system does not need to pay attention to the source and physical location of the data.

The distributed memory grid of the present invention fully considers grid resource data and grid business characteristics, adopts a unified grid resource model and a new storage method, and enables the model to accurately reflect the situation of grid resources, and is reusable, easy to understand, Features such as efficient resource storage and access; at the same time, by utilizing the regional characteristics and voltage level characteristics of power grid resources, the problem of data horizontal segmentation can be well solved, so that resource data access and calculation can minimize the data transmission between distributed nodes , greatly improve efficiency, and at the same time support the associated query of multiple resource data; the distributed memory grid is the product of the integration of distributed technology, memory computing technology and power grid business, and is a distributed memory computing product customized for the grid. Effectively support the fast and efficient processing of power grid resources by the power grid business information system.

Description of drawings

Fig. 1 is a working flow diagram of a method for constructing a distributed memory grid system;

Figure 2 is a schematic diagram of horizontal splitting of power grid resource data;

Figure 3 is a schematic diagram of the memory grid data synchronization method;

Figure 4 is an overall architecture diagram of the data section;

Figure 5 is a diagram of the architecture of the distributed memory grid system;

Figure 6 is a functional architecture diagram of the distributed memory grid.

detailed description

In order to make the technical means, creative features, goals and effects achieved by the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments.

Referring to Fig. 1, the construction method of the distributed memory grid system of the present invention is as follows:

Step 1. Construction of grid resource model

The memory grid resource modeling follows the Common Information Model (CIM) specification in the IEC61970 standard, and uses object-oriented encapsulation, inheritance, object application, etc. to establish a tree-type memory grid model. When performing tree display, it is first necessary to determine the hierarchical relationship of the power grid resources in the model. It is required that there must be a root node, and the parent node of each child node is unique. The power system resources in the power grid have their own characteristics, such as certain inclusiveness in physical characteristics, such as a sub-control area contains multiple substations, a substation contains multiple voltage levels, and a voltage level contains busbars , Switches, switches, loads and other electrical equipment. In order to meet the requirements of inclusiveness, in object-oriented modeling, add parent node and byte node attributes for each class with parent nodes and child nodes, and a parent node attribute is set to the object pointer of the parent class of the power grid resource to which it belongs , and the properties of the sub-nodes are set to the list of grid equipment subclasses it contains.

Step 2. Memory storage structure design

The memory storage structure is designed from two aspects of ease of use and saving memory overhead. In terms of saving memory overhead, general methods such as Key-Value key-value pairs or hash tables are discarded, and arrays are used to store all attribute data of resources. By defining the mapping relationship between attribute name identifiers and array subscripts, data access is realized. The implementation table is as follows:

Suppose some kind of resource data A contains 4 attributes, and the number of records is: 3, then allocate an Object object array with a length of 4 for each record.

attribute 1 attribute 2 attribute 3 attribute 4

array 1 value 11 value 12 value 13 value 14 array 2 value 21 value 22 value 23 value 24 array 3 value 31 value 32 value 33 value 34

Use the Key-Value method to establish the mapping relationship between attribute identifiers and array subscripts as follows:

The attribute (key) corresponds to the array subscript (value)

attribute 1 1 attribute 2 2 attribute 3 3 attribute 4 4

When accessing an attribute value of a record, first find the array corresponding to the record, then find the array subscript corresponding to the attribute according to the relationship between the attribute identifier and the array subscript, and access the corresponding attribute value according to the array subscript.

Step 3. Horizontal splitting of power grid resource data

According to the business logic features contained in the business data object itself, multi-dimensional segmentation is performed according to the attributes of the business data object. See Figure 2 for the schematic diagram of its implementation.

(1) According to the business logic and the data used by the calculation task object model, analyze the optimal N data segmentation attribute fields.

(2) The types and attribute value ranges of the attributes of the N business data objects to be segmented are used as the original input for data segmentation.

(3) Treat the N attributes of the object as the axes of the N-dimensional space, and map to the multi-dimensional space area according to the value range of the object attributes. A data hyperplane that forms a multidimensional space.

(4) Each data hyperplane is mapped to the memory of different computing nodes in the distributed computing cluster to form a distributed memory storage of data.

Step 4. Grid resource data loading and caching

Load grid resource data. When the system starts for the first time, according to the horizontal division method of data, the data in the grid resource database is loaded according to the grid resource memory model, and then distributed and cached in the memory of multiple servers in the cluster. After the data loading is complete, serialize the grid resource data in the cache to form a binary file and save it to the disk. When the system is restarted, the grid data is directly read from the file and read from the grid resource database from the last serialization to The current incremental data.

After the grid resources are distributed and cached, information such as the IP address of the cache server, the regional attribute value of the grid resource, the voltage level value of the grid resource, the occupied space, and the time used for caching are sent to the grid resource management server for unified management.

Step 5. Design of memory grid data synchronization method

Referring to Figure 3, data synchronization is used to synchronize the resource data in the power grid database to the memory grid, and the resource data of the provincial memory grid to the headquarters memory grid, so that the data of the database and the memory grid, the headquarters inland grid and the provincial memory grid are consistent.

Data synchronization comparison includes full synchronization and incremental synchronization. Full synchronization Synchronizes the data of the disk relational database to the object cache of the memory grid in the form of objectification, ensuring the consistency of the object data of the memory grid and the data source data of the disk relational database; Synchronize to the State Grid headquarters and complete the splicing to ensure the synchronization of the memory grids of the headquarters and the memory grids of each province. According to the metadata of the data change record table, the network provincial memory grid completes the incremental update of the object data. After the update, the object data change file is generated at the same time, which is used to record the change information of the data model in the memory grid of the network province, and stored in the change information storage of the comparison server. State Grid Headquarters memory grid accesses the object data change files in the comparison server of each network province, uploads the correspondingly changed data object model from the network province object server to the headquarters memory grid, and completes the update of the changed data object model.

In particular, due to the limited cache of the memory grid, the full amount of data cannot be loaded into the object cache at one time for comparison. Therefore, the data comparison module provides data block comparison technology. The amount of data in some data tables is very large. If you load all of them at once for comparison, it will not only take time but also occupy the resources of the memory grid object cache. All data will first be chunked and prioritized. The division of data blocks is formulated according to business requirements. A data block can contain logically linked or business-related data from different data tables, and unrelated data in the same data table is not classified into the same data block. Each data block is then prioritized according to the real-time performance of the corresponding business and the accuracy of the data. Data blocks with higher priority are processed first during each comparison. It is even possible to increase the frequency of comparisons for particular data blocks. Secondly, create a data comparison object pool in the object server. For each comparison, first load some specified data blocks into the object cache for comparison. After completion, release the data object model in the data comparison object pool, and then start from the disk relationship. The database loads the specified data blocks that need to be compared next.

Step 6. Power Grid Section Design

The data section provides functions such as section management, section generation, and multi-section loading. Section management provides the query of section data, the formulation of section generation planning tasks, and the issuance of section loading instructions. The section generation periodically persists the data of the memory grid according to the scheduled task, and generates the grid data section. Multi-section loading According to the section command, load multiple historical sections into the memory.

The memory grid section is supported by the persistence function of the object-oriented parallel computing framework, and all the section data of each province and headquarters are stored in their respective memory grid servers. Load the local historical section data by sharing the memory with the memory grid server. The overall architecture of the data section is shown in Figure 4.

The memory grid provides a section management tool for querying all historical section information, and at the same time can initiate a section generation request and a section loading request to the data section processor. After receiving relevant requests, the data section processor executes the tasks of section generation and section loading.

In particular, the grid section uses the HDFS distributed file system for permanent and reliable storage. The serialization of data persistence is implemented using the open source Hession component. This component is characterized by high serialization efficiency and short byte streams after serialization.

Step 7. Memory grid access interface design

The memory grid interface is used to provide the grid business information system with a grid resource access mode independent of the physical location of the data, including the grid section interface, the grid data statistical analysis interface, and the data query interface. Through the grid section interface business system, the grid section information can be queried, and the section loading command can be issued to load the specified grid section data into the computer memory. Through the statistical analysis interface, the business system can obtain multi-conditional and multi-dimensional statistical analysis results of headquarters and provincial power grid resources. The data query interface is used to provide the business system with the query of the grid resources of the whole network, and the business system does not need to pay attention to the source and physical location of the data. The grid resource data access interface adopts the PRC remote call protocol, and the transmitted data is a binary stream after serialization of resource objects.

(1) Memory grid system architecture

Memory Grid is built on top of the object-oriented parallel computing framework. Data synchronous replication is performed between the headquarters memory grid and each provincial memory grid, and between the provincial memory grid and the provincial database through the data synchronization comparison module. The memory grid has its own application functions such as data section and memory grid display. The memory grid application interface provides OPC-based statistical analysis, query and other application programming interfaces for other systems such as PMS2.0, see Figure 5.

(2) Memory Grid Functional Architecture

On the basis of OPC, with the memory grid model as the core, the memory grid constructs grid resource data objects and task object pools, and builds related management and application functions on the basis of grid resource object pools, mainly including: data synchronization comparison, data section management, Refer to Figure 6 for memory grid query statistical analysis, memory grid display, and memory grid application interface.

The data synchronization comparison function is responsible for synchronizing the resource data in the grid database to the memory grid, and the resource data of the provincial memory grid to the headquarters memory grid, so that the data of the database and the memory grid, the headquarter's inland grid and the provincial memory grid are consistent. The data section function is responsible for persisting the memory power grid data at a certain moment to generate a data section, providing query statistics of section information, section task management functions, etc. The memory grid management includes the management of the grid memory model and the management of the grid resource objects. The memory grid display provides a multi-dimensional display of the memory grid, including the grid section overview display, grid section data display, and grid section comparison display sub-functions. The memory grid application interface is used to provide the grid section interface, statistical analysis interface, and data query interface to the business system.

Therefore, in the specific design, the present invention fully considers the problems of the prior art, adopts a new weak object storage mode, and carries out object model design by integrating business logic, so that the model can accurately reflect the situation of power grid resources, and at the same time has reusable, Features such as easy to understand, efficient resource storage and access. In addition, the data distribution of the distributed memory grid adopts a hash method based on business attributes, and the business attributes used for hash hashing are optimized and customized for grid data and business characteristics. It can minimize the data transmission between distributed nodes and greatly improve the efficiency.

The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the industry should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and the description only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Variations and improvements are possible, which fall within the scope of the claimed invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims (8)

1. a distributed memory network system construction method, is characterized in that, specifically comprises following step:

(1) build internal memory power network resources model: the common information model specification in IEC61970 standard is followed in the modeling of internal memory power network resources, utilize OO encapsulation, succession and object application, set up the internal memory power network resources model of tree-shaped;

(2) determine memory structure: adopt array mode to store resource all properties data, by the mapping relations of defined attribute name identifiers and array index, determine the position of attribute data in array, realize data access;

(3) level splits power network resources data: will have the power network resources data-mapping of different business property value to different data blocks, by the power network resources data-mapping with identical services property value to together data block;

(4) also buffer memory power network resources data are loaded: when system first time starts, according to the horizontal division mode of data, data in power network resources database are loaded according to internal memory power network resources model, then in distributed caching to cluster in multiple server memory; After Data import completes, power network resources data in buffer memory are carried out serializing, forms binary file and be saved in disk, when start up system again, directly read electric network data from file, and read from serializing last time incremental data up till now from power network resources database; After power network resources distributed caching, by the area attribute value of cache server IP address, power network resources, power network resources electric pressure value, take up room, buffer memory temporal information used is sent in managing power network resources server and carries out unified management;

(5) isochronous memory electric network data: the resource data in electric network database is synchronized to internal memory electrical network, economizes the resource data of internal memory electrical network and be synchronized to general headquarters' internal memory electrical network, makes the data of ground electrical network and province's internal memory electrical network in database and internal memory electrical network, general headquarters be consistent;

(6) internal memory electrical network section is determined: internal memory electrical network section is using objectification parallel computation frame persistence function as support, all profile datas of each province and general headquarters are kept at respective internal memory grid service device, adopt the mode with internal memory grid service device shared drive, load local Historic Section data;

(7) internal memory electrical network access interface is determined: internal memory grid interface is used for the power network resources access mode providing Data Physical position irrelevant to electrical network operating information system, comprises electrical network section interface, electric network data statistical study interface, data-query interfaces.

2. distributed memory network system construction method according to claim 1, it is characterized in that, in step (1), when carrying out tree-shaped displaying, first the hierarchical relationship of power network resources in internal memory power network resources model will be determined, requirement must have a root node, and the father node of each child node is unique.

3. distributed memory network system construction method according to claim 1, is characterized in that, in step (2), the defining method that memory structure is concrete is as follows:

If resource data is A, comprise 4 attributes, record number is: 3, then distribute for every bar record the Object object array that a length is 4,

The mapping relations that employing Key-Value mode sets up attribute-identifier and array index are as follows:

When accessing certain property value of certain record, first find the array that this record is corresponding, then according to attribute-identifier and array index relation, find the array index that this attribute is corresponding, the property value corresponding according to array index access.

4. distributed memory network system construction method according to claim 1, is characterized in that, in step (3), power network resources data level method for splitting is as follows:

(3-1) by calculation task object model according to service logic and the data that use, analyze optimum N number of data cutting attribute field;

(3-2) by the type of N number of service data object attribute to be slit and attribute-value ranges, as the original input of data cutting;

(3-3) N number of attribute of object is regarded as the axle of N dimension space, reflect hyperspace region according to object attribute values scope, form the data lineoid of hyperspace;

(3-4) each data lineoid is mapped to the internal memory of the different computing nodes in Distributed Calculation cluster, and the distributed memory forming data stores.

5. distributed memory network system construction method according to claim 1, it is characterized in that, in step (6), internal memory electrical network section utilizes that HDFS distributed file system is carried out forever, reliable memory, and serializing during data persistence adopts the Hession assembly of increasing income to realize.

6. distributed memory network system construction method according to claim 1, is characterized in that, in step (6), described internal memory electrical network section provides section management, section generates and multibreak loading; The formulation that described section management provides the inquiry of profile data, section generates plan target, section are loaded into assigning of instruction; Described section generates and regularly the data of internal memory electrical network is carried out persistence according to plan target, generates electric network data section; Described multibreak is loaded into according to section instruction, is loaded into multiple Historic Section to internal memory; Internal memory electrical network provides section management tool for inquiring about all Historic Section information, can initiate section simultaneously generate request and section load request to data section processor; Data section processor, after receiving association requests, performs section generation, section loading tasks.

7. distributed memory network system construction method according to claim 1, is characterized in that, in step (7), described internal memory electrical network access interface adopts PRC far call agreement, and the data of transmission are the binary stream after resource object serializing.

8. distributed memory network system construction method according to claim 1, it is characterized in that, in step (7), by described electrical network section interface polls electrical network section information, and assign section load instructions, the electrical network profile data that loading is specified is to calculator memory; Obtain general headquarters by described electric network data statistical study interface and economize the statistic analysis result of power network resources many condition, various dimensions; Thered is provided the inquiry of the whole network power network resources to operation system by described data-query interfaces, operation system does not need source and the physical location of focused data.