[go: up one dir, main page]

CN113157808B - Configuration file synchronization system and method based on memory and lightweight database - Google Patents

Configuration file synchronization system and method based on memory and lightweight database Download PDF

Info

Publication number
CN113157808B
CN113157808B CN202110460354.4A CN202110460354A CN113157808B CN 113157808 B CN113157808 B CN 113157808B CN 202110460354 A CN202110460354 A CN 202110460354A CN 113157808 B CN113157808 B CN 113157808B
Authority
CN
China
Prior art keywords
data
node
directory
file
cluster
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110460354.4A
Other languages
Chinese (zh)
Other versions
CN113157808A (en
Inventor
殷双飞
刘彬彬
马连志
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Institute of Computer Technology and Applications
Original Assignee
Beijing Institute of Computer Technology and Applications
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Institute of Computer Technology and Applications filed Critical Beijing Institute of Computer Technology and Applications
Priority to CN202110460354.4A priority Critical patent/CN113157808B/en
Publication of CN113157808A publication Critical patent/CN113157808A/en
Application granted granted Critical
Publication of CN113157808B publication Critical patent/CN113157808B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2228Indexing structures
    • G06F16/2255Hash tables

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to a configuration file synchronization system and method based on a memory and a lightweight database, and relates to the technical field of data storage. The invention provides a light and rapid configuration file synchronization system and a method for a multi-node cluster or a distributed cluster, which can avoid the single-node fault of a management service and improve the management performance and reliability of a system through the reasonable design of a memory database, a back-end database and a management module; the node adding steps are simplified, and the node adding difficulty is reduced.

Description

Configuration file synchronization system and method based on memory and lightweight database
Technical Field
The invention relates to the technical field of data storage, in particular to a configuration file synchronization system and method based on a memory and a lightweight database.
Background
In the cluster management based on local storage, the control of all nodes of a cluster is generally centralized and unified management through a single node, and configuration data of the cluster is centrally stored on a management node. In cluster management based on distributed storage, configuration files can be stored in a distributed system, but for an uninitialized distributed storage system, the configuration files cannot be stored or shared in the cluster, and moreover, the configuration files are stored based on the distributed system, the magnitude of the storage back end is too heavy, the configuration is complex, for a new node to be added into the cluster, the whole distributed storage system must be installed in advance, and the node can be added into the cluster after the access authorization of the distributed storage system, so that the adding time of the node is obviously prolonged, and the adding difficulty of the node is increased.
Disclosure of Invention
Technical problem to be solved
The technical problem to be solved by the invention is as follows: how to solve the single node fault based on the centralized management of the local storage and reduce the node adding difficulty caused by the configuration synchronization based on the distributed storage.
(II) technical scheme
In order to solve the technical problem, the invention provides a configuration file synchronization system based on a memory and a lightweight database, which is arranged on each cluster node and comprises a memory database, a back-end database and a management module; the memory database is used as a temporary storage position of front-end data and is used for providing data response of an application program; the back-end database is used for providing persistent data storage capacity; the management module is used for providing a data reading and writing function and a node communication function so as to read and write a back-end database, form a complete data copy in the memory database and ensure the consistency of the data state of the cluster directory.
Preferably, the back-end database uses one file as the database, uses the data table as the logical storage body of the file, and each row in the data table is used as one file or directory record and contains the following fields: the ID field represents the ID of the data item, the data identified by the ID has uniqueness, and the field has a self-increment attribute as a main key; the inode field represents the inode number of the file, and for each file or directory, the management module assigns an inode number to the file or directory to identify the file or directory; the name field represents the name of a file or a folder, and is specified by an application program and written by a management module; the type field represents a file type, the value of the value is 0,1 or 2, when the value is 0, the data structure of the file or the directory is just initialized, data is not stored, when the value is 1, the content of the record is a directory, and when the value is 2, the content of the record is a file; the parent field represents the parent directory of the item record, the parent value stores the inode number of the parent directory, and if the item record is located in the root directory, the parent field value is 0; the ctime field indicates the creation time of the file or directory recorded by the item; the mtime field indicates the modification time of the file or the directory recorded by the item; the version field is used for indicating the latest version number of the recorded file or directory and distinguishing whether the current data is the latest data; the data field represents a data entity under the item record, if the item record is a file, the file content is stored, and if the item record is a directory, the field storage content is empty;
the method comprises the steps that a back-end database contains a cluster global version number file in a default mode, the cluster global version number file serves as a first entry and is stored in the back-end database, a version field of the cluster global version number file is used for identifying the current version number of a cluster, and 1 is added to the version field value of the file every fixed time interval or when a cluster directory has data writing and modifying operations.
Preferably, after the management module reads the back-end database, in a data copy structure formed in the memory database, the files and the directories are organized into a tree structure, and the files or the directories can be stored at a root directory of the tree structure, namely, a mount position of a file system; the subdirectory or the file of the directory can be stored in the subdirectory; the method comprises the following steps that a global hash mapping table is contained in a root directory data structure, the global hash mapping table stores hash mapping of all files or directories of a back-end database, and the hash mapping table can search whether a current back-end database contains a currently searched item or not so as to determine whether to continuously search and position the searched item or to immediately return to an upper layer for calling; all the subdirectories except the root directory contain a local hash mapping table, the local hash mapping table stores hash mapping of all the subdirectories and files under the current directory, and the hash mapping is synchronously modified after creating or deleting entries each time.
Preferably, the nodes in the cluster have the roles of a master node and a slave node, the master node is a state consistency control node generated by cluster election, and the slave node is a general functional node; when the management module realizes communication between a master node and a slave node in a cluster, in a data writing process of the master node and the slave node, after a master node initiates a data writing request, a request message is sent to each slave node from the master node, after the slave node receives the message of the master node, a data writing operation is initiated, and after the data writing operation is completed, the data writing operation is returned, so that the data states of the slave node and the master node are kept consistent; when the slave node initiates a data write request, the data will not be written into the cluster directory immediately, but a request message is generated and sent to the master node, after the master node receives the message, the message is sent to other slave nodes except the data write node, and simultaneously a confirmation message is sent to the current slave node, at this moment, the current slave node executes the data write operation, after the master node write request returns, the current slave node completes the data write operation, thereby keeping the data states of the slave nodes consistent with the data state of the master node, and ensuring that the data states of the slave nodes are consistent with the data states of the master node
The invention also provides a configuration file synchronization method based on the memory and the lightweight database, which is realized by utilizing the system and comprises the following steps:
step 1, initializing cluster state
When a cluster is just established, the management module checks a rear-end database of the node, and if the current node has the rear-end database, records in the rear-end database are read to a memory database; if the current node has no back-end database, a new back-end database is created;
under the condition that the node has no back-end database, the management module automatically creates a new back-end database, opens the back-end database, writes a cluster global version number file into the back-end database, sets the inode field of the file to be 0 and sets the parent field to be 0;
under the condition that the node has a back-end database, the directory or the file stored in the back-end database is recombined into a tree-shaped storage structure in the memory database through each field information in the data table; the management module reads records from the back-end database line by line in sequence and creates a global hash mapping table in a root directory structure;
step 2, after the cluster state is initialized, the following steps are executed:
data writing:
when the node writes data into the cluster directory, firstly detecting the operation mode of the current node, and if the node is in a single-node mode, directly writing the data; if the cluster mode is adopted, the cluster state is further detected, if the cluster is in the normal working state, normal writing is carried out, if the cluster is in the abnormal working state, only reading is carried out, and writing is forbidden;
the process of writing data into the cluster directory in the normal working state is as follows:
(1) If the written data is a directory, generating a directory data structure, adding directory information in a global hash mapping table, adding directory information in a parent directory local hash mapping table of the directory, filling parent information, inode information, name information, ctime information, mtime information and version information of the directory, and initializing the local hash mapping table under the directory;
(2) If the written data is a file, generating a file data structure, adding file information into a global hash mapping table, adding directory information into a parent directory local hash mapping table of the file, filling parent information, inode information, name information, ctime information, mtime information and version information of the file, and copying a data entity of the file into a data variable;
(3) If the written directory or file exists, modifying mtime information, version information or data of the corresponding entry;
(4) Updating a version value of the cluster global version number file, wherein the version value is the current value plus 1;
(5) If the current node is the master node, sending the written information to each slave node; if the current node is a slave node, sending the written information to the master node;
(6) And (5) writing the written data to a back-end database.
Preferably, step 2, after the cluster state is initialized, further performing the following steps:
data reading: when the node reads data from the cluster directory, the current node can be directly read no matter the operation mode of the current node is in the single-node mode or the cluster mode, and when the application program reads the data, the management module directly returns the specified data from the memory database to the upper-layer application program.
Preferably, step 2, after the cluster state is initialized, further performing the following steps: updating the lag data:
after a node in the cluster is offline for a period of time, the back-end database or the memory database of the node is in a lagging state, and after the node re-enters the cluster, synchronous operation needs to be executed. Because the node enters a normal working state from an abnormal working state, the node takes the role of a slave node, and the process of updating the hysteresis data is as follows:
(1) The slave node sends heartbeat information to the master node and sends a data synchronization request;
(2) After receiving the data synchronization request, the master node sends a full local hash mapping table of the master node to the slave node;
(3) In the slave node, comparing the records of the master node global hash mapping table with the records of the local node global hash mapping table one by one, and comparing version number information of version fields with the same inode number; if the version number information is consistent, comparing the next piece of information; if the version number information corresponding to the current inode number is inconsistent or the global hash table of the node has no corresponding inode number record, recording the current inode number;
(4) In the slave node, comparing the residual entries of the global hash mapping table of the node with the records of the global hash mapping table of the master node one by one, and if the global hash mapping table of the master node does not contain the entries of the global hash mapping table of the node, deleting the file or the directory (and the file under the directory structure);
(5) The slave node sends a list of state-lagging inode numbers to the master node;
(6) After the master node packages the data, the master node sends an inode number list and a corresponding data entity to the slave nodes as updating data;
(7) And receiving the updated data from the nodes, synchronizing the updated data to the memory database, and simultaneously writing the updated data into the back-end database.
Preferably, the management module sequentially reads the records from the back-end database row by row, and creates the global hash mapping table in the root directory structure in the following specific manner:
(1) If the value of the parent field of the current row is 0 and the value of the type field is 2, the entry is a file under a root directory, and the file information is added into a global hash mapping table;
(2) If the value of the parent field of the current row is 0 and the value of the type field is 1, the entry is indicated to be a directory under a root directory, the information of the directory is added into a global hash mapping table, and an empty local hash mapping table is newly built under the data structure of the directory;
(3) If the value of the parent field of the current row is not 0 and the value of the type field is 2, the entry is a file in a certain subdirectory, and the information of the file is added into a global hash mapping table; checking whether a parent directory of the file exists or not, if so, adding the information of the file into a local hash mapping table of the parent directory, and if not, adding the information of the parent directory into a global hash mapping table, initializing the local hash mapping table of the parent directory, and simultaneously adding the information of the file into the local hash mapping table of the parent directory;
(4) If the value of the parent field of the current row is not 0 and the value of the type field is 1, the entry is a certain subdirectory, and the information of the subdirectory is added into a global hash mapping table; if the directory exists, adding the information of the parent directory into the global hash table, initializing the local hash mapping table of the parent directory, adding the information of the directory into the local hash mapping table of the parent directory, and initializing the local hash mapping table of the parent directory;
(5) If the current directory exists in the global hash table, updating the current entry, wherein the updated content comprises an inode number, a parent number, a name, a ctime, an mtime and a version, and the version value is the version value of the cluster global version number file plus 1.
Preferably, in the method, the cluster node may operate in a single-node mode or in a cluster mode; when a node does not join a cluster, the node is in a single-node mode, the node uses a back-end database to provide persistent data storage capacity in the single-node mode, data in the back-end database is copied to a memory database of the node to provide data access service, after the node joins the cluster, the node is in a cluster mode, in the cluster mode, the data stored in the back-end database of all the nodes are consistent in state, the data copied state in the memory database is consistent, and configuration files stored in a cluster directory are in a synchronous state;
preferably, in the method, when an application program writes a configuration file, a write operation of a user mode file system is triggered, a request is sent to a memory database through a management module to complete temporary storage and is sent to a back-end database to complete permanent storage, in a cluster mode, a written data request is sent to a local node and is synchronously sent to other nodes to form a consistent data storage state, and after synchronization, each node has a complete data copy.
(III) advantageous effects
The invention provides a light and rapid configuration file synchronization system and method for a multi-node cluster or a distributed cluster, which can avoid the single-node fault of a management service and improve the management performance and reliability of a system through the reasonable design of a memory database, a back-end database and a management module; the node adding steps are simplified, and the node adding difficulty is reduced.
Drawings
FIG. 1 is a block diagram of a configuration synchronization design based on a lightweight database according to the present invention;
FIG. 2 is a schematic diagram of field design of the back-end database in the present invention;
FIG. 3 is a schematic diagram of the structural design of the front-end database of the present invention;
fig. 4 is a communication flow diagram of a cluster master-slave node in the present invention.
Detailed Description
In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.
In order to solve the single node fault occurring on the basis of the centralized management of the local storage and reduce the node adding difficulty caused by the configuration synchronization based on the distributed storage, the invention provides a configuration file synchronization system and a method based on a memory and a lightweight database, so as to avoid the single node fault of the management service and improve the management performance and the reliability of the system; the node adding steps are simplified, and the node adding difficulty is reduced.
The invention designs a configuration synchronization system and method based on a memory and a lightweight database. The design aims to provide a light and rapid configuration file synchronization system and method for a multi-node cluster or a distributed cluster, and an efficient file sharing mode is provided for cluster management. The system is arranged on each cluster node and comprises the following parts, as shown in figure 1, a memory database, a back-end database and a management module; the memory database is used as a temporary storage position of front-end data and is used for providing quick data response of an application program (cluster management software); the back-end database is used for providing persistent data storage capacity; the management module is used for providing a data reading and writing function and a node communication function so as to read and write a back-end database, form a complete data copy in the memory database and ensure the consistency of the data state of the cluster directory.
1. Storage structure design of back-end database
In the present invention, the storage of the back-end database uses the lightweight database as the storage engine, uses a file as the database, uses the data table as the logical storage body of the file, and uses each row in the data table as a file or directory record, as shown in fig. 2.
Wherein the ID field represents the ID of the data item, the data identified by the ID has uniqueness, and the field has a self-increment attribute as a primary key; the inode field represents the inode number of the file, and for each file or directory, the management module assigns an inode number to the file or directory to identify the file or directory; the name field represents the name of a file or a folder, and is specified by an application program and written by a management module; the type field indicates the file type, the value of the value is 0,1 or 2, when the value is 0, the data structure of the file or the directory is just initialized, data is not stored, when the value is 1, the content of the record is indicated as the directory, and when the value is 2, the content of the record is indicated as the file; the parent field represents the parent directory of the item record, the value of the parent stores the inode number of the parent directory, and if the item record is located in the root directory, the value of the parent field is 0; the ctime field indicates the creation time of the file or directory recorded by the item; the mtime field indicates the modification time of the file or the directory recorded by the item; the version field is used for indicating the latest version number of the recorded file or directory and distinguishing whether the current data is the latest data; the data field represents the data entity under the entry record, and if the entry record is a file, the file content is stored, and if the entry record is a directory, the field storage content is empty.
The method comprises the steps that a back-end database contains a cluster global version number file as a default, the file is stored in the back-end database as a first entry, a version field of the file is used for identifying the current version number of a cluster, and 1 is added to the version field value of the file every fixed time interval or when a cluster directory has data writing and modifying operations.
2. Storage structure design of memory database
In the present invention, after the management module reads the back-end database, a complete data copy is formed in the memory database, and the files and directories are organized into a tree structure, as shown in fig. 3.
Files or directories can be stored at the root directory of the tree structure, namely the mounting position of the file system; the subdirectory or the file of the directory can be stored in the subdirectory; the root directory data structure comprises a global hash mapping table, the global hash mapping table stores hash mapping of all files or directories of the back-end database, and the hash mapping table can quickly search whether a current back-end database contains a currently searched item or not so as to determine whether to continuously search and locate the searched item or to immediately return to an upper layer for calling; all the subdirectories except the root directory contain a local hash mapping table, the local hash mapping table stores hash mapping of all the subdirectories and files under the current directory, and the hash mapping is synchronously modified after creating or deleting entries each time.
3. Communication design of master-slave nodes realized by management module
The nodes in the cluster have the roles of a master node and a slave node, the master node is a state consistency control node generated by cluster election, and the slave node is a general functional node; in the cluster, the data writing flow of the master node and the slave node is shown in fig. 4, when the master node initiates a data writing request, a request message is sent from the master node to each slave node, after receiving the message of the master node, the slave node initiates a data writing operation, and after writing, the operation is returned, so that the data states of the slave node and the master node are kept consistent (see the solid line part in fig. 4); when a slave node initiates a data write request, data is not written into a cluster directory immediately, but a request message is generated and sent to a master node (see a dotted line part (1) in fig. 4), after the master node receives the message, the message is sent to other slave nodes except the data write node (see a dotted line part (2) in fig. 4), and meanwhile, an acknowledgement message is sent to the current slave node (see a dotted line part (3) in fig. 4), at this moment, the current slave node executes a data write operation, after the master node write request returns, the current slave node completes the data write operation, and therefore the data states of the slave nodes are kept consistent with the data state of the master node.
In the invention, an application program is accessed to a configuration file from a user mode file system, the user mode file system is used for providing a data access interface for cluster management software, a back-end database is used for providing persistent data storage capacity, a memory database is used for providing fast and efficient data access capacity, and a management module is used for providing high-availability data access capacity.
In the invention, the cluster node can operate in a single-node mode and can also operate in a cluster mode; when the node does not join the cluster, the node is in a single-node mode, the node uses the back-end database to provide persistent data storage capacity in the single-node mode, and data in the back-end database is copied to the memory database of the node, so that fast and efficient data access service is provided. When the nodes are added into the cluster, the nodes are in a cluster mode, in the cluster mode, the data states stored in the back-end data base of all the nodes are consistent, the data copying states in the memory database are consistent, and the configuration files stored in the cluster directory are in a synchronous state.
When the application program writes in the configuration file, the write-in operation of the user mode file system is triggered, the request is sent to the memory database through the management module to complete the rapid temporary storage, and meanwhile, the request is sent to the back-end database to complete the permanent storage. In the cluster mode, the written data request is synchronously sent to other nodes while being sent to the node, a consistent data storage state is formed, and after synchronization, each node has a complete data copy.
The configuration file synchronization method implemented based on the configuration file synchronization system based on the memory and the lightweight database comprises the following steps of:
step 1, initializing cluster state
When a cluster is just established, the management module checks a rear-end database of the node, and if the current node has the rear-end database, records in the rear-end database are read to a memory database; and if the current node has no back-end database, creating a new back-end database.
And under the condition that the node has no back-end database, the management module automatically creates a new back-end database, opens the back-end database, writes a cluster global version number file into the back-end database, sets the inode field of the file to be 0 and sets the parent field to be 0.
Under the condition that the node has a back-end database, the directory or the file stored in the back-end database is recombined into a tree-shaped storage structure in the memory database through each field information in the data table; the management module reads records from the back-end database line by line in sequence and creates a global hash mapping table in the root directory structure, specifically as follows:
(1) If the value of the parent field of the current row is 0 and the value of the type field is 2, the entry is a file under a root directory, and the file information is added into a global hash mapping table;
(2) If the value of the parent field of the current row is 0 and the value of the type field is 1, the entry is indicated to be a directory under a root directory, the information of the directory is added into a global hash mapping table, and an empty local hash mapping table is newly built under the data structure of the directory;
(3) If the value of the parent field of the current row is not 0 and the value of the type field is 2, the entry is a file in a certain subdirectory, and the information of the file is added into a global hash mapping table; checking whether a parent directory of the file exists or not, if so, adding the information of the file into a local hash mapping table of the parent directory, and if not, adding the information of the parent directory into a global hash mapping table, initializing the local hash mapping table of the parent directory, and simultaneously adding the information of the file into the local hash mapping table of the parent directory;
(4) If the value of the parent field of the current row is not 0 and the value of the type field is 1, the entry is a certain subdirectory, and the information of the subdirectory is added into a global hash mapping table; if not, adding the information of the father directory in the global hash table, initializing the local hash mapping table of the father directory, and simultaneously adding the information of the directory into the local hash mapping table of the father directory, and initializing the local hash mapping table of the directory;
(5) And if the current directory exists in the global hash table, updating the current entry, wherein the updated content comprises an inode number, a parent number, a name, a ctime, an mtime and a version, and the version value is the sum of the version value of the cluster global version number file and 1.
And 2, executing the following steps after the cluster state is initialized (the following steps are not in sequence and can be executed in parallel):
data writing:
when the node writes data into the cluster directory, firstly detecting the operation mode of the current node, and if the node is in a single-node mode, directly writing the data; if the cluster mode is adopted, the cluster state is further detected, if the cluster is in the normal working state, normal writing is carried out, if the cluster is in the abnormal working state, only reading is carried out, and writing is forbidden;
the process of writing data into the cluster directory in the normal working state is as follows:
(7) If the written data is a directory, generating a directory data structure, adding directory information in a global hash mapping table, adding directory information in a parent directory local hash mapping table of the directory, filling parent information and inode information of the directory (the inode value is the version value of a cluster global version number file plus 1), name information, ctime information, mtime information and version information (the version value is equal to the inode), and initializing the local hash mapping table under the directory;
(8) If the written data is a file, generating a file data structure, adding file information into a global hash mapping table, adding directory information into a parent directory local hash mapping table of the file, filling parent information and inode information of the file (the inode value is the version value of a cluster global version number file plus 1), name information, ctime information, mtime information and version information (the version value is equal to the inode), and copying a data entity of the file into a data variable;
(9) If the written directory or file exists, modifying mtime information, version information (the version value is the version value of the cluster global version number file plus 1) or data of the corresponding entry;
(10) Updating a version value of the cluster global version number file, wherein the version value is the current value plus 1; (ii) a
(11) If the current node is the master node, sending the written information to each slave node; if the current node is a slave node, sending the written information to the master node;
(12) And (5) writing the written data to a back-end database.
Data reading:
when the node reads data from the cluster directory, the data can be directly read no matter the operation mode of the current node is in the single-node mode or the cluster mode. In order to improve the access rate of the data, when the application program reads the data, the management module directly returns the specified data from the memory database to the upper application program.
Updating the lag data:
after a node in the cluster is offline for a period of time, the back-end database or the memory database of the node is in a lagging state, and after the node re-enters the cluster, synchronous operation needs to be executed. Because the node enters a normal working state from an abnormal working state (an offline state), the node plays a role of a slave node, and the process of updating the hysteresis data is as follows:
(8) The slave node sends heartbeat information to the master node and sends a data synchronization request;
(9) After receiving the data synchronization request, the master node sends a full local hash mapping table of the master node to the slave node;
(10) In the slave node, the records of the master node global hash mapping table and the local node global hash mapping table are compared one by one, and the comparison content is version number information of a version field corresponding to the same inode number; if the version number information is consistent, comparing the next piece of information; if the version number information corresponding to the current inode number is inconsistent or the global hash table of the node has no corresponding inode number record, recording the current inode number;
(11) In the slave node, comparing the remaining entries of the global hash mapping table of the slave node with the records of the global hash mapping table of the master node one by one, and if the global hash mapping table of the master node does not contain the entries of the global hash mapping table of the slave node, deleting the file or the directory (and the file under the directory structure);
(12) The slave node sends a list of state-lagging inode numbers to the master node;
(13) After the master node packages the data, the master node sends an inode number list and a corresponding data entity to the slave nodes as updating data;
(14) And receiving the updated data from the nodes, synchronizing the updated data to the memory database, and simultaneously writing the updated data into the back-end database.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.

Claims (9)

1. A configuration file synchronization system based on a memory and a lightweight database is characterized in that the system is arranged on each cluster node and comprises a memory database, a back-end database and a management module; the memory database is used as a temporary storage position of front-end data and is used for providing data response of an application program; the back-end database is used for providing persistent data storage capacity; the management module is used for providing a data reading and writing function and a node communication function so as to read and write a back-end database, form a complete data copy in the memory database and ensure the consistency of the data state of the cluster directory;
the back-end database takes a file as a database, uses a data table as a logical storage body of the file, and each row in the data table is taken as a file or directory record and comprises the following fields: the ID field represents the ID of the data item, the data identified by using the ID has uniqueness, and the field has an autonomy attribute as a main key; the inode field represents the inode number of the file, and for each file or directory, the management module allocates one inode number to the file or directory for identifying the file or directory; the name field represents the name of a file or a folder, and is specified by an application program and written by a management module; the type field represents the file type, the value of the type field is 0,1 or 2, when the value is 0, the data structure of the file or the directory is just initialized, data is not stored, when the value is 1, the content of the record is represented as the directory, and when the value is 2, the content of the record is represented as the file; the parent field represents the parent directory of the item record, the value of the parent stores the inode number of the parent directory, and if the item record is positioned in the root directory, the value of the parent field is 0; the ctime field represents the creation time of the file or the directory recorded by the item; the mtime field indicates the modification time of the file or the directory recorded by the item; the version field is used for indicating the latest version number of the file or the directory recorded by the item and is used for distinguishing whether the current data is the latest data; the data field represents a data entity recorded by the item, if the item is recorded as a file, the content of the file is stored, and if the item is recorded as a directory, the content of the field is empty;
the method comprises the steps that a back-end database contains a cluster global version number file as a default, the cluster global version number file serves as a first entry and is stored in the back-end database, a version field of the cluster global version number file is used for identifying the current version number of a cluster, and 1 is added to the version field value of the file every fixed time interval or a cluster directory has data writing and modifying operations.
2. The system of claim 1, wherein after the management module reads the back-end database, in the structure of the data copy formed in the memory database, the files and directories are organized as a tree structure, and the files or directories can be stored at the root directory of the tree structure, i.e. the mount location of the file system; the subdirectory or the file of the directory can be stored in the subdirectory; the method comprises the following steps that a global hash mapping table is contained in a root directory data structure, the global hash mapping table stores hash mappings of all files or directories of a back-end database, and the hash mapping table can search whether a current back-end database contains a currently searched item or not so as to determine whether to continuously search and locate the searched item or to immediately return to an upper layer for calling; all the subdirectories except the root directory contain a local hash mapping table, the local hash mapping table stores hash mappings of all the subdirectories and files under the current directory, and the hash mappings are synchronously modified after creating or deleting entries each time.
3. The system of claim 2, wherein the nodes within the cluster have a role of master node and slave node, the master node being a state consistency control node resulting from cluster election, the slave node being a general functional node; when the management module realizes the communication between the master node and the slave node in the cluster, in the data writing process of the master node and the slave node, after the master node initiates a data writing request, a request message is sent to each slave node from the master node, after the slave node receives the message of the master node, the data writing operation is initiated, and after the data writing operation is finished, the data writing operation is returned, so that the data states of the slave node and the master node are kept consistent; when the slave node initiates a data write request, the data is not written into the cluster directory immediately, but a request message is generated and sent to the master node, after the master node receives the message, the message is sent to other slave nodes except the data write node, and meanwhile, a confirmation message is sent to the current slave node, at the moment, the current slave node executes data write operation, and after the master node returns the write request, the current slave node completes the data write operation, so that the data states of the slave nodes are kept consistent with the data state of the master node.
4. A method for synchronizing profiles based on internal memory and lightweight database, implemented by using the system of claim 2 or 3, comprising the following steps:
step 1, initializing cluster state
When a cluster is just established, the management module checks a rear-end database of the node, and if the current node has the rear-end database, records in the rear-end database are read to a memory database; if the current node has no back-end database, a new back-end database is created;
under the condition that the node has no back-end database, the management module automatically creates a new back-end database, opens the back-end database, writes a cluster global version number file into the back-end database, sets the inode field of the file to be 0 and sets the parent field to be 0;
under the condition that the node has a back-end database, the directory or the file stored in the back-end database is recombined into a tree-shaped storage structure in the memory database through each field information in the data table; the management module reads records from the back-end database line by line in sequence and creates a global hash mapping table in a root directory structure;
step 2, after the cluster state is initialized, the following steps are executed:
data writing:
when the node writes data into the cluster directory, firstly detecting the operation mode of the current node, and if the node is in a single-node mode, directly writing the data; if the cluster mode is adopted, the cluster state is further detected, if the cluster is in the normal working state, the cluster is normally written, if the cluster is in the abnormal working state, only reading is carried out, and the writing is forbidden;
the process of writing data into the cluster directory in the normal working state is as follows:
(1) If the written data is a directory, generating a directory data structure, adding directory information in the global hash mapping table, adding directory information in the parent directory local hash mapping table of the directory, filling parent information, inode information, name information, ctime information, mtime information and version information of the directory, and initializing the local hash mapping table under the directory;
(2) If the written data is a file, generating a file data structure, adding file information into a global hash mapping table, adding directory information into a parent directory local hash mapping table of the file, filling parent information, inode information, name information, ctime information, mtime information and version information of the file, and copying a data entity of the file into a data variable;
(3) If the written directory or file exists, modifying mtime information, version information or data of the corresponding entry;
(4) Updating a version value of the cluster global version number file, wherein the version value is the current value plus 1;
(5) If the current node is the master node, sending the written information to each slave node; if the current node is a slave node, sending the written information to the master node;
(6) And (5) writing the written data to a back-end database.
5. The method of claim 4, wherein step 2, after initialization of cluster state, further performs the following steps:
data reading: when the node reads data from the cluster directory, the data can be directly read no matter the operation mode of the current node is in a single-node mode or a cluster mode, and when the application program reads the data, the management module directly returns the specified data from the memory database to the upper-layer application program.
6. The method of claim 4, wherein step 2, after the cluster state is initialized, further performs the following steps: updating the lag data:
after a node in a cluster is offline for a period of time, a back-end database or a memory database of the node is in a lag state, and after the node re-enters the cluster, synchronous operation needs to be executed, because the node enters a normal working state from an abnormal working state, the node takes a role as a slave node, and the process of updating lag data is as follows:
(1) The slave node sends heartbeat information to the master node and sends a data synchronization request;
(2) After receiving the data synchronization request, the master node sends a global hash mapping table of the master node to the slave node;
(3) In the slave node, the records of the master node global hash mapping table and the local node global hash mapping table are compared one by one, and the comparison content is version number information of a version field corresponding to the same inode number; if the version number information is consistent, comparing the next piece of information; if the version number information corresponding to the current inode number is inconsistent or the global hash table of the node has no corresponding inode number record, recording the current inode number;
(4) In the slave node, comparing the residual entries of the global hash mapping table of the slave node with the records of the global hash mapping table of the master node one by one, and deleting the file or the directory if the global hash mapping table of the master node does not contain the entries of the global hash mapping table of the slave node;
(5) The slave node sends a list of state-lagging inode numbers to the master node;
(6) After the master node packages the data, the master node sends an inode number list and a corresponding data entity to the slave node as updating data;
(7) And receiving the updated data from the nodes, synchronizing the updated data to the memory database, and simultaneously writing the updated data into the back-end database.
7. The method of claim 4, wherein the management module sequentially reads records row-by-row from the back-end database and creates the global hash map in the root directory structure as follows:
(1) If the value of the parent field of the current row is 0 and the value of the type field is 2, the item is a file under a root directory, and the file information is added into a global hash mapping table;
(2) If the value of the parent field of the current row is 0 and the value of the type field is 1, the item is a directory under a root directory, the information of the directory is added into a global hash mapping table, and an empty local hash mapping table is newly built under the data structure of the directory;
(3) If the field value of the item parent of the current line is not 0 and the field value of the type is 2, the item is a file under a certain subdirectory, and the information of the file is added into a global hash mapping table; checking whether a parent directory of the file exists or not, if so, adding the information of the file into a local hash mapping table of the parent directory, if not, adding the information of the parent directory into a global hash mapping table, initializing the local hash mapping table of the parent directory, and simultaneously adding the information of the file into the local hash mapping table of the parent directory;
(4) If the value of the parent field of the current row is not 0 and the value of the type field is 1, the entry is a certain subdirectory, and the information of the subdirectory is added into a global hash mapping table; checking whether a parent directory of the directory exists or not, if so, adding the information of the directory into a local hash mapping table of the parent directory, creating a local hash mapping table of the directory, if not, adding the information of the parent directory into a global hash table, initializing the local hash mapping table of the parent directory, simultaneously adding the information of the directory into the local hash mapping table of the parent directory, and initializing the local hash mapping table of the directory;
(5) And if the current directory exists in the global hash table, updating the current entry, wherein the updated content comprises an inode number, a parent number, a name, a ctime, an mtime and a version, and the version value is the sum of the version value of the cluster global version number file and 1.
8. The method of claim 4, wherein the cluster nodes are operable in a single node mode or in a cluster mode; when a node does not join a cluster, the node is in a single-node mode, the node uses a back-end database to provide persistent data storage capacity in the single-node mode, data in the back-end database is copied to a memory database of the node to provide data access service, after the node joins the cluster, the node is in a cluster mode, in the cluster mode, the data states stored in the back-end database of all the nodes are consistent, the data copying states in the memory database are consistent, and configuration files stored in a cluster directory are in a synchronous state.
9. The method as claimed in claim 4, wherein in the method, when the application writes in the configuration file, a write operation of the user-mode file system is triggered, the request is sent to the in-memory database through the management module to complete temporary storage and simultaneously sent to the back-end database to complete permanent storage, in the cluster mode, the written data request is sent to the node and simultaneously sent to other nodes to form a consistent data storage state, and after synchronization, each node has a complete data copy.
CN202110460354.4A 2021-04-27 2021-04-27 Configuration file synchronization system and method based on memory and lightweight database Active CN113157808B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110460354.4A CN113157808B (en) 2021-04-27 2021-04-27 Configuration file synchronization system and method based on memory and lightweight database

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110460354.4A CN113157808B (en) 2021-04-27 2021-04-27 Configuration file synchronization system and method based on memory and lightweight database

Publications (2)

Publication Number Publication Date
CN113157808A CN113157808A (en) 2021-07-23
CN113157808B true CN113157808B (en) 2023-02-03

Family

ID=76871526

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110460354.4A Active CN113157808B (en) 2021-04-27 2021-04-27 Configuration file synchronization system and method based on memory and lightweight database

Country Status (1)

Country Link
CN (1) CN113157808B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113938381A (en) * 2021-10-15 2022-01-14 牙木科技股份有限公司 Universal simple network management and network element configuration synchronization method and system
CN113886515B (en) * 2021-10-22 2025-07-11 济南浪潮数据技术有限公司 A cluster node change processing method, device and equipment based on cluster version
CN114048178B (en) * 2021-11-29 2022-07-26 众和空间(北京)科技有限责任公司 A dual-mode data storage and synchronization method

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102508881A (en) * 2011-10-18 2012-06-20 国网电力科学研究院 Method for clustering multiple nodes of memory database of power information system
CN111752488A (en) * 2020-06-30 2020-10-09 重庆紫光华山智安科技有限公司 Management method and device of storage cluster, management node and storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8229945B2 (en) * 2008-03-20 2012-07-24 Schooner Information Technology, Inc. Scalable database management software on a cluster of nodes using a shared-distributed flash memory

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102508881A (en) * 2011-10-18 2012-06-20 国网电力科学研究院 Method for clustering multiple nodes of memory database of power information system
CN111752488A (en) * 2020-06-30 2020-10-09 重庆紫光华山智安科技有限公司 Management method and device of storage cluster, management node and storage medium

Also Published As

Publication number Publication date
CN113157808A (en) 2021-07-23

Similar Documents

Publication Publication Date Title
CN113157808B (en) Configuration file synchronization system and method based on memory and lightweight database
US11704290B2 (en) Methods, devices and systems for maintaining consistency of metadata and data across data centers
US8250033B1 (en) Replication of a data set using differential snapshots
CN101329685B (en) Implementing method of memory database on household gateway
CN110209528B (en) Data backup method, device, server and storage medium
US9149054B2 (en) Prefix-based leaf node storage for database system
CA2139693C (en) Summary catalogs
EP4111324A1 (en) File systems constructed of block objects
CN114281762B (en) A log storage acceleration method, device, device and medium
US9582213B2 (en) Object store architecture for distributed data processing system
CN102012933B (en) Distributed file system and its method for storing data and providing services
US9547706B2 (en) Using colocation hints to facilitate accessing a distributed data storage system
US20160283501A1 (en) Posix-compatible file system, method of creating a file list and storage device
CN111831475A (en) Data backup method and device, node equipment and readable storage medium
CN104657459A (en) Massive data storage method based on file granularity
CN113377292B (en) A stand-alone storage engine
CN102955845A (en) Data access method and device as well as distributed database system
CN110109873A (en) A kind of file management method for message queue
CN114610680B (en) Distributed file system metadata management method, device, equipment and storage medium
CN120123305B (en) Method, product, device and storage medium for multi-host shared file system
US10055139B1 (en) Optimized layout in a two tier storage
CN114048185A (en) Method for transparently packaging, storing and accessing massive small files in distributed file system
CN104516945A (en) Hadoop distributed file system metadata storage method based on relational data base
JP5103786B2 (en) Control program, control device, and control method
Preguiça et al. Few: File management for portable devices

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant