CN119806406A - Data reading and writing method, device, electronic device and storage medium - Google Patents
Data reading and writing method, device, electronic device and storage medium Download PDFInfo
- Publication number
- CN119806406A CN119806406A CN202411865489.9A CN202411865489A CN119806406A CN 119806406 A CN119806406 A CN 119806406A CN 202411865489 A CN202411865489 A CN 202411865489A CN 119806406 A CN119806406 A CN 119806406A
- Authority
- CN
- China
- Prior art keywords
- data
- configuration data
- server side
- request
- distributed storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data reading and writing method, a device, electronic equipment and a storage medium, and relates to the technical field of data processing, wherein the method comprises the following steps: monitoring a data processing request of a server side, wherein the data processing request is used for requesting to store configuration data of the server side, summarizing the configuration data generated by the server side in response to the data processing request, writing the configuration data of the server side into a distributed storage cluster through a data processing interface which is packaged in advance by the distributed storage cluster, and synchronizing the configuration data to a back end for storage through the data processing interface in response to a configuration data reading request stored by the server side corresponding to the back end. The invention realizes the reliable storage of the global configuration data of the server side through the distributed storage cluster, prevents single-point faults, ensures the consistency of the global configuration data and further improves the data synchronization effect of the iSCSI server side.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data reading and writing method, a data reading and writing device, an electronic device, and a storage medium.
Background
ISCSI (INTERNET SMALL Computer SYSTEM INTERFACE ) combines the existing ISCSI interface with the Ethernet technology, and an ISCSI server (Target) and a client (Initiator) are connected based on a TCP/IP protocol, so that the packaged iSCSI data packet can be transmitted on the general Internet, and finally, the iSCSI server is mapped into a storage space (disk) to be provided for the client after connection authentication.
At present, configuration data in an iSCSI server (Target) is distributed and stored, and volume mapping information in the configuration data is stored in a local configuration file, so that if the configuration data needs to be synchronized, the configuration data is obtained by reading and writing the local configuration file, and each synchronization node needs to be operated once, so that the time consumption for executing a command is high, if a node fault exists, the problem of inconsistent configuration data with other nodes after the recovery is possibly caused, and the data synchronization effect of the iSCSI server is affected.
Disclosure of Invention
In view of the above, the present invention aims to provide a data reading and writing method, apparatus, electronic device and storage medium, which solve the problem of low efficiency in obtaining configuration data by reading and writing local configuration files, and implement efficient and reliable data reading and writing.
According to a first aspect of the present invention, there is provided a data read-write method applied to a distributed storage cluster, the method comprising:
Monitoring a data processing request of a server side, wherein the data processing request is used for requesting to store configuration data of the server side;
Summarizing configuration data generated by the server side in response to the data processing request;
Writing the configuration data of the server into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster;
And responding to a request of the server side for reading the configuration data corresponding to the back-end storage, and synchronizing the configuration data to the back-end storage through the data processing interface.
Optionally, the monitoring server side data processing request is used for requesting configuration data of the storage server side, and the data processing request comprises:
monitoring a data processing request sent by a server side by adopting a preset monitoring thread;
And analyzing the data processing request to determine the configuration data of the server side requesting the distributed storage cluster to store the server side.
Optionally, the responding to the data processing request gathers the configuration data generated by the server side, including:
traversing the global configuration of the server side in response to the data processing request, and determining volume mapping configuration information, path number information and authentication information generated by the server side;
and carrying out queue caching on the volume mapping configuration information, the path number information and the authentication information to obtain summarized configuration data.
Optionally, the writing the configuration data of the server side into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster includes:
The method comprises the steps of packaging preset interfaces in a distributed storage cluster into multiple types of data processing interfaces in advance, wherein the data processing interfaces comprise a data writing interface and a data modifying interface;
Responding to a write-in configuration data request of the server side, and calling a data write-in interface corresponding to the write-in configuration data request in the distributed storage cluster;
Writing configuration data of the server side into a preset storage area of the distributed storage cluster through the data writing interface;
responding to the request for modifying the configuration data of the server side, and determining the modification operation of the configuration data of the preset storage area;
calling a data modification interface corresponding to the configuration data modification request in the distributed storage cluster;
And executing the modification operation through the data modification interface, and updating the configuration information of the preset storage area.
Optionally, the data processing interface includes a data reading interface, and the responding to the request of the server side for reading the configuration data corresponding to the back-end storage, synchronizing the configuration data to the back-end storage through the data processing interface includes:
Responding to a request for reading configuration data stored in the back end corresponding to the server end, and calling a data reading interface corresponding to the request for reading configuration data in the distributed storage cluster;
and synchronizing configuration data of a preset storage area of the distributed storage cluster to the back-end storage through the data reading interface.
Optionally, before the monitoring the data processing request of the server side, the method further includes:
The method comprises the steps of pre-ending a distributed storage cluster at a server side, and creating connection between the distributed storage cluster and the server side, wherein the distributed storage cluster comprises a main node and a secondary node for reading and writing data.
Optionally, after the configuration data is synchronized to the back-end storage through the data processing interface in response to a request of the server end for reading the configuration data corresponding to the back-end storage, the method further includes:
monitoring the state of a main node of the distributed storage cluster, wherein the distributed storage cluster comprises a cluster alarm interface;
If the state of the main node is monitored to be abnormal, a target main node is selected from the auxiliary nodes, and the configuration data is backed up to the target main node;
and sending abnormal prompt information to the server side by adopting the cluster alarm interface.
According to a second aspect of the present invention, there is provided a data read-write apparatus for use in a distributed storage cluster, the apparatus comprising:
the monitoring request module is used for monitoring a data processing request of the server side, wherein the data processing request is used for requesting to store configuration data of the server side;
the data summarizing module is used for responding to the data processing request and summarizing the configuration data generated by the server side;
the data writing module is used for writing the configuration data of the server into the distributed storage cluster through a data processing interface which is packaged in advance by the distributed storage cluster;
And the data reading module is used for responding to a request of the server side for reading the configuration data stored in the back end, and synchronizing the configuration data to the back end through the data processing interface.
According to still another aspect of the present invention, there is also provided an electronic apparatus including:
A processor;
a memory for storing the processor-executable instructions;
Wherein the processor is configured to execute the instructions to implement the data read-write method as described above.
According to yet another aspect of the present invention, there is also provided a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data read-write method as described above.
According to the data reading and writing method provided by the embodiment of the invention, the configuration data generated by the server side is summarized by monitoring the data processing request of the server side, responding to the data processing request, the configuration data of the server side is written into the distributed storage cluster through the data processing interface pre-packaged by the distributed storage cluster, and the configuration data is synchronized to the back end for storage through the data processing interface in response to the configuration data reading request stored by the server side corresponding to the back end. The embodiment of the invention realizes the reliable storage of the global configuration data of the server side through the distributed storage cluster, prevents single-point faults, can adapt to the increase of the data volume of the server side, ensures the quick synchronous configuration data of the back-end storage and the server side by responding to the read-write request in time, realizes the efficient, reliable and flexible processing of the configuration data of the server side, ensures the consistency of the global configuration data and further improves the data synchronization effect of the iSCSI server side.
The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 is a flow chart of steps of a data read-write method according to an embodiment of the present invention;
FIG. 2 is a flowchart of step 101 in the data read-write method according to the embodiment of the present invention shown in FIG. 1;
FIG. 3 is a flowchart of step 102 in the data read-write method according to the embodiment of the present invention shown in FIG. 1;
FIG. 4 is a flowchart of step 103 in the data read-write method according to the embodiment of the present invention shown in FIG. 1;
FIG. 5 is a flowchart of step 104 in the data read-write method according to the embodiment of the present invention shown in FIG. 1;
FIG. 6 is a flowchart illustrating steps of another method for reading and writing data according to an embodiment of the present invention;
Fig. 7 is a schematic view of a scenario of a data read-write method according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of a data read-write device according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the following detailed description of the embodiments of the present application will be given with reference to the accompanying drawings. However, those of ordinary skill in the art will understand that in various embodiments of the present application, numerous technical details have been set forth in order to provide a better understanding of the present application. The claimed application may be practiced without these specific details and with various changes and modifications based on the following embodiments. The following embodiments are divided for convenience of description, and should not be construed as limiting the specific implementation of the present application, and the embodiments can be mutually combined and referred to without contradiction.
Referring to fig. 1, a flowchart illustrating steps of a data read-write method according to an embodiment of the present invention is applied to a distributed storage cluster, where the method may include:
Step 101, monitoring a data processing request of a server side, wherein the data processing request is used for requesting to store configuration data of the server side.
In the embodiment of the invention, the iSCSI protocol mainly comprises two parts, i.e., an iSCSI initiator and ISCSI TARGET, wherein the initiator is deployed at a client, the target is a target, and the iSCSI protocol is a process of mapping remote procedure call of iSCSI to an IP protocol. Referring to fig. 7, a schematic view of a scenario of a data read-write method provided by the embodiment of the present invention is shown, in this embodiment, by setting up a distributed storage cluster, global configuration information of ISCSI TARGET is stored in the distributed storage cluster in a key/value manner, when an iscsi server works, read-write of application data is directly written to a back-end storage, configuration data such as volume mapping information, path information, chap information is written to the distributed storage cluster, if configuration data of ISCSI TARGET needs to be obtained in the back-end storage, the back-end storage can read from the distributed storage cluster, and global configuration information of ISCSI TARGET is stored through the distributed storage cluster, so that the problem that the efficiency of reading-writing data from a local file in the conventional method is low is avoided, a larger amount of configuration data can be supported, and the back-end storage can share global configuration information with the iscsi server.
It should be noted that, the distributed storage cluster in this embodiment adopts etcd (etc Distribute), etcd is a KV storage system for sharing the distributed consistency of configuration and service discovery, is a high-availability distributed key value pair storage database, and adopts Raft protocol as a consistency algorithm inside etcd, and Raft protocol algorithm is a distributed consistency algorithm for ensuring data consistency in the distributed system.
Specifically, when the server is started, the distributed storage cluster initializes a data processing request monitoring mechanism, the monitoring mechanism can adopt modes of polling, event driving and the like, the monitoring mechanism continuously monitors the data processing request of the server, and once the request is received, the data processing request is immediately processed, the received data processing request can be analyzed, information such as the type, the content and the target storage position of the request is extracted, and the request is classified into different processing tasks such as a configuration data storage request, a configuration data reading request and the like.
And step 102, summarizing the configuration data generated by the server side in response to the data processing request.
In the embodiment of the invention, after the monitoring mechanism captures the data processing request, the request is responded immediately, the configuration data summarizing flow is started, the generated configuration data is collected from each module or service of the server side, and the collected configuration data is summarized, so that the integrity and consistency of the data are ensured. The aggregation process may include operations such as data merging, deduplication, formatting, and the like, storing the aggregated configuration data in a temporary storage area, such as a memory cache or a local file system, for subsequent processing.
Specifically, the global configuration information related to ISCSI TARGET includes key_name (pool name) and target_name (iscsi server destination name), value_version (initialized to 0), lun_id (volume name), target_ip_list (IP list of binding host), target_ initor _list (binding initiator list), path number information key_path_num (path number), value_max_path_num (maximum path number), min_path_num (minimum path number), and authentication information key_ CHAP _user (CHAP user name), value including incom/outcoming (initialized to 0), password (encrypted password), initor _list (binding initiator list).
And step 103, writing the configuration data of the server side into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster.
In the embodiment of the invention, configuration data of a server side is written into a distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster, wherein a preset interface in the distributed storage cluster is pre-packaged into a plurality of types of data processing interfaces, each data processing interface comprises a data writing interface and a data modifying interface, the distributed storage cluster responds to a configuration data writing request of the server side, a data writing interface corresponding to the configuration data writing request in the distributed storage cluster is called, the configuration data of the server side is written into a preset storage area of the distributed storage cluster through the data writing interface, a configuration data modifying operation on the preset storage area is determined in response to the configuration data modifying request of the server side, a data modifying interface corresponding to the configuration data modifying request in the distributed storage cluster is called, and configuration information of the preset storage area is updated through the data modifying interface.
Step 104, responding to the request of the server side for reading the configuration data stored in the back end, and synchronizing the configuration data to the back end through the data processing interface.
In the embodiment of the invention, when the iSCSI server side works, application data is directly written to the back-end storage, the server side corresponds to the back-end storage and comprises physical storage equipment such as a disk array (RAID) or Virtual storage equipment such as Virtual Disks (Virtual Disks) and the like, the application data of the iSCSI server side can be directly stored in the back-end equipment without specific limitation, global configuration data is required when the back-end storage runs the application data, the distributed storage cluster requests to read the configuration data, and the distributed storage cluster responds to the configuration data reading request corresponding to the back-end storage of the server side and synchronizes the configuration data to the back-end storage through a data processing interface.
According to the data reading and writing method provided by the embodiment of the invention, the configuration data generated by the server side is summarized by monitoring the data processing request of the server side, responding to the data processing request, the configuration data of the server side is written into the distributed storage cluster through the data processing interface pre-packaged by the distributed storage cluster, and the configuration data is synchronized to the back end for storage through the data processing interface in response to the configuration data reading request stored by the server side corresponding to the back end. The embodiment of the invention realizes the reliable storage of the global configuration data of the server side through the distributed storage cluster, prevents single-point faults, can adapt to the increase of the data volume of the server side, ensures the quick synchronous configuration data of the back-end storage and the server side by responding to the read-write request in time, realizes the efficient, reliable and flexible processing of the configuration data of the server side, ensures the consistency of the global configuration data and further improves the data synchronization effect of the iSCSI server side.
Further, referring to fig. 2, a flowchart of step 101 in a data read/write method provided in fig. 1 is shown, where the method is substantially the same as the data read/write method provided in the first embodiment of the present invention, and step 101 may include:
Step 201, monitoring a data processing request sent by a server side by adopting a preset monitoring thread;
step 202, analyzing the data processing request, and determining that the server side requests the distributed storage cluster to store the configuration data of the server side.
It should be noted that, in the embodiment of the present invention, when the server is started, the distributed storage cluster initializes one or more monitoring threads, where the monitoring threads are responsible for monitoring data processing requests of the server, specifically, a specific monitoring port may be allocated to the monitoring threads, so as to ensure that the data processing requests of the server can be sent through the port, the monitoring threads continuously monitor the designated port, and once the data processing requests are received, the data processing requests are immediately cached in the memory or temporary storage for subsequent processing.
In this embodiment, the data processing request of the cache may be preliminarily filtered and classified, the request needing to be processed by the distributed storage cluster is identified, the data processing request is extracted from the cache, the request content is parsed by using a predefined parser, key information such as the request type, the data type, the storage requirement and the like is extracted, the parsed request is verified, the validity and the integrity of the request are ensured, and the type and the storage position of the configuration data to be stored by the server side are determined according to the parsed request information, so that the configuration data is stored in the distributed storage cluster.
According to the embodiment of the invention, the monitoring thread is preset, so that the data processing request of the server side can be captured in real time, the request can be processed in time, the configuration data is stored through the distributed storage cluster, the reliability and redundancy of the data are improved, and single-point faults are prevented.
Further, referring to fig. 3, a flowchart of step 102 in a data read/write method provided in fig. 1 is shown, where the method is substantially the same as the data read/write method provided in the first embodiment of the present invention, and step 102 may include:
Step 301, responding to a data processing request, traversing the global configuration of the server side, and determining volume mapping configuration information, path number information and authentication information generated by the server side;
and step 302, carrying out queue caching on the volume mapping configuration information, the path number information and the authentication information to obtain summarized configuration data.
It should be noted that, in the embodiment of the present invention, the distributed storage cluster responds to the data processing request of the server side, traverses the global configuration of the server side, determines the volume mapping configuration information, the path number information and the authentication information generated by the server side, specifically, when the monitoring thread captures the data processing request, immediately responds to the request, starts the configuration traversal process, traverses the global configuration of the server side, and obtains all relevant configuration information.
Specifically, during traversal, configuration information related to volume mapping is identified and extracted, the configuration information comprises volume names, mapping paths, access rights and the like, traversal is continued, configuration information related to the number of paths is extracted, the number of paths comprises the number of paths, the hierarchical structure of the paths, the access frequency of the paths and the like, authentication information is extracted, configuration information related to authentication is extracted, the authentication information comprises user identity verification, authority control, access tokens and the like, and the volume mapping configuration information, the number of paths and the authentication information are subjected to queue caching to obtain summarized configuration data.
In this embodiment, one or more queue caches are initialized at the server side for temporarily storing the extracted configuration information. The queue cache can manage data in a first-in first-out (FIFO) mode, sequentially enqueues the extracted volume mapping configuration information, path number information and authentication information, each configuration information item serves as an independent queue element, manages the queue, ensures that the capacity and performance of the queue meet requirements, and when all the configuration information is successfully enqueued, gathers the data in the queue, wherein the gathering process can comprise operations such as data merging, deduplication, formatting and the like, so that the final configuration data is ensured to be complete and consistent, and the gathered configuration data is conveniently stored in a storage position appointed by a distributed storage cluster.
The embodiment of the invention ensures that all relevant configuration information is extracted through global configuration traversal, can quickly and accurately respond to a data processing request, generates summarized configuration data, ensures that the configuration information is not lost in the processing process through queue caching, improves the extraction efficiency of the configuration information, and reduces the processing time.
Further, referring to fig. 4, a flowchart of step 103 in a data read/write method provided in fig. 1 is shown, where the method is substantially the same as the data read/write method provided in the first embodiment of the present invention, and step 103 may include:
Step 401, pre-packaging a preset interface in a distributed storage cluster into a plurality of types of data processing interfaces, wherein the data processing interfaces comprise a data writing interface and a data modifying interface;
step 402, responding to a write-in configuration data request of a server side, and calling a data write-in interface corresponding to the write-in configuration data request in a distributed storage cluster;
step 403, writing configuration data of the server side into a preset storage area of the distributed storage cluster through a data writing interface;
Step 404, in response to a request for modifying configuration data from a server, determining a modification operation for configuration data of a preset storage area;
Step 405, calling a data modification interface corresponding to a configuration data modification request in the distributed storage cluster;
in step 406, the data modification interface executes a modification operation to update the configuration information of the preset storage area.
In the embodiment of the invention, a preset interface in a distributed storage cluster is packaged into a plurality of types of data processing interfaces in advance, the data processing interfaces comprise a data writing interface and a data modifying interface, in particular, in the distributed storage cluster, a plurality of types of data processing interfaces are predefined, the data processing interfaces can be divided into a plurality of types according to storage requirements, including a data writing interface, a data modifying interface and a data reading interface, and in the embodiment, the packaged data processing interfaces are comprehensively tested to ensure the stability and performance of the packaged data processing interfaces under different scenes, the interfaces for packaging and storing configuration data comprise a data writing interface, a data reading interface and a data modifying interface, wherein the data writing interface is wirite _iscsi_conf_data (key, value), the data reading interface is read_iscsi_conf_data (key, value), and the data modifying interface is modify_iscsi_conf_data (key, value).
Specifically, a data writing interface corresponding to a writing configuration data request in a distributed storage cluster is called in response to the writing configuration data request of a server side, the distributed storage cluster immediately responds to the request after receiving the writing configuration data request, a data writing flow is started, a data writing interface pre-packaged in the distributed storage cluster is called according to the request content, the data writing interface is executed, configuration data is written in the distributed storage cluster, the configuration data of the server side is written in a preset storage area of the distributed storage cluster through the data writing interface, and a proper preset storage area is selected according to the type and the access requirement of the configuration data, wherein the storage area can be a certain node or a specific storage partition in the distributed storage cluster.
Specifically, in this embodiment, after the distributed storage cluster receives the request for modifying the configuration data, the distributed storage cluster immediately responds to the request, starts a data modification flow, and determines a modification operation to be performed according to the content of the request. The modification operation comprises data updating, data deleting, data adding and the like, the configuration data needing to be modified is positioned in a preset storage area, the accuracy of the modification operation is ensured, a pre-packaged data modification interface in the distributed storage cluster is called according to the type of the modification operation, the data modification interface is executed, the configuration data in the preset storage area is modified, the determined modification operation is executed through the data modification interface, the configuration information in the preset storage area is updated, after the modification operation is completed, the modified configuration data is verified, and the successful execution of the modification operation is ensured.
According to the embodiment of the invention, the data processing interface is packaged in advance, the writing and modification request is responded, the data writing and modification operation is executed, the efficient, reliable and flexible processing of the configuration data of the server side is realized, the reliability and redundancy of the data are improved through the preset storage area of the distributed storage cluster, single-point faults are prevented, the distributed storage cluster can dynamically expand the storage capacity and the processing capacity according to the requirements, the increase of the data volume of the server side is adapted, and the configuration data can be updated quickly and efficiently through reasonable modification operation execution and data verification strategies.
Further, referring to fig. 5, a flowchart of step 104 in a data read/write method provided in fig. 1 is shown, the method is substantially the same as the data read/write method provided in the first embodiment of the present invention, the data processing interface includes a data read interface, and step 104 may include:
step 501, responding to a request for reading configuration data stored in a back end corresponding to a server end, and calling a data reading interface corresponding to the request for reading configuration data in a distributed storage cluster;
step 502, synchronizing configuration data of a preset storage area of the distributed storage cluster to a back-end storage through a data reading interface.
It should be noted that, in the embodiment of the present invention, the distributed storage cluster responds to a request for reading configuration data stored in the back end corresponding to the server end, immediately responds to the request after receiving the request for reading configuration data, starts a data reading flow, invokes a data reading interface corresponding to the request for reading configuration data in the distributed storage cluster, executes the data reading interface, reads configuration data from the distributed storage cluster, specifically, reads configuration data from a preset storage area of the distributed storage cluster through the data reading interface, synchronizes the read configuration data to the back end storage of the server end, and can adopt a full synchronization or incremental synchronization mode to select a proper synchronization policy according to actual requirements, and after the synchronization operation is completed, verifies the synchronized configuration data to ensure successful execution of the synchronization operation.
In the embodiment of the present invention, the back-end storage of the iSCSI server refers to physical or Virtual storage devices that actually store data, where the devices communicate with the client through the iSCSI protocol, and the back-end storage of the iSCSI server includes physical storage devices, such as a disk array (RAID), or Virtual storage devices, such as Virtual Disks (Virtual Disks), where application data of the iSCSI server may be directly stored in the back-end device, without being limited in detail herein.
According to the embodiment of the invention, the reliability and redundancy of data are improved through the preset storage area of the distributed storage cluster, single-point faults are prevented, the distributed storage cluster can dynamically expand the storage capacity and the processing capacity according to the requirements, the method is suitable for the increase of the data quantity of a server side, the configuration data can be ensured to be quickly and efficiently synchronized to the back-end storage through a reasonable synchronization strategy, and the back-end storage and the iscsi server side target share global configuration information.
Referring to fig. 6, a flowchart illustrating steps of another data read/write method according to an embodiment of the present invention is substantially the same as the data read/write method according to the first embodiment of the present invention, where the method may further include:
And 105, a distributed storage cluster is arranged at a server in advance, and connection between the distributed storage cluster and the server is created, wherein the distributed storage cluster comprises a main node and a secondary node for reading and writing data.
In the embodiment of the invention, a distributed storage cluster is arranged at a server in advance, an etcd cluster with multiple nodes is built, the connection between the distributed storage cluster and the server is created, the distributed cluster comprises a main node and a secondary node for reading and writing data, the fault tolerance of the data is ensured, the occurrence of brain cracking is avoided, an odd number of nodes, such as 4 nodes, are deployed, if 2 nodes are different from another 2 nodes in network, the cluster is divided into two network areas, the whole etcd cluster cannot be used, or two main ends, namely the main nodes of the two areas, can not be used, the system can not determine which area is selected for storing configuration data, and if the number of nodes is the odd number, the inside of the etcd is half of the number of normally available nodes is more than half of the total number of the original cluster nodes, and the cluster can normally provide service.
Step 101, monitoring a data processing request of a server side, wherein the data processing request is used for requesting to store configuration data of the server side;
step 102, responding to a data processing request, and summarizing configuration data generated by a server side;
Step 103, writing the configuration data of the server into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster;
Step 104, responding to the request of the server side for reading the configuration data stored in the back end, and synchronizing the configuration data to the back end through the data processing interface.
The embodiment of the invention, on the basis of realizing the beneficial effects of the first embodiment, pre-ends the distributed storage clusters at the server, creates the connection between the distributed storage clusters and the server, provides a storage solution with high availability through the distributed storage clusters, prevents single-point faults, supports dynamic expansion of the distributed storage clusters, and meets the storage requirement of continuously increasing configuration data of the server.
Specifically, after the response to the request for reading the configuration data stored in the back end corresponding to the server end, synchronizing the configuration data to the back end storage through the data processing interface, the method further includes:
firstly, monitoring the state of a main node of the distributed storage cluster, wherein the distributed storage cluster comprises a cluster alarm interface;
secondly, if the state of the main node is abnormal, a target main node is selected from the auxiliary nodes, and the configuration data is backed up to the target main node;
and secondly, sending abnormal prompt information to the server side by adopting the cluster alarm interface.
In the embodiment of the invention, when the distributed storage cluster is started, a main node state monitoring mechanism is initialized, the monitoring mechanism can adopt modes of heartbeat detection, state polling and the like, the main node periodically sends a heartbeat signal to the monitoring mechanism to indicate that the main node normally operates, the monitoring mechanism continuously receives the heartbeat signal and records the state of the main node, and the monitoring mechanism periodically polls the state information of the main node, including memory utilization rate, I/O and the like, so that the health state of the main node is ensured. The method comprises the steps of initializing a cluster alarm interface, sending abnormal prompt information to a server side when the state of a main node is abnormal, if the state of the main node is abnormal, selecting a target main node from auxiliary nodes, backing up configuration data to the target main node, selecting the target main node from auxiliary nodes of a distributed storage cluster, evaluating the selection process based on factors such as the load condition, the health state and the historical performance of the auxiliary nodes, backing up the configuration data in the main node to the target main node, ensuring the integrity and consistency of the data in the backup process, and switching the target auxiliary nodes to the main node after the backup of the configuration data is completed, so as to ensure the normal operation of the distributed storage cluster.
Specifically, the cluster alarm interface is adopted to send abnormal prompt information to the server, after detecting the state abnormality of the main node and completing the selection of the auxiliary node and the backup of configuration data, the abnormal prompt information is generated, the alarm information comprises detailed contents such as abnormal types, abnormal time, processing measures and the like, the cluster alarm interface is called to send the generated abnormal prompt information to the server, and the server can record the sending log of the alarm information, including information such as sending time, receiving party, alarm content and the like, so that the subsequent fault detection is facilitated.
The embodiment of the invention ensures the high availability of the distributed storage cluster by the secondary node selection and the primary node switching of the cluster, prevents single-point faults, configures the backup process of data, ensures the consistency of the data, avoids the loss or damage of the data, can timely send abnormal prompt information to a server through a cluster alarm interface, ensures the quick response and processing of the problem, and realizes the high availability and the data consistency of the distributed storage cluster.
Referring to fig. 8, a schematic structural diagram of a data read-write device according to an embodiment of the present invention is shown, where the device is applied to a distributed storage cluster, and the device includes:
the monitoring request module 601 is configured to monitor a data processing request of a server, where the data processing request is used to request to store configuration data of the server;
a data summarizing module 602, configured to summarize the configuration data generated by the server in response to the data processing request;
A data writing module 603, configured to write the configuration data of the server side into the distributed storage cluster through a data processing interface that is pre-packaged by the distributed storage cluster;
And the data reading module 604 is configured to synchronize the configuration data to the back-end storage through the data processing interface in response to a request of the server end for reading the configuration data corresponding to the back-end storage.
Further, the monitoring request module 601 includes:
the monitoring submodule is used for monitoring a data processing request sent by the server side by adopting a preset monitoring thread;
And the analysis submodule is used for analyzing the data processing request and determining the configuration data of the server side request distributed storage cluster storage server side.
Further, the data summarization module 602 includes:
A traversing submodule, configured to respond to the data processing request, traverse the global configuration of the server side, and determine volume mapping configuration information, path number information and authentication information generated by the server side;
And the caching sub-module is used for carrying out queue caching on the volume mapping configuration information, the path number information and the authentication information to obtain summarized configuration data.
Further, the data writing module 603 includes:
the packaging submodule is used for packaging preset interfaces in the distributed storage cluster into multiple types of data processing interfaces in advance, wherein the data processing interfaces comprise a data writing interface and a data modifying interface;
The first calling sub-module is used for responding to the write-in configuration data request of the server side and calling a data write-in interface corresponding to the write-in configuration data request in the distributed storage cluster;
the writing sub-module is used for writing the configuration data of the server side into a preset storage area of the distributed storage cluster through the data writing interface;
A determining submodule, configured to determine a modification operation of configuration data of a preset storage area in response to a request for modifying the configuration data of the server side;
the second calling sub-module is used for calling a data modification interface corresponding to the configuration data modification request in the distributed storage cluster;
and the modification sub-module is used for executing the modification operation through the data modification interface and updating the configuration information of the preset storage area.
Further, the data processing interface includes a data reading interface, and the data reading module 604 includes:
The third calling sub-module is used for responding to the reading configuration data request stored by the corresponding back end of the server end and calling a data reading interface corresponding to the reading configuration data request in the distributed storage cluster;
and the synchronous sub-module is used for synchronizing the configuration data of the preset storage area of the distributed storage cluster to the back-end storage through the data reading interface.
Further, the device further comprises:
The system comprises a cluster creating module, a data processing module and a data processing module, wherein the cluster creating module is used for pre-arranging a distributed storage cluster at a server and creating connection between the distributed storage cluster and the server, and the distributed storage cluster comprises a main node and a secondary node for reading and writing data.
Further, the device further comprises:
The monitoring node module is used for monitoring the state of a main node of the distributed storage cluster, wherein the distributed storage cluster comprises a cluster alarm interface;
The data backup module is used for selecting a target main node from the auxiliary nodes if the state of the main node is abnormal, and backing up the configuration data to the target main node;
And the abnormality prompting module is used for sending abnormality prompting information to the server side by adopting the cluster alarm interface.
According to the data read-write device provided by the embodiment of the invention, the configuration data generated by the server side is summarized by monitoring the data processing request of the server side, responding to the data processing request, the configuration data of the server side is written into the distributed storage cluster through the data processing interface pre-packaged by the distributed storage cluster, and the configuration data is synchronized to the back end for storage through the data processing interface in response to the configuration data reading request stored by the server side corresponding to the back end. The embodiment of the invention realizes the reliable storage of the global configuration data of the server side through the distributed storage cluster, prevents single-point faults, can adapt to the increase of the data volume of the server side, ensures the quick synchronous configuration data of the back-end storage and the server side by responding to the read-write request in time, realizes the efficient, reliable and flexible processing of the configuration data of the server side, ensures the consistency of the global configuration data and further improves the data synchronization effect of the iSCSI server side.
Referring to fig. 9, an embodiment of the present invention further provides an electronic device, as shown in fig. 9, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, where the processor 701, the communication interface 702, and the memory 703 complete communication with each other through the communication bus 704,
A memory 703 for storing a computer program;
The processor 701 is configured to execute the program stored in the memory 703, and implement the following steps:
Monitoring a data processing request of a server side, wherein the data processing request is used for requesting to store configuration data of the server side;
Summarizing configuration data generated by the server side in response to the data processing request;
Writing the configuration data of the server into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster;
And responding to a request of the server side for reading the configuration data corresponding to the back-end storage, and synchronizing the configuration data to the back-end storage through the data processing interface.
The communication bus mentioned by the above terminal may be a peripheral component interconnect standard (PERIPHERAL COMPONENT INTERCONNECT, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the terminal and other devices.
The memory may include random access memory (Random Access Memory, RAM) or may include non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central Processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), a digital signal processor (DIGITAL SIGNAL Processing, DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable gate array (FPGA) or other Programmable logic device, discrete gate or transistor logic device, or discrete hardware components.
In yet another embodiment of the present invention, a computer readable storage medium is provided, in which instructions are stored, which when run on a computer, cause the computer to perform the data read-write method according to any one of the above embodiments.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data read-write device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk Solid STATE DISK (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (10)
1. A data read-write method, applied to a distributed storage cluster, the method comprising:
Monitoring a data processing request of a server side, wherein the data processing request is used for requesting to store configuration data of the server side;
Summarizing configuration data generated by the server side in response to the data processing request;
Writing the configuration data of the server into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster;
And responding to a request of the server side for reading the configuration data corresponding to the back-end storage, and synchronizing the configuration data to the back-end storage through the data processing interface.
2. The method of claim 1, wherein monitoring the server-side data processing request for storing the server-side configuration data comprises:
monitoring a data processing request sent by a server side by adopting a preset monitoring thread;
And analyzing the data processing request to determine the configuration data of the server side requesting the distributed storage cluster to store the server side.
3. The method of claim 1, wherein the summarizing the server-side generated configuration data in response to the data processing request comprises:
traversing the global configuration of the server side in response to the data processing request, and determining volume mapping configuration information, path number information and authentication information generated by the server side;
and carrying out queue caching on the volume mapping configuration information, the path number information and the authentication information to obtain summarized configuration data.
4. The method of claim 1, wherein writing the server-side configuration data to the distributed storage cluster via a pre-packaged data processing interface of the distributed storage cluster comprises:
The method comprises the steps of packaging preset interfaces in a distributed storage cluster into multiple types of data processing interfaces in advance, wherein the data processing interfaces comprise a data writing interface and a data modifying interface;
Responding to a write-in configuration data request of the server side, and calling a data write-in interface corresponding to the write-in configuration data request in the distributed storage cluster;
Writing configuration data of the server side into a preset storage area of the distributed storage cluster through the data writing interface;
responding to the request for modifying the configuration data of the server side, and determining the modification operation of the configuration data of the preset storage area;
calling a data modification interface corresponding to the configuration data modification request in the distributed storage cluster;
And executing the modification operation through the data modification interface, and updating the configuration information of the preset storage area.
5. The method of claim 1, wherein the data processing interface comprises a data reading interface, and wherein synchronizing the configuration data to the back-end storage via the data processing interface in response to a request for reading configuration data from the back-end storage at the server end comprises:
Responding to a request for reading configuration data stored in the back end corresponding to the server end, and calling a data reading interface corresponding to the request for reading configuration data in the distributed storage cluster;
and synchronizing configuration data of a preset storage area of the distributed storage cluster to the back-end storage through the data reading interface.
6. The method according to claim 1, wherein before the monitoring the data processing request at the server side, further comprises:
The method comprises the steps of pre-ending a distributed storage cluster at a server side, and creating connection between the distributed storage cluster and the server side, wherein the distributed storage cluster comprises a main node and a secondary node for reading and writing data.
7. The method of claim 6, wherein the synchronizing the configuration data to the back-end storage via the data processing interface in response to a request from the server side to read the configuration data corresponding to the back-end storage, further comprises:
monitoring the state of a main node of the distributed storage cluster, wherein the distributed storage cluster comprises a cluster alarm interface;
If the state of the main node is monitored to be abnormal, a target main node is selected from the auxiliary nodes, and the configuration data is backed up to the target main node;
and sending abnormal prompt information to the server side by adopting the cluster alarm interface.
8. A data read-write apparatus for use with a distributed storage cluster, the apparatus comprising:
the monitoring request module is used for monitoring a data processing request of the server side, wherein the data processing request is used for requesting to store configuration data of the server side;
the data summarizing module is used for responding to the data processing request and summarizing the configuration data generated by the server side;
the data writing module is used for writing the configuration data of the server into the distributed storage cluster through a data processing interface which is packaged in advance by the distributed storage cluster;
And the data reading module is used for responding to a request of the server side for reading the configuration data stored in the back end, and synchronizing the configuration data to the back end through the data processing interface.
9. An electronic device, comprising:
A processor;
A memory for storing processor-executable instructions;
Wherein the processor is configured to execute the instructions to implement the data read-write method of any one of claims 1 to 7.
10. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when executed by a processor, implements the data read-write method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411865489.9A CN119806406A (en) | 2024-12-17 | 2024-12-17 | Data reading and writing method, device, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411865489.9A CN119806406A (en) | 2024-12-17 | 2024-12-17 | Data reading and writing method, device, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN119806406A true CN119806406A (en) | 2025-04-11 |
Family
ID=95255277
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411865489.9A Pending CN119806406A (en) | 2024-12-17 | 2024-12-17 | Data reading and writing method, device, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119806406A (en) |
-
2024
- 2024-12-17 CN CN202411865489.9A patent/CN119806406A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10353784B2 (en) | Dynamically adjusting the number of replicas of a file according to the probability that the file will be accessed within a distributed file system | |
| US10712975B2 (en) | Cluster configuration information replication | |
| US10831614B2 (en) | Visualizing restoration operation granularity for a database | |
| US7657796B1 (en) | System and method for distributed storage verification | |
| US6850955B2 (en) | Storage system and control method | |
| US8706694B2 (en) | Continuous data protection of files stored on a remote storage device | |
| US9442952B2 (en) | Metadata structures and related locking techniques to improve performance and scalability in a cluster file system | |
| US9146684B2 (en) | Storage architecture for server flash and storage array operation | |
| US8527561B1 (en) | System and method for implementing a networked file system utilizing a media library | |
| US7730153B1 (en) | Efficient use of NVRAM during takeover in a node cluster | |
| US20070027999A1 (en) | Method for coordinated error tracking and reporting in distributed storage systems | |
| CN101808127B (en) | Data backup method, system and server | |
| CN115145782A (en) | A server switching method, MooseFS system and storage medium | |
| CN116010164A (en) | Method, device and system for backing up data | |
| JP2015527620A (en) | Computer system, server, and data management method | |
| CN106452836A (en) | Method and apparatus for setting host node | |
| US20050086294A1 (en) | Method and apparatus for file replication with a common format | |
| US11217324B2 (en) | Validating data in storage systems | |
| CN119806406A (en) | Data reading and writing method, device, electronic device and storage medium | |
| US7873963B1 (en) | Method and system for detecting languishing messages | |
| CN120492301A (en) | Log collection method, device, electronic device, and storage medium | |
| CN119513189A (en) | Data synchronization device and method | |
| CN117131013A (en) | Data record processing method, distributed storage system and electronic equipment | |
| CN117234837A (en) | Health monitoring method of object storage cluster, related device and storage medium | |
| CN117539383A (en) | Distributed storage architecture for high-load backup system and deployment method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |