CN106446303A - System and method for deploying large-scale cluster file system - Google Patents

Abstract

The invention provides a system and method for deploying a large-scale cluster file system. The method comprises the steps that a Json configuration file is generated by separating out configuration parameters needed in the cluster file system configuration procedure, the configuration file is analyzed through a shell script, and cluster deploying is quickly conducted; meanwhile, when a bottom-layer disk is processed, the disk with a large quantity of nodes is processed by adopting the strategy that the process and the thread are self-adaptively adjusted by a process pool and a thread pool, therefore, the disk processing speed of a large-scale cluster is optimized, and quick deployment of the large-scale cluster is achieved.

Description

Systems and methods for deploying large-scale clustered file systems

technical field

The invention relates to the field of cluster file systems, in particular to the rapid deployment of large-scale cluster file systems.

Background technique

Cluster (cluster) technology is a relatively new technology. Through cluster technology, relatively high benefits in terms of performance, reliability, and flexibility can be obtained at a lower cost. A cluster is a group of independent computers interconnected by a high-speed network that form a group and are managed as a single system. When a client interacts with the cluster, the cluster behaves like a stand-alone server.

With the outbreak and popularization of big data, the Internet and even the Internet of Things, clusters have developed to large-scale and ultra-large-scale. Figure 1 shows the flow chart of traditional cluster deployment, and Figure 2 shows the flow chart of traditional cluster deployment disk processing. Cluster file systems, especially large-scale cluster file system deployments, are most criticized by enterprises for their cumbersome and time-consuming process, and cluster deployment is the most time-consuming. The most important step is disk processing. The existing cluster deployment method uses manual sequential processing. First, the first disk of the first node is processed, and then the second disk of the first node is processed. After the disk is processed, turn to the second node, and then the third node, until all nodes are processed before performing other deployment operations; on the other hand, manually deploying and setting a large number of parameters is prone to errors, resulting in deployment failure, and the traces need to be cleared Redeploy. All of the above factors make the cluster deployment process cumbersome and time-consuming.

Contents of the invention

In order to solve the problems of the technologies described above, the technical solution of the present invention is as follows:

The present invention provides a system for deploying a large-scale cluster file system, including a configuration file generator, a central control unit, and a detector. The configuration file generator forms configuration parameters by separating the configuration parameters required in the cluster file system deployment process. The configuration file generator is connected to the control data input interface of the central control unit to transmit configuration files to the central control unit, and the control data output interface of the central control unit is connected to all nodes of the file system to transmit configuration files and control instructions to each node As well as detection instructions, each node can be a master node or a slave node, and each node is configured with a parsing script for parsing the configuration file transmitted by the central control unit, and the detector is connected to all nodes to detect the configuration of each node Parameters in the process, the detector is also connected to the detection data input interface of the central control unit, and feeds back the configuration parameters of each node to the central control unit.

Further, the configuration file generator is a Json file generator, and the parsing script is a shell script.

Further, after each node to be deployed receives the Json configuration file transmitted by the central control unit, it sends node parameters to the detector, and the detector specifies the master node and slave node according to the node parameters, calculates the number of nodes n, and calculates the number of nodes n And the parameters of each node are sent to the central control unit. The central control unit uses the number of nodes n as the number of node processing processes, and sends control instructions and detection instructions to each node. The control instructions are used to control n nodes to perform parameter configuration at the same time to achieve multi-process configuration.

Further, after receiving the detection instruction from the central control unit, each node sends its CPU core number and disk parameters to the detector, and the detector obtains the CPU core number t of each node and the disk number of each node according to the received data d, and transmit the number of CPU cores t of each node and the number of disks d of each node to the central control unit, and the central control unit uses the smaller one of (2*t _i ) and d _i as the thread on the i-th node number, so that each node performs multi-threaded operations on its own disk in the corresponding process.

Further, the central control unit also stores the node parameters of the cluster deployment and the disk parameters of each node input by the staff, and the central control unit compares the parameters of each node and the disk parameters of each node stored and obtained from the detector, When it is found that a given parameter does not exist, set a dynamic breakpoint and prompt the operator to modify the configuration file, or if the disk given by the parameter does not exist or has been used for other purposes, set a dynamic breakpoint and prompt the operator to modify Configuration file, continue to execute if the detection is correct after modification.

Furthermore, during the process of creating storage services and creating metadata services, the detector synchronously detects the parameters of each node and each disk, and sends them to the central control unit in real time. The central control unit judges whether there is an error in the configuration process according to the configuration parameters. If there is an error, set a breakpoint and prompt for modification, and continue to execute after the modification is detected to be correct.

The present invention also provides a method for deploying a large-scale cluster file system, the steps are as follows:

Step 1, form a Json configuration file by separating the configuration parameters required in the cluster file system deployment process;

Step 2, create a monitoring service;

Step 3, automatically detect the given node of the parameter, and use the node number n as the number of node processing processes to execute multiple processes;

Step 4, automatically detect disk parameters, count the number of disks di of each node, obtain the number of CPU cores ti of each node, and use the smaller one of (2*ti) and di as the number of threads on the i-th node, in the corresponding process Execute multi-threaded operations within;

Step 5, create a storage service;

Step 6, create metadata service;

Step 7, create a file system.

Further, step 3 also includes comparing the stored and detected parameters of each node. When it is found that a given node of a certain parameter does not exist, then set a dynamic breakpoint and prompt the operator to modify the configuration file. After the modification, the detection is correct and then continue to execute .

Further, step 4 also includes comparing the stored and detecting the disk parameters of each node, if the disk given by the parameter does not exist or has been used for other purposes, set a dynamic breakpoint and prompt the operator to modify the configuration file, and the detection is correct after modification Then continue to execute.

Further, step 5 also includes that during the process of creating the storage service, the detector synchronously detects the parameters of each node and each disk, and sends them to the central control unit in real time, and the central control unit judges whether there is an error in the configuration process according to the configuration parameters, and if there is an error Then set a breakpoint and prompt for modification, and continue to execute after the modification is detected to be correct.

Further, step 6 also includes that during the process of creating the metadata service, the detector synchronously detects the parameters of each node and each disk, and sends them to the central control unit in real time, and the central control unit judges whether there is an error in the configuration process according to the configuration parameters. If there is an error, set a breakpoint and prompt for modification, and continue to execute after the modification is detected to be correct.

By separating the configuration parameters required in the cluster file system deployment process, a Json configuration file is formed, and the configuration file is parsed by a shell script to quickly deploy the cluster; at the same time, when processing the underlying disk, the process pool and thread pool are used for adaptive adjustment The strategy of process and thread handles disks of a large number of nodes, optimizes the disk processing speed of large-scale clusters, and realizes the rapid deployment of large-scale clusters.

Description of drawings

Figure 1 shows a flowchart of traditional cluster deployment.

FIG. 2 shows a flow chart of disk deployment in a traditional cluster.

FIG. 3 shows a schematic structural diagram of the cluster deployment system of the present invention.

FIG. 4 shows a flow chart of cluster deployment disk processing in the present invention.

Fig. 5 shows a flow chart of cluster deployment of the present invention.

detailed description

The technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Firstly, the present invention provides a system for deploying a large-scale cluster file system.

As shown in Figure 3, the deployment system of the present invention includes a configuration file generator, a central control unit, and a detector, and the configuration file generator is connected to the control data input interface of the central control unit to transmit the configuration file to the central control unit, and the central control unit The control data output interface of the node is connected to node 1, node 2, node 3~node n, and transmits configuration files, control instructions and detection instructions to node 1, node 2, node 3~node n, and node 1, node 2, node 3~node Node n can be a master node or a slave node, and each node is configured with a parsing script for parsing the configuration file transmitted by the central control unit. The detector is connected to node 1, node 2, node 3 ~ node n for Detect the parameters in the configuration process of node 1, node 2, node 3~node n, the detector is also connected to the detection data input interface of the central control unit, and feedback the configuration parameters of node 1, node 2, node 3~node n to the central control unit .

The configuration file generator can be a Json file generator or an XML configuration file generator, and only the Json file generator will be described in detail below.

The parsing script can be a shell script.

When the system is deployed, firstly, the Json file generator forms the Json configuration file by separating the configuration parameters required in the cluster file system deployment process, and transmits the Json configuration file to the central control unit, and the central control unit transmits the Json configuration file to the required Each node to be deployed sends a startup command to the detector at the same time. After each node to be deployed receives the Json configuration file transmitted by the central control unit, it first sends the node parameters to the detector, and the detector specifies the master node and the slave node according to the node parameters. , calculate the number of nodes n, and send the number of nodes n and the parameters of each node to the central control unit. The central control unit uses the number of nodes n as the number of node processing processes, and sends control instructions and detection instructions to each node. The control instructions are used to control n nodes perform parameter configuration at the same time to realize multi-process configuration.

After receiving the detection instruction from the central control unit, each node sends its CPU core number and disk parameters to the detector, and the detector obtains the CPU core number t of each node and the disk number d of each node according to the received data, and The number of CPU cores t of each node and the number of disks d of each node are sent to the central control unit, and the central control unit uses the smaller one of (2*t _i ) and d _i as the number of threads on the i-th node, so that Each node executes multi-threaded operations on its own disk in the corresponding process.

In addition, the central control unit stores the node parameters of this cluster deployment and the disk parameters of each node input by the staff. The central control unit compares the parameters of each node stored and obtained from the detector. If it exists, set a dynamic breakpoint and prompt the operator to modify the configuration file, and continue to execute after the modification is detected to be correct.

The central control unit compares the disk parameters of each node stored and obtained from the detector. If the disk given by the parameter does not exist or has been used for other purposes, it will set a dynamic breakpoint and prompt the operator to modify the configuration file. After the modification, the detection is correct. Then continue to execute.

The shell script in each node parses the Json configuration, and performs multi-threaded operations on its own disk according to the corresponding process, and starts to create the storage service. After the step of creating the storage service is completed, it starts to create the metadata service. After the step of creating the metadata service is completed Start creating the file system and complete the deployment of the cluster file system.

During the process of creating storage service and metadata service, the detector synchronously detects the parameters of each node and each disk, and sends them to the central control unit in real time. The central control unit judges whether there is an error in the configuration process according to the configuration parameters. If there is an error, set Breakpoint and prompt to modify, and continue to execute after the modification is detected to be correct.

By separating the configuration parameters required in the cluster file system deployment process, a Json configuration file is formed, and the shell script is used to parse the configuration file to quickly deploy the cluster file system.

During the deployment process, the parameter monitoring mechanism will continuously detect the configuration parameters. If a parameter error is found, a dynamic breakpoint will be automatically set and the operator will be prompted to correct the parameter. After the parameter is corrected, the monitoring will continue to execute from the breakpoint, ensuring large-scale A successful deployment of the cluster file system; at the same time, when dealing with the underlying disk, the process pool and thread pool are used to adaptively adjust the process and thread strategy to process the disk of a large number of nodes, and the process is adaptively adjusted through the number of nodes, the number of disks and the number of CPU cores And threads, optimize the disk processing speed of large-scale clusters Optimize the disk processing speed of large-scale clusters, so that the disk processing efficiency can reach the highest.

In addition, the present invention also provides a method for deploying a large-scale cluster file system.

The cluster deployment steps are as follows:

Step 1, separate the parameters to form a configuration file;

Step 2, create a monitoring service;

Step 3, automatically detect the given node of the parameter, and use the node number n as the number of node processing processes to execute multiple processes;

Step 4, automatically detect the disk parameters, count the number of disks di of each node, obtain the number of CPU cores ti of each node, take the smaller one of (2*t _i ) and d _i as the number of threads on the i-th node, and in the corresponding Execute multi-threaded operations within the process;

Step 5, create a storage service;

Step 6, create metadata service;

Step 7, create a file system.

Step 1 is specifically to form a Json configuration file by separating the configuration parameters required in the cluster file system deployment process.

Step 3 also includes comparing the stored and detected parameters of each node, and setting a dynamic breakpoint and prompting the operator to modify the configuration file when it is found that a given node does not exist.

Step 4 also includes comparing the stored and detecting the disk parameters of each node. If the disk given by the parameter does not exist or has been used for other purposes, set a dynamic breakpoint and prompt the operator to modify the configuration file. After the modification, the detection is correct and the execution continues .

Step 5 also includes that during the process of creating the storage service, the detector synchronously detects the parameters of each node and each disk, and sends them to the central control unit in real time. The central control unit judges whether there is an error in the configuration process according to the configuration parameters, and if there is an error, sets the interrupt Click and prompt to modify, and continue to execute after the modification is detected to be correct.

Step 6 also includes that during the process of creating the metadata service, the detector synchronously detects the parameters of each node and each disk, and sends them to the central control unit in real time. The central control unit judges whether there is an error in the configuration process according to the configuration parameters, and if there is an error, set Breakpoint and prompt to modify, and continue to execute after the modification is detected to be correct.

Through the above method, the rapid deployment of the cluster is realized, and the problems of cumbersome and time-consuming deployment methods in the existing methods are solved.

Although some aspects have been described in the context of an apparatus, it is obvious that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of the corresponding block or item or feature of the corresponding apparatus. Some or all of the method steps may be performed by (or using) hardware means such as microprocessors, programmable computers, or electronic circuits. One or more of the most important method steps can be carried out by means of such means.

The implementation may be in hardware or in software or may use a digital storage medium such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory having electronically readable control signals stored thereon. For execution, the electronically readable control signals cooperate (or are capable of cooperating) with a programmable computer system such that the corresponding method is performed. A data carrier may be provided having electronically readable control signals capable of cooperating with a programmable computer system such that the methods described herein are carried out.

The implementation may also take the form of a computer program product having a program code operable to carry out the method when the computer program product is run on a computer. The program code can be stored on a machine readable carrier.

What has been described above is illustrative only, and it is to be understood that modifications and variations in the arrangements and details described herein will be apparent to those skilled in the art. It is therefore the intention to be limited only by the scope of the appended claims rather than by the specific details presented by way of the foregoing description and explanation.

Claims (10)

1. A system for deploying a large-scale cluster file system, characterized in that: it includes a configuration file generator, a central control unit, and a detector, and the configuration file generator separates the required configuration parameters in the cluster file system deployment process , to form a configuration file, the configuration file generator is connected to the control data input interface of the central control unit to transmit the configuration file to the central control unit, the control data output interface of the central control unit is connected to all nodes of the file system, and transmits the configuration file to each node , control instructions and detection instructions, each node can be a master node or a slave node, and each node is configured with a parsing script for parsing the configuration file transmitted by the central control unit, and the detector is connected to all nodes for detection The parameters in the configuration process of each node, the detector is also connected to the detection data input interface of the central control unit, and the configuration parameters of each node are fed back to the central control unit. The configuration file generator is a Json file generator, and the analysis script is a shell script. 2. The system according to claim 1, wherein after each node to be deployed receives the Json configuration file transmitted by the central control unit, it sends the node parameters to the detector, and the detector specifies the master node and the slave node according to the node parameters, and calculates The number of nodes is n, and the number of nodes and parameters of each node are sent to the central control unit. The central control unit uses the number of nodes n as the number of node processing processes, and sends control instructions and detection instructions to each node. The control instructions are used to control n The nodes perform parameter configuration at the same time to realize multi-process configuration. 3. The system according to claim 2, wherein each node sends the number of CPU cores and disk parameters to the detector after receiving the detection instruction from the central control unit, and the detector obtains the CPU of each node according to the received data. The number of cores t and the number of disks d of each node, and the number of CPU cores t of each node and the number of disks d of each node are sent to the central control unit, and the central control unit uses the smaller of (2*t _i ) and d _i One is the number of threads on the i-th node, so that each node performs multi-threaded operations on its own disk in the corresponding process. 4. The system according to claim 3, wherein the central control unit further stores the node parameters of this cluster deployment input by the staff and the disk parameters of each node, and the central control unit compares the parameters stored and obtained from the detector. Node parameters and disk parameters of each node. When a node given by a parameter does not exist, set a dynamic breakpoint and prompt the operator to modify the configuration file, or if the disk given by the parameter does not exist or has been used for other purposes, then Set a dynamic breakpoint and prompt the operator to modify the configuration file, and continue to execute after the modification is detected to be correct. 5. The system according to claim 3, wherein during the process of creating storage services and creating metadata services, the detector synchronously detects the parameters of each node and each disk, and sends them to the central control unit in real time, and the central control unit according to the configuration parameters Determine whether there is an error in the configuration process, if there is an error, set a breakpoint and prompt for modification, and continue execution after the modification is detected to be correct. 6. A method for deploying a large-scale cluster file system, the steps are as follows: Step 1, form a Json configuration file by separating the configuration parameters required in the cluster file system deployment process; Step 2, create a monitoring service; Step 3, automatically detect the given node of the parameter, and use the node number n as the number of node processing processes to execute multiple processes; Step 4, automatically detect disk parameters, count the number of disks di of each node, obtain the number of CPU cores ti of each node, and use the smaller one of (2*ti) and di as the number of threads on the i-th node, in the corresponding process Execute multi-threaded operations within; Step 5, create a storage service; Step 6, create metadata service; Step 7, create a file system. 7. The method according to claim 6, wherein step 3 further comprises by comparing stored and detected node parameters, when a given node of a certain parameter is found not to exist, setting a dynamic breakpoint and prompting the operator to modify the configuration file, continue to execute after the modification is detected to be correct. 8. The method according to claim 6, wherein step 4 also includes comparing the stored and detecting disk parameters of each node, if the disk given by the parameter does not exist or has been used for other purposes, then set a dynamic breakpoint and prompt the operation Personnel modify the configuration file, and continue to execute after the modification is detected to be correct. 9. The method according to claim 6, wherein step 5 further comprises that during the process of creating the storage service, the detector synchronously detects the parameters of each node and each disk, and sends them to the central control unit in real time, and the central control unit judges according to the configuration parameters Whether there is an error in the configuration process, if there is an error, set a breakpoint and prompt for modification, and continue to execute after the modification is detected to be correct. 10. The method according to claim 6, wherein step 6 further comprises that in the process of creating the metadata service, the detector synchronously detects the parameters of each node and each disk, and sends them to the central control unit in real time, and the central control unit according to the configuration parameters Determine whether there is an error in the configuration process, if there is an error, set a breakpoint and prompt for modification, and continue execution after the modification is detected to be correct.