CN1731738A - Automated quick deployment method for large-scale computer cluster node - Google Patents

Abstract

The present invention relates to an automatic rapid deployment method for nodes of a large-scale computer cluster system, which adopts an image-based installation mode and a client/server structure, and uses a pre-boot execution environment PXE protocol, a dynamic host configuration protocol DHCP, a simple file transfer protocol TFTP and Wakeup On LAN technology and other technologies to realize the rapid and automatic deployment of large-scale cluster systems. The method includes five steps of obtaining the image, binding the image and the target node, installing the target node, configuring the target node, and testing, and realizing the complete automation of the cluster system node installation process. Compared with the traditional computer cluster system deployment method, this method greatly simplifies the cluster system node installation process and can greatly improve work efficiency.

Description

Automated Rapid Deployment Method for Large-Scale Computer Cluster System Nodes

Technical field

The invention belongs to the technical field of computer networks, and relates to an automatic rapid deployment method for large-scale computer cluster system nodes, in particular to a method for installing and deploying operating systems and application software on the cluster nodes.

Background technique

High-performance cluster computing systems are currently widely used, and the scale of the cluster is also expanding. During the construction and operation of the cluster, manual repeated installation and upgrade of operating systems and application software packages on a large number of nodes in the cluster will become a difficult task.

In the prior art before the present invention, operating system and application software are generally deployed with manual or ghost (hard disk duplication software developed by Symantec) hard disk recording or NFS (net file system, network file system) shared network installation method. The manual method uses the CD-ROM, and it takes a lot of time and manpower to install each node, and the configuration process of the nodes is very cumbersome and error-prone. The commonly used method in the engineering practice of building a cluster is the method of burning the ghost hard disk. This method needs to disassemble the hard disk of the node to be installed, and requires the target disk to be the same size as the source disk; or the NFS shared network installation method, that is, the A node in the network is used as an NFS server, and the files in the operating system installation disk are copied to this node. The installed node starts through a network card or a floppy disk, and downloads files from the server to complete the installation.

Some existing deployment software such as Oscar (Open Source Cluster Application Resource, an open source cluster application software resource, a set of cluster application software packages developed by the Open Cluster Group) and Systemimager (system mirror) are all aimed at specific clusters. The range is small, the function is relatively single, and the operation is inconvenient.

Contents of Invention

In view of the above-mentioned existing technical situation, the purpose of the present invention is to provide a new automatic deployment method for large-scale computer cluster system nodes, so as to simplify the cluster installation work, improve efficiency, and realize the complete automation of the installation process.

The technical solution for realizing the above object is: the method of the present invention is based on the client/server architecture; the node software installation adopts an image-based mode; the image is a complete system, containing the operating system and application software that need to be installed on the target node All images are stored on the server; the required hardware architecture of this method is provided with a management node as a server, the target node is connected with the management node through a general Ethernet switch, and the target node needs to support the pre-boot execution environment (Preboot Execution Environment) and Wakeup OnLAN (Wakeup OnLAN) network card and motherboard, the target node is connected to the management node as a client in this structure through a general Ethernet switch;

As a server in the architecture, the management node includes the server side of the Preboot eXecutionEnvironment protocol and the server side of the remote replication tool for image transmission; the management node is responsible for creating images, storing target node information, and waking up the target node from the network And provide the services needed for the deployment process. The client in the architecture runs in the memory of the target node, as a deployment agent, responsible for downloading the image from the management node and installing it on the local hard disk. The operation of the deployment agent is independent of the hard disk of the target node. The deployment agent includes the client of the remote copy tool for image transmission; the startup process of the target node complies with Intel's pre-boot execution environment PXE protocol; according to the PXE protocol, a dynamic host is provided on the management node Configuration protocol DHCP service, simple file transfer protocol TFTP (Trivial File Transfer Protocol,) service. For image transfer, the server side of the remote replication tool also needs to be running on the management node.

The specific implementation steps of this method are:

Step 1: Get the image:

Images can be obtained in two ways:

One is to create an image through the installation package, that is, copy the installation package of the operating system and application software in the installation CD to a directory of the management node, and use the installation package to directly create a virtual image on the management node;

The other is to capture the image from a source node, that is, first manually install a node as the source node of the image, the operating system and various application software required are installed on the source node, and then capture the source node from the management node image and store it. This method is suitable for the case where the target node and the source node are isomorphic

Step 2: Bind the image with the target node:

Bind the network identification information of the target node with the image, and determine the corresponding relationship between the image and the target node. One image can correspond to multiple target nodes, but a target node can only correspond to a unique image.

Step 3: Install the target node:

When deploying, first start the DHCP (Dynamic Host Configuration Protocol) service on the management node to provide network identification information for the target node; start the TFTP (Trivial File Transfer Protocol) service to provide services for the target node to download the deployment agent; start The server side of the remote replication tool that transfers the image between the management node and the target node;

The management node starts the target node through Wakeup On LAN technology; after the target node is started through the network card, it first obtains the network identifier from the management node according to the dynamic host configuration protocol DHCP, and then downloads and deploys the agent according to the simple file transfer protocol TFTP and uploads it on the local machine. Run in memory; after the deployment agent is started, first partition the local hard disk, then use the remote copy tool to download the image from the management node to the local hard disk, and complete the installation locally. After the installation is completed, the node will automatically restart (see Figure 3);

Step 4: Configure the target node:

When the target node is installed and started for the first time, run the configuration program to configure the operating system and application software of the target node; after configuration, the target node becomes a cluster node that can be used immediately;

Step 5: Test:

After all the nodes in the cluster are installed, run the test program to test whether the installation and configuration of the operating system and application software are correct.

The advantage of the inventive method is:

A. A single control point is provided for the user. The user only needs to connect to the management node through the terminal, and all control operations are completed on the management node. The target node only needs to be set in the BIOS to start through the network card, and its software installation process is completely automatic; the target node does not need peripheral devices such as keyboard, mouse, monitor, CD-ROM, and floppy drive.

B. Multiple nodes and different images can be installed at the same time. Through the binding between the image and the node, the corresponding relationship between the image and the node is determined, and multiple nodes can be installed at the same time without interfering with each other.

C. The cluster system node operating system and application software package installation and configuration are fully automated, and the target node does not require manual intervention at all. Nodes can be installed in batches with this method, which is suitable for installing large-scale cluster systems.

Description of drawings

Fig. 1: a schematic diagram of the hardware architecture adopted by the method of the present invention.

Figure 2: Flowchart of the main steps of cluster deployment according to the method.

Fig. 3: A schematic diagram of the detailed process of installing a target node with the method of the present invention.

The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments.

Detailed ways

Figure 1 shows the hardware architecture used in this method, "switch" represents a general Ethernet switch, "DHCP" represents "Dynamic Host Configuration Protocol", and "TFTP" represents "Trivial File Transfer Protocol". One of the management nodes acts as a server, providing DHCP service, TFTP service and image transmission service. The target node is connected to the management node through a switch, and the image is downloaded from the management node to complete the installation.

Fig. 2 is a flow chart of main steps of cluster deployment according to this method, showing the main steps of cluster deployment by this method.

The exact meaning of "bootstrap" in Figure 3 is bootstrap. It shows the detailed installation process of a target node, that is, the working process of the PXE protocol. The image server and DHCP server in the figure are both on the management node.

When this method is actually implemented, both the management node and the target node can use ordinary commercial computers. The management node requires input and output devices such as keyboard, mouse, optical drive, floppy drive and monitor, and the target node does not need these input and output devices. . The management node requires a larger hard disk (greater than 80G) to store images. The network card of the target node is required to support PXE and Wakeup On LAN, only in this way can the target node be started from the server, and the installation process of the target node can be fully automated. At present, general commercial computer hardware can support these two functions.

The target node and the management node are connected by Ethernet. Because the image file is usually relatively large, the network bandwidth has a great influence on the efficiency of the method. The bandwidth of the Ethernet is required to be at least 100 Mbps, and the network medium can be twisted pair and optical fiber. The connection method can be through 100M or 1000M general switch. Using a star or point-to-point network model, all target nodes can be directly connected to the management node.

When transferring the image between the management node and the target node, it is necessary to select a suitable remote copy tool. This copy tool can be selected according to the actual situation, and it is related to the operating system of the image. This copy tool needs to be able to read the file system of the image. write control. For example, in the Linux operating system, the remote synchronization software rsync can be selected as a remote replication tool.

In this method, the deployment agent is a micro-operating system with complete functions, and the deployment agent includes the client of the remote replication tool, necessary hard disk and network card drivers, and tools for operating the hard disk. In actual implementation, it can be obtained by customizing Linux. The deployment agent needs to run in the memory of the target node, and be able to recognize the local hard disk and communicate with the server through the network, so it is necessary to compile a new Linux kernel and add hardware drivers according to the different hardware of the target node. Make the root file system image RAMDISK file corresponding to the new kernel, and add necessary tools for operating the hard disk, clients of remote copy tools (such as rsync), driver modules (such as SCSI driver modules), etc.

Specific steps are as follows:

Step 1: Get the Image

There are two ways to obtain an image. One is to create an image on the management node, and the other is to capture an image from a source node. The specific steps of these two methods are introduced below.

Creating an image is to use some resources to generate an image on the management node. These resources include the operating system kernel, application software packages, disk partition files, etc. It is also necessary to do some configuration on the image, that is to say, to write scripts that require some configuration. Then install, reorganize and integrate these resources according to certain rules to form a relatively independent system, which is an image.

The general steps to create an image are as follows:

1) Knock in the image pile

The stub is the base of the image, which requires creating a directory structure and then loading some of the necessary filesystems for the current system. To some extent, creating an image is a copy of the management node operating system, but only a few files of the management node operating system itself are copied, so that there are "stubs" for future creation.

2) Install the system package

After creating the "pile", you can use the "pile" as the basis, that is, change the root directory to the pile, install the packages required by the system, and become a relatively independent file system that has nothing to do with the original file system. This has an image Prototype of the file system.

3) Configuration image

Although there is an image in the second step, the image cannot be used yet, and some configurations, such as disk partitions, need to be performed on the image before such an image can be used. Of course, this configuration can be changed, and file-mode-based images provide more flexibility.

The process of capturing an image is similar to the way of taking photos. First, manually install a source node, install the operating system and necessary application software packages on the source node, and then obtain an image of the source node from the management node without distortion. , the acquisition process is actually a replication process, which can be completed using remote replication tools such as rsync. The steps to capture an image are as follows:

1) Put the system in a quiescent state for valid snapshots

A running operating system needs to maintain its working state, and the data of the working state is saved in the memory and the hard disk. Disk snapshots cannot completely clone the state of the hard disk system. Therefore, when performing image capture on the source node system, the operating system must be properly adjusted to facilitate the smooth implementation of the image capture process. For example, sometimes it is necessary to temporarily delete some identifying information, such as log files, hostnames, IP addresses, etc. Only in this way can the obtained image be a relatively neutral file, which is convenient for porting to other nodes.

2) Capture an undistorted image

First, check whether the source node can capture the image, and then start the image capture agent, read the files in the disk of the source node, and store them in the hard disk of the management node.

3) Configuration image

Perform the necessary configuration on the obtained image. Different from the image creation method, the disk partition information of the source node is also copied when capturing the image, so there is no need to configure the disk partition. The configuration here is mainly the configuration of the application software and operating system in the image.

Step 2: Bind the image to the target node

Bind network information such as the IP address, host name, and MAC address of the target node with the image, and determine the corresponding relationship between the image and the target node. A small database needs to be established to store the correspondence between nodes and images. Among them, the image and the node have a one-to-many relationship, that is, one image can correspond to multiple nodes, and one node can only correspond to one certain image.

Step 3: Install the target node

During deployment, services that need to be started include DHCP service, TFTP service, rsync service, etc. The DHCP server provides network information such as IP address and host name for the target node, and specifies the location of the TFTP server, and notifies the target node to download the startup boot file from the specified location.

The TFTP server provides services for the target node to download and deploy the agent program, and the boot boot file, startup configuration file, Linux kernel and root file system are stored in the root directory of the TFTP server. The target node downloads these files to the local memory for loading and execution.

The rsync service is responsible for managing all images and providing services for transferring images between the management node and the target node. rsync runs as a daemon process on the management node. After the target node starts, it finds the image corresponding to the local machine from the server according to its own host name and other network identification information, and then copies the image from the management node to the local hard disk through rsync.

In this step, the configuration file of the DHCP service needs to be generated first, and the configuration file is generated according to the network information such as the target node IP address, host name, and MAC address stored in the management node database. Then start the DHCP service, TFTP service, and rsync service on the management node.

The target node needs to be connected to the power supply and set to boot through the network card in the BIOS. After starting various services on the management node, send a broadcast packet containing the MAC address of the target node from the management node, and start the target node through Wakeup OnLAN (Wakeup OnLAN) technology.

After receiving the broadcast packet corresponding to the local MAC address, the target node starts through the network card, first obtains the network identification information corresponding to the local machine from the DHCP server according to the local MAC address, and then downloads the network card startup boot program and deployment agent from the TFTP server , after the deployment agent loads and starts in the local memory, it first detects and mounts the local hard disk, and then partitions the local hard disk according to the disk partition configuration file. After the partition is completed, use a remote copy tool such as rsync to download the image corresponding to the local machine from the server and copy it completely to the local hard disk. After the installation is complete, the node will automatically restart.

Step 4: Configure the target node

When the target node is installed and started for the first time, run the configuration program to configure the operating system and application software of the target node. Because the cluster system is a loosely coupled and multi-machine system, each node in the cluster is relatively independent, and each node has an independent operating system and application software. In order for the nodes in the cluster to be able to write to each other and provide servers to the outside world, the nodes must be configured.

In the Linux operating system, the configuration process is to execute a series of scripts, and each application software has a corresponding configuration script, and these scripts have been written into the image during the step of obtaining the image. Since the configuration methods of each application software are different, it is necessary to write a configuration script and store it in the image according to the specific application software. When the node starts for the first time, these scripts are invoked to complete the configuration of the application software. For the Windows operating system, the configuration of the software needs to call some configuration tools, which is implemented in a batch manner.

After configuration, the target node is a cluster node that can be used immediately.

Step 5: Test: After all the nodes in the cluster are installed, run the test program to test whether the installation and configuration of the operating system and application software are correct. The test program is written into the image when configuring the image. It is a test of the basic functions of the application software, such as the parallel computing software MPICH under the Linux system. After completing the configuration of the software, you can run a program written in MPI A small program to check whether the running result is correct.

The test is closely related to the application, and the test program needs to be designed according to different application programs. Since the image is configurable, test programs can be pre-written into the image.

Since the method adopts a client/server (Client/Server) structure, images are all stored on the management node, and all target nodes download images from the management node in parallel. Therefore, under the premise of reliable transmission assumptions and fair bandwidth sharing, the time required for cluster deployment can be estimated based on the network bandwidth. According to the cluster deployment process, the following formula for calculating the deployment time can be obtained:

Deployment time = transfer image preparation time + transfer image time

= Transmission image preparation time + number of nodes × image size / network bandwidth

Expressed in letters: T=C+n×S×8/W (Formula 1)

Among them, T is the deployment time; C is the transfer image preparation time, that is, the time from the start of the network card of the target node to the loading and running of the image deployment agent in memory, which can be considered as a constant; n is the number of target nodes deployed at the same time; S is the image size (MB); W is the network bandwidth (Mb/s).

In the cluster building process of the Shandong Provincial High Performance Computing Center, the cluster deployment software designed according to the method of the present invention is used. The hardware environment of the cluster is as follows:

The management node is Inspur Yingxin NF260 (2 x Xeon 2.4GHz CPU, memory 4GB, SCSI Ultra320, 1000Mb network card). The target node is Inspur NF170 with 2G memory, IDE hard disk, 100Mb network card, no CD-ROM drive, floppy drive, keyboard and display terminal. The backbone network is connected by a Gigabit switch through optical fiber, the management node is connected to the backbone network, and 96 target nodes are divided into 3 subnets, and the subnet is connected to a 100M switch through a twisted pair, and then connected to the backbone network through a 100M switch.

The image acquisition adopts the image capture method, and the obtained image size is 3.7G. Wake up 87 nodes through the network and start up at the same time (9 nodes did not start successfully due to setting reasons), and start from the network card for software deployment. The time it took to finally complete the deployment was 48 minutes. The average transmission rate of the network detected during deployment is 990Mb. The preparation time for image transmission is 150 seconds, then the deployment time calculated according to Formula 1 is T=150+3700×8×87/990=2751 seconds=46 minutes. This time basically coincides with the experimental statistical time, which proves that Formula 1 is correct.

It can be seen from formula 1 that the time required for deployment has a linear relationship with the number of target nodes installed at the same time, and the coefficient of this linear relationship is determined by the size of the image and the network bandwidth. The larger the image, the larger the coefficient, and the longer the deployment time; the larger the network bandwidth, the smaller the coefficient, and the shorter the deployment time.

Claims (4)

1. An automated rapid deployment method for nodes of a large-scale computer cluster system, characterized in that the method is based on the client/server architecture, and the node software installation adopts an image-based method, and the image is a complete system that contains the The operating system and application software package installed on the node; all images are stored on the server; the hardware architecture required by this method is provided with a management node as a server, and the target node is connected to the management node through a general Ethernet switch. A network card and motherboard supporting pre-boot execution environment and wake-up on LAN are required, and the target node is connected to the management node as a client in the structure through a general-purpose Ethernet switch; 1), the management node is the server side in the architecture, including the server side for image transmission and the server side of the pre-boot execution environment protocol, responsible for creating images, managing images, storing target node information, waking up target nodes from the network, and providing deployment Various services required by the process; 2), the target node is the client in the architecture, obtains the deployment agent from the management node, and the deployment agent is responsible for downloading the image from the management node as the server and installing it on the local hard disk; 3), the start-up process of the target node complies with Intel's pre-boot execution environment protocol, and provides dynamic host configuration protocol DHCP, simple file transfer protocol TFTP services and services for transferring images on the management node; 4) When image transmission is performed between the target node and the management node, different remote replication tools based on client/server architecture are used. 2. The method according to claim 1, wherein the specific steps of the automated rapid deployment method for cluster system nodes are: Step 1: Get the Image There are two ways to obtain an image: The first is to create an image through the installation package, that is, copy the installation package of the operating system and application software in the installation CD to a directory of the management node, and use the installation package to directly create a virtual image on the management node; The second is to capture the image from a source node, that is, first manually install a node as the source node of the image, the operating system and various application software required are installed on the source node, and then capture the source node from the management node The image of the image is stored, which is suitable for the case where the target node and the source node are isomorphic; Step 2: Bind the image to the target node Bind the network identification information of the target node with the image, and determine the corresponding relationship between the image and the target node. One image corresponds to multiple target nodes, but one target node can only correspond to a unique image; Step 3: Install the target node When deploying, first start the DHCP service on the management node to provide network identification information for the target node; start the TFTP service to provide services for the target node to download the deployment agent; start the remote image transmission between the management node and the target node The server side of the replication tool; The management node starts the target node through Wakeup On LAN; after the target node is started through the network card, it first obtains the network identifier from the management node according to the dynamic host configuration protocol DHCP, and then downloads the deployment agent according to the simple file transfer protocol TFTP and runs it in the local memory; deployment The agent downloads the image from the management node and completes the installation locally. After the installation is complete, the node restarts automatically; Step 4: Configure the target node When the target node is installed and started for the first time, run the configuration program to configure the operating system and application software of the target node; after configuration, the target node becomes a cluster node for immediate use; Step 5: Test After all the nodes of the cluster system are installed, run the test program to test whether the installation and configuration of the operating system and application software are correct. 3. The method according to claim 1 or 2, characterized in that: the target node is set in the BIOS to start through the network card, and its software installation process is fully automated; the target node does not need a keyboard, mouse, display, CD-ROM, Peripheral devices such as floppy drives; all operations are completed on the management node. 4. The method according to claim 1 or 2, characterized in that: the deployment agent adopts manual customization, it is a perfect micro-operating system, runs in the memory of the target node, and is independent of the hard disk of the target node; the deployment agent It includes the client of the remote copy tool, the necessary hard disk and network card drivers, and the tools for operating the hard disk.

Images