RU2461055C1 - Cluster system with direct channel switching - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: cluster system has master and slave computer modules, having a PICe bus on their motherboards, a switch and interface boards, having 2 ports each - one port operates in transparent port mode and the second in non-transparent port mode. The switch can form a range of addresses in the address space of the PICe bus of the master computer module which covers the entire access window for the slave computer module and can form in each non-transparent port of the interface board, a window with an address in the address region in the master computer module for enabling each slave computer module to access through the address space the memory of other slave computer modules.

EFFECT: high rate of transmitting data and shorter delay time for writing data into foreign memory once.

8 dwg

Description

The invention relates to computer technology and relates to a communication environment, characterized by the use of direct switching of PCI Express lanes without converting the communication protocol.

Clustering is a technology by which two or more servers function as one, representing, from the point of view of the user, a single computing resource. The main goal of clustering is to increase the performance and / or fault tolerance of a cluster system. Cluster systems are used in companies that need reliability and uninterrupted operation of business-critical servers and applications.

Computing clusters built using PCI Express direct switching technology can be used to solve computationally labor-intensive problems in the fields of molecular pharmacology, nanoelectronics, to create new-generation energy complexes, as well as to conduct basic scientific research in astrophysics, microbiology, solid state physics, neuromathematics , tomography of the Earth and in other areas. First of all, from clusters built on the basis of this communication environment, one can expect efficiency in calculations on non-structural and adaptive grids, that is, on problems in which data are complexly scattered in small amounts in the total memory of the system, the mutual arrangement of which is not regular and / or changes in the calculation process. This is a promising, rapidly growing direction in numerical methods, known for its extremely high laboriousness in its implementation on traditional computing clusters.

The development of the PCI Express direct switching network was carried out by the American company OneStopSystems (http://www.onestopsystems.com), but there is no simple and effective way to scale the network without the concentrated central switch that is present in our implementation.

A self-forming supercomputer is known that contains many racks, on each of which there is an array of connectors for connecting conductive power racks of the racks to the power rails of the boards, and an array of boards with microcircuits placed on them, the boards have devices for fixing them on the racks, and the elements of the supercomputer are connected to each other with a friend using fiber-optic and electric buses, there is a construction computer, one or more manipulators controlled by this computer, as well as libraries of cooled boards, opt fiber busbars and electric cables in the form of sets of these elements installed in stores available for manipulators (RU No. 2367125, H05K 7/00, H05K 7/20, G06F 15/18, G06F 1/16, G06F 1/20, publ. 10.09 .2009).

The disadvantage of this solution is the lack of speed.

A known cluster system that contains a leading computing module, which is a computer with a PCIe bus on its motherboard, is connected to a switch, the outputs of which are connected to slave computing modules made with a PCIe bus on their motherboards (see the article “Cluster systems”, posted on the IBS Platformix website on the Internet at http://platformix.ru/eng/IT-Infrastructure/components-IT/computing/clusters-sys/index.wbp, found in May 2011 g.). Adopted as a prototype.

The disadvantage of this solution is the rather large delay in transmitting data on the network. In addition, the design of such a supercomputer is quite complicated and expensive.

The present invention is aimed at achieving a technical result, which consists in increasing the speed of data transfer and reducing the delay time of a single recording of data in a "foreign" memory at a level comparable to similar indicators for data transfer inside a computer.

The indicated technical result is achieved in that the cluster system contains a host computing module, which is a computer with a PCIe bus on its motherboard, connected via an interface board connected to the PCIe bus of the host computing module, with a switch, the outputs of which are connected to the slave computing modules made with a PCIe bus on their motherboards, through interface boards each, connected to the PCIe bus of the corresponding slave computing module, each interface board in full with two ports, one of which is configured to operate in transparent port mode, and the second in opaque port mode, while the host computing module is connected to the transparent port of one interface board that is connected to the switch, and the slave computing modules are connected to opaque the ports of other interface cards that are connected to the switch, the switch is configured to implement the function when you turn on the slave computing modules for initializing the PCIe bus and the slaves connected to it x computing modules with allocation in the opaque port of the corresponding interface board of the address space for each of them in the RAM, and when the host computing module for initializing the PCIe bus and the leading computing module connected to it with allocation in the transparent port of the corresponding interface board of the address space for them in RAM, and the switch is configured to form in the address space of the PCIe bus the leading computing module of the address range ow, covering all access windows to the slave computing modules, and forming, in each opaque port of the interface board, a window with an address in the address area in the master computing module to provide each slave computing module with access to the memory of other slave computing modules through the address space.

These signs are significant and interconnected with the formation of a stable set of essential features sufficient to obtain the desired technical result.

The present invention is illustrated by specific examples of execution, which, however, are not the only possible, but clearly demonstrate the ability to achieve the desired technical result.

Figure 1 - topology of PCI Express;

figure 2 - Multi-Root using NT Bridge;

figure 3 - PCI Express switch with transparent and opaque ports;

4 is a General block diagram of a communication environment;

5 is a General block diagram of a switch;

6 is a block diagram of an interface board;

7 is a block diagram of a PCIe switch (PCIe Switch) IP with transparent and opaque ports;

Fig is a block diagram of a computing cluster of a computer MVS-Express.

According to the present invention, a cluster system is constructed based on direct PCI Express channel switching technology, which implements at the hardware level a common field of large-capacity memories and external devices for all nodes in the cluster. At the same time, the levels of speeds and delays in transmitting data in such a network are comparable to those for data transfer within a computer. So, the delay time for a single record of data in a "foreign" memory is reduced from 3-6 microseconds (for modern clusters) to 200-400 nanoseconds. The data transfer speed will range from 600 MB to 1.5 GB per second, depending on the width of the cables used. The length of the data packet at which peak speed is reached will be reduced from a few kilobytes to tens of bytes. Thus, the prerequisites for building a supercomputer with an architecture similar to the architecture of the original Western supercomputers with a common memory field, but with indicators of availability and low unit cost, are the same as those of supercomputers built on the basis of cluster technologies.

The communication environment of the cluster system is based on the PCI Express (PCIe) computer bus standard (FIG. 1) and has a tree structure. The root of the tree is a special node - Root Complex 1 (from the point of view of the PCIe bus), let's call it Lead. The leaves of the tree are devices connected to the bus, such are the interface cards (IP) of all other computing modules of the VM (let's call them Slave) connected through the switch. The tree tops that are not leaves are commutators, i.e. devices capable of receiving a packet at some input, analyzing the destination address contained in the packet, and depending on what this address is equal to, issuing the packet to one or another output.

For a more convenient consideration of this communication environment, you can consider the PCIe bus, briefly describing its components and general terminology.

PCI Express (or PCIe, or PCI-E) is a computer bus using the software model of the PCI bus and a high-performance physical protocol based on serial data transfer. Unlike the PCI bus, which uses a common bus for data transfer, PCI Express, in general, is a packet network with a star topology, PCI Express devices communicate with each other through an environment formed by switches, with each device directly connected by a point-to-point connection. point with the switch.

The PCI Express (PCIe) bus standard identifies three types of devices. Root Complex I, PCI Switch 2 (PCI Switch), and Endpoint 3 (Endpoint or Device). Root Complex I is a separate subsystem processor that includes a single port or several PCIe ports, one or more CPUs (Central Processing Units), combined with RAM (RAM) and a memory controller, as well as other internal connections and / or bridge functions. Simply put. Root Complex - this device is the common ground of three interfaces: memory bus, PCI Express and processor bus.

PCIe routing is based on memory addressing or unique device numbers (IDs), depending on the type of transaction. Thus, each device (or function in the device) must be identified by a unique number on the PCIe tree bus.

During the initialization of the system, Root Complex performs registration to determine the various available buses, as well as the devices located on each bus, in addition, the address space of the registers and device memory is required. Root Complex allocates bus numbers on all PCIe buses and configures the bus numbers that will be used by the PCIe switch. PCIe behaves as if it were multi PCI-PCI Bridge 4 (Bridge) (Fig. 1, remote insert). Root Complex allocates and configures the memory and address space of the I / O ports for each PCIe switch and end device. The topology of the PCIe is shown in figure 1.

A PCI Express device uses a point-to-point bidirectional serial connection called lane. The connection between the two PCI Express devices is called link and consists of one (called 1x) or several (x2, x4, x8, x12, x16 and x32) lane bidirectional serial connections. Each device must support x1 connection.

PCI-PCI Bridge (RRV) 4 - a device with which you can connect additional buses to the local bus of the computer. Moreover, if, for example, the local bus of the computer is PCIe, then you can connect to it additionally both the same PCIe bus and, for example, the "old" PCI bus. There are various types of bridges. PCI Special Iterest Group (PCISIG) in 1994 developed the basic specifications for RVV bridge architectures. Thus, two main types of bridges are distinguished: “transparent” or “standard” and “opaque” or “built-in” bridges.

The transparent bridge (Transparent Bridge or TV) undergoes standard identification on the PCIe bus during the auto-configuration procedure, after which it becomes transparent with respect to the control processor; accordingly, Root Complex also initializes all devices that are behind this PPB bridge. This type of PPB does not contain any hardware resources, for example, direct memory access devices (DMA) or registers (mailboxes), which would require a separate device driver, and also does not translate addresses from one PCIe bus to another. Thus, the PCIe switch, which has one Upstream port and two Downstream ports, consists of three transparent bridges PPV 4 and a virtual PCI bus (figure 1).

An opaque bridge (Not-Transparent Bridge or NTB) is a bridge whose main task is to separate areas or, you can say, zones subject to scanning and auto-configuration by the local or primary side of the computer. In this case, an opaque bridge is configured, address windows are set, address translation and address register cards for local PCI devices in the built-in bridges from the primary (local) and secondary (backplane bus) sides are determined. The configuration range of local processors is limited by the built-in bridge, and devices on the backplane PCI bus are not searched. Thus, NTB can solve the problem of using multiple processors (figure 2).

Figure 2 shows Root Complex-1 and Root Complex-2, interconnected via a PCIe bus through NTB port 5. The task of NTB port 5 in this case is to shield the Address Domain-1 address space (item 6) from Address Domain-2 (item 7), thus, during auto-configuration of the bus, Root Complex-1 initializes local PCI devices to NTB, i.e. he sees NTB as an endpoint device (Endpoint or Internal Endpoint). Root Complex-2, for its part, also initializes its local PCI devices, also to NTB, and, accordingly, also sees NTB as an end device (Endpoint or External Endpoint). Figure 3 shows a more detailed diagram of the PCIe Bridge, where there are three ports, the upper (Upstream) is configured as a TV port, the lower left (Downstream) of a TV port and the lower right (Downstream) as an NTB port.

The use of opaque bridges also solves the problem of addressing conflicts through address translation. Based on figure 2, consider an example in which Root Complex-1 writes data via the PCIe bus from its local RAM to the RAM of Root Complex-2. The main obstacle is that they have different address spaces. PCIe is a packet network with a star topology. Therefore, in the header of the transmitted packet, you must specify the Requester ID (includes bus, device and function identifiers), Traffic Class identifier, address, routing information and a value that determines the length of this packet or packet to which it should respond requested device. Because the only point of contact between address spaces is NTB, then, respectively, when transferring a packet from Root Complex-1 to Root Complex-2, in the header of the transmitted packet you need to specify the address to which the Internal Endpoint corresponds (Fig. 3). Next, NTB itself translates the address and transfers the packet to the address space of Root Complex-2, where it is delivered to its destination.

A more detailed description of NT Bridge can be found at the following links:

1. http://www.google.com/url?sa=t&source=web&cd=1&ved=0CB0QFjAA&url=http%3A%2F%2Fwww.plxtech.com%2Fpdf%2Ftechnical%2Fexpresslane%2FNTB_Brief_Aprilct.jdf Non-Transporent% 20Bridging & ei = IR4sTYq0BsudOsSZpOQK & usg = AFQjCNHQ_gRT2izo9j9ZRHt1vs7BWgtmKw & cad = rja

2. http://www.eetimes.com/electronics-news/4139182/Non-Transparent-Bridging-Makes-PCI-Express-HA-Friendly

3. http://www.design-reuse.com/articles/8408/non-transparent-bridging-allows-multiprocessor-design-with-pci-express.html

4. http://www.google.com/url?sa=t&source=web&cd=2&ved=0CCYQFjAB&url=http%3A%2F%2Fwww.plxtech.com%2Fpdf%2Ftechnical%2Fexpresslane%2FNontransparentBridging.pdf&rct=jq Transporent% 20Bridging & ei = IR4sTYq0BsudOsSZpOQK & usg = AFQjCNHOjTiobXgalN_cjMmxs5tIHZaffQ & cad = rja

5. http://www.google.com/url?sa=t&source=web&cd=5&ved=0CEMQFjAE&url=ftp%3A%2F%2Fdownload.intel.nl%2Fdesign%2Fintarch%2FPAPERS%2F323328.pdf&rct=j&q=Non- Transporent% 20Bridging & ei = IR4sTYq0BsudOsSZpOQK & usg = AFQjCNGTjUE-PTFKXLwg9xSW1WTVTXxhRA & cad = rja

So, having examined the main components of the PCI Express bus, we can begin to describe the switching environment based on direct switching of PCI Express channels. This solution is an invention.

The general block diagram of a communication medium (CS) based on direct switching of PCIe bus channels is shown in Fig. 4.

COP consists of:

- switch 8;

- computing modules 9 (VM);

- interface cards 10 (IP), which are connected to the PCIe bus VM;

- network wires 11 (SP), with which IP 10 are connected to the switch 8.

The switch 8 (figure 5), which is a PCI Switch 2, performs the function of forwarding packets between the VMs according to the recipient address. From a logical point of view, all communications in the system belong to the point-to-point category and connect all devices directly.

A VM is a computer that has a PCIe bus on its motherboard.

IP 10 (Fig.6) multiport interface board having two ports, one 12 of which operates in the transparent port mode, the other 13 in the opaque port mode. The transparent port is used only to connect the Master (main Root Complex) to the switch. All other nodes (Slaves) use an opaque port to connect to the switch. All IP logic is embedded in the PCIe 2 switch (PCIe Switch in FIG. 6). A more detailed block diagram of the PCIe switch is shown in Fig.7. To organize the logic shown in Fig. 7, our own bit firmware is used. The firmware is loaded using the programmer. During the firmware process, the necessary registers are configured in the NT port - memory bar, the size of which will be equal to the size of the local portion. A local portion is a piece of system memory at a known fixed address that is cut out when the kernel of the operating system (OS) is loaded on the VM.

In this CS, the following order of node loading is required: first all the slaves, then the switch, and the last - the master node.

Turning on the slave VM 9: at the same time, when loading the BIOS on VM 9, the PCIe bus and all devices connected to it are initialized, as well as the address space is allocated for them. Further, during the boot process, the operating system (OS) (Linux is used as the OS) allocates address space for the PCIe bus and devices in RAM. It should be noted that the Internal Endpoint in the NTB IP (Fig. 7) is seen by the VM as an end device on its local PCIe bus.

Next, the switch is loaded by supplying power to it. At this stage, no configurations or memory allocations to any VMs occur, i.e. it just takes some time for the switch to become operational.

After that, power is supplied to the host VM. As in the slave VMs, the BIOS first initializes its bus and devices on it, but since Since the master VM is connected to the switch through a transparent port on its IP, it sees both the switch and the External Endpoint (Fig. 7) IP of all slave VMs. In other words, the Master sees them as end devices on their local bus, and, accordingly, the BIOS initializes its bus and devices, as well as the address space for them. Next, the OS allocates a space of addresses and input / output ports in its RAM. Now the host in the address space of the PCIe bus has N-1 (where N is the total number of all VMs) NT memory bars (extrenal endpoint IP), each of which is the size of a local portion. The master can configure each of these NT ports (slave access windows) to the local portion address in the corresponding slave node. Now, when working with this bar, the master will fall into the local portion of the corresponding slave.

Each of the slaves in the address space of the PCIe bus has 1 NT memory bar (Internal Endpoint), each of which is the size of a local portion. Each of the slaves can configure the corresponding NT port (access window to the master) to the local portion address in the master.

At this stage, the leader sees all the slaves, each slave sees the leader.

Now let the followers see each other. To do this, we select in the address space of the PCIe bus master the minimum naturally aligned range of addresses that covers all access windows to the slaves. Let's call it a cross-access area. We order in each NT port another NT memory bar (cross-access window) the size of this area and configure it to the address of this area in the master. Now each slave can see any other slave in the cross-access area through the cross-access window. With cross-access, the data does not physically go through the master — the master provided the slaves with only the address space, and the routing tables built in hardware at the initial start of PCI Express all drive along the shortest path.

Now any VM can directly access the memory of all slave VMs (the access mode used is Master DMA). We also note that through the use of basic software, reading from someone else's memory is carried out using a program request for a counter record.

The following is a description of an example of the execution of the MVS-Express computer cluster system using this communication medium and K-100 computer. On the example of the block diagram of the computer MVS-Express (Fig. 8), we consider the principle of the communication medium.

The MVS-Express computer contains the following hardware:

- 8 computing nodes, each of which has the following characteristics:

- Processor 2 x Opteron 2382; 7 cores available to the user’s task;

- RAM 16 Gb;

- SATA 320 Gb drive;

- Network card Gigibit Ethernet;

- Interface board;

- Video card nVidia GeForce 285GTX, 240 GPU;

- Server chassis ATX;

- Control machine:

- Processor Intel Core2Duo E8400;

- SATA 320Gb drive;

- DVD

- ATX housing;

- Network cards Gigabit Ethernet;

- Gigabit Ethernet Switch

- PCI Express Switch

The nodes, control machine and switches are located in 24U 600 × 800 cabinets.

The communication environment based on the PCI Express bus (or PCIe) in this block diagram (Fig. 8) consists of a PCI Express switch, interface cards that are inserted into the PCI Express slot in the computing nodes and network wires (standard 10 Gb Ethernet network wires ) for connecting compute nodes to a PCI Express switch. Because Since this communication environment is based on the PCI Express bus, which has a tree structure, you need to assign someone the root of this tree (or the main Root). Accordingly, one of the computing nodes is assigned by this root, we call it Leading. All other N-1 nodes are Slaves.

The PCIe switch has N-1 Downstream ports to which the Slave Node Interface Cards are connected and one Upstream port to which the Lead Node Interface Board is connected.

Each Interface Board has three ports, one of which is configured as an Downstream Not transparent bridge, the other as a Downstream Transparent bridge, and an Upstream port. Accordingly, the first slot is used when connecting the Slave nodes to the Downstream ports of the PCIe switch, the second when connecting the Lead node to the Upstream ports of the PCIe switch, and the third is inserted into the PCIe bus on the motherboard of the node. All interface boards are functionally identical, the only difference is the choice of the downstream port when connected to the switch.

Network connection for access from kiam.ru and internet networks.

Software

- Operating system SuSE Linux Enterprise Server 10SP2;

- User Management Passage Management System (CPSS);

- Compilers C, Fortran;

- Remote access server via SSH / SCP.

Key Features of a PCIe-based Communication Network:

- speed: up to 700 MB / s;

Latency: 1.2 μs

- word output time: ~ 70 ns;

- word reading time: ~ 2.5 μs.

The present invention is industrially applicable, functioning systems are created that are tested and showed high results, fully confirming the possibility of achieving a technical result.

Claims (1)

A cluster system containing a host computing module, which is a computer with a PCIe bus on its motherboard, connected via an interface board connected to the PCIe bus of the host computing module, with a switch whose outputs are connected to slave computing modules made with a PCIe bus on their motherboards boards, through interface boards each connected to the PCIe bus of the corresponding slave computing module, each interface board is made with two ports, one of which is made with the ability to work in the transparent port mode, and the second in the opaque port mode, while the leading computing module is connected to the transparent port of one interface board that is connected to the switch, and the slave computing modules are connected to the opaque ports of other interface boards that are connected to the switch, the switch is configured to implement the function when the slave computing modules for initializing the PCIe bus and the slave computing modules connected to it are turned on with highlighting in opaque th port of the corresponding interface board address space for each of them in the RAM, and when you turn on the host computing module for initializing the PCIe bus and connected to it the host computing module with the allocation in the transparent port of the corresponding interface board address space for them in RAM, and the switch is made with the possibility of the formation in the address space of the PCIe bus of the leading computing module, an address range that covers all access windows to the slave computing module, and the formation in each opaque port of the interface board of a window with an address in the address area in the master computing module to provide each slave computing module with access through the address space to the memory of other slave computing modules.

Images