CN1285993C - Input/output processing unit of storage device - Google Patents

Abstract

The present invention discloses an input/output (IO) processing unit of a storage device, which at least comprises more than one CPU, more than one internal storage and more than one CPU reset module capable of operating independently, more than one bridging chip for protocol conversion, and a disk interface module connected with a storage unit of the storage device, wherein each CPU is connected with the internal storage and the independent CPU reset module; the CPUs are connected with the bridging chips through a high speed protocol channel which can support more than one CPU subsystem, two bridging chips are connected through the high speed protocol channel, and each bridging chip is connected with the disk interface module through a PCI bus. The IO processing unit enhances self-protection capability in the IO process of the storage device and reduces dead halt conditions, and thereby, system usability is improved. The IO processing unit further improves the IO processing speed of the storage device, and thus, the bandwidth bottle necks of the PCI bus and the like are avoided.

Description

Input and output processing device of storage device

technical field

The invention relates to input and output (IO) processing technology, in particular to an IO processing device of a storage device.

Background technique

With the rapid improvement of computer processing speed and storage technology, the IO processing capability of computer equipment has become the main factor affecting its performance and availability. Especially with the continuous expansion of the demand for storage capacity, large-scale storage devices such as disk arrays that can work independently have emerged. This type of storage device itself contains storage units and IO processing devices, which can provide storage for several servers and other devices at the same time. Space and IO services, so the quality of its IO processing device will directly affect the performance of the entire system.

Refer to FIG. 1 for a basic structure of a storage device IO processing device generally used in the industry at present. Including: CPU 101, north bridge chip 102, south bridge chip 103, memory 104 and several Fiber Channel (FC, Fiber Channel) modules 105 for connecting with storage devices such as disk arrays. Wherein, the FC module 105 includes an FC HBA and an FC interface, and the FC interface is used to connect a storage unit composed of a hard disk. The CPU is connected to the North Bridge chip 102 through the CPU front side bus, and the North Bridge chip 102 is connected to the RAM and the South Bridge chip 103 respectively, and all the FC modules 105 are connected to the South Bridge chip 103 through a PCI bus. This structure is similar to that of a PC. Its disadvantage is that it is difficult to improve system availability. Since all FC modules 105 are connected in series on the same PCI bus and processed by one CPU 101, there is no protection measure. Once any link fails , the system will stop working, and the possibility of system crash is high. In addition, the IO bandwidth bottleneck of this structure is very obvious. Since multiple FC interfaces share the same PCI bus bandwidth, because the PCI bandwidth is insufficient, the FC interface cannot reach full load, and the CPU 101, North Bridge chip 102, South Bridge chip 103 and The serial structure of the FC interface also easily causes insufficient IO and mid-end processing capabilities of the CPU 101, making it difficult to improve the system speed.

Contents of the invention

In view of this, the main purpose of the present invention is to provide an IO processing device for a storage device, which enhances the self-protection capability of the storage device during IO, reduces the occurrence of crashes, and thus improves the availability of the system. And further improve the IO processing speed of the storage device to avoid bandwidth bottlenecks such as PCI.

An input and output processing device for a storage device of the present invention at least includes: one or more CPUs that can work independently, memory equal to the number of the CPUs, and CPU reset modules equal to the number of the CPUs; more than one CPU reset module for protocol Converted bridge slices, and a disk interface module that is equal to the number of the bridge slices and is used to connect with the storage unit of the storage device; each CPU is connected with a memory and an independent CPU reset module; more than one CPU can be supported by More than one high-speed protocol channel of the CPU subsystem is connected with the bridge slices, each bridge slice is connected through the high-speed protocol channel, and each bridge slice is connected with the disk interface module through the PCI bus.

The high-speed protocol channel supporting more than one CPU subsystem described in the device is a hypertransport input and output interface channel, the CPU is a CPU with a hypertransport input and output interface, and the bridge slice can convert the hypertransport input and output interface protocol into a bridge slice of the PCI protocol.

The high-speed protocol channel supporting more than one CPU subsystem in the device is an IB interface channel, the bridge is composed of an IB switch and a TCA, the CPU is connected to the IB switch of the bridge through the HCA, and the TCA is connected to the disk interface module.

The CPU reset module of the device is a programmable logic array chip programmed with a reset dog program.

The disk interface module of the device includes an FC chip and an FC interface for fiber channel protocol processing, the FC chip is connected to the bridge through a PCI bus, and the FC interface is connected to the storage unit.

The disk interface module of the device includes an iSCSI chip for iSCSI protocol processing and an iSCSI interface, the iSCSI chip is connected to the bridge through a PCI bus, and the iSCSI interface is connected to the storage unit.

The memory of the device is random access memory.

It can be seen from the above scheme that the IO processing device of the storage device of the present invention adds a protection mechanism to the IO processing device by using two or more CPU subsystems that can work independently, thereby improving the availability of the system, and each The disk interface module independently occupies a PCI bus, which increases the bandwidth and effectively solves the PCI bottleneck problem. The high-speed transmission bus and multiple CPUs work together to greatly improve the IO processing speed of the IO processing device.

Description of drawings

1 is a schematic structural diagram of an existing storage device IO processing device;

Fig. 2 is the structural representation that the present invention adopts double CPU and four FC module embodiments;

Fig. 3 is a schematic diagram of the connection structure between the reset module and the CPU of the present invention;

Fig. 4 adopts double CPU and four FC module embodiment signal flow diagrams for the present invention;

Fig. 5 is the structural representation that the present invention adopts double CPU and double FC module embodiment;

Fig. 6 is the structural representation that the present invention adopts dual CPU and five FC module embodiments;

Fig. 7 is the structural representation of the embodiment of the present invention adopting three CPUs and six FC modules;

FIG. 8 is a schematic structural diagram of an embodiment of the present invention using dual CPUs and four iSCSI modules.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

The invention adopts two or more CPU subsystems, and each CPU is respectively connected with a group of disk interface modules such as FC modules through a high-speed protocol channel and a group of bridges for protocol conversion. When working normally, the CPU subsystems can work together, and each CPU can separately control a part of the bridge and the IO of the disk interface module on it. Moreover, the memory is shared among the CPUs. A CPU can regard the memory of other CPUs, the controlled bridge and the disk interface module as its own IO space. When a certain CPU fails and crashes, other CPUs will take over the work of the CPU, thereby improving the availability of the IO processing device of the storage device.

The structure of the storage device IO processing device of the preferred implementation scheme of the present invention is shown in Figure 2, adopting the structure of two CPUs 101, including: two CPUs 101, two internal memory 104, two CPU reset modules 201 working independently, and Four bridges 202 and four FC modules 105 . The structure is left-right symmetrical, and the four bridges 202 are connected in series through high-speed signal lines of the HyperTransport I/O Interface (HT, HyperTransport I/O Interface) to form a chain of bridges, and each bridge 202 is respectively connected to an FC module 105 through a PCI bus. . The two CPUs 101 are respectively located at the left and right ends of the bridge chain, and are connected to the bridges 202 at the left and right ends through HT high-speed signal lines. A memory 104 is respectively connected to the two CPUs 101, and a CPU reset module 201 for resetting the CPU 101 when it breaks down.

Between the CPU 101 and the bridge chip 202 in this embodiment, and between the bridge chip 202 and the bridge chip 202, the HT protocol is used to realize data interaction. HT is a high-speed channel protocol. This technology not only has the characteristics of high speed and high performance, but also provides a general connection for the system, reduces the number of internal buses in the system, and can be used to connect multiple relatively independent CPU subsystems and realize CPU subsystems. The system's memory 104 is shared.

Therefore, in order to support the HT protocol, it is necessary to use a CPU with its own HT interface. This type of CPU includes, for example, BROADCOM BCM 1250, MIPS1125, MIPS1280 based on MIPS core, and the Hummer series based on X86.

In the present embodiment, CPU 101 adopts Broadcom BCM 1250, and this CPU 101 has two 64-bit MIPS cores of 600MHz～1GHz, and two 64-bit double data rate random access memory (DDR RAM) channels of the highest 200MHz (400Mbit/s) are arranged , the maximum RAM bandwidth is 6.4GB, which can be directly connected to DDR RAM. And the CPU 101 provides a 12.8Gb high-speed LDT HyperTransport interface with a bandwidth of 400M×2×2×8=12.8Gbit/s, which can be directly connected to the bridge chip 202. In addition, the Generic I/O interface and the Reset pin provided on the BROADCOM BCM 1250 can be connected to the CPU reset module 201 of this embodiment.

The memory 104 of the present embodiment adopts DDR RAM.

The function of the bridge 202 in this embodiment is mainly used for the protocol conversion between HT to PCI, adopting the three-way bridge 202 that provides the HT interface, this type of bridge 202 includes two HT interfaces and a PCI interface, three The two interfaces must be able to exchange data with each other, so that the CPU 101 can read and write the registers of the chips on the PCI bus through the bridge chip 202 in an IO space read and write mode. This embodiment adopts the HyperTransport PCI bridge system of API Company. The bridge 202 provides two HT interfaces of 8-bit 400M DDR on the left and right, which can be used for series connection of the bridge 202, and provides a 66M×64-bit PCI interface, which can be passed through the PCI bus. Connect the FC module 105. Moreover, the chip can realize an opaque bridge between the PCI interface and the HT interface, thereby avoiding data crosstalk between the PCI interface and the HT interface.

The FC module 105 is composed of an FC chip and an FC interface in this embodiment. The function of the FC chip is mainly to complete the protocol processing of FC, and the protocol layers processed include FC-0, FC-1, FC-2, FC-3, and FC-4. In this embodiment, the FC chip can use LSI919, which has an embedded ARM processing core for processing the FC protocol, and the internal DMA module can directly send data to the memory 104, and integrates the Serders function of 2G FC inside, directly providing one way two-way 2.125 Differential signal, connected with SFP optical module, externally provides 66MHz×64-bit PCI interface, just can be used with HyperTransport PCI bridge system bridge chip 202.

In the present embodiment, the CPU reset circuit is referring to shown in Figure 3, adopts two sets of independent logic reset dogs (Watchdog) that a programmable logic array chip (FPGA) writes, and each reset dog is reset module 201 and each CPU 101 respectively The Generic I/O interface is connected to the Reset pin, and the two sets of reset logic work independently without interfering with each other. When working, the reset dog of the FPGA outputs a reset signal to the Reset pin of its corresponding CPU 101 every long period of time to reset the CPU 101; the CPU 101 clears the FPGA through the Generic I/O interface every short period of time. The reset dog inside is about to clear the count of the reset dog. Therefore when the CPU 101 has not cleared the dog for a long enough time, the reset dog will reset the CPU 101. Wherein, the time interval for resetting the CPU 101 by the reset dog and the time interval for clearing the dog by the CPU 101 can be set arbitrarily. For example: in this embodiment, CPU 101 can be set to clear the reset dog every 10 milliseconds, and the reset dog resets the CPU 101 every 500 milliseconds, so that when the CPU 101 has not cleared the reset dog for 500 milliseconds in a row, the reset dog of FPGA will output a reset signal to the Reset pin of the CPU 101 to reset the CPU 101. In addition, it takes about 30 seconds for the CPU 101 to start and initialize. In order to prevent the reset dog from sending a reset command to the Reset pin of the CPU 101 during this period, you can set the time interval between the reset dog to send the reset command twice. 1 minute, so as to allow enough time for the CPU 101 to complete the startup and initialization operations.

The following describes the workflow of the storage device IO processing device in this embodiment:

During the power-on initialization process, the left and right CPUs are powered on and self-tested at the same time.

After the left CPU is powered on, it will detect the existence of 4 bridges by accessing the IO address. But the CPU only sets the driver for the left 1 and left 2 bridges, and initializes the registers of the left 1 and left 2 bridges. Then the left CPU detects the left 1FC chip and the left 2FC chip connected to the left 1 and left 2 bridges, and initializes the two FC chips.

The left CPU accesses the physical address space corresponding to the right CPU, detects whether the CPU exists, and if it exists, sets a logical address mapping for the right CPU, and sets a memory area as a communication space with the right CPU. Then the left CPU finishes initialization and starts working normally.

If during the initialization process, the left CPU finds that the right CPU is not in place, for example: crash, damage or unprocessed reasons, then the left CPU initializes the left 3 and left 4 bridges, and the left 3 and left 4 FC chips respectively. Finish initialization and start working normally. At this time, the left CPU independently controls all bridges and FC chips at the same time.

After the left CPU works normally, the left CPU detects the presence of the right CPU at intervals, such as 1 second. The detection method can be to set a space in the communication register of the right CPU to save a number, arrange the right CPU to add 1 to the number every once in a while, such as 1 millisecond, and the left CPU checks the value of the number every 1 second Changes, if the number has not changed compared with the last check, it means that the right CPU may not be in place due to a crash or other reasons.

After the right CPU is powered on, it detects the existence of 4 bridges by accessing the IO address.

Wait for a period of time, such as 1 minute, for the left CPU to complete the initialization process.

Then the right CPU accesses the corresponding physical address space of the left CPU, detects whether the CPU exists, if exists, sets logical address mapping for the left CPU, and sets a memory area as a communication space with the left CPU. At this time, both CPUs are working normally. Through communication, the left CPU will assign the left 3 and left 4 bridges to the right CPU for management, and the right CPU will initialize the left 3 and left 4 bridges and the left 3 and left 4 chips. After initialization, it starts working normally.

If the right CPU finds that the left CPU is not in place during the initialization process, for example: crash, damage or unprocessed reasons, the right CPU will respectively initialize the left 1, left 2, left 3, left 4 bridges and left 1, left 2, Left 3, left 4 chips. After initialization, normal operation begins. At this time, the right CPU independently controls all bridges and FC chips at the same time.

After the right CPU works normally, the same as the left CPU, the right CPU detects the presence of the left CPU every 1 second.

Wherein, in the initialization process of the two CPUs described in this embodiment, each part of the memory is set to allow the other party to read and write. From the perspective of the CPU on one side, a part of the memory on the other side of the CPU and the FC chip are readable and writable. IO space, so using the HT protocol, the two CPUs communicate by reading and writing each other's part of the memory space. In addition, it is also possible to read and write the same register inside the same FPGA for communication, serial port communication or Ethernet communication.

Refer to the data flow direction of the IO processing device of the two CPUs 101 shown in Figure 4 in normal working condition:

Two CPUs 101 work in the multiprocessor (MP) mode, respectively run their respective operating systems, such as: Linux program, two CPUs 101 can simultaneously access 4 bridges 202 and the FC chips hanging on the bridges 202. Under normal circumstances, the CPU 101 on the left only processes the services of FC interface 1 and FC interface 2, and the right CPU 101 only processes the services of FC interface 3 and FC interface 4. In this way, the PCI bandwidth bottleneck at the FC interface is effectively overcome, the bandwidth is increased, and the IO processing speed of the storage device is greatly improved.

FC ports 1 and 4 are forward ports of the system, and are generally connected to FC ports of computer devices such as servers in a storage network. FC ports 2 and 3 are system backward ports, and are generally connected to FC hard disk modules, that is, storage units.

Taking the left half of Figure 4 as an example, the server’s access request first reaches the FC chip through FC interface 1, and the FC chip LSI919 is embedded with an ARM processor core. The PCI burst communication mode reaches the DDR RAM memory 104 through the bridge chip 202 and through the CPU 101. Then CPU 101 processes the access request in internal memory 104, and the processing content includes: judging the legitimacy of request, querying buffer memory (Cache) hit situation, starting to read data request etc. from hard disk, the data after processing is placed in internal memory 104 Then, the FC chip of FC interface 2 takes the data into the chip in DMA mode, and then sends the data to the hard disk module interconnected with FC interface 2.

The two CPUs 101 of the IO processing device are all connected to the two reset dogs of the same FPGA. When a failure such as a crash occurs in the left CPU subsystem, the communication between the two CPUs 101 is interrupted, and the right CPU 101 can pass The detection once per second knows that the left CPU 101 is in place. After detecting that the left CPU 101 is not in place, the right CPU 101 reconfigures the registers of the FC chips of FC interface 1 and FC interface 2, so that the 4 FC chips send all service accesses to the right CPU subsystem for processing. Although the processing capacity of the IO processing device is reduced by about 50%, the bandwidth of the IO processing device is not reduced, and each port can still work, thereby greatly improving the availability of the system.

Due to the failure of the left CPU 101, when the left reset dog of the FPGA is not cleared for 500 milliseconds in succession, the left reset dog will reset the left CPU 101. When the left side CPU 101 resets and works normally again, the FC interface 1 and FC interface 2 are taken over by the left side CPU 101 again, and return to the normal working state.

In this embodiment, the IO processing device adopts two independently working CPU subsystems, which provides double protection for the IO processing of the system, and improves the IO processing capacity by more than two times. In addition, when dual CPUs are used, it can also support two or more FC modules of other numbers.

Referring to FIG. 5, it is a schematic structural diagram of two CPUs 101 supporting two FC modules 105. In this case, one FC interface is the forward interface of the system and is connected to the server in the storage network, and the other FC interface is the backward interface of the system and is connected to the storage unit. Two CPUs 101, one CPU 101 works normally, and the other CPU 101 is in a standby state, and detects the presence of the working CPU 101 in real time, once it detects that the working CPU 101 is not in place, it will immediately take over its work. After the original working CPU 101 is reset, it can be in the standby state, and detect the presence of another CPU 101; it can also take over the work of the current CPU 101, and the current CPU 101 still returns to the standby state.

Referring to FIG. 6, it is a schematic structural diagram of two CPUs 101 supporting five FC modules 105. In this case, two of the five FC modules 105 are system forward interfaces and are connected to the server, and the other three FC modules 105 are used as system backward interfaces to connect to the storage unit. During normal operation, two FC modules 105 can be assigned to one CPU 101, and three FC modules 105 can be assigned to the other CPU 101. When one CPU 101 is not in place due to a fault, the other CPU 101 will quickly control the FC module controlled by the faulty CPU 101. 105 and the bridge chip 202 are initialized, and take over the work of the CPU 101 until the reset of the faulty CPU 101 is completed.

The present invention can also adopt more than two CPU subsystem structures. Referring to FIG. 7 , this is a schematic structural diagram when three CPUs 101 are used to support six FC bridges 202. Each CPU 101 supports two bridge slices 202 and FC modules 105, so that three CPU subsystems can be formed, and a HT switch (Switch) chip 701 is further used to connect the end bridges of the three CPU subsystems through HT high-speed signal lines Slice 202 implements communication among the three CPU subsystems. Priorities can be set for the three CPUs 101. During normal operation, each of the three CPUs 101 controls a part of the bridge 202 and the FC module 105 belonging to itself. The three CPUs 101 work together, and one CPU 101 can use the memory 104 and all other CPUs 101. The controlled bridge slice 202 and the FC module 105 are regarded as their own IO space. When a CPU 101 breaks down and is not in place, the other two CPUs 101 select one to take over the work of the faulty CPU 101 according to the priority order set in advance. If two CPUs 101 are not in place at the same time, the remaining CPU 101 The work of the two CPUs 101 will be replaced. At this time, the IO processing capacity of the IO processing device is only about 30% of the original, but the device bandwidth is not reduced. Any FC interface can still perform IO processing, and the system is still available. After faulty CPU 101 resets, then can recover normal working condition.

When four or more CPUs are used, it is similar to the case of three CPUs.

The FC module mentioned in the present invention may be an electrical interface module, or may refer to an optical interface module. In addition, the present invention can also adopt the iSCSI interface supporting the iSCSI protocol, and use the iSCSI module composed of the iSCSI chip and the iSCSI interface as the disk interface module of the present invention. See Figure 8, which is a schematic diagram of the structure of an IO processing device using four disk interface modules and dual CPUs 101. It is basically the same as the structure of the embodiment shown in Figure 2, except that the iSCSI module 801 in Figure 2 is replaced. The FC module 105 is shown.

In addition, other types of high-speed protocols that can support two or more CPU subsystems can be used for data interaction between the CPU and the bridge chip, and between the bridge chip and the bridge chip.

For example: IB (InfiniBand) interface protocol can be used, which can support multiple CPU subsystems and realize memory sharing. When the IB protocol is used, there is no restriction on the interface type of the CPU, but it needs to be used with IB devices. The combination of CPU and HCA can be used to replace the CPU with HT interface in the case of HT protocol; the combination of IB protocol switch (IB Switch) and TCA can be used to replace the bridge chip in the case of HT protocol, the overall structure of the IO processing device and other chip types Neither change. Among them, HCA, IB Switch and TCA are all IB devices, HCA is used for the connection between the CPU and the IB protocol channel, such chips include Intel's 82808AA chip, etc. The bridge chip to the PCI interface can use chips such as BDC22104 and MT21108. This structure using the IB protocol can achieve the same effect as the case of the HT protocol.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (7)

1, a kind of input and output processing unit of memory device is characterized in that, comprises at least:

The CPU that can work independently more than one, internal memory that equates with described CPU number and the cpu reset module that equates with described CPU number;

Be used to carry out the bridge sheet of protocol conversion more than one, with the disk interface module that is connected with the storage unit of memory device with described bridge sheet number being used for of equating;

Be connected with internal memory and independent CPUs reseting module on each CPU;

The above CPU is connected with the bridge sheet by the high speed protocol passage that can support an above cpu subsystem, connect by described high speed protocol passage between each bridge sheet, and each bridge sheet is connected with the disk interface module by pci bus.

2, according to the described input and output processing unit of claim 1, it is characterized in that, the high speed protocol passage of an above cpu subsystem of described support is the super IO interface passage that transmits, described CPU has the super CPU that transmits IO interface, and described bridge sheet is to transmit the bridge sheet that the IO interface protocol conversion becomes the PCI agreement with surpassing.

3, according to the described input and output processing unit of claim 1, it is characterized in that, the high speed protocol passage of an above cpu subsystem of described support is the IB interface channel, described bridge sheet is made up of IB switch and TCA, CPU is connected with the IB switch of bridge sheet by HCA, and TCA is connected with the disk interface module.

According to the described input and output processing unit of claim 1, it is characterized in that 4, described cpu reset module is to write the programmable logic array chip of the dog program that resets.

5, according to the described input and output processing unit of claim 1, it is characterized in that, described disk interface module includes and is used for FC chip and the FC interface that fiber channel protocol is handled, and the FC chip is connected with described bridge sheet by pci bus, and the FC interface is connected to described storage unit.

6, according to the described input and output processing unit of claim 1, it is characterized in that, described disk interface module includes iSCSI chip and the iSCSI interface that is used for the iSCSI protocol processes, and the iSCSI chip is connected with described bridge sheet by pci bus, and the iSCSI interface is connected to described storage unit.

According to the described input and output processing unit of claim 1, it is characterized in that 7, described internal memory is a random access memory.

Images