CN114428703A - IO fault location method, apparatus, device and computer readable storage medium - Google Patents

Abstract

The embodiment of the invention relates to the technical field of distributed storage, and discloses an IO fault positioning method, device, equipment and a computer readable storage medium. The method comprises the following steps: obtaining IO time delay of a front-end computing node and an IO time delay threshold of an external disk; if the difference value between the IO time delay of the computing node and the IO time delay threshold of the external disk is within a preset numerical range, the IO time delay of the logic unit number and the IO time delay threshold of the logic unit number are obtained; if the IO time delay of the logic unit number is not larger than the IO time delay threshold value, and the difference value between the IO time delay of the calculation node and the IO time delay of the logic unit number is within a preset value range, the IO fault is positioned at the front end; and if the difference value between the IO time delay of the logic unit number and the IO time delay threshold value is within the preset value range, and the difference value between the IO time delay of the calculation node and the IO time delay of the logic unit number is within the preset value range, positioning the IO fault at the rear end. Through the mode, the embodiment of the invention realizes the rapid and accurate positioning of the IO fault in the distributed storage.

Description

IO fault location method, apparatus, device and computer readable storage medium

technical field

Embodiments of the present invention relate to the technical field of distributed storage, and in particular, to an IO fault location method, apparatus, device, and computer-readable storage medium.

Background technique

Distributed storage is a popular storage method at present. It interconnects multiple storage servers through a network to flexibly allocate storage space, which can improve storage efficiency and storage capacity.

A distributed storage system generally includes front-end computing nodes and back-end storage nodes. Various applications can run on the front-end computing nodes, and the back-end storage nodes can save the data of the front-end computing nodes. During the operation of a distributed storage system, IO failures often occur. In order to make the distributed storage system run normally, it is necessary to locate the IO fault to solve the IO fault. In the process of implementing the embodiments of the present invention, the inventor found that locating IO faults in the related art often relies on the alarm information provided by the distributed storage system and the experience of technicians, and the locating means for IO faults is relatively simple, and the locating process is relatively simple. less efficient.

SUMMARY OF THE INVENTION

In view of the above problems, embodiments of the present invention provide an IO fault location method, apparatus, device, and computer-readable storage medium, which are used to solve the problem of low efficiency in the distributed storage IO fault location process in the prior art.

According to an aspect of the embodiments of the present invention, a method for locating an IO fault is provided, where the IO fault is an IO fault of distributed storage, and the method includes:

Obtain the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node;

If the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within the first preset value range, further obtain the IO delay of the logical unit number corresponding to the computing node and all the IO delay threshold of the logical unit number;

If the IO latency of the logical unit number is not greater than the IO latency threshold of the logical unit number, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within the second preset value Within the value range, the IO fault is located at the front end;

If the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within a third preset value range, and the IO delay of the computing node and the IO delay of the logical unit number If the difference between the delays is within the fourth preset value range, the IO fault is located at the back end.

In an optional manner, after locating the IO fault at the front end, the method further includes:

If the computing node is a virtual machine, obtain the IO delay of other computing nodes corresponding to the same logical unit number as the computing node and the IO delay of other virtual machines corresponding to the same physical machine as the computing node, respectively. and the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located;

If it is determined that the IO delay of the other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

If it is judged that the IO delay of the other virtual machines corresponding to the same physical machine as the computing node is normal, and the IO of the other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is judged to be normal If the delay is normal, the IO fault is located on the computing node;

If it is determined that the IO delay of the other virtual machines corresponding to the same physical machine as the computing node is abnormal, and the IO of the other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is determined to be abnormal If the delay is normal, the IO fault is located on the physical machine where the computing node is located.

In an optional manner, after locating the IO fault at the front end, the method further includes:

If the computing node is a physical machine, obtain the IO delay of other computing nodes corresponding to the same logical unit number as the computing node;

If it is determined that the IO delay of the other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

If it is determined that the IO delay of the other computing nodes corresponding to the same logical unit number as the computing node is normal, the IO fault is located on the computing node.

In an optional manner, after locating the IO fault at the backend, the method further includes:

Obtain the ping network delay of each back-end storage node, the average ping network delay of all back-end storage nodes, and the ping network delay threshold of the back-end storage nodes;

If the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the fifth preset value range, and each back-end storage node If the difference between the ping network delay of the back-end storage node and the ping network delay threshold of the back-end storage node is within the sixth preset value range, the IO fault is located in the interconnection network between the back-end storage nodes;

If the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the seventh preset value range, and the preset number of back-end storage nodes The difference between the ping network delay of the back-end storage node and the ping network delay threshold of the back-end storage node is within the eighth preset value range, and the back-end storage nodes other than the preset number of back-end storage nodes have If the ping network delay is not greater than the ping network delay threshold of the back-end storage node, the IO fault is located in the network of the preset number of back-end storage nodes;

If the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the ninth preset value range, obtain all the back-end storage nodes. The IO delay of the built-in disk of the storage node and the IO delay threshold of the built-in disk corresponding to the IO delay of the built-in disk will make the difference between the IO delay of the built-in disk and the IO delay threshold of the built-in disk. The built-in disk whose difference is within the tenth preset value range is identified as an abnormal built-in disk, and the IO fault is located in the abnormal built-in disk.

According to another aspect of the embodiments of the present invention, an apparatus for locating an IO fault is provided, where the IO fault is an IO fault of distributed storage, and the apparatus includes:

a first obtaining module, configured to obtain the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node;

The second obtaining module is configured to obtain the logical unit number corresponding to the computing node if the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within a first preset value range The IO delay and the IO delay threshold of the logical unit number;

A positioning module, used for if the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number Within the second preset value range, the IO fault is located at the front end; if the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within the third preset value range and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within a fourth preset value range, the IO fault is located at the back end.

According to another aspect of the embodiments of the present invention, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete each other through the communication bus. communication between;

The memory is used for storing at least one executable instruction, and the executable instruction enables the processor to perform the operations of the above-mentioned IO fault location method.

According to another aspect of the embodiments of the present invention, an IO fault location device is provided, where the IO fault is an IO fault of distributed storage, and the device includes:

The acquisition module is used to collect IO delay information;

The configuration module is used to configure the IO delay threshold;

a locating module, configured to execute the above-mentioned IO fault locating method, so as to locate the IO fault;

A display module, configured to obtain a result of the positioning module locating the IO fault, and display the result.

In an optional way, the display module is also used for:

Display the IO delay information collected by the collection module, and display the correspondence between the external disk of the front-end computing node and the logical unit number of the back-end storage cluster.

In an optional manner, the configuration module is further used to:

Obtain the configuration information of the front-end computing node and the configuration information of the back-end storage cluster, and send the configuration information of the front-end computing node and the configuration information of the back-end storage cluster to the positioning module;

The locating module is further configured to: locate the IO fault according to the configuration information of the front-end computing node and the configuration information of the back-end storage cluster.

According to yet another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, where at least one executable instruction is stored in the storage medium, and when the executable instruction is executed on a computing device, the computing device executes the above The operation of the IO fault location method.

In this embodiment of the present invention, the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node are obtained; if the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within Within the first preset value range, the IO delay of the logical unit number corresponding to the computing node and the IO delay threshold of the logical unit number are further obtained; if the IO delay of the logical unit number is not greater than all The IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the second preset value range, the IO fault is located at the front end ; If the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within the third preset value range, and the IO delay of the computing node and the logical unit number If the difference between the IO delays is within the fourth preset value range, the IO fault is located at the back end. It can be seen that the embodiment of the present invention can quickly and accurately locate the IO fault by comparing the IO delay of the front-end computing node and the IO delay of the back-end logical unit number.

The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to understand the technical means of the embodiments of the present invention more clearly, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and The advantages can be more clearly understood, and the following specific embodiments of the present invention are given.

Description of drawings

The drawings are only used to illustrate the embodiments and are not considered to be limiting of the present invention. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:

1 shows a schematic flowchart of a method for locating an IO fault provided by an embodiment of the present invention;

FIG. 2 shows a schematic structural diagram of an IO fault location device provided by an embodiment of the present invention;

FIG. 3 shows a schematic structural diagram of a computing device provided by an embodiment of the present invention;

FIG. 4 shows a schematic structural diagram of an IO fault location device provided by an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited by the embodiments set forth herein.

FIG. 1 shows a flowchart of an embodiment of a method for locating an IO fault according to the present invention, and the method is executed by a computing device. In this embodiment of the present invention, an executable instruction is stored in the storage space of the computing device, and the executable instruction can cause the processor to execute the method for locating an IO fault. As shown in Figure 1, the method can locate the IO fault of distributed storage, including the following steps:

Step 110: Obtain the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node.

The front-end computing nodes may be various types of PC servers, for example, the front-end computing nodes may be x86 servers. The external disk of the computing node is located on the storage node on the back-end storage cluster. The back-end storage cluster generally includes multiple storage nodes. Each storage node can include at least one local storage disk. external disk. When the external disks of the compute node perform IO tasks, an IO delay will occur. You can set the IO delay threshold for the external disks of the compute node, and set the IO delay threshold of the external disks of the compute node and the IO delay threshold of the external disks of the compute node. The comparison is made to determine whether the external disk of the computing node times out during the execution of the IO task.

In an implementation manner of the embodiment of the present invention, the read and write delay await value of the external disk of the computing node can be obtained through a relevant operating system command, and the obtained await value is used as the IO delay of the computing node, and the relevant operating system command For example, it can be the iostat command. Further, multiple read and write delay await values of the external disks of the computing node can be obtained at fixed time intervals, the moving average of the await values can be obtained according to the moving average method, and the moving average of the await values can be used as the computing node's moving average. IO delay. The moving average method is a common method for processing data, which can effectively eliminate abnormal sample data, make the moving average of the await value more accurate, and make the IO delay of the final computing node more accurate.

The distributed storage includes a front-end and a back-end, the front-end includes multiple front-end computing nodes, the back-end includes multiple back-end storage nodes, and the front-end and the back-end communicate through the Internet. The front-end computing nodes can run applications, and the storage disks on the back-end storage nodes can save the application data of the applications on the front-end computing nodes. The back-end storage nodes can communicate through the internet between the back-end storage nodes. Each back-end storage node is connected to the internet between the back-end storage nodes through the local network access device. The network access device may include, for example, a network cable and a network card. The IO delay of a computing node can usually include the delay from the computing node to the front-end and back-end interconnection network, the delay from the front-end and back-end interconnection network to the storage node, and the internal delay of the storage node. The internal delay of the storage node usually also It can include the delay of IO data synchronization between storage nodes.

Step 120: If the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within the first preset value range, further obtain the IO time of the logical unit number corresponding to the computing node. delay and the IO delay threshold of the logical unit number.

Among them, the IO delay of the logical unit number can be calculated by the moving average method. First, the difference between the IO delay of the computing node and the IO delay threshold of the external disk can be calculated, and then it can be judged whether the difference is within the first preset value range. If it is determined that the difference is within the first preset value range, it is determined that the IO delay of the computing node is abnormal, and the distributed storage system has an IO failure. The first preset value range can be set according to actual needs. For example, the first preset value range can be set to a value range that is more than 10 times the IO delay threshold of the external disk, that is, when the IO delay of the computing node is greater than that of the external disk. When the IO delay threshold is 11 times, it is determined that the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within the first preset value range.

Among them, the logical unit number (LUN, namely Logical Unit Number) is the storage unit assigned to the front-end computing node by the back-end storage cluster. The back-end storage cluster generally includes multiple logical unit numbers, and the logical unit number of the back-end storage cluster is executing IO The IO delay will be generated during the task. You can set the IO delay threshold for the logical unit number of the backend storage cluster, and compare the IO delay of the logical unit number with the IO delay threshold of the logical unit number to determine that the logical unit number is executing. Whether to time out during the IO task.

Step 130: If the IO latency of the logical unit number is not greater than the IO latency threshold of the logical unit number, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within the first Within the range of two preset values, the IO fault is located at the front end.

Wherein, if it is determined that the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, it is determined that the IO delay of the logical unit number is normal. If it is further judged that the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the second preset value range, it means that an IO fault occurs at the front end, and the IO fault is located at the front end.

The second preset value range can be set according to actual needs. For example, the second preset value range can be set to a value range that is more than 10 times the IO delay of the logical unit number, that is, when the IO delay of the computing node is greater than the logic When the IO delay of the unit number is 11 times, it is determined that the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the second preset value range.

In a preferred implementation of the embodiment of the present invention, after the IO fault is located at the front end in step 130, the method further includes:

Step 131: If the computing node is a virtual machine, obtain the IO delay of other computing nodes corresponding to the same logical unit number as the computing node, the IO delay of other virtual machines corresponding to the same physical machine as the computing node, and the The physical machine where the node is located corresponds to the IO delay of other physical machines with the same logical unit number; if it is judged that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, locate the IO fault at the front end and the back end. If it is judged that the IO delay of other virtual machines corresponding to the same physical machine as the computing node is normal, and the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, then Locate the IO fault on the computing node; if it is judged that the IO delay of other virtual machines corresponding to the same physical machine as the computing node is abnormal, and the IO of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is judged to be abnormal. If the delay is normal, the IO fault is located on the physical machine where the computing node is located.

Wherein, after locating the IO fault at the front end in step 130, it may be further determined whether the computing node is a virtual machine or a physical machine, and if it is determined that the computing node is a virtual machine, step 131 is performed. IO delay of other computing nodes corresponding to the same logical unit number as the computing node, IO delay of other virtual machines corresponding to the same physical machine as the computing node, and other virtual machines corresponding to the same logical unit number as the physical machine where the computing node resides The IO delay of the physical machine can be calculated by the moving average method. After obtaining the IO delay of other computing nodes corresponding to the same logical unit number as the computing node, if it is judged that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, locate the IO fault in the front-end and Backend Internet. When judging whether the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO delay of each other computing node can be compared with the corresponding IO delay threshold, which will make the IO delay greater than the IO delay. The computing node with the delay threshold is identified as the computing node with abnormal IO delay, and the proportion of computing nodes with abnormal IO delay among all other computing nodes is counted. If the proportion of computing nodes with abnormal IO delay among all other computing nodes If the preset ratio is exceeded, it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal.

Among them, after obtaining the IO delay of other virtual machines corresponding to the same physical machine as the computing node and the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located, if it is determined that the computing node corresponds to If the IO delay of other virtual machines on the same physical machine is normal, and it is judged that the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO fault is located on the computing node. If it is judged that the IO delay of other virtual machines corresponding to the same physical machine as the computing node is abnormal, and it is judged that the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, locate the IO fault. on the physical machine where the computing node is located. When judging whether the IO delay of other virtual machines corresponding to the same physical machine as the computing node is normal, the IO delay of each other virtual machine can be compared with the corresponding IO delay threshold, which will make the IO delay greater than the IO delay The virtual machine with the threshold value is identified as a virtual machine with abnormal IO delay. If there is no virtual machine with abnormal IO delay among all other virtual machines, it is judged that the IO delay of other virtual machines corresponding to the same physical machine as the computing node is normal. If there is a virtual machine with abnormal IO delay among all other virtual machines, it is determined that the IO delay of other virtual machines corresponding to the same physical machine as the computing node is abnormal. When judging whether the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, the IO delay of each other physical machine can be compared with the corresponding IO delay threshold, which will make the IO delay A physical machine with a delay greater than the IO delay threshold is identified as a physical machine with an abnormal IO delay. If there is no physical machine with an abnormal IO delay among all other physical machines, it is determined that the physical machine where the computing node is located corresponds to the same logical unit number. The IO delay of other physical machines is normal. If there is a physical machine with abnormal IO delay among all other physical machines, it is judged that the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is abnormal.

Step 132: If the computing node is a physical machine, obtain the IO delay of other computing nodes corresponding to the same logical unit number as the computing node; if it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal , the IO fault is located on the front-end and back-end interconnected networks; if it is judged that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is normal, the IO fault is located on the computing node.

Wherein, after locating the IO fault at the front end in step 130, it may be further determined whether the computing node is a virtual machine or a physical machine, and if it is determined that the computing node is a physical machine, then step 132 is performed. The IO delay of other computing nodes corresponding to the same logical unit number as the computing node can be calculated by the moving average method. After obtaining the IO delay of other computing nodes corresponding to the same logical unit number as the computing node, if it is judged that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, locate the IO fault in the front-end and The back-end interconnection network; if it is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is normal, the IO fault is located on the computing node. When judging whether the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is normal, the IO delay of each other computing node can be compared with the IO delay threshold, so that the IO delay is greater than the IO delay A computing node with a delay threshold is identified as a computing node with abnormal IO delay. If a preset proportion of the computing node IO delays of all other computing nodes corresponding to the same logical unit number as the computing node are abnormal, it is determined that the IO delays of other computing nodes corresponding to the same logical unit number are abnormal, otherwise, It is determined that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is normal, and the preset ratio may be, for example, 90%. Other computing nodes can be virtual machines or physical machines.

Step 140: If the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within the third preset value range, and the difference between the IO delay of the node and the IO delay of the logical unit number is calculated. If the difference is within the fourth preset value range, the IO fault is located at the back end.

Wherein, if it is determined that the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within the third preset value range, it is determined that the IO delay of the logical unit number is abnormal. The third preset value range can be set according to actual needs. For example, the third preset value range can be set to a value range that is more than 10 times the IO delay threshold of the logical unit number, that is, when the IO delay of the logical unit number is greater than the logical unit number When the IO delay threshold of the unit number is 11 times, it is determined that the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within the third preset value range.

Wherein, if it is determined that the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the fourth preset value range, it is determined that the IO delay of the computing node and the IO delay of the logical unit number are compared near. The fourth preset value range can be set according to actual needs. For example, the fourth preset value range can be set to a value range that is less than 0.1 times the IO delay of the logical unit number, that is, when the IO delay of the computing node is smaller than the logical unit number. When the IO delay is 1.1 times, it is judged that the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the fourth preset value range.

In a preferred implementation of the embodiment of the present invention, after the IO fault is located at the back end in step 140, the method further includes:

Step 141: Obtain the ping network delay of each back-end storage node, the average ping network delay of all back-end storage nodes, and the ping network delay threshold of the back-end storage nodes; if the ping network delay of all back-end storage nodes The difference between the average delay and the ping network delay threshold of the back-end storage node is within the fifth preset value range, and the ping network delay of each back-end storage node and the ping network delay of the back-end storage node If the difference between the thresholds is within the sixth preset value range, the IO fault is located in the interconnection network between back-end storage nodes; if the average ping network delay of all back-end storage nodes is the same as the The difference between the ping network delay thresholds of the nodes is within the seventh preset value range, and the difference between the ping network delays of the preset number of back-end storage nodes and the ping network delay thresholds of the back-end storage nodes is within the seventh preset value range. 8. Within the preset value range, if the ping network delay of other back-end storage nodes other than the preset number of back-end storage nodes is not greater than the ping network delay threshold of the back-end storage node, the IO fault is located at the preset number The network of back-end storage nodes; if the difference between the average ping network delay of all back-end storage nodes and the ping network delay threshold of back-end storage nodes is within the ninth preset value range, obtain all The IO delay of the built-in disk of the back-end storage node and the IO delay threshold of the built-in disk corresponding to the IO delay of the built-in disk will make the difference between the IO delay of the built-in disk and the IO delay threshold of the built-in disk in the first The built-in disks within the preset value range are identified as abnormal built-in disks, and the IO fault is located in the abnormal built-in disk.

The ping network delay of each backend storage node and the average of the ping network delays of all backend storage nodes can be calculated by the moving average method. If the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the fifth preset value range, it is determined that the ping network delays of all the back-end storage nodes are The average value of is abnormal, and the fifth preset value range may be set to, for example, a value range that is more than 10 times the ping network delay threshold of the back-end storage node. If the difference between the ping network delay of each back-end storage node and the ping network delay threshold of the back-end storage node is within the sixth preset value range, it is determined that the ping network delay of each back-end storage node is If both are abnormal, the sixth preset value range may be set to, for example, a value range that is more than 10 times the ping network delay threshold of the back-end storage node.

Among them, if the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the seventh preset value range, it is determined that the ping network of all the back-end storage nodes is If the average value of the delay is abnormal, the seventh preset value range may be set to, for example, a value range that is more than 10 times the ping network delay threshold of the backend storage node. If the difference between the ping network delay of the preset number of back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the eighth preset value range, it is determined that the preset number of back-end storage nodes has If the ping network delay is abnormal, the eighth preset value range may be set to, for example, a value range that is more than 10 times the ping network delay threshold of the backend storage node, and the preset number may be set to 1, for example. If the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the ninth preset value range, it is determined that the ping network of all the back-end storage nodes is within the range of the ninth preset value. The average value of the delay is normal, and the ninth preset value range may be, for example, a value range within 0.1 times the ping network delay threshold of the backend storage node. The tenth preset numerical range may be set to, for example, a numerical range that is more than 10 times the IO delay threshold of the built-in disk.

In the embodiment of the present invention, the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node are obtained; if the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within the first preset value Within the set value range, the IO delay of the logical unit number corresponding to the computing node and the IO delay threshold of the logical unit number are further obtained; if the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, and If the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the second preset value range, locate the IO fault at the front end; if the IO delay of the logical unit number and the IO delay of the logical unit number If the difference between the thresholds is within the third preset value range, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the fourth preset value range, the IO fault is located at the backend . It can be seen that the embodiment of the present invention can quickly and accurately locate the IO fault by comparing the IO delay of the front-end computing node and the IO delay of the back-end logical unit number.

FIG. 2 shows a schematic structural diagram of an embodiment of an IO fault locating apparatus according to the present invention. As shown in FIG. 2 , the apparatus 300 includes: a first obtaining module 310 , a second obtaining module 320 and a positioning module 330 .

The first obtaining module 310 is used to obtain the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node;

The second obtaining module 320 is configured to obtain the logic unit corresponding to the computing node if the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within a first preset value range The IO delay of the number and the IO delay threshold of the logical unit number;

The positioning module 330 is configured to, if the IO delay of the logical unit number is not greater than the IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number If the value is within the second preset value range, the IO fault is located at the front end; if the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within the third preset value and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within a fourth preset value range, the IO fault is located at the back end.

In an optional manner, the locating module 330 is further configured to obtain the IO of other computing nodes corresponding to the same logical unit number as the computing node respectively when the computing node is a virtual machine after locating the IO fault at the front end. Delay, IO delay of other virtual machines corresponding to the same physical machine as the computing node, and IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located;

If it is judged that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, locate the IO fault on the front-end and back-end interconnected networks;

If it is judged that the IO delay of other virtual machines corresponding to the same physical machine as the computing node is normal, and the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, locate the IO fault. at the compute node;

If it is judged that the IO delay of other virtual machines corresponding to the same physical machine as the computing node is abnormal, and it is judged that the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is normal, locate the IO fault. On the physical machine where the compute node is located.

In an optional manner, the locating module 330 is further configured to obtain the IO of other computing nodes corresponding to the same logical unit number as the computing node when the computing node is a physical machine after locating the IO fault at the front end. extend;

If it is judged that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is abnormal, locate the IO fault on the front-end and back-end interconnected networks;

If it is judged that the IO delay of other computing nodes corresponding to the same logical unit number as the computing node is normal, the IO fault is located on the computing node.

In an optional manner, the location module 330 is further configured to obtain the ping network delay of each back-end storage node and the ping network delay of all back-end storage nodes after locating the IO fault at the back end. The average value and the ping network delay threshold of the backend storage node;

If the difference between the average ping network delay of all back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the fifth preset value range, and the ping network delay of each back-end storage node is the same as If the difference between the ping network delay thresholds of the back-end storage nodes is within the sixth preset value range, the IO fault is located in the interconnection network between the back-end storage nodes;

If the difference between the average ping network delay of all back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the seventh preset value range, and the ping network time of a preset number of back-end storage nodes The difference between the delay and the ping network delay threshold of the backend storage node is within the eighth preset value range, and the ping network delay of other backend storage nodes other than the preset number of backend storage nodes is not greater than that of the backend The ping network delay threshold of the storage node, the IO fault is located in the network of the preset number of back-end storage nodes;

If the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the ninth preset value range, obtain the built-in disks of all the back-end storage nodes. The IO delay and the IO delay threshold of the built-in disk corresponding to the IO delay of the built-in disk will make the difference between the IO delay of the built-in disk and the IO delay threshold of the built-in disk within the tenth preset value range. The disk is identified as an abnormal built-in disk, and the IO fault is located on the abnormal built-in disk.

In the embodiment of the present invention, the first acquisition module can acquire the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node, and the second acquisition module can obtain the IO delay of the computing node and the IO delay threshold of the computing node. When the difference between the IO delay thresholds of the external disks is within the first preset value range, obtain the IO delay of the logical unit number corresponding to the computing node and the IO delay threshold of the logical unit number, and the positioning module can Compare the IO latency of the logical unit number with the IO latency threshold of the logical unit number, and compare the IO latency of the computing node with the IO latency of the logical unit number, so as to locate the IO fault at the front end or The fault is located in the backend. It can be seen that the IO fault locating device according to the embodiment of the present invention can quickly and accurately locate the IO fault.

FIG. 3 shows a schematic structural diagram of an embodiment of a computing device of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.

As shown in FIG. 3 , the computing device may include: a processor (processor) 402 , a communications interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

The processor 402 , the communication interface 404 , and the memory 406 communicate with each other through the communication bus 408 . The communication interface 404 is used for communicating with network elements of other devices such as clients or other servers. The processor 402 is configured to execute the program 410, and may specifically execute the relevant steps in the above-mentioned embodiments of the method for locating an IO fault.

Specifically, program 410 may include program code, which includes computer-executable instructions.

The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the computing device may be the same type of processors, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 406 is used to store the program 410 . Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

The program 410 can be specifically called by the processor 402 to make the computing device perform the following operations:

Obtain the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node;

If the difference between the IO delay of the computing node and the IO delay threshold of the external disk is within the first preset value range, further obtain the IO delay of the logical unit number corresponding to the computing node and all the IO delay threshold of the logical unit number;

If the IO latency of the logical unit number is not greater than the IO latency threshold of the logical unit number, and the difference between the IO latency of the computing node and the IO latency of the logical unit number is within the second preset value Within the value range, the IO fault is located at the front end;

If the difference between the IO delay of the logical unit number and the IO delay threshold of the logical unit number is within a third preset value range, and the IO delay of the computing node and the IO delay of the logical unit number If the difference between the delays is within the fourth preset value range, the IO fault is located at the back end.

In an optional manner, the program 410 is invoked by the processor 402 to cause the computing device to perform the following operations:

When the computing node is a virtual machine, obtain the IO delay of other computing nodes corresponding to the same logical unit number as the computing node, and the IO delay of other virtual machines corresponding to the same physical machine as the computing node. and the IO delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located;

If it is determined that the IO delay of the other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

If it is judged that the IO delay of the other virtual machines corresponding to the same physical machine as the computing node is normal, and the IO of the other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is judged to be normal If the delay is normal, the IO fault is located on the computing node;

If it is determined that the IO delay of the other virtual machines corresponding to the same physical machine as the computing node is abnormal, and the IO of the other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is determined to be abnormal If the delay is normal, the IO fault is located on the physical machine where the computing node is located.

In an optional manner, the program 410 is invoked by the processor 402 to cause the computing device to perform the following operations:

When the computing node is a physical machine, obtain the IO delay of other computing nodes corresponding to the same logical unit number as the computing node;

If it is determined that the IO delay of the other computing nodes corresponding to the same logical unit number as the computing node is abnormal, the IO fault is located in the interconnection network between the front end and the back end;

If it is determined that the IO delay of the other computing nodes corresponding to the same logical unit number as the computing node is normal, the IO fault is located on the computing node.

In an optional manner, the program 410 is invoked by the processor 402 to cause the computing device to perform the following operations:

Obtain the ping network delay of each back-end storage node, the average ping network delay of all back-end storage nodes, and the ping network delay threshold of the back-end storage nodes;

If the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the fifth preset value range, and each back-end storage node If the difference between the ping network delay of the back-end storage node and the ping network delay threshold of the back-end storage node is within the sixth preset value range, the IO fault is located in the interconnection network between the back-end storage nodes;

If the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the seventh preset value range, and the preset number of back-end storage nodes The difference between the ping network delay of the back-end storage node and the ping network delay threshold of the back-end storage node is within the eighth preset value range, and the back-end storage nodes other than the preset number of back-end storage nodes have If the ping network delay is not greater than the ping network delay threshold of the back-end storage node, the IO fault is located in the network of the preset number of back-end storage nodes;

If the difference between the average of the ping network delays of all the back-end storage nodes and the ping network delay threshold of the back-end storage nodes is within the ninth preset value range, obtain all the back-end storage nodes. The IO delay of the built-in disk of the storage node and the IO delay threshold of the built-in disk corresponding to the IO delay of the built-in disk will make the difference between the IO delay of the built-in disk and the IO delay threshold of the built-in disk. The built-in disk whose difference is within the tenth preset value range is identified as an abnormal built-in disk, and the IO fault is located in the abnormal built-in disk.

In the embodiment of the present invention, the program in the computing device is called by the processor, so that the computing device can execute the acquisition of the IO delay of the front-end computing node and the IO delay threshold of the external disk of the computing node; if the IO delay of the computing node is equal to If the difference between the IO delay thresholds of the external disks is within the first preset value range, the IO delay of the logical unit number corresponding to the computing node and the IO delay threshold of the logical unit number are further obtained; if the IO delay of the logical unit number is If the delay is not greater than the IO delay threshold of the logical unit number, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the second preset value range, the IO fault is located at the front end; if the logic The difference between the IO delay of the unit number and the IO delay threshold of the logical unit number is within the third preset value range, and the difference between the IO delay of the computing node and the IO delay of the logical unit number is within the fourth Within the preset value range, the IO fault is located at the backend. It can be seen that the computing device can quickly read and accurately locate IO faults.

FIG. 4 shows a schematic structural diagram of an embodiment of an IO fault locating device according to the present invention. As shown in Figure 4, the device 500 includes:

The collection module 510 is used for collecting IO delay information.

The IO delay information may include: the IO delay of the front-end computing node, the IO delay of the logical unit number corresponding to the computing node, the ping network delay of each back-end storage node, and, if the computing node is a virtual machine Collects the IO delays of other computing nodes that correspond to the same LUN as the computing node, the IO delays of other virtual machines that correspond to the same physical machine as the computing node, and other physical machines that correspond to the same LUN as the physical machine where the computing node resides. The IO delay of the computer, and, when the current computing node is a physical machine, collects the IO delay of other computing nodes corresponding to the same LUN as the computing node.

The configuration module 520 is configured to configure the IO delay threshold.

The IO delay threshold may include: the IO delay threshold corresponding to the external disk of the computing node, the IO delay threshold of the logical unit number corresponding to the computing node, the ping network delay threshold of the back-end storage node, and, in When the computing node is a virtual machine, it is used to configure the IO delay threshold of other computing nodes corresponding to the same logical unit number as the computing node, the IO delay threshold of other virtual machines corresponding to the same physical machine as the computing node, and the computing node. The IO delay threshold of other physical machines whose physical machine corresponds to the same logical unit number, and, when the current computing node is a physical machine, is used to configure the IO of other computing nodes corresponding to the same logical unit number as the computing node. delay threshold.

The locating module 530 is configured to execute the above-mentioned IO fault locating method to locate the IO fault.

The display module 540 is configured to obtain the result of the location module locating the IO fault, and display the result.

In an optional manner, the collection module 510 is further configured to collect IOPS information, where IOPS is Input/Output Per Second, which refers to the amount of input and output per second of the disk. The collection module can collect the IOPS information of the storage disks of the back-end storage nodes.

In an optional manner, the display module 540 is further configured to display the IO delay information collected by the collection module 510, and to display the correspondence between the external disk of the front-end computing node and the logical unit number of the back-end storage cluster.

In an optional manner, the configuration module 520 is further configured to:

Obtain the configuration information of the front-end computing node and the configuration information of the back-end storage cluster, and send the configuration information of the front-end computing node and the configuration information of the back-end storage cluster to the positioning module 530;

The location module 530 is further configured to: locate the IO fault according to the configuration information of the front-end computing node and the configuration information of the back-end storage cluster.

The configuration information of the front-end computing node may include: the host name of the computing node, the external disk information of the computing node, and the storage cluster information corresponding to the computing node; the configuration information of the back-end storage cluster may include: the storage cluster corresponding to the computing node Name, the number of storage nodes included in the storage cluster corresponding to the computing node, the name of each storage node, the number of logical unit numbers divided by the storage cluster corresponding to the computing node, and other computing nodes that correspond to the same storage cluster as the computing node Node information for the node.

In an optional manner, the configuration module 520 is further configured to obtain the correspondence between the upper and lower layers of the back-end storage cluster, and send the obtained correspondence between the upper and lower layers of the back-end storage cluster to the positioning module 530;

The location module 530 is further configured to: locate the IO fault according to the correspondence between the upper and lower layers of the backend storage cluster.

In an optional manner, the configuration module 520 is further configured to obtain reference information, and configure the IO delay threshold according to the reference information. The reference information may include: the hardware configuration of the computing node, the back-end storage node corresponding to the computing node The hardware configuration and the IO latency requirements of the tasks processed by the computing nodes.

In the embodiment of the present invention, the acquisition module can collect IO delay information, the configuration module can configure the IO delay threshold, the location module can execute the IO fault location method to locate the IO fault, and the display module can obtain the location module to perform the IO fault detection. The results of the positioning, and the results are displayed. It can be seen that the IO fault locating device in the embodiment of the present invention can quickly and accurately locate the IO fault, and display the IO fault locating result, so as to facilitate further analysis of the IO fault.

An embodiment of the present invention provides a computer-readable storage medium, where the storage medium stores at least one executable instruction, and when the executable instruction is run on an IO fault locating device, causes the IO fault locating device to execute any of the above methods The IO fault location method in the embodiment.

An embodiment of the present invention provides a computer program, and the computer program can be invoked by a processor to cause a computing device to execute the method for locating an IO fault in any of the foregoing method embodiments.

An embodiment of the present invention provides a computer program product. The computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are run on a computer, the computer is made to execute any of the above. The IO fault location method in the method embodiment.

The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with teaching based on this. The structure required to construct such a system is apparent from the above description. Furthermore, embodiments of the present invention are not directed to any particular programming language. It is to be understood that various programming languages may be used to implement the inventions described herein, and that the descriptions of specific languages above are intended to disclose the best mode for carrying out the invention.

In the description provided herein, numerous specific details are set forth. It will be understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it is to be understood that in the above description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together into a single implementation in order to simplify the invention and to aid in the understanding of one or more of the various aspects of the invention. examples, figures, or descriptions thereof. This disclosure, however, should not be construed as reflecting an intention that the invention as claimed requires more features than are expressly recited in each claim.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and they may be divided into multiple sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method so disclosed may be employed in any combination unless at least some of such features and/or procedures or elements are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

It should be noted that the above-described embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names. The steps in the above embodiments should not be construed as limitations on the execution order unless otherwise specified.

Claims (10)

1. An IO fault location method is characterized in that the IO fault is an IO fault of distributed storage, and the method comprises the following steps:

obtaining IO time delay of a front-end computing node and an IO time delay threshold of an external disk of the computing node;

if the difference value between the IO time delay of the computing node and the IO time delay threshold of the external disk is within a first preset value range, further acquiring the IO time delay of the logic unit number corresponding to the computing node and the IO time delay threshold of the logic unit number;

if the IO time delay of the logic unit number is not larger than the IO time delay threshold of the logic unit number, and the difference value between the IO time delay of the computing node and the IO time delay of the logic unit number is within a second preset value range, positioning the IO fault at the front end;

and if the difference value between the IO time delay of the logic unit number and the IO time delay threshold of the logic unit number is within a third preset numerical range, and the difference value between the IO time delay of the computing node and the IO time delay of the logic unit number is within a fourth preset numerical range, the IO fault is positioned at the rear end.

2. The method of claim 1, wherein after the locating the IO fault at a front end, the method further comprises:

if the computing node is a virtual machine, respectively acquiring IO time delays of other computing nodes corresponding to the same logic unit number with the computing node, IO time delays of other virtual machines corresponding to the same physical machine with the computing node and IO time delays of other physical machines corresponding to the same logic unit number with the physical machine where the computing node is located;

if the IO time delay of other computing nodes corresponding to the same logic unit number with the computing node is judged to be abnormal, the IO fault is positioned in the interconnection network of the front end and the rear end;

if the IO time delay of other virtual machines corresponding to the same physical machine as the computing node is judged to be normal, and the IO time delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is judged to be normal, the IO fault is positioned at the computing node;

and if the IO time delay of other virtual machines corresponding to the same physical machine as the computing node is judged to be abnormal, and the IO time delay of other physical machines corresponding to the same logical unit number as the physical machine where the computing node is located is judged to be normal, locating the IO fault in the physical machine where the computing node is located.

3. The method of claim 1 or 2, wherein after the locating the IO fault at a front end, the method further comprises:

if the computing node is a physical machine, IO time delay of other computing nodes corresponding to the same logic unit number as the computing node is obtained;

if the IO time delay of other computing nodes corresponding to the same logic unit number with the computing node is judged to be abnormal, the IO fault is positioned in the interconnection network of the front end and the rear end;

and if the IO time delay of the other computing nodes corresponding to the same logic unit number with the computing node is judged to be normal, the IO fault is positioned in the computing node.

4. The method of claim 1, wherein after the locating the IO fault at a backend, the method further comprises:

acquiring ping network delay of each rear-end storage node, an average value of ping network delays of all rear-end storage nodes and a ping network delay threshold of the rear-end storage node;

if the difference value between the average value of the ping network time delays of all the rear-end storage nodes and the ping network time delay threshold value of the rear-end storage node is within a fifth preset numerical range, and the difference value between the ping network time delay of each rear-end storage node and the ping network time delay threshold value of the rear-end storage node is within a sixth preset numerical range, the IO fault is positioned in an interconnection network between the rear-end storage nodes;

if the difference value between the average value of the ping network time delays of all the rear-end storage nodes and the ping network time delay threshold value of the rear-end storage nodes is within a seventh preset numerical range, the difference value between the ping network time delays of the rear-end storage nodes with a preset number and the ping network time delay threshold value of the rear-end storage nodes is within an eighth preset numerical range, and the ping network time delays of other rear-end storage nodes except the rear-end storage nodes with the preset number are not greater than the ping network time delay threshold value of the rear-end storage nodes, locating the IO fault in the network of the rear-end storage nodes with the preset number;

if the difference value between the average value of the ping network time delays of all the rear-end storage nodes and the ping network time delay threshold value of the rear-end storage node is within a ninth preset numerical range, the IO time delay of the built-in disks of all the rear-end storage nodes and the IO time delay threshold value of the built-in disks corresponding to the IO time delay of the built-in disks are obtained, the built-in disks which enable the difference value between the IO time delay of the built-in disks and the IO time delay threshold value of the built-in disks to be within a tenth preset numerical range are identified as abnormal built-in disks, and the IO faults are located on the abnormal built-in disks.

5. An IO fault location device, wherein an IO fault is an IO fault for distributed storage, the device comprising:

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring IO time delay of a front-end computing node and an IO time delay threshold of an external disk of the computing node;

a second obtaining module, configured to obtain an IO delay of a logical unit number corresponding to the compute node and an IO delay threshold of the logical unit number if a difference between the IO delay of the compute node and the IO delay threshold of the external disk is within a first preset numerical range;

the positioning module is used for positioning the IO fault at the front end if the IO time delay of the logic unit number is not larger than the IO time delay threshold of the logic unit number and the difference value between the IO time delay of the computing node and the IO time delay of the logic unit number is within a second preset value range; and if the difference value between the IO time delay of the logic unit number and the IO time delay threshold of the logic unit number is within a third preset numerical range, and the difference value between the IO time delay of the computing node and the IO time delay of the logic unit number is within a fourth preset numerical range, the IO fault is positioned at the rear end.

6. A computing device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is configured to store at least one executable instruction that causes the processor to perform the operations of the IO fault localization method of any one of claims 1-4.

7. An IO fault location device, wherein the IO fault is an IO fault for distributed storage, the device comprising:

the acquisition module is used for acquiring IO time delay information;

the configuration module is used for configuring an IO delay threshold;

a positioning module, configured to execute the IO fault positioning method according to any one of claims 1 to 4, so as to position the IO fault;

and the display module is used for acquiring the result of the positioning module for positioning the IO fault and displaying the result.

8. The apparatus of claim 7, wherein the presentation module is further configured to:

and displaying the IO time delay information acquired by the acquisition module, and displaying the corresponding relation between the external disk of the front-end computing node and the logic unit number of the rear-end storage cluster.

9. The apparatus of claim 7 or 8, wherein the configuration module is further configured to:

acquiring configuration information of a front-end computing node and configuration information of a rear-end storage cluster, and sending the configuration information of the front-end computing node and the configuration information of the rear-end storage cluster to the positioning module;

the positioning module is further configured to: and positioning the IO fault according to the configuration information of the front-end computing node and the configuration information of the rear-end storage cluster.

10. A computer-readable storage medium having stored therein at least one executable instruction that, when executed on a computing device, causes the computing device to perform operations of the IO fault localization method of any one of claims 1-4.

Images