CN118802492A - Fault location method, device, equipment and storage medium of network file system - Google Patents

Abstract

The application relates to the technical field of mobile communication, and provides a fault positioning method, a device, equipment and a storage medium of a network file system, wherein the method comprises the steps of acquiring NFS protocol data packets through tcpdump at a client and a server to obtain original data; performing unique xid field matching on the original data to obtain complete transfer information of each IO atom operation; obtaining average time delay data of corresponding time points based on complete transfer information of all IO atom operations; and comparing the overall average time delay with an overall time delay threshold, and if a first difference value of subtracting the overall time delay threshold from the overall time delay is larger than a first preset threshold, triggering threshold judgment of the server storage average time delay, the server average time delay, the network average time delay and the client average time delay at corresponding time points one by one to obtain an analysis result for feeding back the fault position. The application generates the fine monitoring index, can definitely and completely consume time for the IO path, and realizes the accurate positioning of the fault point.

Description

Fault location method, device, equipment and storage medium of network file system

Technical Field

The present application relates to the field of mobile communication technology, and in particular to a method, device, equipment and storage medium for locating a fault of a network file system.

Background Art

Network File System (NFS) storage based on TCP/IP transmission can easily and quickly realize data sharing and maintain data consistency among multiple servers. It is widely used in enterprise environments to realize sharing of data files or directories between different hosts or even systems.

The existing NFS storage technology features usually use the system's own detection commands for NFS storage. The client can only obtain read and write request data, while the server can obtain data such as the number of calls of all request types and overall latency. There is no effective correlation between the two.

In the existing NFS storage architecture, the actual IO path has many links. The existing performance monitoring methods can roughly determine whether there are performance problems in the storage IO path, but there are still the following shortcomings:

1. The existing monitoring indicators are not precise, and the fault point is difficult to locate quickly;

2. There is a lack of time-consuming information for each link in the IO path, making it difficult to accurately locate the fault point.

Summary of the invention

The embodiments of the present application provide a method, apparatus, device and storage medium for locating a fault of a network file system, so as to solve the technical problems of low accuracy of monitoring indicators and difficulty in locating fault points.

In a first aspect, an embodiment of the present application provides a fault location method for a network file system, comprising: collecting real-time overall delay and real-time server storage delay, collecting NFS protocol data packets through tcpdump on the client and the server to obtain original data; performing unique xid field matching on the original data to obtain complete transmission information of each IO atomic operation; based on the complete transmission information of all IO atomic operations, obtaining average delay data at the corresponding time point; wherein the average delay data includes overall average delay, server storage average delay, server average delay, network average delay and client average delay; determining threshold delay data; wherein the threshold delay data includes overall delay threshold, server storage delay threshold, server delay threshold, network delay threshold and client delay threshold; comparing the overall average delay with the overall delay threshold, if the first difference between the average overall delay and the overall delay threshold is greater than the first preset threshold, then triggering the threshold judgment of the server storage average delay, server average delay, network average delay and client average delay at the corresponding time point one by one, and obtaining an analysis result for feedback of the fault location.

In one embodiment, NFS protocol data packets are collected through tcpdump on the client and server to obtain original data; unique xid field matching is performed on the original data to obtain complete transfer information of each IO atomic operation, including: intercepting RPC call request data packets of the NFS storage client and server through the tcpdump tool, and filtering out all IO data packets of NFS operations on the client and server according to NFS protocol characteristics; wherein the information of each IO data packet includes path, timestamp, source address, destination address, xid field, atomic operation type and data packet content; xid field matching is performed on all IO data packets to obtain complete transfer information of each IO atomic operation with the same xid field, wherein the complete transfer information is the entire IO path, including four processes of IO data packets from the client, into the server, out of the server, and into the client.

In one embodiment, based on the complete transmission information of all IO atomic operations, the average delay data of the corresponding time point is obtained, including: for the complete transmission information of each IO atomic operation, using the entry and exit timestamps of the IO data packets on the client and server, the server time, network time and client time in a unit time interval are calculated; the server time is used as the average server delay; the network time is used as the average network delay; the client time is used as the average client delay; wherein, the server time is the time taken for a complete IO to complete the server entry and server exit; the network time is the time taken for a complete IO request to actually be transmitted back and forth on the network link after completing the four processes of client exit, server entry, server exit, and client entry; the client time is the overall time taken for a complete IO minus the server time and network time.

In one embodiment, the threshold judgments of the server-side storage average delay, server-side average delay, network average delay and client-side average delay at corresponding time points are triggered one by one to obtain analysis results for feedback of the fault location, including: comparing the server-side average delay with the server-side delay threshold; if the second difference between the server-side average delay and the server-side delay threshold is less than or equal to the second preset threshold, the server is normal; if the second difference between the server-side average delay and the server-side delay threshold is greater than the second preset threshold, comparing the server-side storage average delay with the server-side storage delay threshold.

In one embodiment, the server-side storage average latency is compared with the server-side storage latency threshold, including: if the third difference between the server-side storage average latency and the server-side storage latency threshold is less than or equal to the third preset threshold, the metadata access is abnormal; if the third difference between the server-side storage average latency and the server-side storage latency threshold is greater than the third preset threshold, the data storage is abnormal.

In one embodiment, the threshold judgments of the server-side storage average delay, the server-side average delay, the network average delay and the client-side average delay at corresponding time points are triggered one by one to obtain analysis results for feedback of the fault location, including: comparing the network average delay with the network delay threshold, if the fourth difference between the network average delay and the network delay threshold is less than or equal to the fourth preset threshold, the network link is normal; if the fourth difference between the network average delay and the network delay threshold is greater than the fourth preset threshold, the network link is abnormal.

In one embodiment, threshold judgments of the server-side storage average delay, server-side average delay, network average delay, and client-side average delay at corresponding time points are triggered one by one to obtain analysis results for feedback of the fault location, including: comparing the client-side average delay with the client-side delay threshold; if the fifth difference between the client-side average delay and the client-side delay threshold is less than or equal to the fifth preset threshold, the client link is normal; if the fifth difference between the client-side average delay and the client-side delay threshold is greater than the fifth preset threshold, the client link is abnormal.

In the second aspect, the embodiment of the present application provides a fault location device for a network file system, including: a data acquisition module, which is used to collect real-time overall delay and real-time server storage delay, and collect NFS protocol data packets through tcpdump on the client and server to obtain original data; unique xid field matching is performed on the original data to obtain complete transmission information of each IO atomic operation; a data extraction module is used to obtain average delay data at the corresponding time point based on the complete transmission information of all IO atomic operations; wherein the average delay data includes overall average delay, server storage average delay, server average delay, network delay and client average delay; a threshold configuration module, used to determine the threshold delay data; wherein the threshold delay data includes the overall delay threshold, the server storage delay threshold, the server delay threshold, the network delay threshold and the client delay threshold; a performance analysis positioning judgment module, used to compare the overall average delay with the overall delay threshold. If the first difference between the average overall delay and the overall delay threshold is greater than the first preset threshold, the threshold judgments of the server storage average delay, the server average delay, the network average delay and the client average delay at the corresponding time point are triggered one by one to obtain the analysis results for feedback of the fault location.

In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing a computer program, wherein when the processor executes the program, the method for locating the fault of the network file system described in the first aspect is implemented.

In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for locating a fault in a network file system described in the first aspect is implemented.

The fault location method, device, equipment and storage medium of the network file system provided by the embodiment of the present application include collecting NFS protocol data packets through tcpdump on the client and server to obtain original data; matching the original data with a unique xid field to obtain the complete transmission information of each IO atomic operation; obtaining the average delay data of the corresponding time point based on the complete transmission information of all IO atomic operations; comparing the overall average delay with the overall delay threshold, if the first difference between the average overall delay and the overall delay threshold is greater than the first preset threshold, then triggering the threshold judgment of the server storage average delay, server average delay, network average delay and client average delay at the corresponding time point one by one, and obtaining the analysis result for feedback of the fault location. Through the above method, the present application can generate fine monitoring indicators, refine NFS request IO atomic operations, obtain the complete IO path and the delay of each IO path link, so as to clarify the time consumption of the complete IO path and realize accurate location of the fault point.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of a flow chart of a method for locating a fault in a network file system provided in an embodiment of the present application;

FIG2 is a schematic diagram of a deployment architecture of a network file system provided in an embodiment of the present application;

FIG3 is a schematic diagram of a detailed actual IO path of FIG2 provided in an embodiment of the present application;

FIG4 is a schematic diagram of atomic operation path tracking provided by an embodiment of the present application;

FIG5 is a schematic diagram of the time consumption of each link of the IO path provided in an embodiment of the present application;

FIG6 is a schematic diagram of a process flow in a performance analysis and positioning judgment process provided in an embodiment of the present application;

7 is a schematic diagram of the structure of a fault location device for a network file system provided in an embodiment of the present application;

8 is a schematic diagram of the workflow of a fault location device for a network file system provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The network file system fault location method, device, equipment and storage medium provided in the embodiments of the present application can effectively improve the problems of NFS storage prior art in which monitoring indicators are not precise, fault points are difficult to locate quickly, and there is a lack of time-consuming information on each link of the IO path, making it difficult to accurately locate the fault point.

Please refer to Figure 1, which is a flow chart of a method for locating a fault of a network file system provided by an embodiment of the present application. The method for locating a fault of a network file system provided by this embodiment includes steps S110 to S140, and each step is specifically as follows:

S110: Collect the real-time overall latency and the real-time server storage latency, collect NFS protocol data packets through tcpdump on the client and server to obtain the original data; perform unique xid field matching on the original data to obtain complete transmission information of each IO atomic operation.

Please refer to Figures 2-3, Figure 2 is a schematic diagram of the deployment architecture of the network file system provided in the embodiment of the present application. Figure 3 is a schematic diagram of the actual IO path of Figure 2 provided in the embodiment of the present application.

As shown in Figure 2, an NFS server storage serves multiple NFS clients through the TCP/IP network, from which a complete NFS storage IO path model can be abstracted, namely, NFS client-->TCP/IP network-->NFS server file system-->data storage.

As shown in Figure 3, the application completes the request and response through the following five IO paths. It can be seen that the client only has NFS RPC calls, and the real IO occurs on the server: that is, the client application completes the RPC conversion of IO requirements through the file system service process, reaches the server service process through the network to complete the IO requirement restoration of the RPC request, and finally completes the data storage and reading and writing on the server. Details are as follows:

① The application interacts with the NFS client for IO requests: The application initiates an IO request to the NFS file system, and the NFS file system returns the request result to the application.

②. The NFS client performs IO request and RPC request conversion: The NFS file system converts IO requests into RPC network remote call requests through the client service process, and converts the corresponding RPC requests into IO requests of the NFS file system after receiving the RPC response from the server.

③、The NFS client transmits RPC requests to the server: The client and the server complete the RPC request transmission through the TCP/IP network.

④. The NFS server performs RPC request and IO request conversion: The server process converts the received RPC request into an NFS file system IO request, and first requests access to the file system metadata.

⑤. The NFS server completes the IO request data reading and writing: After successfully accessing the file system metadata, the service process formally reads and writes the stored data, and automatically returns a successful result after completion.

Under the NFS storage architecture of related technologies, the client can obtain the remote RPC call volume, average latency and other indicator information of the client's read and write operations through tools such as nfsiostat on the client; on the server, the number of various NFS request operations received by all NFS clients on the NFS storage can be viewed through the nfstat tool; on the server, the sar and iostat tools can also be used to obtain the IO usage of the NFS file system to the data storage, and the data that can be obtained are the IOPS, latency and other data of the read and write operations; however, the NFS client can only obtain the performance data of the read and write requests, and the NFS server can obtain the number of all NFS request types, the overall busyness and other data, and there is no effective correlation between the two:

1. Existing monitoring indicators are not precise, and fault points are difficult to locate quickly

The existing command tools can be used to obtain the overall usage status of a single NFS client and storage service respectively, but cannot perform fine-grained query on the completion status of each IO atomic request from each client during its entire life cycle. As a result, the data of a single client cannot be effectively associated and matched with the overall data of the server. When faced with abnormal access performance, it is impossible to quickly and timely determine which type of IO atomic request of which specific client has a problem.

2. Lack of time-consuming information for each link in the IO path makes it difficult to accurately locate the fault point

The actual IO path of the client is lengthy, and existing methods can only obtain the overall IO delay of the client, which includes the sum of the client delay, network transmission delay and storage delay, but cannot obtain the detailed delay of each segment. Therefore, it is impossible to achieve global IO path concatenation and effectively determine whether the performance problem occurs in a specific IO link of the client, network or storage end, and thus it is impossible to accurately locate the fault.

Therefore, in response to the above problems, the design of this embodiment can improve the precision of NFS monitoring indicators, refine the NFS request IO atomic operations, obtain the complete IO path and the delay of each IO path link, so as to quickly locate and solve the problem.

Optionally, collect NFS protocol data packets through tcpdump on the client and server to obtain original data; perform unique xid field matching on the original data to obtain complete transfer information of each IO atomic operation, including:

The tcpdump tool is used to intercept the RPC call request data packets of the NFS storage client and server, and all IO data packets of the NFS operations of the client and server are filtered out according to the NFS protocol characteristics; the information of each IO data packet includes the path, timestamp, source address, destination address, xid field, atomic operation type and data packet content; the xid field of all IO data packets is matched to obtain the complete transmission information of each IO atomic operation with the same xid field, where the complete transmission information is the entire IO path, including the four processes of IO data packets from the client, into the server, out of the server, and into the client.

S120: Based on the complete transmission information of all IO atomic operations, average latency data of corresponding time points is obtained.

The average latency data includes overall average latency, server storage average latency, server average latency, network average latency and client average latency.

Optionally, based on the complete transfer information of all IO atomic operations, the average latency data at the corresponding time point is obtained, including:

For the complete transmission information of each IO atomic operation, the server time, network time and client time in a unit time interval are calculated using the entry and exit timestamps of the IO data packets on the client and server. The server time is taken as the average server delay. The network time is taken as the average network delay. The client time is taken as the average client delay. Among them, the server time is the time it takes for a complete IO to enter and exit the server. The network time is the time it takes for a complete IO request to be transmitted back and forth on the network link after completing the four processes of client exit, server entry, server exit and client entry. The client time is the overall time of a complete IO minus the server time and network time.

S130: Determine threshold delay data.

The threshold delay data includes the overall delay threshold, server storage delay threshold, server delay threshold, network delay threshold and client delay threshold.

Since the network locations of various clients in the NFS storage architecture are different and the NFS storage configurations are also different, the network latency and server storage latency are also different. Therefore, threshold settings can be generated based on historical data and experience data, providing an important reference for full-link performance analysis.

In some embodiments, the threshold delay data is determined as follows:

Overall latency threshold: adjust different overall IO latency thresholds according to different business environments.

Server storage latency threshold: Set different storage latency thresholds according to different storage models.

Server latency threshold: Generates baseline reference data based on historical data under normal conditions.

Network delay threshold: Generally, the more network links there are and the longer the link is, the higher the threshold is.

Client latency threshold: Generates baseline reference data based on historical data under normal conditions.

S140: Compare the overall average delay with the overall delay threshold. If the first difference between the average overall delay and the overall delay threshold is greater than the first preset threshold, the threshold judgments of the server-side storage average delay, server-side average delay, network average delay and client-side average delay at the corresponding time points are triggered one by one to obtain analysis results for feedback of the fault location.

The embodiment of the present application provides a method for locating a fault of a network file system, including collecting NFS protocol data packets through tcpdump on the client and server to obtain raw data; matching the original data with a unique xid field to obtain complete transmission information of each IO atomic operation; obtaining the average delay data at the corresponding time point based on the complete transmission information of all IO atomic operations; comparing the overall average delay with the overall delay threshold, if the first difference between the average overall delay and the overall delay threshold is greater than the first preset threshold, then triggering the threshold judgment of the server storage average delay, server average delay, network average delay and client average delay at the corresponding time point one by one, and obtaining an analysis result for feedback of the fault location. This embodiment generates fine monitoring indicators, which can clarify the time consumption of the complete IO path and realize accurate positioning of the fault point.

This embodiment uses tcpdump to track network transmission packets on the nfs client and server (storage end) according to the technical principle of NFS protocol based on RPC calls, and uses the unique xid of RPC to track and filter out data packets of various IO atomic operations (such as read, write, rename, remove, create, etc.), as well as the timestamps of the entire IO atomic operation in each link of the nfs server, network and client to obtain the delay consumption data of each segment.

The core point is to track all data packets of the entire IO path, complete detailed statistics of each IO atomic operation through the unique identifier xid in the data packet, and calculate the IO time corresponding to each link of the IO path, and compare it with the set threshold to determine the problem. Now the following is an explanation from three aspects: IO atomic operation tracking, IO path time extraction and time threshold comparison:

1. IO atomic operation tracking: The device first intercepts the data through the tcpdump tool and

The NFS protocol features filter out all IO data packets of the client and server NFS operations. The entire IO path includes four processes: data packets from the client (Co), the server (Si), the server (So), and the client (Ci). Please refer to Figure 4, which is a schematic diagram of atomic operation path tracking provided by an embodiment of the present application.

Taking rename, one of the atomic IO operations in the NFS protocol, as an example, the format of the captured data packet is shown in Table 1. Each data packet contains information such as time node, source and destination address, and operation type. The complete transmission information of each IO atomic operation can be matched through the unique xid field to achieve complete IO path concatenation.

Table 1: Data packet format diagram

2. IO path time extraction: Through the previous step, the complete path information data of any IO atomic operation can be tracked and obtained, as shown in Figure 5, which is a schematic diagram of the time consumption of each link of the IO path provided in an embodiment of the present application.

Using tcpdump, we can capture the complete timestamps of the four points Co, Si, So, and Ci of the data packet. Through St = So-Si, Nt = Si-Co+Ci-So, and Ct = Tt-Nt-St, we can calculate the time consumption data of the three links: server time consumption St, network time consumption Nt, and client time consumption Ct. The specific formula is described in detail as follows.

As can be seen from the above, the entire IO path consists of five links, which can be divided into three links: server, network segment, and client according to the location of each device. Among them:

Server time: St = IO path ④ (NFS server program accesses NFS file system metadata) + IO path ⑤ (NFS server program accesses data storage) time, which is the time taken for a complete IO to complete the server entry (Si) and server exit (So), and is also equal to the file system metadata time Sm plus the IO writing data storage time Sb. That is: St = So-Si = Sm+Sb.

Network time consumption: Nt = the time consumption of IO path ③ (NFS network link transmission), which is actually the time consumption of the round trip transmission on the network link; therefore, it can be calculated by the network time consumption of the server receiving the packet (Si-Co) plus the network time consumption of the client receiving the return packet (Ci-So). That is: Nt = Si-Co + Ci-So.

Client time consumption: Ct = IO path ① (application calls NFS file system) + IO path ② (NFS file system converted to network RPC call) time consumption; therefore, the client time consumption can be obtained by subtracting the network delay Nt and the server delay St from the overall delay Tt. That is: Ct = Tt-Nt-St.

3. IO path time consumption threshold comparison: When NFS storage has performance service abnormalities, the time consumption St, Nt, and Ct of each of the above path links will be compared with the set baseline thresholds. This will enable us to determine where the problem lies in the server, network, or client, thereby achieving accurate analysis and positioning of NFS storage IO link performance.

The above, "the method of obtaining the smallest granularity single atomic IO data by tracking xid in NFS RPC call" and "the method of obtaining the operation time consumption of key links in the three-segment IO path and discovering and locating problems through the change of the three-segment IO time consumption" are the key points of this application, and their specific functions are:

1) Innovatively use the unique identifier xid to match and filter out the NFS protocol data packets of NFS storage operation requests through Tcpdump, and obtain the complete IO data of the entire life cycle of a single IO operation based on RPC calls. The data is based on a single complete IO request, and the data granularity is the finest, which can be used for subsequent extraction, processing and analysis;

2) When obtaining single atomic IO operation data, the entry and exit timestamps of the data packet on the client and server are used to calculate the operation time consumption of the three key links of the IO path within a unit time interval, including client delay (Ct), network delay (Nt), and server delay (St). The performance analysis results are obtained by analyzing and judging with the set threshold, so as to automatically and quickly discover and locate NFS problems.

In one embodiment, the threshold judgments of the server-side storage average delay, server-side average delay, network average delay and client-side average delay at corresponding time points are triggered one by one to obtain analysis results for feedback of the fault location, including: comparing the server-side average delay with the server-side delay threshold; if the second difference between the server-side average delay and the server-side delay threshold is less than or equal to the second preset threshold, the server is normal; if the second difference between the server-side average delay and the server-side delay threshold is greater than the second preset threshold, comparing the server-side storage average delay with the server-side storage delay threshold.

In one embodiment, the server-side storage average latency is compared with the server-side storage latency threshold, including: if the third difference between the server-side storage average latency and the server-side storage latency threshold is less than or equal to the third preset threshold, the metadata access is abnormal; if the third difference between the server-side storage average latency and the server-side storage latency threshold is greater than the third preset threshold, the data storage is abnormal.

In one embodiment, the threshold judgments of the server-side storage average delay, the server-side average delay, the network average delay and the client-side average delay at corresponding time points are triggered one by one to obtain analysis results for feedback of the fault location, including: comparing the network average delay with the network delay threshold, if the fourth difference between the network average delay and the network delay threshold is less than or equal to the fourth preset threshold, the network link is normal; if the fourth difference between the network average delay and the network delay threshold is greater than the fourth preset threshold, the network link is abnormal.

In one embodiment, threshold judgments of the server-side storage average delay, server-side average delay, network average delay, and client-side average delay at corresponding time points are triggered one by one to obtain analysis results for feedback of the fault location, including: comparing the client-side average delay with the client-side delay threshold; if the fifth difference between the client-side average delay and the client-side delay threshold is less than or equal to the fifth preset threshold, the client link is normal; if the fifth difference between the client-side average delay and the client-side delay threshold is greater than the fifth preset threshold, the client link is abnormal.

Please refer to Figure 6, which is a flow chart of the performance analysis and positioning judgment process provided by the embodiment of the present application. First, the overall average delay Tt is compared with its baseline threshold (i.e., the overall delay threshold) Tt_H according to the timing. If it is significantly greater than the baseline threshold, it triggers the judgment of the specific problems of the server, network, and client one by one, and locates the specific abnormal link path. The specific judgment rules are as follows:

A: If Tt>>Tt_H, that is, the overall average latency Tt is much greater than the overall latency threshold Tt_H, then the NFS storage performance service is considered abnormal, and the three major links are judged separately. Otherwise, the entire NFS storage link is normal.

B: If St>>St_H, and Sb>>Sb_H: that is, the average server delay St is much larger than the server delay threshold St_H, and the average server storage delay Sb is much larger than the server storage delay threshold Sb_H, then the server data storage is judged to be abnormal, otherwise the server is normal.

C: If St>>St_H, and Sb≈Sb_H: the average server latency St is much larger than the server latency threshold St_H, but the average server storage latency Sb is normal. Based on actual operation and maintenance experience, it is judged to be caused by abnormal file system metadata access.

D: If Nt>>Nt_H, that is, the average network delay Nt is much greater than the network delay threshold Nt_H, then the network link is judged to be abnormal, otherwise the network link is normal.

E: If Ct>>Ct_H, that is, the average client delay Ct is much greater than the client delay threshold Ct_H, then the client is judged to be abnormal, otherwise the client is normal.

Once the overall average delay Tt is abnormal, the above-mentioned time-consuming judgment method can be used to quickly and accurately obtain five types of positioning results: client abnormality, network abnormality, server-side data storage abnormality, logical lock problem and file system metadata access abnormality.

The following describes a fault location device for a network file system provided in an embodiment of the present application. The fault location device for a network file system described below and the fault location method for a network file system described above can refer to each other.

Please refer to Figure 7, which is a schematic diagram of the structure of a fault location device for a network file system provided in an embodiment of the present application. In this embodiment, the fault location device for a network file system includes:

The data collection module 710 is used to collect the real-time overall delay and the real-time server storage delay, collect NFS protocol data packets through tcpdump on the client and server to obtain the original data; match the unique xid field of the original data to obtain the complete transmission information of each IO atomic operation.

The data extraction module 720 is used to obtain the average delay data of the corresponding time point based on the complete transmission information of all IO atomic operations; the average delay data includes the overall average delay, the server storage average delay, the server average delay, the network delay and the client average delay.

The threshold configuration module 730 is used to determine the threshold delay data; wherein the threshold delay data includes the overall delay threshold, the server storage delay threshold, the server delay threshold, the network delay threshold and the client delay threshold.

The performance analysis and positioning judgment module 740 is used to compare the overall average delay with the overall delay threshold. If the first difference between the average overall delay and the overall delay threshold is greater than the first preset threshold, the threshold judgments of the server-side storage average delay, server-side average delay, network average delay and client-side average delay at the corresponding time points are triggered one by one to obtain analysis results for feedback of the fault location.

The data collection module provided in this embodiment mainly collects the overall IO delay Tt, the server data storage delay Sb through commands, and collects NFS protocol data packets through tcpdump on the client and server using xid matching filtering for subsequent analysis; the data extraction module mainly processes the original data obtained by the collection module, and calculates the average delay of the server, network, client and other path links for all NFS-related atomic operations according to xid for subsequent judgment; the threshold configuration module is mainly used to obtain the normal overall average delay Tt, server storage average delay Sb, server average delay St, network average delay Nt, client average delay Ct as historical baseline data for subsequent comparative analysis and judgment when the NFS service is provided normally; performance analysis and positioning judgment module: by regularly comparing the overall IO delay and baseline data, if it is significantly greater than the baseline threshold, the server storage, server, network, and client delays at the corresponding time points are obtained one by one, and then the specific abnormal links are judged respectively.

In one embodiment, the data acquisition module 710 is used to: intercept the RPC call request data packets of the NFS storage client and server through the tcpdump tool, and filter out all IO data packets of the NFS operations of the client and server according to the NFS protocol characteristics; wherein the information of each IO data packet includes the path, timestamp, source address, destination address, xid field, atomic operation type and data packet content; perform xid field matching on all IO data packets to obtain the complete transmission information of each IO atomic operation with the same xid field, wherein the complete transmission information is the entire IO path, including four processes of IO data packets from the client, into the server, out of the server, and into the client.

In one embodiment, the data extraction module 720 is used to: for the complete transmission information of each IO atomic operation, use the entry and exit timestamps of the IO data packets on the client and server to calculate the server time, network time and client time within a unit time interval; use the server time as the average server delay; use the network time as the average network delay; use the client time as the average client delay; wherein, the server time is the time taken for a complete IO to complete the server entry and server exit; the network time is the time taken for a complete IO request to actually be transmitted back and forth on the network link after completing the four processes of client exit, server entry, server exit and client entry; the client time is the overall time taken for a complete IO minus the server time and network time.

In one embodiment, the performance analysis and positioning judgment module 740 is used to compare the average server delay with the server delay threshold. If the second difference between the average server delay and the server delay threshold is less than or equal to the second preset threshold, the server is normal; if the second difference between the average server delay and the server delay threshold is greater than the second preset threshold, the average server storage delay is compared with the server storage delay threshold.

In one embodiment, the performance analysis positioning judgment module 740 is used to: if the third difference between the server-side storage average delay and the server-side storage delay threshold is less than or equal to the third preset threshold, then the metadata access is abnormal; if the third difference between the server-side storage average delay and the server-side storage delay threshold is greater than the third preset threshold, then the data storage is abnormal.

In one embodiment, the performance analysis positioning judgment module 740 is used to compare the network average delay with the network delay threshold. If the fourth difference between the network average delay and the network delay threshold is less than or equal to the fourth preset threshold, the network link is normal; if the fourth difference between the network average delay and the network delay threshold is greater than the fourth preset threshold, the network link is abnormal.

In one embodiment, the performance analysis positioning judgment module 740 is used to compare the client average delay with the client delay threshold. If the fifth difference between the client average delay and the client delay threshold is less than or equal to the fifth preset threshold, the client link is normal; if the fifth difference between the client average delay and the client delay threshold is greater than the fifth preset threshold, the client link is abnormal.

Please refer to FIG. 8 , which is a schematic diagram of the workflow of the fault location device for the network file system provided in an embodiment of the present application.

Data collection module: responsible for the basic data collection of this device, mainly through nfsiostat, sar/iostat to collect the overall average delay and the server storage average delay Sb; then use the tcpdump tool to intercept the RPC call request data packet of the NFS storage client and server (Table 1), and filter the time point, operation type and other information of the complete path of the IO atomic operation according to the xid field. The basic data types related to the delay that can be summarized are shown in Table 2 below, and the data is saved for further data extraction and processing of the average delay within the unit time interval.

Table 2: Schematic table of acquisition delay data types

Data extraction module: responsible for calculating the overall average latency Tt, server storage average latency Sb, and the time consumption of the three IO links (server average latency St, network average latency Nt, client average latency Ct) in the complete IO path of NFS atomic operation requests based on the basic data of the collection module, for subsequent performance analysis and judgment.

Calculation of overall average latency: The client uses the nfsiostat command tool to collect the average read and write latency Rtt_r and Rtt_w of IO, and can also obtain the number of read and write ops per second.

Therefore, the read and write ratios r% and w% (r%+w%=100%) are calculated according to the ratio of the number of read and write times per unit time, and the overall average delay Tt per unit time is obtained according to the actual read and write ratios: Tt=(Rtt_r*r%)+(Rtt_w*w%).

Calculation of average server storage latency: The server IO finally falls on the disk through two steps: accessing the file system metadata (Sm) and writing to the data storage (Sb). Since the NFS service process is running in kernel mode, the time consumption of accessing the file system metadata (Sm) is not discussed in this proposal. The time consumption of writing to the data storage (Sb) can also be obtained through the command tools sar and iostat on the server to obtain the average wait value. The average latency per unit time can be directly obtained: Sb = wait.

Calculation of average server latency: The server latency is the time it takes for a complete IO to complete the server entry (Si) and server exit (So), so it can be calculated by the difference between the server packet receiving network time (Si) and the server packet return network time (So). Similarly, the number of data packets per unit time n is calculated to be the number of complete IO requests w, and the average server latency per unit time n is finally obtained as:

Calculation of average network delay: The network time is the time it takes for a complete IO request to be transmitted back and forth on the network link after completing the four processes of client out (Co), server in (Si), server out (So), and client in (Ci). Therefore, it can be calculated by adding the network time it takes for the server to receive the packet (Si-Co) to the network time it takes for the client to receive the return packet (Ci-So). Similarly, the number of data packets per unit time n is calculated to be w, and the average network delay per unit time is finally obtained as:

Calculation of average client latency: A complete IO path for an IO request includes four processes: data packets from the client (Co), server (Si), server (So), and client (Ci). Therefore, the average time consumed by the client per unit time n can be calculated by subtracting the difference between the overall latency Tt and the network transmission latency Nt and the server latency St. That is:

Therefore, after data processing by this module, the average overall IO latency Tt, server storage latency Sb, and corresponding server latency within unit time n can be obtained. Network latency Client latency It is used as the core indicator for the next step of performance problem analysis and judgment, and is also accumulated as historical data as threshold reference data.

In summary, the fault location device for a network file system provided in the embodiment of the present application has the following beneficial effects:

1. Collect IO atomic operation indicators and generate fine monitoring indicators: The data collection module and data extraction module in the device of this embodiment use tcpdump to obtain basic data packets, and filter out various data of all qualified atomic IO operations initiated by the NFS protocol based on RPC calls through xid matching as basic atomic data. The client that initiated the IO operation can be clearly identified through the basic atomic operation related information, and combined with the single IO atomic operation type information, after extraction, fine monitoring indicator data including multiple dimensions based on the client and IO type can be obtained.

2. Clarify the time consumption of the complete IO path and realize accurate positioning of the fault point: In the device of this embodiment, the complete IO path and related data of the NFS IO request are obtained through xid. By intercepting the timestamp data of a single complete IO on the client and server, a single atomic IO key time consumption indicator St (storage delay), Nt (network delay), Ct (client delay) data is generated, and the key time consumption indicator data is accumulated and averaged in a certain sampling time interval to obtain the key time consumption indicator data. The key time consumption indicator is compared and analyzed with the baseline threshold through the performance analysis and judgment module, so as to realize the accurate positioning of client anomalies, network anomalies, data storage anomalies, and file system metadata access anomalies.

On the other hand, an embodiment of the present application further provides an electronic device, and FIG9 is a schematic diagram of the physical structure of the electronic device provided by the embodiment of the present application. As shown in FIG9, the electronic device may include: the electronic device may include a memory 920, a processor 910, and a computer program stored in the memory 920 and executable on the processor 910. When the processor 910 executes the program, the fault location method of the network file system provided by the above methods is implemented.

Optionally, the electronic device may further include a communication bus 930 and a communication interface (CommunicationsInterface) 940, wherein the processor 910, the communication interface 940, and the memory 920 communicate with each other via the communication bus 930. The processor 910 may call a computer program in the memory 920 to execute a fault location method for a network file system, the method comprising:

Collect real-time overall delay and real-time server storage delay, collect NFS protocol data packets through tcpdump on the client and server to obtain original data; match the original data with a unique xid field to obtain complete transmission information of each IO atomic operation; based on the complete transmission information of all IO atomic operations, obtain the average delay data at the corresponding time point; the average delay data includes the overall average delay, the server storage average delay, the server average delay, the network average delay and the client average delay; determine the threshold delay data; the threshold delay data includes the overall delay threshold, the server storage delay threshold, the server delay threshold, the network delay threshold and the client delay threshold; compare the overall average delay with the overall delay threshold, if the first difference between the average overall delay and the overall delay threshold is greater than the first preset threshold, then trigger the threshold judgment of the server storage average delay, the server average delay, the network average delay and the client average delay at the corresponding time point one by one, and obtain the analysis result for feedback of the fault location.

In addition, the logic instructions in the above-mentioned memory 920 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

On the other hand, the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the network file system fault location method provided by the above methods. Its steps and principles have been introduced in detail in the above methods and will not be repeated here.

Non-transitory computer-readable storage media can be any available media or data storage devices that can be accessed by a processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drives (SSDs)), etc.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (10)

1. A method for locating a failure of a network file system, comprising:

collecting real-time integral time delay and real-time server storage time delay, and collecting NFS protocol data packets through tcpdump at a client and a server to obtain original data;

performing unique xid field matching on the original data to obtain complete transfer information of each IO atom operation;

Obtaining average time delay data of corresponding time points based on complete transfer information of all IO atom operations; the average delay data comprises an overall average delay, a server storage average delay, a server average delay, a network average delay and a client average delay;

Determining threshold time delay data; the threshold delay data comprises an overall delay threshold, a server storage delay threshold, a server delay threshold, a network delay threshold and a client delay threshold;

And comparing the overall average time delay with the overall time delay threshold, and if the first difference value of subtracting the overall time delay threshold from the average overall time delay is larger than a first preset threshold, triggering threshold judgment of the server storage average time delay, the server average time delay, the network average time delay and the client average time delay of corresponding time points one by one to obtain an analysis result for feeding back the fault position.

2. The method for locating a failure of a network file system according to claim 1, wherein the NFS protocol data packet is collected by tcpdump at a client and a server to obtain original data; performing unique xid field matching on the original data to obtain complete transfer information of each IO atom operation, wherein the method comprises the following steps:

intercepting RPC call request data packets of the NFS storage client and the server through a tcpdump tool, and screening all IO data packets of the NFS operation of the client and the server according to NFS protocol characteristics; the information of each IO data packet comprises a path, a time stamp, a source address, a destination address, an xid field, an atomic operation type and data packet content;

and performing xid field matching on all the IO data packets to obtain complete transfer information of each IO atomic operation with the same xid field, wherein the complete transfer information is a whole IO path and comprises 4 processes of outputting the IO data packets from a client, inputting the IO data packets from a server, outputting the IO data packets from the server and inputting the IO data packets from the client.

3. The method for locating a failure of a network file system according to claim 2, wherein the obtaining average time delay data of the corresponding time points based on the complete transfer information of all IO atomic operations includes:

For the complete transfer information of each IO atom operation, calculating the time consumption of a server, the time consumption of a network and the time consumption of a client in a unit time interval by utilizing the time stamps of the IO data packets in and out of the client and the server;

taking the time consumption of the server as the average time delay of the server;

taking the network time consumption as the network average time delay;

Taking the time consumption of the client as the average time delay of the client;

The time consumption of the server is the time consumption of a complete IO between the completion of the entry of the server and the exit of the server; the network time consumption is the time consumption of actually transmitting a complete IO request in a network link for one round after completing 4 processes of client-side output, server-side input, server-side output and client-side input; the client time consumption is obtained by subtracting the server time consumption and the network time consumption from the whole time consumption of a complete IO.

4. The method for locating a fault in a network file system according to claim 1, wherein the step of triggering the threshold judgment of the average delay of the server, the average delay of the network and the average delay of the client at corresponding time points one by one to obtain the analysis result for feeding back the fault location comprises:

Comparing the average delay of the server with the delay threshold of the server, and if the second difference value of the average delay of the server minus the delay threshold of the server is smaller than or equal to a second preset threshold, the server is normal;

and if the second difference value of the average delay of the server minus the delay threshold of the server is larger than a second preset threshold, comparing the average delay stored by the server with the delay threshold stored by the server.

5. The method for locating a failure of a network file system according to claim 4, wherein comparing the server storage average latency with the server storage latency threshold comprises:

If the third difference value of the average storage delay of the server minus the storage delay threshold of the server is smaller than or equal to a third preset threshold, the metadata access is abnormal;

If the third difference value of the average storage delay of the server minus the storage delay threshold of the server is larger than a third preset threshold, the data storage is abnormal.

6. The method for locating a fault in a network file system according to claim 1, wherein the step of triggering the threshold judgment of the average delay of the server, the average delay of the network and the average delay of the client at corresponding time points one by one to obtain the analysis result for feeding back the fault location comprises:

Comparing the network average time delay with the network time delay threshold, and if the fourth difference value of the network average time delay minus the network time delay threshold is smaller than or equal to a fourth preset threshold, the network link is normal;

If the fourth difference value of the network average time delay minus the network time delay threshold value is larger than a fourth preset threshold value, the network link is abnormal.

7. The method for locating a fault in a network file system according to claim 1, wherein the step of triggering the threshold judgment of the average delay of the server, the average delay of the network and the average delay of the client at corresponding time points one by one to obtain the analysis result for feeding back the fault location comprises:

comparing the client average time delay with the client time delay threshold, and if the fifth difference value of the client average time delay minus the client time delay threshold is smaller than or equal to a fifth preset threshold, the client link is normal;

if the fifth difference value of the client average time delay minus the client time delay threshold is larger than a fifth preset threshold, the client link is abnormal.

8. A failure locator for a network file system, comprising:

The data acquisition module is used for acquiring real-time overall time delay and real-time server storage time delay, and carrying out NFS protocol data packet acquisition on the client and the server through tcpdump so as to obtain original data; performing unique xid field matching on the original data to obtain complete transfer information of each IO atom operation;

The data extraction module is used for obtaining average time delay data of corresponding time points based on complete transmission information of all IO atom operations; the average delay data comprises an overall average delay, a server storage average delay, a server average delay, a network delay and a client average delay;

the threshold configuration module is used for determining threshold time delay data; the threshold delay data comprises an overall delay threshold, a server storage delay threshold, a server delay threshold, a network delay threshold and a client delay threshold;

And the performance analysis positioning judgment module is used for comparing the overall average time delay with the overall time delay threshold, and if the first difference value of subtracting the overall time delay threshold from the average overall time delay is larger than a first preset threshold, triggering the threshold judgment of the server storage average time delay, the server average time delay, the network average time delay and the client average time delay at corresponding time points one by one to obtain an analysis result for feeding back the fault position.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that the processor implements the method of fault localization of a network file system according to any of claims 1 to 7 when executing the computer program.

10. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements a method of fault localization of a network file system according to any one of claims 1 to 7.