JPH11296311A - Fault-tolerant control method for storage devices - Google Patents

Abstract

PROBLEM TO BE SOLVED: To judge a fault part with high precision, to separate the part from a system and to suppress the continuous occurrence of the fault by taking the statistics of the times of the fault occurring in the system at every part and automatically executing a series of operations including the specification of the fault part through the use of it during a system work. SOLUTION: One kind of counter is assigned at every part kind constituting the system per each fault kind in order to count the times of fault occurrence in the respective parts and the times of the fault are individually cumulated at every element number which exists in each kind. When the fault occurs in the data transfer system function of a certain specified channel adaptor(CHA) 3, only the data transfer fault counter of CHA 3 indicates a high value. In this case, a threshold value to be a base concerning the times of the fault at every part is provided. Then, overall judgement is executed at the point of time when the fault counter at a certain part exceeds the base threshold value. When the fault at an individual part is judged after the judgement, the connected part is separated from the system.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0003] On the other hand, in the configuration of a computer system, it is common to have a plurality of logical access paths (hereinafter, logical paths) but also have a physically common part. For example, a bus connection system for connecting each component to a common bus is widely used because the system configuration can be easily changed and expanded. However, having a common part makes it difficult to identify a failure part, for example, an individual failure affects the whole, or one failure occurs simultaneously with another failure.

[0002] The present invention proposes one solution to such a problem of identifying a failure site. By improving the accuracy of specifying the failure site, the failure site can be accurately separated, and as a result, the reliability of the entire system can be improved.

[0003]

2. Description of the Related Art In recent years, there has been an increasing demand for reliability of storage devices. In particular, when constructing a non-stop system, a highly reliable storage device became indispensable. Therefore, each component of the system including the control unit is provided with redundancy, and even if a faulty part is specified and separated, the system operation by the substitute component can be continued.

[0004] In many cases, a faulty part is immediately identified by one fault detection based on the detected fault content, and the part is closed. However, for this purpose, a complicated and sophisticated fault detection configuration is required, and temporary noise that does not occur permanently can be immediately blocked, thereby reducing the redundancy of the system.

On the other hand, there is also a method in which the number of times of failure detection is accumulated for each hardware part, and if the number exceeds a fixed threshold value set in advance, the part is determined as a failure occurrence part and is separated from the system. Temporary noise that does not exceed the threshold does not lead to blockage, so that overblocking can be avoided to some extent. However, in a system that has a system common part and each part is complicatedly intertwined, 1
A plurality of logical paths may be affected by a failure at one location, and the number of failures may be added at a plurality of locations. In such a case, it is difficult to accurately specify a failure part only by a simple threshold value check.

[0006]

What is particularly important in a failure recovery process in a storage device is to extract and separate a failed part while the system is operating. However, in systems that have common parts, such as a bus connection method that connects each component of the system to a common bus,
A single failure may cause a failure to spread throughout the system, and a part that has not failed may temporarily not operate normally. Even in such a case, it is an important issue in securing system reliability how to correctly determine a failure site which is a failure source and to stop the recurrence of the failure.

[0007] When a single failure occurs in a system having a plurality of logical paths, the application of the method of the present invention makes it possible to point out a failed portion with a high probability, and by isolating the failed portion from the system, guarantees continuous operation of the system. Also,
For a part such as a common bus that is complicatedly entangled with each logical path, the failure occurrence status can be continuously monitored even after the failure part is closed, and when the failure does not converge, recovery is performed by recovering the temporarily blocked bus.

[0008]

SUMMARY OF THE INVENTION In order to achieve the above object, the present invention divides a system into a plurality of separable parts, counts the number of occurrences of each fault, and determines a faulty part from comprehensive judgment of the number of faults. Perform identification. Here, the comprehensive judgment means that the failure judgment of one part is not only the number of failure occurrences of that part but also the number of occurrences of failures of other parts and the entire system. If a failure occurs, it is expected that a failure will be detected in each logical path that uses it, and even if the number of failure detections for a single part exceeds the threshold value first, the number will prominently increase compared to other parts. If this happens, there is a suspicion of a common part failure,
The logic is such that the single site is not easily closed.

[0009] Specifically, for example, it is assumed that there are a plurality of paths sharing a certain common part, and the frequency of use of each path is substantially the same. When a failure occurs in a common part, the number of failures in each path should be close to the arithmetic mean of the whole. In the case of a failure in a single part that affects only one path, the number of failures in that path is large, and failures in other paths occur. There should be few times. By calculating the similarity between the number of failures actually counted for each path and the expected result, it is possible to determine whether a failure in the common part is a failure in the single part.

By using the method of the present invention in a storage controller having a plurality of logical paths, it is possible to distinguish between a failure in a common part of a plurality of paths or a failure in an individual part affecting only a single path. If a failure has occurred in an individual part, the logical path is closed and disconnected from the system to suppress the influence on the entire system. When it is determined that a failure has occurred in the common part, if the system has redundancy, it is considered that the system operation can be continued without unnecessary path blockage by performing partial degeneration of the common part.

In addition, according to the method of the present invention, the number of times of occurrence of a failure is counted even after executing the partial degeneracy of the common part, and if the failure does not converge, the degenerated common part is recovered and other parts of the common part or It is possible to occlude individual parts. Even if there is a plurality of common parts that are always used simultaneously and cannot be separated from the number of failure occurrences, if one part is closed first and then the subsequent failure occurrence is monitored,
It is possible to verify whether the faulty site has been correctly excluded.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram of a storage controller to which the present invention is applied. A channel connection system 1 connected to a host computer, a cache memory 2 for temporarily storing input / output data, a channel adapter (hereinafter, CHA) 3 for controlling data transfer between the channel connection system 1 and the cache memory 2, and data storage Array 4, cache memory 2, and disk array 4 as storage media
Disk adapter (hereinafter referred to as DKA) 5 for controlling data transfer between them, shared memory 6 for storing system management information and communication information, etc., and common bus 7 for connecting each of CHA 3 and DKA 5 to cache memory 2 or shared memory 6
Composed of

Each component is provided with redundancy in order to prevent a system operation stoppage due to a failure of a single part. That is, a plurality of CHAs 3 and DKAs exist in the system.

The cache memory 2 and the shared memory 6 have a two-sided structure that can be separated from each other. Here, the cache memory A side 21 and the cache memory B side 2
2, called the shared memory A side 61 and the shared memory B side 62,
In case of any single-sided failure, operation is possible with only one normal side.

The disk array 4 includes a parity disk, and can operate continuously with any one disk closed.

The common bus 7 includes an H bus 71, an L bus 72, and an M bus
The bus 73 includes three buses. The H bus 71 is used for accessing the cache memory 2, and the M bus 73 is used for accessing the shared memory 6. Normally, L bus 72
Is designed to provide high-speed access to the cache memory 2 (a double bus width is realized by simultaneous use) in cooperation with the H bus 71 by designating preset system option information.
Alternatively, it can be used independently for accessing the shared memory 6. When the H bus 71 fails, the L bus 72 is used for accessing the cache memory 2, and when the M bus 73 fails, the shared memory 6 is used.
By using it for access, the system can continue to operate even if one bus fails.

When the system of this embodiment is operating, many accesses using a plurality of access logical paths (hereinafter, logical paths) operate simultaneously. For example, a logical path for transferring data from the channel connection system 1 to the cache memory A surface 21 via a certain CHA 3, a logical path for transferring data from another CHA 3 to the shared memory B surface 62, or a DKA 5 for transferring data from the cache memory B surface 22. Disk array 4 via
Access using various paths, such as a path for transferring data to, operates simultaneously.

During the operation of the system, if a failure occurs in a specific hardware part and the logical path using the part becomes permanently inaccessible, the failure part can be specified and isolated from the system by a fault isolation test. it can. The fault isolation test here is, for example, logic for switching only one part on the failure detection path to another and performing an access attempt, and judging from the result as to whether or not the part is faulty. As an example, from a certain CHA3 to an H bus 71
When performing data transfer to the cache memory 2 in the bus mode in which the CHA and the L bus 72 are used at the same time, when a failure of a data parity error is detected, in the failure isolation processing, the CHA passes through the H bus 71 and the L bus 72, and An access test is performed on the cache A surface 21 and the cache B surface 22. For example, if a failure is detected in both accesses using the H bus 71 and no failure is detected in the two accesses using the L bus 72, the failure of the H bus 71 can be determined. In this case, the H bus 71 is closed, and only the L bus 72 is switched for data transfer, so that the system can continuously operate.

However, actual hardware failures are not always permanent, and temporary failures may occur frequently. It is also conceivable that a failure occurs only at a specific timing and a specific access pattern. At that time, in the failure isolation test, the access test using the failed part is also normally completed, and the failed part cannot be specified in many cases.

Here, the number of times of occurrence of a fault is counted for a case such as a temporary fault other than the one that can be specified by the fault isolation test, and a faulty portion is specified using comprehensive threshold judgment.

First, the counting method of the number of times of occurrence of a failure will be described, and then the logic of the overall threshold value determination will be described.

For convenience of explanation, faults in the storage controller of this embodiment are limited to two types: a data transfer system fault and a shared memory information access fault. In order to count the number of failure occurrences in each part, one type of counter is assigned to each failure type for each part type constituting the system, and the number of failures is separately accumulated for each number of elements in each type. For example, the number of occurrences of a shared memory information access failure in a certain CHA 3, the number of occurrences of a data transfer failure in the cache memory A surface 21, the number of occurrences of a data transfer failure in the L bus 72, and the like are respectively counted.
As a feature of this method, not only the number of faults of the entire system is simply accumulated, but also a logical path used at the time of detecting a fault is analyzed, and each part existing on the path is counted up.

Now, let us consider a case where a data transfer function of a specific CHA 3 breaks down and temporary failures frequently occur. At that time, only the data transfer failure counter of the CHA3 indicates a high value, and the counters of the other CHA3 or DKA5 are not counted up. Since both sides of the cache memory 2 are usually accessed with almost equal probability, data transfer from the failed CHA 3 is distributed to both sides,
Cache memory A side 21 and cache memory B side 22
Are considered to have almost the same value.

Further, consider a case where a failure occurs in the M bus in a system in which the H bus 71 and the L bus 72 are used for data transfer and the M bus 73 is specified for shared memory information access. In this case, the M bus 73 is a complete system common part, and all the operating CHAs 3 and DKA 5 access the shared memory 6 via the M bus 73.
Failures are equally detected in each of the CHAs 3 and DKAs 5, and the number of failures in the shared memory A surface 61 and the shared memory B surface 62 also becomes substantially the same value.

Based on the fact that the number of failure counters has the two above-mentioned tendencies, a determination logic for specifying a failed part using each counter value in the system will be described.

First, a base threshold value is set for the number of failures of each part. If the failure count value does not reach the threshold value, it is determined that there is a possibility that noise may occur even in a normal system, and a comprehensive judgment is made when the failure counter of a certain part exceeds the base threshold value. .

At the time of judgment, all counters of the same type as the counter exceeding the base threshold value are extracted and used as the material for judgment. Here, the counters of the same type indicate counters of other parts which have the same function and operate independently. For example, when the number of data transfer failures of a certain CHA3 exceeds the base threshold, the data transfer failure counter value of another active CHA3 is also extracted. Based on these, it is determined whether the count value of the part where the failure is first detected is significantly larger than the other count values, or whether the value of each counter indicates a close value.

As an example of the determination method, there is a method of comparing, by calculating similarity, how close the actual number of failure times distribution is to the ideal number of times distribution in the above two cases. In the case of an individual part failure, ideally, only the number of failure detections of the part reaches the total value of the extracted counters, and the other counter values become zero. Here, the sum of the actual count value of each part and the distance (square of the difference value) between the ideal value and the ideal value is obtained. The smaller this calculated value is, the closer to the ideal distribution of the individual part failure is. If the distribution is closer to the ideal distribution than a reference value provided in advance, it is determined that a failure has occurred in the relevant part. Similarly, in the case of a common site failure, ideally all count values are equal to the average number of times for each site. Similarly, if the sum of the distance of the ideal value (square of the difference value) is obtained from the actual count value, the similarity with the ideal distribution of the common part failure can be obtained. If it is closer to the reference value or more, it is determined that a failure has occurred in the common part.

Here, in the present embodiment, the comparison is made with the above two types of ideal distributions, but a hardware configuration having another fault distribution is also conceivable. For example, in a case where a failure is detected in a specific part at a probability twice as high as that of another part in a failure of a common part, similar similarity calculation can be performed by preparing a corresponding ideal distribution. Further, a method of counting the actual number of accesses to each part and dynamically reflecting the expected number of failures according to the number of accesses to the ideal distribution can be easily considered.

If it is determined that a failure has occurred in an individual part through the above determination, the part is separated from the system. For example, if it is specified that the cache memory A surface 21 has failed, the system management information is updated so that the cache memory A surface 21 is not used, and the system operation is performed using only the normal cache memory B surface 22.

When it is determined that the common part has failed, the system operation is continued so that the common part is degenerated as much as possible without affecting the whole. For example, when it is determined from the shared memory information access failure that the M bus 73 has failed, the L bus 72 is switched for shared memory access, and the M bus 73 is switched.
Stop access through.

After the failure site is determined and the failure is closed, the count value of the number of failure occurrences is cleared.

There may be a case where a common part is used at the same time and a failed part cannot be uniquely determined. For example,
The system of this embodiment is, as mentioned at the beginning, H bus 71
And the high-speed transfer bus mode combining the L bus 72 can be designated. When such a system option is specified, even if the H bus 71 or the L bus 72 alone fails,
Since the two buses are used at the same time, the same number of failures is counted for the two buses, and it is not possible to determine which bus has failed. In consideration of this case, a logic for closing the H bus 71 first is included. Then, the failure occurrence status after the execution of the blockage is continuously monitored, and when it again exceeds the base threshold and is similarly determined to be a common unit failure, the previous determination result is taken over, and the H bus 71 is recovered. The L bus 72 is controlled to be closed.

The logic for monitoring the occurrence of a failure after the blockage and recovering the blockage once again is not only necessary for separating the plurality of portions used at the same time.
It is clear that it is also effective for correction after erroneous failure point out due to some factor.

[0036]

According to the storage control device to which the method of the present invention is applied,
Performs statistics of the number of failures that occur in the system for each part, and uses it to perform a series of operations including identification of failure parts,
This can be done automatically while the system is running. As a result, even in a system having a common part such as a common bus connection and in which it is difficult to specify a failure part, the failure part can be determined with high accuracy, and as a result, the failure part is separated from the system,
Subsequent failures can be deterred.

[Brief description of the drawings]

FIG. 1 is a block diagram of a storage control device according to an embodiment.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Channel connection system, 2 ... Cache memory, 3 ... CHA (channel adapter), 4 ... Disk array, 5 ... DKA (disk adapter), 6 ... Shared memory, 7 ... Common bus, 21 ... Cache memory A side, 22 ... Cache memory B side,
61: Shared memory A side, 62: Shared memory B side,
71 ... H bus (data transfer dedicated bus), 72
... L bus (switchable bus for data transfer and shared memory access), 73 ... M bus (dedicated bus for shared memory access).

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Hisao Honma 2880 Kozu, Odawara-shi, Kanagawa Prefecture Storage Systems Division, Hitachi, Ltd. (72) Inventor Osamu Sakaguchi 6-81-8 Ouecho, Naka-ku, Yokohama-shi, Hitachi Hitachi Software Engineering Co., Ltd.

Claims (3)

[Claims] In a storage device control unit composed of a plurality of components, the number of fault occurrences is counted for each separable portion by bus connection, and a faulty portion is determined and analyzed by statistical analysis using the count result. Fault-tolerant control system that automatically disconnects from the system. 2. The fault location judging method according to claim 1, wherein the fault occurrence status after the fault location is separated is continuously monitored. Fault-tolerant control method that enables continuous feedback such as recovery. 3. In the case where it is difficult to specify a faulty part by only one fault detection, the control method according to claim 1 or 2 can prevent erroneous determination of a faulty part and avoid a system down. Storage device for the purpose.