JP2008097164A - Fault monitoring method for a system composed of a plurality of functional elements - Google Patents

Abstract

【Task】
In a system in which a large number of functional elements are connected via a network, the reliability of component failure detection and backup is improved, and the system availability is improved.
[Solution]
The above-mentioned problem is solved by determining the failure element and determining whether backup is possible with a plurality of components or by majority vote. Further, by means of mutual monitoring between the sub-elements inside the component, and when the sub-element within the component fails, the other components in the same element can be stopped and the external interface can be separated from the external interface. The reliability of the backup operation can be improved. In addition, it is possible to secure the certainty of the failure element operation stop by means of detecting a failure in the component element among a plurality of sub-elements or by majority vote between the sub-elements and stopping the self-operation.
[Selection] Figure 1

Description

  The present invention relates to a failure monitoring method in a computer system or communication system composed of a plurality of functional elements.

  As a fault monitoring and fault handling method for communication networks consisting of multi-CPU systems, massively parallel computers, and many stations and relay stations, fault monitoring of the active system by a single standby system, backup processing when a fault occurs, etc. (Patent Document 1). In addition, the processing control unit in its own unit is monitored, and when a failure is detected, the processing control unit is reset and restarted (Patent Document 2).

Japanese Patent Laid-Open No. 06-231095 JP 2000-148709 A

  However, the conventional technique does not consider a case where the monitoring target is mistakenly recognized as having failed due to a failure in the failure monitoring circuit on the failure monitoring side. For this reason, if the monitoring mechanism that performs backup or failure processing fails, the entire system may be seriously affected. In addition, in order to prevent the malfunctioning element from adversely affecting other parts, when stopping the operation of the malfunctioning functional element, forced stop means from other elements or manual intervention was necessary. The cost of manual labor has increased.

  An object of the present invention is to provide a problem that the fault monitoring circuit erroneously performs fault processing in the case of a fault in the conventional technique and a means for stopping the operation of the faulty element at a low cost.

  The present invention includes (1) the feature of determining the location of failure and whether or not backup is possible by matching or majority voting among a plurality of components, and (2) a functional element to prevent the failure element from adversely affecting the system. Features of mutual monitoring between constituent sub-elements, stopping the operation of the own element in the event of a failure in the own element, and disconnecting the interface with other elements, and (3) whether or not the operation of the own element can be stopped The sub-elements are matched or majority voted. Furthermore, in a system in which a large number of functional elements are interconnected via a network, as one method for realizing the feature (1), a plurality of functional elements are grouped with two or more network nodes, and the group In particular, it has the characteristics of specifying a failure location, determining whether backup is possible, and performing backup processing.

  In the present invention, since the failure element is specified by a plurality of elements, it is possible to suppress erroneous indications due to a failure of the failure monitoring circuit and the occurrence of an inappropriate backup operation. In addition, mutual monitoring by sub-elements within each functional element and the self-operation stop function can realize interface separation from other parts so that abnormal operation of the self-element does not affect the system. Even if there is no circuit or network for stopping, it is possible to prevent an adverse effect on the system due to a failure element. In addition, since the CPU can be stopped and backed up without manual intervention, system operation costs / maintenance costs can be reduced. The mutual check between the sub-elements usually uses a check circuit such as timer monitoring or data check used at the interface between the sub-elements, so that a large additional circuit is unnecessary.

  Furthermore, in a system in which a large number of functional elements are interconnected via a network, the functional elements and network nodes can be grouped and mutual monitoring and backup can be performed within the group, so a system in which a small number of functional elements are connected via a network. You can build a backup mechanism as easily as

  A mechanism that identifies the failed element with multiple functional elements to prevent erroneous recognition of the backup trigger due to the monitoring circuit failure of the monitoring element and to prevent the backed up element from disturbing the system operation after backup, A system having both the mutual monitoring and self-operation stop function by the sub-elements is the best embodiment. Furthermore, in a system in which a large number of functional elements are connected via a network, several functional elements and network nodes are grouped, and each functional element is connected to a plurality of network nodes in the group and mutually connected within the group. A system that performs monitoring and backup is the best embodiment.

  FIG. 1 is a schematic configuration diagram of a massively parallel computer system according to the first embodiment.

  From this embodiment onward, the operation stop of the own element and the disconnection of the external interface will be referred to as “self FREEZ”.

  The processor units 101 are connected to each other by a network node 102. Two processor units 101 and two network nodes 102 are grouped, and the processor unit 101 is connected to a network node corresponding to its own unit by a signal line group 103. In order to cope with a communication disconnection due to a failure of the node 102, it is also connected to a neighboring network node in the group by another signal line group 104. Network node 102 is also interconnected with network nodes outside the group by network signal 105.

  FIG. 2 is a schematic configuration diagram of the processor unit 101. The main CPU 201 is connected to communication ports 202 and 203 via switching logic 204. In the interaction monitoring table 205, the states of the own unit P (2p) j, pair unit P (2p + 1) j, network node N (2p) j, and pair node N (2p + 1) j are registered (p 0 ~ k). The latch 206 is a latch for holding the self-FREEZ state, and is set when the set signal 211 from the main CPU 201 or the FREEZ assert signals 212 and 213 of both the communication ports 202 and 203 are simultaneously turned ON. This latch is reset at power-on or system reset, but is not shown in this figure.

  FIG. 3 is a schematic configuration diagram of the network node 102 in FIG. Reference numeral 301 denotes a communication port, and the configuration is the same as that of the communication port in the processor unit 101 in FIG. The port needs 4 ports for communication between the network nodes 102 and 2 ports for communication with the processor unit 101. One for communication with the processor unit 101 is a backup port. The SWC 302 is a switching controller that performs network route control. Although the SWC 302 can be duplicated to increase the availability of each network node, in this embodiment, the SWC 302 has a single configuration. This is because it is necessary to stop the operation of the network node when the SWC 302 is abnormal, but communication between the processor units is possible via another network node by a detour path. The circuit 303 is a FREEZ majority circuit, and its logical configuration is shown in FIG. Necessity of self-FREEZ is determined by majority decision of FREEZ approval signal 306 from a total of seven components of communication port 301 and SWC 302. The table 305 is an inter-operation monitoring table. Similarly to FIG. 2, the processor unit P (2p) j, the pair unit P (2p + 1) j, the own network node N (2p) j, N (2p) j The state of pair node N (2p + 1) j is registered.

  FIG. 5 is a schematic flow of failure monitoring / failure processing of the main CPU 201 in FIG. When the power is ON, NC0 (202) is used as a communication port, but the spare port 203 is also polled to check each other's normality. If an abnormality of the working port 202 is detected during communication operation, the spare port 203 is tested.If the spare port 203 is normal, the communication port is switched to 203. Self-freeze and stop operation.

  FIG. 6 is a schematic flow diagram of failure monitoring / processing of the communication ports 202 and 203 in FIG. The left side shows the processing of the working port, and the right side shows the processing of the spare port. Both the active and standby ports monitor the operation of the main CPU (603, 613) as well as the mutual operation monitoring (605, 615). When an abnormality is detected in the main CPU 201, the FREEZ assert signal 212 or 213 is turned ON. However, the processor unit 101 does not FREEZ unless the other communication port detects an abnormality of the main CPU and turns the FREEZ signal ON. Even when the abnormality of the counterpart port is detected, the FREEZ assert signal 212 or 213 from the own port is turned ON (607, 617), but the processor unit 101 is FREEZed only when the FREEZ signal of the counterpart port is turned ON. When an abnormality is detected in the own port test (606, 616), the own port is FREEZed, but details of this part are omitted. Basically, normality is tested by mutual monitoring between the subunits in the own port, self-test, etc., and self-freezes.

  FIG. 7 shows a schematic operation of the SWC (switching controller) 302 in FIG. During normal communication processing, communication with each communication port 301 is performed, or periodic polling is performed at regular time intervals to check the normality of each communication port 301 (702, 703). If an abnormality is detected in one of the NC 301, the SWC 302 confirms its own normality by a self test (705), and then determines whether or not the SWC itself self-freezes based on the number of abnormality of the NC 301. Since there are four communication ports between network nodes, communication with other nodes can be performed if two normally operate. Therefore, when three or more ports are determined to be abnormal, the self-FREEZ signal is turned ON. The self-FREEZ signal is asserted only when both of the communication ports with the processor unit are abnormal. Also, if it is determined in 705 that the own subunit SWC302 is abnormal, the own network node can no longer perform accurate communication, so the own FREEZ signal is turned ON. SWC302 self tests include tests 711-713.

  FIG. 8 is a diagram showing a schematic operation of the communication port 301 in FIG. Since each port communicates with each other via the SWC 302, when there is communication data, the normality of the SWC 302 can be confirmed together with the partner port. When there is no communication data, mutual monitoring is performed by polling to confirm normality. If there is no response from the other party for a certain period of time, a self-test is first performed, and after checking the normality of the self (804), it is distinguished whether the other communication port is abnormal or the SWC 302 is abnormal (806). Although this distinction method is not described in detail, the distinction can be made based on the routing data of the communication data, whether or not the communication response is normally returned from the SWC 302, and the like. Whether the SWC 302 is abnormal or the communication port is abnormal, the self-FREEZ signal 306 output from the local port is turned ON (810). When the own port abnormality is confirmed in the own port normality confirmation (805), a notice of own port abnormality is sent to the other port 301 or SWC 302 (807), and FREEZ of the own port is performed (809). At the time of own port FREEZ, the FREEZ assert signal 306 of the network node 102 is also turned ON. The 804 self-test has items such as 821 to 824, but will not be described in detail. In the case of a runaway program in the SWC, the polling response may be accepted periodically, so the reliability can be improved by performing communication status transition and command / response validity check.

  FIG. 9 is a diagram showing an overview of mutual monitoring / backup processing of the processor unit 101 and the network node 102 in FIG. When the power is turned on or reset, the contents of the mutual monitoring table are set to “Starting” and “Normal” for all elements. During normal communication processing (902), data communication or polling is performed at regular time intervals. When a data abnormality or polling abnormality is detected, a self test is first performed to test the normality of the own element (804, 805). When the normality of the own element is confirmed, the status of the partner element in the mutual monitoring table is set to “abnormal” (906). Next, the mutual monitoring tables 205 and 305 are exchanged with other normal elements. It is checked whether the abnormal element matches the table received from the other element and the mutual monitoring table of the own element (908). If the abnormal element matches, it is determined whether the type of the abnormal element is the same as its own element (909) .If the abnormal element is the processor unit 101, the other processor unit 101, and if the abnormal element is the network node 102, another network node 102 starts the backup operation (910). At this point, the abnormal element detects its own abnormality and self-freezes, so the backup operation does not adversely affect the system. If an abnormality of the own element is detected in the normality determination 905 of the own element, the own element in the mutual monitoring tables 205 and 305 is set to “abnormal” (921), and the abnormality of the own element is notified to other elements. (925), FREEZ of own element. Specific items of the self-test 904 include 931 to 934. In the case of a runaway of the main processing part that is one of the sub-elements within its own element, the normality may not be confirmed only by polling response check. Can improve.

  Since the configuration is as described above, in the first embodiment shown in FIGS. 1 to 9, the abnormal element is specified and whether or not to start the backup operation is determined by majority decision of a plurality of components. The possibility of disturbance can be eliminated. In addition, the sub-elements that make up each element are mutually monitored, the abnormality of the own element is detected, the operation of the own element is stopped and the interface with other elements is shut off at the same time, so the external interface due to runaway of the internal CPU, etc. As a result, it is possible to prevent the entire system from being disturbed. In addition, since a pair of two network nodes and two processor units perform mutual monitoring, a backup system can be configured with a configuration equivalent to a two-node system, even though it is a massively parallel computer system composed of many elements. . As a result, even if one CPU fails, there is no need to stop / replace LSI, MCC, etc., and there is no need to stop the failed CPU or perform manual intervention during backup, thereby reducing system operation costs and maintenance costs.

  FIG. 10 is a configuration diagram of a shared bus connection calculation unit according to the second embodiment of the present invention. The configuration of FIG. 1 is realized by a printed circuit board, MCC, and some highly integrated LSIs. In FIG. 10, N CPUs 1001 perform data transfer and bus use right control via a shared bus 1009. The CCU mutual monitoring table 1002 is a table showing the state of each CPU 1001, and shows the state of each CPU viewed from the corresponding CPU. Data in the CCU mutual monitoring table 1002 is exchanged between the CPUs via the shared bus 1009. The table 1002 also registers its own CPU status. The contents of the table 1002 are checked by each CPU and programmed to be processed by the flow shown in FIG. The CPU that performs backup in the processes 1205 and 1207 in FIG. 12, for example, is performed by the CPU of the number obtained by the equation [(Z + 1) / N] (the remainder of N), where Z is the number of the failed CPU. In addition, it is possible to use a determination method such as a CPU with a low CPU load. FREEZ voting circuit 1003 receives each part monitoring status data output from execution control unit 1005, CPU-BUS control unit 1006, memory access control unit 1007, and IO-BUS control 1008. Whether to perform self-FREEZ (self operation stop and disconnection from the shared bus 1009) is determined under the conditions shown in FIG. Although a specific circuit configuration is not shown, the configuration is the same as the FREEZ decision logic of FIGS. 2 and 4 of the first embodiment. Although details of the monitoring status data of each part are not shown, it has the same structure as the CCU mutual monitoring table 1002.

  As is apparent from Table 1, the execution control 1005 is configured such that its own CPU is FREEZed only when all other parts are recognized as abnormal. In the case of an abnormality of a sub-element other than the execution control unit 1005, if two of the other three are determined to be abnormal, the operation proceeds to the FREEZ operation.

The mutual monitoring method of each part 1005 to 1008 differs depending on the interface with other parts, but there are the following methods. The following (3) is used for mutual check between parts where there is no interface in normal processing. In cases such as program runaway, abnormalities may not be detected in (1) to (3), but can be detected with the validity of the interface response content as in (4).
(1) Check of data output by monitoring part (parity check, ECC, etc.)
(2) Response time monitoring (3) Live signal monitoring from monitoring part (4) Command / response validity check Node1101 in Fig. 11 corresponds to the calculation unit in Fig. 10, and N CPU1001s are installed. Yes. As a whole, a computer system in which M Nodes 1101 are connected by a shared bus.

  The Node 1101 is connected by an inter-node bus 1109, and transmits and receives transactions such as data exchange and bus use right control. The FREEZ majority logic 1103 is a circuit that determines whether or not the own node can be FREEZed (stop of own node operation, bus output between nodes is turned off) based on the FREEZ signal 1105 of the own node transmitted from each CPU 1001 in the node. Although details of the logical configuration are not shown, the configuration is the same as the FREEZ decision logic 2 of FIGS. 2 and 4 of the first embodiment. The self-FREEZ circuit 1104 is a self-FREEZ state holding circuit. When the self-FREEZ latch in this circuit is ON, the self-FREEZ state is entered, the operation of the CPU, etc. is stopped, and the output to the external bus is logically OFF. The circuit 1104 also includes a self-FREEZ latch Reset circuit at the time of system reset. The circuits 1103 and 1104 are mounted in the same components as the BUS control circuit 1106 in order to reduce the number of components.

  FIG. 12 is a schematic process flow of the self-FREEZ circuit. In the normal state, 1201 to 1203 are looped and the state is monitored. When the presence of the abnormal CPU is reported in the CPU mutual monitoring data, the process proceeds to 1204, the mutual monitoring data 1002 is exchanged between the CPUs 1001, and it is checked whether or not the abnormal CPUs recognized by the CPUs match. If they match, backup processing 1205 of the CPU is unconditionally performed. If the abnormal CPUs do not match at 1204, the number of other CPUs that point out that each CPU is abnormal is checked (1206). If there are a plurality of CPUs that point out that one CPU is abnormal and there is only one CPU 1101 that is abnormally pointed out, the CPU 1101 determines that it is normal and moves to a process 1207. Abnormal processing and backup processing of the specified CPU 1101 are performed. If there are multiple CPUs pointed out as abnormal by multiple CPUs, the system abnormal processing 1208-1209 is processed as a multipoint failure or shared BSU failure, and the self-FREEZ of the own node output by the own CPU Turn on signal 1105.

  If a failure occurs in the bus interface circuit inside a certain CPU, the CPU 1001 cannot communicate with all the other CPUs 1001, and the CPU may record all the other CPUs as abnormal and recorded in the mutual monitoring data. is there. However, other CPUs register the CPU as abnormal because the failed CPU is the only abnormal CPU. For this reason, the process proceeds from 1206 to 1207 to start the backup process.

  FIG. 13 shows an operation flow of mutual monitoring between nodes and abnormal node majority decision logic. Steps 1301 to 1307 are the same as the mutual monitoring / majority decision circuit of the CPU in FIG. 12, but in the processing 1308, only the system abnormality processing is performed and the operator is notified without self-freezing.

  In the second embodiment (FIGS. 10 to 13), in a system composed of a plurality of CPUs and Nodes, whether backup is necessary or not is determined by mutual majority vote, so that an erroneous misidentification of the active CPU or Node is prevented and normal operation is in progress. It can prevent erroneous disable of CPU and Node and startup of wrong backup operation, and improve system reliability and operability. In addition, because the self-elements are monitored by mutual monitoring between sub-elements, which are functional elements inside the self-element, and self-freez (suppression of self-operation and disconnection of the external interface) is performed at the time of abnormality, an abnormality caused by another CPU or node Disabling related logic / signal lines between a large number of CPUs / Nodes can be reduced, backup operations can be easily implemented, and LSIs that have been discarded due to some failures in the past The parts can be used for life extension without replacement. Since the second embodiment is connected to a shared bus, one node 1101 is a grouping unit of a CPU and a network node without being aware of it, and claim 4 is realized.

Schematic configuration diagram of massively parallel computer system Schematic configuration diagram of the processor unit Schematic configuration diagram of network node Logical configuration diagram of majority logic General operation flow of main CPU in processor unit Outline operation flow of communication port in processor unit Schematic operation flow of switching controller in network node Outline operation flow of communication port in network node Outline operation flow of failure monitoring and processing of processor unit and network node Schematic diagram of multi-CPU node with shared bus connection Schematic configuration diagram of shared bus connection system of multiple nodes Outline operation flow of failure monitoring and processing in multi-CPU node Outline operation flow of failure monitoring and processing of shared bus connection system of multiple nodes

Explanation of symbols

106: Mutual monitoring / backup unit in massively parallel computer, 205: Mutual monitoring table in processor unit 101, 206: Self FREEZ latch in processor unit 101, 305: Mutual monitoring table in network node 102, 303: Self FREEZ majority logic, 304: Self-freez latch in network node 102, 1002: CPU mutual monitoring table, 1003: CPU1001 self-freez majority logic, 1004: Self-freez latch in CPU1001, 1102: Node mutual monitoring table , 1103: Self-freeze majority logic of Node1101 1104: Self-freeze latch in Node1101

Claims (5)

  In a system in which multiple functional elements are connected via a network and the failed functional element is stopped to perform a degraded operation, whether or not the identification of the failed functional element is consistent among other functional elements A failure monitoring method for a system composed of a plurality of functional elements, which is determined by judgment or majority vote between functional elements, and which backs up the functional elements specified by the determination.   In the system, the sub-functional parts in each functional element are mutually monitored between the sub-functional parts, and if the sub-functional part cannot operate as a single functional element due to abnormality of the sub-functional part, the operation of the own element is stopped and other functions are stopped. 2. The fault monitoring method for a system composed of a plurality of functional elements according to claim 1, wherein the self-element is removed from the system configuration by separating an interface with the element.   3. The plurality of functional elements according to claim 2, wherein the self-operation stop of the functional element is carried out by failure detection recognition of two or more sub-functional parts or by majority vote between the sub-functional parts. Fault monitoring and processing method for configured system   In a system in which four or more functional elements are connected to each other via a network, each of the plurality of functional elements is grouped with two or more network nodes and connected to each other within the group. A failure monitoring method for a system composed of a plurality of functional elements, characterized by monitoring and performing backup in the group when a functional element or network node in the group fails. In the system, the determination of whether or not the identification of the functional element or network node in which the failure has occurred is the same among a plurality of other functional elements or network nodes in the group, or is determined by majority decision in the group 5. The fault monitoring method for a system composed of a plurality of functional elements according to claim 4, wherein the functional elements or network nodes specified by the determination are backed up.

Images