JP2014178793A - Information processing system - Google Patents

Abstract

An information processing system capable of improving reliability is provided.
Processing blocks 310A to 310C that execute the same processing in response to a common input from the outside and output first to third processing results, respectively, and diagnostic blocks 300A to 300C that correspond to the processing blocks, respectively And an external transmission / reception block 10α. Processing blocks 310A-310C include reconfigurable circuit portions 319A-319C. Each of the external transmission / reception block and the diagnosis block includes a majority decision determination unit 14aα and 14b to 14d that determine the majority of the first to third processing results, and a failure diagnosis unit 23aα that performs diagnosis when there is an error in the majority determination result. 23b-23d. Here, each of the failure diagnosis units performs a process of sharing failure diagnosis information generated by reflecting an error with other failure diagnosis units via the first communication unit.
[Selection] Figure 1A

Description

  The present invention relates to an information processing system, for example, an information processing system for a financial institution that requires high reliability.

  For example, Patent Document 1 detects a triple processing system, a majority decision logic unit that performs majority determination of each processing result, and a processing system in which a failure has occurred, in an FPGA (Field Programmable Gate Array), An information processing apparatus including various circuit units for reconfiguring the processing system using circuit configuration information of a normal processing system is shown.

JP 2011-216002 A

  For example, in the field of financial transactions and the like, there is a continuous demand for reducing losses caused by transaction delays, and so far, delays have been reduced by various methods. In recent years, instead of people, algorithmic trading using a computer has become common, and the demand for low delay has progressed to the order of microseconds and nanoseconds in which computers operate. Conventionally, the delay has been reduced by the advancement of software technology and computer system technology, but there is a limit to reducing the delay in the order of microseconds and nanoseconds. Delay (lower latency) is desired.

  Conventionally, in order to reduce the delay using a dedicated circuit, an ASIC (Application Specific Integrated Circuit) has been performed for the processing to be reduced. However, in recent years, due to the development of semiconductor technology, it is common to use FPGAs that can dynamically change hardware wiring and logic at lower cost than ASICs. Since the FPGA supports various I / O technologies and can be connected to a general computer network, it is possible to realize all financial processing conventionally performed by a general computer with the FPGA. . Also, by using an FPGA, it becomes possible to eliminate the processing that was redundant when using a general computer, and to implement only the minimum processing necessary for financial transaction processing on the FPGA. This makes it possible to reduce the delay that could not be achieved with a simple computer.

  However, a new problem occurs by using FPGA. The FPGA stores circuit configuration information in a rewritable memory so that the circuit configuration can be freely programmed. A memory storing the circuit configuration information is called a CRAM (Configuration RAM) or the like. In order to support high performance and high functionality in recent years, SRAM (Static Random Access Memory) suitable for high speed and large capacity is often used for CRAM. Therefore, in CRAM, there is a high possibility that bit inversion occurs due to radiation such as alpha rays and neutron rays. A transient failure caused by this bit inversion is called a soft error. The probability of occurrence of soft errors is increasing due to miniaturization of semiconductor manufacturing processes.

  In order to cope with such a problem, there is an FPGA having a method for detecting and correcting a soft error. However, in such an FPGA, there is a possibility that a delay on the order of milliseconds occurs with the detection of a soft error. Therefore, even in the case of improving the reliability by using such a soft error detection mechanism in the FPGA in the microsecond and nanosecond order financial transaction processing, when the error is detected, it is already wrong. There is a possibility that the processing result is transferred to a user outside the system and the next transaction is performed using the wrong processing result. In financial transaction processing, if the system provides an incorrect processing result, there is a possibility of causing a great deal of damage to the user. Therefore, the soft error detection mechanism in the FPGA is in the order of microseconds or nanoseconds. It will be difficult to utilize in financial transaction processing that requires low latency.

  In such a low-delay environment, in order to detect and correct errors almost in real time, for example, it is conceivable to use a majority voting system as disclosed in Patent Document 1. This is a method of preparing three or more processing systems that perform the same processing, comparing the results of processing in each system, and considering the most probable result as a normal result. In general, the probability of a double failure is very small, so if you perform a triple, even if one system fails, if the other two systems output the correct result, an error is detected by the majority of the output results・ Can be corrected. Also, if the cause of the error is a soft error that has occurred on the FPGA CRAM, this is a transient phenomenon. Therefore, as shown in Patent Document 1, if normal circuit configuration information is input again to the FPGA CRAM, the error is recovered. Is possible.

  However, in the technique of Patent Document 1, there is a possibility that the range in which error detection and correction can be performed by a majority decision becomes partial. Specifically, the error detection and correction of the triple circuit in the FPGA can be performed, and the CRAM can be reconfigured. However, a majority decision logic unit that performs majority decision, an FPGA main body, a device that performs automatic recovery of the FPGA, and the like. When an error having a single point of failure occurs, it is difficult to detect and correct the error with the technique of Patent Document 1, and it is also difficult to reconfigure the CRAM. Due to the nature of financial transaction services, the availability of continuing services after a single failure is essential. In order to realize availability, it is necessary to detect an error even after a single failure occurs, and to stop without leaking the error to the outside at worst.

  The present invention has been made in view of the above, and one of its purposes is to provide an information processing system capable of improving reliability. The above and other objects and novel features of the present invention will be apparent from the description of this specification and the accompanying drawings.

  Of the inventions disclosed in the present application, the outline of a typical embodiment will be briefly described as follows.

  The information processing system according to the present embodiment executes the same processing according to a common input from the outside, and outputs first to third processing results, respectively, and first to third processing blocks. 1st-3rd diagnostic block provided corresponding to each of 3 process blocks, and a 4th diagnostic block are provided. The N-th processing block (N = 1, 2, 3) constructs an N-th configuration information storage unit that stores circuit configuration information and a circuit corresponding to the circuit configuration information. And an Nth reconfigurable circuit unit that outputs a processing result. The Mth diagnosis block (M = 1, 2, 3, 4) receives the first to third processing results, and if there is an error in any of them, the Mth diagnosis block (M = 1, 2, 3, 4) specifies the processing result having the error. An M-th majority determination unit that outputs an M error determination result, an M-th failure diagnosis unit that receives the M-th error determination result and diagnoses a failure, and an M-th communication unit that communicates with other diagnostic blocks And have. Here, the M-th fault diagnosis unit transmits the M-th diagnosis information reflecting the M-th error determination result to the other fault diagnosis unit via the M-th communication unit, and receives the diagnosis information from the other fault diagnosis unit. It has the Mth sharing part which receives via M communication part, respectively.

  Of the inventions disclosed in this application, the effects obtained by the representative embodiments will be briefly described. In the information processing system, it is possible to improve the reliability.

It is a block diagram which shows the example of whole structure of the information processing system in Embodiment 1 of this invention. It is a block diagram which shows the more detailed structural example of the external transmission / reception block in FIG. 1A. It is a block diagram which shows the more detailed structural example of the diagnostic block in FIG. 1A. 2 is a flowchart showing an example of the entire processing content in the information processing system of FIGS. 1A to 1C. In FIG. 2, it is a flowchart which shows an example of the schematic process content of the failure recovery process. FIG. 3 is a flowchart illustrating an example of processing contents of a defective configuration portion diagnosis unit in the information processing system of FIGS. 1A to 1C. FIG. 2 is a flowchart illustrating an example of processing contents of a system failure diagnosis unit in the information processing system of FIGS. 1A to 1C. 2 is a flowchart illustrating an example of processing contents of a failure diagnosis information sharing unit in the information processing system of FIGS. 1A to 1C. 2 is a flowchart illustrating an example of processing contents of a recovery plan determination unit and a recovery plan agreement unit in the information processing system of FIGS. 1A to 1C. FIG. 8 is a table showing an example of determination criteria for determining a recovery plan in the flowchart of FIG. 7. FIG. 2 is a block diagram illustrating a detailed configuration example of a recovery plan execution unit in the external transmission / reception block in the information processing system of FIGS. 1A to 1C. FIG. 2 is a block diagram illustrating a detailed configuration example of a recovery plan execution unit in the diagnostic block in the information processing system of FIGS. 1A to 1C. FIG. 11 is a flowchart illustrating an example of processing contents when the recovery plan execution unit belongs to a recovery destination processing system in the recovery plan execution unit of FIG. 10. FIG. 11 is a flowchart illustrating an example of processing contents when the recovery plan execution unit belongs to the recovery processing system in the recovery plan execution unit of FIG. 10. FIG. 11 is a flowchart illustrating an example of processing contents when the recovery plan execution unit does not belong to either the recovery destination or the recovery processing system in the recovery plan execution unit of FIG. 10. It is a flowchart which shows an example of the processing content of the recovery plan execution part of FIG. 2 is a flowchart illustrating an example of processing contents of a manual recovery process performed by the recovery plan execution unit in the information processing system of FIGS. 1A to 1C. In the information processing system of Drawing 1A-Drawing 1C, it is a state transition diagram showing an example of each mode stored in the mode information, and an example of the relation. In the information processing system of Drawing 1A-Drawing 1C, it is a figure showing an example of the contents stored in the system configuration information. 7 is a flowchart illustrating an example of processing contents different from those in FIG. 6 performed by the failure diagnosis information sharing unit in the information processing system of FIGS. 1A to 1C. FIG. 7 is a sequence diagram illustrating an example of a fault diagnosis information exchange method in the flowchart of FIG. 6. FIG. 8 is a sequence diagram showing an example of an exchange method based on the recovery plan Byzantine agreement protocol in the flowchart of FIG. 7. FIG. 8 is a sequence diagram showing an example of an exchange method based on the recovery plan Byzantine agreement protocol in the flowchart of FIG. 7. It is a block diagram which shows the structural example of the processing block in the information processing system by Embodiment 2 of this invention.

  In the following embodiment, when it is necessary for the sake of convenience, the description will be divided into a plurality of sections or embodiments. However, unless otherwise specified, they are not irrelevant, and one is the other. Some or all of the modifications, details, supplementary explanations, and the like are related. Further, in the following embodiments, when referring to the number of elements (including the number, numerical value, quantity, range, etc.), especially when clearly indicated and when clearly limited to a specific number in principle, etc. Except, it is not limited to the specific number, and may be more or less than the specific number.

  Further, in the following embodiments, the constituent elements (including element steps and the like) are not necessarily indispensable unless otherwise specified and apparently essential in principle. Needless to say. Similarly, in the following embodiments, when referring to the shapes, positional relationships, etc. of the components, etc., the shapes are substantially the same unless otherwise specified, or otherwise apparent in principle. And the like are included. The same applies to the above numerical values and ranges.

<< Summary of Embodiment >>
In the present embodiment, it is assumed that at least one single point of failure is included in the information processing system. For example, as in Patent Document 1, if a reconfigurable circuit unit (typically an FPGA) is provided with a triple processing system and a majority decision determining unit that determines the majority of each processing result, the processing system Instead, when a single point of failure is included in the majority decision determination unit, it becomes difficult to detect and correct an error.

  In addition, in order to improve the reliability of the system, when a failure occurs in the processing system configured by the reconfigurable circuit unit, it is detected and a diagnosis for automatically recovering the processing system in which the failure has occurred It is desirable to provide a block. However, as in the case of the majority decision unit, a single failure point may be included in the diagnosis block instead of the processing system. Therefore, in the present embodiment, an information processing system is constructed in which not only the processing system but also the majority decision unit and the diagnostic block are at least tripled. In this case, since various options can be considered as at least a triple form, it is necessary to consider what form is appropriate.

  Furthermore, as another requirement, for example, in financial transaction processing, etc., there is an internal state (state) such as input data and the previous processing result, and the next processing result changes depending on the internal state (hereinafter referred to as stateful processing). Is used). For example, since the technique of Patent Document 1 assumes a processing system having no internal state (that is, a stateless), when a failure occurs in a certain processing system, the processing system (that is, a reconfigurable circuit unit). Even if the circuit configuration can be reconfigured, it is difficult to recover the internal state. Therefore, it is necessary to consider an appropriate form of the information processing system described above, including the internal state (state) recovery method.

  In view of the above, the information processing system according to the present exemplary embodiment typically includes the following configurations and operations with reference to FIGS. 1A to 1C, 9, and 10, for example. .

  (1) The information processing system according to the present embodiment executes the same processing in response to a common input from the outside, and outputs first to third processing results (310A to 310A, respectively). 310C), first to third diagnosis blocks (300A to 300C) provided corresponding to the first to third processing blocks, respectively, and a fourth diagnosis block (10α). The N-th processing block (N = 1, 2, 3) constructs an N-th configuration information storage unit (318A to 318C) that stores circuit configuration information and a circuit corresponding to the circuit configuration information. The N-th reconfigurable circuit unit (319A to 319C) that outputs the N-th processing result by the above operation. The M-th diagnosis block (M = 1, 2, 3, 4) includes an M-th failure diagnosis unit (14b to 14d, 18aα) and an M-th failure diagnosis unit (18b to 18d, 18aα). 23b to 23d, 23aα) and an M-th communication unit (301A to 301C, 13α) for performing communication with other diagnostic blocks.

  For example, when a single failure occurs in any of the first to third processing blocks, one of the first to third processing results output from the reconfigurable circuit unit in the three processing blocks is another And different. The M-th majority decision determination unit in the M-th diagnosis block inputs the first to third processing results, and if there is an error in any of the first to third processing results, the processing result having an error is displayed. The Mth error determination result including the specified information is output to the Mth fault diagnosis unit. At this time, assuming that the first to fourth diagnostic blocks are normal, all the first to fourth error determination results output from the first to fourth majority determination units are the same. The Mth fault diagnosis unit can determine in which processing block the reconfigurable circuit unit has a single fault based on the Mth error determination result.

  However, a single failure may occur in the majority decision unit instead of the processing block. For example, when a single failure occurs in the first majority decision determination unit, the first failure diagnosis unit determines whether any of the first to third processing blocks has a single failure from the first majority decision determination unit. There is a possibility of receiving the first error determination result. That is, a situation may occur in which only one of the first to fourth failure diagnosis units determines that there is a failure and the remaining failure diagnosis units determine that there is no failure.

  Therefore, the M-th failure diagnosis unit reflects the M-th error determination result received from the M-th majority decision determination unit, generates M-th diagnosis information, and passes the M-th diagnosis information to another failure diagnosis via the M-th communication unit. And receives diagnostic information from other fault diagnosis units via the Mth communication unit. That is, the Mth failure diagnosis unit includes an Mth sharing unit for sharing the first to fourth diagnosis information between the first to fourth failure diagnosis units. By this sharing operation, each failure diagnosis unit can detect a case where a single failure has occurred in the majority decision determination unit.

  Specifically, each failure diagnosis unit can determine that no single failure has occurred in the majority decision determination unit, for example, when the first to fourth diagnosis information is normally shared. On the other hand, when each failure diagnosis unit receives only one of the first to fourth diagnosis information, the failure diagnosis unit is connected to the majority decision unit in the diagnosis block that has transmitted the diagnosis information. It can be determined that a failure may have occurred. Note that, for example, even when a single failure occurs in the communication unit in the diagnostic block, the input operation of the first to third processing results toward the majority decision determination unit in the diagnostic block becomes unstable. As a result, the same situation as when a single failure occurs in the majority decision determination unit may occur. Thus, by having the M-th sharing unit, it is possible to detect a case where a failure has occurred in the majority decision determination unit unlike Patent Document 1, and the reliability of the information processing system can be improved.

  (2) The M-th failure diagnosis unit (23b-23d, 23aα) further includes an M-th recovery plan determination unit (19b-19d, 19aα), and the M-th diagnosis block (300A-300C, 10α) In addition, an M-th recovery plan execution unit (21b to 21d, 21aα) is included. The Mth recovery plan determination unit determines whether any one of the first to third processing blocks has a failure based on the first to fourth diagnosis information shared by the Mth sharing unit. When any one of the first to third processing blocks has a failure, the Mth recovery plan is generated. The Mth recovery plan execution unit executes an automatic recovery process of the recovery destination processing block based on the Mth recovery plan.

  Specifically, the M-th recovery plan determination unit, for example, normally shares the first to fourth diagnosis information, and the first to fourth error determination results included in the first to fourth diagnosis information are If all match, it is determined that the failure is any one of the first to third processing blocks. Then, the Mth recovery plan determination unit determines a recovery plan (automatic recovery plan) for automatically recovering a single failure of the processing block (that is, reconfiguring the reconfigurable circuit unit). In the automatic recovery plan, for example, a recovery destination processing block (that is, a processing block in which a single failure has occurred) and a recovery block processing block (that is, a processing block in which a single failure has occurred) determined from the matching first to fourth error determination results (that is, Normal processing block) information and the like are included.

  On the other hand, for example, when only one of the first to fourth diagnostic information is generated, there is a possibility that a single failure has occurred in the diagnostic block. That is, the case where it arises in the majority decision determination part and communication part which were mentioned above is considered, and not only this but the case where it arises in a fault diagnosis part is also considered. In such a case, since it may be difficult to perform automatic recovery, each failure diagnosis unit defines a recovery plan (manual recovery plan) indicating that manual recovery is necessary. The manual recovery plan includes, for example, information on the location of the failure estimated from the first to fourth failure information (that is, in which diagnostic block the failure has occurred). For example, when only one of the first to fourth diagnostic information is generated, it is estimated that a failure has occurred in the diagnostic block that generated the one diagnostic information. The M-th recovery plan execution unit executes an automatic recovery process based on the automatic recovery plan when an automatic recovery plan is transmitted from the M-th failure diagnosis unit, and when a manual recovery plan is transmitted, an external recovery plan is executed. A notification indicating that manual recovery is necessary is performed.

  (3) During the period in which automatic recovery processing is performed by the recovery plan execution unit, the remaining two processing blocks excluding the recovery destination processing block (one of which is also a recovery block) are external The same processing according to the common input from is continued as it is. At this time, the majority decision determination unit receives the two processing results from the two processing blocks, and outputs the same processing result to the outside when the two processing results are the same. On the other hand, when the two processing results are different, the majority decision determining unit stops the system.

  In addition, during the period from when manual recovery is notified by the recovery plan execution unit to when manual recovery corresponding to this is completed, the processing block corresponding to the normal diagnostic block corresponds to the common external input. The same processing is continued as it is. As described above, the normal diagnosis block can be specified based on the manual recovery plan from the failure diagnosis unit. In this way, the information processing system performs the degenerate operation using the remaining two processing blocks even if a failure occurs in one processing block. In this degenerate operation period, it is difficult for the information processing system to correct an error using the majority decision determination unit, but it is possible to detect an error using the majority decision determination unit. As a result, it is possible to prevent an erroneous processing result from leaking to the outside while continuing the operation of the information processing system, and to improve the reliability of the information processing system.

  (4) The M-th failure diagnosis unit (23b-23d, 23aα) further includes an M-th recovery plan agreement unit (20b-20d, 20aα). The Mth recovery plan agreement unit exchanges the Mth recovery plan generated by the Mth recovery plan determination unit with another recovery plan determination unit using the Byzantine agreement protocol via the Mth communication unit. The M-th recovery plan determined based on the exchange result is output to the M-th recovery plan execution unit. For example, when a single failure occurs in the recovery plan determination unit, or when a Byzantine failure occurs in the communication unit or the like, there is a possibility that each failure diagnosis unit may determine an incorrect recovery plan. Therefore, each failure diagnosis unit agrees on a recovery plan using a Byzantine agreement protocol.

  Specifically, first, the Mth recovery plan agreement unit exchanges the Mth recovery plan generated by the Mth recovery plan determination unit with another failure diagnosis unit by a Byzantine agreement protocol. Next, when all of the first to fourth recovery plans obtained by exchanging this Byzantine agreement protocol match, the Mth recovery plan agreement unit determines that the recovery plan is the same as the recovery plan to the Mth recovery plan execution unit. Send. On the other hand, when the first to fourth recovery plans all match, there is a possibility that a failure exists in the diagnostic block including a Byzantine failure. In this case, for example, the Mth recovery plan agreement unit manually selects a recovery plan obtained by agreement from the first to fourth recovery plans (for example, three recovery plans that match among the four recovery plans). The recovery plan is transmitted to the Mth recovery plan execution unit.

  The Mth recovery plan execution unit executes the automatic recovery process based on the automatic recovery plan when the recovery plan that matches from the Mth recovery plan agreement unit is transmitted and is the automatic recovery plan. On the other hand, when a manual recovery plan is transmitted from the Mth recovery plan consensus unit, the Mth recovery plan execution unit performs an external notification indicating that manual recovery is necessary. In this way, by performing the automatic recovery process based on the automatic recovery plan after passing through the Byzantine agreement protocol, it is possible to prevent a situation in which an erroneous automatic recovery process is performed, and to improve the reliability. For example, if a diagnostic block determines a wrong recovery plan due to its own single failure, or if a Byzantine failure occurs in a diagnostic block, a consistent recovery plan cannot be obtained by the Byzantine agreement protocol. The automatic recovery process that is erroneous as the processing system is not executed.

  In addition, even if the first to fourth recovery plans do not all match, the same manual recovery plan is established among the three diagnostic blocks belonging to the system that has not failed by the agreement using the Byzantine agreement protocol. It becomes possible to determine. And based on this manual recovery plan, a normal diagnostic block can be specified from each diagnostic block, and it becomes possible to perform the above-mentioned degenerate operation using the normal diagnostic block.

  (5) When the recovery plan execution unit executes automatic recovery processing based on the automatic recovery plan, assuming that the processing content in each processing block is stateful processing, automatic recovery of the reconfigurable circuit unit In addition to processing (circuit reconfiguration), the state must also be restored. Therefore, the Nth processing block (310A to 310C) further includes an Nth state storage unit (314A to 314C) that holds a state obtained in the process of executing the process of the Nth reconfigurable circuit unit. Then, the M-th recovery plan execution unit (21b to 21d, 21aα), when executing the automatic recovery processing of the recovery destination processing block, in addition to the storage information of the configuration information storage unit in the recovery destination processing block, The stored information in the state storage unit is recovered. Thus, unlike Patent Document 1, it is possible to realize automatic recovery processing even in an information processing system for a financial institution including stateful processing.

  (6) When the storage information in the configuration information storage unit in the recovery destination processing block is recovered, there is no particular limitation. For example, the process of copying the storage information in the configuration information storage unit in the recovery processing block Alternatively, separately, the circuit configuration information may be stored in a nonvolatile memory or the like, and the stored information may be copied. On the other hand, when the storage information in the state storage unit in the recovery destination processing block is recovered, for example, the following configuration may be provided.

  The Nth state storage units (314A to 314C) include Nth state saving units (315A1 to 315C1) that hold the state at the first time point when the state recovery starts. The fourth recovery plan execution unit (21aα) includes a data holding unit (218aα) that sequentially holds data input from the outside after the first time point. Each of the first to third recovery plan execution units (21b to 21d) includes a configuration recovery unit (214b to 214d) that recovers the storage information of the configuration information storage unit in the recovery destination processing block, and the storage of the state storage unit. A state recovery unit (215b to 215d) for recovering information. First, the state recovery unit sets a state held by the state saving unit in the processing block for recovery and restoration in the state storage unit in the recovery destination processing block. Next, after the storage information in the configuration information storage unit in the recovery destination processing block is recovered, the data held in the data holding unit in the fourth recovery plan execution unit is sequentially output toward the recovery destination processing block. As a result, the processing block recovered from the failure sequentially approaches the same state as the normal processing block including the state, and when it reaches the same state as the normal processing block, it completely recovers from the failure including the state. To do.

  (7) The fourth diagnosis block (10α) may be provided, for example, in the external transmission / reception block (10α) that serves as an interface with the outside. And the 4th majority decision part (14a (alpha)) in a 4th diagnostic block determines the process result output outside by performing majority decision with respect to a 1st-3rd process result. That is, the final processing result for output to the outside is determined by the fourth majority decision determination unit. However, in this case, when a failure occurs in the fourth majority decision determination unit, it is difficult to output a correct processing result to the outside. Therefore, it is desirable to make the external transmission / reception block redundant including the normal use (10α) and the backup use (10β). The presence / absence of a failure in the fourth majority determination unit can be determined based on, for example, the sharing result of the first to fourth diagnostic information in the Mth sharing unit as described above. Therefore, switching between normal use and backup use may be performed based on the determination result.

  In addition, assuming the failure of the majority decision unit, for example, between the first to fourth majority decision units, the processing result obtained by each majority decision is agreed by the Byzantine, and thereby the final for output to the outside A method for determining a typical processing result can be considered. However, in this case, with the Byzantine agreement, there is a possibility that the latency when outputting the processing result to the outside increases and the high speed of the information processing system is hindered. Thus, by adopting a method in which the final processing result is determined by one majority deciding unit, and by making this one majority deciding unit (external transmission / reception block including it) redundant, high speed and It becomes possible to achieve both reliability.

  As described above, by using the information processing system according to the present embodiment, typically, improvement in reliability can be realized. For example, in an information processing system in which components including a reconfigurable circuit such as an FPGA are at least tripled, even after a single failure occurs in any component, the same stateful processing as before the failure without stopping the information processing Can be executed continuously.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

(Embodiment 1)
<< Overall configuration of information processing system >>
FIG. 1A is a block diagram showing an example of the overall configuration of the information processing system according to Embodiment 1 of the present invention. 1B is a block diagram showing a more detailed configuration example of the external transmission / reception block in FIG. 1A, and FIG. 1C is a block diagram showing a more detailed configuration example of the diagnostic block in FIG. 1A. The information processing system in FIG. 1A includes external transmission / reception blocks 10α and 10β, diagnostic blocks 300A, 300B, and 300C, processing blocks 310A, 310B, and 310C, and communication paths L1 and L2.

  The external transmission / reception blocks 10α and 10β are HA (High Availability) systems (in other words, redundant systems) composed of blocks having the same configuration, the external transmission / reception block 10α is an execution system (for normal use), and the external transmission / reception block 10β is a standby system (backup). For). Although the external transmission / reception blocks 10α and 10β are HA systems having the same configuration, the present embodiment is not limited to this as long as the external transmission / reception block has a function capable of operating normally even after a single failure.

  The diagnosis blocks 300A, 300B, and 300C have the same configuration, and are blocks for diagnosing and recovering the triple processing blocks 310A, 310B, and 310C, respectively. The diagnostic blocks 300A, 300B, and 300C are configured by, for example, a reconfigurable circuit such as an FPGA, a computer system that executes program processing by a CPU (Central Processing Unit), or a combination of these. Is possible. The diagnosis blocks 300A, 300B, and 300C are triple blocks having the same configuration, but the present embodiment is not limited to this as long as the single failure point in the diagnosis block is tripled.

  The processing blocks 310A, 310B, and 310C have the same configuration, are tripled in the same way as the diagnostic blocks 300A, 300B, and 300C, and are blocks for processing input data transmitted in common from the external transmission / reception blocks 10α and 10β. It is. The processing blocks 310A, 310B, and 310C are triple blocks having the same configuration, but the present embodiment is not limited to this as long as the single failure point in the processing block is tripled.

  The communication path L1 is a communication path for performing communication between the external transmission / reception blocks 10α and 10β and an external information processing apparatus using the information processing system of FIG. 1A. The communication path L2 is a communication path for performing internal communication among the external transmission / reception blocks 10α and 10β, the diagnosis blocks 300A, 300B, and 300C and the processing blocks 310A, 310B, and 310C. For example, it is assumed that the components constituting the communication paths L1 and L2 are made redundant so that highly reliable communication can be performed. However, the present embodiment is not limited to this as long as the communication paths L1 and L2 are sufficiently reliable.

  The external transmission / reception block 10α includes an external communication unit 11α, an input data input unit 12α, a first communication unit 13α, and a failure diagnosis recovery system aα. The failure diagnosis recovery system aα includes a majority decision determination unit 14aα, a failure diagnosis unit 23aα, and a recovery plan execution unit 21aα. More specifically, the failure diagnosis unit 23aα includes a system failure diagnosis unit 16aα, a failure diagnosis information sharing unit 18aα, a recovery plan determination unit 19aα, and a recovery plan agreement unit 20aα as illustrated in FIG. 1B. More specifically, the fault diagnosis recovery system aα further includes system configuration information 17aα and mode information 22aα as shown in FIG. 1B.

  Although not shown, the external transmission / reception block 10β similarly includes an external communication unit 11β, an input data input unit 12β, a first communication unit 13β, and a failure diagnosis recovery system aβ. The failure diagnosis recovery system aβ includes a majority decision determination unit 14aβ, a failure diagnosis unit 23aβ, and a recovery plan execution unit 21aβ. More specifically, the failure diagnosis unit 23aβ includes a system failure diagnosis unit 16aβ, a failure diagnosis information sharing unit 18aβ, a recovery plan determination unit 19aβ, and a recovery plan agreement unit 20aβ as in FIG. 1B. More specifically, the fault diagnosis recovery system aβ further includes system configuration information 17aβ and mode information 22aβ as in FIG. 1B.

  The diagnosis block 300A includes a first communication unit 301A, a second communication unit 303A, and a failure diagnosis recovery system b. The failure diagnosis recovery system b includes a majority decision determination unit 14b, a failure diagnosis unit 23b, and a recovery plan execution unit 21b. More specifically, as shown in FIG. 1C, the failure diagnosis unit 23b includes a configuration failure location diagnosis unit 302A, a system failure diagnosis unit 16b, a failure diagnosis information sharing unit 18b, a recovery plan determination unit 19b, and a recovery plan. It has an agreement part 20b. More specifically, the failure diagnosis recovery system b further includes system configuration information 17b and mode information 22b as shown in FIG. 1C.

  The diagnostic blocks 300B and 300C also have the same configuration as the diagnostic block 300A, although detailed illustration is omitted. That is, the diagnostic block 300B includes the respective constituent elements (301B to 303B, c, 14c, 16c to 21c, 23c) corresponding to the respective constituent elements (301A to 303A, b, 14b, 16b to 21b, 23b) in the diagnostic block 300A. ). Similarly, the diagnosis block 300C includes each component (301C to 303C, d, 14d, 16d to 21d, corresponding to each component (301A to 303A, b, 14b, 16b to 21b, 23b) in the diagnosis block 300A, respectively. 23d).

  The processing block 310A includes a first communication unit 311A, a stateful processing unit 312A, a state storage unit 314A, a state storage management unit 313A, a second communication unit 316A, a configuration information storage unit 318A, and a configuration information storage management unit. 317A. The processing block 310A is configured by, for example, an FPGA. Here, the stateful processing unit 312A, the first communication unit 311A, and the state storage management unit 313A are configured by a reconfigurable circuit unit 319A, and the circuit configuration of the reconfigurable circuit unit 319A is stored in the configuration information storage unit 318A. It can be set arbitrarily according to the information. The configuration information storage unit 318A is generally called a CRAM (Configuration RAM) or the like, and is configured by an SRAM or the like, for example.

  The configuration information storage management unit 317A performs control when the diagnostic block 300A accesses the configuration information storage unit 318A via the second communication units 303A and 316A. That is, the configuration information storage management unit 317A performs control when the diagnostic block 300A performs reconfiguration of the circuit of the reconfigurable circuit unit 319A. The state storage unit 314A appropriately stores states generated in the process of the stateful processing unit 312A. The state storage management unit 313A performs control when storing this state. The state storage management unit 313A also performs control when the diagnostic block 300A accesses the state storage unit 314A via the second communication units 303A and 316A. Although details will be described later, the state storage unit 314A includes a first bank 315A1 and a second bank 315A2 in order to realize state recovery.

  The processing blocks 310B and 310C also have the same configuration as the processing block 310A, although detailed illustration is omitted. That is, the processing block 310B includes constituent elements (311B to 314B, 315B1, 315B2, 316B to 319B) respectively corresponding to the constituent elements (311A to 314A, 315A1, 315A2, 316A to 319A) in the processing block 310A. Similarly, the processing block 310C includes constituent elements (311C to 314C, 315C1, 315C2, 316C to 319C) respectively corresponding to the constituent elements (311A to 314A, 315A1, 315A2, 316A to 319A) in the processing block 310A. .

  Thus, as shown in FIG. 1A, the information processing system according to the first embodiment includes the diagnosis block 300A and the processing block 310A as the processing system AA, the diagnosis block 300B and the processing block 310B as the processing system BB, and the diagnosis block 300C. A triple processing system is provided with the processing block 310C as the processing system CC. By performing triple processing, three processing results for error detection and error correction can be obtained even after a single failure occurs. That is, the stateful processing units 312A to 312C in the processing system AA to the processing system CC perform the same processing by setting the same circuit configuration, and the processing results are transmitted via the first communication units 311A to 311C. Output. The information processing system according to the first embodiment includes quadruple fault diagnosis / recovery systems aα (or aβ) and b to d for performing fault diagnosis and recovery. By performing quadruplexing, it is possible to cope with a case where the failure diagnosis recovery system itself has a single failure including a Byzantine failure.

<< Overall operation of information processing system >>
FIG. 2 is a flowchart showing an example of the entire processing contents in the information processing system of FIGS. 1A to 1C. In FIG. 2, first, the external communication unit 11α receives the input data received from the communication path L1, and transmits it to the input data input unit 12α. Based on the information stored in the system configuration information 17aα, the input data input unit 12α broadcasts the input data to the first communication units 311A to 311C included in the processing blocks 310A to 310C of each processing system (step S201).

  Next, the stateful processing units 312A to 312C receive the input data via the first communication units 311A to 311C and perform the same processing (step S202). Subsequently, since the internal state (state) of the stateful processing units 312A to 312C changes as a result of step S202, the stateful processing units 312A to 312C transmit the states to the state storage management units 313A to 313C, respectively. The state storage management units 313A to 313C store the received state information in the state storage units 314A to 314C (step S203).

  Next, the stateful processing units 312A to 312C transmit output data as processing results to the first communication units 311A to 311C. The first communication units 311A to 311C broadcast the output data to the first communication unit 13α of the external transmission / reception block 10α and the first communication units 301A to 301C of the diagnosis blocks 300A to 300C. The first communication unit 13α and the first communication units 301A to 301C receive the three processing results (output data) received from the first communication units 311A to 311C, and send them to the majority decision determination unit 14aα and the majority decision determination units 14b to 14d, respectively. Transmit (step S204).

  The majority decision determination units 14aα and 14b to 14d compare the received three processing results (output data) and determine whether there is any different data (step S205). When no different data exists, that is, when it is determined that all three pieces of output data are the same, only the majority decision unit 14aα transmits the same output data to the output data input unit 15α. The output data input unit 15α sets the other party that has transmitted the input data corresponding to the output data as a transmission destination, and transmits the output data to the external communication unit 11α. The external communication unit 11α transmits output data using the communication path L1. On the other hand, the majority decision determining units 14b to 14d discard the output data (step S206). After step S206 is completed, normal processing is continuously performed by repeating the processing from step S201 again.

  As described above, in the information processing systems of FIGS. 1A to 1C, the processing result (output data) determined by the majority decision by the majority decision determination unit 14aα is output to the outside. Therefore, if a failure occurs in the majority decision determination unit 14aα, it is difficult to output a correct processing result to the outside. Therefore, here, a configuration is adopted in which duplicated majority decision units 14aα and 14aβ (external transmission / reception blocks 10α and 10β) are provided, and when a failure occurs on one side, the other is switched to the other side. Although details will be described later, the presence / absence of a failure such as the majority decision determination unit 14aα can be determined by the failure diagnosis units 23aα and 23b to 23d.

  When it is determined in step S205 in FIG. 2 that at least one piece of different data exists, that is, there is an error in the three processing results (output data), the majority decision determining units 14aα and 14b to 14d output three outputs. It is determined whether two or more identical data exist among the data (step S207). When it is determined in step S207 that two or more identical data exist, the majority determination units 14aα and 14b to 14d output an error determination result for specifying a processing result having an error to the failure diagnosis units 23aα and 23b to 23d. Then, the execution of the failure recovery process is started (step S208).

  Details of the failure recovery processing will be described in the description of FIG. 3 and subsequent figures, but the error determination results and output data of the majority decision determination units 14aα and 14b to 14d are the system failure diagnosis units 16aα and 16b to 16d, and the configuration failure location diagnosis unit. It is transmitted to 302A-302C, and the failure status is diagnosed. Thereafter, sharing of failure diagnosis information obtained from the diagnosis result, determination of the recovery plan, and agreement of the recovery plan proceed, and finally the recovery plan is executed. Since these processes are executed in parallel with the flowchart of FIG. 2, if two or more identical data exist and an error can be corrected, failure recovery is performed while continuing normal processing. It is possible to do.

  In parallel with the process of step S208 in FIG. 2, the majority decision determination units 14aα and 14b to 14d correct errors. Specifically, two of the three output data that are the same are regarded as normal output data, and only the majority decision unit 14aα transmits the normal output data to the output data input unit 15α. The output data input unit 15α sets the other party that has transmitted the input data corresponding to the output data as the transmission destination, and transmits the output data via the external communication unit 11α and the communication path L1 (step S209). If it is determined in step S207 in FIG. 2 that two or more identical data do not exist, that is, all the output data are different, the majority decision determining units 14aα and 14b to 14d stop the information processing system ( Step S210).

<Outline of failure recovery processing>
FIG. 3 is a flowchart showing an example of a schematic processing content of the failure recovery processing in FIG. If it is determined in step S207 of FIG. 2 that two or more identical data exist and the error can be corrected, the flowchart of FIG. 3 is executed as the failure recovery processing in step S208. In FIG. 3, first, the failure diagnosis units 23aα, 23b to 23d determine whether or not their own failure diagnosis recovery systems aα, b to d are in the failure recovery mode (step S301). The detailed mode will be described later with reference to FIG.

  The failure diagnosis unit 23aα, 23b-23d terminates the process when the mode information 22aα, 22b-22d is in the failure recovery mode, and shifts to the failure recovery mode when in the normal mode (step S302). Specifically, the failure diagnosis units 23aα and 23b to 23d rewrite the mode information 22aα and 22b to 22d to the failure recovery mode.

  Next, the error determination results and output data of the majority decision determination units 14aα and 14b to 14d are transmitted to the system failure diagnosis units 16aα and 16b to 16d and the configuration failure location diagnosis units 302A to 302C, and the failure diagnosis is performed. (Step S303). Details of the diagnosis performed by the system failure diagnosis units 16aα and 16b to 16d will be described later with reference to FIG. 5, and details of the diagnosis performed by the configuration failure point diagnosis units 302A to 302C will be described later with reference to FIG. To explain the outline, these diagnosis units diagnose failure locations and failure states in processing blocks and systems (specifically, the first communication unit), for example, which processing block has a failure, Alternatively, fault diagnosis information indicating which diagnostic block or external transmission / reception block has a fault is created. Then, these diagnosis units pass the created failure diagnosis information to the failure diagnosis information sharing units 18aα and 18b to 18d.

  Next, the fault diagnosis information sharing unit 18aα transmits the fault diagnosis information created based on the diagnosis result of the system fault diagnosis unit 16aα to the other fault diagnosis information sharing units 18b to 18d. The failure diagnosis information sharing unit 18b transmits the failure diagnosis information created based on the diagnosis results of the system failure diagnosis unit 16b and the configuration failure location diagnosis unit 302A to the other failure diagnosis information sharing units 18aα, 18c, and 18d. Similarly, the fault diagnosis information sharing units 18c and 18d also transmit fault diagnosis information. That is, each of the failure diagnosis information sharing units 18aα and 18b to 18d transmits its own failure diagnosis information to other failure diagnosis information sharing units, and receives failure diagnosis information of other failure diagnosis information sharing units. Thus, fault diagnosis information is exchanged and shared (step S304).

  Although specific processing will be described later with reference to FIG. 6, each failure diagnosis recovery system aα, b to d only detects a failure in each failure diagnosis recovery system when the processing of step S303 in FIG. 3 is completed. There is a possibility of erroneous detection depending on the location of the actual failure. The erroneous detection can occur, for example, when a failure occurs in the majority decision determination unit or the like. Therefore, the fault diagnosis information sharing units 18aα, 18b to 18d share fault diagnosis information with each other in order to grasp the status of other fault diagnosis recovery systems, and share the respective fault diagnosis information with the recovery plan determination unit 19aα, It passes to 19b-19d.

  Here, for example, when there is no failure in the processing blocks 310A to 310C, the majority decision determination units 14aα and 14b to 14d in each failure diagnosis recovery system do not output the error determination result, but reflect the error determination result and the like. Failure diagnosis information generated is not generated. However, there is a situation in which only a fault diagnosis recovery system detects a fault, and fault diagnosis information associated therewith is transmitted to a fault diagnosis information sharing unit in a fault diagnosis recovery system other than the fault diagnosis recovery system that detected the fault. Can happen. Specifically, for example, a failure occurs in the majority decision determination unit, the system failure diagnosis unit, the defective configuration diagnosis unit, or the first communication unit in the failure diagnosis and recovery system that detects the failure. Can happen.

  Therefore, although details will be described later with reference to FIG. 18, the fault diagnosis information sharing units 18aα and 18b to 18d of each fault diagnosis / recovery system always perform other faults not only in step S304 in the fault recovery mode but also in the normal mode. Wait for failure diagnosis information from the diagnosis recovery system and respond to the failure. Thereby, for example, the fault diagnosis recovery system aα detects a fault even though the other fault diagnosis recovery systems b to c do not detect the fault, and the fault diagnosis of the other fault diagnosis recovery systems b to d Even when the information sharing units 18b to 18d receive failure diagnosis information for the failure, it is possible to appropriately deal with the failure. Specifically, in such a case, the diagnosis blocks 300A to 300C determine that a failure has occurred in the failure diagnosis recovery system aα, and, for example, output a signal indicating that the failure diagnosis recovery system aα has a failure. It transmits to the external transmission / reception block 10β. In response to this, the external transmission / reception block 10β takes over all processing of the external transmission / reception block 10α.

  Next, in FIG. 3, the recovery plan determination units 19aα and 19b to 19d determine a recovery plan to be executed from each failure diagnosis information shared by the failure diagnosis information sharing unit (step S305). The specific processing will be described later with reference to FIGS. 7 and 8. However, when it is automatically diagnosed that recovery is possible, a processing block for recovery and a processing block for recovery are determined, and an automatic recovery plan including this information is established. , To the recovery plan agreement unit 20aα, 20b-20d. In addition, when it is diagnosed that automatic recovery is impossible, a manual recovery plan including failure diagnosis information (that is, information specifying a diagnosis block in which a failure has occurred) is passed to the recovery plan agreement units 20aα, 20b to 20d. .

  Subsequently, the recovery plan agreement units 20aα and 20b to 20d make an agreement between the recovery plan agreement units to execute the recovery plan passed from the recovery plan determination units 19aα and 19b to 19d (step S306). The specific processing will be described later with reference to FIGS. 7 and 8. This consensus process generates an incorrect recovery plan due to, for example, a failure in the recovery plan determination unit or a Byzantine failure in the first communication unit. Can be prevented.

  Next, the recovery plan execution units 21aα and 21b to 21d execute the agreed recovery plan (step S307). Specific processing will be described later with reference to FIGS. 11 to 15, but when the recovery can be normally performed, the mode is changed to the normal mode and the original triple system is restored. If recovery cannot be performed normally, the failure recovery process is interrupted and manual recovery is encouraged.

《Processing contents of faulty location diagnosis unit》
FIG. 4 is a flowchart illustrating an example of processing contents of the defective configuration portion diagnosis unit in the information processing system of FIGS. 1A to 1C. The flowchart of FIG. 4 represents the processing contents of the defective configuration portion diagnosis units 302A to 302C in step S303 of FIG. In FIG. 4, first, the configuration failure location diagnosis units 302A to 302C determine whether or not they are faulty systems based on the error determination results passed from the majority decision determination units 14b to 14d (in other words, there is a possibility that a failure has occurred in their processing block). Whether or not there is a characteristic) is determined (step S401). Then, the defective configuration portion diagnosis units 302A to 302C determine whether or not the faulty system is under its own management (step S402).

  Specifically, for example, when the majority decision determination units 14b to 14d differ from the processing systems BB and CC only in the processing result (output data) from the processing system AA, the error determination result with the processing system AA as the fault system Is output. In this case, the misconfiguration point diagnosis unit 302A determines that it is a fault system and the fault system is under its control, and the misconfiguration point diagnosis units 302B and 302C have the processing system AA as a fault system, It is determined that the fault system is not under its own management.

  If the configuration failure location diagnosis units 302A to 302C determine that the faulty system is not under their control, the processing ends. On the other hand, if the configuration failure location diagnosis units 302A to 302C determine that the faulty system is under their own management, the configuration information storage management is performed via the second communication units 303A to 303C and the second communication units 316A to 316C. Information on the fault location is requested to the units 317A to 317C (step S403). Actually, any one of the configuration failure location diagnosis units 302A to 302C (for example, 302A) requests the configuration information storage management unit (for example, 317A) corresponding to itself for information on the failure location. .

  Subsequently, the configuration information storage management units 317A to 317C inspect the configuration information storage units 318A to 318C, collect information on the failure location, and respond to the configuration failure location diagnosis units 302A to 302C (step S404). Note that the processing is actually performed in any one of the processing systems AA to CC. In addition, as specific processing contents, for example, a method using a soft error diagnostic circuit or a storage information stored in a configuration information storage unit in a processing block for recovery included in a recovery plan is compared and determined as an expected value. A method or a method in which correct stored information is stored in a nonvolatile memory or the like in advance and the stored information is compared and determined as an expected value.

  Next, the configuration failure location diagnosis units 302A to 302C determine whether or not there is a response from the configuration information storage management units 317A to 317C within a certain time (step S405). If there is a response from the configuration information storage management units 317A to 317C for information regarding the fault location within a certain time, the configuration failure location diagnosis units 302A to 302C pass the information to the fault diagnosis information sharing units 18b to 18d (step S406). ). On the other hand, if there is no response from the configuration information storage management units 317A to 317C within a certain period of time, the configuration failure location diagnosis units 302A to 302C pass information that the reconfiguration is impossible to the fault diagnosis information sharing units 18b to 18d (step) S407).

<< Processing contents of system fault diagnosis part >>
FIG. 5 is a flowchart illustrating an example of processing contents of the system failure diagnosis unit in the information processing systems of FIGS. 1A to 1C. The flowchart of FIG. 5 represents the processing contents of the system failure diagnosis units 16aα and 16b to 16d in step S303 of FIG. In FIG. 5, first, the system failure diagnosis units 16aα, 16b to 16d check whether or not the first communication units 13α and 301A to 301C included in other failure diagnosis recovery systems respond (step S501). Specifically, taking the system failure diagnosis unit 16aα as an example, the system failure diagnosis unit 16aα responds using a tool such as Ping to the first communication units 301A to 301C via the first communication unit 13α, for example. Check if there is.

  Next, when there is no response from the first communication unit, the system failure diagnosis unit 16aα, 16b to 16d passes information indicating that communication is not possible to the failure diagnosis information sharing unit 18aα, 18b to 18d, and ends the process (step S502). . On the other hand, when there is a response from all the first communication units, information indicating that communication is possible is passed to the fault diagnosis information sharing units 18aα, 18b to 18d (step S503). Thus, it is determined whether or not the cause of the failure indicated by the error determination result from the majority determination unit is in any of the first communication units 13α and 301A to 301C.

《Processing contents of failure diagnosis information sharing unit》
FIG. 6 is a flowchart illustrating an example of processing contents of the failure diagnosis information sharing unit in the information processing systems of FIGS. 1A to 1C. The flowchart in FIG. 6 shows the processing contents performed in the fault diagnosis information sharing units 18aα and 18b to 18d in order to grasp the status of other fault diagnosis and recovery systems in step S304 in FIG. In FIG. 6, first, the fault diagnosis information sharing units 18aα and 18b to 18d check whether communication is possible from the diagnosis results received from the system fault diagnosis units 16aα and 16b to 16d (step S601). Specifically, as shown in step S503 in FIG. 5, it is determined that communication is possible only when information indicating that communication is possible is received.

  Next, when it is determined that communication is possible, the fault diagnosis information sharing units 18aα and 18b to 18d exchange their fault diagnosis information between the respective fault diagnosis information sharing units by broadcasting their own fault diagnosis information. (Step S602). A detailed exchange method will be described later with reference to FIG. Subsequently, the failure diagnosis information sharing units 18aα and 18b to 18d check whether or not the exchange has been completed normally (step S603). Specifically, it is checked whether there is a timeout or a data error.

  When the exchange ends normally, the fault diagnosis information sharing units 18aα, 18b to 18d pass the fault diagnosis information acquired by the exchange to the recovery plan determination units 19aα, 19b to 19d, and the processing is ended (step S604). On the other hand, if the exchange has not been completed normally, or if it is determined in step S601 that communication is not possible, the failure diagnosis information sharing units 18aα and 18b to 18d create a manual recovery plan, and the recovery plan execution unit 21aα. , 21b to 21d, and the process is terminated (step S605).

  For example, when a failure occurs in the majority decision determination unit, the failure diagnosis information sharing unit corresponding to the majority decision determination unit in which the failure has occurred does not receive the failure diagnosis information from the other failure diagnosis information sharing unit. The process timed out, and the process proceeds to step S605. At this time, the other failure diagnosis information sharing unit recognizes the existence of the majority decision determination unit in which the failure has occurred according to the flow of FIG.

《Processing contents of recovery plan judgment unit and recovery plan agreement unit》
FIG. 7 is a flowchart illustrating an example of processing contents of the recovery plan determination unit and the recovery plan agreement unit in the information processing system of FIGS. 1A to 1C. FIG. 8 is a table showing an example of determination criteria for determining a recovery plan in the flowchart of FIG. 7 and 8 show the processing contents performed in the recovery plan determination units 19aα and 19b to 19d and the recovery plan agreement units 20aα and 20b to 20d in order to determine and agree with the recovery plan in steps S305 and S306 of FIG. Is expressed.

  In FIG. 7, first, the recovery plan determination units 19aα, 19b to 19d determine a recovery plan (step S701). Specifically, first, in the previous stage, for each fault diagnosis recovery system aα, b to d, an error determination result from its own majority decision determination unit, a diagnosis result in its own configuration failure point diagnosis unit, Fault diagnosis information is generated separately and independently based on the diagnosis result of the system fault diagnosis unit of the system. Each failure diagnosis information determines which processing block has failed or which diagnosis block or external transmission / reception block has failed by judgment for each failure diagnosis recovery system aα, b to d. It will be a thing. Thereafter, each failure diagnosis information is shared by the failure diagnosis information sharing units 18aα and 18b-18. Each of the recovery plan determination units 19aα, 19b to 19d refers to each shared failure diagnosis information, and whether the failure can be automatically recovered or not automatically recovered based on the table 800 of FIG. Determine.

  In the table 800 of FIG. 8, it is determined that automatic recovery is possible if any one of the three processing blocks 310A to 310C (reconfigurable circuit units 319A to 319C) has a failure, and otherwise. It is determined that automatic recovery is impossible. For example, a configuration in which all pieces of information (in other words, each error determination result from each majority decision unit) that specify a failed processing block included in each shared failure diagnosis information match and the presence of a failure in the processing block corresponds When it is verified by the defective part diagnosis unit and no communication failure is detected by each system failure diagnosis unit, it is determined that automatic recovery is possible. The recovery plan determination units 19aα and 19b to 19d generate an automatic recovery plan when it is determined that automatic recovery is possible, and generate a manual recovery plan when it is determined that automatic recovery is impossible.

  The automatic recovery plan includes information on a recovery destination processing block representing a processing block having a failure, and information on a processing block for time restoration selected from processing blocks having no failure. The processing block for this restoration is selected based on the priority order set in advance in the recovery plan determination units 19aα, 19b to 19d. Further, the manual recovery plan includes information on the location of the failure estimated from each failure diagnosis information (that is, in which diagnosis block the failure has occurred). For example, if any one of the four shared fault diagnosis information is inconsistent with the other, it is estimated that a fault has occurred in the diagnostic block corresponding to this one fault diagnosis information. Is done.

  Next, the generated recovery plan is passed to the recovery plan agreement units 20aα, 20b to 20d. The recovery plan agreement units 20aα and 20b to 20d exchange recovery plans according to the Byzantine agreement protocol (step S702). The detailed replacement method will be described later with reference to FIGS. 20 and 21. By using such a Byzantine agreement protocol, even if one Byzantine disorder exists, the remaining three faults excluding the line in which the Byzantine disorder occurs are excluded. The recovery plan of the diagnostic recovery system can be made the same. That is, the state transition after determining the recovery plan can be made the same in the remaining three fault diagnosis recovery systems except for the system in which the Byzantine failure occurs.

  As a result, for example, even if a system in which a Byzantine failure occurs does not actually cause a failure, even if it enters into the execution of automatic failure recovery processing, Since the three failure diagnosis and recovery systems do not execute the automatic failure recovery processing, the automatic failure recovery processing results in an error. In this case, by appropriately performing error processing, for example, it is possible to perform degenerate operation or the like using the remaining three fault diagnosis recovery systems in which no Byzantine fault has occurred and the corresponding processing systems.

  Subsequently, the recovery plan agreement units 20aα and 20b to 20d check whether or not the replacement of the recovery plan has ended normally (step S703). In the case of normal termination, the recovery plan agreement units 20aα, 20b to 20d compare the replaced recovery plans and check whether the recovery plans of all the failure diagnosis recovery systems match (step S704). If the recovery plan does not match, a Byzantine failure has occurred somewhere, and by looking at the recovery plan match, it is possible to prevent failure recovery processing in an unstable system environment due to Byzantine failure .

  When the recovery plans of all the failure diagnosis recovery systems match, the recovery plan agreement units 20aα, 20b-20d pass the matched recovery plans to the recovery plan execution units 21aα, 21b-21d (step S705). On the other hand, when the recovery plans of all failure diagnosis recovery systems do not correspond, the recovery plan agreement units 20aα, 20b to 20d change the recovery plan to a manual recovery plan and change it to the recovery plan execution unit. 21a [alpha], 21b to 21d (step S707). In addition, even when the exchange of the recovery plan does not end normally in step S703, the recovery plan agreement units 20aα and 20b to 20d change the recovery plan to a manual recovery plan and change it to the recovery plan execution unit 21aα, 21b to 21d (step S706).

<< Detailed configuration of recovery plan execution part >>
FIG. 9 is a block diagram showing a detailed configuration example of the recovery plan execution unit in the external transmission / reception block in the information processing system of FIGS. 1A to 1C. Here, the external transmission / reception block 10α will be described as a representative, but the same applies to the external transmission / reception block 10β. The recovery plan execution unit 21aα illustrated in FIG. 9 includes a recovery control unit 211aα, a manual recovery plan notification unit 216aα, an input data re-input unit 217aα, and an input data re-input queue 218aα. Although details will be described later with reference to FIGS. 11 to 15, the recovery control unit 211aα controls the manual recovery plan notification unit 216aα and the input data re-input unit 217aα to execute the manual recovery plan and the automatic recovery plan.

  The input data re-input unit 217aα stores input data necessary for state recovery in the input data re-input queue 218aα, and inputs the input data in order to the processing block of the recovery destination when necessary. . When the received recovery plan is a manual recovery plan, the manual recovery plan notification unit 216aα notifies the outside as manual recovery plan information.

  FIG. 10 is a block diagram illustrating a detailed configuration example of the recovery plan execution unit in the diagnosis block in the information processing system of FIGS. 1A to 1C. Here, the diagnosis block 300A will be described as a representative, but the diagnosis blocks 300B and 300C are the same. The recovery plan execution unit 21b illustrated in FIG. 10 includes a recovery control unit 211b, a reconfiguration data generation unit 212b, circuit data 213b, a configuration recovery unit 214b, a state recovery unit 215b, and a manual recovery plan notification unit 216b. Is done. Although details will be described later with reference to FIGS. 11 to 15, the recovery control unit 211b controls the reconfiguration data generation unit 212b, the configuration recovery unit 214b, the state recovery unit 215b, and the manual recovery plan notification unit 216b.

  For example, the reconfiguration data generation unit 212b generates reconfiguration data of a fault location to be reconfigured from the fault location information obtained by the configuration recovery unit 214b and the circuit data 213b. Further, the configuration recovery unit 214b transmits the reconfiguration data to the configuration information storage management unit 317A, and performs automatic recovery (circuit reconfiguration) of the reconfigurable circuit unit 319A. After the reconfiguration of the circuit of the reconfigurable circuit unit 319A is completed, the state recovery unit 215b copies the state from the recovery block having a normal state and transfers the state to the first bank through the state storage management unit 313A. Overwrite 315A1.

  When the received recovery plan is a manual recovery plan, the manual recovery plan notifying unit 216b notifies the outside as manual recovery plan information. Here, the circuit data 213b exists in the recovery plan execution unit 21b. However, as long as the circuit data 213b is tripled between the processing systems, the circuit data 213b may exist in other different locations in the processing block and the diagnostic block.

<< Detailed operation of the recovery plan execution unit (in the recovery destination processing system) >>
FIG. 11 is a flowchart showing an example of processing contents when the recovery plan execution unit belongs to the recovery destination processing system in the recovery plan execution unit of FIG. The flowchart in FIG. 11 is executed as the failure recovery process in step S307 in FIG. In FIG. 11, first, the recovery plans input to the recovery plan execution units 21b to 21d are received by the recovery control units 211b to 211d, and the recovery control units 211b to 211d check whether the recovery plan is a manual recovery plan. (Step S1101). If the recovery plan is not a manual recovery plan, the process proceeds to step S1102, and automatic recovery processing based on the automatic recovery plan is performed. Hereinafter, a case where the recovery destination processing system indicated by the automatic recovery plan is the processing system AA and the recovery processing system is the processing system BB will be described as an example.

  In step S1102, the recovery control unit 211b broadcasts a reconfiguration start notification to the other recovery control units 211aα, 211c, and 211d using the first communication unit 301A and the communication path L2. Next, the recovery control unit 211b reconfigures the reconfigurable circuit unit 319A (step S1103).

  Specifically, first, the recovery control unit 211b requests information on the failure location from the configuration recovery unit 214b. The configuration recovery unit 214b acquires failure location information in the same manner as the processing of steps S403 to S405 of the configuration failure location diagnosis unit 302A. If acquisition succeeds, the configuration recovery unit 214b sends the failure location information to the reconfiguration data generation unit 212b. hand over. The reconfiguration data generation unit 212b uses the failure location information and the circuit data 213b to generate reconfiguration data for a location that needs to be reconfigured and passes it to the configuration recovery unit 214b. The configuration recovery unit 214b sends the reconfiguration data to the configuration information storage management unit 317A using the second communication units 303A and 316A. The configuration information storage management unit 317A overwrites the configuration information storage unit 318A based on the reconfiguration data, and restores the normal circuit. When the restoration is normally completed, the configuration information storage management unit 317A notifies the recovery control unit 211b via the configuration recovery unit 214b.

  Subsequently, the recovery control unit 211b checks whether the reconfiguration has been completed normally (step S1104). When the reconfiguration is normally completed, the recovery control unit 211b broadcasts a state recovery start notification to the other recovery control units 211aα, 211c, and 211d using the first communication unit 301A and the communication path L2. Next, the recovery control unit 211b transfers control to the state recovery unit 215b, and receives a normal state and overwrites an existing state (step S1106).

  Specifically, the state recovery unit 215b communicates with the state recovery unit 215c in the normal recovery processing block using the first communication unit 301A and the communication path L2, and stores the normal state in the processing block. receive. The state recovery unit 215b transmits the received normal state to the state storage management unit 313A using the second communication units 303A and 316A. The state storage management unit 313A overwrites the received normal state on the state storage unit 314A. After the completion of overwriting, the state storage management unit 313A notifies the state recovery unit 215b of the fact using the second communication units 303A and 316A.

  Next, the state recovery unit 215b checks whether or not the state recovery has ended normally (step S1107). When the state recovery is completed normally, the state recovery unit 215b notifies the recovery control unit 211b of the fact, and the recovery control unit 211b notifies the other recovery control units 211aα, 211c, and 211d of the end of the state recovery. Broadcast using one communication unit 301A and the communication path L2 (state S1108). At this time, as will be described in detail later with reference to FIG. 14, the input data re-insertion unit 217aα of the external transmission / reception block 10α that has received the state recovery end notification is received during the automatic recovery process and is still processing. Start re-inputting the input data that has not been done. The stateful processing unit 312A sequentially processes the input data that has been input again.

  Subsequently, the recovery control unit 211b determines whether the result of processing the input data sent from the input data re-input unit 217aα has caught up with the result of processing by another normal processing system with respect to the majority decision determining unit 14b. Confirmation is made (step S1109). As a result of the confirmation, if it has not caught up, the recovery control unit 211b executes Step S1109 once again. As a result of the confirmation, if it has caught up, the recovery control unit 211b performs a recovery completion agreement with the other recovery control units 211aα, 211c, and 211d in the same manner as the above-described recovery plan agreement (step S1110). ). Next, the recovery control unit 211b checks whether or not the recovery completion agreement has been normally obtained (step S1111). When the recovery completion agreement is normally performed, the recovery control unit 211b shifts to the normal mode assuming that the recovery is completed (step S1112). Specifically, the recovery control unit 211b overwrites the mode information 22b to the normal mode, and ends the automatic recovery process.

  In step S1104, if the reconfiguration is not completed normally, the recovery control unit 211b creates a manual recovery plan including that fact (step S1113). In step S1107, when the state recovery is not normally completed, the recovery control unit 211b creates a manual recovery plan including the fact (step S1114). In step S1111, when the recovery completion agreement is not normally performed, the recovery control unit 211b creates a manual recovery plan including the fact (step S1115). In step S1101, if the recovery plan is a manual recovery plan, or if a manual recovery plan is created in steps S1113 to S1115, details will be described later with reference to FIG. 15, but the recovery controllers 211aα, 211b to 211d, and The manual recovery plan notification units 216aα and 216b to 216c perform manual recovery processing based on the manual recovery plan (step S1116).

<< Detailed operation of the recovery plan execution unit (in the recovery system) >>
FIG. 12 is a flowchart illustrating an example of the processing contents when the recovery plan execution unit belongs to the processing system of the time restoration in the recovery plan execution unit of FIG. The flowchart in FIG. 12 is executed as the failure recovery process in step S307 in FIG. In FIG. 12, first, the recovery plans input to the recovery plan execution units 21b to 21d are received by the recovery control units 211b to 211d, and the recovery control units 211b to 211d check whether the recovery plan is a manual recovery plan. (Step S1201). If the recovery plan is not a manual recovery plan, the process proceeds to step S1202, and automatic recovery processing based on the automatic recovery plan is performed. Hereinafter, a case where the recovery destination processing system indicated by the automatic recovery plan is the processing system AA and the recovery processing system is the processing system BB will be described as an example.

  In step S1202, the recovery control unit 211c waits until the recovery control unit 211b of the recovery destination processing system issues a reconfiguration start notification until time-out, and checks whether the time-out has occurred (step S1202). When the reconfiguration start notification can be received before the time-out, the recovery control unit 211c waits until the recovery destination recovery control unit 211b notifies the start of the state recovery until time-out, and checks whether the time-out has occurred. (Step S1203).

  When the state recovery start notification can be received before the time-out, the recovery control unit 211c passes control to the state recovery unit 215c. The state recovery unit 215c requests the state storage management unit 313B to switch the state write destination to the second bank 315B2 using the second communication units 303B and 316B (step S1204). Next, the state recovery unit 215c transmits the state on the first bank 315B1 to the recovery destination state recovery unit 215b using the first communication unit 301B and the communication path L2 (step S1205).

  Specifically, the state recovery unit 215c requests the state storage management unit 313B to read the state on the first bank 315B1 using the second communication units 303B and 316B. The state storage management unit 313B reads the state on the first bank 315B1, and transmits the state to the state recovery unit 215c using the second communication units 316B and 303B. The state recovery unit 215c transmits the transmitted normal state to the recovery destination state recovery unit 215b using the first communication unit 301B and the communication path L2, and returns control to the recovery control unit 211c.

  Next, the recovery control unit 211c waits until the recovery control unit 211b of the recovery destination issues a state recovery end notification until time-out, and checks whether or not the time-out has occurred (step S1206). When the end notification of the state recovery can be received before the time-out, the recovery control unit 211c passes control to the state recovery unit 215c. The state recovery unit 215c requests the state storage management unit 313B to use the second communication units 303B and 316B to return the state write destination to the first bank 315B1 (step S1207). The details will be described later with reference to FIG. 14. At this time, the input data re-input unit 217aα included in the external transmission / reception block 10α that has received the state recovery end notification is received during the automatic recovery process, and the processing is still in progress. Start re-inputting the input data that has not been done.

  Subsequently, the recovery control unit 211c processes the input data sent from the input data re-input unit 217aα by the processing result of the normal processing system including itself to the majority determination unit 14c. It is confirmed whether or not the processing result of the processing system has been caught up (step S1208). As a result of the confirmation, if it is not caught up, the recovery control unit 211c executes Step S1208 once again. As a result of the confirmation, if it is caught up, the recovery control unit 211c performs a recovery completion agreement with the other recovery control units 211aα, 211b, and 211d in the same manner as the above-described recovery plan agreement (step S1209). ). Next, the recovery control unit 211c checks whether or not the recovery completion agreement has been normally obtained (step S1210). When the recovery completion agreement is normally performed, the recovery control unit 211c shifts to the normal mode on the assumption that the recovery is completed (step S1211). Specifically, the recovery control unit 211c overwrites the mode information 22c to the normal mode, and ends the automatic recovery process.

  If a time-out occurs in step S1202, the recovery control unit 211c creates a manual recovery plan including that fact (step S1212). When a time-out occurs in step S1203, the recovery control unit 211c creates a manual recovery plan including that fact (step S1213). When a time-out occurs in step S1206, the recovery control unit 211c creates a manual recovery plan including that fact (step S1214). In step S1210, when the recovery completion agreement is not normally performed, the recovery control unit 211c creates a manual recovery plan including the fact (step S1215). In step S1201, if the recovery plan is a manual recovery plan, or if a manual recovery plan is created in steps S1212 to S1215, the details will be described later with reference to FIG. 15, but recovery controllers 211aα, 211b to 211d, and The manual recovery plan notification units 216aα and 216b to 216c perform manual recovery processing based on the manual recovery plan (step S1216).

<< Detailed operation of the recovery plan execution unit (in the recovery destination and non-restore processing systems) >>
FIG. 13 is a flowchart illustrating an example of processing contents when the recovery plan execution unit of FIG. 10 does not belong to either the recovery destination or the recovery system. The flowchart in FIG. 13 is executed as the failure recovery process in step S307 in FIG. In FIG. 13, first, the recovery plans input to the recovery plan execution units 21b to 21d are received by the recovery control units 211b to 211d, and the recovery control units 211b to 211d check whether the recovery plan is a manual recovery plan. (Step S1301). If the recovery plan is not a manual recovery plan, the process proceeds to step S1302, and automatic recovery processing based on the automatic recovery plan is performed. Hereinafter, a case where the recovery destination processing system indicated by the automatic recovery plan is the processing system AA and the recovery processing system is the processing system BB will be described as an example.

  In step S1302, the recovery control unit 211d waits for the recovery destination recovery control unit 211b to notify the start of reconfiguration until time-out, and checks whether time-out has occurred (step S1302). When the reconfiguration start notification can be received before the time-out, the recovery control unit 211d waits until the recovery destination recovery control unit 211b performs the state recovery start notification until the time-out, and checks whether the time-out has occurred. (Step S1303). When the state recovery start notification can be received before the time-out, the recovery control unit 211d waits until the recovery destination recovery control unit 211b notifies the end of the state recovery until time-out, and checks whether the time-out has occurred ( Step S1304).

  When the state recovery end notification can be received before the timeout, the input data re-input unit 217aα of the external transmission / reception block 10α having received the state recovery end notification will be described later in detail with reference to FIG. Re-input of the input data received during the process and not yet processed is started. The recovery control unit 211d is a recovery destination processing system in which the processing result of the normal processing system including itself is processing the input data sent from the input data re-input unit 217aα to the majority decision determining unit 14d. It is confirmed whether or not the processing result has been caught up (step S1305). As a result of the confirmation, if it is not caught up, the recovery control unit 211d executes Step S1305 once again. As a result of the confirmation, if the catch-up is overtaken, the recovery control unit 211d makes a recovery completion agreement with the other recovery control units 211aα, 211b, and 211c in the same manner as the above-described recovery plan agreement (step S1306). ). Next, the recovery control unit 211d confirms whether or not the recovery completion agreement has been normally obtained (step S1307). When the recovery completion agreement is normally performed, the recovery control unit 211d shifts to the normal mode assuming that the recovery is completed (step S1308). Specifically, the recovery control unit 211d overwrites the mode information 22d to the normal mode, and ends the automatic recovery process.

  If a time-out occurs in step S1302, the recovery control unit 211d creates a manual recovery plan including the fact (step S1309). When a time-out occurs in step S1303, the recovery control unit 211d creates a manual recovery plan including that fact (step S1310). When a time-out occurs in step S1304, the recovery control unit 211d creates a manual recovery plan including that fact (step S1311). In step S1307, when the recovery completion agreement is not normally performed, the recovery control unit 211d creates a manual recovery plan including the fact (step S1312). In step S1301, if the recovery plan is a manual recovery plan, or if a manual recovery plan is created in steps S1309 to S1312, the details will be described later with reference to FIG. 15, but recovery controllers 211aα, 211b to 211d, and The manual recovery plan notification units 216aα and 216b to 216c perform manual recovery processing based on the manual recovery plan (step S1313).

<< Detailed operation of recovery plan execution unit (in external transmission / reception block) >>
FIG. 14 is a flowchart showing an example of processing contents of the recovery plan execution unit of FIG. The flowchart in FIG. 14 is executed as the failure recovery process in step S307 in FIG. In FIG. 14, first, the recovery plan input to the recovery plan execution unit 21aα is received by the recovery control unit 211aα, and the recovery control unit 211aα checks whether the recovery plan is a manual recovery plan (step S1401). If the recovery plan is not a manual recovery plan, the process proceeds to step S1402, and automatic recovery processing based on the automatic recovery plan is performed. Hereinafter, a case where the recovery destination processing system indicated by the automatic recovery plan is the processing system AA and the recovery processing system is the processing system BB will be described as an example.

  In step S1402, the recovery control unit 211aα stops transmitting input data to the first communication unit 311A in the recovery destination processing system (step S1402). Next, the recovery control unit 211aα waits until the recovery destination recovery control unit 211b notifies the start of reconfiguration until time-out, and checks whether time-out has occurred (step S1403). If the reconfiguration start notification can be received before the time-out, the recovery control unit 211aα waits until the recovery destination recovery control unit 211b notifies the start of the state recovery until the time-out, and checks whether the time-out has occurred ( Step S1404).

  When the state recovery start notification can be received before the time-out, the recovery control unit 211aα captures the input data from the input data input unit 12α in the input data reinput unit 217aα and stores it in the input data reinput queue 218aα. It requests to continue (step S1405). Subsequently, the recovery control unit 211aα waits until the recovery destination recovery control unit 211b notifies the end of the state recovery until time-out, and checks whether time-out has occurred (step S1406). When the state recovery end notification can be received before the time-out, the recovery control unit 211aα requests the input data input unit 12α to reduce the input data input speed to the normal processing system (step S1407).

  Next, the recovery control unit 211aα requests the input data re-input unit 217aα to retrieve one input data from the input data re-input queue 218aα and send it to the recovery destination processing system (step S1408). Subsequently, the recovery control unit 211aα gives the majority decision determination unit 14aα the processing result of the recovery destination processing system that has processed the input data sent from the input data re-input unit 217aα, from other normal processing systems. It is confirmed whether or not it has caught up with the processing result (step S1409).

  As a result of the confirmation, if it has not caught up, the recovery control unit 211aα proceeds to step S1408 again. As a result of the confirmation, if it has caught up, the recovery control unit 211aα stops “capturing input data from the input data input unit 12α and storing it in the input data reinput queue 218aα” in the input data reinput unit 217aα. Request. Furthermore, the recovery control unit 211aα requests the input data input unit 12α to return the input data input speed to the normal processing system to the normal speed (step S1410).

  Subsequently, the recovery control unit 211aα performs recovery completion agreement with the other recovery controls 211b to 211d in the same manner as the recovery plan agreement described above (step S1411). Next, the recovery control unit 211aα confirms whether or not the recovery completion agreement has been normally obtained (step S1412). When the recovery completion agreement is normally performed, the recovery control unit 211aα shifts to the normal mode assuming that the recovery is completed (step S1413). Specifically, the recovery control unit 211aα overwrites the mode information 22aα to the normal mode, and ends the automatic recovery process.

  When a time-out occurs in step S1403, the recovery control unit 211aα creates a manual recovery plan including that fact (step S1414). When time-out occurs in step S1404, the recovery control unit 211aα creates a manual recovery plan including that fact (step S1415). When a time-out occurs in step S1406, the recovery control unit 211aα creates a manual recovery plan including the fact (step S1416). In step S1412, if the recovery completion agreement is not normally performed, the recovery control unit 211aα creates a manual recovery plan including the fact (step S1417). In step S1401, if the recovery plan is a manual recovery plan, or if a manual recovery plan is created in steps S1414 to S1417, details will be described later with reference to FIG. 15, but recovery controllers 211aα, 211b to 211d, and The manual recovery plan notification units 216aα and 216b to 216c perform manual recovery processing based on the manual recovery plan (step S1418).

<< Detailed operation of the recovery plan execution unit (during manual recovery processing) >>
FIG. 15 is a flowchart illustrating an example of the processing content of the manual recovery process performed by the recovery plan execution unit in the information processing systems of FIGS. 1A to 1C. The flowchart in FIG. 15 is executed by the recovery plan execution units 21aα and 21b to 21d as the manual recovery process in steps S1116, S1216, S1313, and S1418 described above.

  In FIG. 15, first, the manual recovery plan notification units 216aα and 216b to 216d use the manual recovery plan received via the recovery control units 211aα and 211b to 211d as the manual recovery plan information, for example, logs provided in each processing system. The data is transmitted to a recording unit (not shown) or a monitoring device (not shown) connected to the communication path L2 (step S1501). Next, the recovery control units 211aα, 211b to 211d determine whether they are a faulty system from the information in the manual recovery plan. If it is determined that it is a faulty system, any of the recovery control units 211aα, 211b to 211d corresponding to the faulty system rewrites the mode information 22aα and 22b to 22d to the manual recovery waiting mode (step S1503).

  Subsequently, for example, an administrator who manages the monitoring apparatus performs a manual recovery process based on the manual recovery plan information (step S1504). Next, when any of the recovery control units 211aα, 211b to 211d corresponding to the failure system has completed the manual recovery process and is notified by the administrator or the like, the mode information 22aα, 22b to 22d is manually recovered. The mode is rewritten to the completed mode (step S1505). Next, one of the recovery control units 211aα, 211b to 211d corresponding to the faulty system notifies the other recovery control units 211aα, 211b to 211d of the transition to the manual recovery completed mode (step S1506). After that, any of the recovery controllers 211aα, 211b to 211d corresponding to the failure system rewrites the mode information 22aα, 22b to 22d to the failure recovery mode, and proceeds to the above-described step S301 (step S1507).

  On the other hand, if it is not determined in step S1502 that the system is a faulty system, the recovery controllers 211aα, 211b to 211d corresponding to the non-failed system rewrite the mode information 22aα and 22b to 22d to the degenerate operation mode (step S1508). ). Next, the recovery control units 211aα and 211b to 211d corresponding to the non-failed system wait for notification of transition from the failed system to the manual recovery completed mode, and check whether the notification has arrived (step S1509). If the notification has not arrived, step S1509 is executed once again. When the notification arrives, the recovery control units 211aα, 211b to 211d corresponding to the non-failure system rewrite the mode information 22aα and 22b to 22d to the failure recovery mode, and proceed to Step S301 described above (Step S1510).

<Details of mode information>
FIG. 16 is a state transition diagram showing an example of each mode stored in the mode information and an example of the relationship in the information processing system of FIGS. 1A to 1C. As shown in FIG. 16, the modes stored in the mode information 22aα, 22b-22d are the normal mode 160, the failure recovery mode 161, the manual recovery wait mode 162, the manual recovery completed mode 163, and the degenerate operation mode 164. Consists of

  Transition 160161 from the normal mode 160 to the failure recovery mode 161 is performed in step S302. The transition 161160 from the failure recovery mode 161 to the normal mode 160 is performed by Step S1112, Step S1211, Step S1308, and Step S1413. The transition 161116 from the failure recovery mode 161 to the manual recovery wait mode 162 is performed in step S1503. The transition 161164 from the failure recovery mode 161 to the degenerate operation mode 164 is performed in step S1508. The transition 162163 from the manual recovery wait mode 162 to the manual recovery completed mode 163 is performed in step S1505. Transition 163161 from the manual recovery completed mode 163 to the failure recovery mode 161 is performed in step S1507. Transition 164161 from the degenerate operation mode 164 to the failure recovery mode 161 is performed in step S1510.

  For example, the mode information 22aα and 22b to 22d are both in the failure recovery mode 161 while the automatic recovery process is being performed, and are in the normal mode 160 when the automatic recovery process is completed. Further, for example, while the manual recovery process is being performed, the mode information of the fault system that is the recovery destination of the manual recovery process is the manual recovery waiting mode 162, and the other mode information is the degenerate operation mode 164. When the manual recovery process of the fault system is completed, the mode information of the fault system becomes the manual recovery completed mode 163, and the other mode information becomes the fault recovery mode 161 in response to this. Thereafter, the mode information of the fault system becomes the normal mode 160 through the fault recovery mode 161, and the other mode information also becomes the normal mode 160.

  Each processing system whose mode information is the degenerate operation mode 164 performs the degenerate operation. That is, for example, when a failure requiring a manual recovery process occurs in the processing system AA, the mode information of the processing system BB, the processing system CC, and the external transmission / reception block 10α are all in the degenerate operation mode 164, and the processing system BB and the processing system The system CC continues the process according to the input data from the external transmission / reception block 10α as it is. At this time, the majority decision determination unit 14aα of the external transmission / reception block 10α receives two processing results from the two processing systems, and if the two processing results are the same, the same processing result is externally transmitted. Output to. On the other hand, the majority decision determination unit 14aα stops the system or the like when the two processing results are different.

  Further, while the mode information is the failure recovery mode 161 and the automatic recovery processing is being performed in each processing system, the remaining two processing systems excluding the recovery destination processing system and the external transmission / reception block are in the degenerate operation described above. The degenerate operation is performed in the same manner as in the mode 164. During this degenerate operation period, it is difficult for the information processing system to correct the error using the majority decision determination unit, but it is possible to detect the error. As a result, it is possible to prevent an erroneous processing result from leaking to the outside while continuing the operation of the information processing system, and to improve the reliability of the information processing system from the viewpoint of fail-safe or fail software. Become.

<< Details of system configuration information >>
FIG. 17 is a diagram illustrating an example of contents stored in the system configuration information in the information processing systems of FIGS. 1A to 1C. As shown in FIG. 17, the system configuration information 17aα, 17b to 17d includes, for example, connection relationships between the external transmission / reception blocks 10α and 10β, the diagnosis blocks 300A to 300C, and the processing blocks 310A to 310C shown in FIG. Is retained. The system configuration information 17aα, 17b to 17d is used, for example, when performing communication or resolving the dependency between systems when a failure occurs.

<Other processing contents of the fault diagnosis information sharing unit>
FIG. 18 is a flowchart illustrating an example of processing contents different from those in FIG. 6 performed by the failure diagnosis information sharing unit in the information processing systems of FIGS. 1A to 1C. 6 is executed by the failure diagnosis information sharing units 18aα and 18b to 18d in the failure recovery mode, but FIG. 18 is in the normal mode (that is, no error determination result is received from the majority decision determination unit). ) It is executed by the fault diagnosis information sharing unit 18aα, 18b to 18d, and is executed in parallel with the flowchart of FIG.

  In FIG. 18, the fault diagnosis information sharing units 18aα, 18b to 18d wait for fault diagnosis information from other fault diagnosis recovery systems (step S1801). When fault diagnosis information is sent from another fault diagnosis recovery system, the fault diagnosis information sharing units 18aα and 18b to 18d execute the above-described flowchart of FIG. 3 (step S1802). Here, for example, it is assumed that a failure occurs in the majority decision determination unit 14aα in the failure diagnosis recovery system aα, and failure diagnosis information is sent from the failure diagnosis recovery system aα toward the failure diagnosis information sharing units 18b to 18d. .

  In this case, each failure diagnosis recovery system b to d shifts to the failure recovery mode in step S302 of FIG. 3, and generates failure diagnosis information in step S303. The failure diagnosis information includes, for example, information indicating that no failure is detected by its own majority decision unit, or information indicating that failure diagnosis information is received only from the failure diagnosis recovery system aα. Thereafter, in step S304, the respective fault diagnosis information sharing units 18aα and 18b to 18d share the fault diagnosis information. In this case, the failure diagnosis information sharing units 18b to 18d agree that no failure is detected or that failure diagnosis information is received only from the failure diagnosis recovery system aα, and only failure diagnosis information of the failure diagnosis information sharing unit 18aα is obtained. Will be different. In this case, the failure diagnosis information sharing units 18b to 18d (or the recovery plan determination units 19b to 19d) determine that the failure diagnosis recovery system aα has a failure, notify the external transmission / reception block 10β to that effect, Switching from the transmission / reception block 10α to the external transmission / reception block 10β is performed.

  Further, for example, it is assumed that a failure occurs in the majority decision determination unit 14b in the failure diagnosis recovery system b, and failure diagnosis information is sent from the failure diagnosis recovery system b to the failure diagnosis information sharing units 18aα, 18c, and 18d. To do. In this case, each failure diagnosis recovery system aα, c, d shifts to the failure recovery mode in step S302 of FIG. 3, and generates failure diagnosis information in step S303. The fault diagnosis information has, for example, content indicating that no fault is detected by its own majority determination unit, or content indicating that fault diagnosis information has been received only from the fault diagnosis recovery system b.

  Thereafter, in step S304 of FIG. 3, the fault diagnosis information sharing units 18aα and 18b to 18d share the fault diagnosis information. In this case, the failure diagnosis information sharing unit 18aα, 18c, 18d agrees that a failure is not detected or that failure diagnosis information is received only from the failure diagnosis recovery system b. Only the information will be different. In this case, in step S305 in FIG. 3, the recovery plan determination units 19aα, 19c, and 19d determine that there is a failure in the diagnostic block 300A, and create a manual recovery plan indicating that fact. Thereafter, the processes of steps S306 and S307 are executed.

  Here, since it is desirable to switch the external transmission / reception block early when there is a failure in the failure diagnosis and recovery system aα, the failure diagnosis information sharing units 18b to 18d (or the recovery plan determination units 19b to 19d) perform the switching. The explanation is given with an example of instructing. However, depending on the case, it is also possible to use a process in which the recovery plan execution unit instructs the switching after the processes of steps S305 to S307 in FIG. Further, in some cases, a process may be used in which the fault diagnosis information sharing units 18b to 18d are instructed to switch the external transmission / reception block when receiving only the fault diagnosis information from the fault diagnosis recovery system aα. Is possible. In any case, failure diagnosis information is shared by the failure diagnosis information sharing units 18aα, 18b to 18d, so that failure diagnosis information is generated from only one of the failure diagnosis information sharing units (ie, It is possible to detect a situation in which a failure has occurred in the majority decision unit or the like.

  Further, the flowchart of FIG. 3 may be activated in parallel from two steps of step S208 and step S1802. Therefore, step S301 and step S302 need to be executed as a single process so that two steps operating in parallel do not simultaneously shift to the failure recovery mode.

《How to exchange fault diagnosis information in the fault diagnosis information sharing unit》
FIG. 19 is a sequence diagram illustrating an example of a method for exchanging fault diagnosis information in the flowchart of FIG. As shown in FIG. 19, each failure diagnosis recovery system aα, b-d uses the data (in this case, failure diagnosis information) A190, B190-D190 generated by itself in step S602 of FIG. Data is exchanged by broadcasting to the recovery system. Thereafter, each of the fault diagnosis recovery systems aα, b to d combines the data 191 obtained by the exchange with the data generated by itself (any one of A190, B190 to D190) in step S603 of FIG. If all of A190 to D190 are aligned, it is determined to be successful.

《Recovery plan exchange method in Recovery Plan Agreement Department》
20 and 21 are sequence diagrams illustrating an example of an exchange method based on the recovery plan Byzantine agreement protocol in the flowchart of FIG. 7. FIG. 20 shows the processing contents when the data (here, the recovery plan) generated by the system a is shared using the Byzantine agreement protocol. Byzantine agreement protocol is an agreement between n tasks, that is, when sharing the same data, there is a failure in m tasks, but it is not clear which task has a failure. This is a protocol for sharing the same data among the tasks other than the task in which the failure has occurred, and obtaining a correct agreement. In order to share data, one task sends data to the other (n-1) tasks. At this time, an agreement is established when the following three conditions are established.

  First, an agreement is established regarding the received data among all the tasks having no failure. Second, if the transmission task is normal, the data agreed in the first condition matches the actually transmitted data. Third, when m tasks contain faults under the first and second conditions, the Byzantine agreement is established because the relation n ≧ 3m + 1 is established. Is the time.

  The Byzantine agreement protocol is executed for reliable sharing of the recovery plan in step S702 of FIG. In FIG. 20, data (here, the recovery plan) created by the system a is indicated by A (200). Here, the description will be made on the assumption that the system b is a type of Byzantine failure that transmits wrong data when transmitting data. First, the system a broadcasts data A (200) to the other systems b to d as the first communication. A (200) received by the systems b to d is represented as Ai (i = 1, 2, 3) in the sense that it is the i-th received data with respect to A (200). Therefore, A1 (201) is set. When correctly received, A1, A2, and A3 are all the same data.

  Next, the systems b to d that have received the data A1 (201) transmit A1 (201) to itself and the remaining two systems other than the system a as the second communication. Since the systems c and d operate normally, they can receive the correct data A2 (202) and A3 (203), but the system b receives erroneous data (here, A ′, A in the transmission). Since the Byzantine failure is a type that transmits “”), the systems c and d receive erroneous data A′3 (204) and A ″ 3 (205).

  Next, the systems b to d make a majority decision on three data received from different paths, A1 (201) A2 (202) and A3 (203) or A′3 (204) or A ″ 3 (205). As a result of the determination, A206 to A208 are obtained.By using the Byzantine agreement protocol in this way, the lines c and d receive erroneous data A′3 (204) and A ″ 3 (205) from the line b. Nevertheless, the correct results A (207) and A (208) can be obtained.

  In FIG. 21, using the Byzantine agreement protocol, each of the systems a, b, c, and d has its own data (here, a recovery plan) A (210), B (similar to the processing contents shown in FIG. 211), C (212), and D (213). In FIG. 21, description will be made on the assumption that the system a is a Byzantine failure that transmits wrong data when data is transmitted. That is, assuming a single failure, for example, the first communication unit 13α, the failure diagnosis information sharing unit 18aα, and the like will be described when a Byzantine failure occurs.

  As described above with reference to FIG. 20, each of the systems a to d receives three pieces of data for data other than its own system according to the Byzantine agreement protocol. In this example, the system a transmits erroneous data A ″ toward the system b, and erroneous data A ′ toward the systems b and c. The system a also transmits data B from the system b. In response, the erroneous data B ′ is transmitted to the systems c and d, the erroneous data C ′ is transmitted to the systems b and d in response to the data C from the system c, and the data D from the system d is transmitted. In response, erroneous data D ′ is transmitted to the systems b and c.

  Thereby, a part of erroneous data is included in the data received by the systems b to d. However, after the exchange of data is completed, the majority of the three data received from different routes of the systems a to d is determined. As a result, the system a is A, B, C, D, and the systems b to d are A ′. , B, C, D can be obtained. Thus, by the Byzantine agreement protocol, the data exchanged in all the lines can be made the same in at least the lines other than the line in which the Byzantine disorder has occurred. That is, when a Byzantine disorder occurs in the line a, the lines b to d may not obtain the data A of the line a correctly, but can correctly obtain the data B to D of the lines b to d.

  As a result, the three systems b to d other than the system in which the Byzantine failure occurs are based on the coincidence of the data (here, the recovery plans) B to D, and the operation after the occurrence of the failure (manual operation with the system a as the failure occurrence location) Recovery processing, degenerate operation, etc.) can be executed correctly. If there is a Byzantine failure, correct automatic recovery processing may not be performed. Therefore, as shown in FIG. 21, for example, when there is a need for a majority decision such as making a majority decision based on two data C and one data C ′ without obtaining three data C. In step S704 of FIG. 7, it is desirable to process the recovery plans as inconsistent.

  As described above, by using the information processing system of the first embodiment, it is typically possible to improve reliability.

(Embodiment 2)
<< Configuration of Processing Block (Modification) >>
FIG. 22 is a block diagram showing a configuration example of the processing block in the information processing system according to the second embodiment of the present invention. A processing block 310A shown in FIG. 22 is a modification of the processing block 310A in the information processing system of FIG. 1 described above. Here, the processing block 310A of FIG. 1 will be described as a representative, but the configuration example of FIG. 22 is also applied to the processing blocks 310B and 310C of FIG.

  The processing block 310A shown in FIG. 22 differs from the processing block of FIG. 1 in the configuration in the reconfigurable circuit unit 319A, and the other configurations are the same as those in FIG. A reconfigurable circuit unit 319A shown in FIG. 22 includes triple stateful processing units [1] 312A1 to [3] 312A3 instead of the stateful processing unit 312A in the reconfigurable circuit unit of FIG. It has.

  The stateful processing unit [1] 312A1 to [3] 312A3 uses the input data input via the first communication unit 311A and the state input from the state storage unit 314A via the state storage management unit 313A. The same processing is performed, and the processing result (output data) and state obtained thereby are output to majority decision section 320A. The majority decision determination unit 320A performs a majority decision on the three processing results (output data), outputs the processing result (output data) obtained thereby through the first communication unit 311A, and also determines the three states. The majority decision is made, and the state obtained thereby is written to the state storage unit 314A via the state storage management unit 313A.

  As described above, the information processing system according to the second embodiment makes a majority decision in the reconfigurable circuit unit, and also makes a majority decision in the external transmission / reception block 10α and the diagnostic blocks 300A to 300C in FIG. It is configured to perform a two-step majority decision. When such a configuration example is used, for example, even when a failure occurs in any one of the stateful processing units [1] 312A1 to [3] 312A3, information processing is performed without performing recovery processing, degenerate operation, or the like. Processing as a system can be performed continuously. In other words, the fault tolerance of the processing block can be apparently improved.

  However, if a failure occurs in the majority decision unit 320A in the reconfigurable circuit unit, recovery processing is required. In addition, the information processing system according to the second embodiment may cause an increase in circuit scale or latency in the reconfigurable circuit unit as compared with the information processing system in FIG. 1 according to the first embodiment. Therefore, it is considered that the information processing system of FIG. 1 in the first embodiment is more desirable than the information processing system of the second embodiment in terms of the balance of circuit scale, latency, and reliability.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiments. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to one having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. . Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  For example, here, in order to cope with a single failure including a Byzantine failure, three processing blocks 310A to 310C and four failure diagnosis recovery systems aα, b to d are provided. In this case, for example, five processing blocks for enabling majority decision and seven fault diagnosis recovery systems for obtaining a Byzantine agreement when two faults occur may be provided.

10α, 10β External transmission / reception block 11α, 11β External communication unit 12α, 12β Input data input unit 13α, 13β, 301A to 301C First communication unit 14aα, 14aβ, 14b-14d Majority determination unit 15α, 15β Output data input unit 16aα, 16aβ , 16b to 16d System failure diagnosis unit 17aα, 17aβ, 17b to 17d System configuration information 18aα, 18aβ, 18b to 18d Failure diagnosis information sharing unit 19aα, 19aβ, 19b to 19d Recovery plan determination unit 20aα, 20aβ, 20b to 20d Recovery plan Agreement unit 211aα Recovery control unit 211b Recovery control unit 212b Reconfiguration data generation unit 213b Circuit data 214b Configuration recovery unit 215b State recovery unit 216aα Manual recovery plan notification unit 216b Manual recovery plan notification unit 217aα Re-input unit 218aα Input data re-input queue 21aα, 21aβ, 21b to 21d Recovery plan execution unit 22aα, 22aβ, 22b to 22d Mode information 23aα, 23aβ, 23b to 23d Fault diagnosis unit 300A to 300C Diagnosis block 302A to 302C Configuration failure Location diagnosis unit 303A to 303C Second communication unit 310A to 310C Processing block 311A to 311C First communication unit 312A, 312A1 to 321A3 Stateful processing unit 313A to 313C State storage management unit 314A to 314C State storage unit 315A1 to 315C1 First bank 315A2 315C2 Second bank 316A to 316C Second communication unit 317A to 317C Configuration information storage management unit 318A to 318C Configuration information storage unit 320A Majority determination unit AA to CC processing Integrated L1, L2 channel aα, aβ, b~d fault diagnosis recovery system

Claims (11)

First to third processing blocks that execute the same processing according to a common input from the outside and output the first to third processing results, respectively;
First to third diagnostic blocks provided corresponding to the first to third processing blocks, respectively;
A fourth diagnostic block,
The Nth processing block (N = 1, 2, 3) is
An Nth configuration information storage unit for storing circuit configuration information;
An N-th reconfigurable circuit unit that constructs a circuit according to the configuration information of the circuit and outputs the N-th processing result according to the operation of the circuit;
The Mth diagnostic block (M = 1, 2, 3, 4)
When the first to third processing results are input and there is an error in any of the first to third processing results, an Mth error determination result specifying the processing result having the error is output. An M-th majority decision determination unit;
An Mth fault diagnosis unit that receives the Mth error determination result and diagnoses a fault;
An Mth communication unit for communicating with other diagnostic blocks,
The Mth failure diagnosis unit transmits Mth diagnosis information reflecting the Mth error determination result to another failure diagnosis unit through the Mth communication unit, and receives diagnosis information from the other failure diagnosis unit. An information processing system having an M-th sharing unit that respectively receives the M-th communication unit. The information processing system according to claim 1,
The M-th failure diagnosis unit further determines whether any of the first to third processing blocks has a failure based on the first to fourth diagnosis information shared by the M-th sharing unit. If any of the first to third processing blocks is faulty, the recovery destination and the information on the processing block to be restored that are determined from the first to third processing blocks are included. An Mth recovery plan determination unit for generating an Mth recovery plan;
The M-th diagnosis block further includes an M-th recovery plan execution unit that executes an automatic recovery process of the recovery-destination processing block based on the M-th recovery plan. The information processing system according to claim 2,
While the M-th recovery plan execution unit is executing automatic recovery processing of the recovery destination processing block, processing blocks other than the recovery destination processing block are processed in accordance with the common input from the outside. An information processing system that executes The information processing system according to claim 3.
The M-th failure diagnosing unit further transmits the M-th recovery plan generated by the M-th recovery plan determination unit to another recovery plan determination unit using the Byzantine agreement protocol via the M-th communication unit. And an M-th recovery plan consensus unit that outputs the M-th recovery plan determined based on the exchange result to the M-th recovery plan execution unit. The information processing system according to claim 4,
The Nth processing block further includes an Nth state storage unit that holds a state obtained in the process of executing the processing of the Nth reconfigurable circuit unit,
When the M-th recovery plan execution unit executes automatic recovery processing of the recovery destination processing block, in addition to the storage information of the configuration information storage unit in the recovery destination processing block, the storage information of the state storage unit An information processing system to recover. The information processing system according to claim 5,
The Nth state storage unit includes an Nth state saving unit that holds a state at a first time point,
The fourth recovery plan execution unit includes a data holding unit that sequentially holds data input from the outside after the first time point,
Each of the first to third recovery plan execution units,
A configuration recovery unit for recovering the storage information of the configuration information storage unit in the recovery destination processing block;
The state stored in the state saving unit in the recovery processing block is set for the state storage unit in the recovery destination processing block, and the storage information in the configuration information storage unit in the recovery destination processing block is An information processing system, comprising: a state recovery unit that sequentially outputs data held in the data holding unit in the fourth recovery plan execution unit toward the recovery destination processing block after recovery. The information processing system according to claim 4,
The fourth diagnostic block is provided in an external transmission / reception block serving as an interface with the outside,
The fourth majority decision unit in the fourth diagnosis block determines a processing result to be output to the outside by performing a majority decision on the first to third processing results,
The external transmission / reception block is made redundant including normal use and backup use,
Switching between the normal use and the backup use is performed based on the first to fourth diagnosis information shared by the Mth sharing unit. First to third processing blocks that execute the same processing according to a common input from the outside and output the first to third processing results, respectively;
First to third diagnostic blocks provided corresponding to the first to third processing blocks, respectively;
A fourth diagnostic block,
The Nth processing block (N = 1, 2, 3) is
An Nth configuration information storage unit for storing circuit configuration information;
Constructing a circuit according to the configuration information of the circuit and outputting the N-th processing result according to the operation of the circuit;
An Nth state storage unit that holds a state obtained in the process of executing the process of the Nth reconfigurable circuit unit,
The Mth diagnostic block (M = 1, 2, 3, 4)
When the first to third processing results are input and there is an error in any of the first to third processing results, an Mth error determination result specifying the processing result having the error is output. An M-th majority decision determination unit;
When the M-th error determination result is input, a failure is diagnosed, and if any of the first to third processing blocks is diagnosed as having a failure, a recovery destination and a recovery processing block An M-th failure diagnosis unit that generates an M-th recovery plan including the information of:
An Mth communication unit for communicating with other diagnostic blocks;
An M-th recovery plan execution unit that executes an automatic recovery process of the recovery destination processing block based on the M-th recovery plan generated by the M-th fault diagnosis unit;
The Mth failure diagnosis unit
The M-th diagnosis information reflecting the M-th error determination result is transmitted to another failure diagnosis unit via the M-th communication unit, and the diagnosis information from the other failure diagnosis unit is transmitted via the M-th communication unit. An M-th sharing unit for receiving each;
An M-th recovery plan determination unit that determines the M-th recovery plan based on the first to fourth diagnosis information shared by the M-th sharing unit;
When the M-th recovery plan execution unit executes automatic recovery processing of the recovery destination processing block, in addition to the storage information of the configuration information storage unit in the recovery destination processing block, the storage information of the state storage unit An information processing system to recover. The information processing system according to claim 8,
The Nth state storage unit includes an Nth state saving unit that holds a state at a first time point,
The fourth recovery plan execution unit includes a data holding unit that sequentially holds data input from the outside after the first time point,
Each of the first to third recovery plan execution units,
A configuration recovery unit for recovering the storage information of the configuration information storage unit in the recovery destination processing block;
The state stored in the state saving unit in the recovery processing block is set for the state storage unit in the recovery destination processing block, and the storage information in the configuration information storage unit in the recovery destination processing block is An information processing system, comprising: a state recovery unit that sequentially outputs data held in the data holding unit in the fourth recovery plan execution unit toward the recovery destination processing block after recovery. The information processing system according to claim 9,
The M-th failure diagnosing unit further transmits the M-th recovery plan generated by the M-th recovery plan determination unit to another recovery plan determination unit using the Byzantine agreement protocol via the M-th communication unit. And an M-th recovery plan consensus unit that outputs the M-th recovery plan determined based on the exchange result to the M-th recovery plan execution unit. The information processing system according to claim 9,
The fourth diagnostic block is provided in an external transmission / reception block serving as an interface with the outside,
The fourth majority decision unit in the fourth diagnosis block determines a processing result to be output to the outside by performing a majority decision on the first to third processing results,
The external transmission / reception block is made redundant including normal use and backup use,
Switching between the normal use and the backup use is performed based on the contents of the first to fourth diagnostic information shared by the Mth sharing unit and the match / mismatch.

Images