[go: up one dir, main page]

WO2014068774A1 - Information processing device, arithmetic processing device, and counter synchronization method - Google Patents

Information processing device, arithmetic processing device, and counter synchronization method Download PDF

Info

Publication number
WO2014068774A1
WO2014068774A1 PCT/JP2012/078543 JP2012078543W WO2014068774A1 WO 2014068774 A1 WO2014068774 A1 WO 2014068774A1 JP 2012078543 W JP2012078543 W JP 2012078543W WO 2014068774 A1 WO2014068774 A1 WO 2014068774A1
Authority
WO
WIPO (PCT)
Prior art keywords
arithmetic processing
cpu
tsc
value
counter
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2012/078543
Other languages
French (fr)
Japanese (ja)
Inventor
石田健祐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujitsu Ltd
Original Assignee
Fujitsu Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujitsu Ltd filed Critical Fujitsu Ltd
Priority to PCT/JP2012/078543 priority Critical patent/WO2014068774A1/en
Publication of WO2014068774A1 publication Critical patent/WO2014068774A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/52Program synchronisation; Mutual exclusion, e.g. by means of semaphores

Definitions

  • the present invention relates to an arithmetic processing apparatus equipped with a counter for counting a clock supplied from the outside as a timer.
  • partition function is installed in many computers (information processing devices) such as servers.
  • the partition function is a function that divides resources of one information processing apparatus into processing units (groups) called a plurality of partitions and operates independently for each partition.
  • each partition an arbitrary OS (Operating System) or application program (hereinafter abbreviated as “application”) can be executed.
  • each partition can be operated as one independent information processing apparatus. Since one information processing apparatus can be used as a plurality of information processing apparatuses, the partition function is useful for suppressing the number of information processing apparatuses installed.
  • the resources allocated to each partition often include a plurality of arithmetic processing devices that execute programs. This is because each partition usually requires high processing power.
  • the arithmetic processing device is, for example, a CPU (Central Processing Unit).
  • CPU Central Processing Unit
  • CPU is used as a general term for arithmetic processing devices.
  • each CPU In a partition to which a plurality of CPUs are assigned, each CPU is used for executing a program. In such a partition, one task may be executed in a distributed manner by a plurality of CPUs.
  • This TSC is a free running counter installed in the CPU, and counts the total number of clocks supplied from the outside after starting the operation. The count is cyclically incremented (incremented) from an initial value of 0 to a maximum value corresponding to the number of bits (for example, 64 bits).
  • TSC • 0 is set in TSC by resetting the installed CPU.
  • the TSC of each CPU is usually supplied with the clock from the same clock source so that the count-up time interval does not differ for each TSC. For this reason, each CPU can appropriately check the elapsed time by resetting each CPU belonging to the same partition within an allowable range.
  • HPET High Performance Event Timer
  • LSI Large Scale Integration
  • ICH I / O Controller Hub
  • multi-core CPUs are widely used as CPUs. Each core executes a thread which is the smallest execution unit of a program in parallel. Each core executing a thread may check the elapsed time. Therefore, in a partition having a multi-core CPU, many accesses to HPET are likely to be concentrated. Access concentration reduces processing performance. For this reason, at present, TSC is used rather than HPET for checking the elapsed time.
  • the partition function includes a dynamic partition (DP) function.
  • This DP function refers to the dynamic addition (Hot-add) of hardware resources such as CPU and memory to the partition where the OS is running, or the dynamic replacement (Hot-replace) of existing hardware resources. It is a function that makes it possible.
  • a hardware resource to be added or exchanged is usually modularized so that addition or exchange can be easily performed.
  • modularized hardware resources are referred to as “processing modules”.
  • the CPU present in the processing module is reset when it is newly installed for addition or replacement.
  • the TSC in the CPU existing in the newly installed processing module starts counting up from the initial value.
  • the processing module is installed without shutting down. Conventionally, a reset to a CPU existing in a newly installed processing module is automatically performed by recognition of the processing module. For this reason, the TSC value in the CPU existing in the newly installed processing module is usually greatly different from the TSC value in the CPU existing in another processing module belonging to the same partition.
  • one CPU does not necessarily execute all tasks that require confirmation of elapsed time. For this reason, when the difference between the TSC value in the CPU existing in the newly installed processing module and the TSC value in the CPU existing in another processing module belonging to the same partition is not within the allowable range, It is very likely that a task that cannot be handled properly will occur. Therefore, the TSC value in the CPU existing in the newly installed processing module is synchronized with the TSC value in the CPU existing in another processing module belonging to the same partition within the allowable range. There is a need.
  • the TSC value can be rewritten not only for reading but for only the lower bits.
  • the overall value is not always appropriate.
  • the TSC is a counter that counts up by input of a clock even when the installed processing unit is not operating.
  • a difference more than the time required to write the value of one TSC to another TSC occurs. The time is the total time required for reading a value from a certain TSC, transferring the read value, writing the read value to another TSC, and the like.
  • the above difference may be outside the allowable range depending on the task to be executed.
  • the difference increases due to the presence of a heavily loaded CPU or the concentration of data transfer. Therefore, it seems important to synchronize the TSC value in the newly added CPU so as to be more accurate with the TSC value in the operating CPU.
  • an object of the present invention is to provide a technique for synchronizing a newly added TSC value in a CPU with a TSC value in an operating CPU.
  • One system to which the present invention is applied includes one or more arithmetic processing devices equipped with a counter that cyclically counts the total number of clocks supplied from the outside after the operation starts, and one or more arithmetic processing devices.
  • a first arithmetic processing device that is not included is newly added, each of the first arithmetic processing device and each of the arithmetic processing devices in operation among the one or more arithmetic processing devices Stop control means for stopping all mounted counters, reading means for reading the value of the counter mounted in one of the arithmetic processing units after stopping by the stop control means, and values read by the reading means
  • the value of a counter such as a TSC in a CPU that is newly added can be synchronized with the value of a counter such as a TSC in an operating CPU.
  • FIG. 10 is a sequence diagram showing an operation flow of an MMB, an existing system board, and a newly added system board when a new system board is added to a partition. It is a figure explaining the synchronization method of TSC by this embodiment. It is a flowchart showing the process which each CPU performs when a system board is added. It is a flowchart showing the process which each CPU performs when a system board is added (continuation).
  • FIG. 1 is a diagram illustrating a configuration example of the information processing apparatus according to the present embodiment.
  • the information processing apparatus according to the present embodiment includes a plurality of system boards (SB) 1 (1-1 to 1-4) and a plurality of IO (Input / Output) units 2 (2-1 to 2). -4), a crossbar switch 3, an MMB (ManageMent Board) 4, and a LAN 5.
  • SB system boards
  • IO Input / Output
  • MMB ManageMent Board
  • the system board 1 is a processing module including one or more CPUs 11 and a memory 12.
  • the IO unit 2 is a processing module that includes one or more IO devices 21.
  • the IO device 21 there is a storage storing an OS (Operating System) 21a.
  • OS Operating System
  • the crossbar switch 3 is a relay device that connects the system boats 1, the IO units 2, and the system boards 1 and the IO units 2.
  • the MMB 4 is a management dedicated unit that manages the entire information processing apparatus.
  • the MMB 4 provides a user interface to the connected terminal device, and enables operation management such as hardware status monitoring, configuration information display, error information display, partition management, network management, and power supply control.
  • the reason why the system boards 1 and the MMB 4 are connected by the LAN 5 is to perform a plurality of complicated operation management processes through one physical medium.
  • FIG. 1 shows three partitions 6-1 to 6-3 as the partition 6 managed by the MMB 4.
  • the partition 6-1 includes two system boards 1-1 and 1-2 and one IO unit 2-1.
  • the partition 6-2 includes one system board 1-2 and two IO units 2-2 and 2-3
  • the partition 6-3 includes one system board 1-4 and IO unit, respectively. 2-4.
  • FIG. 2 is a diagram for explaining a schematic configuration example of a system board and a CPU mounted on the system board.
  • a CPU 11 equipped with a TSC 110 is employed.
  • each partition 6 one CPU 11 does not necessarily execute all tasks for which the elapsed time needs to be confirmed. Therefore, a common clock generator (“CLOCK Source” in FIG. 2) is used so that the difference in value between the TSCs 110 of the CPU 11 mounted on each system board 1 (1-1 to 1-n) does not change with time.
  • the clock is supplied from 200 to the CPU 11 of each system board 1.
  • the clock generator 200 is provided in the housing, but the arrangement location is not limited as long as it is in the housing. For example, the clock generator 200 may be mounted on any system board 1 or any IO unit 2.
  • Each system board 1 is equipped with a south bridge 13.
  • An HPET 130 is mounted on the south bridge 13.
  • FIG. 3 is a diagram illustrating a configuration example of the system board in more detail.
  • each system board 1 includes, in addition to the CPU 11, the memory 12, and the south bridge 13, a BMC (Baseboard Management Controller) 14, an FWH (Firm-Ware Hub) 15, an MC (Memory Controller) 16, And I2C (Inter-Integrated Circuit) GPIO (General Purpose Input / Output) 17.
  • BMC Baseboard Management Controller
  • FWH Firm-Ware Hub
  • MC Memory Controller
  • I2C Inter-Integrated Circuit
  • GPIO General Purpose Input / Output
  • the configuration of each system board 1 other than the number of CPUs 11 is not limited to that shown in FIG.
  • the BMC 14 is a processing device that performs various management processes including monitoring of the state of the own system board 1 and power on / off.
  • the BMC 14 is connected to the MMB 4 via the LAN 5, and requests from the MMB 4 are processed by the BMC 14.
  • the south bridge 13 is connected by an LPC (Low Pin Count) bus.
  • the BMC 14 When the BMC 14 is connected to the LAN 5, the BMC 14 transmits the configuration information of the own system board 1 to the MMB 4. Thereby, the MMB 4 recognizes the configuration including the number of CPUs 11 of the newly added system board 1 and reflects it in the control.
  • the south bridge 13 is connected to the CPU 11 via a DMI (Desktop Management Interface) bus, for example, and performs interrupts such as SMI (System Management Interrupt) and SCI (System Control Interrupt) to the CPU 11 in accordance with instructions from the BMC 14.
  • the FWH 15 connected to the south bridge 13 stores a BIOS (Basic Input / Output System) 15 a executed by each CPU 11.
  • the SMI is processed by the BIOS 15a, and the SCI is processed by the OS 21a that is read from the IO device 21 and executed.
  • MC 16 accesses memory 12 according to an instruction from CPU 11. Communication via the crossbar switch 3 is performed via the CPU 11.
  • the I2C GPIO 17 is connected to the MMB 4 via the I2C bus.
  • the MMB 4 can reset the CPU 11 via the I2C GPIO 17.
  • the TSC 110 of the CPU 11 that has been reset starts counting up with 0 as an initial value.
  • the reset by the MMB 4 is performed on the CPUs 11 on all the system boards 1 to be turned on for each partition 6. Thereby, the TSCs 110 of the CPUs 11 on all the system boards 1 that are simultaneously turned on are synchronized.
  • the reset is performed by recognizing the BMC 14 newly connected to the LAN 5, for example.
  • the value of the TSC 110 of the CPU 11 that is newly reset is usually significantly different from the value of the TSC 110 of the CPU 11 that has already been reset.
  • the synchronization which makes the value of TSC110 of CPU11 newly reset correspond to the value of TSC110 of CPU11 already reset within an allowable range as follows is realized.
  • the system board 1-2 is newly added to the partition 6-1 and will be specifically described with reference to FIG.
  • FIG. 4 is a sequence diagram showing an operation flow of the MMB, the existing system board, and the newly added system board when a new system board is added to the partition.
  • the system board 1-1 is indicated as “Initial SB”
  • the system board 1-2 is indicated as “Hot-added SB”.
  • the system board 1-1 represents the BMC 14, the south bridge 13, the I2C GPIO 17, and the CPU 11.
  • the CPU 11 is divided into a TSC 110, a BIOS 15a, and an OS 21a.
  • the system board 1-2 represents the BMC 14, the I2C GPIO 17, and the CPU 11.
  • the south bridge 13 is represented on the system board 1-1 and is not represented on the system board 1-2. This is because only one south bridge 13 is valid in each partition 6. Since the south bridge 13 of the system board 1-1 is enabled, the south bridge 13 of the system board 1-2 is disabled.
  • the BMC 14 of each system board 1 is powered on when mounted on the information processing apparatus. From this, the BMC 14 of the system board 1-2 notifies the MMB 4 that the system board 1-2 on which the BMC 14 is mounted is newly added by connection with the LAN 5. Next, the MMB 4 is requested to start processing for adding the system board 1-2 mounted from the OS 21a to the partition 6-1 (system board HotAdd) (S1). This process can be instructed directly from the user interface on the MMB 4 as well as from the OS 21a. In response to the HotAdd start instruction of the system board, the MMB 4 instructs the BMC 14 to activate the CPU 11 of the system board 1-2 and reset the activated CPU 11 (S2, S3).
  • system board HotAdd system board HotAdd
  • the MMB 4 instructs the CPU 11 of the system board 1-1 to perform processing for incorporating the system board 1-2 into the partition 6-1 as hardware (S4 to S6).
  • This instruction is given by the SMI via the BMC 14 and the south bridge 13. More specifically, for example, the BMC 14 asserts an SMI signal output from the mounted GPIO (not shown) to the south bridge 14 in response to an instruction from the MMB 4, and the south bridge 14 asserts the SMI by asserting the signal. I do.
  • the instruction by the SMI is processed by the BIOS 15a executed by the CPU 11, more specifically, by the SMI handler (Handler) which is a program incorporated in the BIOS 15a.
  • the SMI handler here updates the setting related to the connection target of its own partition 6-1 and causes the CPU 11 of the system board 1-2 to perform the setting related to the connection target (S7). Thereafter, the SMI handler notifies the MMB 4 of the end of the instructed process (S8). This notification is not shown in detail in FIG. 4, but is performed via, for example, the south bridge 13 and the BMC 14.
  • the MMB 4 instructs the CPU 11 of the system board 1-1 to stop the operations of all the TSCs 110 existing in the partition 6-1 (S11 to S13). This instruction is performed by the SMI via the BMC 14 and the south bridge 13 as described above.
  • the instruction by the SMI is processed by the SMI handler of the BIOS 15a executed by the CPU 11.
  • the SMI handler stops the TSC 110 of its own CPU 11, and instructs the CPU 11 on the system board 1-2 to stop the TSC 110 (S14).
  • An instruction to the CPU 11 on the system board 1-2 is made through the crossbar switch 3.
  • the SMI handler reads the value of the TSC 110 of its own CPU 11 (S15), and instructs the CPU 11 of the system board 1-2 to write the read value to the TSC 110 (S16). Thereafter, the SMI handler restarts the operation of the TSC 110 of its own CPU 11, and instructs the CPU 11 of the system board 1-2 to resume the operation of the TSC 110 (S17). In this manner, after all the operations of the TSC 110 existing in the partition 6-1 are resumed, the MMB 4 is notified that the synchronization of the TSC 110 is completed (S18). By this notification, the operation related to the synchronization of the TSC 110 is completed.
  • the MMB 4 notified that the synchronization of the TSC 110 is completed performs a process of activating the added system board 1-2 as a resource of the partition 6-1. Therefore, the MMB 4 causes the CPU 11 existing in the partition 6-1 to perform SCI for setting the configuration of the partition 6-1 from the MMB 4 (S21 to S23).
  • This instruction is given by SCI via the BMC 14 and the south bridge 13.
  • the BMC 14 asserts an SCI signal output from the mounted GPIO to the south bridge 14 in response to an instruction from the MMB 4, and the south bridge 14 performs SCI by asserting the signal.
  • the SCI activates ACPI (Advanced Configuration and Power Interface) of OS 21a.
  • the activated ACPI performs processing for logically incorporating resources (CPU and memory) on the system board 1-2 into the partition 6-1. Thereby, thereafter, in the partition 6-1, an application operation using both CPUs and memory resources of the two system boards 1-1 and 1-2 becomes possible.
  • FIG. 5 is a diagram for explaining a TSC synchronization method according to the present embodiment. Next, the synchronization method realized by the operation shown in FIG. 4 will be specifically described with reference to FIG.
  • “CPU From initial Power On” and “Hot-added CPU” shown in FIG. 5 represent the CPU 11 of the system board 1-1 and the CPU 11 of the system board 1-2, respectively.
  • a change in the value of the TSC 110 mounted on the corresponding CPU 11 is represented by a broken line.
  • a straight line extending under the notation “TSC” represents a time axis and a value of 0, and a broken line drawn on the right side of the straight line represents a change in the value of the corresponding TSC 110.
  • TSC time axis and a value of 0
  • a broken line drawn on the right side of the straight line represents a change in the value of the corresponding TSC 110.
  • a line extending diagonally to the right starting from the straight line is bent to the left beyond the time A. This indicates that the TSC 110 counts up using 0 as an initial value, and after the value reaches the maximum value, the value becomes 0 again.
  • S14 to S17 represented as symbols have the same meaning as the symbols illustrated in FIG. That is, the same reference numerals are used for the same operations as in FIG.
  • the “BIOS SMI handler” is a program executed by the CPU 11 of the system board 1-1.
  • “OS operation” represents a transition of a program executed by the CPU 11 of the system board 1-1.
  • the TSC 110 of the CPU 11 of the newly installed system board 1-2 starts counting up from time A.
  • the value held by the TSC 110 of the CPU 11 of the system board 1-1 is significantly different from zero. Since the values are usually so different, it is necessary to match the value of one of the TSCs 110 with the other. However, the value of the TSC 110 of the CPU 11 of the system board 1-1 that is already operating cannot be changed in view of the existence of the task being executed. Therefore, the value to be changed is the TSC 110 of the CPU 11 of the system board 1-2.
  • the MMB 4 recognizes the installation of the system board 1-2 through communication with the BMC 14 and performs SMI. Thereby, in the CPU 11 of the system board 1-1, the control is transferred from the OS 21a to the BIOS 15a at time B.
  • the SMI handler is activated, and the activated SMI handler stops the TSC 110 of the CPU 11 and the TSC 110 of the CPU 11 of the system board 1-2 at time C (S14).
  • the value of TSC110 of each CPU11 after that is represented by a line parallel to the time axis.
  • the SMI handler After stopping the TSC 110 of each CPU 11, the SMI handler reads the value of the TSC 110 of its own CPU 11 (S15), and writes the read value to the TSC 110 of the CPU 11 of the system board 1-2 at time D (S16). As a result, the value of the TSC 110 of the CPU 11 of the system board 1-2 changes from the previous value to the value of the TSC 110 of the CPU 11 of the system board 1-1.
  • the SMI handler restarts the operation of the TSC 110 of each CPU 11 at time E after the writing is completed (S17).
  • the TSC 110 of each CPU 11 at the time of resumption is the same value, and the difference in timing at which the TSC 110 of each CPU 11 resumes operation can be within an allowable range. For this reason, the TSC 110 mounted on each CPU 11 can be appropriately synchronized.
  • control passes from the BIOS 15a to the OS 21a.
  • FIGS. 6A and 6B are flowcharts showing processing executed by each CPU when a system board is added.
  • the processing shown in FIGS. 6A and 6B is processing when the TSC 110 of each CPU 11 is synchronized, and is realized when each CPU 110 executes the BIOS 15a.
  • the synchronization of the TSC 110 of each CPU 11 performed in S14 to S17 is realized by a method different from the method shown in FIG.
  • the process performed in order to synchronize the value of TSC110 of each CPU11 including CPU11 on the system board 1 added is demonstrated in detail.
  • “Master” shown in FIG. 6A is the CPU 11 that plays a role of leading the synchronization of the TSC 110, and “Slave” means all other CPUs 11 including the CPU 11 on the newly installed system board 1. . 6A and 6B, the CPU 11 on the newly installed system board 1 is handled in the same manner as the CPUs 11 other than the master.
  • FIG. 6A for the sake of convenience, only one process with the same content executed by the master and slave CPUs 11 is shown. A process that represents only one and a process that is basically the same will not be described separately for a master and a slave.
  • the MMB 4 that has recognized the newly installed system board 1 performs SMI. Thereby, all the CPUs 11 activate the SMI handler of the BIOS 15a (SM1, SS1). With the activation of the SMI handler, all the CPUs 11 transition to SMM (System Management Mode).
  • SMI System Management Mode
  • the activation of the SMI handler (BIOS 15 a) in each CPU 11 is performed by first activating the CPU 11 of the system board 1 in which the south bridge 13 is valid, and causing the CPU 11 to activate all other CPUs 11. Can be made. Alternatively, the south bridges 13 of all the system boards 1 may be validated, and the SMI handlers may be activated in all the CPUs 11 according to instructions from the MMB 4.
  • each CPU 11 executing the SMI handler of the BIOS 15a.
  • the reason for causing each CPU 11 to execute the SMI handler is that the SMI handler can be basically activated in any situation. Depending on the status of each CPU 11, it is possible to prevent the timing at which each CPU 11 starts processing or the timing at which the processing ends from greatly shifting. This is an advantage in performing synchronization of the TSC 110 more appropriately.
  • Each CPU 11 that has activated the SMI handler waits for all other CPUs 11 to activate the SMI handler (SM2, SS2). Thereafter, each CPU 11 recognizes whether its own role is master or slave (SM3, SS3). As a result, the CPU 11 that has been recognized as a master subsequently performs processing as a master, and the CPU 11 that has been recognized as a slave thereafter performs processing as a slave.
  • the recognition itself may be performed from the identification information assigned to the CPU 11, for example. More specifically, the CPU 11 having the smallest CPU number among the CPU numbers assigned as identification information may be set as the master.
  • the master CPU 11 checks whether all slaves have participated, for example, by waiting for the elapse of a predetermined time (SM4). Next, the master CPU 11 confirms the SMI factor (SM5), and determines whether the confirmed factor is for the synchronization request of the TSC 110 (SM6). When SMI is performed for the synchronization request, the determination of SM6 is YES and the process proceeds to SM7. When the SMI is performed for a purpose other than the synchronization request, the determination in SM6 is NO, and then the master CPU 11 executes processing for the confirmed request. Since the explanation of the processing is not particularly important, it is omitted here.
  • a TSC stop process for stopping the TSC 110 of all the CPUs 11 is performed.
  • the master CPU 11 secures a work area in the memory 12, for example, and designates a process to be executed by each slave CPU 11, an instruction to start execution of the process, and the like through the work area. It has become.
  • the slave CPU 11 executes a designated process, the slave CPU 11 writes that fact in the work area.
  • the work area is described as “Msg_box”.
  • the work area is referred to as a “message box”.
  • the “start flag” shown in FIGS. 6A and 6B is information for instructing the start of execution of the process, and “set start flag” means that the start of the execution of the process has been instructed.
  • the “end flag” is information for notifying that the slave CPU 11 has executed the process, and “setting the end flag” means that the process has been executed.
  • One start flag is sufficient, but an end flag is required for each slave CPU 11. When there is only one start flag, each slave CPU 11 may execute the process on condition that the start flag is set and the end flag corresponding to the CPU 11 is reset.
  • the designation of the process to be executed represents, for example, storage of a command to be executed, storage of information used by the slave CPU 11 for processing, or address information indicating the storage location, or a command to be executed next in the SMI handler. This can be done by address information or the like. Other methods may be used.
  • the CPU 11 of each slave recognizes and executes the process to be executed by polling to confirm the contents of the message box.
  • SM7 the master CPU 11 stops the TSC 110 of its own CPU 11.
  • the master CPU 11 writes, for example, a command instructing to stop the TSC 110 in a message box (SM8), sets all start flags, and resets all end flags (SM9).
  • the master CPU 11 waits until it can be confirmed that the all end flag is set (SM10).
  • the slave CPU 11 after executing SS3 waits for the start flag corresponding to the CPU 11 in the message box to be set (SS4).
  • SS4 the determination of SS4 is Yes and the process proceeds to SS5.
  • the slave CPU 11 stops the TSC 110 of its own CPU 11 according to the command stored in the message box. Next, the slave CPU 11 sets an end flag corresponding to the CPU 11 in the message box (SS6). Thereafter, the slave CPU 11 proceeds to SS7 in order to wait until the next process is executed.
  • SM10 When the CPU 11 of all slaves sets the end flag, the determination of SM10 is YES and the process proceeds to SM11.
  • SM11 to SM14 a TSC write process is performed to write the value of the TSC 110 of the master CPU 11 to the TSC 110 of all the slave CPUs 11.
  • the master CPU 11 reads the value of the TSC 110 of its own CPU 11. Next, the master CPU 11 writes, for example, a command for instructing writing of a value to the TSC 110 and the read value in the message box (SM12), sets all start flags, and resets all end flags (SM13). . Thereafter, the master CPU 11 waits until it can be confirmed that the all end flag has been set (SM14).
  • the CPU 11 of the slave that has shifted to SS7 shifts to SS8 by setting a start flag corresponding to its own CPU 11 in the message box.
  • the slave CPU 11 writes the value stored in the message box into the TSC 110 of the CPU 11 in accordance with the command in the message box (SS8).
  • the slave CPU 11 sets an end flag corresponding to the CPU 11 in the message box (SS9).
  • the slave CPU 11 shifts to SS10 to wait until the next process is executed.
  • the master CPU 11 writes, for example, a command instructing to resume the operation of the TSC 110 in a message box.
  • the master CPU 11 sets all start flags and resets all end flags (SM16).
  • the master CPU 11 resumes the operation of the TSC 110 of the own CPU 11 (SM17). Thereafter, the master CPU 11 waits until it can be confirmed that the all end flag is set (SM18).
  • the CPU 11 of the slave that has moved to SS10 moves to SS11 by setting a start flag corresponding to its own CPU 11 in the message box.
  • the slave CPU 11 resumes the operation of the TSC 110 of the own CPU 11 in accordance with the command in the message box.
  • the slave CPU 11 sets an end flag corresponding to its own CPU 11 in the message box (SS12).
  • the slave CPU 11 proceeds to SS13 in order to wait until the next process is executed.
  • the master CPU 11 writes, for example, a command instructing the end of SMM in a message box.
  • the master CPU 11 sets all start flags in the message box and resets all end flags (SM20). Thereafter, the master CPU 11 waits until it can be confirmed that the all end flag has been set (SM21).
  • the CPU 11 of the slave that has shifted to SS13 shifts to SS14 according to the setting of the start flag in the message box.
  • the slave CPU 11 performs a process for returning to the state before the SMM transition according to the command in the message box.
  • the slave CPU 11 sets an end flag corresponding to its own CPU 11 in the message box (SS15). Thereafter, the slave CPU 11 ends the series of processing and returns to the state before the SMM transition.
  • SM21 When the CPU 11 of all slaves sets the end flag, the determination of SM21 is YES and the process proceeds to SM21.
  • the master CPU 11 performs processing for returning to the state before the SMM transition and notifies the MMB 4 that the synchronization of the TSC 110 is completed. Thereafter, the master CPU 11 ends a series of processing and returns to the state before the SMM transition.
  • the target of synchronization is TSC 110, but another counter may be used. Any information processing apparatus or a counter that exists in the partition 6 and is handled in the same manner as the TSC 110 can be set as a synchronization target.
  • the CPU (system board) HotAdd is used as an example of synchronization.
  • the partition is activated, the CPU 11 is reset to the CPU 11 due to a control delay between the MMB 4 and the GPIO 17 of each system board 1 shown in FIG.
  • TSC synchronization of all the CPUs 11 in the partition 6 can be guaranteed by performing TSC synchronization processing by SMI before starting the OS. Before starting the OS, the TSC is not used, so no problem occurs in the TSC synchronization processing.
  • the stop of the TSC 110 is performed by executing a command incorporated in the SMI handler, but if possible in hardware, it may be performed by cutting off the clock supply to the TSC 110.
  • the mechanism for stopping may be determined according to the counter to be synchronized.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Hardware Redundancy (AREA)

Abstract

A system in which the present invention has been applied has one or more arithmetic processing devices provided with counters for cyclically counting the total number of clocks supplied from outside after operation has started. The system is furthermore has: a stop control means for stopping all the counters provided to the first arithmetic processing device and to the respective arithmetic processing devices that are in operation from among the one or more arithmetic processing devices when a first arithmetic processing device, which is not included among the one or more arithmetic processing devices, has been added; a reading means for reading the value of a counter provided to one of the arithmetic processing units after the arithmetic processing unit has been stopped by the stop control means; a writing means for writing the value read by the reading means to the counter provided to the first arithmetic processing unit; and a resume control means for resuming operation of all the counters that were stopped by the stop control means, after the value has been written by the writing means.

Description

情報処理装置、演算処理装置、及びカウンタ同期方法Information processing apparatus, arithmetic processing apparatus, and counter synchronization method

 本発明は、外部から供給されるクロックをカウントするカウンタをタイマとして搭載した演算処理装置に関する。 The present invention relates to an arithmetic processing apparatus equipped with a counter for counting a clock supplied from the outside as a timer.

 近年、サーバ等のコンピュータ(情報処理装置)の多くに、パーティション機能が搭載されている。このパーティション機能とは、1台の情報処理装置のリソースを複数のパーティションと呼ばれる処理単位(グループ)に分割し、パーティション毎に独立して動作させる機能である。 In recent years, a partition function is installed in many computers (information processing devices) such as servers. The partition function is a function that divides resources of one information processing apparatus into processing units (groups) called a plurality of partitions and operates independently for each partition.

 各パーティションでは、任意のOS(Operating System)、或いはアプリケーション・プログラム(以降「アプリケーション」と略記)を実行させることができる。それにより、各パーティションは、それぞれ1台の独立した情報処理装置として動作させることができる。1台の情報処理装置を複数台の情報処理装置として使用できることから、パーティション機能は、情報処理装置の設置台数を抑制するうえで有用である。 In each partition, an arbitrary OS (Operating System) or application program (hereinafter abbreviated as “application”) can be executed. Thereby, each partition can be operated as one independent information processing apparatus. Since one information processing apparatus can be used as a plurality of information processing apparatuses, the partition function is useful for suppressing the number of information processing apparatuses installed.

 各パーティションに割り当てられるリソースには、プログラムを実行する演算処理装置が複数、含まれることが多い。これは、各パーティションには高い処理能力が求められるのが普通だからである。演算処理装置とは、例えばCPU(Central Processing Unit)である。以降、CPUは演算処理装置の総称として用いる。 The resources allocated to each partition often include a plurality of arithmetic processing devices that execute programs. This is because each partition usually requires high processing power. The arithmetic processing device is, for example, a CPU (Central Processing Unit). Hereinafter, CPU is used as a general term for arithmetic processing devices.

 複数のCPUが割り当てられたパーティションでは、各CPUはそれぞれプログラムの実行に用いられる。そのようなパーティションでは、1個のタスクを複数のCPUが分散して実行する場合もある。 In a partition to which a plurality of CPUs are assigned, each CPU is used for executing a program. In such a partition, one task may be executed in a distributed manner by a plurality of CPUs.

 タスクのなかには、そのタスクの実行時間、或いは待ち時間等の経過時間を確認するタスクが存在する。そのような経過時間を確認する必要のあるタスクを、1個のCPUが全て実行するとは限らない。このため、各パーティションでは、各CPUは経過時間の確認を必要に応じて行うようになっている。その経過時間の確認には、普通、TSC(Time Stamp Counter)が用いられる。 There are tasks that check the elapsed time such as the execution time or waiting time of tasks. A single CPU does not always execute such a task that requires confirmation of the elapsed time. For this reason, in each partition, each CPU checks the elapsed time as necessary. To confirm the elapsed time, TSC (Time Stamp Counter) is usually used.

 このTSCは、CPU内に搭載されたフリー・ランニング・カウンタであり、動作を開始した後に外部から供給されるクロックの総数をカウントする。そのカウントは、初期値である0からそのビット数(例えば64ビット)に応じた最大値までのカウントアップ(インクリメント)をサイクリックに行う。 This TSC is a free running counter installed in the CPU, and counts the total number of clocks supplied from the outside after starting the operation. The count is cyclically incremented (incremented) from an initial value of 0 to a maximum value corresponding to the number of bits (for example, 64 bits).

 TSCには、搭載されたCPUのリセットにより0がセットされる。また、各CPUのTSCには、TSC毎にカウントアップの時間間隔が異ならないように、同一のクロックソースからクロックが供給されるようになっているのが普通である。このようなことから、同じパーティションに属する各CPUへのリセットを許容範囲内で行うことにより、各CPUは経過時間の確認を適切に行うことができる。 • 0 is set in TSC by resetting the installed CPU. The TSC of each CPU is usually supplied with the clock from the same clock source so that the count-up time interval does not differ for each TSC. For this reason, each CPU can appropriately check the elapsed time by resetting each CPU belonging to the same partition within an allowable range.

 サウスブリッジ、及びICH(I/O Controller Hub)等の相互接続用のLSI(Large Scale Integration)チップには、HPET(High Performance Event Timer)と呼ばれる高精度タイマが搭載されている。各CPUによる経過時間の確認には、このHPETを用いることができる。 High-precision timers called HPET (High Performance Event Timer) are mounted on LSI (Large Scale Integration) chips for interconnection such as South Bridge and ICH (I / O Controller Hub). This HPET can be used for confirmation of the elapsed time by each CPU.

 近年、CPUとしてはマルチコアのCPUが広く用いられている。各コアはそれぞれ並行して、プログラムの最小の実行単位であるスレッドを実行する。スレッドを実行する各コアは、経過時間の確認を行う可能性がある。そのため、マルチコアのCPUを備えたパーティションでは、HPETへの多くのアクセスが集中しやすい。アクセスの集中は、処理性能を低下させる。このことから、現在では、経過時間の確認には、HPETよりもTSCのほうが用いられるようになっている。 In recent years, multi-core CPUs are widely used as CPUs. Each core executes a thread which is the smallest execution unit of a program in parallel. Each core executing a thread may check the elapsed time. Therefore, in a partition having a multi-core CPU, many accesses to HPET are likely to be concentrated. Access concentration reduces processing performance. For this reason, at present, TSC is used rather than HPET for checking the elapsed time.

 パーティション機能には、ダイナミックパーティション(DP)機能がある。このDP機能とは、OSが稼働中のパーティションへのCPUやメモリ等のハードウェアリソースの動的な追加(Hot-add)、或いは既存のハードウェアリソースの動的な交換(Hot-replace)を可能にする機能である。DP機能を搭載した情報処理装置では、追加、或いは交換の対象となるハードウェアリソースをモジュール化し、追加、及び交換を容易に行えるようにしているのが普通である。以降、便宜的に、モジュール化されたハードウェアリソースは「処理モジュール」と表記する。 The partition function includes a dynamic partition (DP) function. This DP function refers to the dynamic addition (Hot-add) of hardware resources such as CPU and memory to the partition where the OS is running, or the dynamic replacement (Hot-replace) of existing hardware resources. It is a function that makes it possible. In an information processing apparatus equipped with a DP function, a hardware resource to be added or exchanged is usually modularized so that addition or exchange can be easily performed. Hereinafter, for convenience, modularized hardware resources are referred to as “processing modules”.

 処理モジュールに存在するCPUは、追加、或いは交換のために新たに搭載された際にリセットされる。それにより、新たに搭載された処理モジュールに存在するCPU内のTSCは、初期値からのカウントアップを開始する。 The CPU present in the processing module is reset when it is newly installed for addition or replacement. As a result, the TSC in the CPU existing in the newly installed processing module starts counting up from the initial value.

 処理モジュールの搭載は、シャットダウンせずに行われる。新たに搭載された処理モジュールに存在するCPUへのリセットは、従来、その処理モジュールの認識により自動的に行われている。そのため、新たに搭載された処理モジュールに存在するCPU内のTSCの値は、同じパーティションに属する他の処理モジュールに存在するCPU内のTSCの値とは大きく異なるのが普通となっている。 The processing module is installed without shutting down. Conventionally, a reset to a CPU existing in a newly installed processing module is automatically performed by recognition of the processing module. For this reason, the TSC value in the CPU existing in the newly installed processing module is usually greatly different from the TSC value in the CPU existing in another processing module belonging to the same partition.

 上記のように、パーティションでは、経過時間を確認する必要のあるタスクを、1個のCPUが全て実行するとは限らない。このため、新たに搭載された処理モジュールに存在するCPU内のTSCの値と、同じパーティションに属する他の処理モジュールに存在するCPU内のTSCの値との差が許容範囲内でなかった場合、適切に処理できないタスクが発生する可能性は非常に高い。このことから、新たに搭載された処理モジュールに存在するCPU内のTSCの値は、同じパーティションに属する他の処理モジュールに存在するCPU内のTSCの値と許容範囲内で一致するように同期させる必要がある。 As described above, in a partition, one CPU does not necessarily execute all tasks that require confirmation of elapsed time. For this reason, when the difference between the TSC value in the CPU existing in the newly installed processing module and the TSC value in the CPU existing in another processing module belonging to the same partition is not within the allowable range, It is very likely that a task that cannot be handled properly will occur. Therefore, the TSC value in the CPU existing in the newly installed processing module is synchronized with the TSC value in the CPU existing in another processing module belonging to the same partition within the allowable range. There is a need.

 従来、TSCの値は、読み出すだけでなく、下位分のビットのみを対象に書き換えることができるようになっている。しかし、新たに搭載された処理モジュールに存在するCPU内のTSCの下位分のビットにのみ適切な値を書き込んでも、全体の値が適切となるとは限らない。例え他の処理モジュールに存在するCPU内のTSCから全ビット分の値を読み出し、読み出した値を新たに搭載された処理モジュールに存在するCPU内のTSCに書き込んだとしても、それらのTSCの値に比較的に大きい差が発生する可能性がある。これは、TSCは、搭載された演算処理装置が動作していなくともクロックの入力によってカウントアップを行うカウンタだからである。それにより、或るTSCの値を別のTSCに書き込むのに要する時間以上の差が発生する。その時間とは、或るTSCからの値の読み出し、読み出した値の転送、及び読み出した値の別のTSCへの書き込み、等にそれぞれ要する時間の合計である。 Conventionally, the TSC value can be rewritten not only for reading but for only the lower bits. However, even if an appropriate value is written only in the lower bits of the TSC in the CPU existing in the newly installed processing module, the overall value is not always appropriate. Even if all bit values are read from the TSC in the CPU existing in another processing module, and the read value is written to the TSC in the CPU existing in the newly installed processing module, the values of those TSC A relatively large difference may occur. This is because the TSC is a counter that counts up by input of a clock even when the installed processing unit is not operating. As a result, a difference more than the time required to write the value of one TSC to another TSC occurs. The time is the total time required for reading a value from a certain TSC, transferring the read value, writing the read value to another TSC, and the like.

 上記差は、実行するタスクによっては許容範囲外となる可能性がある。その差は、負荷の重いCPUの存在、或いはデータ転送の集中、等により大きくなる。このことから、新たに追加されるCPU内のTSCの値を、動作中のCPU内のTSCの値とより正確となるように同期させることは重要と思われる。 * The above difference may be outside the allowable range depending on the task to be executed. The difference increases due to the presence of a heavily loaded CPU or the concentration of data transfer. Therefore, it seems important to synchronize the TSC value in the newly added CPU so as to be more accurate with the TSC value in the operating CPU.

特表2008-518367号公報Special table 2008-518367 gazette 特開平6-332569号公報JP-A-6-332569 特開平10-228397号公報Japanese Patent Application Laid-Open No. 10-228397

 1側面では、本発明は、新たに追加されるCPU内のTSCの値を、動作中のCPU内のTSCの値と同期させるための技術を提供することを目的とする。 In one aspect, an object of the present invention is to provide a technique for synchronizing a newly added TSC value in a CPU with a TSC value in an operating CPU.

 本発明を適用した1システムは、動作を開始した後に外部から供給されるクロックの総数をサイクリックにカウントするカウンタを搭載した1つ以上の演算処理装置と、1つ以上の演算処理装置には含まれない演算処理装置である第1の演算処理装置が新たに追加された場合に、第1の演算処理装置、及び1つ以上の演算処理装置のなかで動作中の各演算処理装置にそれぞれ搭載されたカウンタを全て停止させる停止制御手段と、停止制御手段により停止させた後、各演算処理装置のうちの1つに搭載されたカウンタの値を読み出す読出手段と、読出手段が読み出した値を第1の演算処理装置に搭載されたカウンタに書き込む書込手段と、書込手段が値を書き込んだ後、停止制御手段により停止させた全てのカウンタの動作を再開させる再開制御手段と、を有する。 One system to which the present invention is applied includes one or more arithmetic processing devices equipped with a counter that cyclically counts the total number of clocks supplied from the outside after the operation starts, and one or more arithmetic processing devices. When a first arithmetic processing device that is not included is newly added, each of the first arithmetic processing device and each of the arithmetic processing devices in operation among the one or more arithmetic processing devices Stop control means for stopping all mounted counters, reading means for reading the value of the counter mounted in one of the arithmetic processing units after stopping by the stop control means, and values read by the reading means Writing means to the counter mounted on the first arithmetic processing unit, and after the writing means writes the value, the operation of all the counters stopped by the stop control means is restarted. Having a control means.

 本発明を適用した1システムでは、新たに追加されるCPU内のTSC等のカウンタの値を、動作中のCPU内のTSC等のカウンタの値と同期させることができる。 In one system to which the present invention is applied, the value of a counter such as a TSC in a CPU that is newly added can be synchronized with the value of a counter such as a TSC in an operating CPU.

本実施形態による情報処理装置の構成例を説明する図である。It is a figure explaining the structural example of the information processing apparatus by this embodiment. システムボードの概略構成例、及びそのシステムボードに搭載されるCPUを説明する図である。It is a figure explaining the schematic structural example of a system board, and CPU mounted in the system board. システムボードの構成例をより詳細に説明する図である。It is a figure explaining the structural example of a system board in detail. パーティションに新たにシステムボードが追加される場合に、MMB、既存のシステムボード、及び新たに追加されるシステムボードの動作の流れを表すシーケンス図である。FIG. 10 is a sequence diagram showing an operation flow of an MMB, an existing system board, and a newly added system board when a new system board is added to a partition. 本実施形態によるTSCの同期方法を説明する図である。It is a figure explaining the synchronization method of TSC by this embodiment. システムボードが追加される場合に、各CPUが実行する処理を表すフローチャートである。It is a flowchart showing the process which each CPU performs when a system board is added. システムボードが追加される場合に、各CPUが実行する処理を表すフローチャートである(続き)。It is a flowchart showing the process which each CPU performs when a system board is added (continuation).

 以下、本発明の実施形態について、図面を参照しながら詳細に説明する。
 図1は、本実施形態による情報処理装置の構成例を説明する図である。図1に表すように、本実施形態による情報処理装置は、複数のシステムボード(SB)1(1-1~1-4)、複数のIO(Input/Output)ユニット2(2-1~2-4)、クロスバスイッチ3、MMB(ManageMent Board)4及びLAN5を備えている。
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration example of the information processing apparatus according to the present embodiment. As shown in FIG. 1, the information processing apparatus according to the present embodiment includes a plurality of system boards (SB) 1 (1-1 to 1-4) and a plurality of IO (Input / Output) units 2 (2-1 to 2). -4), a crossbar switch 3, an MMB (ManageMent Board) 4, and a LAN 5.

 システムボード1は、1つ以上のCPU11、及びメモリ12を備えた処理モジュールである。IOユニット2は、1つ以上のIO装置21を備えた処理モジュールである。IO装置21のなかには、OS(Operating System)21aを格納したストレージが存在する。以降、IOユニット2に搭載されたIO装置21としてはOS21aを格納したストレージ21のみを想定する。 The system board 1 is a processing module including one or more CPUs 11 and a memory 12. The IO unit 2 is a processing module that includes one or more IO devices 21. In the IO device 21, there is a storage storing an OS (Operating System) 21a. Hereinafter, only the storage 21 storing the OS 21a is assumed as the IO device 21 mounted in the IO unit 2.

 クロスバスイッチ3は、各システムボート1間、各IOユニット2間、及び各システムボード1と各IOユニット2間を接続する中継装置である。MMB4は、情報処理装置全体を管理する管理専用ユニットである。このMMB4は、接続された端末装置にユーザインターフェイスを提供し、ハードウェアの状態監視、構成情報表示、エラー情報表示、パーティション管理、ネットワーク管理、及び電源制御などの運用管理を可能にさせる。各システムボード1とMMB4とをLAN5により接続させているのは、複数の複雑な運用管理処理を1つの物理メディアを通して行うようにするためである。図1には、MMB4によって管理されるパーティション6として、3つのパーティション6-1~6-3を表している。パーティション6-1は、2つのシステムボード1-1、1-2、及び1つのIOユニット2-1を備えている。同様に、パーティション6-2は、1つのシステムボード1-2、及び2つのIOユニット2-2、2-3を備え、パーティション6-3は、それぞれ1つのシステムボード1-4、及びIOユニット2-4を備えている。 The crossbar switch 3 is a relay device that connects the system boats 1, the IO units 2, and the system boards 1 and the IO units 2. The MMB 4 is a management dedicated unit that manages the entire information processing apparatus. The MMB 4 provides a user interface to the connected terminal device, and enables operation management such as hardware status monitoring, configuration information display, error information display, partition management, network management, and power supply control. The reason why the system boards 1 and the MMB 4 are connected by the LAN 5 is to perform a plurality of complicated operation management processes through one physical medium. FIG. 1 shows three partitions 6-1 to 6-3 as the partition 6 managed by the MMB 4. The partition 6-1 includes two system boards 1-1 and 1-2 and one IO unit 2-1. Similarly, the partition 6-2 includes one system board 1-2 and two IO units 2-2 and 2-3, and the partition 6-3 includes one system board 1-4 and IO unit, respectively. 2-4.

 図2は、システムボードの概略構成例、及びそのシステムボードに搭載されるCPUを説明する図である。CPU11には、TSC110を搭載したものが採用されている。 FIG. 2 is a diagram for explaining a schematic configuration example of a system board and a CPU mounted on the system board. A CPU 11 equipped with a TSC 110 is employed.

 各パーティション6では、経過時間を確認する必要のあるタスクを、1個のCPU11が全て実行するとは限らない。そのため、各システムボード1(1-1~1-n)に搭載されたCPU11のTSC110の間で値の差が時間経過に従って変化しないように、共通のクロック発生器(図2中「CLOCK Source」と表記)200からクロックが各システムボード1のCPU11に供給される。このクロック発生器200は、筐体内に設けられているが、筐体内であれば配置場所は限定されない。例えばクロック発生器200は、何れかのシステムボード1、或いは何れかのIOユニット2に搭載されていても良い。 In each partition 6, one CPU 11 does not necessarily execute all tasks for which the elapsed time needs to be confirmed. Therefore, a common clock generator (“CLOCK Source” in FIG. 2) is used so that the difference in value between the TSCs 110 of the CPU 11 mounted on each system board 1 (1-1 to 1-n) does not change with time. The clock is supplied from 200 to the CPU 11 of each system board 1. The clock generator 200 is provided in the housing, but the arrangement location is not limited as long as it is in the housing. For example, the clock generator 200 may be mounted on any system board 1 or any IO unit 2.

 各システムボード1には、サウスブリッジ13が搭載されている。このサウスブリッジ13には、HPET130が搭載されている。 Each system board 1 is equipped with a south bridge 13. An HPET 130 is mounted on the south bridge 13.

 図3は、システムボードの構成例をより詳細に説明する図である。
 各システムボード1は、図3に表すように、CPU11、メモリ12、及びサウスブリッジ13に加えて、BMC(Baseboard Management Controller)14、FWH(Firm-Ware Hub)15、MC(Memory Controller)16、及びI2C(Inter-Integrated Circuit) GPIO(General Purpose Input/Output)17を備えている。各システムボード1には、複数個のCPU11を搭載することができるが、ここでは説明上、便宜的に、各システムボード1に搭載されたCPU11は1個のみと想定する。CPU11の個数以外の各システムボード1の構成も図3に表すようなものに限定されない。
FIG. 3 is a diagram illustrating a configuration example of the system board in more detail.
As shown in FIG. 3, each system board 1 includes, in addition to the CPU 11, the memory 12, and the south bridge 13, a BMC (Baseboard Management Controller) 14, an FWH (Firm-Ware Hub) 15, an MC (Memory Controller) 16, And I2C (Inter-Integrated Circuit) GPIO (General Purpose Input / Output) 17. Although a plurality of CPUs 11 can be mounted on each system board 1, for convenience of explanation, it is assumed here that only one CPU 11 is mounted on each system board 1. The configuration of each system board 1 other than the number of CPUs 11 is not limited to that shown in FIG.

 BMC14は、自システムボード1の状態の監視、電源のオン/オフ等を含む各種管理上の処理を行う処理装置である。BMC14は、MMB4とLAN5を介して接続され、MMB4からの要求はBMC14によって処理される。サウスブリッジ13とは、LPC(Low Pin Count)バスにより接続されている。 The BMC 14 is a processing device that performs various management processes including monitoring of the state of the own system board 1 and power on / off. The BMC 14 is connected to the MMB 4 via the LAN 5, and requests from the MMB 4 are processed by the BMC 14. The south bridge 13 is connected by an LPC (Low Pin Count) bus.

 BMC14は、LAN5と接続された場合、自システムボード1の構成情報をMMB4に送信する。それにより、MMB4は、新たに追加されたシステムボード1のCPU11の数を含む構成を認識し、制御に反映させる。 When the BMC 14 is connected to the LAN 5, the BMC 14 transmits the configuration information of the own system board 1 to the MMB 4. Thereby, the MMB 4 recognizes the configuration including the number of CPUs 11 of the newly added system board 1 and reflects it in the control.

 サウスブリッジ13は、CPU11と例えばDMI(Desktop Management Interface)バスを介して接続され、BMC14からの指示に従い、SMI(System Management Interrupt)及びSCI(System Control Interrupt)等の割り込みをCPU11に対して行う。そのサウスブリッジ13に接続されたFWH15には、各CPU11が実行するBIOS(Basic Input/Output System)15aが格納されている。SMIは、BIOS15aによって処理され、SCIは、IO装置21から読み出されて実行されるOS21aによって処理される。 The south bridge 13 is connected to the CPU 11 via a DMI (Desktop Management Interface) bus, for example, and performs interrupts such as SMI (System Management Interrupt) and SCI (System Control Interrupt) to the CPU 11 in accordance with instructions from the BMC 14. The FWH 15 connected to the south bridge 13 stores a BIOS (Basic Input / Output System) 15 a executed by each CPU 11. The SMI is processed by the BIOS 15a, and the SCI is processed by the OS 21a that is read from the IO device 21 and executed.

 サウスブリッジ13は、各パーティション6で1個のみ有効とされる。図3では、システムボード1-1のサウスブリッジ13のみが有効であり、システムボード1-2のサウスブリッジ13は無効となっている。 Only one South Bridge 13 is valid for each partition 6. In FIG. 3, only the south bridge 13 of the system board 1-1 is valid, and the south bridge 13 of the system board 1-2 is invalid.

 MC16は、CPU11からの指示に従ってメモリ12にアクセスする。クロスバスイッチ3を介した通信は、CPU11を介して行われる。 MC 16 accesses memory 12 according to an instruction from CPU 11. Communication via the crossbar switch 3 is performed via the CPU 11.

 I2C GPIO17は、MMB4とI2Cバスを介して接続されている。MMB4は、I2C GPIO17を介して、CPU11をリセットすることができる。 The I2C GPIO 17 is connected to the MMB 4 via the I2C bus. The MMB 4 can reset the CPU 11 via the I2C GPIO 17.

 このリセットが行われたCPU11のTSC110は、0を初期値としたカウントアップを開始する。MMB4によるリセットは、パーティション6毎に、電源をオンさせる全てのシステムボード1上のCPU11に対して行われる。それにより、同時に電源をオンさせる全てのシステムボード1上のCPU11のTSC110は、同期したものとなっている。 The TSC 110 of the CPU 11 that has been reset starts counting up with 0 as an initial value. The reset by the MMB 4 is performed on the CPUs 11 on all the system boards 1 to be turned on for each partition 6. Thereby, the TSCs 110 of the CPUs 11 on all the system boards 1 that are simultaneously turned on are synchronized.

 しかし、ホットアッド、或いはホットリプレースによる新たなシステムボード1のパーティション6への追加では、リセットは、例えばLAN5に新たに接続されたBMC14を認識することで行われる。そのため、新たにリセットされるCPU11のTSC110の値は、既にリセットされたCPU11のTSC110の値とは大きく異なるのが普通である。このことから、本実施形態では、以下のように、新たにリセットされるCPU11のTSC110の値を、既にリセットされたCPU11のTSC110の値と許容範囲内で一致させる同期を実現させる。ここでは、システムボード1-2がパーティション6-1に新たに追加されることを想定し、図4を参照して具体的に説明する。 However, when a new system board 1 is added to the partition 6 by hot add or hot replacement, the reset is performed by recognizing the BMC 14 newly connected to the LAN 5, for example. For this reason, the value of the TSC 110 of the CPU 11 that is newly reset is usually significantly different from the value of the TSC 110 of the CPU 11 that has already been reset. From this, in this embodiment, the synchronization which makes the value of TSC110 of CPU11 newly reset correspond to the value of TSC110 of CPU11 already reset within an allowable range as follows is realized. Here, it is assumed that the system board 1-2 is newly added to the partition 6-1 and will be specifically described with reference to FIG.

 その図4は、パーティションに新たにシステムボードが追加される場合に、MMB、既存のシステムボード、及び新たに追加されるシステムボードの動作の流れを表すシーケンス図である。図4中、システムボード1-1は「Initial SB」、システムボード1-2は「Hot-added SB」と表記している。システムボード1-1では、BMC14、サウスブリッジ13、I2C GPIO17、及びCPU11を表している。CPU11は、TSC110、BIOS15a、及びOS21aに分けている。システムボード1-2では、BMC14、I2C GPIO17、及びCPU11を表している。 FIG. 4 is a sequence diagram showing an operation flow of the MMB, the existing system board, and the newly added system board when a new system board is added to the partition. In FIG. 4, the system board 1-1 is indicated as “Initial SB”, and the system board 1-2 is indicated as “Hot-added SB”. The system board 1-1 represents the BMC 14, the south bridge 13, the I2C GPIO 17, and the CPU 11. The CPU 11 is divided into a TSC 110, a BIOS 15a, and an OS 21a. The system board 1-2 represents the BMC 14, the I2C GPIO 17, and the CPU 11.

 図4では、サウスブリッジ13はシステムボード1-1に表し、システムボード1-2には表していない。これは、サウスブリッジ13は各パーティション6で1つのみ有効とされるからである。システムボード1-1のサウスブリッジ13が有効となっていることから、システムボード1-2のサウスブリッジ13は無効とされる。 In FIG. 4, the south bridge 13 is represented on the system board 1-1 and is not represented on the system board 1-2. This is because only one south bridge 13 is valid in each partition 6. Since the south bridge 13 of the system board 1-1 is enabled, the south bridge 13 of the system board 1-2 is disabled.

 各システムボード1のBMC14は、CPU11とは異なり、情報処理装置への搭載によって電源がオンとされる。このことから、システムボード1-2のBMC14は、LAN5との接続により、自身が搭載されたシステムボード1-2が新たに追加された旨をMMB4に通知する。次にOS21aから搭載されたシステムボード1-2をパーティション6-1に追加する処理(システムボードのHotAdd)の開始をMMB4に依頼する(S1)。なお、この処理はOS21aからだけではなく、MMB4上のユーザインターフェースからも直接指示することができる。そのシステムボードのHotAddの開始指示により、MMB4は、BMC14に指示して、システムボード1-2のCPU11を起動させ、起動したCPU11をリセットさせる(S2、S3)。 Unlike the CPU 11, the BMC 14 of each system board 1 is powered on when mounted on the information processing apparatus. From this, the BMC 14 of the system board 1-2 notifies the MMB 4 that the system board 1-2 on which the BMC 14 is mounted is newly added by connection with the LAN 5. Next, the MMB 4 is requested to start processing for adding the system board 1-2 mounted from the OS 21a to the partition 6-1 (system board HotAdd) (S1). This process can be instructed directly from the user interface on the MMB 4 as well as from the OS 21a. In response to the HotAdd start instruction of the system board, the MMB 4 instructs the BMC 14 to activate the CPU 11 of the system board 1-2 and reset the activated CPU 11 (S2, S3).

 次にMMB4は、システムボード1-1のCPU11に、パーティション6-1にシステムボード1-2をハードウェアとして組み入れるための処理の実施を指示する(S4~S6)。この指示は、BMC14、及びサウスブリッジ13を介したSMIによって行われる。より具体的には、例えばBMC14は、MMB4からの指示により、搭載された不図示のGPIOからサウスブリッジ14に出力されるSMI用の信号をアサートさせ、サウスブリッジ14は、その信号のアサートによってSMIを行う。 Next, the MMB 4 instructs the CPU 11 of the system board 1-1 to perform processing for incorporating the system board 1-2 into the partition 6-1 as hardware (S4 to S6). This instruction is given by the SMI via the BMC 14 and the south bridge 13. More specifically, for example, the BMC 14 asserts an SMI signal output from the mounted GPIO (not shown) to the south bridge 14 in response to an instruction from the MMB 4, and the south bridge 14 asserts the SMI by asserting the signal. I do.

 そのSMIによる指示は、CPU11が実行するBIOS15a、より具体的には、BIOS15aに組み込まれたプログラムであるSMIハンドラ(Handler)によって処理される。ここでのSMIハンドラは、自身のパーティション6-1の接続対象に係わる設定を更新すると共に、システムボード1-2のCPU11に接続対象に係わる設定を行わせる(S7)。その後、SMIハンドラは、指示された処理の終了をMMB4に通知する(S8)。この通知は、図4には詳細に表していないが、例えばサウスブリッジ13及びBMC14を介して行われる。 The instruction by the SMI is processed by the BIOS 15a executed by the CPU 11, more specifically, by the SMI handler (Handler) which is a program incorporated in the BIOS 15a. The SMI handler here updates the setting related to the connection target of its own partition 6-1 and causes the CPU 11 of the system board 1-2 to perform the setting related to the connection target (S7). Thereafter, the SMI handler notifies the MMB 4 of the end of the instructed process (S8). This notification is not shown in detail in FIG. 4, but is performed via, for example, the south bridge 13 and the BMC 14.

 上記S4~S8は、システムボード1-2の追加するために行われる。その後は、システムボード1-2のCPU11に搭載されたTSC110を同期させるための処理が行われる。 The above S4 to S8 are performed to add the system board 1-2. Thereafter, processing for synchronizing the TSC 110 mounted on the CPU 11 of the system board 1-2 is performed.

 MMB4は、システムボード1-1のCPU11に、パーティション6-1に存在する全てのTSC110の動作を停止させるための指示を行う(S11~S13)。この指示は、上記と同様に、BMC14、及びサウスブリッジ13を介したSMIによって行われる。 The MMB 4 instructs the CPU 11 of the system board 1-1 to stop the operations of all the TSCs 110 existing in the partition 6-1 (S11 to S13). This instruction is performed by the SMI via the BMC 14 and the south bridge 13 as described above.

 そのSMIによる指示は、CPU11が実行するBIOS15aのSMIハンドラによって処理される。SMIハンドラは、自CPU11のTSC110を停止させると共に、システムボード1-2上のCPU11に指示して、TSC110を停止させる(S14)。システムボード1-2上のCPU11への指示は、クロスバスイッチ3を介して行われる。 The instruction by the SMI is processed by the SMI handler of the BIOS 15a executed by the CPU 11. The SMI handler stops the TSC 110 of its own CPU 11, and instructs the CPU 11 on the system board 1-2 to stop the TSC 110 (S14). An instruction to the CPU 11 on the system board 1-2 is made through the crossbar switch 3.

 次にSMIハンドラは、自CPU11のTSC110の値を読み出し(S15)、読み出した値のTSC110への書き込みをシステムボード1-2のCPU11に指示する(S16)。その後、SMIハンドラは、自CPU11のTSC110の動作を再開させると共に、システムボード1-2のCPU11にTSC110の動作の再開を指示する(S17)。そのようにして、パーティション6-1に存在するTSC110の動作を全て再開させた後、MMB4にTSC110の同期が完了した旨を通知する(S18)。その通知により、TSC110の同期に係わる動作が完了する。 Next, the SMI handler reads the value of the TSC 110 of its own CPU 11 (S15), and instructs the CPU 11 of the system board 1-2 to write the read value to the TSC 110 (S16). Thereafter, the SMI handler restarts the operation of the TSC 110 of its own CPU 11, and instructs the CPU 11 of the system board 1-2 to resume the operation of the TSC 110 (S17). In this manner, after all the operations of the TSC 110 existing in the partition 6-1 are resumed, the MMB 4 is notified that the synchronization of the TSC 110 is completed (S18). By this notification, the operation related to the synchronization of the TSC 110 is completed.

 TSC110の同期が完了した旨を通知されたMMB4は、追加されたシステムボード1-2をパーティション6-1のリソースとして活性化させる処理を行う。そのためにMMB4は、MMB4からパーティション6-1に存在する各CPU11に、パーティション6-1の構成を設定させるためのSCIを行わせる(S21~S23)。この指示は、BMC14、及びサウスブリッジ13を介したSCIによって行われる。例えばBMC14は、MMB4からの指示により、搭載されたGPIOからサウスブリッジ14に出力されるSCI用の信号をアサートさせ、サウスブリッジ14は、その信号のアサートによってSCIを行う。 The MMB 4 notified that the synchronization of the TSC 110 is completed performs a process of activating the added system board 1-2 as a resource of the partition 6-1. Therefore, the MMB 4 causes the CPU 11 existing in the partition 6-1 to perform SCI for setting the configuration of the partition 6-1 from the MMB 4 (S21 to S23). This instruction is given by SCI via the BMC 14 and the south bridge 13. For example, the BMC 14 asserts an SCI signal output from the mounted GPIO to the south bridge 14 in response to an instruction from the MMB 4, and the south bridge 14 performs SCI by asserting the signal.

 そのSCIは、OS21aのACPI(Advanced Configuration and Power Interface)を起動させる。起動されたACPIは、システムボード1-2上のリソース(CPUやメモリ)をパーティション6-1に論理的に組み込むための処理を行う。それにより、以降、パーティション6-1では、2つのシステムボード1-1及び1-2両方のCPU、およびメモリリソースを使用したアプリケーション動作が可能となる。 The SCI activates ACPI (Advanced Configuration and Power Interface) of OS 21a. The activated ACPI performs processing for logically incorporating resources (CPU and memory) on the system board 1-2 into the partition 6-1. Thereby, thereafter, in the partition 6-1, an application operation using both CPUs and memory resources of the two system boards 1-1 and 1-2 becomes possible.

 図5は、本実施形態によるTSCの同期方法を説明する図である。次に図5を参照し、上記図4に表す動作によって実現される同期方法について具体的に説明する。 FIG. 5 is a diagram for explaining a TSC synchronization method according to the present embodiment. Next, the synchronization method realized by the operation shown in FIG. 4 will be specifically described with reference to FIG.

 図5に表記の「CPU From initial Power On」及び「Hot-added CPU」はそれぞれ、システムボード1-1のCPU11、及びシステムボード1-2のCPU11を表している。各表記の下には、対応するCPU11に搭載されたTSC110の値の変化を折れ線で表している。「TSC」の表記の下に延びる直線は、時間軸、及び0の値を表し、その直線の右側に描く折れ線が対応するTSC110の値の変化を表している。例えばシステムボード1-1のCPU11では、直線を起点に右斜めに延びる線が時刻Aを超えて左に折れ曲がっている。これは、TSC110では、0を初期値としてカウントアップが行われ、その値が最大値となった後、その値が再び0となることを表している。 5. “CPU From initial Power On” and “Hot-added CPU” shown in FIG. 5 represent the CPU 11 of the system board 1-1 and the CPU 11 of the system board 1-2, respectively. Below each notation, a change in the value of the TSC 110 mounted on the corresponding CPU 11 is represented by a broken line. A straight line extending under the notation “TSC” represents a time axis and a value of 0, and a broken line drawn on the right side of the straight line represents a change in the value of the corresponding TSC 110. For example, in the CPU 11 of the system board 1-1, a line extending diagonally to the right starting from the straight line is bent to the left beyond the time A. This indicates that the TSC 110 counts up using 0 as an initial value, and after the value reaches the maximum value, the value becomes 0 again.

 図5に符号として表記のS14~S17は、図4に表す符号と同じ意味である。つまり図4と同じ動作には同一の符号を用いている。また、「BIOS SMI handler」は、システムボード1-1のCPU11が実行するプログラムである。「OS operation」では、システムボード1-1のCPU11が実行するプログラムの変移を表している。 5, S14 to S17 represented as symbols have the same meaning as the symbols illustrated in FIG. That is, the same reference numerals are used for the same operations as in FIG. The “BIOS SMI handler” is a program executed by the CPU 11 of the system board 1-1. “OS operation” represents a transition of a program executed by the CPU 11 of the system board 1-1.

 図5では、新たに搭載されたシステムボード1-2のCPU11のTSC110は、時刻Aからカウントアップを開始する。その時刻Aでは、システムボード1-1のCPU11のTSC110が保持する値は0とは大きく異なっている。そのように値が大きく異なっているのが普通であることから、何れかのTSC110の値を他方に合わせる必要がある。しかし、既に動作しているシステムボード1-1のCPU11のTSC110の値は、実行中のタスクの存在を考えれば変更することはできない。そのため、値の変更対象は、システムボード1-2のCPU11のTSC110となる。 In FIG. 5, the TSC 110 of the CPU 11 of the newly installed system board 1-2 starts counting up from time A. At the time A, the value held by the TSC 110 of the CPU 11 of the system board 1-1 is significantly different from zero. Since the values are usually so different, it is necessary to match the value of one of the TSCs 110 with the other. However, the value of the TSC 110 of the CPU 11 of the system board 1-1 that is already operating cannot be changed in view of the existence of the task being executed. Therefore, the value to be changed is the TSC 110 of the CPU 11 of the system board 1-2.

 MMB4は、システムボード1-2の搭載をBMC14との通信により認識し、SMIを行わせる。それにより、システムボード1-1のCPU11では、時刻Bで制御がOS21aからBIOS15aに渡っている。 The MMB 4 recognizes the installation of the system board 1-2 through communication with the BMC 14 and performs SMI. Thereby, in the CPU 11 of the system board 1-1, the control is transferred from the OS 21a to the BIOS 15a at time B.

 制御が渡ったBIOS15aでは、SMIハンドラが起動され、起動されたSMIハンドラは、自CPU11のTSC110、及びシステムボード1-2のCPU11のTSC110を時刻Cで停止させる(S14)。それにより、以降の各CPU11のTSC110の値は、時間軸と平行な線によって表される。 In the BIOS 15a to which the control has passed, the SMI handler is activated, and the activated SMI handler stops the TSC 110 of the CPU 11 and the TSC 110 of the CPU 11 of the system board 1-2 at time C (S14). Thereby, the value of TSC110 of each CPU11 after that is represented by a line parallel to the time axis.

 SMIハンドラは、各CPU11のTSC110を停止させた後、自CPU11のTSC110の値を読み出し(S15)、読み出した値をシステムボード1-2のCPU11のTSC110に時刻Dで書き込ませる(S16)。それにより、システムボード1-2のCPU11のTSC110の値は、それまでの値からシステムボード1-1のCPU11のTSC110の値に変化する。 After stopping the TSC 110 of each CPU 11, the SMI handler reads the value of the TSC 110 of its own CPU 11 (S15), and writes the read value to the TSC 110 of the CPU 11 of the system board 1-2 at time D (S16). As a result, the value of the TSC 110 of the CPU 11 of the system board 1-2 changes from the previous value to the value of the TSC 110 of the CPU 11 of the system board 1-1.

 SMIハンドラは、書き込みが終了した後、時刻Eで各CPU11のTSC110の動作を再開させる(S17)。再開時における各CPU11のTSC110は同じ値であり、各CPU11のTSC110が動作を再開するタイミングの差は許容範囲内に収めることができる。このため、各CPU11が搭載したTSC110は適切に同期させることができる。再開後、時刻Fで制御はBIOS15aからOS21aに渡っている。 The SMI handler restarts the operation of the TSC 110 of each CPU 11 at time E after the writing is completed (S17). The TSC 110 of each CPU 11 at the time of resumption is the same value, and the difference in timing at which the TSC 110 of each CPU 11 resumes operation can be within an allowable range. For this reason, the TSC 110 mounted on each CPU 11 can be appropriately synchronized. After restarting, at time F, control passes from the BIOS 15a to the OS 21a.

 図6A及び図6Bは、システムボードが追加される場合に、各CPUが実行する処理を表すフローチャートである。図6A及び図6Bに表す処理は、各CPU11のTSC110を同期させる場合の処理であり、各CPU110が、BIOS15aを実行することで実現される。図6A及び図6Bに表す処理では、図4に表す方法とは異なる方法により、S14~S17によって行われる各CPU11のTSC110の同期を実現させる。次に図6A及び図6Bを参照して、追加されるシステムボード1上のCPU11を含む各CPU11のTSC110の値を同期させるために実行される処理について詳細に説明する。 6A and 6B are flowcharts showing processing executed by each CPU when a system board is added. The processing shown in FIGS. 6A and 6B is processing when the TSC 110 of each CPU 11 is synchronized, and is realized when each CPU 110 executes the BIOS 15a. In the processing shown in FIGS. 6A and 6B, the synchronization of the TSC 110 of each CPU 11 performed in S14 to S17 is realized by a method different from the method shown in FIG. Next, with reference to FIG. 6A and FIG. 6B, the process performed in order to synchronize the value of TSC110 of each CPU11 including CPU11 on the system board 1 added is demonstrated in detail.

 図6Aに表記の「Master」は、TSC110の同期を主導する役割を持つCPU11であり、「Slave」は、新たに搭載されたシステムボード1上のCPU11を含む他の全てのCPU11のことである。図6A及び図6Bに表す処理では、新たに搭載されたシステムボード1上のCPU11はマスター以外の他のCPU11と同様に扱うようになっている。 “Master” shown in FIG. 6A is the CPU 11 that plays a role of leading the synchronization of the TSC 110, and “Slave” means all other CPUs 11 including the CPU 11 on the newly installed system board 1. . 6A and 6B, the CPU 11 on the newly installed system board 1 is handled in the same manner as the CPUs 11 other than the master.

 図6Aでは、便宜的に、マスター、スレーブの各CPU11が実行する同じ内容の処理は、1つのみ表している。1つのみ表した処理、及び内容が基本的に同じ処理については、マスター、スレーブに分けての説明は行わない。 In FIG. 6A, for the sake of convenience, only one process with the same content executed by the master and slave CPUs 11 is shown. A process that represents only one and a process that is basically the same will not be described separately for a master and a slave.

 新たに搭載されたシステムボード1を認識したMMB4は、SMIを行わせる。それにより、全CPU11はBIOS15aのSMIハンドラを起動させる(SM1、SS1)。SMIハンドラの起動により、全CPU11はSMM(System Management Mode)に遷移する。 The MMB 4 that has recognized the newly installed system board 1 performs SMI. Thereby, all the CPUs 11 activate the SMI handler of the BIOS 15a (SM1, SS1). With the activation of the SMI handler, all the CPUs 11 transition to SMM (System Management Mode).

 各CPU11でのSMIハンドラ(BIOS15a)の起動は、始めにサウスブリッジ13が有効となっているシステムボード1のCPU11を起動させ、そのCPU11に他の全てのCPU11にSMIハンドラを起動させることで行わせることができる。或いは、全てのシステムボード1のサウスブリッジ13を有効とさせ、MMB4からの指示により全てのCPU11にSMIハンドラを起動させるようにしても良い。 The activation of the SMI handler (BIOS 15 a) in each CPU 11 is performed by first activating the CPU 11 of the system board 1 in which the south bridge 13 is valid, and causing the CPU 11 to activate all other CPUs 11. Can be made. Alternatively, the south bridges 13 of all the system boards 1 may be validated, and the SMI handlers may be activated in all the CPUs 11 according to instructions from the MMB 4.

 図6A及び図6Bに表す処理は、BIOS15aのSMIハンドラを各CPU11が実行することで実現される。各CPU11にSMIハンドラを実行させるのは、SMIハンドラは基本的にどのような状況でも起動させることができるからである。各CPU11の状況によって、各CPU11が処理を開始するタイミング、或いはその処理が終了するタイミングが大きくずれることも抑えることができる。これはTSC110の同期をより適切に行ううえでの利点である。 6A and 6B is realized by each CPU 11 executing the SMI handler of the BIOS 15a. The reason for causing each CPU 11 to execute the SMI handler is that the SMI handler can be basically activated in any situation. Depending on the status of each CPU 11, it is possible to prevent the timing at which each CPU 11 starts processing or the timing at which the processing ends from greatly shifting. This is an advantage in performing synchronization of the TSC 110 more appropriately.

 SMIハンドラを起動させた各CPU11は、他のCPU11が全てSMIハンドラを起動するのを待つ待ち合わせを行う(SM2、SS2)。その後、各CPU11は、自身の役割がマスター、スレーブのなかの何れであるかの認識を行う(SM3、SS3)。それにより、マスターと認識したCPU11は、以降、マスターとしての処理を行い、スレーブと認識したCPU11は、以降、スレーブとしての処理を行う。その認識自体は、例えばCPU11に割り当てられた識別情報から行わせれば良い。より具体的には、識別情報として割り当てられたCPU番号のなかで最小のCPU番号を持つCPU11をマスターとすれば良い。 Each CPU 11 that has activated the SMI handler waits for all other CPUs 11 to activate the SMI handler (SM2, SS2). Thereafter, each CPU 11 recognizes whether its own role is master or slave (SM3, SS3). As a result, the CPU 11 that has been recognized as a master subsequently performs processing as a master, and the CPU 11 that has been recognized as a slave thereafter performs processing as a slave. The recognition itself may be performed from the identification information assigned to the CPU 11, for example. More specifically, the CPU 11 having the smallest CPU number among the CPU numbers assigned as identification information may be set as the master.

 マスターのCPU11は、例えば予め定めた時間の経過を待つことにより、全スレーブが参加したか否かの確認を行う(SM4)。次にマスターのCPU11は、SMI要因を確認し(SM5)、確認した要因がTSC110の同期要求のためのものか否かを判定する(SM6)。SMIがその同期要求のために行われた場合、SM6の判定はYESとなってSM7に移行する。SMIがその同期要求以外のために行われた場合、SM6の判定はNOとなり、次にマスターのCPU11は確認した要求のための処理を実行する。その処理についての説明は特に重要でないことから、ここでは省略する。 The master CPU 11 checks whether all slaves have participated, for example, by waiting for the elapse of a predetermined time (SM4). Next, the master CPU 11 confirms the SMI factor (SM5), and determines whether the confirmed factor is for the synchronization request of the TSC 110 (SM6). When SMI is performed for the synchronization request, the determination of SM6 is YES and the process proceeds to SM7. When the SMI is performed for a purpose other than the synchronization request, the determination in SM6 is NO, and then the master CPU 11 executes processing for the confirmed request. Since the explanation of the processing is not particularly important, it is omitted here.

 SM7~SM10では、全CPU11のTSC110を停止させるためのTSC停止処理が行われる。 In SM7 to SM10, a TSC stop process for stopping the TSC 110 of all the CPUs 11 is performed.

 本実施形態では、マスターのCPU11は、例えばメモリ12に作業領域を確保し、その作業領域を介して、各スレーブのCPU11に実行させる処理の指定、その処理の実行開始の指示、等を行うようになっている。スレーブのCPU11は、指定された処理を実行した場合に、その旨をその作業領域に書き込むようになっている。 In this embodiment, the master CPU 11 secures a work area in the memory 12, for example, and designates a process to be executed by each slave CPU 11, an instruction to start execution of the process, and the like through the work area. It has become. When the slave CPU 11 executes a designated process, the slave CPU 11 writes that fact in the work area.

 図6A及び図6Bでは、その作業領域は「Msg_box」と表記している。以降、その作業領域は「メッセージボックス」と表記する。 In FIG. 6A and FIG. 6B, the work area is described as “Msg_box”. Hereinafter, the work area is referred to as a “message box”.

 図6A及び図6Bに表記の「開始フラグ」は、処理の実行開始を指示するための情報であり、「開始フラグのセット」は、処理の実行開始を指示したことを意味する。また、「終了フラグ」は、スレーブのCPU11が処理を実行した旨を通知するための情報であり、「終了フラグのセット」は、処理を実行したことを意味する。開始フラグは1つで良いが、終了フラグはスレーブのCPU11毎に必要である。開始フラグが1つであった場合、各スレーブのCPU11は、開始フラグがセットであり、且つ自CPU11に対応する終了フラグがリセットであることを条件に、処理を実行すれば良い。ここでは、便宜的に、開始フラグ、及び終了フラグは共にスレーブのCPU11の数分、存在すると想定する。 The “start flag” shown in FIGS. 6A and 6B is information for instructing the start of execution of the process, and “set start flag” means that the start of the execution of the process has been instructed. The “end flag” is information for notifying that the slave CPU 11 has executed the process, and “setting the end flag” means that the process has been executed. One start flag is sufficient, but an end flag is required for each slave CPU 11. When there is only one start flag, each slave CPU 11 may execute the process on condition that the start flag is set and the end flag corresponding to the CPU 11 is reset. Here, for convenience, it is assumed that there are as many start flags and end flags as there are slave CPUs 11.

 実行させる処理の指定は、例えば実行すべきコマンドの格納、スレーブのCPU11が処理に用いる情報自体、若しくはその保存場所を表すアドレス情報の格納、或いはSMIハンドラのなかで次に実行すべきコマンドを表すアドレス情報等により行うことができる。それら以外の方法を用いても良い。 The designation of the process to be executed represents, for example, storage of a command to be executed, storage of information used by the slave CPU 11 for processing, or address information indicating the storage location, or a command to be executed next in the SMI handler. This can be done by address information or the like. Other methods may be used.

 上記のようなメッセージボックスを用いる場合、各スレーブのCPU11は、そのメッセージボックスの内容を確認するポーリングにより、自身が実行すべき処理を認識し、実行することになる。 When using the message box as described above, the CPU 11 of each slave recognizes and executes the process to be executed by polling to confirm the contents of the message box.

 SM7では、マスターのCPU11は、自CPU11のTSC110を停止させる。次にマスターのCPU11は、メッセージボックスに、例えばTSC110の停止を指示するコマンドを書き込み(SM8)、更に全開始フラグをセットし、全終了フラグはリセットする(SM9)。その後、マスターのCPU11は、全終了フラグがセットされたのを確認できるまで待つ(SM10)。 In SM7, the master CPU 11 stops the TSC 110 of its own CPU 11. Next, the master CPU 11 writes, for example, a command instructing to stop the TSC 110 in a message box (SM8), sets all start flags, and resets all end flags (SM9). After that, the master CPU 11 waits until it can be confirmed that the all end flag is set (SM10).

 SS3を実行した後のスレーブのCPU11は、メッセージボックス内の自CPU11に対応する開始フラグがセットされるのを待つ(SS4)。その開始フラグのセットが確認されると、SS4の判定はYesとなってSS5に移行する。 The slave CPU 11 after executing SS3 waits for the start flag corresponding to the CPU 11 in the message box to be set (SS4). When the set of the start flag is confirmed, the determination of SS4 is Yes and the process proceeds to SS5.

 SS5では、スレーブのCPU11は、メッセージボックスに格納されたコマンドに従い、自CPU11のTSC110を停止させる。次にスレーブのCPU11は、メッセージボックス内の自CPU11に対応する終了フラグをセットする(SS6)。その後、スレーブのCPU11は、次の処理を実行すべき状況となるまで待つためにSS7に移行する。 In SS5, the slave CPU 11 stops the TSC 110 of its own CPU 11 according to the command stored in the message box. Next, the slave CPU 11 sets an end flag corresponding to the CPU 11 in the message box (SS6). Thereafter, the slave CPU 11 proceeds to SS7 in order to wait until the next process is executed.

 全スレーブのCPU11が終了フラグをセットした場合、上記SM10の判定がYESとなってSM11に移行する。SM11~SM14では、マスターのCPU11のTSC110の値を全スレーブのCPU11のTSC110に書き込ませるためのTSC書き込み処理が行われる。 When the CPU 11 of all slaves sets the end flag, the determination of SM10 is YES and the process proceeds to SM11. In SM11 to SM14, a TSC write process is performed to write the value of the TSC 110 of the master CPU 11 to the TSC 110 of all the slave CPUs 11.

 先ず、SM11では、マスターのCPU11は、自CPU11のTSC110の値を読み出す。次にマスターのCPU11は、メッセージボックスに、例えばTSC110への値の書き込みを指示するコマンド、及び読み出した値を書き込み(SM12)、更に全開始フラグをセットし、全終了フラグはリセットする(SM13)。その後、マスターのCPU11は、全終了フラグがセットされたのを確認できるまで待つ(SM14)。 First, in SM11, the master CPU 11 reads the value of the TSC 110 of its own CPU 11. Next, the master CPU 11 writes, for example, a command for instructing writing of a value to the TSC 110 and the read value in the message box (SM12), sets all start flags, and resets all end flags (SM13). . Thereafter, the master CPU 11 waits until it can be confirmed that the all end flag has been set (SM14).

 SS7に移行したスレーブのCPU11は、メッセージボックス内の自CPU11に対応する開始フラグのセットにより、SS8に移行する。そのSS8では、スレーブのCPU11は、メッセージボックス内のコマンドに従い、そのメッセージボックスに格納されていた値を自CPU11のTSC110に書き込む(SS8)。次にスレーブのCPU11は、メッセージボックス内の自CPU11に対応する終了フラグをセットする(SS9)。その後、スレーブのCPU11は、次の処理を実行すべき状況となるまで待つためにSS10に移行する。 The CPU 11 of the slave that has shifted to SS7 shifts to SS8 by setting a start flag corresponding to its own CPU 11 in the message box. In SS8, the slave CPU 11 writes the value stored in the message box into the TSC 110 of the CPU 11 in accordance with the command in the message box (SS8). Next, the slave CPU 11 sets an end flag corresponding to the CPU 11 in the message box (SS9). Thereafter, the slave CPU 11 shifts to SS10 to wait until the next process is executed.

 全スレーブのCPU11が終了フラグをセットした場合、上記SM14の判定がYESとなってSM15に移行する。SM15~SM18では、全CPU11のTSC110の動作を再開させるためのTSC再開処理が行われる。 When the CPU 11 of all slaves sets the end flag, the determination of SM14 is YES and the process proceeds to SM15. In SM15 to SM18, TSC restart processing for restarting the operation of the TSC 110 of all the CPUs 11 is performed.

 先ず、SM15では、マスターのCPU11は、メッセージボックスに、例えばTSC110の動作の再開を指示するコマンドを書き込む。次にマスターのCPU11は、全開始フラグをセットし、全終了フラグはリセットする(SM16)。更にマスターのCPU11は、自CPU11のTSC110の動作を再開させる(SM17)。その後、マスターのCPU11は、全終了フラグがセットされたのを確認できるまで待つ(SM18)。 First, in SM15, the master CPU 11 writes, for example, a command instructing to resume the operation of the TSC 110 in a message box. Next, the master CPU 11 sets all start flags and resets all end flags (SM16). Further, the master CPU 11 resumes the operation of the TSC 110 of the own CPU 11 (SM17). Thereafter, the master CPU 11 waits until it can be confirmed that the all end flag is set (SM18).

 SS10に移行したスレーブのCPU11は、メッセージボックス内の自CPU11に対応する開始フラグのセットにより、SS11に移行する。そのSS11では、スレーブのCPU11は、メッセージボックス内のコマンドに従い、自CPU11のTSC110の動作を再開させる。次にスレーブのCPU11は、メッセージボックス内の自CPU11に対応する終了フラグをセットする(SS12)。その後、スレーブのCPU11は、次の処理を実行すべき状況となるまで待つためにSS13に移行する。 The CPU 11 of the slave that has moved to SS10 moves to SS11 by setting a start flag corresponding to its own CPU 11 in the message box. In the SS 11, the slave CPU 11 resumes the operation of the TSC 110 of the own CPU 11 in accordance with the command in the message box. Next, the slave CPU 11 sets an end flag corresponding to its own CPU 11 in the message box (SS12). Thereafter, the slave CPU 11 proceeds to SS13 in order to wait until the next process is executed.

 全スレーブのCPU11が終了フラグをセットした場合、上記SM18の判定がYESとなってSM19に移行する。SM19~SM22では、全CPU11をSMMからSMM遷移前の状態に戻すためのSMIレジューム処理が行われる。 When the CPU 11 of all slaves sets the end flag, the determination of SM18 is YES and the process proceeds to SM19. In SM19 to SM22, SMI resume processing for returning all the CPUs 11 from the SMM to the state before the SMM transition is performed.

 先ず、SM19では、マスターのCPU11は、メッセージボックスに、例えばSMMの終了を指示するコマンドを書き込む。次にマスターのCPU11は、メッセージボックスの全開始フラグをセットし、全終了フラグはリセットする(SM20)。その後、マスターのCPU11は、全終了フラグがセットされたのを確認できるまで待つ(SM21)。 First, in SM 19, the master CPU 11 writes, for example, a command instructing the end of SMM in a message box. Next, the master CPU 11 sets all start flags in the message box and resets all end flags (SM20). Thereafter, the master CPU 11 waits until it can be confirmed that the all end flag has been set (SM21).

 SS13に移行したスレーブのCPU11は、メッセージボックス内の開始フラグのセットにより、SS14に移行する。そのSS14では、スレーブのCPU11は、メッセージボックス内のコマンドに従い、SMM遷移前の状態に復帰するための処理を行う。次にスレーブのCPU11は、メッセージボックス内の自CPU11に対応する終了フラグをセットする(SS15)。その後、スレーブのCPU11は、一連の処理を終了し、SMM遷移前の状態に復帰する。 The CPU 11 of the slave that has shifted to SS13 shifts to SS14 according to the setting of the start flag in the message box. In the SS 14, the slave CPU 11 performs a process for returning to the state before the SMM transition according to the command in the message box. Next, the slave CPU 11 sets an end flag corresponding to its own CPU 11 in the message box (SS15). Thereafter, the slave CPU 11 ends the series of processing and returns to the state before the SMM transition.

 全スレーブのCPU11が終了フラグをセットした場合、上記SM21の判定がYESとなってSM21に移行する。SM21では、マスターのCPU11は、SMM遷移前の状態に復帰するための処理を行うと共に、MMB4にTSC110の同期が完了した旨の通知を行う。その後、マスターのCPU11は、一連の処理を終了し、SMM遷移前の状態に復帰する。 When the CPU 11 of all slaves sets the end flag, the determination of SM21 is YES and the process proceeds to SM21. In SM 21, the master CPU 11 performs processing for returning to the state before the SMM transition and notifies the MMB 4 that the synchronization of the TSC 110 is completed. Thereafter, the master CPU 11 ends a series of processing and returns to the state before the SMM transition.

 なお、本実施形態では、同期の対象をTSC110としているが、別のカウンタを対象にしても良い。情報処理装置、或いはパーティション6に複数、存在し、且つTSC110と同様に扱われるようなカウンタであれば、同期の対象とすることができる。
 また、本実施形態では同期の事例をCPU(システムボード)のHotAddとしているが、パーティション起動時であっても図3に示すMMB4と各システムボード1のGPIO17間の制御の遅れによってCPU11へのリセットタイミングが大きくズレ、同期が崩れることが予想されるケースを想定して、OS起動前にSMIによるTSC同期処理を行うことで、パーティション6内の全CPU11のTSCの同期を保証できる。OS起動前は、TSCは使用しないためTSCの同期処理に問題は発生しない。
In this embodiment, the target of synchronization is TSC 110, but another counter may be used. Any information processing apparatus or a counter that exists in the partition 6 and is handled in the same manner as the TSC 110 can be set as a synchronization target.
In this embodiment, the CPU (system board) HotAdd is used as an example of synchronization. However, even when the partition is activated, the CPU 11 is reset to the CPU 11 due to a control delay between the MMB 4 and the GPIO 17 of each system board 1 shown in FIG. Assuming a case where the timing is greatly shifted and the synchronization is expected to be lost, TSC synchronization of all the CPUs 11 in the partition 6 can be guaranteed by performing TSC synchronization processing by SMI before starting the OS. Before starting the OS, the TSC is not used, so no problem occurs in the TSC synchronization processing.

 TSC110の停止は、SMIハンドラに組み込んだコマンドの実行により行うようになっているが、ハードウェア的に可能であれば、TSC110へのクロック供給を遮断させることで行うようにしても良い。停止させる仕組みは、同期の対象とするカウンタに応じて決定すれば良い。 The stop of the TSC 110 is performed by executing a command incorporated in the SMI handler, but if possible in hardware, it may be performed by cutting off the clock supply to the TSC 110. The mechanism for stopping may be determined according to the counter to be synchronized.

Claims (5)

 動作を開始した後に外部から供給されるクロックの総数をサイクリックにカウントするカウンタを搭載した1つ以上の演算処理装置と、
 前記1つ以上の演算処理装置には含まれない演算処理装置である第1の演算処理装置が新たに追加された場合に、前記第1の演算処理装置、及び前記1つ以上の演算処理装置のなかで動作中の各演算処理装置にそれぞれ搭載された前記カウンタを全て停止させる停止制御手段と、
 前記停止制御手段により停止させた後、前記各演算処理装置のうちの1つに搭載されたカウンタの値を読み出す読出手段と、
 前記読出手段が読み出した値を前記第1の演算処理装置に搭載されたカウンタに書き込む書込手段と、
 前記書込手段が前記値を書き込んだ後、前記停止制御手段により停止させた全てのカウンタの動作を再開させる再開制御手段と、
 を有することを特徴とする情報処理装置。
One or more arithmetic processing units equipped with a counter that cyclically counts the total number of clocks supplied from the outside after starting operation;
When a first arithmetic processing device which is an arithmetic processing device not included in the one or more arithmetic processing devices is newly added, the first arithmetic processing device and the one or more arithmetic processing devices Stop control means for stopping all the counters mounted in each of the arithmetic processing devices in operation,
A reading means for reading a value of a counter mounted on one of the arithmetic processing units after being stopped by the stop control means;
Writing means for writing the value read by the reading means into a counter mounted in the first arithmetic processing unit;
After the writing means writes the value, restart control means for restarting the operation of all the counters stopped by the stop control means;
An information processing apparatus comprising:
 前記第1の演算処理装置を認識する認識手段を更に有し、
 前記停止制御手段は、前記第1の演算処理装置を前記認識手段が認識した場合に、前記カウンタを全て停止させる、
 ことを特徴とする請求項1記載の情報処理装置。
Recognizing means for recognizing the first arithmetic processing unit;
The stop control means stops all the counters when the recognition means recognizes the first arithmetic processing unit;
The information processing apparatus according to claim 1.
 前記1つ以上の演算処理装置が複数の演算処理装置であり、前記複数の演算処理装置が複数のグループに分けられている場合に、前記認識手段は、グループ毎に、前記第1の演算処理装置を認識する、
 ことを特徴とする請求項2記載の情報処理装置。
When the one or more arithmetic processing devices are a plurality of arithmetic processing devices, and the plurality of arithmetic processing devices are divided into a plurality of groups, the recognition means performs the first arithmetic processing for each group. Recognize the device,
The information processing apparatus according to claim 2.
 動作を開始した後に外部から供給されるクロックの総数をサイクリックにカウントするカウンタと、
 前記前記カウンタを停止させる停止手段と、
 外部から指定された値を前記カウンタに書き込む書込手段と、
 前記停止手段により停止させたカウンタの動作を再開させる再開手段と、
 を有することを特徴とする演算処理装置。
A counter that cyclically counts the total number of externally supplied clocks after starting operation;
Stop means for stopping the counter;
Writing means for writing an externally designated value into the counter;
Restarting means for restarting the operation of the counter stopped by the stopping means;
An arithmetic processing apparatus comprising:
 動作を開始した後に外部から供給されるクロックの総数をサイクリックにカウントするカウンタを搭載した1つ以上の演算処理装置を備えた情報処理装置に新たに第1の演算処理装置を追加する場合に、
 前記第1の演算処理装置、及び前記1つ以上の演算処理装置のなかで動作中の各演算処理装置に搭載された前記カウンタを停止させ、
 前記各演算処理装置のうちの1つに搭載されたカウンタの値を読み出させ、
 該読み出された値を、前記第1の演算処理装置に搭載のカウンタに書き込ませ、
 前記第1の演算処理装置、及び前記各演算処理装置にそれぞれ搭載されたカウンタの動作を開始させる、
 ことを特徴とするカウンタ同期方法。
When a first arithmetic processing device is newly added to an information processing device including one or more arithmetic processing devices equipped with a counter that cyclically counts the total number of clocks supplied from the outside after the operation is started ,
Stopping the counter mounted on each of the first arithmetic processing units and the one or more arithmetic processing units that are operating,
Read the value of the counter mounted on one of the arithmetic processing units,
The read value is written in a counter mounted on the first arithmetic processing unit,
Starting the operation of the first arithmetic processing unit and a counter mounted in each arithmetic processing unit,
And a counter synchronization method.
PCT/JP2012/078543 2012-11-02 2012-11-02 Information processing device, arithmetic processing device, and counter synchronization method Ceased WO2014068774A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2012/078543 WO2014068774A1 (en) 2012-11-02 2012-11-02 Information processing device, arithmetic processing device, and counter synchronization method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2012/078543 WO2014068774A1 (en) 2012-11-02 2012-11-02 Information processing device, arithmetic processing device, and counter synchronization method

Publications (1)

Publication Number Publication Date
WO2014068774A1 true WO2014068774A1 (en) 2014-05-08

Family

ID=50626737

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/078543 Ceased WO2014068774A1 (en) 2012-11-02 2012-11-02 Information processing device, arithmetic processing device, and counter synchronization method

Country Status (1)

Country Link
WO (1) WO2014068774A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021174287A (en) * 2020-04-27 2021-11-01 富士通株式会社 Information processing device and cooperation method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS5552169A (en) * 1978-10-11 1980-04-16 Nec Corp Multiplex processor system
JPH06332569A (en) * 1993-05-26 1994-12-02 Nippon Telegr & Teleph Corp <Ntt> Method and device for matching real time timer
JP2002049605A (en) * 2000-08-02 2002-02-15 Fujitsu Ltd Timer adjustment system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS5552169A (en) * 1978-10-11 1980-04-16 Nec Corp Multiplex processor system
JPH06332569A (en) * 1993-05-26 1994-12-02 Nippon Telegr & Teleph Corp <Ntt> Method and device for matching real time timer
JP2002049605A (en) * 2000-08-02 2002-02-15 Fujitsu Ltd Timer adjustment system

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021174287A (en) * 2020-04-27 2021-11-01 富士通株式会社 Information processing device and cooperation method
JP7380403B2 (en) 2020-04-27 2023-11-15 富士通株式会社 Information processing device and cooperation method

Similar Documents

Publication Publication Date Title
CN115686872B (en) BMC-based memory resource processing equipment, method, device and medium
US9389976B2 (en) Distributed persistent memory using asynchronous streaming of log records
KR101754496B1 (en) Leverage offload programming model for local checkpoints
JP4934642B2 (en) Computer system
CN110083494B (en) Method and apparatus for managing hardware errors in a multi-core environment
US9910664B2 (en) System and method of online firmware update for baseboard management controller (BMC) devices
US20170168756A1 (en) Storage transactions
US8190805B2 (en) Information processing apparatus and method for reconfiguring the information processing apparatus
TW201339969A (en) Management method and system for start servers in data center
KR20150067332A (en) Inter-core communication apparatus and method
CN102289402A (en) Monitoring and managing method based on physical multi-partition computer architecture
US9811404B2 (en) Information processing system and method
US7441150B2 (en) Fault tolerant computer system and interrupt control method for the same
US20190324751A1 (en) Technologies for ensuring functional safety of an electronic device
US11366679B2 (en) Guest operating system wake-up method, device, electronic apparatus, and readable medium
CN105677373A (en) Node hot plug method and NUMA node
JP2001022599A (en) Fault tolerant system, fault tolerant processing method, and fault tolerant control program recording medium
WO2014068774A1 (en) Information processing device, arithmetic processing device, and counter synchronization method
CN103488505A (en) Patching method, device and system
JP2012108853A (en) System, device and method for starting digital signal processor
JP2011113163A (en) Inter-end point communication control device and method in io access communication system
TWI881478B (en) System power synchronization control method
CN112199230A (en) Storage controller supporting multi-core system exception handling
JP5970846B2 (en) Computer system and computer system control method
JP6449671B2 (en) Core I/O failover control system and core I/O failover control method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12887837

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12887837

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP