CN111736996A - A process persistence method and device for distributed non-volatile memory system - Google Patents

Abstract

The invention discloses a process persistence method and device for a distributed non-volatile memory system. In this method, a process copy of the persistent process running on the master node is periodically generated, and the copy is distributed to each slave node through the network; the master node and each slave node store the process copy in the PM module, so that The secondary node can restore and rebuild the persistent process based on the process copy. The invention mainly aims at the problem that the scientific computing process runs for a long time without reliability guarantee in a large-scale distributed system, and utilizes the characteristics of PM non-volatile, fast speed and large capacity to automatically maintain the process checkpoint and process copy, and can automatically maintain the process checkpoint and process copy. Automatic recovery when the system fails, has good market prospects and application value.

Description

A process persistence method and device for distributed non-volatile memory system

technical field

The invention relates to a computer system structure technology, in particular to a process persistence method and device for a distributed non-volatile memory system.

Background technique

In various scientific computing fields such as high-energy physics research, nuclear weapon design, aerospace vehicle design, prediction and decision-making of the national economy, energy exploration, medium and long-term weather forecasting, satellite image processing, intelligence analysis, industrial simulation, etc. Storage requirements, distributed high-performance computing clusters, as the main solution for scientific computing, provide relatively strong computing power support and storage support. However, in the field of scientific computing, a single computing task usually takes a long period to complete, and many computing tasks require no downtime or crash in the middle. Computational efficiency in scientific computing. To this end, the present invention provides a solution for continuing the execution of a computing task from a breakpoint, specifically a method and device for persisting a process in a computing task in a distributed persistent memory storage system. Distributed high-performance computing cluster is the third largest paradigm of human scientific research, and it is the concentrated expression of national scientific and technological strength. Therefore, the method and device provided by the present invention have extremely strong practical significance and application value.

In this technical field, the following technical terms are involved:

DRAM: Dynamic Random Access Memory, dynamic random access memory, that is, the memory that is widely used at this stage, which is characterized by information loss after power failure, and theoretically unlimited read and write life.

DDR4 SDRAM: Double-Data-Rate Fourth Generation Synchronous Dynamic Random Access Memory, the fourth generation of double data rate synchronous dynamic random access memory, is the latest generation of computer memory specifications, providing lower voltage, power consumption and higher performance than previous generations. high bandwidth.

PM: PersistentMemory, non-volatile memory or persistent memory, STTRAM, PCM, RRAM all belong to PM, which is characterized by the fact that the information is not lost after the power is turned off, and it can be byte addressable; but the number of reads and writes is limited, and the read and write are not Symmetrical, depending on the material, the traditional PM write time is about 2 to 4 times as long as its read time, and the PM read time is 1 to 3 times slower than DRAM. Equivalent to existing DDR4 DRAM.

PCB: Process Control Block, the process control block, the data structure used in the operating system to manage the process. Each process has a corresponding PCB, which stores information such as the running status, scheduling priority, memory allocation, process page table, and open files of the process.

PSU: PowerSupplyUnit, power supply unit, converts high-voltage alternating current into low-voltage direct current to supply power to various parts of the computer. The power supply unit of most modern desktop computers follows the ATX standard formulated by Intel Corporation, which defines parameters such as the size of the power supply unit, the shape of the interface, and the meaning of pins.

PWR_OK: Power-Good, a signal output by the power supply unit to a pin of the power supply interface of the motherboard in the ATX standard. When the signal is high, it indicates that the output current of the power supply unit has been stabilized and can supply power to the device normally. The ATX standard stipulates that when the external power supply is disconnected, the output current of the power supply unit should remain stable for at least 1 millisecond after PWR_OK falls back to a low level.

IPMI: Intelligent Platform Management Interface, intelligent platform management interface, a subsystem in the computer that is independent of the native operating system, BIOS, processor, etc., used to monitor and manage the running status of each computer device, and can exchange information with the outside world through an out-of-band management network.

BMC: Baseboard Management Controller, a microcontroller embedded in the server motherboard, the temperature sensor value, fan speed and other information are reported to the outside world through the out-of-band management network when abnormal parameters occur. It is the core component of IPMI.

ipmitool: IntelligentPlatform Management Interface Tool, an intelligent platform management interface tool, a software tool for controlling the IPMI system, which can realize functions such as checking the running status of the machine and turning it on and off remotely.

DETECT_PWR: In the process exception mining module, the power failure detection unit sends a signal to the exception processing unit after detecting that the power supply unit is powered off.

DETECT_CRASH: In the process exception mining module, the program crash detection unit sends a signal to the exception handling unit after detecting a program crash.

DETECT_IMBALANCE: In the process field mining module, the load balancing detection unit sends a signal to the exception processing unit when it detects that the load of each node is unbalanced.

undo method: a data writing method using undo log. When updating data in a non-volatile storage medium, first save the original data in the undo log, and then modify the data itself. If the writing process is interrupted abnormally, the original data value is restored using the undo log to prevent the system from entering an inconsistent state.

Distributed non-volatile memory system: Physically, it consists of several machine nodes configured with PM and DRAM memory. Each node is relatively independent and has its own processor, memory, operating system and other resources. The nodes are connected through TCP/IB, and then a complete distributed non-volatile memory system is formed through distributed system software. The TCP and IB (InfiniBand) are all network communication protocols. TCP is a commonly used protocol with reliable transmission characteristics. IB is a network protocol with extremely high bandwidth and extremely low latency, which is very suitable for nodes in distributed systems. Communication with nodes and between nodes and backing stores. Usually, in a distributed system, the redundant storage method of master and copy is used to ensure the availability and reliability of data. Figure 1 shows the overall architecture diagram of a distributed non-volatile memory system. Node1 is the master node, Node2 and Node3 are secondary nodes, and data1_x and data1_xx are copies of data1.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a process persistence method and device for a distributed non-volatile memory system based on the shortcomings of the above-mentioned prior art. The method and device enable scientific computing tasks to achieve from The breakpoint continues execution of the target.

The realization of the object of the present invention is accomplished by the following technical solutions:

A process persistence method for a distributed non-volatile memory system is applied to a system composed of multiple computing nodes; each of the computer nodes is connected through network communication; for any persistent process, its execution process includes the following step:

Select a computing node as the primary node, and additionally select at least one computing node as a secondary node; and initialize the persistence process in the primary node, and create a process copy in the PM module of the primary node simultaneously;

During the running of the persistent process, the process copy is updated at predetermined time intervals and distributed to each of the secondary nodes, and the secondary node stores the process copy of the persistent process in its PM module ;

When the process abnormality mining module of the master node detects that the persistence process is faulty, it selects a target migration node from the secondary node, and selects a target migration node according to the process copy stored in the target migration node in the target migration node. The persistent process is resumed in the migration node.

A further improvement of the present invention is that, in the process of initializing the persistence process, the process persistence unit of the master node creates a process structure in the PM module as the process copy; PCB, execution process first address, node ID, timestamp, process virtual address space copy; the process virtual address space copy adopts a balanced binary tree data structure, and each virtual page in the virtual address space is a balanced binary tree node.

A further improvement of the present invention is that in the process of initializing the persistent process, the process copy is initialized; the initialization process includes the following steps:

(S11) setting the verification mark to be unavailable;

(S12) determine the timestamp according to the current time; determine the node ID according to the ID of the current master node;

(S13) the PCB of the persistent process, the first address of the execution process are written into the process copy;

(S14) traverse the virtual address space of the persistent process, store each virtual page in use in the process virtual address space copy, and reset the SOFT_DIRTY flag bit of the virtual page;

(S15) Adjust the persistent process to a runnable state, and set the verification flag to be available.

A further improvement of the present invention is that in the process of updating the process copy, the process virtual address space copy is updated by means of incremental update.

A further improvement of the present invention is that the process of updating the process copy includes the following steps:

(S21) Suspend the persistent process, and set the verification identifier of the process replica to be unavailable;

(S22) updating the timestamp of the process replica according to the current time;

(S23) the PCB of the persistent process, the first address of the execution process are written into the process copy;

(S24) Traverse the virtual address space of the persistent process, update or insert the changed virtual page in the virtual address space copy of the process, and reset the change flag of each virtual page in the virtual address space ;

(S25) Adjust the persistent process to a runnable state, and set the verification flag to be available.

A further improvement of the present invention is that when the process abnormality mining module of the master node detects that the persistent process is faulty, the process abnormality mining module forwards the detected failure information to the failover module of the master node.

A further improvement of the present invention is that the response process of the failover module to the failure information includes the following steps:

(S31) The failover module on the primary node collects the running status of each secondary node, and selects the target migration node according to the running status; the running status includes CPU usage and available memory;

(S32) sending a migration instruction to each of the target migration nodes, requesting that the above-mentioned persistent process to be migrated be restored on the target migration node;

(S33) After the target migration node receives the migration instruction and the persistent process that needs to be recovered, its process recovery unit searches for the process copy of the persistent process in its own PM module, and checks its verification identifier. If the identifier is unavailable, the process recovers Failed; if there are multiple process copies of the same process, select the process copy with the latest timestamp;

(S34) The process recovery unit running on the target migration node rebuilds the persistent process on the node according to the process copy, and uses the node as a new master node.

A further improvement of the present invention is that the process of selecting the target migration node includes the following steps:

(S311) If the fault type is node power failure or program crash, perform the following process for each persistent process on the faulty node: select the available memory and the available memory among all the running secondary nodes in the current distributed system. The node with the lowest CPU usage among the nodes accommodating the migration process is used as the target migration node; if the available memory of all nodes is not enough to accommodate the migration process, the node with the largest available memory is selected as the target migration node, and jumps to step (S32) ;

(S312) If the fault type is unbalanced load, perform the following process on a randomly selected persistent process on the faulty node: from all running secondary nodes in the current distributed system, select available memory and can accommodate migration The node with the lowest CPU usage among the process nodes is used as the target migration node; if the available memory of all nodes is not enough to accommodate the migration process, the node with the largest available memory is selected as the target migration node, and jumps to step (S32).

A further improvement of the present invention is that the process of rebuilding the persistent process includes the following steps:

(S341) establishing a new process at the target migration node, and initializing the process PCB according to the process replica structure;

(S342) traverse the process virtual address space copy of the process copy, restore the process address space according to the page address and the flag position of the balanced binary tree node in the process virtual address space copy, and restore the memory data of the process according to the page content of the balanced binary tree node;

(S343) open the corresponding file according to the open file table in the process copy;

(S344) Mark the new process as runnable, insert it into the scheduling queue, and start running.

The present invention also includes a process persistence device for a distributed non-volatile memory system, which is arranged in the computing nodes of the distributed non-volatile memory system, and each of the computing nodes is connected through network communication; the process persistence device Including process persistence module, process exception mining module and failover module; among them:

The process persistence module is set in the master node where the persistence process is located, and is configured to periodically record the state and execution progress of the persistence process to generate a process copy, and distribute the process copy to several slaves. In the node; the master node and each of the secondary nodes save the process copy in the PM module;

The process abnormality mining module is set in the master node where the persistent process is located, and is configured to detect the failure of the master node, and when a failure is detected, forward the failure information to the failover module of the master node; The types of the failure include node power failure, process crash, and load imbalance;

A failover module is deployed on both the primary node and the secondary node;

The failover module on the primary node is configured to, after receiving the fault information, select a target migration node from the secondary nodes according to the fault information and the operating status of each secondary node, and send a migration instruction to the target migration node, requiring Restore the above persistent process to be migrated on the target migration node;

The failover module on the secondary node is configured to, after receiving the migration instruction, rebuild the persistence process on the secondary node, and use the node as a new primary node.

A further improvement of the present invention is that the process persistence module includes a persistence creation unit and a persistence update unit;

The persistent creation unit is configured to create a process copy of the persistent process on the master node; the process copy includes a verification identifier, a process PCB, an execution process first address, a node ID, a timestamp, and a copy of the process virtual address space; The process virtual address space copy adopts a balanced binary tree data structure, and each virtual page in the virtual address space is a balanced binary tree node;

The persistent updating unit is configured to periodically update the process copy, and send the updated process copy to each secondary node.

A further improvement of the present invention is that the fault types detected by the process abnormality mining module include node power failure, process crash, and load imbalance; the process abnormality mining module includes a power failure detection unit, a program crash detection unit, a load balance detection unit, and exception handling. unit;

The power failure detection unit is configured to transmit the power failure fault information to the exception processing unit when the master node is powered off;

The program crash detection unit is configured to capture the return value when the persistent process exits, and send process crash fault information to the exception handling unit when the return value is abnormal;

The load balancing detection unit is configured to poll the load coefficients of the primary node and each of the secondary nodes, and issue a load imbalance to the exception processing unit when the difference between the load coefficients of any two nodes is greater than a threshold accident details;

The exception processing unit is configured to receive fault information of the power failure detection unit, the program crash detection unit, and the load balancing detection unit, and forward the fault information to the failover module of the master node.

A further improvement of the present invention is that the failover module includes a node information statistics unit, a migration decision unit and a process recovery unit;

The node information statistics unit is configured to collect the running status of the primary node and each of the secondary nodes;

The migration decision unit is configured to, when the master node where it is located detects a failure, determine a persistent process that needs to perform failover, and select a target migration node for process migration according to the running state of each computing node in the distributed system ;

The process recovery unit on the master node is configured to send a migration instruction to the target migration node according to the migration decision unit determining the migration persistence process and the corresponding target migration node;

The process recovery unit on the target migration node is configured to search for a process copy from the PM module of the node after receiving the migration instruction, and rebuild the persistent process according to the process copy.

The advantages of the present invention are: the present invention is mainly aimed at the problem that the scientific computing process runs for a long time without reliability guarantee in a large-scale distributed system, and utilizes the characteristics of PM non-volatile, fast speed and large capacity to automatically maintain the process checkpoint and Process copy, and can automatically recover when the system fails, with good market prospects and application value.

Description of drawings

Fig. 1 is a schematic diagram of the overall architecture of a distributed system equipped with PM;

2 is a schematic diagram of the overall architecture of a process persistence device for a distributed non-volatile memory system;

3 is a schematic diagram of a process persistence execution module and its constituent units;

Figure 4 is a schematic diagram of the machine constituent unit of the process abnormality mining module

Fig. 5 is the structural representation of process virtual address space copy;

FIG. 6 is a schematic diagram of a process of inserting a node into a process address space balanced binary tree.

Detailed ways

As shown in FIG. 2 , an embodiment of the present invention includes a process persistence device for a distributed non-volatile memory system, which is arranged in computing nodes of the distributed non-volatile memory system, and the computing nodes are connected through network communication. For a persistent process that needs to run persistently, in the distributed non-volatile memory system, there is one computing node as the master node, and two or more computing nodes as the secondary nodes. Different persistent processes can have different master nodes. The process persistence device includes a process persistence module, a process exception mining module and a failover module. specific:

The process persistence module is set in the primary node where the persistence process is located, and is configured to periodically record the state and execution progress of the persistence process to generate a process copy, and distribute the process copy to each secondary node. The master node and each slave node save the process copy in the PM module, so as to rebuild the persistent process in the slave node according to the process copy. A necessary condition for generating a process copy is that the persistent process has been called and executed at least once.

In implementation, each process copy is equivalent to a checkpoint. The process of updating the process copy by the process persistence module adopts the method of appending to periodically update the status and execution progress of each process copy of the persistence process, so that the execution process on the primary node and the backup process on the secondary node reach a consistent state. and execution progress.

The process abnormality mining module is set in the main node where the persistent process is located, and its main function is to detect the failure of the main node and forward the failure information to the failover module of the main node; the type of failure of the main node includes node failure. Power outages, process crashes, load imbalances. In terms of implementation, the transmission of abnormal conditions adopts an interrupt mechanism or an event-triggered mechanism.

The failover module is deployed on the primary node and the secondary node. Its main function is to determine the type of abnormal situation and transfer the abnormal process to the appropriate node to continue running. In terms of implementation, the persistent process is rebuilt on the secondary node according to the process copy backed up at the last checkpoint, and then the secondary node is reset to the primary node, and the backup process is reset to the execution process. The failover modules in the primary and secondary nodes perform different functions, specifically:

The failover module on the primary node is configured to, after receiving the fault information, select the target migration node from the secondary nodes according to the fault information and the operating status of each secondary node, and send a migration instruction to the target migration node, requesting that the target migration node be migrated. Restore the above persistent process to be migrated on the node;

The failover module on the secondary node is configured to, after receiving the migration instruction, rebuild the persistence process on the secondary node, and use the node as a new primary node.

The above process persistence module, process exception mining module and failover module may be software modules implemented by pure software, hardware modules implemented by dedicated hardware, or functional modules implemented by combining software and hardware.

In some embodiments, the process persistence module includes a persistence creation unit and a persistence update unit, both of which operate according to the Crash consistency guarantee mechanism.

The persistence creation unit is configured to create a process copy of the persistent process on the master node; the process copy includes the verification identifier, the process PCB, the first address of the execution process, the node ID, the timestamp, and the copy of the process virtual address space. Based on the above, the entire persistence can be rebuilt. The copy of the virtual address space of the process adopts a balanced binary tree data structure, and each virtual page in the virtual address space is a balanced binary tree node. In the process of implementation, the process copy can be implemented by using the data structure Process_struct.

The persistent updating unit is configured to periodically update the process copy, and send the updated process copy to each secondary node. In the process of updating the process copy, it is transmitted to the secondary node through TCP/IB in an incremental manner, that is, the backup process data structure Process_struct_back is generated on the secondary node. For the same execution process, the persistent update unit saves the two checkpoints with the latest timestamps in the primary and secondary nodes in the form of undo.

Crash consistency guarantee mechanism refers to the first address of the execution process shown in Figure 3, the organization relationship of the linear area in the process address space of the execution process, the global page directory of the execution process, and the content of the CR3 register of the execution process by undo. For persistence, the above content will be backed up to the process copy. The content of the user data page of the executing process is updated according to the copy-on-write method, and the update mainly refers to the linear area organization relationship in the address space of the executing process.

As shown in Figure 4, the fault types detected by the process exception mining module include node power failure, process crash, and load imbalance; the process exception mining module includes a power failure detection unit, a program crash detection unit, a load balance detection unit, and an exception processing unit.

The power failure detection unit is configured to transmit the power failure fault information to the exception processing unit when the main node is powered off; the power failure detection unit is composed of a dedicated microcontroller, and the microcontroller runs during the node operation In the process, the PWR_OK signal of the power supply unit of the node is monitored, and when the signal drops to a low level, it is determined that a power failure occurs, and the DETECT_PWR signal is sent to other nodes through the out-of-band management network. The power failure detection unit can also continuously poll the working status of the power supply of the slave node from the master node by means of ipmitool. The node's exception handling unit sends the DETECT_PWR signal.

The program crash detection unit is configured to capture the return value when the persistent process exits, and send process crash fault information to the exception processing unit when the return value is abnormal; the program crash detection unit When the returned value is captured, when the captured return value is an abnormal value (such as a segmentation fault SIGSEGV, a user-defined abnormal return value), it is judged that the persistence process has crashed, exits abnormally, and sends a DETECT_CRASH signal to other nodes. The program crash detection unit may also detect kernel crashes. When the Linux kernel crashes, the unit executes the kexec mechanism, starts the backup kernel, and sends the DETECT_CRASH signal to other nodes in the backup kernel.

The load balancing detection unit is configured to poll the load coefficients of the primary node and each of the secondary nodes, and issue a load imbalance to the exception processing unit when the difference between the load coefficients of any two nodes is greater than a threshold Fault information; the load balancing detection unit running on the master node polls the CPU usage of the slave node and the CPU usage cpu_usage and memory usage memory_usage of the node through the software ipmitool. At the same time, it allows administrators to customize the weight of CPU usage cpu_weight and the weight of memory usage memory_weight. The load balancing detection unit calculates a load coefficient for each node, and the specific calculation method is cpu_usage×cpu_weight+memory_usage×memory_weight. When the load factor of any two nodes differs by more than 40%, it is determined that the current system is in a state of unbalanced load, and the DETECT_IMBALANCED signal is transmitted to the exception processing unit.

The exception processing unit is configured to receive fault information of the power failure detection unit, the program crash detection unit, and the load balancing detection unit, and forward the fault information to the failover module of the master node.

The failover module includes a node information statistics unit, a migration decision unit and a process recovery unit. The failover module is used to start the process failover process after receiving the failure signal from the process abnormality mining module, which specifically includes: judging the failure type, selecting the corresponding node and the persistence process accordingly, and selecting the target migration node to carry out the process. Migrate and continue running on the target migration node.

The node information statistics unit is configured to collect the running status of the primary node and each of the secondary nodes, and provide a reference for the migration decision unit. The node information statistics unit uses ipmitool to query the current running status of the master node itself and other nodes. The running status includes cpu_usage (CPU usage) and memory_available (available memory), and sends them to the migration decision unit.

The migration decision unit is configured to determine the persistent process that needs to perform failover when the master node where it is located detects a failure, and selects the target migration node for process migration according to the running status of each computing node in the distributed system; the decision-making principle is: :

If the fault signal sent by the process exception mining module is DETECT_PWR, all persistent processes on the faulty node must be migrated; if the fault signal is DETECT_CRASH, the crashed persistent process needs to be migrated; if the fault signal is DETECT_IMBALANCE, random Select a persistent process running on the failed node as the migration process. The node running status data required by the migration decision unit is provided by the node information statistics unit in the aforementioned master node. After knowing the CPU usage cpu_usage and available memory memory_available of each node, for each process that needs to be migrated, select the following algorithm according to the following algorithm: Target migration node: Among all the running secondary nodes in the current distributed system, select the node with the lowest cpu_usage among the nodes whose memory_available can accommodate the migration process as the target migration node; if the memory_available of all the secondary nodes is not enough to accommodate the migration process, Then select the secondary node with the largest memory_available as the target migration node. After the persistent process to be migrated and the target migration node are selected, the migration instruction is sent to the process recovery unit running on the master node.

The process recovery unit on the master node is configured to send a migration instruction to the target migration node according to the migration decision unit deciding the migration persistence process and the corresponding target migration node.

The process recovery unit on the target migration node is configured to search for a process copy from the PM module of the node after receiving the migration instruction, and rebuild the persistent process according to the process copy. During the rebuilding process, find the Process_struct (process copy) of the migration process on the current node, rebuild the PCB information, user data pages, open files, etc. of the process according to the content of the Process_struct (process copy), and resume the execution of the process.

The embodiment of the present invention also includes a process persistence method for a distributed non-volatile memory system, which is applied to a system composed of multiple computing nodes; the computer nodes are connected through network communication; it is characterized in that, for The execution of any persistent process includes the following steps:

(S1) Select one computing node as the primary node, and additionally select at least one computing node as the secondary node (usually 2 to 3 secondary nodes are selected); and initialize the persistence process in the primary node, and at the same time in the Create a process copy (Process_struct data structure) in the PM module of the master node;

(S2) During the running of the persistent process, the process copy is updated every predetermined time and distributed to each of the secondary nodes, and the secondary node stores the process copy of the persistent process in its in the PM module;

(S3) When the process abnormality mining module of the master node detects that the persistence process is faulty, select a target migration node from the secondary node, and select a target migration node according to the process copy stored in the target migration node. The persistent process is restored in the target migration node.

In the process of initializing the persistence process, the process persistence unit of the master node creates a process structure in the PM module as the process copy; the process copy includes a verification identifier, a process PCB, an execution process first address, and a node ID. , timestamp, copy of the process virtual address space. The copy of the process virtual address space adopts a balanced binary tree data structure, and each virtual page in the virtual address space is a balanced binary tree node, and its structure is shown in Figure 5.

In step (S1), in the process of initializing the persistence process, the user designates the process P as the persistence process when creating the process, and the linker links it with the program of the process persistence execution module. When the process completes the initialization work and enters the TASK_RUNNING state, a process Process_struct data structure (process copy) data structure is created in the PM of the master node, and then the process copy (Process_struct data structure) is initialized; the initialization process includes The following steps:

(S11) setting the check mark as unavailable (setting 0);

(S12) determine the timestamp according to the current time; determine the node ID according to the ID of the current master node;

(S13) the PCB of the persistent process, the first address of the execution process are written into the process copy;

(S14) Traverse the virtual address space of the persistent process, store each virtual page in use in the virtual address space copy of the process, and reset the SOFT_DIRTY flag bit of the virtual page; the specific steps are shown in Figure 6 Show;

(S15) Adjust the persistent process to a runnable (TASK_RUNNING) state, and set the verification flag to be available (set to 1). The check mark is used to indicate whether the process copy is available. If the check mark is unavailable, it means that the process copy is interrupted during the backup process, indicating that the process copy is incomplete.

In this embodiment, in the process of updating the process copy, the process virtual address space copy is updated by means of incremental update, which makes this method only need to create a complete process copy of the persistent process once. The period of updating the process copy is set by the user in the initialization phase of the persistent process, and the process of updating the process copy includes the following steps:

(S21) Suspend the persistent process, and set the verification identifier of the process replica to be unavailable;

(S22) according to the current time, the timestamp of the process copy is updated, and the CPU cache is written back to the memory;

(S23) the PCB of the persistent process, the first address of the execution process are written into the process copy;

(S24) Traverse the virtual address space of the persistent process, update or insert the virtual page that has been changed (SOFT_DIRTY flag = 1) in the copy of the virtual address space of the process, and update or insert each virtual page in the virtual address space The change flag (soft_dirty) of the virtual page is reset (set to 0); the specific steps are shown in Figure 6;

(S25) Adjust the persistent process to a runnable (TASK_RUNNING) state, and set the verification flag to be available. After the update is completed, after the data structure is sent to the secondary node through TCP/IB, the user process continues to run normally.

When the process abnormality mining module of the master node detects that the persistent process is faulty, the process abnormality mining module forwards the detected failure information to the failover module of the master node. The failover module's response to failure information includes the following steps:

(S31) The failover module on the primary node collects the running status of each secondary node, and selects the target migration node according to the running status; the running status includes CPU usage (cpu_usage) and available memory (memory_available); selecting the The process of migrating a node to a target includes the following steps:

(S311) If the failure type is node power failure or program crash, the following process is respectively performed for each persistent process on the failed node: among all the running secondary nodes in the current distributed system, select the available memory (memory_available ) and the node with the lowest CPU usage (cpu_usage) among the nodes that can accommodate the migration process as the target migration node; if the available memory (memory_available) of all nodes is not enough to accommodate the migration process, the node with the largest available memory (memory_available) is selected as the target migration node. The target migrates the node, and jumps to step (S32).

(S312) If the fault type is load imbalance, perform the following process on a randomly selected persistent process on the faulty node: select an available memory (memory_available) and The node with the lowest CPU usage (cpu_usage) among the nodes that can accommodate the migration process is selected as the target migration node; if the available memory (memory_available) of all nodes is not enough to accommodate the migration process, the node with the largest available memory (memory_available) is selected as the target migration node node, and jump to step (S32).

(S32) sending a migration instruction to each target migration node, requesting that the above-mentioned persistent process to be migrated be restored on the target migration node;

(S33) After the target migration node receives the migration instruction and the persistent process that needs to be recovered, its process recovery unit searches for the process copy of the persistent process in its own PM module, and checks its verification identifier. If the identifier is unavailable, the process recovers Failed; if there are multiple process copies of the same process, select the process copy with the latest timestamp;

(S34) The process recovery unit running on the target migration node rebuilds the persistent process on the node according to the process copy, and uses the node as a new master node. The process of rebuilding the persistence process includes the following steps:

(S341) establishing a new process at the target migration node, and initializing the process PCB according to the process replica structure;

(S342) traverse the process virtual address space copy of the process copy, restore the process address space according to the page address and the flag position of the balanced binary tree node in the process virtual address space copy, and restore the memory data of the process according to the page content of the balanced binary tree node;

(S343) Open the corresponding file according to the open file table in the process copy, where the target migration node is required to provide the same file resources and device drivers as the original master node;

(S344) Mark the new process as a runnable (TASK_RUNNING) state, insert it into the scheduling queue, and start running.

Specific embodiments in the implementation process of the present invention will be described below with reference to the accompanying drawings.

Specific embodiment one:

In this example, we assume that the device is running on a distributed system with a cluster with three nodes with node IDs: master, slave0, slave1. The nodes run the Linux operating system, and each node has sufficient DRAM and PM. The update period of the process Process_struct data structure (process copy) is set to 60 seconds. During the running of the persistence process, the slave0 node was powered off/program crashed. Since node power down/program crash is the same in effect, this embodiment does not differentiate between them.

The following steps are specifically described in conjunction with the device as follows:

Step 1: The user uses the interface provided by the device on the distributed system, executes the executable file, and performs scientific computing. The operating system assigns it to the slave0 node, uses it as the master node, uses mater and slave1 as the secondary node, and creates a persistent process on this node, and the process PID is 100;

Step 2: When the process is created and enters the TASK_RUNNABLE (runnable) state, it starts to create the process Process_struct data structure (process copy). The specific steps are as follows:

Step 2.1: The module creates an empty process Process_struct data structure (process copy) on the PM module of the slave0 node (master node), and the verification flag is initially set to 0;

Step 2.2: Set the timestamp of the process Process_struct data structure (process copy) to the current system time, set the node ID to slave0, and save the process PCB to the process control block area. Traverse each VMA (virtual address space) of the process, establish a balanced binary tree node for each memory page in it, and insert the address space balanced binary tree. The specific insertion process is shown in Figure 6;

Step 2.3: Write 4 to /proc/$pid/clear_refs to reset the SOFT_DIRTY bit of all memory pages of the process;

Step 2.4: Set the verification flag of the process Process_struct data structure (process copy), indicating that the Process_struct data structure (process copy) is completely available, and distribute the file to the master node and slave1 node through TCP/IB (for this persistence process, the master node and the slave1 node are all secondary nodes), which are stored in their PM modules respectively;

Step 3: Wake up the process so it starts running normally. After that, the process Process_struct data structure (process copy) is updated every 60 seconds. The specific steps are as follows:

Step 3.1: Every 60 seconds, the persistent update unit suspends the process, finds the process Process_struct data structure (process copy), and sets its verification flag to 0;

Step 3.2: Set the timestamp of the process Process_struct data structure (process copy) as the current system time, and save the process PCB to the process control block field. Traverse each VMA (virtual address space) of the process, and for each memory page in it, if its SOFT_DIRTY bit is 1, update/insert the corresponding address space balanced binary tree node. The specific update/insert process is shown in Figure 6 ;

Step 3.3: Write 4 to /proc/$pid/clear_refs to reset the SOFT_DIRTY bit of all memory pages of the process;

Step 3.4: Set the verification flag of the process Process_struct data structure (process copy), indicating that the Process_struct data structure (process copy) is completely available, and distribute the data structure to the master node and slave1 node through TCP/IB, and save them in the in its PM module;

Step 4: The power failure/program crash occurs on the slave0 node during operation, which is captured by the process fault mining module on the slave0 node, and sends the DETECT_PWR/DETECT_CRASH fault signal to the failover module running on the slave0 node;

Step 5: The process migration module searches the PM for the process Process_struct data structure (process copy) whose node ID is slave0, and finds one file altogether;

Step 6: The node information statistics unit collects the current cluster operating status, and learns that the current master node cpu_usage is 40%, the slave1 node cpu_usage is 30%, and the remaining memory is sufficient. Therefore, the slave1 node is selected as the target migration node and sent to the slave1 node. instruction on which to start the process recovery process;

Step 7: The slave1 node receives the slave0 command and resumes the process of the slave0 node. The specific steps are as follows:

Step 7.1: The slave1 node searches its PM for the Process_struct data structure (process copy) of the process whose node ID is slave0, finds a data structure, verifies that its complete identification is correct, and starts to read its content;

Step 7.2: The slave1 node establishes a new process in itself, and initializes the PCB of the new process according to the process control block part of the process Process_struct data structure (process copy). Here, the slave1 node is required to provide the same required files and device resources for the process;

Step 7.3: Traverse the address space balanced binary tree, apply for and map a memory page at the corresponding virtual address for each node in it, and fill in the page content;

Step 7.4: Mark the process as a persistent process, and then the persistent update unit on the slave1 node will continue to maintain the Process_struct data structure (process copy) of the process;

Step 7.5: Wake up the process and keep it running on slave1 node.

At this point, the migration of the persistence process is complete. The persistent process on the faulty slave0 node is not destroyed by the failure, but continues to run from a closer checkpoint on the slave1 node, which reflects the protection capability of the present invention for the process that takes a long time in the distributed system .

Specific embodiment two:

In this example, we assume that the device is running on a distributed system with a cluster, which has three nodes: master, slave0, slave1, the nodes run Linux operating system, and each node has sufficient DRAM and PM . The update period of the process Process_struct data structure (process copy) is set to 60 seconds. During the running of the persistence process, the system load is unbalanced.

The following steps are specifically described in conjunction with the device as follows:

Step 1: The user uses the interface provided by the device on the distributed system, executes the executable file, and performs scientific computing. The operating system root assigns it to the slave1 node, uses it as the master node, and uses mater and slave0 as the secondary node, and creates a persistent process on this node, and the process PID is 200;

Step 2: When the process is created and enters the TASK_RUNNABLE (runnable) state, it starts to create the process Process_struct data structure (process copy). The specific steps are as follows:

Step 2.1: The module creates an empty process Process_struct data structure (process copy) on the PM of the slave1 node, and the verification flag is initially set to 0;

Step 2.2: Set the timestamp of the process Process_struct data structure (process copy) to the current system time, set the node ID to slave1, and save the process PCB to the process control block area. Traverse each VMA (virtual address space) of the process, establish a balanced binary tree node for each memory page in it, and insert the address space balanced binary tree. The specific insertion process is shown in Figure 6;

Step 2.3: Write 4 to /proc/$pid/clear_refs to reset the SOFT_DIRTY bit of all memory pages of the process;

Step 2.4: Set the verification flag of the Process_struct data structure (process copy) of the process, indicating that the Process_struct data structure (process copy) is completely available, and distribute the file to the master node and slave0 node through TCP/IB, and save them in their in PM;

Step 3: Wake up the process so it starts running normally. After that, the process Process_struct data structure (process copy) is updated every 60 seconds. The specific steps are as follows:

Step 3.1: Every 60 seconds, the persistent update unit suspends the process, finds the process Process_struct data structure (process copy), and sets its verification flag to 0;

Step 3.2: Set the timestamp of the process Process_struct data structure (process copy) as the current system time, and save the process PCB to the process control block area. Traverse each VMA (virtual address space) of the process, and for each memory page in it, if its SOFT_DIRTY bit is 1, update/insert the corresponding address space balanced binary tree node. The specific update/insert process is shown in Figure 6 ;

Step 3.3: Write 4 to /proc/$pid/clear_refs to reset the SOFT_DIRTY bit of all memory pages of the process;

Step 3.4: Set the check mark of the process Process_struct data structure (process copy), indicating that the Process_struct data structure (process copy) is completely available, and distribute the data structure to the master node and slave0 node through TCP/IB, and save them in the in its PM;

Step 4: At a certain moment, the abnormal process mining module running on the slave1 node detects that the system has a load imbalance. At this time, the cpu_usage of the master node, slave0 node, and slave1 node are 40%, 30%, and 50% respectively. The process exception mining module running on the master node sends the DETECT_IMBALANCED signal to the failover module;

Step 5: The process migration module searches the PM for the process Process_struct data structure (process copy) whose node ID is slave1, and finds one data structure altogether;

Step 6: The node information statistics unit collects the current cluster operating status, and learns that the current master node cpu_usage is 40%, the slave0 node cpu_usage is 30%, and the remaining memory is sufficient, so the slave0 node is selected as the target migration node, and the slave0 node is sent to the slave0 node. instruction, start the process migration process on it, and send an instruction to the slave1 node to stop the running of the persistent process on it;

Step 7: The slave0 node receives the instruction of slave1 and restores the process of the slave1 node. The specific steps are as follows:

Step 7.1: The slave0 node searches its PM for the Process_struct data structure (process copy) of the process whose node ID is slave1, finds a data structure, verifies that its complete identification is correct, and starts to read its content;

Step 7.2: The slave0 node establishes a new process in itself, and initializes the process PCB according to the process control block part of the process Process_struct data structure (process copy). Here, the slave0 node is required to provide the same required files and device resources for the process;

Step 7.3: Traverse the address space balanced binary tree, apply for and map a memory page at the corresponding virtual address for each node in it, and fill in the page content;

Step 7.4: Mark the process as a persistent process, and then the persistent update unit on the slave0 node will continue to maintain the Process_struct data structure (process copy) of the process;

Step 7.5: Wake up the process and keep it running on the slave0 node.

At this point, the migration of the persistence process is complete. When the load imbalance phenomenon occurs in each node of the distributed system, the method and device of the present invention can migrate the process from the high-load node to the low-load node, so as to achieve the effect of load balancing, which shows that the scope of application of the present invention is not limited to The protection of persistent processes on power failure/program crash nodes can also be applied to the implementation of distributed system load balancing.

The above embodiments of the present invention do not constitute a limitation on the protection scope of the present invention. Any modifications, substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (13)

1. A process persistence method for a distributed non-volatile memory system is applied to a system consisting of a plurality of computing nodes; all the computer nodes are connected through network communication; the method is characterized in that for any persistent process, the execution process comprises the following steps:

selecting a computing node as a main node, and selecting at least one computing node as a secondary node; initializing the persistent process in the main node, and creating a process copy in a PM module of the main node;

in the running process of the persistent process, updating the process copy every other preset time, and distributing the process copy to each secondary node, wherein the secondary node stores the process copy of the persistent process in a PM module of the secondary node;

when the process abnormity mining module of the main node detects that the persistent process has a fault, a target migration node is selected from the auxiliary node, and the persistent process is recovered in the target migration node according to the process copy stored in the target migration node.

2. The process persistence method for the distributed non-volatile memory system according to claim 1, wherein in a process of initializing the persistent process, the process persistence unit of the master node creates a process structure in the PM module as the process copy; the process copy comprises a check identifier, a process PCB, an execution process initial address, a node ID, a timestamp and a process virtual address space copy; the process virtual address space copy adopts a balanced binary tree data structure, and each virtual page in the virtual address space is a balanced binary tree node.

3. The process persistence method for the distributed non-volatile memory system according to claim 1, wherein in a process of initializing the persistent process, a process copy is initialized; the initialization process comprises the following steps:

(S11) setting the check mark as unavailable;

(S12) determining the timestamp from the current time; determining the node ID according to the ID of the current main node;

(S13) writing the PCB of the persisted process, the executing process first address, to the process copy;

(S14) traversing the virtual address space of the persisted process, storing each in-use virtual page into a copy of the process virtual address space, and resetting the SOFT _ direct flag of that virtual page;

(S15) adjusting the persistence process to a runnable state and making the check mark available.

4. The process persistence method for the distributed non-volatile memory system according to claim 2, wherein in the process of updating the process copy, the process virtual address space copy is updated in an incremental update manner.

5. The method for persisting processes to a distributed non-volatile memory system according to claim 4, wherein the updating the process copy includes:

(S21) suspending the persisted process and setting a check flag of the process copy as unavailable;

(S22) updating the timestamp of the process copy according to the current time;

(S23) writing the PCB of the persisted process, the executing process first address, to the process copy;

(S24) traversing the virtual address space of the persistent process, updating or inserting the changed virtual page in the process virtual address space copy, and resetting the change flag of each virtual page in the virtual address space;

(S25) adjusting the persistence process to a runnable state and making the check mark available.

6. The process persistence method for the distributed non-volatile memory system according to claim 1, wherein when the process exception mining module of the master node detects that the persistent process has a failure, the process exception mining module forwards the failure information detected by the process exception mining module to the failover module of the master node.

7. The method for persisting processes in a distributed non-volatile memory system according to claim 6, wherein the step of responding to the failure information by the failover module includes:

(S31) the fault transfer module on the main node collects the running state of each secondary node and selects the target transfer node according to the running state; the running state comprises the CPU utilization rate and the available memory;

(S32) sending a migration instruction to each target migration node, and requiring the target migration node to recover the persistent process to be migrated;

(S33) after the target migration node receives the migration instruction and the persistent process needing to be recovered, the process recovery unit searches the process copy of the persistent process in the PM module of the target migration node, checks the check identifier of the process copy, and if the check identifier is unavailable, the process recovery fails; if a plurality of process copies of the same process exist, selecting the process copy with the latest timestamp;

(S34) the process recovery unit running on the target migration node rebuilds the persistent process on the node according to the process copy and takes the node as a new master node.

8. The method for persisting processes in a distributed non-volatile memory system according to claim 7, wherein the step of selecting the target migration node includes the steps of:

(S311) if the failure type is node power-off or program crash, respectively performing the following processes for each persistent process on the failed node: selecting a node with the lowest CPU utilization rate from all running auxiliary nodes in the current distributed system as a target migration node, wherein the available memory and the node which can accommodate a migration process are the nodes with the lowest CPU utilization rate; if the available memories of all the nodes are not enough to accommodate the migration process, selecting the node with the largest available memory as a target migration node, and jumping to the step (S32);

(S312) if the failure type is load imbalance, performing the following process for a randomly selected persistency process on the failed node: selecting a node with the lowest CPU utilization rate from all running auxiliary nodes in the current distributed system as a target migration node, wherein the available memory and the node which can accommodate a migration process are the nodes with the lowest CPU utilization rate; if the available memories of all the nodes are not enough to accommodate the migration process, the node with the largest available memory is selected as the target migration node, and the step is skipped to (S32).

9. The method for persisting processes in a distributed non-volatile memory system according to claim 7, wherein the process of rebuilding the persisted process includes the following steps:

(S341) establishing a new process at the target migration node, and constructing an initialization process PCB according to the process copy;

(S342) traversing the process virtual address space copy of the process copy, recovering the process address space according to the page address and the flag bit of the balanced binary tree node in the process virtual address space copy, and recovering the memory data of the process according to the page content of the balanced binary tree node;

(S343) opening a corresponding file according to the file opening table in the process copy;

(S344) mark the new process as runnable, insert the scheduling queue, and start running.

10. A process persistence device facing a distributed nonvolatile memory system is arranged in computing nodes of the distributed nonvolatile memory system, and the computing nodes are in communication connection through a network; the process persistence device is characterized by comprising a process persistence module, a process exception mining module and a fault transfer module; wherein:

the process persistence module is arranged in a main node where a persistence process is located, and is configured to periodically record the state and the execution progress of the persistence process to generate process copies and distribute the process copies to a plurality of secondary nodes; the primary node and each secondary node store the process copy in a PM module;

the process exception mining module is arranged in a main node where a persistent process is located and is configured to detect a fault of the main node and forward fault information to the fault transfer module of the main node when the fault is detected; the types of the faults comprise node outage, process crash and load unbalance;

fault transfer modules are deployed on the main node and the secondary node;

the fault transfer module on the main node is configured to select a target migration node from the auxiliary nodes according to the fault information and the running state of each auxiliary node after receiving the fault information, and send a migration instruction to the target migration node to request the target migration node to recover the persistent process to be migrated;

and the fault transfer module on the secondary node is configured to rebuild the persistent process on the secondary node after receiving the migration instruction, and take the node as a new primary node.

11. The device for persisting processes in a distributed non-volatile memory system according to claim 10, wherein the process persisting module includes a persisting creation unit and a persisting update unit;

the persistence creation unit is configured to create a process copy of a persistent process on a primary node; the process copy comprises a check identifier, a process PCB, an execution process initial address, a node ID, a timestamp and a process virtual address space copy; the process virtual address space copy adopts a balanced binary tree data structure, and each virtual page in the virtual address space is a balanced binary tree node;

the persistent updating unit is configured to periodically update the process copy and send the updated process copy to each secondary node.

12. The device for persisting processes for a distributed non-volatile memory system according to claim 10, wherein the types of failures detected by the process anomaly mining module include node power failure, process crash, and load imbalance; the process exception mining module comprises a power failure detection unit, a program crash detection unit, a load balancing detection unit and an exception handling unit;

the power failure detection unit is configured to transmit power failure fault information to the exception handling unit when the main node is powered off;

the program crash detection unit is configured to capture a return value when the persistent process exits, and send process crash fault information to the exception handling unit when the return value is abnormal;

the load balancing detection unit is configured to poll the load coefficients of the main node and the auxiliary nodes, and send load imbalance fault information to the exception handling unit when the difference value of the load coefficients of any two nodes is greater than a threshold value;

the exception handling unit is configured to receive fault information of the power failure detection unit, the program crash detection unit and the load balancing detection unit, and forward the fault information to a fault transfer module of the main node.

13. The process persistence method for the distributed non-volatile memory system according to claim 12, wherein the failover module includes a node information statistics unit, a migration decision unit, and a process recovery unit;

the node information statistical unit is configured to collect the operation states of the main node and each of the secondary nodes;

the migration decision unit is configured to determine a persistent process which needs to be subjected to fault transfer when the main node where the migration decision unit is located detects a fault, and select a target migration node for process migration according to the running state of each computing node in the distributed system;

the process recovery unit on the main node is configured to send a migration instruction to the target migration node according to the persistent process determined to be migrated by the migration decision unit and the corresponding target migration node;

and the process recovery unit on the target migration node is configured to find a process copy from the PM module of the node after receiving the migration instruction, and rebuild the persistent process according to the process copy.

Images