CN111930493A - NodeManager state management method, device and computing device in cluster - Google Patents

Abstract

Embodiments of the present invention relate to the technical field of distributed resource management and scheduling systems, and disclose a NodeManager state management method, device and computing device in a cluster. The method includes: collecting network load information of a cluster, and evaluating the hardware state of the cluster according to the network load information; determining the health state of the nodes in the cluster according to the evaluation result; NodeManager goes offline. Through the above method, the embodiment of the present invention realizes the NodeManager pre-judgment and automatic offline, guarantees the stable operation of the system, and avoids the situation of Container allocation failure and task failure caused by the node host being preempted by multiple applications. .

Description

NodeManager state management method, device and computing device in cluster

technical field

Embodiments of the present invention relate to the technical field of distributed resource management and scheduling systems, and in particular, to a method, an apparatus, and a computing device for managing the state of a NodeManager in a cluster.

Background technique

With the development of computer technology, various data-intensive application computing frameworks are emerging, such as MpaReduce, Spark, S4, Storm, etc. When adopting a computing framework, factors such as resource utilization, operation and maintenance costs, data sharing, etc. are generally considered. The user generally hopes to deploy all these computing frameworks into a common cluster, so that they can share the resources of the cluster and have a good understanding of the resources. for unified use. In this way, a unified resource management and scheduling platform was born, and its typical representative is YARN (Yet Another Resource Negotiator, another resource coordinator).

YARN is divided into ResouceManager (Global Resource Manager, RM) and NodeManager (Node Manager, NM) roles, where ResouceManager is mainly responsible for global allocation and management. NodeManager is responsible for resource allocation and management of a single node. After the NodeManager accepts the task, it can assign the Application Master and Container. When the host resource is not exclusively owned by YARN, it may cause the resource application failure of the ResouceManager.

In the prior art, YARN resource allocation only uses CPU and memory as computing resources, which are divided in advance in the form of yarn-site.xml configuration when the cluster is started. The connection between ResouceManager and NodeManager is maintained through heartbeat, and cannot be used for network operations. Make judgments to allocate resources. In addition, the Impala of the MPP architecture will also be deployed on the host of the Hadoop cluster, but the resource allocation is not managed according to YARN. When executing the MPP aggregation query, a large amount of data will be accumulated in the memory. At this time, if you continue to follow the configured memory If it is applied with the CPU, it will cause the Container allocation to fail and the task to fail. The memory occupied by the instant query is relatively high, but the usage time is relatively short. If all are reserved, it will cause waste of YARN. Therefore, this method cannot adapt to the situation where the node host is preempted by multiple applications.

SUMMARY OF THE INVENTION

In view of the above problems, the embodiments of the present invention provide a subscription database scaling method, apparatus, and computing device based on the TimesTen memory library, which overcome the above problems or at least partially solve the above problems.

According to an aspect of the embodiments of the present invention, a method for managing the state of a NodeManager in a cluster is provided, and the method includes:

Collect network load information of the cluster, and evaluate the hardware status of the cluster according to the network load information;

Determine the health status of the nodes in the cluster according to the evaluation result;

When the state of the node is unhealthy, the NodeManager is offline.

In an optional manner, the collecting network load information of the cluster, and evaluating the hardware status of the cluster according to the network load information, further includes:

Collect network load information of the cluster;

The network latency of the cluster is evaluated based on the network load information, and the disk status of the cluster is evaluated.

In an optional manner, when the host resource is not exclusive to YARN, the method further includes:

Evaluate CPU usage and memory usage;

The determining the health status of the nodes in the cluster according to the evaluation result further includes:

The health status of the nodes in the cluster is determined according to the results of the evaluation of the network latency, disk status, CPU usage and memory usage.

In an optional manner, when the host resource is exclusive to YARN, the method further includes:

When the network delay exceeds a preset value, the network delay of the cluster is evaluated in combination with the historical network delay and the health status records of the corresponding nodes.

In an optional manner, the method further includes:

Reconfigure CPU resources and memory resources;

When it is determined that the state of the nodes in the cluster is healthy according to the evaluation of the hardware state of the cluster, modify the parameters of the NodeManager configuration file to the reconfigured values;

An online operation is performed on the NodeManager.

In an optional manner, the evaluating the network delay of the cluster according to the network load information further includes:

Collect the request queuing time and processing time of the RPC queue through the JMX interface monitored by JMX in Hadoop;

After summing the request queuing time of all nodes, take the average value to obtain the reference queuing time, and take the processing time of the first host as the reference processing time;

Determine whether the network delay of the first host is greater than the reference queue time, or whether the network delay of the second host is greater than the reference processing time;

The determining the health status of the nodes in the cluster according to the evaluation result further includes:

When the network delay of the first host is greater than the reference queue time, or the network delay of the second host is greater than the reference processing time, it is determined that the state of the node is unhealthy.

In an optional manner, the evaluating the disk status of the cluster further includes:

View disk health through scripts;

Determine whether the disk reports an error;

The determining the health status of the nodes in the cluster according to the evaluation result further includes:

When a certain disk in the disk of the cluster reports an error, it is determined that the state of the node is unhealthy.

In an optional manner, the evaluating the CPU usage further includes:

Calculate the total number of cores N of the current CPU through the script, determine the current utilization rate p of the CPU not used by YARN, and the number of cores M of the CPU allocated by NodeManager;

Subtract the product of N and (1-p) from M to obtain a score for the evaluation of the CPU usage;

The determining the health status of the nodes in the cluster according to the evaluation result further includes:

When the evaluated score of the CPU usage exceeds a preset CPU usage threshold, it is determined that the state of the node is unhealthy.

In an optional manner, the evaluating the memory usage further includes:

Obtain the total amount of memory through the script, the total amount of memory allocated in NodeManager and the usage of system processes;

Determine whether the difference between the total amount of memory and the system process usage is greater than the total amount of memory allocated in the NodeManager;

The determining the health status of the nodes in the cluster according to the evaluation result further includes:

When the difference between the total amount of memory and the usage amount of the system process is not greater than the total amount of memory allocated in the NodeManager, it is determined that the state of the node is unhealthy.

According to another aspect of the embodiments of the present invention, there is provided a NodeManager state management device in a cluster, the device comprising:

an evaluation module, configured to collect network load information of the cluster, and evaluate the hardware state of the cluster according to the network load information;

a determining module, configured to determine the health status of the nodes in the cluster according to the evaluation result;

The management module is used for offline operation of the NodeManager when the state of the node is unhealthy.

According to another aspect of the embodiments of the present invention, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete each other through the communication bus. communication between;

The memory is used for storing at least one executable instruction, and the executable instruction enables the processor to perform the operations of the above-mentioned NodeManager state management method in a cluster.

According to another aspect of the embodiments of the present invention, a computer storage medium is provided, where at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute the foregoing method for managing NodeManager status in a cluster .

The embodiment of the present invention automatically collects and evaluates the hardware state of the cluster, determines the health state of the nodes in the cluster according to the evaluation result, and performs offline operation on the NodeManager when the state of the node is unhealthy, so as to realize the prediction before the NodeManager fails. and automatic offline, which ensures the stable operation of the system; at the same time, the embodiment of the present invention does not only evaluate the health status of the node based on the memory and CPU status in the configuration, and avoids the failure of Container allocation caused by the preemption of multiple applications on the node host. This in turn causes the task to fail.

The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to understand the technical means of the embodiments of the present invention more clearly, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and The advantages can be more clearly understood, and the following specific embodiments of the present invention are given.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for the purpose of illustrating preferred embodiments only and are not to be considered limiting of the invention. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:

1 shows a flowchart of a method for managing NodeManager state in a cluster provided by an embodiment of the present invention;

2 shows a flowchart of a method for managing NodeManager state in a cluster provided by another embodiment of the present invention;

3 shows a flowchart of a method for managing NodeManager state in a cluster provided by another embodiment of the present invention;

4 shows a flowchart of a method for managing NodeManager state in a cluster provided by yet another embodiment of the present invention;

FIG. 5 shows a flowchart of a method for managing NodeManager state in a cluster provided by a specific application instance of Embodiment 1 of the present invention;

6 shows a schematic structural diagram of a NodeManager state management device in a cluster provided by an embodiment of the present invention;

FIG. 7 shows a schematic structural diagram of a computing device provided by an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present invention will be more thoroughly understood, and will fully convey the scope of the invention to those skilled in the art.

1 shows a flowchart of a method for managing NodeManager state in a cluster provided by an embodiment of the present invention. The method is applied to computing devices, such as a server in a communication network, a management computer in a unified resource management and scheduling platform of a cluster, and the like. As shown in Figure 1, the method includes the following steps:

Step 110: Collect network load information of the cluster, and evaluate the hardware status of the cluster according to the network load information.

In this step, the hardware status includes network delay, disk status, and the like. Generally, when the host resource YARN is exclusively used, the network status can be judged by automatically collecting the network load information, and the hard disk status can also be judged to evaluate the hardware status of the cluster. This step further includes:

Step A1: Collect network load information of the cluster;

Step A2: Evaluate the network delay of the cluster according to the network load information, and evaluate the disk status of the cluster.

Step 120: Determine the health status of the nodes in the cluster according to the evaluation result.

Among them, according to the evaluation result, it is judged whether there is a risk of downtime. If so, it is determined that the health status of the nodes in the cluster is unhealthy, and further processing is required. The result of the evaluation may be whether the hardware status of the above-mentioned cluster complies with the preset condition, and when it complies, it is determined that the status of the nodes in the cluster is unhealthy. Alternatively, the evaluation result may also be a score, and when the evaluated score is greater than or less than a preset threshold, it is determined that the state of the node in the cluster is unhealthy. It can be understood that, in step 110, the hardware state includes one or more hardware states, and at this time, if the evaluation result of a certain hardware state meets the preset condition or the evaluation score is greater than or less than the preset threshold, then It is determined that the node status of the cluster is unhealthy without determining the health status of the node according to the evaluation result of the overall hardware.

Step 130: Perform an offline operation on the NodeManager when the state of the node is unhealthy.

In this step, the NodeManager is taken offline according to the current status of the node without affecting the service, so as to ensure the stable operation of the system. It can be understood that when the conditions are restored, the NodeManager can also be modified to appropriate parameters and resumed online, which will be described in detail later.

FIG. 2 shows a flowchart of a method for managing the state of a NodeManager in a cluster provided by another embodiment of the present invention. In this embodiment, the host resource is not exclusively owned by YARN. As shown in Figure 2, the method includes the following steps:

Step 110: Collect network load information of the cluster, and evaluate the hardware status of the cluster according to the network load information.

Step 210: Evaluate the CPU usage and memory usage when the host resource is not exclusive to YARN.

Judging whether the host resource is exclusive to YARN belongs to the judgment of the software process.

Step 120: Determine the health status of the nodes in the cluster according to the results of the evaluation of the network delay, disk status, CPU usage and memory usage.

At this time, there are multiple items to be evaluated, and when the evaluation result of one of the items meets the preset condition or the evaluation score is greater than or less than the preset threshold, it can be determined that the node status of the cluster is unhealthy, and there is no need to base the The results of the evaluation of all items determine the health status of the node. For example, only if the network delay is greater than a preset threshold, it can be determined that the state of the nodes in the cluster is unhealthy.

Step 130: Perform an offline operation on the NodeManager when the state of the node is unhealthy.

Wherein, step 110 , step 120 and step 130 are the same as those in the foregoing embodiments, and reference may be made to the detailed descriptions of the foregoing embodiments, which will not be repeated here.

In this embodiment, when the host resources are not exclusively owned by YARN, by analyzing the current utilization rate of CPU and memory resources, and fully considering the priorities of other applications, the current node status can be reasonably evaluated, and the current node status can be used without affecting services. Take the node offline and restore it.

FIG. 3 shows a flowchart of a method for managing the state of a NodeManager in a cluster provided by another embodiment of the present invention. In this embodiment, the host resource is exclusively owned by YARN and the network delay is too large. As shown in Figure 3, the method includes the following steps:

Step 110: Collect network load information of the cluster, and evaluate the hardware status of the cluster according to the network load information.

Among them, the hardware state includes network latency.

Step 310: When the host resource is exclusive to YARN and the network delay exceeds a preset value, the network delay of the cluster is evaluated in combination with the historical network delay and the health status records of the corresponding nodes.

In this step, when the host resource is exclusively owned by YARN, if the current network traffic is too large, other services may be occupying the bandwidth, not the node status is unhealthy. operation, may lead to unnecessary offline and reduce the efficiency of system operation. Therefore, you can refer to the historical record information, which includes various network delays and records of whether the node is healthy at that time. If the network delay exceeds a preset value, the historical network delay and its corresponding node health status records are combined to assist in evaluating the network delay of the cluster. If the status of nodes exceeding a certain proportion (eg, 80%) in the historical records are healthy under similar network delay conditions, it can be determined that the network delay is normal at this time.

Step 120: Determine the health status of the nodes in the cluster according to the result of the network delay evaluation.

Step 130: Perform an offline operation on the NodeManager when the state of the node is unhealthy.

Wherein, step 110 , step 120 and step 130 are the same as those in the foregoing embodiments, and reference may be made to the detailed descriptions of the foregoing embodiments, which will not be repeated here.

In this embodiment, when the host resource is exclusively owned by YARN and the current network traffic is too large, other services may be occupying the bandwidth. At this time, it is comprehensively judged whether to send the NodeManager offline command according to the historical traffic peak, so as to avoid false logoffs. Wire.

FIG. 4 shows a flowchart of a method for managing the state of a NodeManager in a cluster provided by yet another embodiment of the present invention. In this embodiment, after the NodeManager goes offline, when the conditions are restored, the NodeManager is modified to appropriate parameters and then goes online again. As shown in Figure 4, the method includes the following steps:

Step 110: Collect network load information of the cluster, and evaluate the hardware status of the cluster according to the network load information.

Step 120: Determine the health status of the nodes in the cluster according to the evaluation result.

Step 130: Perform an offline operation on the NodeManager when the state of the node is unhealthy.

Step 440: Reconfigure CPU resources and memory resources.

In this step, the dynamic allocation and utilization of resources can be achieved by modifying the values of yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores in the yarn-site.xml configuration file through the program.

The CPU resources can be reconfigured by the following steps:

1. Obtain the idle time of each CPU through the system stat command, and evaluate the overall CPU usage;

2. Calculate and remove the CPU time used by NodeManager;

3. Calculate the proportion of idle CPU according to the proportion, and combine the number of physical CPU cores N to obtain the number of cores that should be allocated to the CPU. Among them, the proportion of idle CPU is: user, nice, system, idle four CPU total proportion; should be The calculation formula of the number of cores allocated to the CPU is: Pf ₁ +Pf ₂ +Pf ₃ +…+Pf _n , where Pf ₁ refers to the idle ratio of CPU core 1… Pf _n refers to the idle ratio of CPU core n proportion.

The memory resources can be reconfigured by the following steps:

1. Count the preset –X _mx value M _x when the Java process of NodeManager is started, where X _mx refers to the maximum heap memory occupied by the Java process startup;

2. Count the total memory M _t of the current system as a whole, and the total memory M _u occupied by the current system;

3. Calculate the total amount of memory M _s that needs to be allocated, and the calculation formula is: M _s =M _t -(M _x +M _u )

Step 450: When it is determined that the state of the nodes in the cluster is healthy according to the evaluation of the hardware state of the cluster, modify the parameters of the NodeManager configuration file to the reconfigured values.

Step 460: Perform an online operation on the NodeManager.

In this embodiment, the yarn-site.xml configuration of the node can be modified according to the current operating status of the node after going offline, and memory and CPU are allocated in a more flexible manner when the node is brought back online.

In the above embodiment, how to evaluate the network delay of the cluster, how to evaluate the disk status of the cluster, how to evaluate the CPU usage, and how to evaluate the memory usage according to the network load information in the above embodiment The assessment is described in further detail.

In some embodiments, in the above step A2, evaluating the network delay of the cluster according to the network load information includes the following steps:

Step A21: Collect the request queuing time and processing time of the RPC queue through the JMX interface monitored by JMX in Hadoop;

Step A22: Summing the request queuing times of all nodes and taking an average value to obtain the reference queuing time, and taking the processing time of the first host as the reference processing time;

Step A23: Determine whether the network delay of the first host is greater than the reference queue time, or whether the network delay of the second host is greater than the reference processing time;

At this time, the determining the health status of the nodes in the cluster according to the evaluation result further includes:

When the network delay of the first host is greater than the reference queue time, or the network delay of the second host is greater than the reference processing time, it is determined that the state of the node is unhealthy.

Specifically, the network delay score of the first host and the network delay score of the second host can be calculated; wherein the network delay score of the first host is greater than the reference when the network delay of the first host The queue time is the difference between the network delay of the first host and the reference queue time, otherwise it is 0: the network delay score of the second host is when the network delay of the second host is greater than the reference processing time is the difference between the network delay of the second host and the reference processing time, otherwise it is 0; the network delay score of the first host and the network delay score of the second host are added to obtain the cluster the network latency score. When the network delay score of the cluster is not 0, it is determined that the state of the node is unhealthy.

This embodiment collects RpcQueueTimeAvgTime and RpcProcessingTimeAvgTime through the JMX interface monitored by JMX in Hadoop, so as to obtain the request queuing time and processing time of the RPC queue, and obtain the average time of the normal operation of the cluster. The specific calculation can refer to the following formula:

Reference queue time Tq=(Tq ₁ +Tq ₂ +Tq ₃ +...+Tq _N )/N

Reference processing time Tp=(Tp ₁ +Tp ₁ +Tp ₁ +...+Tp ₁ )/N

The current network delay time is judged as: (t ₁ -Tq)>0? (t ₁ -Tq): 0+(t ₂ -Tp)>0? (t ₂ -Tp):0

Among them, N represents N nodes, t ₁ represents the network delay of host one, and t ₂ represents the network delay of host two.

When the network delay time exceeds a preset value (for example, 0.8s), it is determined that the node state is unhealthy.

In some embodiments, in the above step A2, evaluating the disk status of the cluster includes the following steps:

Step A21': Check the disk operating status through a script;

Step A22': determine whether the disk reports an error;

At this time, the determining the health status of the nodes in the cluster according to the evaluation result further includes:

When a certain disk in the disk of the cluster reports an error, it is determined that the state of the node is unhealthy.

Specifically, when a certain disk does not report an error, the disk status score of the disk is 100; when a certain disk reports an error, the disk status score of the disk is 0; when any disk status score is 0, all The evaluation score of the disk state of the cluster is 0, and when the state score of all the disks is 100, the evaluation score of the disk state of the cluster is 100. When the evaluation score of the disk state of the cluster is 0, it is determined that the state of the node is unhealthy.

In this embodiment, the running status of the disk can be checked by executing the smartctl-H sdaN script (the built-in check script of linux), and the evaluation score is 100 points when the disk does not report an error, and the evaluation score is 0 when the disk reports an error. The final score of the disk is: 0∈(D ₁ ,D ₂ ,D ₃ ,…,D _n )? 0:100, where D ₁ is the score for disk 1...D _n is the score for disk n. When the disk score is 0, the node status is determined to be unhealthy.

In some embodiments, in the above step A2, when the host resource is not exclusive to YARN, evaluating the CPU usage includes the following steps:

Step B1: Calculate the total number of cores N of the current CPU through the script, determine the current usage rate p of the CPU not used by YARN, and the number of cores M of the CPU allocated by NodeManager;

Step B2: Subtract the product of N and (1-p) from M to obtain the evaluation score of the CPU usage.

At this time, determining the health status of the nodes in the cluster according to the result of the CPU usage evaluation, further comprising:

When the evaluated score of the CPU usage exceeds a preset CPU usage threshold, it is determined that the state of the node is unhealthy.

In this embodiment, the total number of cores N of the current CPU can be calculated by checking /proc/stat (the file occupied by the current system is displayed by the Linux system), and the CPU usage p and the number of cores M allocated by NodeManager can be determined, and then the CPU score can be determined. is M-N*(1-p). When the CPU usage score exceeds a preset value (eg, 75% or 80%), the node state is determined to be unhealthy.

In some embodiments, in the above step A2, when the host resource is not exclusive to YARN, evaluating the memory usage includes the following steps:

Step C1: Obtain the total amount of memory, the total amount of memory allocated in NodeManager and the amount of system process usage through the script;

Step C2: Determine whether the difference between the total amount of memory and the usage amount of the system process is greater than the total amount of memory allocated in the NodeManager.

At this time, determining the health status of the nodes in the cluster according to the result of the evaluation of the memory usage, further comprising:

When the difference between the total amount of memory and the usage amount of the system process is not greater than the total amount of memory allocated in the NodeManager, it is determined that the state of the node is unhealthy.

Specifically, when the difference between the total amount of memory and the amount of system process usage is greater than the total amount of memory allocated in the NodeManager, the evaluation score of the memory usage rate is 100, otherwise the memory usage rate The evaluation score is 0; when the memory usage evaluation score is 0, it is determined that the state of the node is unhealthy.

In this embodiment, the total amount of memory amem, the total amount of memory allocated in Nodemanager nmem, and the amount of system process usage smem are obtained through the /proc/meminfo (Linux system displays files currently occupied by the system) file. Then the memory score is amem-smem>nmem? 100:0. When the memory usage score is 0, the node status is determined to be unhealthy.

The following describes the embodiment of the present invention in further detail through a specific application example. FIG. 5 shows a flowchart of a method for managing the state of a NodeManager in a cluster provided by a specific application example of the embodiment of the present invention. As shown in Figure 5, the method includes the following steps:

Step 510: Evaluate the network delay and disk bad blocks of the cluster to obtain a network score and a disk score.

Step 520: Determine whether the host resource is exclusive to YARN; if so, go to Step 540; otherwise, go to Step 530;

Step 530: Evaluate the CPU usage and memory usage of the cluster to obtain a CPU score and a memory score.

Step 540 : Determine whether any one of the above-mentioned scores satisfies the respective preset conditions; if yes, go to Step 550 ; otherwise, go to Step 560 .

Step 550: Take the NodeManager offline and modify its configuration.

Before performing this step, set the node status to unhealthy.

Step 560: Continue to run.

Before performing this step, set the node status to healthy.

In this embodiment, scoring is performed by comprehensively judging the real-time status of the network, disk, CPU, and memory of the host. When a certain score meets a certain score, the NodeManager role of the host is offline. When the conditions are restored, for example Disk repair, memory occupancy reduction, and network delay meet the online conditions, that is, when it is determined that the status of the nodes in the cluster is healthy according to the evaluation of the hardware status of the cluster, the NodeManager is modified to appropriate parameters to resume online.

The embodiment of the present invention automatically collects and evaluates the hardware state of the cluster, determines the health state of the nodes in the cluster according to the evaluation result, and performs offline operation on the NodeManager when the state of the node is unhealthy, so as to realize the prediction before the NodeManager fails. and automatic offline, which ensures the stable operation of the system; at the same time, the embodiment of the present invention does not only evaluate the health status of the node based on the memory and CPU status in the configuration, and avoids the failure of Container allocation caused by the preemption of multiple applications on the node host. This in turn causes the task to fail.

FIG. 6 shows a schematic structural diagram of a NodeManager state management apparatus in a cluster provided by an embodiment of the present invention. As shown in FIG. 6 , the apparatus 600 includes an evaluation module 610 , a determination module 620 and a management module 630 .

The evaluation module 610 is used to collect the network load information of the cluster, and to evaluate the hardware state of the cluster according to the network load information; the determination module 620 is used to determine the health state of the nodes in the cluster according to the evaluation result; the management module 630 is used to perform an offline operation on the NodeManager when the state of the node is unhealthy.

In an optional manner, the evaluation module 610 is further configured to:

Collect network load information of the cluster;

The network latency of the cluster is evaluated based on the network load information, and the disk status of the cluster is evaluated.

In an optional manner, when the host resource is not exclusive to YARN, the evaluation module 610 is further configured to:

Evaluate CPU usage and memory usage;

The determining module 620 is further configured to:

The health status of the nodes in the cluster is determined according to the results of the evaluation of the network latency, disk status, CPU usage and memory usage.

In an optional manner, when the host resource is exclusive to YARN, the evaluation module 610 is further configured to:

When the network delay exceeds a preset value, the network delay of the cluster is evaluated in combination with the historical network delay and the health status records of the corresponding nodes.

In an optional manner, the device further includes:

a configuration module 640 for reconfiguring CPU resources and memory resources;

A modification module 650, configured to modify the parameters of the NodeManager configuration file to the reconfigured values when the state of the nodes in the cluster is determined to be healthy according to the evaluation of the hardware state of the cluster;

The management module 630 is further configured to perform an online operation on the NodeManager.

In an optional manner, the evaluation module 610 is further configured to:

Collect the request queuing time and processing time of the RPC queue through the JMX interface monitored by JMX in Hadoop;

After summing the request queuing time of all nodes, take the average value to obtain the reference queuing time, and take the processing time of the first host as the reference processing time;

Determine whether the network delay of the first host is greater than the reference queue time, or whether the network delay of the second host is greater than the reference processing time;

The determining module 620 is further configured to:

When the network delay of the first host is greater than the reference queue time, or the network delay of the second host is greater than the reference processing time, it is determined that the state of the node is unhealthy.

In an optional manner, the evaluation module 610 is further configured to:

View disk health through scripts;

Determine whether the disk reports an error;

The determining module 620 is further configured to:

When a certain disk in the disk of the cluster reports an error, it is determined that the state of the node is unhealthy.

In an optional manner, the evaluation module 610 is further configured to:

Calculate the total number of cores N of the current CPU through the script, determine the current utilization rate p of the CPU not used by YARN, and the number of cores M of the CPU allocated by NodeManager;

Subtract the product of N and (1-p) from M to obtain a score for the evaluation of the CPU usage;

The determining module 620 is further configured to:

When the evaluated score of the CPU usage exceeds a preset CPU usage threshold, it is determined that the state of the node is unhealthy.

In an optional manner, the evaluation module 610 is further configured to:

Obtain the total amount of memory through the script, the total amount of memory allocated in NodeManager and the usage of system processes;

Determine whether the difference between the total amount of memory and the system process usage is greater than the total amount of memory allocated in the NodeManager;

The determining module 620 is further configured to:

When the difference between the total amount of memory and the usage amount of the system process is not greater than the total amount of memory allocated in the NodeManager, it is determined that the state of the node is unhealthy.

The embodiment of the present invention automatically collects and evaluates the hardware state of the cluster, determines the health state of the nodes in the cluster according to the evaluation result, and performs offline operation on the NodeManager when the state of the node is unhealthy, so as to realize the prediction before the NodeManager fails. and automatic offline, which ensures the stable operation of the system; at the same time, the embodiment of the present invention does not only evaluate the health status of the node based on the memory and CPU status in the configuration, and avoids the failure of Container allocation caused by the preemption of multiple applications on the node host. This in turn causes the task to fail.

An embodiment of the present invention provides a computer storage medium, where at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute the method for managing the state of a NodeManager in a cluster in any of the foregoing method embodiments.

The embodiment of the present invention automatically collects and evaluates the hardware state of the cluster, determines the health state of the nodes in the cluster according to the evaluation result, and performs offline operation on the NodeManager when the state of the node is unhealthy, so as to realize the prediction before the NodeManager fails. and automatic offline, which ensures the stable operation of the system; at the same time, the embodiment of the present invention does not only evaluate the health status of the node based on the memory and CPU status in the configuration, and avoids the failure of Container allocation caused by the preemption of multiple applications on the node host. This in turn causes the task to fail.

An embodiment of the present invention provides a computer program product, the computer program product includes a computer program stored on a computer storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, causes the computer to The method for managing the state of a NodeManager in a cluster in any of the foregoing method embodiments is executed.

The embodiment of the present invention automatically collects and evaluates the hardware state of the cluster, determines the health state of the nodes in the cluster according to the evaluation result, and performs offline operation on the NodeManager when the state of the node is unhealthy, so as to realize the prediction before the NodeManager fails. and automatic offline, which ensures the stable operation of the system; at the same time, the embodiment of the present invention does not only evaluate the health status of the node based on the memory and CPU status in the configuration, and avoids the failure of Container allocation caused by the preemption of multiple applications on the node host. This in turn causes the task to fail.

FIG. 7 shows a schematic structural diagram of a computing device provided by an embodiment of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

As shown in FIG. 7 , the computing device may include: a processor (processor) 702 , a communications interface (Communications Interface) 704 , a memory (memory) 706 , and a communication bus 708 .

The processor 702 , the communication interface 704 , and the memory 706 communicate with each other through the communication bus 708 . The communication interface 704 is used to communicate with network elements of other devices such as clients or other servers. The processor 702 is configured to execute the program 710, and may specifically execute the method for managing the state of a NodeManager in a cluster in any of the foregoing method embodiments.

Specifically, the program 710 may include program code including computer operation instructions.

The processor 702 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the computing device may be the same type of processors, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 706 is used to store the program 710 . Memory 706 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

The embodiment of the present invention automatically collects and evaluates the hardware state of the cluster, determines the health state of the nodes in the cluster according to the evaluation result, and performs offline operation on the NodeManager when the state of the node is unhealthy, so as to realize the prediction before the NodeManager fails. and automatic offline, which ensures the stable operation of the system; at the same time, the embodiment of the present invention does not only evaluate the health status of the node based on the memory and CPU status in the configuration, and avoids the failure of Container allocation caused by the preemption of multiple applications on the node host. This in turn causes the task to fail.

The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with teaching based on this. The structure required to construct such a system is apparent from the above description. Furthermore, embodiments of the present invention are not directed to any particular programming language. It is to be understood that various programming languages may be used to implement the inventions described herein, and that the descriptions of specific languages above are intended to disclose the best mode for carrying out the invention.

In the description provided herein, numerous specific details are set forth. It will be understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it is to be understood that, in the above description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together into a single implementation in order to simplify the invention and to aid in the understanding of one or more of the various aspects of the invention. examples, figures, or descriptions thereof. This disclosure, however, should not be construed as reflecting an intention that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and further they may be divided into multiple sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method so disclosed may be employed in any combination, unless at least some of such features and/or procedures or elements are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, it will be understood by those skilled in the art that although some of the embodiments herein include certain features, but not others, included in other embodiments, that combinations of features of the different embodiments are intended to be within the scope of the present invention And form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

It should be noted that the above-described embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names. The steps in the above embodiments should not be construed as limitations on the execution order unless otherwise specified.

Claims (12)

1. A method for managing NodeManager state in a cluster, wherein the method comprises: Collect network load information of the cluster, and evaluate the hardware status of the cluster according to the network load information; Determine the health status of the nodes in the cluster according to the evaluation result; When the state of the node is unhealthy, the NodeManager is offline. 2 . The method according to claim 1 , wherein the collecting network load information of the cluster, and evaluating the hardware status of the cluster according to the network load information, further comprises: 2 . Collect network load information of the cluster; The network latency of the cluster is evaluated based on the network load information, and the disk status of the cluster is evaluated. 3. The method according to claim 2, wherein when the host resource is not exclusive to YARN, the method further comprises: Evaluate CPU usage and memory usage; The determining the health status of the nodes in the cluster according to the evaluation result further includes: The health status of the nodes in the cluster is determined according to the results of the evaluation of the network latency, disk status, CPU usage and memory usage. 4. The method according to claim 2, wherein when the host resource is exclusive to YARN, the method further comprises: When the network delay exceeds a preset value, the network delay of the cluster is evaluated in combination with the historical network delay and the health status records of the corresponding nodes. 5. The method according to any one of claims 1-4, wherein the method further comprises: Reconfigure CPU resources and memory resources; When it is determined that the state of the nodes in the cluster is healthy according to the evaluation of the hardware state of the cluster, modify the parameters of the NodeManager configuration file to the reconfigured values; Perform an online operation on the NodeManager. 6. The method according to claim 2, wherein the evaluating the network delay of the cluster according to the network load information further comprises: Collect the request queuing time and processing time of the RPC queue through the JMX interface monitored by JMX in Hadoop; After summing the request queuing time of all nodes, take the average value to obtain the reference queuing time, and take the processing time of the first host as the reference processing time; Determine whether the network delay of the first host is greater than the reference queue time, or whether the network delay of the second host is greater than the reference processing time; The determining the health status of the nodes in the cluster according to the evaluation result further includes: When the network delay of the first host is greater than the reference queue time, or the network delay of the second host is greater than the reference processing time, it is determined that the state of the node is unhealthy. 7. The method according to claim 2, wherein the evaluating the disk status of the cluster further comprises: View disk health through script; Determine whether the disk reports an error; The determining the health status of the nodes in the cluster according to the evaluation result further includes: When a certain disk in the disk of the cluster reports an error, it is determined that the state of the node is unhealthy. 8. The method according to claim 3, wherein the evaluating the CPU usage further comprises: Calculate the total number of cores N of the current CPU through the script, determine the current utilization rate p of the CPU not used by YARN, and the number of cores M of the CPU allocated by NodeManager; Subtract the product of N and (1-p) from M to obtain a score for the evaluation of the CPU usage; The determining the health status of the nodes in the cluster according to the evaluation result further includes: When the evaluated score of the CPU usage exceeds a preset CPU usage threshold, it is determined that the state of the node is unhealthy. 9. The method according to claim 3, wherein the evaluating the memory usage further comprises: Obtain the total amount of memory through the script, the total amount of memory allocated in NodeManager and the usage of system processes; Determine whether the difference between the total amount of memory and the system process usage is greater than the total amount of memory allocated in the NodeManager; The determining the health status of the nodes in the cluster according to the evaluation result further includes: When the difference between the total amount of memory and the usage amount of the system process is not greater than the total amount of memory allocated in the NodeManager, it is determined that the state of the node is unhealthy. 10. A NodeManager state management device in a cluster, wherein the device comprises: an evaluation module, configured to collect network load information of the cluster, and evaluate the hardware state of the cluster according to the network load information; a determining module, configured to determine the health status of the nodes in the cluster according to the evaluation result; The management module is used for offline operation of the NodeManager when the state of the node is unhealthy. 11. A computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the method for managing the state of a NodeManager in a cluster according to any one of claims 1-9. 12. A computer storage medium, wherein the storage medium stores at least one executable instruction, and the executable instruction causes the processor to execute the NodeManager in the cluster according to any one of claims 1-9 State management methods.

Images