TWI506456B - System and method for dispatching hadoop jobs in multi-cluster environment - Google Patents

Description

Work distribution system and method based on Hadoop multi-cluster environment

The invention relates to a work assignment system and method based on Hadoop multi-cluster environment, in particular to a combination cluster feature analysis, work feature analysis, analysis and judgment of cluster and operation tasks, and computing tasks through execution environment selection. Delivery to achieve a simple and efficient deployment of service computing resources.

In recent years, due to the large amount of informationization, the average enterprise and government agencies are facing an explosive growth in data volume. In the fields of data storage, database or data retrieval and data exploration, the same problem is encountered. The huge and time-consuming work that has been done is no longer burdened by a supercomputer, but instead is directed to computing through a large number of group computers, so as to obtain maximum benefits. Today's information field uses cloud-based services to provide decentralized computing to solve the above problems, including Apache Hadoop as one of the main open source solutions.

Hadoop actually makes a decentralized operation processing framework called MapReduce. By distributing the data processing work to each node on the network, each node periodically reports the completed work and status updates. Come back, and then achieve a large-scale data operation analysis. Processing here Under the framework, the scheduling and assignment of work is preset to the FIFO (First In First Out) algorithm. Although the architecture is simple, it ignores the difference in the essential requirements of the computing work, which may cause a certain task to occupy resources for a long time. In addition, whether the adjustment of the system parameters can be consistent with the essential requirements of the computing work is another important factor in the Hadoop system, but if this condition is met, the user often needs to re-work for different operations. Set the overall system environment parameters so that the overall system performance and operation can match the needs of computing work. It can be seen that there are still many shortcomings in the above-mentioned methods, which are not a good designer, but need to be improved. How can a large number of group computers perform calculations at the same time, but they can optimize the computing work. The inventors of the present invention have improved and innovated in view of the shortcomings derived from the above-mentioned conventional methods, and after years of painstaking research. Finally, I successfully developed this system based on Hadoop multi-cluster environment.

The object of the present invention is to provide an apparatus and system, in particular, a distributed computer cluster applied to a plurality of large-scale data processing, which can select the best execution according to the characteristics of the execution program, the characteristics of the data to be processed, and the dynamic behavior of the computer cluster. surroundings. It can reduce the scheduling waiting time of work with different computing characteristics, effectively speed up the calculation and analysis, and improve the overall resource utilization efficiency.

A work distribution system and method based on the Hadoop multi-cluster environment that achieves the above objects, and utilizes a combination of a cluster feature and a monitoring module, a work data and a program analysis module, and an execution environment selection module to provide optimization. The Hadoop multi-cluster environment work dispatching system performs large data computing services for users. The method is to grasp the cluster characteristics and monitor the cluster operation. Shape, analysis of the operating data characteristics and program operation characteristics and other influencing parameters, and then the operation to find the most appropriate cluster, and then through the execution environment selection module to find the corresponding cluster, and the user work, including user program and input data is sent to The corresponding cluster is executed.

1‧‧‧Work assignment system

11‧‧‧Characteristic database module

12‧‧‧ Cluster Feature Module

13‧‧‧ Cluster Monitoring Module

14‧‧‧Work data analysis module

15‧‧‧Working program analysis module

16‧‧‧Execution Environment Selection Module

2‧‧‧User interface

3‧‧‧Client

4‧‧‧ Input data

5‧‧‧Mini Cluster

6‧‧‧ Host cluster

1 is a schematic diagram of a work distribution system based on a Hadoop multi-cluster environment of the present invention; FIG. 2 is a flowchart of operation of a work distribution system based on a Hadoop multi-cluster environment; FIG. 3 is a Hadoop multi-cluster environment according to the present invention; The execution environment selection flow chart of the work distribution system.

FIG. 1 is a schematic structural diagram of an implementation example of a work distribution system and method based on a Hadoop multi-cluster environment according to the present invention, including: a feature database module 11 for storing a cluster feature module 12 described later. , the cluster monitoring module 13, the work data analysis module 14, the work program analysis module 15 matrix equation; a cluster feature module 12, is used to collect static features that the cluster does not change over time, and clustered The static feature matrix equation is used to describe the collected static features; a cluster monitoring module 13 is used to periodically collect the dynamic features of each cluster and analyze the dynamic characteristic curves to establish a cluster dynamic feature matrix equation to describe the cluster feature analysis results. ; A work data analysis module 14 is configured to collect static features that do not change over time during work execution, and describe the collected static features by working static feature matrix equations; a work program analysis module 15, The method for analyzing the use of resources by the user program during execution is mainly used to establish a working dynamic characteristic matrix equation to describe the user program behavior feature; an execution environment selection module 16 is configured to analyze the module 15 and the cluster feature by the working program. The cluster equation established by the module 12 selects the cluster most suitable for the user's work and sends it to the corresponding cluster; the working process of the work distribution system based on the Hadoop multi-cluster environment of the present invention is shown in FIG. 2, and the customer works ( The work distribution system 1 including the client program 3 and the input data 4) is sent to the Hadoop multi-cluster environment through the user operation interface 2, and the work distribution system 1 finds the most suitable cluster by the customer work characteristics and the cluster 6 characteristics, and sends it to the cluster. This cluster is executed, and the description of each module in the work distribution system 1 is as follows.

First, the cluster monitoring module 13 periodically collects the dynamic characteristics of each cluster (such as CPU frequency (GHz), Disk space, Memory usage), and analyzes the dynamic characteristic curve to convert the analysis result into a cluster dynamic characteristic matrix equation. And stored in the feature database module 11. For example, periodically collect n dynamic features of N clusters (C_1...C_N), such as CPU usage (%) per second (%), usage (%) of Disk space, etc., and represent them in a matrix:

Each cluster takes a time interval ( t ₁ ~ t _k ), where k is the total number of intervals, and the average usage of each time interval is calculated and expressed in an n × k matrix:

Each cluster is stored in the feature database module 11 at a 1×n matrix of each time interval.

The cluster feature module 12 is mainly responsible for collecting static features that are not changed over time in the cluster, such as CPU core number, CPU frequency (GHz), Disk space size, Memory size, etc., and converting the collected data into clusters. The static feature matrix equation is stored in the feature database module 11, and the matrix equation of the i-th cluster is represented by a 1xn matrix:

E.g:

When a new cluster is added to the system, the cluster feature module 12 collects its static features and also stores them in the feature database module 11.

The work profile analysis module 14 collects static features that do not change over time during the execution of the work, such as the total amount of data, the total number of data, the format of the data format, whether it is compressed, etc., describing the static characteristics of the work. When a new job enters the work distribution system, the work data analysis module 14 collects its static features, and converts the collected data into a working static feature matrix equation stored in the feature database module 11, working static characteristic matrix equation Expressed in a 1xn matrix: [Js ₁ Js ₂ Js ₃ ... Js _n ] (6)

For example: [Total data] _1×n (7)

The work program analysis module 15 is used to analyze the situation in which the client 3 uses resources during execution, and is a sub-module in the work feature analysis module, which is mainly used to establish a matrix equation to describe the behavior characteristics of the client 3; After the user operation interface 2 submits the storage path of the client 3 and the input data 4, the work program analysis module 15 takes a fixed number of data from the input data 4 as a sample, and uploads the client 3 and the input data sample to the mini. Cluster 5, requires the Mini Cluster 5 startup program to start processing input data samples, and record the case where the client 3 uses the Mini Cluster 5 resource when processing a fixed number of data samples (eg, CPU, memory, file read and write requirements). , network packet read and write requirements) and time spent, and the collected data is converted into a working dynamic characteristic matrix equation stored in the feature database module 11, the working dynamic characteristic matrix equation is represented by a 1xn matrix, where Jd _n is the nth working dynamic characteristic parameter: [Jd ₁ Jd ₂ Jd ₃ ... Jd _n ] (8)

E.g:

The operation flow of the execution environment selection module 16 is as shown in FIG. 3 . First, the cluster monitoring module 13 , the cluster feature module 12 , the work data analysis module 14 and the work program analysis module 15 are obtained from the feature repository module 11 . Analysis cluster static characteristic matrix equation Cluster dynamic characteristic matrix equation The working static characteristic matrix equation [Js ₁ Js ₂ Js ₃ ... Js _n ] and the working dynamic characteristic matrix equation [Jd ₁ Jd ₂ Jd ₃ ... Jd _n ] and calculate the user program characteristic matrix equation and the cluster feature matrix corresponding to each cluster The equation is as shown in equations (10) and (11):

Where F ₁ ^job represents the first feature in the user program characteristic matrix equation, and the value is the result of multiplying the first term Js ₁ of the working static feature matrix equation with the first term Jd _{1 of the} working dynamic characteristic matrix equation, and so on. , there are n eigenvalues, and Then it is the first feature of the cluster feature matrix equation representing the i-th cluster, and there are also n eigenvalues. Since the value of the cluster dynamic feature matrix equation is the average usage of the cluster at the time, we need the remainder of the cluster. Usage rate, so take (1- Calculate the remaining usage of the cluster, the value of which is the first term of the cluster static characteristic matrix equation Residual usage rate with the first cluster dynamic feature matrix (1- Multiply, here is cluster 1 as an example, and the cluster characteristic matrix equation is In order to avoid confusion, we denote the user program characteristic matrix equation with J and Ci denote the cluster feature matrix equation of the i-th cluster. As shown in Figure 3, the second step is to perform clustering action on Ci. First, filter the clusters that are not suitable. Since some user program features have a lower limit value, if the feature value corresponding to the cluster is lower than the lower limit value, the user program cannot be executed on the cluster. For example, the user program feature has disk usage, if the disk in the cluster feature This cluster is not suitable for executing this user program when the remaining amount is lower than the disk usage required by the user program. In the discriminating unsuitable cluster, the user program characteristic matrix equation and the cluster feature matrix equation can be compared. If the element belongs to the feature with the lower limit value, and the cluster feature matrix equation element is smaller than the user program characteristic matrix equation, the cluster feature matrix is represented. the equation for the corresponding cluster is not currently executed user program, so it is not suitable for the set of cluster _cluster unsuitable as a first representation (12) represented by the formula:

Where Cluster _all represents a set of all cluster feature matrix equations, and L represents a feature set with a lower limit value. The jth element of the cluster characteristic matrix equation representing the C ⁱ cluster, and F _j ^job is the jth element of the user program characteristic matrix equation. After filtering out the unsuitable cluster, it is divided into the highest priority for the remaining cluster characteristic equations. Cluster feature matrix set and sub-priority cluster feature matrix set, first highest priority cluster feature matrix equation set here is defined as cluster feature matrix equations, each feature element satisfies all elements of the user program characteristic equation, and the remaining cluster feature matrix equation This is the set of priority cluster feature matrix equations, which are defined as follows:

Cluster _{Second priorty} =Cluster _All -(Cluster _{First priorty} U Cluster _Unsuitable ) (14)

Cluster _{first priorty} is the highest priority cluster feature matrix equation set and Cluster _{second priorty} is the sub-priority cluster feature matrix equation set. After the cluster feature matrix equations are grouped, the next step is to select the target cluster, and the target cluster can be divided into the following steps:

A. Check the set of the most preferred cluster feature matrix equations. If it is not an empty set, select the most suitable cluster feature matrix equation from the set. Here, the user program feature matrix equation can be regarded as a vector existing in n-dimensional space, and the highest priority cluster. The set of characteristic matrix equations can also be regarded as multiple sets of vectors existing in n space, so the distance of the vector is used as the basis for selection; generally speaking, the larger the distance, the more resources the cluster will have for the user to execute, but The purpose of the present invention is to reduce the wait to run time. In order to avoid a large number of user programs being configured to a specific cluster, the execution efficiency is lowered, so that the vector distance is selected closest to the user at that time. The cluster of program execution, as shown in (15): Where dist(C ⁱ , J) is the vector distance between the cluster feature matrix equation and the user program feature matrix equation. The algorithm is as follows:

B. If there is no cluster characteristic equation in the set of the most preferential cluster feature matrix, then the selection is made from the set of sub-priority cluster feature matrix, in which the cluster set of the user program cannot be fully satisfied but can be successfully completed. The user's work needs, the selection method here is the same as the first step, and the matrix equations are regarded as the vectors existing in the n-dimensional space. In order to avoid the user program execution time is too long due to the difference between the cluster features and the user program features, this is too long. The same is to choose the minimum distance between the two vector spaces as the basis for selection, as shown in (17):

C. If there is no cluster characteristic equation in the set of the highest priority and the second priority cluster feature matrix, it means that all the existing clusters are not suitable for performing user work. At this time, the execution environment selection module returns the user work request and notifies the use. By.

If the most suitable cluster feature matrix equation is found, the execution environment selection module 16 finds the corresponding cluster from the selected cluster feature matrix equation, and sends the client work, including the client 3 and the input data 4, to the corresponding cluster to execute. .

The system of work assignment based on Hadoop multi-cluster environment provided by the invention has the following advantages when compared with other conventional technologies:

1. The invention can provide an optimized execution environment to the user according to the characteristics of the data to be processed, the characteristics of the computing program and the dynamic behavior of the computer cluster, effectively reducing the waiting time of the work, and providing a feasible, reliable and efficient computing service.

2. The system and method for the work assignment of the present invention can fully utilize the hardware resources of the computing device according to the characteristics of the data to be processed, reduce the cost of computing service construction, ensure the stability and reliability of the service, and solve the difference in the essential requirements of the computing work. The problem, and thus the overall service speed and efficiency, its economic benefits are very obvious.

The detailed description of the preferred embodiments of the present invention is intended to be limited to the scope of the invention, and is not intended to limit the scope of the invention. The patent scope of this case.

In summary, this case is not only innovative in terms of space type, but also can enhance the above-mentioned multiple functions compared with the customary items, and should fully comply with the novelty and progress. The statutory invention patent requirements of the step, apply in accordance with the law, and ask your office to approve the invention patent application, in order to invent invention, to the sense of virtue.

1‧‧‧Work assignment system

11‧‧‧Characteristic database module

12‧‧‧ Cluster Feature Module

13‧‧‧ Cluster Monitoring Module

14‧‧‧Work data analysis module

15‧‧‧Working program analysis module

16‧‧‧Execution Environment Selection Module

2‧‧‧User interface

3‧‧‧Client

4‧‧‧ Input data

5‧‧‧Mini Cluster

6‧‧‧ Host cluster

Claims (7)

A system for job assignment based on Hadoop multi-cluster environment includes: a feature database module for storing static and dynamic feature matrix equations of clusters and working static and dynamic feature matrix equations; a cluster feature module, It is mainly responsible for analyzing the static characteristics of each cluster; a cluster monitoring module is mainly responsible for analyzing the dynamic characteristics of each cluster; a working data analysis module is mainly responsible for analyzing the static characteristics of the computing work; a working program analysis module is used for Analyze the situation in which the user program uses resources during execution; and an execution environment selection module is used to select the cluster that is most suitable for the user's work and send it to the corresponding cluster for execution. According to the Hadoop multi-cluster environment-based work dispatching system described in claim 1, the cluster monitoring module periodically collects dynamic features of each cluster, analyzes the dynamic characteristic curves, and converts the analysis results into a cluster dynamic feature matrix. The equation is stored in the feature database module. The system of work assignment based on Hadoop multi-cluster environment according to claim 1, wherein the cluster feature module is mainly responsible for analyzing static features that do not change over time in the cluster, and establishing a matrix equation to describe static features of the cluster; When a new cluster is added to the system, the cluster feature module analyzes its static features and converts the data into a matrix equation stored in the feature database module. The system of work assignment based on the Hadoop multi-cluster environment described in claim 1, wherein the work data analysis module is mainly responsible for analyzing data characteristics and static features in the execution of the calculation work, and establishing a matrix equation to describe the work. The static feature; when a new job enters the work distribution system, the data analysis module analyzes its static features and converts the data into a matrix equation stored in the feature database module. The system for scheduling work based on a Hadoop multi-cluster environment according to claim 1, wherein the worker program analysis module is configured to analyze a situation in which a client uses resources when processing data, and spends time, and converts the collected data into The working dynamic characteristic matrix equation is stored in the feature database module. The system for scheduling work based on a Hadoop multi-cluster environment according to claim 1, wherein the execution environment selection module obtains a cluster monitoring module, a cluster feature module, a work data analysis module, and a work program from a feature database module. Analyze the results of the module analysis and use the user program feature matrix equation to work with the user, including the user program and input data to the corresponding cluster. A work assignment method based on a Hadoop multi-cluster environment includes the following steps: obtaining a result of a cluster monitoring module, a cluster feature module, a work data analysis module, and a work program analysis module from a feature database module; calculating a user program feature The matrix equation and the cluster feature matrix equation corresponding to each cluster; the cluster feature matrix equations corresponding to each cluster are classified into the highest priority cluster feature matrix equation set, the second priority cluster feature matrix equation set and the unsuitable cluster feature matrix through the user program characteristic matrix equation A set of equations; wherein the user program characteristic matrix equation and each cluster feature matrix equation are compared, and if the element belongs to a feature having a lower limit value, and the cluster feature matrix equation element is smaller than the user program feature matrix equation, the cluster feature matrix is classified as unsuitable Equation; filter out discomfort After the cluster characteristic matrix equation is combined, all the elements of the cluster characteristic matrix equation satisfy the user program characteristic equation and all the elements are classified into the highest priority cluster feature matrix equation set, and the remaining cluster feature matrix equation classifies the sub-priority cluster feature matrix equation set; If the set of the most preferential cluster feature matrix is not an empty set, the most suitable cluster feature matrix equation is selected according to the user program feature matrix equation from the highest priority cluster feature matrix equation set; if the most preferential cluster feature matrix equation set is an empty set, then check Whether the set of sub-priority cluster feature matrix is an empty set, if not an empty set, then select a suitable cluster feature matrix equation; calculate the corresponding cluster by the selected cluster feature matrix equation, and work the user, including user program and input The data is sent to the corresponding cluster for execution; if the highest priority and the second priority cluster feature matrix are all empty sets, it means that all existing clusters are not suitable for performing user work. User job requirements, and notify the user.