CN111176840A - Distributed task allocation optimization method and device, storage medium and electronic device - Google Patents
Distributed task allocation optimization method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN111176840A CN111176840A CN201911330857.9A CN201911330857A CN111176840A CN 111176840 A CN111176840 A CN 111176840A CN 201911330857 A CN201911330857 A CN 201911330857A CN 111176840 A CN111176840 A CN 111176840A
- Authority
- CN
- China
- Prior art keywords
- node
- task
- tasks
- distributed system
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
Abstract
The invention provides a distributed task allocation optimization method and a distributed task allocation optimization device, wherein the method comprises the following steps: determining a second node set in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein each node in the second node set is a node allowing execution of a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set, wherein the degree of association parameter is used for indicating the degree of association between the current task to be distributed and the task received by each node in the second node set; and distributing the current task to be distributed to the target node with the highest association degree in the second node set for execution.
Description
Technical Field
The invention relates to the field of computers, in particular to a distributed task allocation optimization method and device, a storage medium and an electronic device.
Background
In the related technology, when a scheduling system of a distributed system performs node scheduling, the resource load conditions of all nodes are mainly calculated through a load balancing algorithm, then the nodes with low load utilization rate are preferentially added into a scheduling queue, and tasks are randomly distributed to the nodes added into the queue.
In the node allocation mode in the related art, for nodes added into the scheduling queue, the problem of unequal high and low task execution efficiency still exists due to different load utilization rates, and if the random allocation mode is simply adopted, the task execution effect is good or bad, and the optimization of the allocation optimization of the distributed tasks cannot be realized.
Aiming at the problems that in the related art, when a scheduling system of a distributed system performs node scheduling, due to the difference of load utilization rates of nodes, distribution optimization of distributed tasks is still unreasonable and the like, an effective technical scheme is not provided.
Disclosure of Invention
The embodiment of the invention provides a distributed task allocation optimization method and a distributed task allocation optimization device, which are used for at least solving the problems that in the related art, when a scheduling system of a distributed system performs node scheduling, due to the difference of load utilization rates of nodes, the distributed task allocation optimization is still unreasonable and the like.
According to an embodiment of the present invention, there is provided a distributed task allocation optimization method, including: determining a second node set in a first node set formed by all nodes of the distributed system according to state information of each node in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set; the association degree parameter is used for indicating the association degree between the current task to be distributed and the task received by each node in the second node set; and optimizing the distribution of the current distributed tasks to be distributed to the target node with the highest association degree in the second node set for execution.
In this embodiment of the present invention, the state information of each node in the distributed system includes: the number of uncompleted tasks, the average load parameter within a first predetermined time period, and the estimated completion time of the uncompleted tasks; determining a second node set in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein the determining comprises the following steps: acquiring state information of each node in the distributed system; determining a performance evaluation value of each node in the distributed system according to the state information; determining the second set of nodes in the first set of nodes in the distributed system according to the performance evaluation value.
In an embodiment of the present invention, the determining, according to the performance evaluation value, the second node set in the first node set in the distributed system includes: comparing the performance evaluation value of each node in the distributed system with a preset threshold value; and determining a set formed by nodes of which the performance evaluation values are smaller than the preset threshold value as the second node set.
In an embodiment of the present invention, the determining a performance evaluation value of each node in the distributed system includes: normalizing the number of the uncompleted tasks to obtain the normalized number of the uncompleted tasks; determining a first product of a first weight and the number of uncompleted tasks after the normalization, determining a second product of a second weight and the average load parameter, and determining a third product of a third weight and the estimated completion time; wherein the first weight is greater than the second weight, which is greater than the third weight; determining a sum of the first product, the second product, and the third product as the performance evaluation value.
In this embodiment of the present invention, the determining a parameter of a degree of association between a current task to be allocated and each node in the second node set includes: acquiring the maximum task number of each node in the second node set within second preset time, determining the maximum task number as a first task number, acquiring the average value of the task numbers of each node in the second node set within the second preset time, determining the average value as a second task number, acquiring the task number included in a task set to be allocated in the distributed system, and determining the task number included in the task set to be allocated as a third task number; the task set to be allocated comprises the current task to be allocated; determining a ratio of the third number of tasks to the second number of tasks; determining a covariance value between the ratio and the first task number, and determining the covariance value as the association degree parameter.
In this embodiment of the present invention, allocating the task to be currently allocated to the target node with the highest degree of association in the second node set, and performing the task includes: arranging each node in the second node set according to a first preset sequence according to the magnitude of the association degree parameter to obtain a target node set arranged in the first preset sequence; determining a node corresponding to the maximum value of the association degree parameter in the target node set as the target node; and distributing the current task to be distributed to the target node for execution.
In the embodiment of the present invention, each task to be allocated in the task sets to be allocated in the distributed system is arranged in a task pool in the distributed system according to a second preset sequence, the current task to be allocated is any one task to be allocated in the task sets to be allocated, and the task pool is used for storing the task sets to be allocated in the distributed system.
According to another embodiment of the present invention, there is also provided a distributed task allocation optimizing apparatus including: a first determining module, configured to determine a second node set in a first node set composed of all nodes in the distributed system according to state information of each node in the distributed system, where each node in the second node set is a node that is allowed to execute a task to be allocated in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; a second determining module, configured to determine a degree of association parameter between a current task to be allocated and each node in the second node set, where the degree of association parameter is used to indicate a degree of association between the current task to be allocated and a task that has been received by each node in the second node set; and the distribution module is used for distributing the current task to be distributed to the target node with the highest correlation degree in the second node set for execution.
There is also provided, in accordance with another embodiment of the present invention, a computer-readable storage medium including a stored program, wherein the program when executed performs the method for optimizing distribution of distributed tasks as set forth in any one of the above.
There is also provided, in accordance with another embodiment of the present invention, an electronic apparatus, the storage medium including a stored program, wherein the program executes, when running, the method for optimizing distribution of distributed tasks according to any one of the above.
According to the invention, a second node set is determined in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set; the association degree parameter is used for indicating the association degree between the current task to be distributed and the task received by each node in the second node set; and optimizing the distribution of the current distributed tasks to be distributed to the target node with the highest association degree in the second node set for execution. By adopting the technical scheme, the problems that in the related art, when the dispatching system of the distributed system carries out node dispatching, the distributed tasks are still unreasonable in distribution optimization due to different load utilization rates of the nodes and the like are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a hardware structure block diagram of a distributed task allocation optimization method according to an embodiment of the present invention;
FIG. 2 is a flow chart of an alternative method for optimizing the distribution of distributed tasks according to an embodiment of the present invention;
FIG. 3 is a flow diagram of an alternative method for optimizing the distribution of distributed tasks according to an embodiment of the invention;
FIG. 4 is a schematic diagram of an alternative distributed system according to embodiments of the present invention;
FIG. 5 is a block diagram of an alternative method for optimizing the distribution of distributed tasks, according to an embodiment of the present invention;
fig. 6 is a block diagram of another alternative first determination module according to an embodiment of the invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method provided by the embodiment of the application can be applied to a distributed system for execution. Fig. 1 is a block diagram of a hardware configuration of a distributed task allocation optimization apparatus according to an embodiment of the present invention. As shown in fig. 1, the distributed task allocation optimization apparatus 10 may include one or more processors 102 (only one is shown in fig. 1) (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data, and optionally, the intelligent door lock may further include a transmission device 106 for communication function and an input/output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the above-described distributed task allocation optimizing device. For example, the distributed task allocation optimization apparatus 10 may also include more or fewer components than shown in FIG. 1, or have a different configuration with equivalent functionality to that shown in FIG. 1 or with more functionality than that shown in FIG. 1.
The memory 104 may be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to the distributed task allocation optimization method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a Network adapter (NIC), which can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In this embodiment, a method for optimizing distribution of distributed tasks running on the intelligent door lock is provided, and fig. 2 is a flowchart of an optional method for optimizing distribution of distributed tasks according to an embodiment of the present invention, as shown in fig. 2, the method includes the following steps:
step S202, according to the state information of each node in the distributed system, determining a second node set in a first node set formed by all nodes in the distributed system, wherein each node in the second node set is a node allowing execution of a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
step S204, determining a degree of association parameter between the current task to be distributed and each node in the second node set, wherein the degree of association parameter is used for indicating the degree of association between the current task to be distributed and the task received by each node in the second node set;
step S206, the task to be currently allocated is allocated to the target node with the highest association degree in the second node set for execution.
According to the invention, a second node set is determined in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set; the association degree parameter is used for indicating the association degree between the current task to be distributed and the task received by each node in the second node set; and optimizing the distribution of the current distributed tasks to be distributed to the target node with the highest association degree in the second node set for execution. By adopting the technical scheme, the problems that in the related art, when the dispatching system of the distributed system carries out node dispatching, the distributed tasks are still unreasonable in distribution optimization due to different load utilization rates of the nodes and the like are solved.
In step S202, the state information of each node in the distributed system includes: the number of uncompleted tasks, the average load parameter within a first predetermined time period, and the estimated completion time of the uncompleted tasks; determining a second node set in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein the determining comprises the following steps: acquiring state information of each node in the distributed system; determining a performance evaluation value of each node in the distributed system according to the state information; determining the second set of nodes in the first set of nodes in the distributed system according to the performance evaluation value.
Optionally, the determining, according to the performance evaluation value, the second node set in the first node set in the distributed system includes: comparing the performance evaluation value of each node in the distributed system with a preset threshold value; and determining a set formed by nodes of which the performance evaluation values are smaller than the preset threshold value as the second node set.
There are various implementations of determining the performance evaluation value, and in an alternative embodiment, the determining the performance evaluation value of each node in the distributed system includes: normalizing the number of the uncompleted tasks to obtain the normalized number of the uncompleted tasks; determining a first product of a first weight and the number of uncompleted tasks after the normalization, determining a second product of a second weight and the average load parameter, and determining a third product of a third weight and the estimated completion time; wherein the first weight is greater than the second weight, which is greater than the third weight; determining a sum of the first product, the second product, and the third product as the performance evaluation value.
In optional embodiments, the determining a parameter of a degree of association between the current task to be distributed and each node in the second node set includes: acquiring the maximum task number of each node in the second node set within second preset time, determining the maximum task number as a first task number, acquiring the average value of the task numbers of each node in the second node set within the second preset time, determining the average value as a second task number, acquiring the task number included in a task set to be allocated in the distributed system, and determining the task number included in the task set to be allocated as a third task number; the task set to be allocated comprises the current task to be allocated; determining a ratio of the third number of tasks to the second number of tasks; determining a covariance value between the ratio and the first task number, and determining the covariance value as the association degree parameter.
In this embodiment of the present invention, allocating the current task to be allocated to the target node with the highest association degree in the second node set for execution includes: arranging each node in the second node set according to a first preset sequence according to the magnitude of the association degree parameter to obtain a target node set arranged in the first preset sequence; determining a node corresponding to the maximum value of the association degree parameter in the target node set as the target node; and distributing the current task to be distributed to the target node for execution.
Optionally, each task to be allocated in the task set to be allocated in the distributed system is arranged in a task pool in the distributed system according to a second preset sequence, the current task to be allocated is any one task to be allocated in the task set to be allocated, and the task pool is used for storing the task set to be allocated in the distributed system.
The following describes a flow of a task allocation method with reference to an alternative example, as shown in fig. 3, the method includes the following steps:
step S302, acquiring real-time task execution data of each node in the system;
when a new task in the system task pool is distributed, considering the load balance condition of the nodes to be distributed, the task execution conditions of all the nodes are analyzed, the task execution conditions of all the nodes are comprehensively considered, and the task is dynamically distributed according to the actual conditions of all the nodes. The phenomenon that when a node has too many tasks which are not finished at present, the node load is too heavy is avoided.
As shown in fig. 4, an average load parameter L of each node in a first node set composed of all nodes in the distributed system may be obtained, where the average load parameter L is an average number of processes in a first preset time period, and the distributed system can preferentially allocate tasks to nodes with low average load; the time parameter index of each node in the distributed system can be obtained, for example, the estimated completion time sum of the uncompleted task of each node is obtained, the estimated completion time sum of the uncompleted task can accurately reflect the capability of the node to execute the new task, if the time for the node to process the uncompleted task is long, the node is not allocated with the new task, and the parameter index can be determined based on the uncompleted task data, the CPU usage data and the content usage data of the node through comprehensive calculation; the number M of uncompleted tasks of each node in all nodes in the distributed system can be obtained.
Step S304, calculating the comprehensive performance index of the node based on the number M of unfinished tasks of the node, the average load parameter L of the node and the estimated finish time sum T of unfinished tasks of the node;
in an alternative embodiment, the overall performance index of the node (corresponding to the performance assessment value described above) is represented by Q, and the threshold Q for the schedulable task is determined at the same timei(corresponding to the above-mentioned preset threshold), only if the Q value is less than a defined threshold QiAs shown in FIG. 4, the Q value is made smaller than a defined threshold QiAnd adding the corresponding node into a scheduling queue (corresponding to the second node set), wherein the node comprehensive performance index exceeds a threshold value, the average load of the identified node is higher, the number of uncompleted tasks is larger, and the estimated completion time sum of the uncompleted tasks of the node is larger.
By the method and the device, the calculation parameters of the sum of the estimated completion time of the uncompleted tasks of the nodes are increased, and the actual task execution capacity of the nodes is comprehensively considered. The nodes in the distributed system have different CPU frequencies, memory sizes and the like, the processing capacities of different nodes for the same task are different, the actual task execution capacity of the nodes is comprehensively considered, the dynamic task allocation can be carried out according to the actual condition of each node, the phenomenon that when the number of tasks which are not completed by one node at present is too large, the node load is too heavy is avoided, and the task allocation reasonability in the distributed system is improved.
Optionally, the following is a performance evaluation value calculation process, which is specifically described as follows:
Q＝M′k1+Lk2+Tk3(1)
wherein, M' is log (M), which is a result obtained after normalization processing is performed on the number M of uncompleted tasks;
m is the number of unfinished tasks of the node;
t is the sum of estimated completion time of uncompleted tasks;
l is an average load parameter of each node in the distributed system in a first preset time period, and only the nodes with low average load rate are assigned with tasks in limited consideration;
k1、k2、k3weight values set according to the degree of importance of the index, which should satisfy k1＞k2＞k3. Wherein k is1Corresponding to the first weight, k1Corresponding to the second weight, k1Corresponding to the third weight described above.
Step S306, based on the comprehensive performance indexes of all the nodes calculated in step S304, the set schedulable task threshold value Q is passed throughiComparing, determining nodes capable of distributing tasks, and adding the screened nodes into a scheduling queue (corresponding to the second node set);
in an alternative embodiment, the task threshold Q may be schedulediCan be set according to the computing capability of the computing equipment of the nodes in the system when the task threshold value Q can be schedulediToo high a definition may result in some idle nodesScheduling cannot be obtained; when schedulable task threshold QiWhen the definition is too low, the nodes receive too many tasks, and the load among the nodes is unbalanced. The node number and the task number of the system need to be comprehensively determined according to the ratio of the node number and the task number of the system.
Step S308, carrying out secondary optimization on the nodes added into the scheduling queue, calculating the association degree index value between the nodes in the queue and the tasks to be distributed, grouping according to the tasks to be distributed, determining a node set corresponding to each task to be distributed, and determining the optimal execution node (corresponding to the target node) of the task to be distributed based on the determined node set;
by executing step S306, the nodes that can perform task allocation are added to the scheduling queue, but the nodes included in the scheduling queue are not necessarily the optimal nodes for executing the tasks to be allocated.
In an optional embodiment, the process of determining the optimal execution node specifically includes the following steps:
step 1, determining a historical load parameter X (corresponding to the first task quantity) of each node in a scheduling queue;
the historical load parameter can be the maximum value of the task allocation quantity of the node in a historical period of time;
X＝max(Xi) (2)
step 2, calculating the ratio Y of the task to be distributed to the historical distribution task quantity average value of the node;
calculating the ratio of the task quantity of the tasks to be distributed to the historical distribution task quantity average value of the nodes (corresponding to the ratio of the third task quantity to the second task quantity);
step 3, calculating a covariance value between the historical load parameter X and the ratio Y (corresponding to a covariance value between the ratio and the first task quantity);
the covariance is defined as:
COV(X，Y)＝E[(X-E(X))(Y-E(Y))](3)
the equivalent calculation formula is: COV (X, Y) ═ E (X-Y) -E (X) E (Y).
The covariance value between each task to be allocated and the node in the scheduling queue can be obtained through the formula (3), and the covariance value can reflect the correlation degree between the historical allocation task amount of the node and the task amount of the task to be allocated.
Step 4, grouping the covariance values obtained by calculation by taking the tasks to be distributed as units; obtaining a set of covariance values, for example, Vc ═ { C1, C2, C3, … … }, where C1, C2, and C3 are covariance values between each node in the scheduling queue and the task to be allocated;
then, in each combination, respectively selecting nodes corresponding to S values with the covariance values at the front to form node combinations Vj; for example, Vj ═ N1, N2, N3, … …, Ns }.
Assuming that S is 10 and the number of tasks to be distributed is 10, the node combination formed by each task to be distributed is 1x10 matrix size; the 10 tasks to be assigned may be formed as a matrix of 10 x 10.
5. Selecting the node with the smallest sequencing j in the combination Vj as the optimal execution node of the task to be distributed;
in this way, each task to be distributed uniquely determines an optimal node to which the corresponding task to be distributed is distributed.
Step S310, the scheduling module allocates tasks to the optimal nodes determined in the step S308 and deletes the allocated tasks from the task pool;
when the scheduling module allocates a certain task to be allocated to the determined optimal node from the task pool for execution, the task pool can automatically delete the task, and the task is prevented from being repeatedly allocated.
According to the embodiment, by introducing the time parameter index, the estimated completion time sum of the uncompleted tasks of the node can be calculated, the capability of the node for executing the new task can be accurately reflected, if the time for the node to process the uncompleted tasks is long, the new task is not distributed to the node, the association degree parameter of the node and the task to be distributed can be determined through covariance value calculation, the association degree of the node and the task to be distributed is improved, and the reasonability of the task to be distributed in a distributed system is improved.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The present embodiment further provides a distributed task allocation optimization apparatus, which is applied to a distributed system, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and the description of the apparatus is omitted for brevity. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 5 is a block diagram of an alternative apparatus for optimizing distribution of distributed tasks according to an embodiment of the present invention, as shown in fig. 5, the apparatus includes:
a first determining module 502, configured to determine, according to state information of each node in the distributed system, a second node set from the first node set composed of all nodes in the distributed system, where each node in the second node set is a node that is allowed to execute a task to be allocated in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
a second determining module 504, configured to determine a degree of association parameter between a current task to be allocated and each node in the second node set, where the degree of association parameter is used to indicate a degree of association between the current task to be allocated and a task that has been received by each node in the second node set;
and an allocating module 506, configured to allocate the current task to be allocated to a target node with the highest association degree in the second node set for execution.
According to the invention, a second node set is determined in a first node set formed by all nodes of the distributed system according to the state information of each node in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set; determining a degree of association parameter between the current task to be distributed and each node in the second node set; the association degree parameter is used for indicating the association degree between the current task to be distributed and the task received by each node in the second node set; and optimizing the distribution of the current distributed tasks to be distributed to the target node with the highest association degree in the second node set for execution. By adopting the technical scheme, the problems that in the related art, when the dispatching system of the distributed system carries out node dispatching, the distributed tasks are still unreasonable in distribution optimization due to different load utilization rates of the nodes and the like are solved.
In the embodiment of the present invention, as shown in fig. 6, the state information of each node in the distributed system includes: the number of uncompleted tasks, the average load parameter within a first predetermined time period, and the estimated completion time of the uncompleted tasks; the first determining module 502 includes: an obtaining unit 5021, configured to obtain state information of each node in the distributed system; a determining unit 5022, configured to determine a performance evaluation value of each node in the distributed system according to the state information; the determining unit is further configured to determine the second node set from the first node set in the distributed system according to the performance evaluation value.
In this embodiment of the present invention, the determining unit 5022 is further configured to compare the performance evaluation value of each node in the distributed system with a preset threshold; and determining a set formed by nodes of which the performance evaluation values are smaller than the preset threshold value as the second node set.
In this embodiment of the present invention, the determining unit 5022 is further configured to normalize the number of the uncompleted tasks, so as to obtain the number of the uncompleted tasks after normalization; determining a first product of a first weight and the number of uncompleted tasks after the normalization, determining a second product of a second weight and the average load parameter, and determining a third product of a third weight and the estimated completion time; wherein the first weight is greater than the second weight, which is greater than the third weight; determining a sum of the first product, the second product, and the third product as the performance evaluation value.
In this embodiment of the present invention, the second determining module 504 is further configured to obtain a maximum task number of tasks allocated to each node in the second node set within a second predetermined time, determine the maximum task number as a first task number, obtain a mean value of the task numbers of tasks allocated to each node in the second node set within the second predetermined time, determine the mean value as a second task number, obtain a task number included in a task set to be allocated in the distributed system, and determine the task number included in the task set to be allocated as a third task number; the task set to be allocated comprises the current task to be allocated; determining a ratio of the third number of tasks to the second number of tasks; determining a covariance value between the ratio and the first task number, and determining the covariance value as the association degree parameter.
In this embodiment of the present invention, the allocating module 506 is further configured to arrange each node in the second node set according to a first preset order according to the magnitude of the association degree parameter, so as to obtain a target node set arranged in the first preset order; determining a node corresponding to the maximum value of the association degree parameter in the target node set as the target node; and distributing the current task to be distributed to the target node for execution.
In the embodiment of the invention, each task to be allocated in the task sets to be allocated in the distributed system is arranged in the task pool in the distributed system according to a second preset sequence, the current task to be allocated is any one task to be allocated in the task sets to be allocated, and the task pool is used for storing the task sets to be allocated in the distributed system.
Embodiments of the present invention also provide a computer-readable storage medium including a stored program, wherein the program performs any one of the methods described above when executed.
Alternatively, in the present embodiment, the storage medium may be configured to store program codes for performing the following steps:
s1, according to the state information of each node in the distributed system, determining a second node set in a first node set formed by all nodes in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
s2, determining a degree of association parameter between the current task to be distributed and each node in the second node set, wherein the degree of association parameter is used for indicating the degree of association between the current task to be distributed and the task received by each node in the second node set;
and S3, distributing the current task to be distributed to the target node with the highest correlation degree in the second node set for execution.
Optionally, in this embodiment, the storage medium may include, but is not limited to: various media capable of storing program codes, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, according to the state information of each node in the distributed system, determining a second node set in a first node set formed by all nodes in the distributed system, wherein each node in the second node set is a node which is allowed to execute a task to be distributed in the distributed system, and the number of nodes included in the second node set is less than or equal to the number of nodes included in the first node set;
s2, determining a degree of association parameter between the current task to be distributed and each node in the second node set, wherein the degree of association parameter is used for indicating the degree of association between the current task to be distributed and the task received by each node in the second node set;
and S3, distributing the current task to be distributed to the target node with the highest correlation degree in the second node set for execution.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911330857.9A CN111176840B (en) | 2019-12-20 | 2019-12-20 | Distributed task allocation optimization method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911330857.9A CN111176840B (en) | 2019-12-20 | 2019-12-20 | Distributed task allocation optimization method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111176840A true CN111176840A (en) | 2020-05-19 |
| CN111176840B CN111176840B (en) | 2023-11-28 |
Family
ID=70655616
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911330857.9A Active CN111176840B (en) | 2019-12-20 | 2019-12-20 | Distributed task allocation optimization method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111176840B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111737751A (en) * | 2020-07-17 | 2020-10-02 | 支付宝(杭州)信息技术有限公司 | Method and device for realizing privacy-protected distributed data processing |
| CN112162846A (en) * | 2020-11-27 | 2021-01-01 | 腾讯科技（深圳）有限公司 | Transaction processing method, device and computer readable storage medium |
| CN112948106A (en) * | 2020-09-07 | 2021-06-11 | 深圳市明源云科技有限公司 | Task allocation method and device |
| CN114900518A (en) * | 2022-04-02 | 2022-08-12 | 中国光大银行股份有限公司 | Task allocation method, device, medium and electronic equipment for directed distributed network |
| CN115665364A (en) * | 2022-10-17 | 2023-01-31 | 上海理想信息产业（集团）有限公司 | Video conferencing method, device, system and equipment for cloud-edge-device collaborative computing |
| CN117591039A (en) * | 2024-01-18 | 2024-02-23 | 济南浪潮数据技术有限公司 | Distributed storage methods, systems, equipment and media |
| CN118982125A (en) * | 2024-07-22 | 2024-11-19 | 北京科杰科技有限公司 | A task management method based on time estimation |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120084400A1 (en) * | 2010-07-23 | 2012-04-05 | Almadi Soloman M | Machines, Computer Program Products, and Computer-Implemented Methods Providing an Integrated Node for Data Acquisition and Control |
| CN102831012A (en) * | 2011-06-16 | 2012-12-19 | 日立(中国)研究开发有限公司 | Task scheduling device and task scheduling method in multimode distributive system |
| CN107155197A (en) * | 2017-05-31 | 2017-09-12 | 北京邮电大学 | Distributed storage method, device and the electronic equipment cooperated based on multi-hop |
| WO2019170011A1 (en) * | 2018-03-07 | 2019-09-12 | 杭州海康威视系统技术有限公司 | Task allocation method and device, and distributed storage system |
| CN110287245A (en) * | 2019-05-15 | 2019-09-27 | 北方工业大学 | Method and system for scheduling and executing distributed ETL (extract transform load) tasks |
| US20190324819A1 (en) * | 2016-12-30 | 2019-10-24 | Huawei Technologies Co., Ltd. | Distributed-system task assignment method and apparatus |
| CN110390345A (en) * | 2018-04-20 | 2019-10-29 | 复旦大学 | A method for adaptive resource scheduling of big data clusters based on cloud platform |
| CN110428124A (en) * | 2019-06-10 | 2019-11-08 | 平安科技（深圳）有限公司 | Method for allocating tasks, task allocation apparatus, storage medium and computer equipment |
-
2019
- 2019-12-20 CN CN201911330857.9A patent/CN111176840B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120084400A1 (en) * | 2010-07-23 | 2012-04-05 | Almadi Soloman M | Machines, Computer Program Products, and Computer-Implemented Methods Providing an Integrated Node for Data Acquisition and Control |
| CN102831012A (en) * | 2011-06-16 | 2012-12-19 | 日立(中国)研究开发有限公司 | Task scheduling device and task scheduling method in multimode distributive system |
| US20190324819A1 (en) * | 2016-12-30 | 2019-10-24 | Huawei Technologies Co., Ltd. | Distributed-system task assignment method and apparatus |
| CN107155197A (en) * | 2017-05-31 | 2017-09-12 | 北京邮电大学 | Distributed storage method, device and the electronic equipment cooperated based on multi-hop |
| WO2019170011A1 (en) * | 2018-03-07 | 2019-09-12 | 杭州海康威视系统技术有限公司 | Task allocation method and device, and distributed storage system |
| CN110390345A (en) * | 2018-04-20 | 2019-10-29 | 复旦大学 | A method for adaptive resource scheduling of big data clusters based on cloud platform |
| CN110287245A (en) * | 2019-05-15 | 2019-09-27 | 北方工业大学 | Method and system for scheduling and executing distributed ETL (extract transform load) tasks |
| CN110428124A (en) * | 2019-06-10 | 2019-11-08 | 平安科技（深圳）有限公司 | Method for allocating tasks, task allocation apparatus, storage medium and computer equipment |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111737751A (en) * | 2020-07-17 | 2020-10-02 | 支付宝(杭州)信息技术有限公司 | Method and device for realizing privacy-protected distributed data processing |
| CN112948106A (en) * | 2020-09-07 | 2021-06-11 | 深圳市明源云科技有限公司 | Task allocation method and device |
| CN112948106B (en) * | 2020-09-07 | 2024-05-31 | 深圳市明源云科技有限公司 | Task allocation method and device |
| CN112162846A (en) * | 2020-11-27 | 2021-01-01 | 腾讯科技（深圳）有限公司 | Transaction processing method, device and computer readable storage medium |
| CN112162846B (en) * | 2020-11-27 | 2021-04-09 | 腾讯科技（深圳）有限公司 | Transaction processing method, device and computer readable storage medium |
| CN114900518A (en) * | 2022-04-02 | 2022-08-12 | 中国光大银行股份有限公司 | Task allocation method, device, medium and electronic equipment for directed distributed network |
| CN115665364A (en) * | 2022-10-17 | 2023-01-31 | 上海理想信息产业（集团）有限公司 | Video conferencing method, device, system and equipment for cloud-edge-device collaborative computing |
| CN117591039A (en) * | 2024-01-18 | 2024-02-23 | 济南浪潮数据技术有限公司 | Distributed storage methods, systems, equipment and media |
| CN118982125A (en) * | 2024-07-22 | 2024-11-19 | 北京科杰科技有限公司 | A task management method based on time estimation |
| CN118982125B (en) * | 2024-07-22 | 2025-02-21 | 北京科杰科技有限公司 | Task management method based on time estimation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111176840B (en) | 2023-11-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111176840B (en) | Distributed task allocation optimization method and device, storage medium and electronic device | |
| CN112162865A (en) | Server scheduling method and device and server | |
| CN111176852A (en) | Resource allocation method, device, chip and computer readable storage medium | |
| CN114968510A (en) | Multi-target dynamic task scheduling method and system based on improved ant colony algorithm | |
| CN112817728B (en) | Task scheduling method, network device and storage medium | |
| CN109002358A (en) | Mobile terminal software adaptive optimization dispatching method based on deeply study | |
| CN111813523A (en) | Duration pre-estimation model generation method, system resource scheduling method, device, electronic equipment and storage medium | |
| CN104484233B (en) | A kind of resource allocation methods | |
| CN110502321A (en) | A kind of resource regulating method and system | |
| CN104657217B (en) | A kind of cloud environment method for scheduling task based on non-homogeneous grain-size classification | |
| CN102662764A (en) | Dynamic cloud computing resource optimization allocation method based on semi-Markov decision process (SMDP) | |
| CN113419863B (en) | A data distribution and processing method and device based on node capability | |
| CN117032937B (en) | Task scheduling method based on GPU, electronic device and storage medium | |
| CN118796380A (en) | Virtual machine resource scheduling method and device | |
| Bensalem et al. | Scaling serverless functions in edge networks: A reinforcement learning approach | |
| CN119784088A (en) | Work order allocation method, device and electronic equipment | |
| CN111611076B (en) | Fair distribution method for mobile edge computing shared resources under task deployment constraint | |
| CN118740844B (en) | Method and device for determining execution node, storage medium and electronic device | |
| CN108200185B (en) | Method and device for realizing load balance | |
| CN117806806B (en) | Task partial unloading scheduling method, terminal device and storage medium | |
| CN111124439A (en) | Intelligent dynamic unloading algorithm with cloud edge cooperation | |
| CN106888237B (en) | A data scheduling method and system | |
| CN115250274A (en) | Node cooperation-based mobile ad hoc network computing unloading method | |
| Hu | Hybrid dynamic scheduling of mapreduce and spark services based on the profit model in the cloud computing platform | |
| CN118714076B (en) | Intelligent networking method of routers based on Internet of Things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |