CN107807815B - Method and apparatus for distributed processing tasks - Google Patents

Abstract

The present invention provides a method and device for distributed processing tasks, wherein the method includes: receiving a task request for a task to be executed; according to the task request, selecting a task compilation and deployment node that executes the to-be-executed task, and deploying the The to-be-executed task is sent to the task compilation and deployment node for execution. The method and device for distributed processing tasks provided by the embodiments of the present invention process multiple task requests by establishing a task compilation and deployment node set including the maximum concurrent task compilation and deployment nodes, and can create task compilation and deployment nodes in real time according to the number of processed tasks In order to meet the processing requirements of large-scale request tasks, it effectively solves the bottleneck problem of a single task compilation and deployment node when dealing with large-scale tasks.

Description

Method and apparatus for distributed processing tasks

technical field

The present invention relates to the field of computer technology, and in particular, to a method and device for distributed processing tasks.

Background technique

Continuous integration tools can be used to monitor ongoing repetitive work, including: continuous software release/testing projects and monitoring work performed by external calls. Each task compilation and deployment node in the continuous integration tool can be used to realize the processing of large-scale deployment and compilation tasks. In a distributed task compilation and deployment system built by continuous integration tools, task compilation and deployment nodes are prone to fall into performance bottlenecks when processing large-scale deployment and compilation tasks, or the task pressure on a single task compilation and deployment node is too large.

Take Jenkins as an example, Jenkins is a continuous integration tool developed based on Java. The entire environment of Jenkins' distributed construction requires a central server (Jenkins Master node) and several test machines (Slave nodes). Among them, the Jenkins Master node is responsible for task scheduling and the preservation of metadata and scheduling results, and the Jenkins Slave node is responsible for task scheduling. Specific execution, and the result is returned to the Jenkins Master node, so that the same set of code or project can be compiled and deployed in different environments.

Regarding the architecture of Jenkins, there are the following three forms in the prior art:

(1) Jenkins stand-alone mode

As shown in Figure 1, this mode means that the deployed Jenkins server is not only a Jenkins Master node but also a Jenkins Slave node. The Jenkins server will execute these tasks locally when it receives a build or deployment request.

(2) Combine a single Jenkins Master and static multiple Jenkins Slave modes

As shown in Figure 2, this mode refers to deploying a Jenkins Master node and multiple Jenkins Slave nodes. These Jenkins Slave nodes are created in advance and registered on the Jenkins Master node for JenkinsMaster scheduling. When the Jenkins Master receives the request, it will select the Jenkins Slave and schedule the task to be executed on the Jenkins Slave. After the execution is completed, the metadata and results of the task are saved on the Jenkins Master.

(3) Combining a single Jenkins Master and dynamic multiple Jenkins Slave modes

As shown in Figure 3, this mode refers to deploying a Jenkins Master node and multiple dynamic Jenkins Slave nodes. The Jenkins Slave node is created when a task is executed, and is dynamically registered on the Jenkins Master node for Jenkins Master scheduling. When the Jenkins Master receives the request, it will create a suitable Jenkins Slave node through the K8s plug-in, and schedule the task to the Jenkins Slave node for execution. After the execution is completed, the Jenkins Slave node will be destroyed, and the metadata and results of the task will be saved in the Jenkins Master. superior.

However, there are the following disadvantages in the prior art:

(1) About Jenkins stand-alone mode

Since Jenkins single-machine mode is only suitable for processing small-scale tasks, when the number of tasks increases, all pending tasks will be executed on one machine, thereby increasing the machine load and eventually causing the server to crash.

(2) About combining a single Master and static multiple Jenkins Slave modes

Although the static slave mode can avoid the problem of executing a large number of tasks on the same machine in the aforementioned single-machine mode, the number of slave nodes is still determined, and manual registration is required every time a slave node is added, so the operation is very troublesome, and When the task exceeds a certain scale, the pressure on the Master node will be very large.

(3) About the collection of a single Master and dynamic multiple Jenkins Slave modes

Although the dynamic creation of Jenkins Slave mode can save the trouble of manual registration, and the Jenkins Slave node does not need to exist all the time, only when there is a task request, but this method still cannot solve the problem of processing large-scale tasks, that is, when the task scale When the processing capacity is exceeded (for example, more than 10,000), the performance of a single Jenkins Master will be very poor, which directly causes the Jenkins Master node to fail to work.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present invention provide a method and apparatus for distributed processing tasks, so as to solve the bottleneck problem in the prior art when a single task compilation and deployment node faces large-scale tasks.

In order to achieve the above object, according to an aspect of the embodiments of the present invention, a method for distributed processing tasks is provided, and the method includes:

Receive task requests for tasks to be executed;

According to the task request, a task compilation and deployment node that executes the to-be-executed task is selected, and the to-be-executed task is sent to the task compilation and deployment node for execution.

Optionally, the method further includes: before receiving a task request for a task to be executed, creating a preset number of task compilation and deployment nodes, wherein the preset number refers to a maximum number of concurrent tasks not less than the maximum number of tasks to be executed. quantity.

Optionally, in the step of creating a preset number of task compilation and deployment nodes, the method includes: creating, by a cluster management system, a task compilation and deployment node set including a preset number of the task compilation and deployment nodes.

Optionally, the method further includes: after creating the preset number of task compilation and deployment nodes, saving the information of the task compilation and deployment nodes in a database, where the information includes at least one of the following: each The IP address of the server of each task compilation and deployment node, the running state, and the number of tasks that have been executed to complete the task, wherein the running state includes an idle state and a working state.

Optionally, according to the task request, selecting a task compilation and deployment node that executes the to-be-executed task, and sending the to-be-executed task to the task compilation and deployment node for execution includes:

After receiving the task request that the task to be executed is a compilation task, save the task request information of the compilation task in the database, and select the task request that meets the preset conditions from the database according to the task request of the compilation task. The task compilation and deployment node sends the task request of the compilation task to the task compilation and deployment node that meets the preset conditions for execution.

Optionally, according to the task request, selecting a task compilation and deployment node that executes the to-be-executed task, and sending the to-be-executed task to the task compilation and deployment node for execution further includes:

Before receiving the task request that the task to be executed is the compilation task, according to the received task request of creating the task, the task request information of the creating task is stored in the database.

Optionally, selecting a task compilation and deployment node that meets preset conditions from the database according to the task request of the compilation task includes:

Obtaining a list of task compilation and deployment nodes whose running state is the idle state saved in the database, and selecting the task compilation and deployment node with the least number of tasks that have completed the task from the list; and

If there is no task compilation and deployment node whose running state is idle state in the database, a new task compilation and deployment node is created by the cluster management system as the task compilation and deployment node for executing the compilation task.

Optionally, the method further includes: after selecting a task compilation and deployment node that executes the task to be executed, setting the running state of the task compilation and deployment node to a working state; and

After the execution of the to-be-executed task is completed, the result information of the to-be-executed task is saved, and the running state of the task compilation and deployment node is set to an idle state.

In one aspect of the embodiments of the present invention, an apparatus for distributed processing tasks is provided, and the apparatus includes:

The receiving module is used to receive the task request of the task to be executed;

The execution module is configured to select, according to the task request, a task compilation and deployment node that executes the to-be-executed task, and send the to-be-executed task to the task compilation and deployment node for execution.

Optionally, the device further includes: a creation module, configured to create a preset number of task compilation and deployment nodes before receiving a task request for a task to be executed, wherein the preset number refers to not less than the task to be executed. The maximum number of concurrency.

Optionally, the creating module is further configured to: create a task compilation and deployment section set including a preset number of the task compilation and deployment nodes through the cluster management system.

Optionally, the apparatus further includes: a storage module, configured to save the information of the task compilation and deployment nodes in a database after the preset number of task compilation and deployment nodes are created, wherein the information includes at least the following: One item: the IP address of the server of each task compilation and deployment node, the running state, and the number of tasks that have completed the task, wherein the running state includes an idle state and a working state.

Optionally, the execution module is further configured to: save the task request information of the compilation task in the database, and select a task compilation that meets preset conditions from the database according to the task request of the compilation task. The deployment node, and then send the task request of the compilation task to the task compilation and deployment node that meets the preset conditions for execution.

Optionally, the execution module is further configured to: before receiving the task request that the task to be executed is the compilation task, according to the received task request for creating the task, save the task request information of the creating task to the database middle.

Optionally, the execution module is further configured to: obtain a list of task compilation and deployment nodes that are stored in the database and whose running state is an idle state, and select a task compilation with the least number of tasks that have completed tasks from the list. deploying nodes; and

If there is no task compilation and deployment node whose running state is idle state in the database, a new task compilation and deployment node is created by the cluster management system as the task compilation and deployment node for executing the compilation task.

Optionally, the device further includes: a setting module, configured to set the running state of the task compiling and deploying node to a working state after selecting the task compiling and deploying node that executes the to-be-executed task; After the task is executed, the result information of the to-be-executed task is saved, and the running state of the task compilation and deployment node is set to an idle state.

An embodiment of the present invention further provides a terminal device for distributed processing tasks, including: one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more programs A plurality of processors execute, such that the one or more processors implement the method for distributing processing tasks according to an embodiment of the present invention.

Embodiments of the present invention further provide a computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, implements the method for distributed processing tasks according to the embodiments of the present invention.

The method and device for distributed processing tasks provided by the embodiments of the present invention can solve the bottleneck problem of a single task compiling and deploying node when dealing with large-scale tasks; by creating a task compiling and deploying node that includes the maximum concurrent number of task compiling and deploying nodes A set of nodes, so that the distributed system can have multiple task compilation and deployment nodes so as to process multiple task requests; by saving the information of the created task compilation and deployment nodes in the database, the processing according to the tasks to be executed can be processed. The requirement is convenient to select the task compilation and deployment node to execute the pending task; select the task compilation and deployment node that meets the preset conditions from the database according to the task request of the compilation task, and then send the task request of the compilation task to the task that meets the preset conditions. Compile and deploy nodes for execution, so that the task compilation and deployment nodes that meet the conditions can be selected to execute the corresponding tasks to be executed; by saving the information of the task compilation and deployment nodes in the database, the data persistence of the task compilation and deployment node information is realized, so that Data can be easily obtained and/or viewed at any time during subsequent processing; the list of deployment nodes can be compiled and deployed by obtaining the tasks saved in the database whose running state is idle, and the task compilation with the least number of tasks that have completed tasks is selected from the list. deployment node; and if there is no task compilation deployment node whose running state is an idle state in the database, a new task compilation deployment node is created by the cluster management system as the task compilation deployment node for executing the compilation task, so that a task compilation deployment node that satisfies the predetermined requirements can be selected. Conditional tasks compile and deploy nodes to execute pending tasks.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort. In the attached image:

Fig. 1 is the schematic diagram of the Jenkins stand-alone mode in the architecture of Jenkins in the prior art;

Fig. 2 is a schematic diagram of combining a single Master and static multiple slave modes in the architecture of Jenkins in the prior art;

3 is a schematic diagram of combining a single Master and dynamic multiple slave modes in the architecture of Jenkins in the prior art;

4 is a schematic diagram of a main flow of a method for distributed processing tasks according to an embodiment of the present invention;

5 is a schematic diagram of main modules of an apparatus for distributed processing tasks according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a computer system suitable for implementing a terminal device according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In this article, it is to be understood that among the terms involved:

Jenkins: It is an open source software project and a continuous integration tool developed based on Java to monitor continuous repetitive work. It aims to provide an open and easy-to-use software platform to make continuous software integration possible.

Kubernetes: referred to as k8s, is an open source platform for automating container operations. It is a system for implementing cluster management of containerized applications, including deployment, scheduling, and scaling between node clusters. Kubernetes not only supports Docker, but also Rocket, another container technology.

MySQL: It is a relational database management system, developed by the Swedish MySQL AB company, and is currently a product of Oracle. MySQL is one of the most popular relational database management systems. In terms of WEB applications, MySQL is the best RDBMS (Relational Database Management System) application software.

Exemplary method

As shown in Figure 4, a method for distributed processing tasks is provided, and the method includes:

Step S401: receiving a task request for a task to be executed;

Step S402: According to the task request, select a task compilation and deployment node Jenkins Master that executes the task to be performed, and send the to-be-executed task to the task compilation and deployment node for execution. In the embodiment of the present invention, the task compilation and deployment node may be, but is not limited to, the Jenkins Master in the Jenkins continuous integration tool. The following takes Jenkins Master as an example to describe the creation of task compilation and deployment nodes and task processing.

Optionally, the method further includes: before receiving the task request for the task to be executed, creating a preset number of Jenkins Masters, where the preset number refers to a number that is not less than the maximum concurrent number of the to-be-executed task.

Optionally, in the step of creating a preset number of Jenkins Masters, including:

A JenkinsMaster collection containing a preset number of said Jenkins Masters is created by the Kubernetes system.

Optionally, the method further includes: after creating the preset number of Jenkins Masters, saving the information of the Jenkins Master into a database, wherein the information includes at least one of the following: The IP address of the server, the running status, and the number of tasks that have been executed and completed, wherein the running status includes an idle status and a working status.

Optionally, according to the task request, selecting the Jenkins Master that executes the task to be executed, and sending the task to be executed to the Jenkins Master for execution includes: receiving a task request that the task to be executed is a compilation task After that, the task request information of the compilation task is saved in the database, and a Jenkins Master that meets the preset conditions is selected from the database according to the task request of the compilation task, and then the task request of the compilation task is selected from the database. Send it to the Jenkins Master that meets the preset conditions for execution.

Optionally, according to the task request, selecting the Jenkins Master that executes the task to be executed, and sending the task to be executed to the Jenkins Master for execution further includes:

Before receiving the task request that the task to be executed is the compilation task, according to the received task request of creating the task, the task request information of the creating task is stored in the database.

Optionally, selecting a Jenkins Master that meets preset conditions from the database according to the task request of the compilation task includes:

Acquiring a list of Jenkins Masters whose running states are idle states saved in the database, and selecting from the list the Jenkins Master that has the least number of tasks that have completed tasks; and

If there is no Jenkins Master whose running state is in the idle state in the database, a new Jenkins Master is created through the Kubernetes system as the Jenkins Master that executes the compilation task.

Optionally, the method further includes: after selecting the Jenkins Master that executes the task to be executed, setting the running state of the Jenkins Master to a working state; and

After the execution of the to-be-executed task is completed, the result information of the to-be-executed task is saved, and the running state of the Jenkins Master is set to an idle state.

Example

The present invention will be specifically described below with reference to a specific embodiment, however, it should be noted that the specific embodiment is only for better describing the present invention, and does not constitute an improper limitation of the present invention.

First, according to the processing requirements of the task, a Jenkins Master pool (Jenkins Master Pool) containing a preset number of JenkinsMasters is created through the Kubernetes system, and the information of each Jenkins Master in the Jenkins Master collection is saved to the Server table of the Mysql database. The information recorded in the Server table includes at least one or more of the IP address of the server of each Master, whether the status of the Master is idle, and the number of tasks that the Master has performed.

Then, after the application program interface (API) receives the task request, the request type of the task request is determined.

If the task request is a "create task" task request, insert a record into the Job table in the Mysql database. Specifically, since the native Jenkins will save the task locally in the form of text, only the local Jenkins will query and obtain this task, and the purpose of inserting a record in the embodiment of the present invention is to save this task information To the database, so that the task can be obtained from the database when the task is compiled in the future, and then select a Jenkins Master from the Jenkins Master collection (Jenkins Master Pool) to execute the task;

If the task request is a "compile task" request, first insert a build record into the Build table in the Mysql database. Specifically, the purpose of saving the build record to the Mysql database is to facilitate users to view these records in the future. Even if the Jenkins Master that executed the build record at that time is destroyed, we can still view this record from the Mysql database. Then, select a Master that satisfies the preset conditions from the Server table to process the task. Specifically, the preset conditions are as follows:

First, filter out the idle Master list from the Server table;

Second, in the list of idle Jenkins Masters, sort them according to the number of completed tasks, and select the Jenkins Master with the smallest number of completed tasks as the JenkinsMaster that meets the preset conditions;

Third, if there is no idle Jenkins Master in the Server table, a new Jenkins Master is dynamically created through the Kubernetes system, the new Jenkins Master handles the task, and the new Jenkins Master record is inserted into the Server table.

That is to say, when the received task request includes the task to be executed and the number of tasks is greater than the preset number, the Kubernetes system continues to create the number of tasks and the number of tasks in the Jenkins Master set through the Kubernetes system. The difference between the preset number of said Jenkins Masters.

Finally, after finding a Jenkins Master that satisfies the preset conditions, set the status in the record information of the Master in the Server table to "working status", and after the task is executed, save the result information of the task to the Build table , and then set the state of the Jenkins Master to "idle state" for subsequent use.

The method for distributed processing tasks provided by the embodiments of the present invention can prepend large-scale tasks to the database from a single Jenkins Master node, and then evenly schedule these tasks to each Jenkins Master node, and process the single Jenkins Master node. The form of large-scale tasks is converted into multiple Jenkins Master nodes to process small-scale tasks at the same time, which effectively solves the bottleneck problem encountered when a single Jenkins Master node handles multiple tasks.

In addition, when performing a "compile" or "deployment" task, the distributed system only needs to process the corresponding request received, and does not need to respond and judge the source of the pending task, thereby decoupling the pending task from the Jenkins service .

As shown in FIG. 5 , it is a schematic diagram of main modules of an apparatus for distributed processing tasks according to an embodiment of the present invention; wherein, the apparatus includes:

A receiving module 501, configured to receive a task request of a task to be executed;

The execution module 502 is configured to select a JenkinsMaster that executes the to-be-executed task according to the task request, and send the to-be-executed task to the Jenkins Master for execution.

Optionally, the device further includes: a creation module for creating a preset number of Jenkins Masters before receiving a task request for a task to be executed, wherein the preset number refers to a maximum number of not less than the task to be executed The number of concurrent counts.

Optionally, the creation module is further configured to: create a Jenkins Master set including a preset number of the Jenkins Master through the Kubernetes system.

Optionally, the device further includes: a storage module for saving the information of the Jenkins Master in a database after creating the preset number of Jenkins Masters, wherein the information includes at least one of the following: The IP address of the server of each of the Jenkins Master, the running status, and the number of tasks that have been executed to complete the task, wherein the running status includes an idle status and a working status.

Optionally, the execution module is further configured to: save the task request information of the compilation task in the database, and select a JenkinsMaster that meets preset conditions from the database according to the task request of the compilation task, Then, the task request of the compilation task is sent to the Jenkins Master that meets the preset conditions for execution.

Optionally, the execution module is further configured to: before receiving the task request that the task to be executed is the compilation task, according to the received task request for creating the task, save the task request information of the creating task to the database middle.

Optionally, the execution module is further configured to: obtain a list of Jenkins Masters whose running states are idle states saved in the database, and select from the list the Jenkins Master that has performed the least number of tasks that have completed tasks; and

If there is no Jenkins Master whose running state is in the idle state in the database, a new Jenkins Master is created through the Kubernetes system as the Jenkins Master that executes the compilation task.

Optionally, the device further includes: a setting module, configured to set the running state of the Jenkins Master to a working state after selecting the JenkinsMaster that executes the task to be executed; and after the execution of the task to be executed is completed, The result information of the to-be-executed task is saved, and the running state of the Jenkins Master is set to an idle state.

Since the device for distributed processing tasks provided by the present invention is a device corresponding to the above method, it is not repeated here.

Referring to FIG. 6 below, it shows a schematic structural diagram of a computer system 600 suitable for implementing a terminal device according to an embodiment of the present invention. The terminal device shown in FIG. 6 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in FIG. 6, a computer system 600 includes a central processing unit (CPU) 601, which can be loaded into a random access memory (RAM) 603 according to a program stored in a read only memory (ROM) 602 or a program from a storage section 608 Instead, various appropriate actions and processes are performed. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601 , the ROM 602 , and the RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

The following components are connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc. ; and a communication section 609 including a network interface card such as a LAN card, a modem, and the like. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I/O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs in accordance with the disclosed embodiments of the present invention. For example, embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 609 and/or installed from the removable medium 611 . When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the system of the present invention are executed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules can also be set in the processor, for example, it can be described as: a processor includes a receiving module 501 and an executing module 502 . Among them, the names of these modules do not constitute a limitation on the module itself under certain circumstances.

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by a device, the device includes:

Step S401: receiving a task request for a task to be executed;

Step S402: According to the task request, select the Jenkins Master that executes the task to be executed, and send the task to be executed to the Jenkins Master for execution.

The method and device for distributed processing tasks provided by the embodiments of the present invention can solve the bottleneck problem of a single task compiling and deploying node when dealing with large-scale tasks; by creating a task compiling and deploying node that includes the maximum concurrent number of task compiling and deploying nodes A set of nodes, so that the distributed system can have multiple task compilation and deployment nodes so as to process multiple task requests; by saving the information of the created task compilation and deployment nodes in the database, the processing according to the tasks to be executed can be processed. The requirement is convenient to select the task compilation and deployment node to execute the pending task; select the task compilation and deployment node that meets the preset conditions from the database according to the task request of the compilation task, and then send the task request of the compilation task to the task that meets the preset conditions. Compile and deploy nodes for execution, so that the task compilation and deployment nodes that meet the conditions can be selected to execute the corresponding tasks to be executed; by saving the information of the task compilation and deployment nodes in the database, the data persistence of the task compilation and deployment node information is realized, so that Data can be easily obtained and/or viewed at any time during subsequent processing; the list of deployment nodes can be compiled and deployed by obtaining the tasks saved in the database whose running state is idle, and the task compilation with the least number of tasks that have completed tasks is selected from the list. deployment node; and if there is no task compilation deployment node whose running state is an idle state in the database, a new task compilation deployment node is created by the cluster management system as the task compilation deployment node for executing the compilation task, so that a task compilation deployment node that satisfies the predetermined requirements can be selected. Conditional tasks compile and deploy nodes to execute pending tasks.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (16)

1. a method for distributed processing task, is characterized in that, comprises: Receive task requests for tasks to be executed; According to the task request, select a task compilation and deployment node that executes the to-be-executed task, and send the to-be-executed task to the task compilation and deployment node for execution; According to the task request, selecting a task compilation and deployment node that executes the task to be executed, and sending the task to be executed to the task compilation and deployment node for execution includes: receiving a task request that the task to be executed is a compilation task After that, the task request information of the compilation task is stored in the database, and a task compilation and deployment node that meets the preset conditions is selected from the database according to the task request of the compilation task, and then the task request of the compilation task is selected from the database. Send to the task compilation and deployment node that meets the preset conditions for execution. 2. The method of claim 1, further comprising: Before receiving a task request for a task to be executed, a preset number of task compilation and deployment nodes are created, where the preset number refers to a number that is not less than the maximum concurrent number of the to-be-executed task. 3. The method according to claim 2, wherein, in the step of creating a preset number of task compilation and deployment nodes, comprising: A task compilation and deployment node set including a preset number of task compilation and deployment nodes is created through the cluster management system. 4. The method of claim 3, further comprising: After the preset number of task compilation and deployment nodes are created, the information of the task compilation and deployment nodes is stored in a database, wherein the information includes at least one of the following: the IP address of the server of each task compilation and deployment node Address, running state, and the number of tasks that have been executed and completed, wherein the running state includes idle state and working state. 5. The method according to claim 1, wherein, according to the task request, a task compilation and deployment node that executes the to-be-executed task is selected, and the to-be-executed task is sent to the task compilation and deployment node for execution. Execution also includes: Before receiving the task request that the task to be executed is the compilation task, according to the received task request of creating the task, the task request information of the creating task is stored in the database. 6. The method according to claim 1, wherein selecting a task compilation and deployment node that meets preset conditions from the database according to the task request of the compilation task comprises: Obtaining a list of task compilation and deployment nodes whose running state is the idle state saved in the database, and selecting the task compilation and deployment node with the least number of tasks that have completed the task from the list; and If there is no task compilation and deployment node whose running state is in the idle state in the database, a new task compilation and deployment node is created through the Kubernetes system as the task compilation and deployment node for executing the compilation task. 7. The method according to any one of claims 1 to 6, further comprising: After selecting the task compilation and deployment node that executes the task to be executed, setting the running state of the task compilation and deployment node to a working state; and After the execution of the to-be-executed task is completed, the result information of the to-be-executed task is saved, and the running state of the task compilation and deployment node is set to an idle state. 8. A device for distributed processing tasks, comprising: The receiving module is used to receive the task request of the task to be executed; an execution module, configured to select a task compilation and deployment node that executes the to-be-executed task according to the task request, and send the to-be-executed task to the task compilation and deployment node for execution; The execution module is further configured to: save the task request information of the compilation task in a database, select a task compilation and deployment node that meets preset conditions from the database according to the task request of the compilation task, and then compile the compilation task. The task request of the task is sent to the task compilation and deployment node that meets the preset condition for execution. 9. The apparatus of claim 8, further comprising: The creation module is configured to create a preset number of task compilation and deployment nodes before receiving a task request for a task to be executed, where the preset number refers to a number that is not less than the maximum concurrent number of the to-be-executed task. 10. The apparatus according to claim 9, wherein the creation module is further used for: A task compilation and deployment node set including a preset number of task compilation and deployment nodes is created through the cluster management system. 11. The apparatus of claim 10, further comprising: a storage module, configured to save the information of the task compilation and deployment nodes in a database after the preset number of task compilation and deployment nodes are created, wherein the information includes at least one of the following contents: each of the tasks Compile the IP address, running status, and number of completed tasks of the server where the node is deployed, where the running status includes an idle status and a working status. 12. The apparatus according to claim 8, wherein the execution module is further configured to: Before receiving the task request that the task to be executed is the compilation task, according to the received task request of creating the task, the task request information of the creating task is stored in the database. 13. The device according to claim 10, wherein the execution module is further configured to: Obtaining a list of task compilation and deployment nodes whose running state is the idle state saved in the database, and selecting the task compilation and deployment node with the least number of tasks that have completed the task from the list; and If there is no task compilation and deployment node whose running state is in the idle state in the database, a new task compilation and deployment node is created by the cluster management system as the task compilation and deployment node for executing the compilation task. 14. The device according to any one of claims 8 to 13, further comprising: a setting module, configured to set the running state of the task compiling and deploying node to a working state after selecting a task compiling and deploying node that executes the task to be executed; and After the execution of the to-be-executed task is completed, the result information of the to-be-executed task is saved, and the running state of the task compilation and deployment node is set to an idle state. 15. A terminal device for distributed processing tasks, comprising: one or more processors; storage means for storing one or more programs, The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7. 16. A computer-readable medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the method according to any one of claims 1-7 is implemented.

Images