CN111831408A - Asynchronous task processing method and device, electronic equipment and medium - Google Patents
Asynchronous task processing method and device, electronic equipment and medium Download PDFInfo
- Publication number
- CN111831408A CN111831408A CN202010616454.7A CN202010616454A CN111831408A CN 111831408 A CN111831408 A CN 111831408A CN 202010616454 A CN202010616454 A CN 202010616454A CN 111831408 A CN111831408 A CN 111831408A
- Authority
- CN
- China
- Prior art keywords
- task
- computing
- asynchronous
- tasks
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides an asynchronous task processing method, an asynchronous task processing device, electronic equipment and a computer readable medium. The method comprises the following steps: receiving asynchronous tasks submitted by a user, establishing an asynchronous task queue, creating and updating a task table and a task assignment table in real time, selecting corresponding computing nodes to distribute computing tasks according to task information of the asynchronous tasks aiming at each asynchronous task, and collecting corresponding task computing results and sending the corresponding task computing results to the user when the computing nodes complete the computing tasks so as to complete task execution requests of the user. In the method, because the corresponding computing nodes are selected to distribute the computing tasks according to the task information of the asynchronous tasks and the storage and state management of different types of asynchronous tasks are realized through the task table and the task assignment table, the task scheduling function can be quickly realized without deploying an independent service system for each type of asynchronous task, and compared with the prior art, the method has the advantages of reducing network overhead and improving processing efficiency.
Description
Technical Field
The present application relates to the field of computer application technologies, and in particular, to an asynchronous task processing method and apparatus, an electronic device, and a computer-readable medium.
Background
Asynchronous task queues are concepts that are presented with respect to common synchronous computing tasks.
The synchronous computing task is that when a large number of tasks requiring a large amount of computing time are received, the computer serially processes the tasks according to a certain priority order and returns corresponding task processing results. In the synchronous calculation task, the user has to wait for the completion of the execution of the previous calculation task to know whether the own request is correctly received by the server, and most feasibly, the user cannot perform other operations during the waiting time and only waits for the return of the result, because the user needs to resend the request until the request is successful if the request is not correctly received by the server. This time is intolerable for a normal network request, since it results in a very large network overhead and requests are timed out frequently.
Therefore, the concept of asynchronous task queues is proposed, which encourages that when a large number of computing tasks are received, the tasks are saved and immediately returned to the sender whether the task request has been correctly received, then the tasks are executed one by one in the background, and the sender is informed of the result after the tasks are executed. Compared with the time of task execution, the time from the initiation of the request to the receipt of the network request from the server to confirm whether the request has been received is tolerable or even negligible. This is why asynchronous task queuing techniques are now widely used in a variety of modern computer system architectures and network services.
With the advent of the mobile internet, cloud computing and big data era, when the existing asynchronous task processing method is oriented to application scenarios with large amount of computing tasks and more task categories or computing properties, an independent service system needs to be deployed for maintenance and update work of each category of computing tasks or each online application, which causes high network overhead and low processing efficiency.
Disclosure of Invention
The application aims to provide an asynchronous task processing method and device, an electronic device and a computer readable medium.
A first aspect of the present application provides an asynchronous task processing method, including:
receiving an asynchronous task submitted by a user, and establishing an asynchronous task queue;
creating and updating a task table and a task assignment table in real time, wherein the task table is used for recording task information of each asynchronous task, and the task assignment table is used for recording computing node allocation information and task computing results of each asynchronous task;
aiming at each asynchronous task, selecting a corresponding computing node according to task information of the asynchronous task to distribute a computing task;
and when the computing node finishes the computing task, collecting a corresponding task computing result and sending the task computing result to the user.
A second aspect of the present application provides an asynchronous task processing apparatus, including:
the receiving module is used for receiving the asynchronous tasks submitted by the user and establishing an asynchronous task queue;
the system comprises a creating module, a task assigning module and a task processing module, wherein the creating module is used for creating and updating a task table and a task assigning table in real time, the task table is used for recording task information of each asynchronous task, and the task assigning table is used for recording computing node distribution information and task computing results of each asynchronous task;
the computing module is used for selecting a corresponding computing node to distribute computing tasks according to the task information of each asynchronous task;
and the sending module is used for collecting corresponding task calculation results and sending the task calculation results to the user when the calculation nodes finish the calculation tasks.
A third aspect of the present application provides an electronic device comprising: memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the computer program when executing the computer program to perform the method of the first aspect of the application.
A fourth aspect of the present application provides a computer readable medium having computer readable instructions stored thereon which are executable by a processor to implement the method of the first aspect of the present application.
Compared with the prior art, the asynchronous task processing method, the asynchronous task processing device, the electronic equipment and the asynchronous task processing medium provided by the application receive asynchronous tasks submitted by users, establish an asynchronous task queue, create and update a task table and a task assignment table in real time, select corresponding computing nodes to distribute computing tasks according to task information of the asynchronous tasks, and collect corresponding task computing results and send the computing results to the users to complete task execution requests of the users when the computing nodes complete the computing tasks. In the method, because the corresponding computing nodes are selected to distribute the computing tasks according to the task information of the asynchronous tasks and the storage and state management of different types of asynchronous tasks are realized through the task table and the task assignment table, the task scheduling function can be quickly realized without deploying an independent service system for each type of asynchronous task, and compared with the prior art, the method has the advantages of reducing network overhead and improving processing efficiency.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 illustrates a flow diagram of a method of asynchronous task processing provided by some embodiments of the present application;
FIG. 2 is a schematic diagram of the architecture of a processing device employing the asynchronous task processing method of the present application;
FIG. 3 illustrates a schematic diagram of an asynchronous task processing device provided by some embodiments of the present application;
FIG. 4 illustrates a schematic diagram of an electronic device provided by some embodiments of the present application;
FIG. 5 illustrates a schematic diagram of a computer-readable medium provided by some embodiments of the present application.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
It is to be noted that, unless otherwise specified, technical or scientific terms used herein shall have the ordinary meaning as understood by those skilled in the art to which this application belongs.
In addition, the terms "first" and "second", etc. are used to distinguish different objects, rather than to describe a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
In order to solve the problems of high network overhead and low processing efficiency of the conventional asynchronous task processing method, embodiments of the present application provide an asynchronous task processing method and apparatus, an electronic device, and a computer readable medium, which are described below with reference to the accompanying drawings.
Referring to fig. 1, a flowchart of an asynchronous task processing method provided in some embodiments of the present application is shown, where the asynchronous task processing method may be applied to an asynchronous task processing device (or server) as shown in fig. 2, and the device may include two major modules, namely a task queue scheduling module and a computation task processing module, where the computation task processing module is composed of a plurality of computation nodes.
Referring to fig. 2, the task queue scheduling module mainly functions to provide a uniform service interface to the outside, provide computing resource service for a user (i.e., a task request initiator), select a corresponding computing node according to a service class required by the user, and then allocate a computing task to the selected computing node. When the computing node completes the computing task, the task queue scheduling module can also collect the corresponding task computing result and send the result to the user. The computing task processing module is composed of a plurality of computing nodes with computing capacity, and each computing node can be in the form of a server, a virtual machine or a service process and provides computing resources for the task queue scheduling module.
It should be noted that: the device is asynchronous task queue oriented, so there is not much limit to the time difference from the initiation of a computation task request to the reception of the computation result of the task, and all asynchronous tasks are executed sequentially in the form of a FIFO (first-in first-out) queue and return the result.
As shown in fig. 1, the asynchronous task processing method may include the steps of:
step S101: and receiving the asynchronous tasks submitted by the user and establishing an asynchronous task queue.
In this embodiment, first, the task queue scheduling module receives and persistently stores all submitted asynchronous tasks, and establishes an asynchronous task queue.
Step S102: and creating and updating the task table and the task assignment table in real time.
According to an embodiment of the present application, step S102 may be implemented as: creating the task table and a task assignment table based on a MongoDB database; and updating the task table and the task assignment table in the MongoDB database at regular time based on the thread library of python.
The task table is used for recording task information of each asynchronous task, and the task assignment table is used for recording computing node allocation information and task computing results of each asynchronous task.
Specifically, the task information may include task identification, task content, task type, and task status. The task identification means that a unique task ID is allocated to each independent task and an index is generated, so that a desired task can be found quickly, and the service efficiency is improved. The task content refers to specific content of a task submitted by a user, for example, complex calculation and conversion of a large number of pictures are required, and AI (artificial intelligence) detection of diseases is performed according to medical image materials. Task type refers to the type of task content, such as different categories of disease detection. The task state refers to the stage of task processing, e.g., whether it is completed, how much it is completed.
Specifically, the computing node allocation information indicates to which computing nodes the task is allocated to process, and the task computing result indicates a computing result after the task is processed by the computing nodes.
In order to be compatible with asynchronous tasks of various different formats and types, and simultaneously allow for the expansion of subsequent functions or performances, the step S102 can be implemented by using a database Mongodb based on distributed file storage. The core here lies in that two tables are separated, so that the situation that one task needs to be assigned to a plurality of computing nodes under the condition that the processing fails or a plurality of computing nodes are required to be cooperatively completed can be effectively solved. Meanwhile, because the Mongodb is a resident non-relational database and is assisted by the index, the Mongodb can be matched with a background to process the scheduling task very efficiently.
Step S103: and aiming at each asynchronous task, selecting a corresponding computing node according to the task information of the asynchronous task to distribute the computing task.
In this embodiment, for different asynchronous tasks, a computing node adept at processing the type of task may be selected to process, that is, a corresponding computing node is selected to distribute computing tasks according to task information of the asynchronous task.
According to some embodiments of the present application, the method may further comprise the steps of: monitoring the states of all the computing nodes to obtain real-time states of all the computing nodes, where the states of the computing nodes may include: offline, online idle, online busy.
Specifically, according to some embodiments of the present application, the step of monitoring the states of all the computing nodes may be implemented as: setting a corresponding daemon process for each computing node, wherein the daemon process is used for monitoring the state of the current computing node; and updating a state table corresponding to the computing node at regular time based on the thread library of python, wherein the state table is used for recording the state of each computing node.
Accordingly, according to some embodiments of the present application, step S103 may be specifically implemented as: and selecting corresponding computing nodes to distribute the computing tasks according to the task information of the asynchronous tasks and the states of the computing nodes.
In this embodiment, when selecting a compute node to allocate a compute task, not only the task itself but also the states of the compute nodes are considered, which is to allocate the compute task according to an optimal policy according to different states of each compute node. The computing task only selects an online idle computing node to assign the task, once the computing task is successfully distributed, the task queue scheduling module sets the state of the computing node to be online busy, and further blocks the distribution of a new computing task to the node, and further resource conflict caused by the fact that a single computing node processes a plurality of tasks at the same time is prevented. The method can be realized by adopting a thread library based on python and updating the state table of the computing node at regular time through a multithread programming technology.
It should be noted that: because the computing resources are limited, but the computing tasks are diversified, in order to ensure that each computing node can be effectively utilized, the phenomenon that some computing tasks occupy too long time of resources or an unknown error occurs to cause that the task processing process is in a false dead state and can not be normally exited is prevented. In the embodiment, the timeout limit of task processing is realized by adopting the reading library and the time library based on python for encapsulation, and the execution time of the computing task is ensured not to exceed the threshold value.
Moreover, in order to prevent the computing task process from being stuck or accidentally exited due to the occurrence of the abnormality, and to further enhance the stability of the computing nodes, in this embodiment, a corresponding daemon process is set for each computing node itself, so as to monitor the state of the current computing node. If the computing task process is accidentally exited or is judged to be offline by the scheduling system, the daemon process restarts the computing task process and forces the daemon process to register the online state with the task queue scheduling module again. The embodiment adopts a multiprocessing library based on python, and enables the daemon process of the computing node and the process of the processing task to be independent of each other through a multi-process programming technology, but can communicate with each other through states.
Step S104: and when the computing node finishes the computing task, collecting a corresponding task computing result and sending the task computing result to the user.
In this embodiment, when the computing node completes the computing task and returns the task computing result, the task queue scheduling module receives and stores the task computing result, updates the corresponding task state, and forwards the task computing result to the user who submitted the task. Specifically, the calculation results can be stored by utilizing Mongodb, and the task calculation results are sent in a mode of sending an HTTPPOST request to the user.
According to some embodiments of the present application, the method may further comprise the steps of: the status of each computing task assigned to a compute node is monitored. And for each calculation task, if the calculation task is not finished after the preset execution time, reallocating the calculation task to the calculation nodes.
The embodiment prevents the situation that the processing result can not be received all the time due to some unknown reasons after part of tasks are assigned for processing, thereby well protecting the stability and the robustness of the whole task queue. The method can be realized by adopting a thread library based on python and updating the Mongodb database at regular time through a multithread programming technology.
According to some embodiments of the present application, the method may further comprise the steps of: and generating an access log of the user, wherein the access log records all operations of the user.
In the embodiment, in order to record the condition of the whole task queue scheduling module, when an unknown error occurs or a history record needs to be searched, effective data support can be provided. The patent adopts a logging module based on python to realize the retention of access logs from users of all parties.
According to some embodiments of the present application, a computing node may register node information with a task queue scheduling module, and the computing node sends its health status (e.g., heartbeat time) to the task queue scheduling module through a preset health status reporting mechanism (e.g., a heartbeat mechanism). According to the health status reporting mechanism, when an abnormal computing node is found (for example, the heartbeat time of a certain computing node is greater than a normal heartbeat time interval), the task queue scheduling module deletes the node information of the abnormal computing node, and redistributes the incomplete computing task of the node as a new task, preferably, the task is distributed to the computing node with a smaller number of current tasks, so as to balance the task load among the computing nodes. According to the embodiment, a task recovery mechanism is added, the task allocation safety is improved, and the task scheduling throughput is effectively improved.
Therefore, the asynchronous task processing method is mainly oriented to application scenes with large calculation task amount, more task types or calculation properties and lower requirement on server response time, for example, the asynchronous task processing method can be used for constructing a calculation platform or an operation maintenance platform and the like, an independent service system is not required to be deployed for each type of calculation task or each online application maintenance and update work, the task scheduling function can be rapidly realized, and the task queue scheduling module has a calculation node state monitoring function, so that the maintenance cost is lower, and the efficiency is higher. Particularly, when the AI detection platform is oriented to various different types of diseases, under the condition that medical image materials, AI detection algorithms and models required by different diseases are different even the realization technology is different, the method can conveniently and efficiently expand detection services of more different types of diseases, and the conventional medical image detection time also needs the working time from half a day to one day, namely the response time requirement on the returned result is not high, so the method can well serve the scenes of the type and reduce the operation cost of the medical system.
According to the asynchronous task processing method, the corresponding computing nodes are selected to distribute the computing tasks according to the task information of the asynchronous tasks, and the storage and state management of different types of asynchronous tasks are achieved through the task table and the task assignment table, so that task scheduling and processing functions can be achieved quickly without deploying an independent service system for each type of asynchronous task, compared with the prior art, network overhead is reduced, and processing efficiency is improved.
In the foregoing embodiment, an asynchronous task processing method is provided, and correspondingly, the present application also provides an asynchronous task processing apparatus. The asynchronous task processing device provided by the embodiment of the application can implement the asynchronous task processing method, and the asynchronous task processing device can be implemented through software, hardware or a combination of software and hardware. For example, the asynchronous task processing device may comprise integrated or separate functional modules or units to perform the corresponding steps of the above-described method. Referring to fig. 3, a schematic diagram of an asynchronous task processing device according to some embodiments of the present application is shown. Since the apparatus embodiments are substantially similar to the method embodiments, they are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for relevant points. The device embodiments described below are merely illustrative.
As shown in fig. 3, the asynchronous task processing device 10 may include:
the receiving module 101 is configured to receive an asynchronous task submitted by a user and establish an asynchronous task queue;
the creating module 102 is configured to create and update a task table and a task assignment table in real time, where the task table is used to record task information of each asynchronous task, and the task assignment table is used to record computation node allocation information and task computation results of each asynchronous task;
the computing module 103 is configured to select, for each asynchronous task, a corresponding computing node according to task information of the asynchronous task to allocate a computing task;
and the sending module 104 is configured to collect corresponding task calculation results when the computing node completes the calculation task, and send the corresponding task calculation results to the user.
In some implementations of embodiments of the present application, the task information includes task identification, task content, task type, and task status.
In some implementations of embodiments of the present application, the apparatus 10 further comprises:
a monitoring module to:
the status of each computing task assigned to a compute node is monitored.
For each computing task, if the computing task is not completed after the preset execution time, the computing task is redistributed to computing nodes.
In some implementations of embodiments of the present application, the monitoring module is further configured to:
the states of all the computing nodes are monitored.
The calculation module 103 is specifically configured to:
and selecting corresponding computing nodes to distribute the computing tasks according to the task information of the asynchronous tasks and the states of the computing nodes.
In some implementations of embodiments of the present application, the state of the compute node includes: offline, online idle, online busy.
In some implementations of embodiments of the present application, the apparatus 10 further comprises:
and the log module is used for generating an access log of the user, and the access log records all operations of the user.
The asynchronous task processing device 10 provided in the embodiment of the present application has the same beneficial effects as the asynchronous task processing method provided in the foregoing embodiment of the present application.
The embodiment of the present application further provides an electronic device corresponding to the asynchronous task processing method provided in the foregoing embodiment, where the electronic device may be a mobile phone, a notebook computer, a tablet computer, a desktop computer, or the like, so as to execute the asynchronous task processing method.
Referring to fig. 4, a schematic diagram of an electronic device provided in some embodiments of the present application is shown. As shown in fig. 4, the electronic device 20 includes: the system comprises a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected through the bus 202; the memory 201 stores a computer program that can be executed on the processor 200, and the processor 200 executes the asynchronous task processing method provided by any one of the foregoing embodiments when executing the computer program.
The Memory 201 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 203 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
The processor 200 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 200. The Processor 200 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the method in combination with the hardware thereof.
The electronic device provided by the embodiment of the application and the asynchronous task processing method provided by the embodiment of the application have the same inventive concept and have the same beneficial effects as the method adopted, operated or realized by the electronic device.
Referring to fig. 5, a computer-readable storage medium is shown as an optical disc 30, on which a computer program (i.e., a program product) is stored, where the computer program is executed by a processor to perform the asynchronous task processing method provided in any of the foregoing embodiments.
It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other optical and magnetic storage media, which are not described in detail herein.
The computer-readable storage medium provided by the above-mentioned embodiment of the present application and the asynchronous task processing method provided by the embodiment of the present application have the same beneficial effects as the method adopted, executed or implemented by the application program stored in the computer-readable storage medium.
It should be noted that the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the present disclosure, and the present disclosure should be construed as being covered by the claims and the specification.
Claims (10)
1. An asynchronous task processing method, comprising:
receiving an asynchronous task submitted by a user, and establishing an asynchronous task queue;
creating and updating a task table and a task assignment table in real time, wherein the task table is used for recording task information of each asynchronous task, and the task assignment table is used for recording computing node allocation information and task computing results of each asynchronous task;
aiming at each asynchronous task, selecting a corresponding computing node according to task information of the asynchronous task to distribute a computing task;
and when the computing node finishes the computing task, collecting a corresponding task computing result and sending the task computing result to the user.
2. The method of claim 1, further comprising:
monitoring the state of each computing task distributed to the computing nodes;
for each computing task, if the computing task is not completed after the preset execution time, the computing task is redistributed to computing nodes.
3. The method of claim 1, further comprising:
monitoring the states of all the computing nodes;
the selecting the corresponding computing node according to the task type of the asynchronous task to distribute the computing task comprises:
and selecting corresponding computing nodes to distribute the computing tasks according to the task information of the asynchronous tasks and the states of the computing nodes.
4. The method of claim 1, wherein creating and updating in real-time a task table and a task assignment table comprises:
creating the task table and a task assignment table based on a MongoDB database;
and updating the task table and the task assignment table in the MongoDB database at regular time based on the thread library of python.
5. The method of claim 3, wherein monitoring the status of all computing nodes comprises:
setting a corresponding daemon process for each computing node, wherein the daemon process is used for monitoring the state of the current computing node;
and updating a state table corresponding to the computing node at regular time based on the thread library of python, wherein the state table is used for recording the state of each computing node.
6. An asynchronous task processing apparatus, comprising:
the receiving module is used for receiving the asynchronous tasks submitted by the user and establishing an asynchronous task queue;
the system comprises a creating module, a task assigning module and a task processing module, wherein the creating module is used for creating and updating a task table and a task assigning table in real time, the task table is used for recording task information of each asynchronous task, and the task assigning table is used for recording computing node distribution information and task computing results of each asynchronous task;
the computing module is used for selecting a corresponding computing node to distribute computing tasks according to the task information of each asynchronous task;
and the sending module is used for collecting corresponding task calculation results and sending the task calculation results to the user when the calculation nodes finish the calculation tasks.
7. The apparatus of claim 6, further comprising:
the monitoring module is used for monitoring the state of each computing task distributed to the computing nodes; for each computing task, if the computing task is not completed after the preset execution time, the computing task is redistributed to computing nodes.
8. The apparatus of claim 6, wherein the monitoring module is further configured to:
monitoring the states of all the computing nodes;
correspondingly, the calculation module is specifically configured to:
and selecting corresponding computing nodes to distribute the computing tasks according to the task information of the asynchronous tasks and the states of the computing nodes.
9. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method according to any of claims 1 to 5.
10. A computer readable medium having computer readable instructions stored thereon which are executable by a processor to implement the method of any one of claims 1 to 5.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010616454.7A CN111831408A (en) | 2020-06-30 | 2020-06-30 | Asynchronous task processing method and device, electronic equipment and medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010616454.7A CN111831408A (en) | 2020-06-30 | 2020-06-30 | Asynchronous task processing method and device, electronic equipment and medium |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111831408A true CN111831408A (en) | 2020-10-27 |
Family
ID=72899914
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010616454.7A Pending CN111831408A (en) | 2020-06-30 | 2020-06-30 | Asynchronous task processing method and device, electronic equipment and medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111831408A (en) |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112486654A (en) * | 2020-12-08 | 2021-03-12 | 浙江诺诺网络科技有限公司 | Asynchronous slow task scheduling method, device, equipment and medium |
CN113051077A (en) * | 2021-03-26 | 2021-06-29 | 深圳市优必选科技股份有限公司 | User request processing method and device, terminal equipment and storage medium |
CN114327827A (en) * | 2021-12-28 | 2022-04-12 | 中国联合网络通信集团有限公司 | Task processing method and device and storage medium |
CN114625505A (en) * | 2022-03-11 | 2022-06-14 | 北京京东振世信息技术有限公司 | Waybill task execution method and device, electronic equipment and computer readable medium |
CN114936096A (en) * | 2021-11-09 | 2022-08-23 | 北京百度网讯科技有限公司 | Method and apparatus for scheduling operator |
CN114968532A (en) * | 2022-06-02 | 2022-08-30 | 北京达佳互联信息技术有限公司 | Task execution method and apparatus, electronic device, and computer-readable storage medium |
CN115098255A (en) * | 2022-06-17 | 2022-09-23 | 特赞(上海)信息科技有限公司 | Design method and system of distributed file asynchronous processing service and electronic equipment |
CN115665256A (en) * | 2022-10-26 | 2023-01-31 | 中国农业银行股份有限公司 | A task processing method, device, electronic device and storage medium |
CN115840680A (en) * | 2022-12-23 | 2023-03-24 | 中国电子科技集团公司第二十九研究所 | Multi-cooperative task monitoring system based on message bus |
WO2025065987A1 (en) * | 2023-09-29 | 2025-04-03 | Huawei Technologies Co., Ltd. | Method, apparatus and system for managing mission instance |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102063336A (en) * | 2011-01-12 | 2011-05-18 | 国网电力科学研究院 | Distributed computing multiple application function asynchronous concurrent scheduling method |
CN103092698A (en) * | 2012-12-24 | 2013-05-08 | 中国科学院深圳先进技术研究院 | System and method of cloud computing application automatic deployment |
-
2020
- 2020-06-30 CN CN202010616454.7A patent/CN111831408A/en active Pending
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102063336A (en) * | 2011-01-12 | 2011-05-18 | 国网电力科学研究院 | Distributed computing multiple application function asynchronous concurrent scheduling method |
CN103092698A (en) * | 2012-12-24 | 2013-05-08 | 中国科学院深圳先进技术研究院 | System and method of cloud computing application automatic deployment |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112486654A (en) * | 2020-12-08 | 2021-03-12 | 浙江诺诺网络科技有限公司 | Asynchronous slow task scheduling method, device, equipment and medium |
CN113051077A (en) * | 2021-03-26 | 2021-06-29 | 深圳市优必选科技股份有限公司 | User request processing method and device, terminal equipment and storage medium |
CN114936096A (en) * | 2021-11-09 | 2022-08-23 | 北京百度网讯科技有限公司 | Method and apparatus for scheduling operator |
CN114327827A (en) * | 2021-12-28 | 2022-04-12 | 中国联合网络通信集团有限公司 | Task processing method and device and storage medium |
CN114327827B (en) * | 2021-12-28 | 2025-03-21 | 中国联合网络通信集团有限公司 | A task processing method, device and storage medium |
CN114625505A (en) * | 2022-03-11 | 2022-06-14 | 北京京东振世信息技术有限公司 | Waybill task execution method and device, electronic equipment and computer readable medium |
CN114968532A (en) * | 2022-06-02 | 2022-08-30 | 北京达佳互联信息技术有限公司 | Task execution method and apparatus, electronic device, and computer-readable storage medium |
CN115098255A (en) * | 2022-06-17 | 2022-09-23 | 特赞(上海)信息科技有限公司 | Design method and system of distributed file asynchronous processing service and electronic equipment |
CN115665256A (en) * | 2022-10-26 | 2023-01-31 | 中国农业银行股份有限公司 | A task processing method, device, electronic device and storage medium |
CN115840680A (en) * | 2022-12-23 | 2023-03-24 | 中国电子科技集团公司第二十九研究所 | Multi-cooperative task monitoring system based on message bus |
WO2025065987A1 (en) * | 2023-09-29 | 2025-04-03 | Huawei Technologies Co., Ltd. | Method, apparatus and system for managing mission instance |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111831408A (en) | Asynchronous task processing method and device, electronic equipment and medium | |
WO2020211579A1 (en) | Processing method, device and system for distributed bulk processing system | |
US8533334B2 (en) | Message binding processing technique | |
CN108647104B (en) | Request processing method, server and computer readable storage medium | |
CN111625331A (en) | Task scheduling method, device, platform, server and storage medium | |
CN113886069B (en) | Resource allocation method, device, electronic device and storage medium | |
CN109033814B (en) | Intelligent contract triggering method, device, equipment and storage medium | |
WO2019056771A1 (en) | Distributed storage system upgrade management method and device, and distributed storage system | |
CN116483546B (en) | Distributed training task scheduling method, device, equipment and storage medium | |
CN112860387A (en) | Distributed task scheduling method and device, computer equipment and storage medium | |
WO2025176140A1 (en) | In-process resource control method and apparatus, and device and storage medium | |
CN112486638A (en) | Method, apparatus, device and storage medium for executing processing task | |
US10970098B2 (en) | Methods for sharing input-output device for process automation on a computing machine and devices thereof | |
CN112631994A (en) | Data migration method and system | |
CN113703930A (en) | Task scheduling method, device and system and computer readable storage medium | |
CN113626173B (en) | Scheduling method, scheduling device and storage medium | |
CN119829238A (en) | Task execution method, task execution device, storage medium, electronic device, and program product | |
CN119440879A (en) | Message processing method, device, equipment and storage medium | |
CN106598706B (en) | Method and device for improving stability of server and server | |
CN114911538A (en) | Starting method of running system and computing equipment | |
US11797342B2 (en) | Method and supporting node for supporting process scheduling in a cloud system | |
CN114390104A (en) | Process forensics system, method, apparatus, computer device and medium | |
CN114546279B (en) | IO request prediction method and device, storage node and readable storage medium | |
CN114896022B (en) | Method, device, equipment and storage medium for creating a virtual machine | |
CN109992374A (en) | An application instance management method, device and computer-readable storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |