CN111290841A - Task scheduling method and device, computing equipment and storage medium - Google Patents
Task scheduling method and device, computing equipment and storage medium Download PDFInfo
- Publication number
- CN111290841A CN111290841A CN201811501565.2A CN201811501565A CN111290841A CN 111290841 A CN111290841 A CN 111290841A CN 201811501565 A CN201811501565 A CN 201811501565A CN 111290841 A CN111290841 A CN 111290841A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- group
- processing
- fragment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The application discloses a task scheduling method, a task scheduling device, a computing device and a storage medium. The task scheduling method comprises the following steps: executing multiple generation operations to generate a plurality of task scheduling plans of data to be processed, wherein each task scheduling plan is used for describing an execution plan of processing tasks of a plurality of data fragments of the data to be processed, and the data fragments are stored in a cluster; and selecting one task scheduling plan with the shortest execution time from the plurality of task scheduling plans as a task execution mode for the data to be processed. The task scheduling scheme can heuristically find various task scheduling plans, and enables the locally executed tasks in different task scheduling plans to be different in amount, so that the task scheduling plan with the highest execution efficiency is selected as a task execution mode, and further load balance of cluster processing to-be-processed data is improved, and task execution efficiency is improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a task scheduling method and apparatus, a computing device, and a storage medium.
Background
As data processing technology evolves, clusters may be used to store data and to process data in a distributed manner. For example, some application schemes may utilize clusters to store and transcode video. To improve data processing efficiency, the cluster may apply a task scheduling policy to allocate data processing tasks. Clusters can be classified into homogeneous and heterogeneous modes. The homogeneous mode cluster means that the computing power of each computing node in the cluster is basically consistent. The cluster of heterogeneous patterns may include computing nodes of different computing capabilities. The data processing scheme may be applied in clusters of heterogeneous or homogeneous patterns. However, the load balancing of the existing task scheduling strategies needs to be improved.
Disclosure of Invention
According to an aspect of the present application, there is provided a task scheduling method, including: executing multiple generation operations to generate a plurality of task scheduling plans of data to be processed, wherein each task scheduling plan is used for describing an execution plan of processing tasks of a plurality of data fragments of the data to be processed, and the data fragments are stored in a cluster; and selecting one task scheduling plan with the shortest execution time from the plurality of task scheduling plans as a task execution mode for the data to be processed. Wherein, the generation operation executed each time comprises: dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different generation operations is different; allocating a processing task of any data fragment in the first data fragment group to a computing node storing the data fragment in the cluster, and using a task allocation mode of the data fragment in the first data fragment group as a first task allocation plan; allocating the processing task of any data fragment in the second data fragment group to a computing node which does not store the data fragment in the cluster, and using the task allocation mode of the data fragment in the second data fragment group as a second task allocation plan; and taking the distribution mode comprising the first task distribution plan and the second task distribution plan as a task scheduling plan generated by the current generation operation.
In some embodiments, the data to be processed is a video to be transcoded, the data slices are video slices, and the processing task amount of each data slice in the data slices is the transcoding task amount of each video slice.
In some embodiments, the method further comprises: before generating the plurality of task scheduling plans, acquiring the processing task amount of each data fragment in a plurality of data fragments of the data to be processed; and acquiring the computing capacity of each computing node in the cluster and the data transmission bandwidth among the computing nodes, wherein the computing capacity of each computing node is used for describing the task amount processed by each computing node in unit time.
In some embodiments, dividing the plurality of data slices into a first group of data slices and a second group of data slices comprises: in a first generation operation of the multiple generation operations, taking a total task amount of the multiple data slices as a total task amount of the first data slice group, and dividing the multiple data slices into a first data slice group and a second data slice group according to the total task amount of the first data slice group, wherein the first data slice group comprises the multiple data slices, and the second data slice group is empty; in any generation operation except the first generation operation of the multiple generation operations, the total task quantity of the first data slice group is set to be reduced by a quantity threshold value compared with the total task quantity of the first data slice group in the last generation operation, and the multiple data slices are divided into the first data slice group and the second data slice group according to the set total task quantity of the first data slice group.
In some embodiments, the dividing the plurality of data slices into the first data slice group and the second data slice group according to the set total task amount of the first data slice group includes: selecting a set of data slices from the plurality of data slices as the first data slice group by taking the set total task amount of the first data slice group as a reference; and adding the data slices which do not belong to the first data slice group in the plurality of data slices into a second data slice group.
In some embodiments, the allocating the processing task of any data slice in the first group of data slices to one computing node in the cluster that stores the data slice comprises: according to the processing task amount of the data fragments in the first data fragment group and the computing capacity of the computing nodes in the cluster, the processing task of any data fragment in the first data fragment group is distributed to one computing node which can finish the processing task of the data fragment at the earliest time in a plurality of computing nodes storing the data fragment.
In some embodiments, the allocating, according to the processing task amount of each data slice in a first data slice group and the computing capacity of each computing node in the cluster, a processing task of any data slice in the first data slice group to a computing node that completes the processing task of the data slice earliest in a plurality of computing nodes that store the data slice includes: and for the data fragments in the first data fragment group, sequentially determining the computing nodes corresponding to the processing tasks of the data fragments according to the descending order of the processing task amount. For any data fragment in the first data fragment, determining a computing node corresponding to a processing task of the data fragment includes: determining a plurality of computing nodes in the cluster, wherein the copies of the data fragments are stored in the cluster; determining an expected completion time point of each computing node in the plurality of computing nodes storing the copies of the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is as follows: the computing node adds a time value obtained by adding a task processing time length of the processing task of the data fragment to a starting time point of the processing task of the data fragment, and the task processing time length of each computing node is as follows: the ratio of the processing task amount of the data fragment to the computing capability of the computing node, and the sum of the starting time of the computing node to the processing task of the data fragment; and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes storing the copies of the data fragment as the computing node corresponding to the processing task of the data fragment.
In some embodiments, the allocating the processing task of any data slice in the second group of data slices to a computing node in the cluster that does not store the data slice includes: according to the processing task amount of the data fragments in the second data fragment group, the computing capacity of the computing nodes in the cluster and the data transmission bandwidth among the computing nodes, the processing task of any data fragment in the second data fragment group is distributed to one computing node which can finish the processing task of the data fragment at the earliest in the plurality of computing nodes which do not store the data fragment.
In some embodiments, the allocating, according to the processing task amount of a data slice in a second data slice group, the computing capabilities of the computing nodes in the cluster, and the transmission bandwidth between the computing nodes, a processing task of any data slice in the second data slice group to a computing node that completes the processing task of the data slice earliest among the computing nodes that do not store the data slice includes: and for the data fragments in the second data fragment group, sequentially determining the computing nodes corresponding to the processing tasks of the data fragments according to the descending order of the processing task amount. For any data fragment in the second data fragment, determining a computing node corresponding to a processing task of the data fragment includes: determining a plurality of computing nodes in the cluster that do not store the data fragment; determining an expected completion time point of each computing node in the plurality of computing nodes which do not store the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is as follows: the computing node adds a time value obtained by adding a task processing time length of the processing task of the data fragment to a starting time point of the processing task of the data fragment, and the task processing time length of each computing node is as follows: the sum of the ratio of the processing task amount of the data fragment to the computing capability of the computing node, the transmission time length of the data fragment to the computing node and the starting time length of the computing node for processing the data fragment; and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes as the computing node corresponding to the processing task of the data fragment.
In some embodiments, the selecting, from the plurality of task scheduling plans, one task scheduling plan with the shortest execution duration as a task execution mode for the to-be-processed data includes: for a task scheduling plan generated by any secondary generating operation, determining the completion time of each computing node in the cluster executing a task according to the task scheduling plan; selecting the maximum completion time length from the completion time lengths corresponding to the computing nodes as the execution time length corresponding to the task scheduling plan; and selecting one task scheduling plan with the shortest execution time as a task execution mode for the data to be processed.
According to an aspect of the present application, there is provided a task scheduling apparatus, comprising:
the system comprises a plan generating unit, a task scheduling unit and a task scheduling unit, wherein the plan generating unit is used for executing multiple generation operations and generating a plurality of task scheduling plans of data to be processed, each task scheduling plan is used for describing an execution plan of processing tasks of a plurality of data fragments of the data to be processed, and the data fragments are stored in a cluster; and
and the plan selection unit is used for selecting one task scheduling plan with the shortest execution time from the plurality of task scheduling plans as a task execution mode for the data to be processed.
Wherein the generation operation executed each time by the plan generation unit includes: dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different generation operations is different; allocating a processing task of any data fragment in the first data fragment group to a computing node storing the data fragment in the cluster, and using a task allocation mode of the data fragment in the first data fragment group as a first task allocation plan; allocating the processing task of any data fragment in the second data fragment group to a computing node which does not store the data fragment in the cluster, and using the task allocation mode of the data fragment in the second data fragment group as a second task allocation plan; and taking the distribution mode comprising the first task distribution plan and the second task distribution plan as a task scheduling plan generated by the current generation operation.
According to an aspect of the application, there is provided a computing device comprising: a processor; a memory; and one or more programs stored in the memory and configured to be executed by the processor, the one or more programs including instructions for performing a task scheduling method according to the present application.
According to an aspect of the present application, there is provided a storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform a task scheduling method according to the present application.
In conclusion, according to the task scheduling scheme of the present application, various task scheduling plans can be heuristically discovered, and the amount of locally executed tasks in different task scheduling plans is different, so that the task scheduling plan with the highest execution efficiency is selected as a task execution mode, thereby improving load balance of cluster processing of data to be processed and improving task execution efficiency. In particular, the task scheduling scheme fully considers the transmission time of data among different computing nodes, so that the modeling accuracy of the task scheduling scheme is improved, and the execution efficiency of the data to be processed is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
FIG. 1A illustrates a schematic diagram of an application scenario 100 in accordance with some embodiments of the present application;
FIG. 1B is a schematic diagram illustrating a hierarchy of videos in accordance with some embodiments of the present application;
FIG. 2 illustrates a flow diagram of a task scheduling method 200 according to some embodiments of the present application;
FIG. 3 illustrates a flow diagram of a method 300 of performing a generate operation according to some embodiments of the present application;
FIG. 4 illustrates a flow diagram of a method 400 of partitioning a group of data slices according to some embodiments of the present application;
FIG. 5 illustrates a flow diagram of a method 500 of determining a compute node corresponding to a processing task of a data slice according to some embodiments of the present application;
FIG. 6 illustrates a flow diagram of a method 600 of determining a compute node corresponding to a processing task for a data slice according to some embodiments of the present application;
FIG. 7 illustrates a flow diagram of a method 700 of selecting a task scheduling plan, according to some embodiments of the present application;
FIG. 8 illustrates a flow diagram of a task scheduling method 800 according to some embodiments of the present application;
FIG. 9 illustrates a schematic diagram of a task scheduler 900 according to some embodiments of the application; and
FIG. 10 illustrates a block diagram of the components of a computing device according to some embodiments of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In some application scenarios, a cluster may store various data to be processed and may perform processing tasks on the data to be processed. The cluster can balance the task load of each computing node by using a task scheduling scheme, so that the data processing efficiency of the cluster is improved. Here, the data to be processed is, for example, a video to be transcoded, but is not limited thereto. In some embodiments, the data to be processed may include a plurality of data slices. The data slices may be, for example, video slices into which the video content is split. The data slices may be stored in a distributed manner in the cluster. The cluster may distribute the processing tasks for the data slices based on a task scheduling scheme to balance the load. An application scenario of the task scheduling scheme according to the present application is exemplarily described below by taking a video processing scenario as an example. FIG. 1A illustrates a schematic diagram of an application scenario 100 according to some embodiments of the present application. As shown in fig. 1A, the application scenario includes a video storage device 110, a segmentation device 120, a prediction device 130, a cluster 140, and a merge device 150.
The video storage device 110 may be, for example, a user terminal device or a server or the like capable of providing video to be transcoded. Here, the user terminal device may include, for example, but is not limited to, a camera, a palmtop computer, a wearable computing device, a Personal Digital Assistant (PDA), a tablet computer, a laptop computer, a desktop computer, a mobile phone, a smartphone, an Enhanced General Packet Radio Service (EGPRS) mobile phone, a media player, a navigation device, a game console, a television, or a combination of any two or more of these or other data processing devices. The server may be, for example, a hardware-independent server node or a virtual server, etc. The video storage device 110 may provide various video content such as movies, television shows, art programs, sporting events, and user live broadcasts.
Splitting device 120 may split the video content from video storage device 110 to obtain a plurality of video slices. For a piece of video content, the splitting device 120 may employ various suitable splitting strategies to control the number and size of video slices. Here, the dividing device 120 may be implemented as a server or a user terminal device. FIG. 1B shows a schematic diagram of a hierarchy of videos for some embodiments of the present application. The splitting device 120 may divide the video into a sequence of video slices. A sequence of video slices may comprise, for example, video slices 1, 2, and 3. Video slice 2 may include header information for the video slice and a number of groups of pictures (GOP), for example, including groups of pictures 1, 2, and 3. The image group 1 may for example comprise header information of the image group and a plurality of image frames, for example comprising image frames 1, 2 and 3. The image frame 1 may comprise a plurality of groups of macroblocks (abbreviated as GOBs), for example groups of macroblocks 1, 2, 3. The Macroblock group 3 may include, for example, a plurality of macroblocks (abbreviated as MBs), such as macroblocks 1, 2, 3, 4, and 5. Each macroblock may include a luminance pixel block and a chrominance pixel block.
The prediction device 130 may predict the amount of transcoding tasks for each video slice from the segmentation device 120. The amount of transcoding tasks may also be referred to herein as transcoding complexity. The transcoding complexity of a video slice may be expressed as the product of the computational power of a computing device processing the video slice and the required transcoding time. The computing power of a computing device is used to describe the computing speed of the computing device when processing a computing task. The prediction period 130 may be implemented as a server or a user terminal device.
Cluster 140 may include a plurality of server nodes. The cluster 140 may be any of a variety of server cluster architectures, which are not limited in this application. From a software perspective, cluster 140 may be, for example, a cluster of Hadoop-based servers. In addition, from a hardware perspective, the cluster 140 may be a homogeneous model cluster (i.e., the computing capabilities of different computing nodes are consistent) or a heterogeneous model cluster (i.e., the computing capabilities of different computing nodes may be different). The cluster 140 may store video fragments of the video to be transcoded in a distributed manner, and may also manage description information of the video to be transcoded. Here, the description information may include a transcoding task amount of each video slice of the video to be transcoded. Cluster 140 may utilize server nodes for transcoding operations. In some embodiments, the server nodes in cluster 140 may include a management node 141 and a plurality of compute nodes (e.g., 142, 143, and 144). The computing node may be configured to store the video slices and perform transcoding operations on the video slices. In this way, the cluster 140 may generate video slices in various bitrate formats. The rate formats may be, for example, low definition, standard definition, 720P, 1080P, etc., from a resolution point of view. The management node 141 may generate a transcoding scheduling policy according to the transcoding task amount of each video slice, and control the computing node to perform transcoding operation.
The merge device 150 may splice the video slices transcoded by the cluster 140 into a complete video. The merge device 150 may also transmit the complete video to the video storage device 110. The combining device 150 may be implemented as a server or a user terminal device.
The devices in the application scenario 100 may communicate over one or more networks (not shown). Examples of the one or more networks include a Local Area Network (LAN) and a Wide Area Network (WAN), among others. Embodiments of the application may implement one or more networks using any network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Global System for Mobile communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, WiFi, Voice Over IP (VOIP), Wi-MAX, or any other suitable communication protocol.
FIG. 2 illustrates a flow diagram of a task scheduling method 200 according to some embodiments of the present application. Method 200 may be performed by various data processing clusters, but is not limited thereto. Taking the application scenario of fig. 1 as an example, the method 200 may be performed by a management node 141 in the cluster 140.
In step S201, a plurality of generation operations are performed, and a plurality of task scheduling plans for data to be processed are generated. Each generation operation may generate a task schedule. Each task scheduling plan is used for describing an execution plan of processing tasks of a plurality of data fragments of data to be processed. Multiple data slices may be stored in cluster 140. In some embodiments, the data to be processed is, for example, video to be transcoded. The plurality of data slices are a plurality of video slices. The processing task amount of each data fragment is the transcoding task amount of each video fragment. To improve storage security, cluster 140 may store multiple copies of each video slice.
It should be noted that, in step S201, each generation operation may be performed sequentially, or multiple generation operations may be performed in parallel, which is not limited in this application. In some embodiments, the manner in which step S201 performs the generating operation at any time may be implemented as method 300.
As shown in fig. 3, in step S301, a plurality of data slices of data to be processed are divided into a first data slice group and a second data slice group. And the total task quantity of the first data fragment groups corresponding to different secondary generating operations is different. The amount of processing tasks for one data slice may also be referred to as computational complexity. A data slice computational complexity may be expressed as the product of the computational power of a computing device processing the data slice and the required computational time. In some embodiments, cluster 140 may obtain description information from predictive device 130. The description information may include the transcoding task volume of the video slice. In some embodiments, the cluster 140 may analyze the transcoding task volume of the video segments, so as to obtain the transcoding task volume of each video segment in the video to be transcoded. It is further noted that, in the embodiments of the present application, various partitioning strategies may be adopted to partition the total task amount of the first data slice group in different generation operations. For example, embodiments of the present application may be such that: the total task amount of the first data slice groups corresponding to the multiple generation operations meets the arithmetic progression when the first data slice groups are sequenced. The difference value of the equal difference sequence is, for example, the smallest processing task amount among the processing task amounts of the plurality of data slices.
In step S302, a processing task of any data slice in the first data slice group is allocated to one computing node storing the data slice in the cluster 140, and a task allocation manner of the data slice in the first data slice group is used as a first task allocation plan. Step S302 may employ various load balancing policies to distribute the processing tasks of the data slices in the first data slice group to the computing nodes in the cluster 140.
In step S303, a processing task of any data slice in the second data slice group is allocated to a computing node in the cluster 140 that does not store the data slice, and a task allocation manner of the data slice in the second data slice group is used as a second task allocation plan. Step S303 may adopt various load balancing strategies to distribute the processing tasks of the data slices in the first data slice group to the computing nodes in the cluster 140.
In step S304, the assignment method including the first task assignment plan and the second task assignment plan is set as one task scheduling plan generated in the current generation operation.
In summary, through the combination of steps S302 and S303, the method 300 may locally execute the processing task corresponding to the first data slice group, and remotely execute the processing task corresponding to the second data slice group. On this basis, through the combination of steps S301 to S303, step S201 may obtain multiple task schedules by using multiple generation operations, and make the locally executed task amounts in different task schedules different (i.e., the total task amount of the first data slice group is different).
In step S202, one task scheduling plan with the shortest execution duration is selected from the plurality of task scheduling plans as a task execution manner for the to-be-processed data.
In summary, through the combination of steps S201 and S202, the task scheduling method 200 may heuristically find various task scheduling plans, and make the locally executed tasks in different task scheduling plans different in amount, so as to select the task scheduling plan with the highest execution efficiency as the task execution manner, thereby improving the load balance of the cluster 140 for processing the data to be processed and improving the task execution efficiency.
In some embodiments, the operation manner of dividing the first data slice group and the second data slice group in step S301 may include steps S3011 and S3012. In the first generation operation of the multiple generation operations, step S301 may execute step S3011, taking the total task volume of the multiple data slices as the total task volume of the first data slice group, and dividing the multiple data slices into the first data slice group and the second data slice group according to the total task volume of the first data slice group. Wherein the first group of data slices comprises the plurality of data slices and the second group of data slices is empty. In other words, the data slice to be processed is divided into the first group of data slices in its entirety.
In any of the generation operations other than the first generation operation described above, step S301 may perform step S3012, and set the total task amount of the first data slice group to be decreased by an amount threshold from the total task amount of the first data slice group in the last generation operation. When the generating operation is performed iteratively a plurality of times, the amount threshold may be a step value for adjusting the total task amount of the first data slice group. For example, step S201 may iteratively perform 4 generation operations. The total task amount of the first data fragment group set in each secondary generating operation is C1、C2、C3、C4. The quantity threshold may representIs CstepFor example, the smallest processing task amount among the processing task amounts of the plurality of data slices to be processed. C1And C2Has a difference of Cstep. Similarly C2And C3Also has a difference of Cstep. On this basis, step S3012 divides the plurality of data slices into a first data slice group and a second data slice group according to the set total task amount of the first data slice group. In some embodiments, the total task size of the first group of data slices may be set to zero (i.e. the data slices are all allocated into the second group of data slices).
In some embodiments, step S3012 is implemented as method 400. In step S401, a set of data slices is selected from the plurality of data slices as a first data slice group based on the set total task amount of the first data slice group. In other words, the set total task amount of the first data slice group is a reference value, and S401 may make the actual total task amount of the selected data slice close to the set total task amount. For example, step S401 may be such that the task volume of the selected set of data slices is as close as possible to and does not exceed the set total task volume.
In step S402, a data slice that does not belong to the first data slice group among all data slices corresponding to the data to be processed is added to the second data slice group. To more visually explain the operation of step S3012, the following example is given. For example, the data slices to be processed include a total of N data slices. Set of data slices V ═ V (V)1，v2，v3，…，vN) A set of data sizes for a data slice may be denoted as S ═ S (S)1，s2，s3，…，sN) The set of processing task amounts for each data slice may be represented as C (C)1，c2，c3，…，cN). Wherein the data is sliced viHas a data size of siThe processing task amount is ci. i is an integer and has a value range of [1, N]. The first group of data slices may be denoted as L and the first group of data slices may be denoted as R. The total task size of the first data slice group L may be denoted CLSecond data scoreThe total task volume of a slice group R can be represented as CR。
The total task amount of all data fragments corresponding to the data to be processed can be represented as Csum：
In some embodiments, a more specific implementation of step S302 is: the processing task of any data slice in the first data slice group is allocated to one of the plurality of computing nodes storing the data slice, which can complete the processing task of the data slice earliest, according to the processing task amount of the data slice in the first data slice group and the computing capacity of the computing nodes in the cluster 140. Here, step S302 may enable the processing tasks of the data slices in the first data slice group to be locally executed and enable the load balancing of the computing nodes in the cluster 140 that contain the data slices in the first data slice group by allocating the processing tasks of the data slices to the computing node that completes the data earliest.
In some embodiments, for the data slices in the first data slice group, step S302 may sequentially determine, in descending order of the processing task amount, the computing nodes corresponding to the processing tasks of the data slices. Here, step S302 may avoid processing the idle state after some computing nodes in the cluster 140 complete task processing in a descending order of the processing task amount, thereby improving load balance of the cluster 140. For any one of the first data slices, step S302 may determine, by executing the method 500, a computing node corresponding to a processing task of the data slice.
As shown in fig. 5, in step S501, a plurality of compute nodes in the cluster 140 that store copies of the data slice are determined. For example, data slice a includes 3 copies. The 3 copies are stored at compute nodes b1, b2, and b3, respectively, of cluster 140. Step S501 may determine, for data slice A, compute nodes b1, b2, and b3 as candidate nodes to process the processing tasks of data slice A.
In step S502, the sub-system storing the data slice is determinedIn the plurality of computing nodes, each computing node is used for expecting the completion time point of the processing task of the data fragmentation. For example, suppose that the compute nodes b1, b2, and b3 perform the processing tasks of data slice a, respectively. Step S502 may determine the expected completion time of each of the compute nodes b1, b2, and b 3. Wherein the expected completion time point of the processing task of the data fragment by each computing node is as follows: and the time value is obtained by adding the task processing time length of the processing task of the data fragment to the starting time point of the processing task of the data fragment executed by the computing node. The task processing duration of each computing node is: the ratio of the processing task amount of the data fragment to the computing capability of the computing node, and the sum of the starting time of the computing node to the processing task of the data fragment. Taking data slice A as an example, the starting time point t of the computing node b1 for the data slice A may be based on the total amount of tasks C that the computing node b1 is to perform before performing the processing task of the data slice AtIs determined by the total execution time period T. Starting time t ═ t0+T。t0Indicating the start of the first processing of a data slice by compute node b 1.
Calculating the duration T of the anyware processing of the node b1 to the data fragment AA＝cA/pb1+Ob1Wherein c isAIndicating the amount of processing tasks for data slice a. p is a radical ofb1Representing the computing power of computing node b 1. O isb1And the starting time of the processing task of processing one data fragment by the computing node is shown.
In step S503, one of the plurality of computing nodes storing the copy of the data fragment is selected as the computing node corresponding to the processing task of the data fragment, the computing node having the earliest expected completion time point. Upon determining that computing node b2 may complete the processing task of data slice a earliest, step S503 may assign the processing task of data slice a to computing node b2, i.e., select computing node b2 as the computing node corresponding to the processing task of data slice a.
In summary, the method 500 may allocate the processing tasks of the data fragments in the first data fragment group to the local computing nodes for execution, and may determine the task allocation manner of each data fragment in a manner of minimum completion time (i.e., a manner of selecting the computing node that completes the data fragment earliest in step S503), so as to maintain load balancing of the cluster 140.
In some embodiments, step S303 may allocate the processing task of any data slice in the second data slice group to one of the plurality of computing nodes that does not store the data slice, which can complete the processing task of the data slice earliest, according to the processing task amount of the data slice in the second data slice group, the computing capabilities of the computing nodes in the cluster 140, and the data transmission bandwidth between the computing nodes.
In some embodiments, for the data slices in the second data slice group, step S303 may sequentially determine, according to a descending order of the transcoding task amount, the computing nodes corresponding to the processing tasks of the data slices. By determining the computing nodes corresponding to the data fragments in the second data fragment group according to the descending order of the transcoding task amount, step S303 can prevent some computing nodes in the cluster 140 from processing an idle state after completing task processing, thereby improving load balance of the cluster 140. For any data slice in the second data slice, step S303 may determine, by executing the method 600, a computing node corresponding to the processing task of the data slice.
As shown in FIG. 6, in step S601, a number of compute nodes in the cluster 140 that do not store the data slice are determined. For example, the set of compute nodes J of cluster 140 is (J1, J2, …, jm), and m is a positive integer. For example, the compute nodes in J that store data slice B are J2, J3, and J4. Step S601 may take the remaining compute nodes in set J, excluding compute nodes J2, J3, and J4, as alternative compute nodes for the processing task of combing data slice B.
In step S602, an expected completion time point of a processing task for the data slice by each of the plurality of computing nodes that do not store the data slice is determined. Wherein the expected completion time point of the processing task of the data fragment by each computing node is as follows: and the time value is obtained by adding the task processing time length of the processing task of the data fragment to the starting time point of the processing task of the data fragment executed by the computing node. The task processing duration of each computing node is: the sum of the ratio of the processing task amount of the data fragment to the computing capability of the computing node, the transmission time of the data fragment to the computing node and the starting time of the transcoding task of the data fragment by the computing node.
To calculate node j1Processing tasks for processing data slice B, compute node j1Task processing duration TB＝cB/pj1+Oj1+dB,j1Wherein, cBRepresenting the amount of processing tasks, p, of a data slice Bj1Represents the computing power, O, of computing node j1j1Indicating the start time of the processing task of the computing node j1 on the data slice B. Here, the starting time of any one computing node for different data fragments may be a constant. dB,j1Indicating the transmission duration of the data slice B to the computing node j 1.
In step S603, one of the plurality of computing nodes that do not store the data slice is selected as the computing node corresponding to the processing task of the data slice, the computing node having the earliest expected completion time point.
In summary, the method 600 may consider the data transmission time of the data slice when the processing task of the data slice in the second data slice group is allocated according to the minimum completion time mode, so as to improve the prediction accuracy of the completion duration of the task scheduling plan (i.e., improve the modeling accuracy of the task scheduling plan), and further improve the load balance of the cluster and the processing efficiency of the data to be processed.
In some embodiments, step S202 may be implemented as method 700.
As shown in fig. 7, in step S701, for the task scheduling plan generated by any secondary generation operation, the completion time of executing the task according to the task scheduling plan by each computing node in the cluster 140 is determined. For example, the computing node set J of the cluster 140 is (J1, J2, …, jm), and the total task amount allocated on the computing node ji is Cji. i is a positive integer and has a value range of [1, m]. Compute node ji execution CjiIs Tji。
In step S702, the largest completion time length is selected from the completion time lengths corresponding to the respective computation nodes as the execution time length corresponding to the task scheduling plan. Here, the execution time period refers to a processing time period for the cluster 140 to complete the task scheduling plan. For example, for the task scheduling plan P, the computation node in the set J with the shortest completion time is J5. The completion duration of computing node j5 is Tj5. Step S702 may convert Tj5As the execution time length of the task scheduling plan P.
In step S703, a task scheduling plan with the shortest execution duration is selected as the task execution mode for the data to be processed. In summary, the method 700 may compare the execution durations of the plurality of task scheduling plans, so as to select the task scheduling plan with the shortest execution duration as the task execution manner of the data to be processed, thereby improving the processing efficiency of the data to be processed.
FIG. 8 illustrates a flow diagram of a method 800 of task scheduling according to some embodiments of the present application. The task scheduling method 800 may be performed by the cluster 140, for example.
As shown in fig. 8, in step S801, a processing task amount of each data slice in a plurality of data slices of the data to be processed is acquired.
In step S802, the computing power of each computing node in the cluster 140 and the data transmission bandwidth between the computing nodes are obtained, where the computing power of each computing node is used to describe the task amount processed by each computing node in a unit time.
In step S803, a plurality of generation operations are performed to generate a plurality of task scheduling plans for the data to be processed. Each generation operation may generate a task schedule. More specific implementation of step S803 is consistent with step S201, and is not described here.
In step S804, one task scheduling plan with the shortest execution duration is selected from the plurality of task scheduling plans as a task execution manner for the to-be-processed data. More specific implementation of step S804 is consistent with step S202, and is not described herein again.
Fig. 9 illustrates a schematic diagram of a task scheduler 900 according to some embodiments of the application. The management node 141 of the cluster 140 may comprise, for example, a task scheduler 900. The apparatus 900 may comprise a plan generating unit 901 and a plan selecting unit 902.
The plan generating unit 901 may perform a plurality of generating operations to generate a plurality of task scheduling plans for data to be processed. Each task scheduling plan is used for describing an execution plan of processing tasks of a plurality of data fragments of the data to be processed. The plurality of data slices are stored in a cluster.
In some embodiments, the generation operation performed by the plan generation unit 901 each time includes: dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different generation operations is different; allocating a processing task of any data fragment in the first data fragment group to a computing node storing the data fragment in the cluster, and using a task allocation mode of the data fragment in the first data fragment group as a first task allocation plan; allocating the processing task of any data fragment in the second data fragment group to a computing node which does not store the data fragment in the cluster, and using the task allocation mode of the data fragment in the second data fragment group as a second task allocation plan; and taking the distribution mode comprising the first task distribution plan and the second task distribution plan as a task scheduling plan generated by the current generation operation.
In some embodiments, the data to be processed is a video to be transcoded, the data slices are video slices, and the processing task amount of each data slice in the data slices is the transcoding task amount of each video slice.
In some embodiments, the apparatus 900 may further include a first obtaining unit 903 and a second obtaining unit 904. Before generating the plurality of task scheduling plans, the first obtaining unit 903 may obtain a processing task amount of each data slice in a plurality of data slices of the data to be processed. The second obtaining unit 904 may obtain the computing power of each computing node in the cluster 140 and the data transmission bandwidth between each computing node. The computing capacity of each computing node is used for describing the task amount processed by each computing node in unit time.
In some embodiments, the plan generating unit 901 may divide the plurality of data slices into the first data slice group and the second data slice group according to the total task amount of the first data slice group, using the total task amount of the plurality of data slices as the total task amount of the first data slice group in the first generating operation of the plurality of generating operations. Wherein the first group of data slices comprises the plurality of data slices, and the second group of data slices is empty. In any one of the generation operations other than the first generation operation of the multiple generation operations, the plan generation unit 901 may set the total task amount of the first data slice group to be decreased by an amount threshold from the total task amount of the first data slice group in the last generation operation, and divide the plurality of data slices into the first data slice group and the second data slice group according to the set total task amount of the first data slice group.
In some embodiments, the plan generating unit 901 selects a set of data slices from the plurality of data slices as the first data slice group based on the set total task amount of the first data slice group. The plan generation unit 901 may further add a data slice, which does not belong to the first data slice group, of the plurality of data slices to the second data slice group.
In some embodiments, the plan generating unit 901 allocates the processing task of any one data slice in the first data slice group to one of the plurality of computing nodes storing the data slice, which can complete the processing task of the data slice earliest, according to the processing task amount of the data slice in the first data slice group and the computing capacity of the computing nodes in the cluster.
In some embodiments, for the data slices in the first data slice group, the plan generating unit 901 sequentially determines the computing nodes corresponding to the processing tasks of the data slices in the descending order of the processing task amount. For any data fragment in the first data fragment, the plan generation unit 901 may determine the computing node corresponding to the processing task of the data fragment according to the following manner: determining a plurality of computing nodes in the cluster, wherein the copies of the data fragments are stored in the cluster; determining an expected completion time point of each computing node in the plurality of computing nodes storing the copies of the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is as follows: the computing node adds a time value obtained by adding a task processing time length of the processing task of the data fragment to a starting time point of the processing task of the data fragment, and the task processing time length of each computing node is as follows: the ratio of the processing task amount of the data fragment to the computing capability of the computing node, and the sum of the starting time of the computing node to the processing task of the data fragment; and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes storing the copies of the data fragment as the computing node corresponding to the processing task of the data fragment.
In some embodiments, the plan generating unit 901 may allocate the processing task of any data slice in the second data slice group to one of the plurality of computing nodes that does not store the data slice and is capable of completing the processing task of the data slice earliest according to the processing task amount of the data slice in the second data slice group, the computing capabilities of the computing nodes in the cluster, and the data transmission bandwidth between the computing nodes.
In some embodiments, for the data slices in the second data slice group, the plan generation unit 901 may sequentially determine, according to the descending order of the processing task amount, the computing nodes corresponding to the processing tasks of the data slices. For any data fragment in the second data fragment, the plan generation unit 901 may determine the computing node corresponding to the processing task of the data fragment according to the following manner: determining a plurality of computing nodes in the cluster that do not store the data fragment; determining an expected completion time point of each computing node in the plurality of computing nodes which do not store the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is as follows: the computing node adds a time value obtained by adding a task processing time length of the processing task of the data fragment to a starting time point of the processing task of the data fragment, and the task processing time length of each computing node is as follows: the sum of the ratio of the processing task amount of the data fragment to the computing capability of the computing node, the transmission time length of the data fragment to the computing node and the starting time length of the computing node for processing the data fragment; and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes as the computing node corresponding to the processing task of the data fragment.
The plan selecting unit 902 may select one task scheduling plan with the shortest execution time from the plurality of task scheduling plans as a task execution mode for the to-be-processed data.
In some embodiments, for any secondary generation operation generated task schedule, the plan selection unit 902 determines a completion time for each computing node in the cluster to execute a task according to the task schedule. The plan selection unit 902 may further select a maximum completion time length from the completion time lengths corresponding to the respective computing nodes as an execution time length corresponding to the task scheduling plan. The plan selecting unit 902 may further select a task scheduling plan with the shortest execution time as a task execution mode for the to-be-processed data.
In summary, the task scheduling apparatus 900 according to the present application may heuristically find various task scheduling plans, and make the locally executed tasks in different task scheduling plans different in quantity, so as to select the task scheduling plan with the highest execution efficiency as the task execution manner, thereby improving load balancing of cluster processing of data to be processed and improving task execution efficiency. In particular, the task scheduling device 900 of the present application fully considers the transmission time of data between different computing nodes, thereby improving the modeling accuracy of the task scheduling scheme, and further improving the execution efficiency of the data to be processed.
FIG. 10 illustrates a block diagram of the components of a computing device. Here, the computing device may be implemented, for example, as a management node 141 of a cluster 140. As shown in fig. 10, the computing device includes one or more processors (CPUs) 1002, a communications module 1004, a memory 1006, a user interface 1010, and a communications bus 1008 for interconnecting these components.
The processor 1002 can receive and transmit data via the communication module 1004 to enable network communications and/or local communications.
The user interface 1010 includes one or more output devices 1012 including one or more speakers and/or one or more visual displays. The user interface 1410 also includes one or more input devices 1014. The user interface 1010 may receive, for example, but not limited to, an instruction of a remote controller.
The memory 1006 may be a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; or non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 1006 stores a set of instructions executable by the processor 1002, including: an operating system 1016, and applications 1018.
Operating system 1016 includes programs for handling various basic system services and for performing hardware dependent tasks. In some embodiments, application 1018 may comprise task scheduler 900 shown in FIG. 9.
In addition, each of the embodiments of the present application can be realized by a data processing program executed by a data processing apparatus such as a computer. It is clear that a data processing program constitutes the present application.
Further, the data processing program, which is generally stored in one storage medium, is executed by directly reading the program out of the storage medium or by installing or copying the program into a storage device (such as a hard disk and/or a memory) of the data processing device. Such a storage medium therefore also constitutes the present application. The storage medium may use any type of recording means, such as a paper storage medium (e.g., paper tape, etc.), a magnetic storage medium (e.g., a flexible disk, a hard disk, a flash memory, etc.), an optical storage medium (e.g., a CD-ROM, etc.), a magneto-optical storage medium (e.g., an MO, etc.), and the like.
The present application therefore also discloses a non-volatile storage medium having stored therein a data processing program for executing any one of the embodiments of the task scheduling method described above.
In addition, the method steps described in this application may be implemented by hardware, for example, logic gates, switches, Application Specific Integrated Circuits (ASICs), programmable logic controllers, embedded microcontrollers, and the like, in addition to data processing programs. Such hardware capable of implementing the methods described herein may also constitute the present application.
The above description is only exemplary of the present application and should not be taken as limiting the present application, and any modifications, equivalents, improvements and the like that are made within the spirit and principle of the present application should be included in the scope of the present application.
Claims (13)
1. A method for task scheduling, the method comprising:
executing multiple generation operations to generate a plurality of task scheduling plans of data to be processed, wherein each task scheduling plan is used for describing an execution plan of processing tasks of a plurality of data fragments of the data to be processed, and the data fragments are stored in a cluster; and
selecting one task scheduling plan with the shortest execution time from the plurality of task scheduling plans as a task execution mode for the data to be processed;
wherein, the generation operation executed each time comprises:
dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different generation operations is different;
allocating a processing task of any data fragment in the first data fragment group to a computing node storing the data fragment in the cluster, and using a task allocation mode of the data fragment in the first data fragment group as a first task allocation plan;
allocating the processing task of any data fragment in the second data fragment group to a computing node which does not store the data fragment in the cluster, and using the task allocation mode of the data fragment in the second data fragment group as a second task allocation plan; and
and taking the distribution mode comprising the first task distribution plan and the second task distribution plan as a task scheduling plan generated by the current generation operation.
2. The method of claim 1, wherein the data to be processed is a video to be transcoded, the data slices are video slices, and a processing task amount of each data slice in the data slices is a transcoding task amount of each video slice.
3. The method of claim 1, wherein the method further comprises:
before generating the plurality of task scheduling plans, acquiring the processing task amount of each data fragment in a plurality of data fragments of the data to be processed;
and acquiring the computing capacity of each computing node in the cluster and the data transmission bandwidth among the computing nodes, wherein the computing capacity of each computing node is used for describing the task amount processed by each computing node in unit time.
4. The method of claim 1, wherein dividing the plurality of data slices into a first group of data slices and a second group of data slices comprises:
in a first generation operation of the multiple generation operations, taking a total task amount of the multiple data slices as a total task amount of the first data slice group, and dividing the multiple data slices into a first data slice group and a second data slice group according to the total task amount of the first data slice group, wherein the first data slice group comprises the multiple data slices, and the second data slice group is empty;
in any generation operation except the first generation operation of the multiple generation operations, the total task quantity of the first data slice group is set to be reduced by a quantity threshold value compared with the total task quantity of the first data slice group in the last generation operation, and the multiple data slices are divided into the first data slice group and the second data slice group according to the set total task quantity of the first data slice group.
5. The method of claim 4, wherein the dividing the plurality of data slices into a first group of data slices and a second group of data slices according to the set total workload of the first group of data slices comprises:
selecting a set of data slices from the plurality of data slices as the first data slice group by taking the set total task amount of the first data slice group as a reference;
and adding the data slices which do not belong to the first data slice group in the plurality of data slices into a second data slice group.
6. The method of any of claims 1-5, wherein the assigning the processing task of any of the first group of data slices to a compute node in the cluster that stores the data slice comprises:
according to the processing task amount of the data fragments in the first data fragment group and the computing capacity of the computing nodes in the cluster, the processing task of any data fragment in the first data fragment group is distributed to one computing node which can finish the processing task of the data fragment at the earliest time in a plurality of computing nodes storing the data fragment.
7. The method of claim 6, wherein the allocating the processing task of any one data slice in the first data slice group to one of the plurality of computing nodes storing the data slice that completes the processing task of the data slice earliest according to the processing task amount of each data slice in the first data slice group and the computing capacity of each computing node in the cluster comprises:
for the data fragments in the first data fragment group, sequentially determining the computing nodes corresponding to the processing tasks of the data fragments according to the descending order of the processing task amount;
for any data fragment in the first data fragment, determining a computing node corresponding to a processing task of the data fragment includes:
determining a plurality of computing nodes in the cluster, wherein the copies of the data fragments are stored in the cluster;
determining an expected completion time point of each computing node in the plurality of computing nodes storing the copies of the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is as follows: the computing node adds a time value obtained by adding a task processing time length of the processing task of the data fragment to a starting time point of the processing task of the data fragment, and the task processing time length of each computing node is as follows: the ratio of the processing task amount of the data fragment to the computing capability of the computing node, and the sum of the starting time of the computing node to the processing task of the data fragment;
and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes storing the copies of the data fragment as the computing node corresponding to the processing task of the data fragment.
8. The method of any of claims 1-5, wherein the assigning the processing task of any of the data slices in the second group of data slices to a compute node in the cluster that does not store the data slice comprises:
according to the processing task amount of the data fragments in the second data fragment group, the computing capacity of the computing nodes in the cluster and the data transmission bandwidth among the computing nodes, the processing task of any data fragment in the second data fragment group is distributed to one computing node which can finish the processing task of the data fragment at the earliest in the plurality of computing nodes which do not store the data fragment.
9. The method of claim 8, wherein the allocating the processing task of any data slice in the second data slice group to one of the plurality of computing nodes that does not store the data slice at the earliest time to complete the processing task of the data slice according to the processing task amount of the data slice in the second data slice group, the computing capabilities of the computing nodes in the cluster, and the transmission bandwidth among the computing nodes comprises:
for the data fragments in the second data fragment group, sequentially determining the computing nodes corresponding to the processing tasks of the data fragments according to the descending order of the processing task amount;
for any data fragment in the second data fragment, determining a computing node corresponding to a processing task of the data fragment includes:
determining a plurality of computing nodes in the cluster that do not store the data fragment;
determining an expected completion time point of each computing node in the plurality of computing nodes which do not store the data fragment for the processing task of the data fragment, wherein the expected completion time point of each computing node for the processing task of the data fragment is as follows: the computing node adds a time value obtained by adding a task processing time length of the processing task of the data fragment to a starting time point of the processing task of the data fragment, and the task processing time length of each computing node is as follows: the sum of the ratio of the processing task amount of the data fragment to the computing capability of the computing node, the transmission time length of the data fragment to the computing node and the starting time length of the computing node for processing the data fragment;
and selecting one computing node with the earliest expected completion time point from the plurality of computing nodes as the computing node corresponding to the processing task of the data fragment.
10. The method according to claim 1, wherein the selecting one of the task schedules with the shortest execution duration as the task execution mode for the data to be processed comprises:
for a task scheduling plan generated by any secondary generating operation, determining the completion time of each computing node in the cluster executing a task according to the task scheduling plan;
selecting the maximum completion time length from the completion time lengths corresponding to the computing nodes as the execution time length corresponding to the task scheduling plan;
and selecting one task scheduling plan with the shortest execution time as a task execution mode for the data to be processed.
11. A task scheduling apparatus, characterized in that the apparatus comprises:
the system comprises a plan generating unit, a task scheduling unit and a task scheduling unit, wherein the plan generating unit is used for executing multiple generation operations and generating a plurality of task scheduling plans of data to be processed, each task scheduling plan is used for describing an execution plan of processing tasks of a plurality of data fragments of the data to be processed, and the data fragments are stored in a cluster; and
the plan selection unit is used for selecting one task scheduling plan with the shortest execution time from the plurality of task scheduling plans as a task execution mode for the data to be processed;
wherein the generation operation executed each time by the plan generation unit includes:
dividing the plurality of data fragments into a first data fragment group and a second data fragment group, wherein the total task amount of the first data fragment group corresponding to different generation operations is different;
allocating a processing task of any data fragment in the first data fragment group to a computing node storing the data fragment in the cluster, and using a task allocation mode of the data fragment in the first data fragment group as a first task allocation plan;
allocating the processing task of any data fragment in the second data fragment group to a computing node which does not store the data fragment in the cluster, and using the task allocation mode of the data fragment in the second data fragment group as a second task allocation plan; and
and taking the distribution mode comprising the first task distribution plan and the second task distribution plan as a task scheduling plan generated by the current generation operation.
12. A computing device, comprising:
a processor;
a memory; and
one or more programs stored in the memory and configured to be executed by the processor, the one or more programs including instructions for performing the method of any of claims 1-10.
13. A storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the method of any of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811501565.2A CN111290841B (en) | 2018-12-10 | 2018-12-10 | Task scheduling method, device, computing equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811501565.2A CN111290841B (en) | 2018-12-10 | 2018-12-10 | Task scheduling method, device, computing equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111290841A true CN111290841A (en) | 2020-06-16 |
| CN111290841B CN111290841B (en) | 2024-04-05 |
Family
ID=71020760
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811501565.2A Active CN111290841B (en) | 2018-12-10 | 2018-12-10 | Task scheduling method, device, computing equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111290841B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813524A (en) * | 2020-07-09 | 2020-10-23 | 北京奇艺世纪科技有限公司 | Task execution method and device, electronic equipment and storage medium |
| CN112346845A (en) * | 2021-01-08 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Method, device and equipment for scheduling coding tasks and storage medium |
| CN114510540A (en) * | 2022-04-19 | 2022-05-17 | 北京微芯感知科技有限公司 | Data processing method, calculation storage separation system and block chain network architecture |
| CN114862606A (en) * | 2022-06-13 | 2022-08-05 | 新疆益盛鑫创展科技有限公司 | Insurance information processing method and device based on cloud service |
| CN114866334A (en) * | 2022-06-09 | 2022-08-05 | 中国工商银行股份有限公司 | Data fusion processing method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7693813B1 (en) * | 2007-03-30 | 2010-04-06 | Google Inc. | Index server architecture using tiered and sharded phrase posting lists |
| CN103226467A (en) * | 2013-05-23 | 2013-07-31 | 中国人民解放军国防科学技术大学 | Data parallel processing method and system as well as load balancing scheduler |
| CN104182279A (en) * | 2014-02-26 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Task scheduling method, device and system |
| US9087012B1 (en) * | 2014-06-04 | 2015-07-21 | Pure Storage, Inc. | Disaster recovery at high reliability in a storage cluster |
| CN105828105A (en) * | 2015-12-10 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed environment-based video transcoding system and video transcoding method |
| WO2017008477A1 (en) * | 2015-07-14 | 2017-01-19 | 杭州海康威视数字技术股份有限公司 | Cluster video analysis method and system |
| CN108391142A (en) * | 2018-03-30 | 2018-08-10 | 腾讯科技（深圳）有限公司 | A kind of method and relevant device of video source modeling |
-
2018
- 2018-12-10 CN CN201811501565.2A patent/CN111290841B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7693813B1 (en) * | 2007-03-30 | 2010-04-06 | Google Inc. | Index server architecture using tiered and sharded phrase posting lists |
| CN103226467A (en) * | 2013-05-23 | 2013-07-31 | 中国人民解放军国防科学技术大学 | Data parallel processing method and system as well as load balancing scheduler |
| CN104182279A (en) * | 2014-02-26 | 2014-12-03 | 无锡天脉聚源传媒科技有限公司 | Task scheduling method, device and system |
| US9087012B1 (en) * | 2014-06-04 | 2015-07-21 | Pure Storage, Inc. | Disaster recovery at high reliability in a storage cluster |
| WO2017008477A1 (en) * | 2015-07-14 | 2017-01-19 | 杭州海康威视数字技术股份有限公司 | Cluster video analysis method and system |
| CN105828105A (en) * | 2015-12-10 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed environment-based video transcoding system and video transcoding method |
| CN108391142A (en) * | 2018-03-30 | 2018-08-10 | 腾讯科技（深圳）有限公司 | A kind of method and relevant device of video source modeling |
Non-Patent Citations (1)
| Title |
|---|
| 朱洁;李雯睿;王江平;赵红;: "基于节点集计算能力差异的Hadoop自适应任务调度算法", 计算机应用, no. 04, 10 April 2016 (2016-04-10) * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813524A (en) * | 2020-07-09 | 2020-10-23 | 北京奇艺世纪科技有限公司 | Task execution method and device, electronic equipment and storage medium |
| CN111813524B (en) * | 2020-07-09 | 2023-09-08 | 北京奇艺世纪科技有限公司 | Task execution method and device, electronic equipment and storage medium |
| CN112346845A (en) * | 2021-01-08 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Method, device and equipment for scheduling coding tasks and storage medium |
| CN112346845B (en) * | 2021-01-08 | 2021-04-16 | 腾讯科技（深圳）有限公司 | Method, device and equipment for scheduling coding tasks and storage medium |
| CN114510540A (en) * | 2022-04-19 | 2022-05-17 | 北京微芯感知科技有限公司 | Data processing method, calculation storage separation system and block chain network architecture |
| CN114866334A (en) * | 2022-06-09 | 2022-08-05 | 中国工商银行股份有限公司 | Data fusion processing method and device |
| CN114866334B (en) * | 2022-06-09 | 2023-11-24 | 中国工商银行股份有限公司 | Data fusion processing method and device |
| CN114862606A (en) * | 2022-06-13 | 2022-08-05 | 新疆益盛鑫创展科技有限公司 | Insurance information processing method and device based on cloud service |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111290841B (en) | 2024-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111290841B (en) | Task scheduling method, device, computing equipment and storage medium | |
| CN110769278B (en) | Distributed video transcoding method and system | |
| US10298969B2 (en) | Architecture and method for high performance on demand video transcoding | |
| US9407944B1 (en) | Resource allocation optimization for cloud-based video processing | |
| Jokhio et al. | Prediction-based dynamic resource allocation for video transcoding in cloud computing | |
| CN109791504B (en) | Dynamic resource configuration for application containers | |
| US12075099B2 (en) | System for high performance on-demand video transcoding | |
| Reddy et al. | Qos-Aware Video Streaming Based Admission Control And Scheduling For Video Transcoding In Cloud Computing | |
| US11516152B2 (en) | First-in first-out function for segmented data stream processing | |
| US20150133214A1 (en) | Video encoding based on areas of interest | |
| CN108965884B (en) | Distribution method of transcoding tasks, scheduling device and transcoding device | |
| Li et al. | CVSS: A cost-efficient and QoS-aware video streaming using cloud services | |
| US20090144423A1 (en) | Network traffic prioritization | |
| JP2017050001A (en) | System and method for effective neural network deployment | |
| US11775350B2 (en) | Compute resource estimation for function implementation on computing platform | |
| Meskar et al. | Fair multi-resource allocation with external resource for mobile edge computing | |
| WO2024212767A1 (en) | Video pushing method and apparatus, device, and storage medium | |
| US20100104010A1 (en) | Real-time rate-control method for video encoder chip | |
| US10681398B1 (en) | Video encoding based on viewer feedback | |
| CN118368259B (en) | Network resource allocation method, device, electronic equipment and storage medium | |
| TW201445989A (en) | System and method for encoding and decoding data | |
| CN116962518A (en) | Resource request method, device and storage medium | |
| Lee et al. | Quality-aware transcoding task allocation under limited power in live-streaming systems | |
| Farhad et al. | Dynamic resource provisioning for video transcoding in iaas cloud | |
| Mavromoustakis et al. | Dynamic cloud resource migration for efficient 3D video processing in mobile computing environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |