WO2018137611A1 - Task processing method, node, medium, device and content distribution network - Google Patents

Abstract

Disclosed are a task processing method, a node, a medium, a device and a content distribution network. The method comprising a sub-node of a first type acquiring a task to be processed allocated by a task scheduling node to a task processing node where the sub-node of the first type is located; acquiring data to be processed corresponding to the task to be processed from a storage node; splitting the data to be processed into a plurality of pieces of sub-data; allocating the plurality of pieces of sub-data to at least one sub-node of a second type in the task processing node where the sub-node of the first type is located, for data processing; and merging the sub-data processed by each sub-node of the second type in the task processing node where the sub-node of the first type is located. By arranging the sub-node of the first type, the present invention means that the sub-node of the second type can directly acquire data from the sub-node of the first type. Since the sub-node of the first type split the data, each sub-node of the second type only needs to acquire, from the sub-node of the first type, data they themselves need to process, reducing network load.

Description

Task processing methods, nodes, media, devices, and content distribution networks

This application claims the priority of the Chinese Patent Application entitled "Task Processing Method, Node, Medium, Equipment, and Content Distribution Network" submitted by the Chinese Patent Office on January 24, 2017, with the application number of 201710059992.9, the entire contents of which are incorporated by reference. Combined in this application.

Technical field

The embodiments of the present invention are not limited to the field of computing technologies, and particularly relate to a task processing method, a node, a medium, a device, and a content distribution network.

Background technique

With the development of technology, distributed computing technology has also been widely used. For example, in the field of video transcoding, a video transcoding task is often assigned to multiple processing nodes for transcoding.

When the existing distributed computing technology performs task processing, each processing node needs to obtain complete pending data from the storage system separately, and then select some data for processing, which undoubtedly increases the network burden. When a processing node fails to process data, another processing node needs to re-acquire complete pending data from the storage system, which also increases the network burden.

How to solve this technical problem is still a technical problem that needs to be solved urgently.

Summary of the invention

The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.

An object of the embodiments of the present invention is to provide a task processing method, a node, a medium, a device, and a content distribution network, so as to reduce network load.

The specific technical solutions are as follows:

A task processing method is provided in a content distribution network, where the content distribution network includes: a storage node, a task scheduling node, and a plurality of task processing nodes, each task processing node includes: a first type of child node And a plurality of second type of child nodes, wherein the devices corresponding to the second type of child nodes are different, and the method includes:

The first type of child node obtains a to-be-processed task assigned by the task scheduling node to the task processing node where the first type of child node is located;

The first type of child node acquires to-be-processed data corresponding to the to-be-processed task from the storage node;

The first type of child node splits the data to be processed into a plurality of child data; the first type of child node allocates a plurality of child data to at least one second type of child node in the task processing node where the first type of child node is located. data processing;

The first type of child node merges the sub-data processed by each second type of child node in the task processing node where the first type of child node is located.

The above method also has the following characteristics:

The method further includes: the first type of child node sends the merged data to the storage node. Optionally, the task scheduling node is assigned to the first type in the first type of child node.

Before the task of the task node of the child node is processed, the method further includes:

The task scheduling node obtains working state information of each task processing node and distance information of each task processing node to the storage node;

The task scheduling node determines a task processing node that executes the to-be-processed task according to the working state information and the distance information, and allocates the to-be-processed task to the determined task processing node.

The above method also has the following characteristics:

The distance between each child node in each task processing node is less than the first preset distance.

The above method also has the following characteristics:

After the first type of child node allocates the plurality of child data to the at least one second type of child node in the task processing node where the first type of child node is located, the method further includes:

The first type of child node obtains a processing result of data processing of each second type of child node in the task processing node where the first type of child node is located;

The first type of child node allocates data processed by the failed second type of child node to the task processing node where the first type of child node is located, except for the second type of child node whose processing result is a failure. Two types of child nodes perform data processing.

The embodiment of the present invention further provides a task processing node, which is located in a content distribution network, and each task processing node includes: a first type of child node and a plurality of second type of child nodes, wherein each second type of child node corresponds to The devices of the first type include: a task obtaining unit, a data obtaining unit, a data splitting unit, a data distributing unit, and a data combining unit.

a task obtaining unit, configured to obtain a task to be processed that is allocated by the task scheduling node to the task processing node where the first type of child node is located;

a data obtaining unit, configured to acquire, from the storage node, data to be processed corresponding to the task to be processed;

a data splitting unit, configured to split the to-be-processed data into multiple sub-data; and a data distribution unit, configured to allocate multiple sub-data to the first-type sub-node

At least one second type of child node in the task processing node performs data processing;

The data merging unit is configured to merge the sub-data processed by each second type of child node in the task processing node where the first type of child node is located.

The above nodes also have the following characteristics:

The first type of child node further includes: a data sending unit, configured to send the merged data to the storage node.

The above nodes also have the following characteristics:

The distance between each child node in each task processing node is less than the first preset distance.

The above nodes also have the following characteristics:

The first type of child node further includes: a processing result obtaining unit and a redistribution unit,

a processing result obtaining unit, configured to obtain, after the data distribution unit allocates the plurality of sub-data to the at least one second type of child node in the task processing node where the first type of child node is located, to obtain the first type of child node The processing result of data processing by each second type of child node in the task processing node;

a redistribution unit, configured to allocate, to the task processing node where the first type of child node is located, the data processed by the failed second type child node, other than the second type child node whose processing result is a failure Two types of child nodes perform data processing.

A content distribution network provided by an embodiment of the present invention includes: a storage node, a task scheduling node, and a plurality of the foregoing task processing nodes,

The task scheduling node obtains load information of each task processing node and distance information of each task processing node to the storage node, determines a task processing node that executes the to-be-processed task according to the load information and the distance information, and allocates the to-be-processed task to the determined task processing. node;

The storage node stores the to-be-processed data corresponding to the task to be processed.

The embodiment of the present invention provides a computer readable storage medium, where the computer program stores a computer program, and when the program is executed by the processor, the steps of the foregoing method are implemented.

A computer device provided by an embodiment of the present invention includes a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements the steps of the foregoing method when executing the program.

The embodiment of the present invention provides a task processing method, a node, and a content distribution network. The first type of child node obtains a task to be processed that is assigned by the task scheduling node to the task processing node where the first type of child node is located; Data to be processed corresponding to the task to be processed; splitting the data to be processed into a plurality of sub-data; assigning a plurality of sub-data to at least one second-type sub-node in the task processing node where the first-type sub-node is located for data processing And merging the processed sub-data of each second type of child node in the task processing node where the first type of child node is located. The invention adopts the setting of the first type of child node, so that the second type of child node can directly acquire data from the first type of child node, and since the first type of child node performs data splitting, each second type of child node only You need to get the data you need to process from the first type of child nodes, which reduces the network burden. At the same time, since the number of data transmissions between the node and the storage system is greatly reduced, the probability that the data transmission process is affected by the network instability factor is smaller, and the stability of the data transmission is improved.

Of course, any product or method embodying the present invention necessarily does not necessarily require all of the advantages described above to be achieved at the same time.

DRAWINGS

The accompanying drawings are intended to provide a further understanding of the embodiments of the embodiments of the invention Improper limitations. In the drawing:

FIG. 1 is a schematic structural diagram of a content distribution network according to an embodiment of the present disclosure;

2 is a flowchart of a task processing method according to an embodiment of the present invention;

FIG. 3 is a flowchart of another task processing method according to an embodiment of the present invention;

FIG. 4 is a flowchart of another task processing method according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a first type of child node according to an embodiment of the present invention.

detailed description

The embodiments of the present invention will be further described with reference to the drawings and specific embodiments.

As shown in FIG. 1 , a task processing method provided by an embodiment of the present invention is applied to a content distribution network, where the content distribution network may include: a storage node 001, a task scheduling node 003, and a plurality of task processing nodes 002, each task The processing node 002 includes: a first type of child node 100 and a plurality of second type child nodes 200. The devices corresponding to the second type of child nodes 200 are different. As shown in FIG. 2, the method may include:

S100, the first type of child node 100 obtains a task to be processed that the task scheduling node 003 assigns to the task processing node 002 where the first type of child node 100 is located;

The storage node 001, the task scheduling node 003, and the plurality of task processing nodes 002 can be connected to each other.

Specifically, the distance between each sub-node in each task processing node may be smaller than the first preset distance, or each sub-node in each task processing node is located in the same local area network. In this way, each sub-node within each task processing node can communicate efficiently. The task processing node may be a cache node in the content distribution network.

Specifically, multiple devices in a certain area may be planned as multiple child nodes in a task processing node according to service requirements, for example, planning a communication device of a telecommunication carrier in a certain city as multiple tasks in a task processing node. node.

Optionally, there may be multiple task processing nodes 002 that are in communication with the storage node 001. The first type of child nodes 100 and the plurality of second type of child nodes 200 in each task processing node 002 may be located in a preset geographic area. in.

S200. The first type of child node 100 acquires to-be-processed data corresponding to the to-be-processed task from the storage node 001.

The to-be-processed task may be a video transcoding task, and the to-be-processed data may be a video file. The distance between the storage node 001 and the task processing node 002 may be greater than the second preset distance. It may be understood that, for cost reasons, the storage node 001 and the task processing node 002 are often not disposed in the same area, and the distance between the two is far. . In this case, after obtaining the to-be-processed data from the storage node 001, the first-type child node 100 of the embodiment of the present invention can directly allocate the split to each second-type child node 200 without using each second. The type sub-node 200 obtains the data to be processed from the storage node 001, respectively, which reduces the amount of data transmitted and the occupation of the network bandwidth.

S300. The first type of child node 100 splits the to-be-processed data into multiple sub-data.

S400, the first type of child node 100 allocates a plurality of sub-data to at least one second type of child node 200 in the task processing node 002 where the first type of child node 100 is located for data processing;

S500. The first type of child node 100 merges the sub-data processed by each second type of child node 200 in the task processing node 002 where the first type of child node 100 is located.

In other embodiments of the present invention, after the step S500, the first type of child node 100 may send the merged data to the storage node 001.

The task processing method provided by the embodiment of the present invention enables the second type of child node to directly acquire data from the first type of child node through the setting of the first type of child node, and at the same time, the data is removed due to the first type of child node. Therefore, each second type of child node only needs to obtain the data that it needs to process from the first type of child node, which greatly reduces the network burden. At the same time, since the number of data transmissions between the node and the storage system is greatly reduced, the probability that the data transmission process is affected by the network instability factor is smaller, and the stability of the data transmission is improved.

As shown in FIG. 3, in another task processing method provided by the embodiment of the present invention, before the step S100, the method may further include:

S001, the task scheduling node 003 obtains the working state information of each task processing node 002 and the distance information of each task processing node 002 to the storage node 001;

S002. The task scheduling node 003 determines, according to the working state information and the distance information, a task processing node that executes the to-be-processed task, and allocates the to-be-processed task to the determined task processing node.

Specifically, when the first type of child node 100 cannot obtain the working state information of a certain type of child node, the first type of child node 100 determines that the second type of child node is a child node of the unexecutable task. The situation that the first type of child node 100 cannot obtain the working state information of the second type of child node is different, for example, the network connection of the first type child node 100 and the second type child node has a problem, or The second type of child node is down.

The working state of the second type child node 200 may include: load information of the second type child node 200, and the like. Optionally, when the load of the second type of child node 200 is lower than the preset load, the second type of child node 200 may be determined to be a second type of child node of the executable task and the task to be processed is allocated to the child node.

Optionally, after the task scheduling node 003 assigns the to-be-processed task to the determined task processing node, the processing result of the task processing node that processes the to-be-processed task may be obtained. When the processing result is a failure, the process returns to step S001. In this way, the feedback mechanism ensures the successful execution of the task and improves the reliability of the system.

As shown in FIG. 4, another task processing method provided by the embodiment of the present invention may further include: after step S400:

S410. The first type of child node obtains a processing result of data processing of each second type of child node in the task processing node where the first type of child node is located;

S420. The first type of child node allocates data processed by the failed second type of child node to the task processing node where the first type of child node is located, except for the second type of child node whose processing result is a failure. The second type of child node performs data processing.

After the data is reassigned to the other second type of child nodes in step S420, step S410 receives the processing results of the other second type of child nodes and continues to determine whether the processing results are all successful. Through this feedback mechanism, the data is successfully processed, the reliability of the system is improved, and the success rate of task execution is improved.

The task processing method provided by the embodiment of the present invention, when the second type of child node fails to process data, the other second type of child node does not need to acquire data from the storage node, but directly fails the previous processing by the first type of child node. The sub-data is sent to another second type of sub-node, which greatly reduces the amount of data transmitted, reduces the network burden, improves the concurrent execution of the task, and improves the data throughput capability of the system.

The embodiment of the present invention further provides a computer readable storage medium, where the computer program stores a computer program, and when the program is executed by the processor, the steps of the foregoing method are implemented.

A computer device further includes a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements the steps of the method when executing the program.

Corresponding to the foregoing method embodiment, the embodiment of the present invention further provides a task processing node. A task processing node is provided in the content distribution network. As shown in FIG. 1 , the content distribution network may include: a storage node 001, a task scheduling node 003, and a plurality of task processing nodes 002, each task processing. The node 002 includes: a first type of child node 100 and a plurality of second type child nodes 200, wherein each second type of child node 200 corresponds to different devices. As shown in FIG. 5, the first type of child node 100 may include : task obtaining unit 110, data obtaining unit 120, data splitting unit 130, data allocating unit 140, and data combining unit 150,

The task obtaining unit 110 is configured to obtain a task to be processed that the task scheduling node 003 assigns to the task processing node 002 where the first type of child node 100 is located;

The storage node 001, the task scheduling node 003, and the plurality of task processing nodes 002 can be connected to each other.

Specifically, the distance between each sub-node in each task processing node may be smaller than the first preset distance, such as a CDN edge node of the same content distribution network, or the same equipment room, or the same area, or each task processing node. Each child node is located in the same local area network. In this way, each sub-node within each task processing node can communicate efficiently. The task processing node may be a cache node in the content distribution network.

Specifically, multiple devices in a certain area may be planned as multiple child nodes in a task processing node according to service requirements, for example, planning a communication device of a telecommunication carrier in a certain city as multiple tasks in a task processing node. node.

Optionally, there may be multiple task processing nodes 002 that are in communication with the storage node 001. The first type of child nodes 100 and the plurality of second type of child nodes 200 in each task processing node 002 may be located in a preset geographic area. in.

The data obtaining unit 120 is configured to acquire, from the storage node 001, data to be processed corresponding to the task to be processed;

The to-be-processed task may be a video transcoding task, and the to-be-processed data may be a video file. The distance between the storage node 001 and the task processing node 002 may be greater than the second preset distance. It may be understood that, for cost reasons, the storage node 001 and the task processing node 002 are often not disposed in the same area, and the distance between the two is far. . In this case, after obtaining the to-be-processed data from the storage node 001, the first-type child node 100 of the embodiment of the present invention can directly allocate the split to each second-type child node 200 without using each second. The type sub-node 200 obtains the data to be processed from the storage node 001, respectively, which reduces the amount of data transmitted and the occupation of the network bandwidth.

a data splitting unit 130, configured to split the data to be processed into multiple sub-data;

a data distribution unit 140, configured to allocate a plurality of sub-data to at least one second-type sub-node 200 in the task processing node 002 where the first-type child node 100 is located for data processing;

The data merging unit 150 is configured to merge the sub-data processed by the second type sub-nodes 200 in the task processing node 002 where the first-type child node 100 is located.

The first type of child node 100 may further include: a processing result obtaining unit and a redistribution unit.

a processing result obtaining unit, configured to obtain the first type of child node after the data distribution unit 140 allocates the plurality of sub-data to the at least one second type of child node in the task processing node where the first type of child node is located for data processing The processing result of data processing performed by each second type of child node in the task processing node;

a redistribution unit, configured to allocate, to the task processing node where the first type of child node is located, the data processed by the failed second type child node, other than the second type child node whose processing result is a failure Two types of child nodes perform data processing.

After the redistribution unit reassigns the data to the other second type of child nodes, the processing result obtaining unit receives the processing result of the other second type of child nodes and continues to determine whether the processing result is successful.

In other embodiments of the present invention, the first type of child node 100 may further include: a data sending unit, configured to send the merged data to the storage node 001.

The task processing node provided by the embodiment of the present invention enables the second type of child node to directly acquire data from the first type of child node through the setting of the first type of child node, and at the same time, the data is removed due to the first type of child node. Therefore, each second type of child node only needs to obtain the data that it needs to process from the first type of child node, which greatly reduces the network burden.

As shown in FIG. 1 , an embodiment of the present invention further provides a content distribution network, which may include: a storage node 001, a task scheduling node 003, and a plurality of task processing nodes 002 described in the foregoing embodiments.

The task scheduling node 003 obtains the working state information of each task processing node 002 and the distance information of each task processing node 002 to the storage node 001, and determines the task processing node 002 that executes the to-be-processed task according to the load information and the distance information, and allocates the to-be-processed task. Processing node 002 for the determined task;

The storage node 001 stores the to-be-processed data corresponding to the task to be processed.

Specifically, when the first type of child node 100 cannot obtain the working state information of a certain type of child node, the first type of child node 100 determines that the second type of child node is a child node of the unexecutable task. The situation that the first type of child node 100 cannot obtain the working state information of the second type of child node is different, for example, the network connection of the first type child node 100 and the second type child node has a problem, or The second type of child node is down.

The working state of the second type child node 200 may include: load information of the second type child node 200, and the like. Optionally, when the load of the second type of child node 200 is lower than the preset load, the second type of child node 200 may be determined to be a second type of child node of the executable task and the task to be processed is allocated to the child node.

Specifically, the task scheduling node 003 can allocate a to-be-processed task to each task processing node according to the network condition, the geographical distribution, the number of processing nodes, the processing power of the processing node, and the load condition of each task processing node.

The content distribution network provided by the embodiment of the present invention enables the second type of child node to directly acquire data from the first type of child node through the setting of the first type of child node, and at the same time, the data is removed due to the first type of child node. Therefore, each second type of child node only needs to obtain the data that it needs to process from the first type of child node, which greatly reduces the network burden.

A person skilled in the art should understand that the technical solutions of the present invention may be modified or equivalent, without departing from the spirit and scope of the present invention, and should be included in the scope of the claims.

Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and functional blocks/units of the methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical The components work together. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and nonvolatile, implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules or other data. Sex, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or may Any other medium used to store the desired information and that can be accessed by the computer. Moreover, it is well known to those skilled in the art that communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. .

Industrial applicability

In the embodiment of the present invention, the setting of the first type of child node enables the second type of child node to directly obtain data from the first type of child node, and since the first type of child node performs data splitting, each second The type child node only needs to obtain the data that it needs to process from the first type of child node, which effectively reduces the network burden.

Claims (12)

A task processing method is applied to a content distribution network, where the content distribution network includes: a storage node, a task scheduling node, and a plurality of task processing nodes, each of the task processing nodes includes: a first type of child node and a plurality of a second type of child node, wherein each device of the second type of child node is different, and the method includes: Obtaining, by the first type of child node, a task to be processed that is allocated by the task scheduling node to a task processing node where the first type of child node is located; The first type of child node acquires to-be-processed data corresponding to the to-be-processed task from the storage node; The first type of child node splits the to-be-processed data into a plurality of child data; the first type of child node allocates the plurality of child data to at least one of task processing nodes where the first type of child node is located The second type of child node performs data processing; The first type of child node merges the sub-data processed by each of the second type of child nodes in the task processing node where the first type of child node is located. The method of claim 1, wherein the method further comprises the first type of child node transmitting the combined data to the storage node. The method according to claim 1, wherein before the first type of child node obtains a task to be processed that is allocated to the task processing node where the first type of child node is located, the method further includes: The task scheduling node obtains working state information of each task processing node and distance information of each task processing node to the storage node; The task scheduling node determines, according to the working state information and the distance information, a task processing node that executes a task to be processed, and allocates the to-be-processed task to the determined task processing node. The method of claim 1 wherein the distance between each of the child nodes within each of the task processing nodes is less than a first predetermined distance. The method according to any one of claims 1 to 4, wherein the first type of child node assigns the plurality of child data to at least one of the task processing nodes in which the first type of child node is located After the data processing of the two types of child nodes, the method further includes: Obtaining, by the first type of child node, a processing result of performing data processing by each of the second type of child nodes in the task processing node where the first type of child node is located; The first type of child node allocates data processed by the failed second type of child node to the second type of child in the task processing node where the first type of child node is located, except that the processing result is a failure. Other second types of child nodes outside the node perform data processing. [Correct according to Rule 26 09.02.2018]
A task processing node is located in a content distribution network, where the task processing node includes: a first type of child node and a plurality of second type of child nodes, wherein each of the second type of child nodes corresponds to different devices. The first type of child node includes: a task obtaining unit, a data obtaining unit, a data splitting unit, a data assigning unit, and a data combining unit, and the task obtaining unit is configured to obtain the task scheduling node and assign the first type to the child. a task to be processed by the task processing node where the node is located; the data obtaining unit is configured to acquire data to be processed corresponding to the to-be-processed task from the storage node; the data splitting unit is configured to: The data to be processed is split into a plurality of sub-data; the data distribution unit is configured to allocate the plurality of sub-data to at least one of the second-type sub-nodes in the task processing node where the first-type sub-node is located to perform data The data merging unit is configured to: each of the second type of child nodes in the task processing node where the first type of child node is located After processing the sub-data merging. The task processing node according to claim 6, wherein the first type of child node further comprises: a data sending unit, configured to send the merged data to the storage node. The task processing node of claim 6, wherein a distance between each of the child nodes within the task processing node is less than a first predetermined distance. The task processing node according to any one of claims 6 to 8, wherein the first type of child node further comprises: a processing result obtaining unit and a redistributing unit; The processing result obtaining unit is configured to: after the data distribution unit allocates the plurality of sub-data to at least one of the second type of child nodes in the task processing node where the first type of child node is located, perform data processing, Obtaining a processing result of data processing by each of the second type of child nodes in the task processing node where the first type of child node is located; The redistribution unit is configured to allocate data processed by the failed second type of child node to the second type of the task processing node where the first type of child node is located, except that the processing result is a failure. Other second types of child nodes outside the child node perform data processing. A content distribution network, wherein the content distribution network includes: a storage node, a task scheduling node, and a plurality of task processing nodes according to any one of claims 6 to The task scheduling node obtains load information of each task processing node and distance information of each task processing node to the storage node, and determines a task processing node that executes a task to be processed according to the load information and the distance information, and the The pending task is assigned to the determined task processing node; The storage node stores to-be-processed data corresponding to the to-be-processed task. A computer readable storage medium having stored thereon a computer program, the program being executed by a processor to perform the steps of the method of any one of claims 1 to 5. A computer device, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, the processor executing the program to implement the claims 1 to 5 The steps of any of the methods described.

Images