CN107070976A - A kind of data transmission method - Google Patents

Abstract

The invention discloses a data transmission method, relates to the technical field of data processing, and can ensure the accuracy of data flow distribution, thereby ensuring the accuracy and high efficiency of distributed computing. The data transmission method includes: obtaining an output data stream from an upstream node; performing persistent backup of the output data stream according to the data stream name; and sending the output data stream to On the shared memory queue corresponding to the downstream nodes one by one; sending all the data streams in the shared memory queue.

Description

A method of data transmission

technical field

The invention relates to the field of data processing, in particular to a data transmission method.

Background technique

With the rapid development of information technology, applications based on data processing and data analysis are widely welcomed and concerned. A large number of information sources have brought about explosive growth in data scale, and complex calculations on massive data have far exceeded the processing power of a single computer, thus promoting the research on distributed systems and their key technologies. In distributed computing, the massive data requiring complex calculations is divided into small pieces and then distributed to multiple computers for parallel processing, and the local calculation results are integrated to obtain the final result.

Distributed computing means that data needs to be distributed or forwarded multiple times to send the corresponding data stream to the corresponding computer or downstream node for processing. Therefore, it is necessary to ensure the accuracy of data stream distribution and the accuracy of distributed computing , Efficiency.

Contents of the invention

The technical problem to be solved by the present invention is to provide a data transmission method, which can ensure the accuracy of data flow distribution, thereby ensuring the accuracy and high efficiency of distributed computing.

In order to solve the problems of the technologies described above, the present invention adopts the following technical solutions:

An embodiment of the present invention provides a data transmission method, the data transmission method comprising:

Get the output data stream from the upstream node;

Perform persistent backup of the output data stream according to the data stream name;

Sending the output data stream to a shared memory queue corresponding to the downstream node one by one according to the subscription relationship between the output data stream and the downstream node;

Send the data stream in the shared memory queue.

Preferably, the shared memory queues one-to-one corresponding to the downstream nodes are named after the downstream nodes.

Preferably, the data transmission method also includes:

Set the expiration time;

record the start time of persistent backup of the output data stream;

Get the current time;

If the time difference between the current time and the start time is greater than or equal to the expiration time, the output data stream is deleted.

Preferably, obtaining the output data stream from the upstream node includes:

Listen to the specified port and add the specified port to the epoll handle;

When receiving the output data stream name from the upstream node, open the named pipe with the read flag;

Add the file descriptor of the famous pipeline to the epoll loop;

Loop reads output data streams from upstream nodes from the named pipe.

Preferably, performing persistent backup of the output data stream according to the data stream name includes:

Data items in the shared memory queue are stored in key-value form.

Preferably, the key value is the timestamp of the data item.

Preferably, sending all data streams in the shared memory queue includes:

Encapsulate the downstream node name into a task and put it in the task queue;

Take out the downstream node name in the task queue, and take out the data from the corresponding shared memory queue;

Send data according to the configuration of the downstream node described in Zookeeper.

Preferably, said sending data according to the configuration of said downstream node in Zookeeper includes:

Obtaining the traffic setting of the downstream node;

Based on the traffic settings, data is sent.

The invention provides a data transmission method. When receiving an output data stream from an upstream node, the data stream can be saved on the disk according to the name of the output data stream for persistent backup, and the output data stream can be subscribed according to the output data stream and the downstream node. Relationship, send the output stream to the shared memory queue named after the downstream node, and send it to the downstream node uniformly. This makes it more targeted and more accurate when outputting data streams.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only of the present invention. For some embodiments, those skilled in the art can also obtain other drawings based on these drawings without creative effort.

FIG. 1 is a schematic flowchart of a data transmission method provided by an embodiment of the present invention.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

An embodiment of the present invention provides a data transmission method, as shown in Figure 1, the data transmission method includes:

Step S1. Obtain the output data stream from the upstream node.

Specifically, in the embodiment of the present invention, by listening to a specified port for a long time and adding it to the epoll handle, when the output data stream name (outstream name) sent by the upstream node is received, the well-known pipe is opened with the read (READ) flag ( FIFO pipeline), and add the pipeline file descriptor to the epoll loop.

Step S2, performing persistent backup of the output data stream according to the data stream name.

In order to increase the robustness of the framework, a data persistence module is added to the node, which acts as a disk queue and follows the semantics of queue first-in-first-out (FIFO). This module saves the processed output data stream on disk. In order to avoid disk capacity expansion, the persistence module assigns each data item a certain expiration time and periodically deletes them from the disk.

Performing persistent backup is beneficial to solve the problem of mismatching speeds of upstream and downstream nodes processing data streams. Especially when the sending speed of the upstream node is much faster than the receiving and processing speed of the downstream node, a large number of data packets will accumulate in the buffer of the upstream node and cannot be sent, which will cause the memory usage to expand and make the internal operation of the upstream node slow. The persistent backup has a layer of buffer between the upstream and downstream nodes, which helps to alleviate the problems caused by the difference in the processing speed of the upstream and downstream nodes.

In addition, in order to improve the processing speed of distributed computing, when each computer performs local computing, the cutting, fusion and processing of data streams can be completed in memory, ensuring extremely fast processing speed and response speed. But this also brings disadvantages, that is, compared to the long-term data "landed" on the disk, the memory data is a kind of unstable data. When one of the computers restarts due to a failure or directly crashes, the data in the memory All will disappear, causing accidents to spread in distributed computing and affecting the accuracy of system calculation results.

Furthermore, many streaming computing systems do not support the function of resuming transmission from breakpoints, because it is assumed that the data flow passes through the system once. When a node fails or the calculation logic fails, the original data flow can only be re-passed through the system to perform a calculation again. , not only wastes resources, especially when the data flow is unique and cannot be reproduced, the final output result will not include this part of the data flow, and the accuracy of the system will be greatly reduced. Therefore, in the embodiment of the present invention, a persistent backup is added to address this disadvantage, so as to support the functions of resuming transmission after interruption and error retransmission.

In the embodiment of the present invention, the persistent backup can also provide the functions of resuming transmission from breakpoints and error retransmission. In the existing data flow computing system, the data flow passes through the system memory completely at one time. Since the data cannot be restored and reproduced, it is very sensitive to faults and has poor fault tolerance. This requires the two parties involved in the data transmission to conduct an agreement before formally transmitting the data stream, and the downstream node notifies the upstream node of the location where the previous data stream transmission terminated. In the embodiment of the present invention, the location transmission protocol can be used to locate the previous transmission location of the data flow, and the data at the corresponding location can be retrieved from the persistent backup, so as to realize the restoration of the data flow.

Specifically, on the basis of step S1, the output data flow processed by the upstream node is continuously read from the specified pipeline, and stored in a disk queue named after the output data flow for persistent backup. The data items in the disk queue are stored in the form of key-value, and the key value is the timestamp of the data item, which is convenient for quickly locating the data item from the hard disk queue when an error is retransmitted. According to the characteristics of asynchronous reading and writing of epoll, a state machine is implemented, a simple data transmission protocol similar to TCP/IP, which provides support for data resuming and ensuring the accuracy and authenticity of data.

epoll is a poll improved by the Linux kernel to handle a large number of file descriptors. It is an enhanced version of the multiplexed IO interface select/poll under Linux. The CPU utilization of the system. Another reason is that when acquiring an event, it does not need to traverse the entire set of descriptors being listened to. It only needs to traverse the set of descriptors that are asynchronously awakened by the kernel IO event and added to the Ready queue. In addition to providing level trigger (Level Triggered) of IO events such as select/poll, epoll also provides edge trigger (Edge Triggered), which makes it possible for user space programs to cache IO status, reduce epoll_wait/epoll_pwait calls, and improve application performance. program efficiency.

In the embodiment of the present invention, an efficient disk queue can be realized through the open source software Tokyo Cabinet, thereby realizing efficient persistent backup. Tokyo Cabinet (TC for short) is a data storage engine written in C language. It stores data in the form of key-value, supports Hash, B+ tree, Hash Table and other data structures, and reads and writes very fast.

Step S3. According to the subscription relationship between the output data stream and the downstream node, the output data stream is sent to the shared memory queue corresponding to the downstream node one by one.

By querying the subscription relationship between the output data stream and the downstream node, the data that has been backed up in the disk queue is sent to the shared memory queue named after the downstream node (node). Obviously, each downstream node corresponds to a unique shared memory queue. In order to ensure the consistency when receiving and sending data, in the embodiment of the present invention, the downstream node name can be packaged into a task and placed in the task queue. By parsing the task type as "internal" (receiving data from the upstream node is an "external" task, Getting data from the shared memory queue is an "internal" task), take out the downstream node name in the task, take out the data from the corresponding shared memory queue, and send the data according to the configuration (IP) of the corresponding downstream node in Zookeeper. Shared message queues are also a means of interprocess communication (IPC) in unix-like systems, where multiple processes can read or add messages to the queue. The shared message queue is a linked list of messages that persists with the kernel, and the messages in it have a prescribed format, a specific type, and a corresponding priority.

Step S4, sending the data stream in the shared memory queue.

After the downstream nodes corresponding to the data flows in the shared memory queue are found, the data in the shared memory queue can be sent.

The embodiment of the present invention provides a data transmission method. When the output data stream from the upstream node is received, the data stream can be saved on the disk according to the name of the output data stream for persistent backup, and the data stream can be stored according to the name of the output data stream and the downstream node. Subscription relationship, send the output stream to the shared memory queue named after the downstream node, and send it to the downstream node uniformly. This makes it more targeted and more accurate when outputting data streams.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (8)

1. A data transmission method, characterized in that, comprising: Get the output data stream from the upstream node; Perform persistent backup of the output data stream according to the data stream name; Sending the output data stream to a shared memory queue corresponding to the downstream node one by one according to the subscription relationship between the output data stream and the downstream node; Send the data stream in the shared memory queue. 2. The data transmission method according to claim 1, wherein: The shared memory queues corresponding to the downstream nodes one by one are named after the downstream nodes. 3. The data transmission method according to claim 1, further comprising: Set the expiration time; record the start time of persistent backup of the output data stream; Get the current time; If the time difference between the current time and the start time is greater than or equal to the expiration time, the output data stream is deleted. 4. The data transmission method according to claim 1, wherein obtaining the output data stream from the upstream node comprises: Listen to the specified port and add the specified port to the epoll handle; When receiving the output data stream name from the upstream node, open the named pipe with the read flag; Add the file descriptor of the famous pipeline to the epoll loop; Loop reads output data streams from upstream nodes from the named pipe. 5. The data transmission method according to claim 1, wherein the persistent backup of the output data stream according to the data stream name comprises: Data items in the shared memory queue are stored in key-value form. 6. The data transmission method according to claim 5, wherein the key value is a timestamp of a data item. 7. The data transmission method according to claim 1, wherein sending all data streams in the shared memory queue comprises: Encapsulate the downstream node name into a task and put it in the task queue; Take out the downstream node name in the task queue, and take out the data from the corresponding shared memory queue; Send data according to the configuration of the downstream node described in Zookeeper. 8. The data transmission method according to claim 7, wherein the sending data according to the configuration of the downstream node in Zookeeper comprises: Obtaining the traffic setting of the downstream node; Based on the traffic settings, data is sent.