CN111917812B - Data transmission control method, device, equipment and storage medium - Google Patents

Abstract

The embodiments of the present invention disclose a data transmission control method, device, device and storage medium. The method includes: splitting data sources in the acquired transmission configuration information to obtain a split result matching the number of parallel processing; generating and splitting results according to the destination source in the transmission configuration information and a local computing operator The corresponding local transmission configuration information, and distribute the local transmission configuration information to the target proxy device in the proxy device set; receive the transmission process information of each target node in the transmission stream reported by the target proxy device, and monitor the transmission stream according to the transmission process information, and/or the real-time operational status of the target node. The technical solutions of the embodiments of the present invention can further strengthen the supervision and control of the transmission flow and the node during the data transmission process, so as to realize the dynamic adjustment of the transmission flow or the transmission control mode of the target node during the data transmission process, so as to further Improve data transmission efficiency.

Description

Data transmission control method, device, device and storage medium

technical field

Embodiments of the present invention relate to a data transmission control technology, and in particular, to a data transmission control method, apparatus, device, and storage medium.

Background technique

In the Internet industry, the era of "big data" has come, and various companies and organizations are trying their best to research and build their own big data processing platforms to mine all kinds of useful information. These large-scale data need to be obtained from various application systems. For large-scale companies, there are many application systems, and data generation, storage methods and specifications are different, especially the log data of application systems.

Therefore, how to construct an effective log transmission system to reliably and real-time transmit large-scale log data to the big data platform for data processing is an important problem that needs to be solved urgently.

SUMMARY OF THE INVENTION

The embodiments of the present invention provide a data transmission control method, device, device and storage medium, which strengthen the supervision and control of the transmission flow and nodes while allocating data to multiple nodes in the transmission flow for parallel transmission. .

In a first aspect, an embodiment of the present invention provides a data transmission control method, including:

Obtain transmission configuration information, and perform split processing on the data sources in the transmission configuration information, to obtain a split result matching the set parallel processing quantity;

According to the destination source and the local computing operator in the transmission configuration information, local transmission configuration information corresponding to each split result is generated, and the local transmission configuration is distributed to at least one target proxy device in the proxy device set information;

Wherein, each of the local transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission;

Receive the transmission process information of each target node in the transmission stream reported by the at least one target proxy device, and monitor the transmission process and/or the real-time running state of the target node according to the transmission process information.

In a second aspect, an embodiment of the present invention further provides a data transmission control device, including:

a data source splitting module, used for acquiring transmission configuration information, and performing split processing on the data sources in the transmission configuration information to obtain a split result matching the set number of parallel processing;

The local transmission configuration information distribution module is configured to generate local transmission configuration information corresponding to each of the split results according to the destination source and the local computing operator in the transmission configuration information, and send the information to at least one of the proxy device sets. the target proxy device distributes the local transmission configuration information;

Wherein, each of the local transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission;

A real-time operating state monitoring module, configured to receive the transmission process information of each target node in the transmission stream periodically reported by the at least one target proxy device, and monitor the transmission stream and/or the target according to the transmission process information The real-time running status of the node.

In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implementing the program as described in the present invention when the processor executes the program The data transmission control method described in any one of the embodiments.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the data transmission control method as described in any of the embodiments of the present invention .

Embodiments of the present invention provide a data transmission control method, device, device, and storage medium. By splitting data sources in transmission configuration information, local transmission configuration information corresponding to each split result is generated and distributed to an agent The target proxy device in the device set instructs each target proxy device to run the corresponding target node, and generates a transport stream for data transmission in combination; and monitors the transmission process information of each target node in the transport stream reported by the target proxy device. By means of the transport stream, and/or the real-time running status of the target node, by uniformly deploying the proxy devices in the proxy device set, a transport stream composed of multiple target nodes can be quickly generated to realize the parallel transmission of the data source, and through the Real-time acquisition of information on the transmission of each target node can further strengthen the supervision and control of the transmission stream and nodes during the data transmission process, and realize the transmission control method of the transmission stream or the target node during the data transmission process. Dynamic adjustment to further improve data transmission efficiency.

Description of drawings

1 is a flowchart of a data transmission control method in Embodiment 1 of the present invention;

2 is a flowchart of a data transmission control method in Embodiment 2 of the present invention;

3 is a flowchart of a data transmission control method in Embodiment 3 of the present invention;

4 is a structural diagram of a data transmission control device in Embodiment 4 of the present invention;

FIG. 5 is a schematic structural diagram of a computer device in Embodiment 5 of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all structures related to the present invention.

In addition, it should be noted that, for the convenience of description, the drawings only show some but not all of the contents related to the present invention. Before discussing the exemplary embodiments in greater detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart depicts various operations (or steps) as a sequential process, many of the operations may be performed in parallel, concurrently, or concurrently. Additionally, the order of operations can be rearranged. The process may be terminated when its operation is complete, but may also have additional steps not included in the figures. The processes may correspond to methods, functions, procedures, subroutines, subroutines, and the like.

In order to facilitate the understanding of the technical solutions of the embodiments of the present invention, first, the implementation mechanisms of the embodiments of the present invention are briefly described. The technical solutions of the embodiments of the present invention mainly involve the following hardware devices: multiple data source storage locations, multiple destination source storage locations, a management device (typically, it can be implemented through a cluster), and a set of proxy devices, wherein the set of proxy devices includes Including multiple proxy devices. The methods of the embodiments of the present invention are mainly completed by the above-mentioned management device.

Among them, when there is a data source (typically, log data) stored in the set storage location that needs to be backed up or transmitted to the destination source storage location, a transmission configuration information may be sent to the management device, and the transmission configuration information indicates that the transmission needs to be transmitted. The data source, the destination source storage location to which the data source needs to be transmitted, and the data processing method of the data source during the transmission process. Based on the transmission configuration information, the management device divides the data source into small units of data with a set parallel processing quantity, so as to perform parallel transmission of the data source. In a specific example, the data source contains 10,000 pieces of data. If the pre-specified number of parallel processing is 10, the above-mentioned data source can be divided into 10 small unit data of 1,000 pieces of data. Afterwards, the management device may select one or more proxy devices from multiple proxy device sets, and send a corresponding amount of small unit data, so that the proxy device starts a corresponding process (that is, according to the received number of small unit data) node) respectively transmits each small unit data to the corresponding destination source storage location. In a specific example, the management device allocates three small cell data to a selected proxy device, and after receiving the above small cell data, the proxy device starts three threads (nodes) to store the above three small cell data Transfer to the destination source storage location separately. It should be noted that the set of nodes opened when transmitting the data source together constitutes a transmission stream.

Example 1

FIG. 1 is a flowchart of a data transmission control method according to Embodiment 1 of the present invention. This embodiment is applicable to the case where a data source is concurrently transmitted to a destination source storage location through one or more proxy devices. It can be executed by the data transmission control device provided by the embodiment of the present invention, and the device can be implemented in software and/or hardware. The device can be integrated into a server or a server cluster, and the server or server cluster is used as a management device. One or more proxy devices in the collection are used together. As shown in Figure 1, the method of this embodiment specifically includes:

S110. Acquire transmission configuration information, and perform split processing on the data sources in the transmission configuration information to obtain a split result matching the set number of parallel processing.

As mentioned above, the transmission configuration information is configuration information sent to the management device. The transmission configuration information may include: data source information, destination source information, and local computing operator information.

Wherein, the data source information is used to specify the storage location of the data source to be transmitted, etc., through the data source information, the management device can be made to obtain the data source to be transmitted. Typically, the data source information may include: a log cluster path or a log path. Meanwhile, the data format of the data source may be: log, file, secondary file or various types of database files, etc.

Destination source information, which is used to specify the destination source storage location to which the data source is transmitted. Typically, the destination source information may include: hdfs (Hadoop Distributed File System, distributed file system) path, BigPipe (redesigned basis Dynamic web service system) path, kafka (an open source stream processing platform developed by the Apache Software Foundation) path and file path, etc.

The local computing operator information is used to specify the processing method or data conversion method of the data source to be transmitted by the node. For example, in the process of backing up the data source from the data source storage location to the destination source storage location, in order to adapt to its In the storage form of the destination source storage location, the data format needs to be converted for the data source, and the specific data format conversion method is determined by the local computing operator.

The above-mentioned transmission configuration information is mainly configured in the configuration interface pushed by the management device to the user, and the user can flexibly select data source information, destination source information and local computing operator information according to actual needs.

After the management device acquires the transmission configuration information, it can perform split processing on the data sources in the transmission configuration information, and obtain a split result matching the set number of parallel processing. The splitting process may be data splitting or data set splitting. The purpose of the specific splitting operation is to divide a data source with a huge amount of data into multiple local data sources with a small amount of data, so as to achieve parallelization data processing.

As mentioned above, the data source needs to be split and processed according to a preset number of parallel processing to obtain multiple split results (that is, the aforementioned small unit data). The number of parallel processing may be preset according to the actual load situation or the amount of data included in the data source, or may be directly specified by the user, which is not limited in this embodiment.

S120. Generate local transmission configuration information corresponding to each split result according to the destination source and local computing operator in the transmission configuration information, and distribute the local transmission configuration information to at least one target proxy device in the proxy device set Transfer configuration information.

Wherein, each of the partial transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission.

In this embodiment, each partial transmission configuration information corresponds to a split result, and the partial transmission configuration information includes: data source information, destination source information, and local computing algorithms of some data sources included in the corresponding split result. sub information.

Among them, each target node starts the process corresponding to the received local transmission configuration information as a node according to the quantity value of the received local transmission configuration information, and allocates a corresponding local transmission configuration information for each node for processing, and each node is allocated according to the assigned value. The data source information, destination source information and local computing operator information of some data sources included in the local transmission configuration information of the data source can be correspondingly established three processes (process), which are executed respectively: 1. According to the data source information, the partial data source information 2. Perform data conversion on the read data source according to the local computing operator; and 3. Write the data after data conversion into the destination source storage location corresponding to the destination source information.

In this embodiment, it is necessary to establish a node in the at least one target node that matches the quantity value of the local transmission configuration information, and perform parallel data transmission processing on all data sources corresponding to the transmission configuration information. The nodes collectively form a transport stream for transporting all data sources.

The quantity of local transmission configuration information allocated to each target node may be the same or different. Specifically, the number of local transmission configuration information allocated by each node may be determined according to the actual load of each target node, or all local transmission configuration information may be directly equally distributed to each target node, which is not limited in this embodiment. .

S130. Receive the transmission process information of each target node in the transmission stream reported by the at least one target proxy device, and monitor the transmission process and/or the real-time running state of the target node according to the transmission process information.

In this embodiment, in order to strengthen the effective monitoring of the transmission stream and each target node, it is necessary to control each target proxy device to report the transmission process information of each node in the transmission stream in real time.

Typically, each target proxy device can be specified to report the transmission process information of the target node started by itself at regular intervals, such as 5 minutes or 10 minutes, or the management device can report to each The target proxy device sends a reporting instruction, so that each target proxy device reports the transmission process information of the respective activated target nodes according to the received reporting instruction.

The transmission process information specifically refers to the transmission progress of the data source in the local transmission configuration information allocated by a target node. For example, if a target node is allocated 1000 data for transmission, the target node is currently At the time point, 100 pieces of data have been successfully transmitted to the destination source storage location, then at the current time point, the transmission process information reported by the target node is 100/1000=10%.

Wherein, the real-time running state refers to the information used to describe the transmission state of the target node or the transmission stream to the data source, and the real-time running state may include: normal running state, blocking state, slow running state, pause state and abnormal interruption status, etc.

In this embodiment, the real-time running status of each target node or the entire transmission stream can be monitored according to the transmission process information reported by the target node of the target proxy device at different times. For example, if the transmission process information of a target node keeps at 10% within 10 minutes, it can be determined that the target node is currently blocked. If all target nodes or more than 80% of the target nodes in the transmission flow are within 10 minutes, If the increment of the transmission process information does not exceed 2%, it can be determined that the transmission stream is currently in a blocking state.

In this embodiment, based on the real-time monitoring of the transmission process information of each target node in the transport stream, the real-time running status of each target node or the entire transport stream can be dynamically obtained, and then the target node or the transport stream can be quickly and accurately performed. The adjustment of the transmission state to improve the transmission efficiency of the data source.

An embodiment of the present invention provides a data transmission control method. By splitting data sources in transmission configuration information, local transmission configuration information corresponding to each split result is generated and distributed to target proxy devices in a set of proxy devices. , to instruct each target proxy device to run the corresponding target node, and combine to generate a transport stream for data transmission; and monitor the transport stream and/or the target node according to the transmission process information of each target node in the transport stream reported by the target proxy device The real-time running state of the node, through the unified deployment of the proxy devices in the proxy device set, can quickly generate a transmission stream composed of multiple target nodes to realize the parallel transmission of the data source, and through the transmission of each target node. The real-time acquisition of information can further strengthen the supervision and control of the transmission flow and nodes during the data transmission process, and realize the dynamic adjustment of the transmission flow or the transmission control method of the target node during the data transmission process, so as to further improve the data transmission. transmission efficiency.

On the basis of the foregoing embodiments, after receiving the transmission process information of each target node in the transmission stream reported by the at least one target proxy device, the method may further include: performing the processing on the transmission process information of each target node. Update storage; wherein, the stored transmission process information is used to perform retrospective retransmission or breakpoint retransmission of data that has been transmitted by each target node in the restarted transmission stream after the transmission stream is abnormally interrupted .

In this embodiment, the management device may further update and store the received transmission process information of each target node. The so-called update storage means that after receiving the transmission process information reported by each target node, it stores the information sent by each target node. The latest transmission process information, for example, when the transmission process information reported by the target node A for the first time at 2019.04.26.11:00:00 is 5%, the 5% is first stored. When the transmission progress information reported at :05:00 is 30%, the current transmission progress information of the target node is updated and stored as 30%.

The advantage of this setting is that when the transport stream is abnormally interrupted, in the restarted transport stream, the data that has been transmitted by each target node can be retrospectively retransmitted or retransmitted at a breakpoint, so as to avoid data that has already been transmitted. The retransmission of the completed data further improves the data transmission efficiency and shortens the data transmission time.

Among them, the so-called backtracking and retransmission refers to backtracking to the last data transmission location, starting from this location, and continuing to transmit subsequent data; the so-called resuming transmission from a breakpoint means that data is transmitted from the place where the file was interrupted last time. Both of the above two methods can ensure that the data is not retransmitted from the initial position, but can select a new starting point for data transmission according to the current transmission progress of the target node, and thus can reduce the repetition of repeated data after an abnormal situation occurs. transmission.

On the basis of the above embodiments, the local computing operator may be componentized configuration information; wherein, the local computing operator may include: a custom processing component uploaded by a user, and/or a standard system default configuration Handling components.

In this embodiment, in order to further improve the flexibility of managing device functions, the componentized (plug-in) management of the transmission configuration information is implemented. The user can add a custom processing component to the transmission configuration information by adding a plug-in. In order to add new data processing functions that can meet the needs of users, it is convenient to add and adapt to heterogeneous data transmission modes.

Of course, those skilled in the art can understand that, in addition to the above-mentioned local computing operators that can be componentized configuration information, the data source can also be componentized configuration information, thereby enabling users to customize the introduction of unstructured data Correspondingly, when the user needs to set the transmission configuration information, he only needs to add a custom data source configuration component in the configuration page of the transmission configuration information (the main implementation method can be, writing code to inherit the parent class provided in advance by the system, to obtain a new class as a data source configuration component), and specify a custom data source based on the data source component, so as to further enrich the versatility and expansibility of the management device of the embodiment of the present invention.

Embodiment 2

FIG. 2 is a flowchart of a data transmission control method according to Embodiment 2 of the present invention. This embodiment is optimized on the basis of the above-mentioned embodiment. In this embodiment, according to the transmission process information, the The transport stream, and/or the real-time running state of the target node, is embodied as: separately counting the transport process information of each of the target nodes in the transport stream at at least two monitoring time points; Statistical results, calculate the transmission rate and transmission completion degree corresponding to each of the target nodes respectively; determine the real-time operating status of each of the target nodes according to the transmission rate and the transmission completion degree; Statistical calculation is performed on the real-time running state of the transport stream, and the statistical calculation result is used as the real-time running state of the transport stream.

Correspondingly, the method of the embodiment of the present invention includes:

S210: Acquire transmission configuration information, and perform split processing on the data sources in the transmission configuration information to obtain a split result matching the set parallel processing quantity.

S220. Generate local transmission configuration information corresponding to each split result according to the destination source and local computing operator in the transmission configuration information, and distribute the local transmission configuration information to at least one target proxy device in the proxy device set Transfer configuration information.

Wherein, each of the partial transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission.

S230. Receive the transmission process information of each target node in the transmission stream reported by the at least one target proxy device.

S240. Statistics of the transmission process information of each of the target nodes in the transmission stream at at least two monitoring time points are performed separately.

The monitoring time point specifically refers to the time point when the management device receives the transmission process information of the target node reported by the agent device.

In this embodiment, by acquiring the transmission process information of each target node reported by the agent device at each monitoring time point, the transmission process information of each target node at two or more consecutive monitoring time points can be summarized.

S250. Calculate the transmission rate and the transmission completion degree corresponding to each of the target nodes according to the statistical result of the transmission process information.

Wherein, according to the data transmission increment of each node at two or more consecutive time points, and the time interval determined by the above two or more time points, the transmission rate of each target node can be determined, and the The latest transmission process information of each target node is used as the transmission completion degree of each target node.

For example, if the transmission progress information reported by target node A at 2019.04.26.11:00:00 is 5%, and the transmission progress information reported by target node A at 2019.04.26.11:05:00 is 30%, it can be determined that target node A is at 5% 25% of the data is transmitted within minutes. According to the total amount of data of the target node, for example: 5M, it can be determined that the data transmission increment within 5 minutes is 1M, and then the transmission rate of the target node A can be determined to be 1/300 ( M/s).

S260. Determine the real-time running state of each target node according to the transmission rate and the transmission completion degree.

S270. Perform statistical calculation on the real-time running state of each target node, and use the statistical calculation result as the real-time running state of the transport stream.

In this embodiment, a statistical calculation rule may be preset, and based on the statistical calculation rule, the real-time running state of each target node is statistically calculated, and the statistical calculation result is obtained as the real-time running state of the transport stream. For example, count the number of target nodes whose transmission speed is greater than or equal to the first speed threshold, the number of target nodes located between the first speed and the second speed, and the number of target nodes whose transmission speed is less than or equal to the second speed, wherein the first The speed is greater than the second speed.

Alternatively, according to the transmission speed of each target node and the transmission completion degree, the transmission speed of the entire transmission stream (the average value of the transmission speed of the target node) and the transmission completion degree (the average value of the transmission completion degree of each target node) can be calculated, as the statistical calculation result.

Of course, those skilled in the art may also adopt other methods to perform statistical calculation on the real-time running state of each of the target nodes, and obtain corresponding statistical calculation results to measure the real-time running state of the transport stream.

An embodiment of the present invention provides a data transmission control method. By splitting data sources in transmission configuration information, local transmission configuration information corresponding to each split result is generated and distributed to target proxy devices in a set of proxy devices. , to instruct each target proxy device to run the corresponding target node, and combine to generate a transport stream for data transmission; and monitor the transport stream and/or the target node according to the transmission process information of each target node in the transport stream reported by the target proxy device The real-time running state of the node, through the unified deployment of the proxy devices in the proxy device set, can quickly generate a transmission stream composed of multiple target nodes to realize the parallel transmission of the data source, and through the transmission of each target node. The real-time acquisition of information can further strengthen the supervision and control of the transmission flow and nodes during the data transmission process, and realize the dynamic adjustment of the transmission flow or the transmission control method of the target node during the data transmission process, so as to further improve the data transmission. transmission efficiency.

Based on the foregoing embodiments, before distributing the local transmission configuration information to at least one target proxy device in the set of proxy devices, the method may further include:

The at least one target proxy device is selected in the proxy device set according to the number of running nodes of each proxy device in the proxy device set and the adopted load balancing algorithm.

On the basis of the above embodiments, the data transmission control method can be implemented by a server cluster composed of at least two standard servers;

Wherein, the server cluster is used to select a matching standard server for the obtained Transfer configuration information for processing.

The advantage of this setting is that each standard server and each target proxy device can be deployed in a distributed manner through a cluster to build a large-scale data transmission system.

Embodiment 3

FIG. 3 is a flowchart of a data transmission control method according to Embodiment 3 of the present invention. This embodiment is optimized on the basis of the foregoing embodiment. In this embodiment, after distributing the local transmission configuration information to at least one target proxy device in the proxy device set, the method further specifically includes: receiving node status information of each target node reported by the at least one target proxy device ;

Correspondingly, after monitoring the transport stream and/or the real-time running status of the target node according to the transport process information, the method further specifically includes: according to the transport stream and/or the real-time running status of each target node , and the node status information of each target node, select a matching transmission control mode, and perform transmission control on the transmission stream and/or the target node.

Correspondingly, the method of this embodiment may include:

S310: Acquire transmission configuration information, and perform split processing on the data sources in the transmission configuration information to obtain a split result matching the set number of parallel processing.

S320. According to the destination source and the local computing operator in the transmission configuration information, generate local transmission configuration information corresponding to each of the split results, and distribute the local transmission configuration information to at least one target proxy device in the proxy device set Transfer configuration information.

Wherein, each of the partial transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission.

S330. Receive the transmission process information of each target node in the transmission stream reported by the at least one target proxy device.

S340. Receive node status information of each target node reported by the at least one target proxy device.

S350. Statistics of the transmission process information of each of the target nodes in the transmission stream at at least two monitoring time points are performed separately.

S360. Calculate the transmission rate and the transmission completion degree corresponding to each of the target nodes according to the statistical result of the transmission process information.

S370. Determine the real-time running state of each target node according to the transmission rate and the transmission completion degree.

S380. Perform statistical calculation on the real-time running state of each target node, and use the statistical calculation result as the real-time running state of the transport stream.

S390. According to the transport stream, and/or the real-time running status of each target node, and the node status information of each target node, select a matching transmission control mode, and control the transport stream and/or the target node. Carry out transport control.

In an optional implementation manner of this embodiment, the transmission control mode for the transport stream includes at least one of the following: restart, pause, delete, and modify;

The transmission control mode for the target node includes at least one of the following: restart, pause, and restart after pause.

In this embodiment, a corresponding transmission control mode may be selected to perform transmission control on the transport stream or the target node according to the real-time running state of the transport stream or the target node.

In a specific example, if a target node is currently in a blocked state, the target node can be restarted, or the target node can be suspended first and then restarted after a period of time, so as to achieve the purpose of the above transmission control. It is to improve the current blocking state of the target node.

On the basis of the foregoing embodiments, the transmission configuration information may further include: an option for enabling at least one additional function;

Correspondingly, after acquiring the transmission configuration information, it may further include: acquiring a target accessory function included in the transmission configuration information that is selected to be enabled; The target additional function.

The advantage of this setting is that the user can customize his own task framework according to the actual data upload requirements, and the customized content (target attachment function) can be used as an independent component and run in the proxy device independently.

Optionally, the additional functions may include at least one of the following: a notification function, an alarm function, a self-check function, and an integrity detection function.

Among them, the notification function specifically means that when a transmission stream completes data, it will notify the system where the destination source is located that the transmission has ended. There are two main ways of notification, one is real-time (calling the downstream interface), One is delayed (an identification file is issued, and the destination source scans the document by itself); the alarm function specifically refers to the management device scanning the running status of the proxy device, monitoring the processing pressure of the proxy device, and finding that the proxy device has a processing abnormality. , send an alarm signal to the corresponding proxy device; the self-check function refers to providing a log query method, specifically, the management device reads the log interface of the proxy device, and can view the activity status of the proxy device reported by the proxy device in the management device. .

Embodiment 4

FIG. 4 is a schematic structural diagram of a data transmission control device according to Embodiment 4 of the present invention. As shown in FIG. 4 , the device includes: a data source splitting module 410, a local transmission configuration information distribution module 420, and a real-time operation Status monitoring module 430 . in:

The data source splitting module 410 is configured to acquire transmission configuration information, and perform split processing on the data sources in the transmission configuration information to obtain a split result matching the set number of parallel processing.

The local transmission configuration information distribution module 420 is configured to generate local transmission configuration information corresponding to each of the split results according to the destination source and the local computing operator in the transmission configuration information, and send the information to at least one of the proxy device sets. A target proxy device distributes the local transport configuration information.

Wherein, each of the local transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission;

A real-time operating state monitoring module 430, configured to receive the transmission process information of each target node in the transmission stream periodically reported by the at least one target proxy device, and monitor the transmission stream according to the transmission process information, and/or The real-time running status of the target node.

An embodiment of the present invention provides a data transmission control device, which generates local transmission configuration information corresponding to each split result by splitting data sources in transmission configuration information and distributes it to target proxy devices in a proxy device set , to instruct each target proxy device to run the corresponding target node, and combine to generate a transport stream for data transmission; and monitor the transport stream and/or the target node according to the transmission process information of each target node in the transport stream reported by the target proxy device The real-time running state of the node, through the unified deployment of the proxy devices in the proxy device set, can quickly generate a transmission stream composed of multiple target nodes to realize the parallel transmission of the data source, and through the transmission of each target node. The real-time acquisition of information can further strengthen the supervision and control of the transmission flow and nodes during the data transmission process, and realize the dynamic adjustment of the transmission flow or the transmission control method of the target node during the data transmission process, so as to further improve the data transmission. transmission efficiency.

On the basis of the above embodiments, the real-time operating state monitoring module 430 can be specifically used for:

separately collect statistics on the transmission process information of each of the target nodes in the transmission stream at at least two monitoring time points;

According to the statistical result of the transmission process information, calculate the transmission rate and transmission completion degree corresponding to each of the target nodes respectively;

According to the transmission rate and the transmission completion degree, determine the real-time operating state of each of the target nodes;

Perform statistical calculation on the real-time running state of each target node, and use the statistical calculation result as the real-time running state of the transport stream.

On the basis of the above embodiments, it can also include:

a node state information receiving module, configured to receive the node state information of each target node reported by the at least one target proxy device after distributing the local transmission configuration information to at least one target proxy device in the proxy device set;

Correspondingly, it may also include: a transmission control module, configured to monitor the transmission flow and/or the real-time running state of the target node according to the transmission process information Based on the real-time running state of the target node and the node state information of each target node, a matching transmission control mode is selected to perform transmission control on the transmission stream and/or the target node.

Based on the foregoing embodiments, the transmission control manner for the transport stream may include at least one of the following: restart, pause, delete, and modify; and/or

The transmission control mode for the target node may include at least one of the following: restart, pause, and restart after pause.

On the basis of the above embodiments, it can also include:

a progress information storage module, configured to update and store the transmission process information of each target node after receiving the transmission process information of each target node in the transmission stream reported by the at least one target proxy device;

The stored transmission process information is used to perform retrospective retransmission or breakpoint retransmission of data that has been transmitted by each target node in the restarted transmission stream after the transmission stream is abnormally interrupted.

On the basis of the above embodiments, it can also include:

The target proxy device selection module is configured to, before distributing the local transmission configuration information to at least one target proxy device in the proxy device set, according to the number of running nodes of each proxy device in the proxy device set and the adopted load balancing algorithm , selecting the at least one target proxy device in the proxy device set.

Based on the above embodiments, the local computing operator may be componentized configuration information;

Wherein, the local computing operator may include: a custom processing component uploaded by a user, and/or a standard processing component configured by default in the system.

On the basis of the foregoing embodiments, the transmission configuration information may further include: an option for enabling at least one additional function;

Correspondingly, it may further include: an additional function enabling module, configured to obtain the selected and enabled target accessory function included in the transmission configuration information after obtaining the transmission configuration information;

A service program matching the target additional function is started to provide the target additional function during data transmission.

Based on the above embodiments, the additional functions may include at least one of the following: a notification function, an alarm function, a self-check function, and an integrity detection function.

On the basis of the above embodiments, the data transmission control device is implemented by a server cluster composed of at least two standard servers;

Wherein, the server cluster is used to select a matching standard server for the obtained Transfer configuration information for processing.

The above data transmission control apparatus can execute the data transmission control method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the executed data transmission control method.

Embodiment 5

FIG. 5 is a schematic structural diagram of a computer device according to Embodiment 5 of the present invention. Figure 5 shows a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 5 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present invention.

As shown in FIG. 5, computer device 12 takes the form of a general-purpose computing device. Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16 , system memory 28 , and a bus 18 connecting various system components including system memory 28 and processing unit 16 .

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect ( PCI) bus.

Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including both volatile and nonvolatile media, removable and non-removable media.

System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 . Computer device 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 5, commonly referred to as a "hard disk drive"). Although not shown in Figure 5, a disk drive may be provided for reading and writing to removable non-volatile magnetic disks (eg "floppy disks"), as well as removable non-volatile optical disks (eg CD-ROM, DVD-ROM) or other optical media) to read and write optical drives. In these cases, each drive may be connected to bus 18 through one or more data media interfaces. Memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present invention.

A program/utility 40 having a set (at least one) of program modules 42, which may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other programs Modules and program data, each or some combination of these examples may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the described embodiments of the present invention.

Computer device 12 may also communicate with one or more external devices 14 (eg, keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable a user to interact with computer device 12, and/or communicate with Any device (eg, network card, modem, etc.) that enables the computer device 12 to communicate with one or more other computing devices. Such communication may take place through input/output (I/O) interface 22 . Also, the computer device 12 may communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through a network adapter 20 . As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18 . It should be understood that, although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tapes drives and data backup storage systems.

The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the data transmission control method provided by the embodiments of the present invention.

That is: when the processing unit executes the program, it realizes: obtains the transmission configuration information, and performs split processing on the data sources in the transmission configuration information, so as to obtain a split result matching the set parallel processing quantity;

According to the destination source and the local computing operator in the transmission configuration information, local transmission configuration information corresponding to each split result is generated, and the local transmission configuration is distributed to at least one target proxy device in the proxy device set information;

Wherein, each of the local transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission;

Receive the transmission process information of each target node in the transmission stream reported by the at least one target proxy device, and monitor the transmission process and/or the real-time running state of the target node according to the transmission process information.

Embodiment 6

Embodiment 6 of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the data transmission control method provided by all the inventive embodiments of the present application:

That is: when the program is executed by the processor, it is realized that the transmission configuration information is obtained, and the data source in the transmission configuration information is split and processed to obtain a split result matching the set parallel processing quantity;

According to the destination source and the local computing operator in the transmission configuration information, local transmission configuration information corresponding to each split result is generated, and the local transmission configuration is distributed to at least one target proxy device in the proxy device set information;

Wherein, each of the local transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission;

Receive the transmission process information of each target node in the transmission stream reported by the at least one target proxy device, and monitor the transmission process and/or the real-time running state of the target node according to the transmission process information.

Any combination of one or more computer-readable media may be employed. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (a non-exhaustive list) of computer readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), Erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signal in baseband or as part of a carrier wave, with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .

Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention. The scope is determined by the scope of the appended claims.

Claims (12)

1. a transmission control method of data, is characterized in that, comprises: Obtain transmission configuration information, and perform split processing on the data sources in the transmission configuration information to obtain a split result matching the set number of parallel processing; the transmission configuration information is used to indicate the storage location of the data source, and the data The storage location of the destination source to which the source needs to be transmitted, and the data processing method of the data source during the transmission process; According to the destination source and the local computing operator in the transmission configuration information, local transmission configuration information corresponding to each split result is generated, and the local transmission configuration is distributed to at least one target proxy device in the proxy device set information; Wherein, each of the local transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission; Receive the transmission process information of each target node in the transmission stream reported by the at least one target proxy device, and monitor the transmission process and/or the real-time running state of the target node according to the transmission process information; Wherein, monitoring the transmission stream and/or the real-time running status of the target node according to the transmission process information includes: separately collect statistics on the transmission process information of each of the target nodes in the transmission stream at at least two monitoring time points; According to the statistical result of the transmission process information, calculate the transmission rate and transmission completion degree corresponding to each of the target nodes respectively; According to the transmission rate and the transmission completion degree, determine the real-time operating state of each of the target nodes; Perform statistical calculation on the real-time running state of each target node, and use the statistical calculation result as the real-time running state of the transport stream. 2. The method according to claim 1, wherein after distributing the local transmission configuration information to at least one target proxy device in the set of proxy devices, the method further comprises: receiving the node status information of each target node reported by the at least one target proxy device; After monitoring the transmission flow and/or the real-time running state of the target node according to the transmission process information, the method further includes: According to the transport stream, and/or the real-time operating status of each target node, and the node status information of each target node, select a matching transmission control mode to transmit the transport stream and/or target node control. 3. method according to claim 2, is characterized in that: The transmission control mode for the transport stream includes at least one of the following: restart, pause, delete, and modify; and/or The transmission control mode for the target node includes at least one of the following: restart, pause, and restart after pause. 4. The method according to claim 1, wherein after receiving the transmission process information of each target node in the transmission stream reported by the at least one target proxy device, the method further comprises: updating and storing the transmission process information of each of the target nodes; The stored transmission process information is used to perform retrospective retransmission or breakpoint retransmission of data that has been transmitted by each target node in the restarted transmission stream after the transmission stream is abnormally interrupted. 5. The method according to claim 1, wherein before distributing the local transmission configuration information to at least one target proxy device in the set of proxy devices, the method further comprises: The at least one target proxy device is selected in the proxy device set according to the number of running nodes of each proxy device in the proxy device set and the adopted load balancing algorithm. 6. The method according to any one of claims 1-5, wherein the local computing operator is componentized configuration information; Wherein, the local computing operator includes: a custom processing component uploaded by a user, and/or a standard processing component configured by default in the system. 7. The method according to any one of claims 1-5, wherein the transmission configuration information further comprises: an enabling option of at least one additional function; After obtaining the transmission configuration information, it also includes: Obtain the target accessory function that is selected to be enabled and included in the transmission configuration information; A service program matching the target additional function is started to provide the target additional function during data transmission. 8. The method according to claim 7, wherein the additional functions include at least one of the following: a notification function, an alarm function, a self-check function, and an integrity detection function. 9. The method according to any one of claims 1-5, wherein: The data transmission control method is implemented by a server cluster composed of at least two standard servers; Wherein, the server cluster is used to select a matching standard server for the obtained Transfer configuration information for processing. 10. A data transmission control device, comprising: A data source splitting module, configured to obtain transmission configuration information, and perform split processing on the data sources in the transmission configuration information to obtain a split result matching the set number of parallel processing; the transmission configuration information is used for Indicate the storage location of the data source, the storage location of the destination source to which the data source needs to be transmitted, and the data processing method of the data source during the transmission process; The local transmission configuration information distribution module is configured to generate local transmission configuration information corresponding to each of the split results according to the destination source and the local computing operator in the transmission configuration information, and send the information to at least one of the proxy device sets. the target proxy device distributes the local transmission configuration information; Wherein, each of the local transmission configuration information is used to instruct each of the target proxy devices to run at least one target node respectively, and to combine and generate a transmission stream matching the transmission configuration information for data transmission; A real-time operating state monitoring module, configured to receive the transmission process information of each target node in the transmission stream periodically reported by the at least one target proxy device, and monitor the transmission stream and/or the target according to the transmission process information The real-time running status of the node; Among them, the real-time operating status monitoring module is specifically used for: separately collect statistics on the transmission process information of each of the target nodes in the transmission stream at at least two monitoring time points; According to the statistical result of the transmission process information, calculate the transmission rate and transmission completion degree corresponding to each of the target nodes respectively; According to the transmission rate and the transmission completion degree, determine the real-time operating state of each of the target nodes; Perform statistical calculation on the real-time running state of each target node, and use the statistical calculation result as the real-time running state of the transport stream. 11. A computer device, comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements any of claims 1-9 when the processor executes the program. a described method. 12. A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the method according to any one of claims 1-9 is implemented.

Images