CN118842766A - Government cloud multi-source data transmission detection system and method - Google Patents

Abstract

The invention relates to the technical field of data transmission, and discloses a government cloud multisource data transmission detection system and method, wherein the system comprises the following steps: a data transmitting end and a data receiving end; the data transmitting terminal comprises: a to-be-transmitted database for storing all to-be-transmitted data; a segmentation module for segmenting the data to be transmitted and endowing the segmented data with a check code; a queue module for determining transmission priority and retransmitting data packets; determining a verification module needing to resend the missing data packet according to the missing form; the data receiving terminal comprises: a verification module for confirming the missing data packet and generating a missing form; and the feedback module is used for sending the missing form and the recombination module is used for recombining the missing form into complete data. The invention obviously improves the reliability, efficiency and safety of data transmission and the integrity of the data in the transmission process by comprehensively applying the policies of data segmentation, verification, priority adjustment, compression policy, missing data packet management and the like.

Description

A government cloud multi-source data transmission detection system and method

Technical Field

The present invention relates to the field of data transmission technology, and in particular to a government cloud multi-source data transmission detection system and method.

Background Art

Government cloud multi-source data transmission refers to the process of transmitting and integrating data from different sources through the network in the government cloud environment to achieve data sharing and collaborative application. The government cloud is an information platform built by relevant departments and institutions based on cloud computing technology to centrally manage and share various government data. Multi-source data transmission involves data exchange and integration between several data sources.

The multi-source data transmission in the government cloud mainly relies on traditional transmission protocols and methods. Although they can realize the basic data transmission function, they can only perform simple verification of data packets, making it difficult to fully guarantee the integrity of data during transmission and accurately control the transmission process. Once a data packet is lost or damaged, the existing mechanism cannot detect it in time and handle it effectively.

Therefore, it is particularly important to design a government cloud multi-source data transmission detection system or method to ensure data integrity.

Summary of the invention

In view of this, the present invention proposes a government cloud multi-source data transmission detection system and method, aiming to solve the problem that it is difficult to ensure the data integrity of government cloud multi-source data transmission in current technology.

On the one hand, the present invention proposes a government cloud multi-source data transmission detection system, including: a data sending end and a data receiving end;

The data sending end comprises:

A to-be-transmitted database, configured to receive and store all to-be-transmitted data;

A segmentation module is configured to obtain the data to be transmitted in the database to be transmitted, segment the data to be transmitted into a plurality of data packets, and bundle a check code for each of the data packets;

A queue module is configured to sort the data packets according to the transmission priority according to the throughput and the queuing delay of the data packets at the data sending end, and send the data packets and a confirmation form including the data packets and the check code bundled with the data packets according to the transmission priority; the queue module is also configured to resend the missing data packets;

The verification module is configured to request the data receiving end to send a missing form after the data packet is successfully sent, and determine the missing data packet to be resent according to the missing form; the missing form includes the name of the missing data packet and the corresponding check code;

The data receiving end comprises:

A verification module, configured to receive the data packet and the check code, confirm the missing data packet and generate the missing form;

A feedback module, configured to send the missing form to the data sending end;

The reassembly module is configured to reassemble the received data packets into complete data in original order.

Furthermore, when the database to be transmitted receives and stores all the data to be transmitted, it includes:

Acquire the received data to be transmitted and record the receiving time, pre-set a character number threshold for the data to be transmitted, and compare the actual number of characters of the data to be transmitted with the character number threshold; when the actual number of characters is less than the character number threshold, do not process the data to be transmitted; when the actual number of characters is greater than or equal to the character number threshold, add the data to be transmitted to a compression list and determine whether to compress it in turn;

The CPU idle rate is obtained through the real-time CPU usage rate of the computer, and the CPU occupancy rate of each data to be transmitted in the compression list is calculated. When the CPU idle rate is greater than the CPU occupancy rate of the data to be transmitted, the data to be transmitted is backed up and compression is started. When the CPU idle rate is less than or equal to the CPU occupancy rate of the data to be transmitted, the data to be transmitted is moved back one order in the compression list.

Furthermore, when the database to be transmitted receives and stores all the data to be transmitted, it also includes:

A processing time threshold of the database to be transmitted is pre-set, and the real-time processing time is calculated based on the difference between the real-time time and the receiving time. When the real-time processing time is greater than or equal to the processing time threshold, it is determined whether the data to be transmitted has completed compression; if the judgment result is yes, the compressed data to be transmitted is transmitted to the segmentation module; if the judgment result is no, the compression is terminated and the backup is transmitted to the segmentation module.

Further, the data to be transmitted is segmented into a plurality of data packets, and a check code is bundled with each of the data packets, including:

The number of data packets into which the single data to be transmitted is divided is calculated according to the network bandwidth, network delay, packet loss rate and the size of the single data to be transmitted, and the number of data packets satisfies the following relationship:

;

Wherein, N is the number of data packets into which the single data to be transmitted is divided, S is the size of the single data to be transmitted, w _B is the network bandwidth influence coefficient, w _L is the network delay influence coefficient, w _R is the packet loss rate influence coefficient, Indicates that the combination is rounded up;

After determining the number of data packets into which the single data to be transmitted is divided, the data to be transmitted is segmented; after the segmentation is completed, a check code of the data packet is generated based on the SHA-256 check algorithm.

Further, generating a checksum of the data packet based on the SHA-256 checksum algorithm includes:

Read the data content in the segmented data packet, input the data content into the SHA-256 verification algorithm and generate a hash value of a fixed length as the verification code, and attach the generated verification code to the metadata part of the data packet, the metadata part includes a data packet sequence number, a data packet size and a verification code, the data packet sequence number is used for sequence information when reassembling the data packet, and the data packet size is used to verify whether it is lost or damaged during transmission.

Furthermore, the network bandwidth impact coefficient is determined by the following method:

The actual bandwidth of the current network is obtained by measuring, the maximum bandwidth is obtained by historical network bandwidth, and the ratio of the actual bandwidth to the maximum bandwidth is calculated and used as the network bandwidth influence coefficient;

The network delay impact coefficient is determined by the following method:

Measure the actual delay of the network, obtain the minimum delay through the historical network delay, calculate the ratio of the actual delay to the minimum delay, and measure the minimum delay jitter based on the actual delay jitter obtained by the delay jitter measurement tool and the historical record; determine the network delay impact coefficient according to the ratio of the actual delay to the minimum delay and the product of the ratio of the actual delay jitter to the minimum delay jitter, and the network delay impact coefficient satisfies the following relationship:

;

The packet loss rate influence coefficient is determined by the following formula:

;

in, is the actual delay, is the minimum delay, is the actual delay jitter, is the minimum delay jitter; R is the packet loss rate, and 0≤R≤1.

Further, the data packets are sorted according to the transmission priority according to the throughput and the queuing delay of the data packets at the data sending end, including:

Obtaining the throughputs of the data receiving end and the data sending end and recording them as receiving throughput and sending throughput respectively, and determining the actual throughput according to the relationship between the receiving throughput and the sending throughput;

Obtaining the queuing delay of each of the data packets, and sorting the corresponding data packets according to the queuing delay from small to large in terms of transmission priority;

After the sorting is completed, the relationship between the actual throughput and the size of the data packet is determined. When the actual throughput is greater than the size of the data packet, the transmission priority of the data packet is relegated one place.

Further, a queuing delay threshold is preset, and according to the magnitude relationship between the current queuing delay and the queuing delay threshold, it is determined whether to stop the transmission priority of the data packet and retreat one position;

When the queuing delay is greater than or equal to the queuing delay threshold, the transmission priority adjustment of the data packet is stopped; when the queuing delay is less than the queuing delay threshold, the transmission priority of the data packet continues to be relegated one place.

Furthermore, the data packet size is used to verify whether loss or damage occurs during transmission, including:

The data receiving end compares the size of the received data packet with the record in the metadata. If the size of the received data packet is different from the record in the metadata, it is determined that the data packet is lost or damaged during transmission, and the data sending end resends the missing or damaged data packet.

On the other hand, the present invention also proposes a method applied to the above-mentioned government cloud multi-source data transmission detection system, the method comprising:

S1: Receive and store all data to be transmitted;

S2: Acquire the data to be transmitted in the database to be transmitted, segment the data to be transmitted into a plurality of data packets, and bundle a check code for each data packet;

S3: sorting the data packets according to the throughput and the queuing delay of the data packets at the data sending end according to the transmission priority, sending the data packets and a confirmation form including the data packets and the checksum bundled with the data packets according to the transmission priority; when the data packets are lost or damaged during the transmission process, resending the data packets;

S4: After the data packet is successfully sent, the data receiving end is requested to send a missing list, and the missing data packet to be resent is determined according to the missing list; the missing list includes the name of the missing data packet and the corresponding check code;

S5: receiving the data packet and the check code, confirming the missing data packet and generating the missing table;

S6: Sending the missing form to the data sending end;

S7: Reassemble the received data packets into complete data in the original order.

Compared with the prior art, the present invention has the following beneficial effects:

By segmenting the data at the data sending end and adding a checksum to each data packet, it is ensured that the loss, damage or tampering of the data packet can be detected during the transmission process, ensuring the integrity and accuracy of the data. The system can detect the loss and damage of data packets in real time, and resend the missing data packets in time through the queue module to reduce the impact of data loss on the business. The segmentation module calculates the number of data packets based on factors such as network bandwidth, delay and packet loss rate, realizes the reasonable segmentation of the data to be transmitted, and improves the efficiency and stability of data transmission. The queue module adjusts the transmission priority of the data packet according to the queuing delay and throughput of the data packet, optimizes the transmission order of the data packet, and improves the transmission efficiency of the system. The database to be transmitted determines whether to compress the data to be transmitted through the real-time CPU usage of the computer, reasonably utilizes system resources, and improves the efficiency of data processing. The data receiving end can automatically generate a missing form and send it to the data sending end to ensure the accurate identification and rapid resending of the missing data packets. The reorganization module at the data receiving end can reorganize the received segmented data packets into complete data in the original order to ensure the integrity and accuracy of the data sequence.

BRIEF DESCRIPTION OF THE DRAWINGS

Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

FIG1 is a functional framework diagram of a multi-source data transmission detection system for a government cloud according to an embodiment of the present invention;

FIG2 is a flow chart of a method for detecting multi-source data transmission in a government cloud according to an embodiment of the present invention.

DETAILED DESCRIPTION

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

As shown in FIG1 , an embodiment of the present invention provides a government cloud multi-source data transmission detection system, including: a data sending end and a data receiving end;

The data sending end includes:

A to-be-transmitted database, configured to receive and store all to-be-transmitted data;

A segmentation module is configured to obtain the data to be transmitted in the database to be transmitted, segment the data to be transmitted into a plurality of data packets, and bundle a checksum for each data packet;

A queue module is configured to sort the data packets according to the throughput and the queuing delay of the data packets at the data transmission end, and send the data packets and the confirmation form containing the data packets and the checksums bundled with the data packets according to the transmission priority; the queue module is also configured to resend the missing data packets;

The verification module is configured to request the data receiving end to send a missing form after the data packet is successfully sent, and determine the missing data packet to be resent according to the missing form; the missing form includes the name of the missing data packet and the corresponding check code;

The data receiving end includes:

A verification module is configured to receive a data packet and a check code, confirm a missing data packet and generate a missing form;

a feedback module configured to send the missing form to a data sending end;

The reassembly module is configured to reassemble the received data packets into complete data in original order.

It should be noted that the database to be transmitted is a data storage module in the system, which is responsible for receiving and storing all data to be transmitted. It ensures that all data to be transmitted can be effectively managed during the transmission process. The database to be transmitted provides the system with a data storage basis, ensures the integrity and accessibility of all data to be transmitted, and lays the foundation for the subsequent processing and transmission of data. The segmentation module obtains the data to be transmitted from the database to be transmitted, segments it into several data packets, and generates and attaches a checksum to each data packet. The checksum is generated using a hash algorithm such as SHA-256 to ensure the integrity of the data. The large data block is divided into small data packets suitable for network transmission, which effectively improves the efficiency of data transmission. The checksum is used to verify the integrity of the data at the receiving end to prevent the data from being tampered with or damaged during transmission. The queue module prioritizes the data packets according to the throughput and the queuing delay of the data packets. It sends the data packets according to the calculated priority and comes with a confirmation form. The queue module is also responsible for resending the data packets lost during the transmission process. The queue module can optimize the transmission order of data packets and improve the efficiency of network utilization. It ensures that important data packets are processed first, and at the same time, the lost data packets are processed through the resending mechanism to ensure the reliability of data transmission. After the data packet is successfully sent, the verification module requests the data receiving end to send a missing form. The missing form contains the name and check code of the missing data packet, which is used to confirm the data packet that needs to be resent. It detects the loss during the data packet transmission process and requires the receiving end to report the missing data to ensure that all data packets can be successfully received in the end. The verification module in the data receiving end receives the data packet and the check code, checks the integrity of the data packet, and generates a missing form. The missing form records the information of the lost or damaged data packet. The integrity of the received data packet is verified, and a missing form is generated to report the transmission problem to ensure that any data loss can be discovered and processed in time. The feedback module sends the missing form back to the data sending end to notify it of the data packet that needs to be resent. The information of the missing data is fed back to the data sending end, and the data packet resending process is started to ensure the complete reception of the data. The reassembly module reassembles the received data packets into complete data in the order of the original data. The function is to integrate the received segmented data into the original data in order to ensure the integrity and correctness of the final data.

The embodiment of the present invention achieves high efficiency and reliability of government cloud data transmission through the following methods:

Data segmentation and verification: By segmenting the data and adding a verification code, it is ensured that the data is not tampered with during the transmission process, which can effectively deal with the problem of data loss and damage. Dynamic priority adjustment: By sorting and adjusting the priority of data packets, the efficiency of data transmission and the utilization of network resources are improved. Automated missing data processing: By automatically generating missing forms and feedback missing data, it is ensured that the data lost during the data transmission process can be resent in time. Complete data reorganization: The received data packets are reorganized in order to ensure the integrity and accuracy of the final data.

In some embodiments of the present application, when the to-be-transmitted database receives and stores all the data to be transmitted, it includes:

The received data to be transmitted is obtained and the receiving time is recorded, a character number threshold of the data to be transmitted is pre-set, and the actual number of characters of the data to be transmitted is compared with the character number threshold; when the actual number of characters is less than the character number threshold, the data to be transmitted is not processed; when the actual number of characters is greater than or equal to the character number threshold, the data to be transmitted is added to a compression list and it is determined in turn whether to compress it;

The CPU idle rate is obtained through the real-time CPU usage of the computer, and the CPU occupancy rate of each data to be transmitted in the compression list is calculated. When the CPU idle rate is greater than the CPU occupancy rate of the data to be transmitted, the data to be transmitted is backed up and compression is started. When the CPU idle rate is less than or equal to the CPU occupancy rate of the data to be transmitted, the data to be transmitted is moved back one order in the compression list.

It should be noted that the database to be transmitted ensures that the receiving time of each data to be transmitted is recorded, so as to facilitate the subsequent processing and scheduling according to the receiving time of the data. For the comparison of the character number threshold: the system pre-sets a character number threshold to determine the size of the data. The system compares the actual number of characters of the data to be transmitted with the threshold. When the actual number of characters is less than the threshold, the data is not processed; when the actual number of characters is greater than or equal to the threshold, the data will be added to the compression list. Avoid unnecessary compression processing of data less than the threshold, thereby saving system resources and improving processing efficiency. Only larger data will be considered for compression to reduce the burden on the system. At the same time, the system monitors the CPU usage of the computer in real time and calculates the CPU idle rate. According to the CPU occupancy rate of each data to be transmitted in the compression list, decide whether to compress the data. The specific approach is: when the CPU idle rate is higher than the CPU occupancy rate of the data to be transmitted, start compressing the data and keep its backup; when the CPU idle rate is lower than or equal to the CPU occupancy rate of the data, push the data back one order in the compression list and wait for the system resources to be more abundant before processing. Dynamically adjust the timing of data compression to ensure that compression is performed when system resources are sufficient and avoid taking up too much CPU resources when resources are tight. This can optimize the data compression process and improve data processing efficiency without affecting system performance.

This process optimizes the compression strategy of the transmitted data in the following ways:

Data screening: By setting the character count threshold, only large data is compressed, avoiding unnecessary processing of small data and saving system resources. Dynamic resource management: Using real-time CPU usage and idle rate, dynamically adjust the timing of data compression. Ensure that the system compresses when resources are sufficient, avoiding the impact on system performance when resources are tight. Optimize data processing: By adjusting the processing order of data in the compression list, flexible processing is achieved under different system load conditions, improving data processing efficiency and overall system performance.

In some embodiments of the present application, when the to-be-transmitted database receives and stores all the data to be transmitted, it also includes:

The processing time threshold of the database to be transmitted is set in advance, and the real-time processing time is calculated based on the difference between the real-time time and the receiving time. When the real-time processing time is greater than or equal to the processing time threshold, it is determined whether the data to be transmitted has been compressed; if the judgment result is yes, the compressed data to be transmitted is transmitted to the segmentation module; if the judgment result is no, the compression is terminated and the backup is transmitted to the segmentation module.

It should be noted that the system pre-sets a processing time threshold to determine the time limit for data compression. This threshold defines the maximum allowed time from data reception to data processing (including compression). Setting a processing time threshold helps manage the timeliness of data processing and ensures that data compression or other processing operations are completed within the specified time range to meet system performance and response time requirements. The system calculates the real-time processing time based on the difference between the current time and the data reception time. This process determines the time interval from data reception to the current moment. The calculation of the real-time processing time is used to monitor the progress of data processing and ensure that the data processing operation is completed within the set time range. The real-time processing time is compared with the preset processing time threshold. When the real-time processing time is greater than or equal to the processing time threshold, the system determines whether data compression has been completed. If data compression has been completed, the compressed data is transmitted to the segmentation module; if data compression has not been completed, the compression process is terminated and the backup data is transmitted to the segmentation module. This ensures that the system can transmit the data to the segmentation module in a timely manner even if time limits are encountered during the compression process. Terminating compression and transmitting backup data is to avoid affecting the overall data transmission process due to compression delays, thereby maintaining the efficiency and stability of the system.

This process optimizes the flow of data to be transmitted by:

Time management: Through the preset processing time threshold and real-time processing time calculation, the system can effectively monitor and manage the data processing progress to ensure that data processing is completed within a reasonable time.

Flexible processing: When the processing time exceeds the threshold, the system can flexibly adjust the data processing strategy to avoid data transmission delays caused by incomplete compression. By terminating compression and transmitting backup data, the continuity and timeliness of data transmission are maintained.

Improve efficiency: Ensure that data is processed and transmitted within the specified time, reduce performance bottlenecks that may be caused by data processing delays, and improve the efficiency and response speed of the overall system.

In some embodiments of the present application, the data to be transmitted is segmented into a plurality of data packets, and a check code is bundled with each data packet, including:

The number of data packets into which a single data to be transmitted is divided is calculated based on the network bandwidth, network delay, packet loss rate, and the size of a single data to be transmitted. The number of data packets satisfies the following relationship:

;

Among them, N is the number of data packets into which a single data to be transmitted is divided, S is the size of a single data to be transmitted, w _B is the network bandwidth influence coefficient, w _L is the network delay influence coefficient, w _R is the packet loss rate influence coefficient, Indicates that the combination is rounded up;

After determining the number of data packets into which a single data to be transmitted is divided, the data to be transmitted is segmented; after the segmentation is completed, a checksum of the data packet is generated based on the SHA-256 checksum algorithm.

It should be noted that by considering factors such as network bandwidth, network delay and packet loss rate, the number of data packets can be reasonably determined to optimize the efficiency of data transmission and reduce data loss or delay during transmission.

After determining the number of data packets, the data to be transmitted is segmented into several data packets. The size and number of each data packet are adjusted based on the results of the previous calculation to adapt to the network conditions. Dividing large data blocks into small data packets suitable for network transmission ensures that the size of each data packet is moderate, effectively improving network transmission efficiency and reducing network congestion caused by a single large data packet.

The SHA-256 checksum algorithm is used to generate a checksum for each data packet. The specific process includes reading the segmented data packet content, inputting the data content into the SHA-256 algorithm, and generating a fixed-length hash value as the checksum. The generated checksum is then attached to the metadata portion of the data packet. The checksum is used to verify the integrity of the data packet at the data receiving end to ensure that the data has not been tampered with or damaged during transmission. The SHA-256 algorithm provides a high-strength checksum that can effectively prevent data tampering and error detection.

The present application embodiment achieves efficient data transmission and verification through the following methods:

Optimize network conditions: By comprehensively considering factors such as network bandwidth, network delay and packet loss rate to calculate the number of data packets, the data segmentation strategy is more in line with the actual network conditions, improving the reliability of data transmission.

Segment processing: Reasonably segment large data blocks to optimize network transmission efficiency and reduce network congestion while ensuring smooth data transmission.

Strong verification protection: Use the SHA-256 algorithm to generate a verification code to ensure the integrity and accuracy of each data packet and prevent data errors or tampering during transmission.

In some embodiments of the present application, generating a checksum of a data packet based on a SHA-256 checksum algorithm includes:

Read the data content in the segmented data packet, input the data content into the SHA-256 verification algorithm and generate a fixed-length hash value as a verification code, and attach the generated verification code to the metadata part of the data packet. The metadata part includes the data packet sequence number, data packet size and verification code. The data packet sequence number is used for the sequence information when reassembling the data packet, and the data packet size is used to verify whether it is lost or damaged during transmission.

It can be explained that the data content is read from the segmented data packets. Each data packet contains its own data content, which has been extracted from the original data to be transmitted during the segmentation process. The actual content of the data packet is obtained to provide the necessary input data for the subsequent generation of the checksum.

Input the read data content into the SHA-256 verification algorithm. SHA-256 is a cryptographic hash function that inputs data of any length and generates a fixed-length 256-bit (32-byte) hash value. Generate a unique hash value (i.e., verification code) for the data content to verify the integrity of the data during transmission.

The SHA-256 algorithm outputs a fixed-length 256-bit hash value. This hash value is attached to the metadata part of the data packet as the checksum of the data packet. The fixed-length hash value ensures the consistency and reliability of the data content during verification, so that the receiving end can use the same verification algorithm to verify the integrity of the data packet.

The generated SHA-256 checksum is attached to the metadata part of the data packet. The metadata part contains: Data packet sequence number: identifies the order of the data packet in the entire data transmission. Data packet size: records the actual size of the data packet. Checksum: a fixed-length hash value generated by the SHA-256 algorithm. Function: The metadata part provides the necessary information for the sequence reorganization and integrity verification of the data packet. The data packet sequence number is used to reorganize the order of the data packet, the data packet size is used to check whether data loss or damage occurs during the transmission process, and the checksum is used to verify the integrity of the data. Integrity verification: The checksum generated by SHA-256 can effectively detect whether the data packet has been tampered with or damaged during the transmission process. The receiving end can use the same SHA-256 algorithm to verify the received data packet to ensure the integrity of the data. Sequence management: The data packet sequence number enables the receiving end to correctly reorganize the data packet in sequence to ensure the correct order of the data. Data verification: The data packet size is used to verify whether there is a problem with data loss or damage during the transmission process. For example, if the size of the received data packet does not match the record in the metadata, it means that the data packet may have a problem during the transmission process. Through the above processing process, the embodiment of the present application realizes efficient integrity verification and sequence management of the data packet. The SHA-256 checksum algorithm provides strong data integrity protection, ensuring that data is not tampered with or damaged during transmission. The additional metadata section not only helps reassemble the data packet, but is also used to check the size of the data and verify the integrity of the transmission process, thereby improving the reliability and accuracy of the data transmission system.

In some embodiments of the present application, the network bandwidth impact coefficient is determined by the following method:

The actual bandwidth of the current network is obtained through measurement, the maximum bandwidth is obtained through historical network bandwidth, and the ratio of the actual bandwidth to the maximum bandwidth is calculated and used as the network bandwidth impact coefficient;

The network delay impact factor is determined by the following method:

Measure the actual delay of the network, obtain the minimum delay through the historical network delay, calculate the ratio of the actual delay to the minimum delay, and measure the minimum delay jitter based on the actual delay jitter obtained by the delay jitter measurement tool and the historical record; determine the network delay impact coefficient based on the ratio of the actual delay to the minimum delay and the product of the ratio of the actual delay jitter to the minimum delay jitter. The network delay impact coefficient satisfies the following relationship:

;

The packet loss rate impact coefficient is determined by the following formula:

;

in, is the actual delay, For minimum delay, is the actual delay jitter, is the minimum delay jitter; R is the packet loss rate, and 0≤R≤1.

In some embodiments of the present application, the data packets are sorted according to the transmission priority according to the throughput and the queuing delay of the data packets at the data sending end, including:

Obtain the throughput of the data receiving end and the data sending end and record them as the receiving throughput and the sending throughput respectively, and determine the actual throughput according to the relationship between the receiving throughput and the sending throughput;

Obtain the queuing delay of each data packet, and sort the corresponding data packets according to the queuing delay from small to large.

After the sorting is completed, the relationship between the actual throughput and the packet size is determined. When the actual throughput is greater than the packet size, the transmission priority of the packet is relegated one place.

It should be noted that the method for determining the network bandwidth impact coefficient is: obtain the actual bandwidth of the current network, usually measured using a network bandwidth test tool. Obtain the maximum bandwidth of the historical network bandwidth: obtain the maximum bandwidth of the network in the past records by analyzing the historical network data. Calculate the bandwidth impact coefficient: determine the network bandwidth impact coefficient based on the ratio of the actual bandwidth to the maximum bandwidth. The specific calculation method is as follows: The network bandwidth impact coefficient reflects the proportion of the current network bandwidth relative to the historical maximum bandwidth, helping to adjust the data transmission strategy to adapt to the actual situation of the network bandwidth.

Method for determining the network delay impact coefficient: Get the actual delay of the current network, usually measured using a network delay test tool. Get the minimum delay of historical network delay: Get the minimum delay in past records by analyzing historical network data. Measure actual delay jitter: Get the actual delay jitter, which reflects the instability of network delay. Get the minimum value of historical delay jitter: Get the minimum delay jitter in historical records by analyzing historical network data. This coefficient reflects the impact of network delay and delay fluctuation on data transmission.

Method for determining the packet loss rate impact coefficient: The packet loss rate usually refers to the ratio of data packets lost during transmission to the total number of sent data packets, ranging from 0 to 1. This coefficient is directly equal to the packet loss rate R, because the packet loss rate itself directly reflects the degree of data packet loss in the network.

Network bandwidth impact coefficient: The ratio of current bandwidth to maximum bandwidth reflects the utilization of network bandwidth. Network delay impact coefficient: The actual delay, minimum delay, actual delay jitter, and minimum delay jitter are combined to evaluate the impact of network delay and its fluctuation on data transmission. Packet loss rate impact coefficient: The packet loss rate is directly used to indicate the packet loss of the network.

In some embodiments of the present application, a queuing delay threshold is preset, and according to the magnitude relationship between the current queuing delay and the queuing delay threshold, it is determined whether to stop the transmission priority of the data packet and retreat one position;

When the queuing delay is greater than or equal to the queuing delay threshold, the transmission priority adjustment of the data packet is stopped; when the queuing delay is less than the queuing delay threshold, the transmission priority of the data packet continues to be relegated one place.

It should be noted that the actual throughput of the data receiving end is measured, that is, the amount of data that can be successfully received per unit time. This data is usually obtained through network monitoring tools. Get the throughput of the data sending end: measure the actual throughput of the data sending end, that is, the amount of data that can be successfully sent per unit time. This data is also obtained through network monitoring tools. The actual throughput depends on the lower throughput of the receiving end and the sending end, which reflects the bottleneck of data transmission.

Queuing delay sorting: Queuing delay refers to the time a data packet waits for transmission at the sender. This value can be recorded through monitoring tools or at the data sender. According to the queuing delay of each data packet, the data packets are sorted from small to large. This means that the data packets with smaller queuing delay will be processed first.

Adjust transmission priority: Perform preliminary priority sorting of data packets based on queuing delay, and send data packets with smaller queuing delay first. Determine the relationship between actual throughput and data packet size: Data packet size: the actual size of each data packet. Adjust priority: If the actual throughput is greater than the data packet size (that is, the throughput that the current network can handle is greater than the size of the data packet), the priority of the data packet is moved down one place. The reason for this is that when the network throughput is large, the data packet may be able to be transmitted quickly, so the impact on the queuing delay is small.

Example Process

Measuring Throughput

Receive throughput: 500 MB/s; Send throughput: 450 MB/s; Actual throughput: 450 MB/s (minimum)

Calculate the queuing delay of packets

Packet A: 2 ms; Packet B: 5 ms; Packet C: 1 ms

Preliminary sorting

The packet with the smallest queuing delay (packet C) takes priority. Packet A takes second place. Packet B takes last place.

Priority Adjustment

Assume that the packet size is smaller than the actual throughput (for example, the packet size is 100 KB and the actual throughput is 450MB/s). Because the actual throughput is much larger than the packet size, the priority of all packets does not need to be adjusted.

This prioritization method aims to achieve the following effects by comprehensively considering throughput and queuing delay: Optimize data transmission efficiency: Improve transmission efficiency by giving priority to sending packets with smaller queuing delay. Adapt to network conditions: Dynamically adjust the priority according to the actual throughput to adapt to data transmission requirements under different network conditions. Reduce transmission delay: Optimizing the sorting can reduce the transmission delay of data packets and improve the overall data transmission performance.

In some embodiments of the present application, the data packet size is used to verify whether loss or damage occurs during transmission, including:

The data receiving end compares the size of the received data packet with the record in the metadata. If the size of the received data packet is different from the record in the metadata, it is determined that the data packet is lost or damaged during transmission, and the data sending end resends the missing or damaged data packet.

It should be noted that the packet size verification process

Size record of received data packets: Each data packet is attached with metadata when it is sent. The metadata records the size, sequence number, checksum, etc. of the data packet. When receiving a data packet, the data receiving end will check the metadata in the data packet to confirm its correctness and integrity.

Comparison of received data packet sizes: At the data receiving end, the actual received data packet size is recorded. The actual received data packet size is compared with the size recorded in the metadata of the data packet.

Determine loss or damage: If the size of the actual received data packet is different from the record in the metadata, it is determined that the data packet may have been lost or damaged during transmission. Packet loss: If the size of the received data packet is 0 or very small, it may mean that the data packet is lost. Packet damage: If the actual size of the data packet is significantly different from the expected size, it may mean that the data packet has been damaged during transmission. Record missing or damaged situations: Record the detailed information of the missing or damaged data packet (such as the data packet sequence number, expected size, etc.) in the missing form.

To resend missing or corrupted packets:

Generate missing form: The data receiving end generates a missing form to record the sequence number and other relevant information of the data packets that need to be resent. Feedback to the data sending end: Send the missing form to the data sending end. Data sending end resends: The data sending end resends the missing or damaged data packets according to the information in the missing form.

This mechanism can effectively detect whether data is lost or damaged during transmission by comparing the actual size of the data packet with the expected size. By automatically generating a missing table and resending the data packet, the system can ensure the integrity and accuracy of data transmission, thereby improving the reliability of overall data transmission.

As shown in FIG2 , an embodiment of the present invention further provides a method for detecting multi-source data transmission in a government cloud, which is applied to the above-mentioned multi-source data transmission detection system in a government cloud. The method includes:

S1: Receive and store all data to be transmitted;

S2: Obtain the data to be transmitted in the database to be transmitted, segment the data to be transmitted into several data packets, and bundle a checksum for each data packet;

S3: sorting the data packets according to the throughput and the queuing delay of the data packets at the data sending end, sending the data packets and the confirmation form containing the data packets and the checksum bundled with the data packets according to the transmission priority; resending the data packets when the data packets are lost or damaged during the transmission process;

S4: After the data packet is successfully sent, the data receiving end is requested to send a missing form, and the missing data packet to be resent is determined according to the missing form; the missing form includes the name of the missing data packet and the corresponding check code;

S5: receiving the data packet and the checksum, confirming the missing data packet and generating a missing form;

S6: Send the missing form to the data sending end;

S7: Reassemble the received data packets into complete data in the original order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims (10)

1. A government cloud multisource data transmission detection system, the system comprising: a data transmitting end and a data receiving end;

The data transmitting terminal comprises:

a to-be-transmitted database configured to receive and store all data to be transmitted;

the segmentation module is configured to acquire the data to be transmitted in the data base to be transmitted, segment the data to be transmitted into a plurality of data packets, and bind a check code for each data packet;

The queue module is configured to sort transmission priorities of the data packets according to throughput and queuing delay of the data packets of the data transmitting end, and transmit the data packets and a confirmation form containing the data packets and the check codes bundled by the data packets according to the transmission priorities; the queue module is further configured to retransmit the missing data packet;

The verification module is configured to request to send a missing form to the data receiving end after the data packet is successfully sent, and determine the missing data packet to be retransmitted according to the missing form; the missing form comprises the name of the missing data packet and a corresponding check code;

The data receiving terminal comprises:

A verification module configured to receive the data packet and a check code, confirm the missing data packet, and generate the missing form;

the feedback module is configured to send the missing form to the data sending end;

And the reorganization module is configured to reorganize the received data packets into complete data according to an original sequence.

2. The government cloud multi-source data transmission detection system according to claim 1, wherein when the waiting database receives and stores all the data to be transmitted, the system comprises:

acquiring the received data to be transmitted, recording the receiving time, presetting a character number threshold of the data to be transmitted, and comparing the actual character number of the data to be transmitted with the character number threshold; when the actual character number is smaller than the character number threshold, not processing the data to be transmitted, and when the actual character number is larger than or equal to the character number threshold, adding a compression list into the data to be transmitted and sequentially judging whether to compress or not;

And acquiring CPU idle rate through the CPU utilization rate in real time of a computer, calculating the CPU occupancy rate of each data to be transmitted in the compression list, when the CPU idle rate is larger than the CPU occupancy rate of the data to be transmitted, reserving backup for the data to be transmitted and starting compression, and when the CPU idle rate is smaller than or equal to the CPU occupancy rate of the data to be transmitted, reversing the data to be transmitted in the compression list by one order.

3. The government cloud multi-source data transmission and detection system according to claim 2, wherein when the waiting database receives and stores all the data to be transmitted, the system further comprises:

Presetting a processing time threshold of a to-be-transmitted database, calculating the real-time processing time according to the difference value between the real-time and the receiving time, and judging whether the to-be-transmitted data is compressed or not when the real-time processing time is greater than or equal to the processing time threshold; and if the judgment result is yes, transmitting the data to be transmitted after the compression is completed to the segmentation module, and if the judgment result is no, terminating the compression and transmitting the backup to the segmentation module.

4. The government cloud multi-source data transmission detection system according to claim 2, wherein the segmenting the data to be transmitted into a plurality of data packets and binding check codes for each data packet comprises:

Calculating the number of data packets dividing the single data to be transmitted according to the network bandwidth, the network delay, the packet loss rate and the size of the single data to be transmitted, wherein the number of the data packets satisfies the following relation:

；

Wherein N is the number of data packets dividing single data to be transmitted, S is the size of single data to be transmitted, w _B is the network bandwidth influence coefficient, w _L is the network delay influence coefficient, w _R is the packet loss rate influence coefficient, Representing rounding up the bond;

After determining the number of data packets dividing the single data to be transmitted, segmenting the data to be transmitted; and generating a check code of the data packet based on an SHA-256 check algorithm after segmentation is completed.

5. The government cloud multi-source data transmission and detection system according to claim 4, wherein generating the check code of the data packet based on SHA-256 check algorithm comprises:

Reading the data content in the segmented data packet, inputting the data content into an SHA-256 verification algorithm, generating a hash value with a fixed length as the verification code, and attaching the generated verification code to a metadata part of the data packet, wherein the metadata part comprises a data packet sequence number, a data packet size and the verification code, the data packet sequence number is used for recombining sequence information of the data packet, and the data packet size is used for verifying whether the data packet is lost or damaged in the transmission process.

6. The government cloud multi-source data transmission and detection system according to claim 4, wherein the network bandwidth influence coefficient is determined by the following method:

Acquiring the actual bandwidth of the current network through measurement, acquiring the maximum bandwidth through historical network bandwidth, and calculating the ratio of the actual bandwidth to the maximum bandwidth to be used as the network bandwidth influence coefficient;

The network delay influence coefficient is determined by the following method:

Measuring the actual delay of a network, acquiring the minimum delay through the delay of a historical network, calculating the ratio of the actual delay to the minimum delay, and measuring the minimum delay jitter based on the actual delay jitter and the historical record acquired by a delay jitter measuring tool; determining a network delay influence coefficient according to the product of the ratio of the actual delay to the minimum delay and the ratio of the actual delay jitter to the minimum delay jitter, wherein the network delay influence coefficient meets the following relation:

；

The packet loss rate influence coefficient is determined by the following formula:

；

wherein, For the actual delay to be mentioned,In order for the minimum delay to be a function of the minimum delay,For the actual delay jitter to be mentioned,Is the minimum delay jitter; r is packet loss rate, and R is more than or equal to 0 and less than or equal to 1.

7. The government cloud multi-source data transmission detection system according to claim 5, wherein the sequencing of transmission priorities of the data packets according to throughput and queuing delay of the data packets at the data transmitting end comprises:

The throughput of the data receiving end and the data transmitting end is obtained and respectively recorded as receiving throughput and transmitting throughput, and the actual throughput is determined according to the magnitude relation between the receiving throughput and the transmitting throughput;

The queuing time delay of each data packet is obtained, and the corresponding data packets are sequenced according to the queuing time delay from small to large;

And after the sorting is finished, judging the relation between the actual throughput and the size of the data packet, and when the actual throughput is larger than the size of the data packet, reversing the transmission priority of the data packet by one ranking.

8. The government cloud multi-source data transmission detection system according to claim 7, wherein a queuing delay threshold is preset, and whether to stop the transmission priority of the data packet to back by one ranking is judged according to the size relation between the current queuing delay and the queuing delay threshold;

Stopping the transmission priority adjustment of the data packet when the queuing delay is greater than or equal to a queuing delay threshold; and when the queuing delay is smaller than the queuing delay threshold, continuing to back the transmission priority of the data packet by one ranking.

9. The government cloud multi-source data transmission and detection system of claim 8, wherein the data packet size is used to verify whether loss or damage occurs during transmission, comprising:

And the data receiving end compares the size of the received data packet with the record in the metadata, if the size of the received data packet is different from the record in the metadata, the data receiving end judges that the data packet is lost or damaged in the transmission process, and the data transmitting end resends the data packet with the loss or damage.

10. The government cloud multisource data transmission detection method applied to the government cloud multisource data transmission detection system according to any one of claims 1 to 9, wherein the method comprises the following steps:

S1: receiving and storing all data to be transmitted;

S2: acquiring the data to be transmitted in the data to be transmitted, segmenting the data to be transmitted into a plurality of data packets, and binding a check code for each data packet;

S3: sequencing the transmission priority of the data packets according to throughput and queuing delay of the data packets of the data transmitting end, and transmitting the data packets and a confirmation form containing the data packets and the verification codes bundled by the data packets according to the transmission priority; retransmitting the data packet when the data packet is lost or damaged in the transmission process;

s4: after the data packet is successfully transmitted, a missing form is required to be transmitted to the data receiving end, and the missing data packet to be retransmitted is determined according to the missing form; the missing form comprises the name of the missing data packet and a corresponding check code;

S5: receiving the data packet and the check code, confirming the missing data packet and generating the missing form;

S6: transmitting the missing form to the data transmitting end;

s7: and reorganizing the received data packets into complete data according to the original sequence.