CN104753815A - Flow cache based transparent transmission acceleration network and method - Google Patents
Flow cache based transparent transmission acceleration network and method Download PDFInfo
- Publication number
- CN104753815A CN104753815A CN201510131557.3A CN201510131557A CN104753815A CN 104753815 A CN104753815 A CN 104753815A CN 201510131557 A CN201510131557 A CN 201510131557A CN 104753815 A CN104753815 A CN 104753815A
- Authority
- CN
- China
- Prior art keywords
- data
- key value
- node server
- fragmentation
- disappearance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
本发明提供了一种基于流缓存的透明传输加速方法,该包括:数据入口节点服务器对接收客户端或者源站发送的数据流进行分片,得到多个数据分片;该数据入口节点服务器对每个该数据分片进行处理,得到能表示该数据分片的关键值;该数据入口节点服务器将所有关键值发送给数据出口节点服务器;该数据出口节点服务器根据该些关键值还原成对应的数据分片;该数据出口节点服务器将该多个数据分片拼接成完整的数据发送给源站或者客户端。
The present invention provides a transparent transmission acceleration method based on stream cache, which includes: the data entry node server fragments the data flow sent by the receiving client or the source station to obtain multiple data fragments; the data entry node server Each of the data fragments is processed to obtain key values that can represent the data fragments; the data entry node server sends all key values to the data exit node server; the data exit node server restores the corresponding key values according to these key values Data sharding: the data export node server splices the multiple data shards into complete data and sends it to the source station or client.
Description
技术领域technical field
本发明涉及网络数据透明传输加速技术,尤其涉及各种应用协议的TCP数据传输的加速技术。The invention relates to an acceleration technology for transparent transmission of network data, in particular to an acceleration technology for TCP data transmission of various application protocols.
背景技术Background technique
网络数据透明传输技术是指在传输过程中,传输网络对外界透明,不管传输的业务如何,透明传输网络只负责将需要传送的业务传送到目的节点,同时保证传输的质量即可,而不对传输的业务进行处理。Network data transparent transmission technology means that during the transmission process, the transmission network is transparent to the outside world. Regardless of the business to be transmitted, the transparent transmission network is only responsible for transmitting the business that needs to be transmitted to the destination node, while ensuring the quality of the transmission. business is processed.
因此,现有的透明传输技术存在以下几个主要弊端:Therefore, the existing transparent transmission technology has the following major drawbacks:
一、当客户端多次上传或者下载同一个文件时,由于透明传输网络的关系,应用加速网络不解析应用层协议,只进行透明传输,因此,并不知道客户请求的数据是什么,所以每次都需要传输完整的数据,使得传输时间长。图1示出了现有技术的透明传输网络框架。从图中可以看出,客户端与边缘节点之间、各节点之间、或节点与源站服务器之间传输的都是完整的数据。1. When the client uploads or downloads the same file multiple times, due to the relationship of the transparent transmission network, the application acceleration network does not analyze the application layer protocol, but only performs transparent transmission. Therefore, it does not know what data the client requests, so each Complete data needs to be transmitted every time, making the transmission time long. Fig. 1 shows a transparent transmission network framework in the prior art. It can be seen from the figure that all data is transmitted between the client and the edge node, between nodes, or between the node and the origin server.
二、即使是HTTP缓存,但如果客户端请求的是不同URL的文件,而这些文件虽然URL不同,但实质只有一小部分的数据变化,存在很多重复数据,那么缓存依然无法命中。也就是说,尽管存在大量的数据重复,但由于不同的URL,使得客户端仍然无法从缓存中获取数据,必须要回源站去取数据,并传输完整的数据。2. Even if it is an HTTP cache, if the client requests files with different URLs, and although the URLs of these files are different, only a small part of the data changes in essence, and there are many duplicate data, then the cache still cannot be hit. That is to say, although there is a large amount of data duplication, the client still cannot obtain data from the cache due to different URLs, and must go back to the origin site to retrieve the data and transmit the complete data.
三、现有的流缓存应用,一般应用于双边加速,例如,通过增加客户端网关和源端网关来进行加速。对于这种双边加速,需要客户端和源端两端预先同步数据流仓库。但对于应用加速网络而言,由于存在多对多的网络环境,预先同步数据流仓库则可能浪费空间、耗费资源。另外新增的数据流缓存必须等待同步之后,才可以命中,而且必须管理哪一台缓存了哪些数据,管理复杂度高。3. Existing stream caching applications are generally applied to bilateral acceleration, for example, by adding client gateways and source gateways for acceleration. For this kind of bilateral acceleration, the client and the source end need to synchronize the data flow warehouse in advance. But for the application acceleration network, due to the many-to-many network environment, pre-synchronizing the data flow warehouse may waste space and consume resources. In addition, the newly added data stream cache must wait for synchronization before it can be hit, and it is necessary to manage which data is cached by which device, and the management complexity is high.
因此,亟需一种能解决现有的网络数据透明传输技术中的上述弊端的透明加速方法。Therefore, there is an urgent need for a transparent acceleration method that can solve the above drawbacks in the existing network data transparent transmission technology.
发明内容Contents of the invention
为了减少透明传输网络中的数据传输时间、降低传输过程的数据量、节省网络带宽,本发明提供了一种基于流缓存的透明传输加速网络以及方法。In order to reduce the data transmission time in the transparent transmission network, reduce the amount of data in the transmission process, and save network bandwidth, the present invention provides a transparent transmission acceleration network and method based on stream cache.
本发明提供了一种基于流缓存的透明传输加速方法,该方法包括以下步骤:The present invention provides a method for accelerating transparent transmission based on stream cache, the method comprising the following steps:
数据入口节点服务器对接收客户端或者源站发送的数据流进行分片,得到多个数据分片;The data entry node server fragments the data stream sent by the receiving client or the source station to obtain multiple data fragments;
该数据入口节点服务器对每个该数据分片进行处理,得到能表示该数据分片的关键值;The data entry node server processes each data fragment to obtain a key value that can represent the data fragment;
该数据入口节点服务器将所有关键值发送给数据出口节点服务器;The data entry node server sends all key values to the data exit node server;
该数据出口节点服务器根据该些关键值还原成对应的数据分片;The data export node server restores the corresponding data fragments according to these key values;
该数据出口节点服务器将该多个数据分片拼接成完整的数据发送给源站或者客户端。The data egress node server splices the multiple data fragments into complete data and sends it to the source station or the client.
在一个实施例中,该数据出口节点服务器将该多个数据分片拼接成完整的数据发送给终端的步骤还包括:In one embodiment, the step of the data egress node server splicing the plurality of data fragments into complete data and sending it to the terminal further includes:
该数据出口节点服务器判断本地是否缓存有与该关键值对应的数据分片;The data export node server judges whether there is a data fragment corresponding to the key value in the local cache;
若该数据出口节点服务器判断出本地不缓存有与该关键值对应的数据分片,则向该数据入口节点返回表示缺失该关键值的信息,所述数据入口节点服务器在接收到该表示缺失关键值的信息后向所述数据出口节点服务器发送该缺失的关键值和对该缺失的关键值所对应的数据分片;所述数据出口节点服务器利用该缺失的关键值所对应的数据分片以拼接完整的数据发送给终端,并同时缓存该缺失的关键值所对应的数据分片;If the data egress node server determines that there is no data fragment corresponding to the key value in the local cache, it returns information indicating that the key value is missing to the data ingress node, and the data ingress node server receives the information indicating that the key is missing Value information to the data export node server to send the missing key value and the data fragmentation corresponding to the missing key value; the data export node server uses the data fragmentation corresponding to the missing key value to Stitch the complete data and send it to the terminal, and at the same time cache the data fragment corresponding to the missing key value;
若该数据出口节点服务器判断出本地缓存有与该关键值对应的数据分片,则将该多个数据分片拼接成完整的数据发送给终端。If the data egress node server determines that there is a data fragment corresponding to the key value in the local cache, then the multiple data fragments are spliced into complete data and sent to the terminal.
在一个实施例中,该数据入口节点服务器对每个该数据分片进行处理,得到能表示该数据分片的关键值的步骤包括:In one embodiment, the data entry node server processes each data fragment, and the step of obtaining a key value that can represent the data fragment includes:
该数据入口节点服务器对每个该数据分片进行编码。The data entry node server encodes each of the data fragments.
在一个实施例中,每个数据分片可以是不定长的。In one embodiment, each data segment may be of variable length.
在一个实施例中,在该数据出口节点服务器根据该些关键值还原成对应的数据分片的步骤之前,还包括以下步骤:In one embodiment, before the data egress node server restores the corresponding data fragments according to the key values, the following steps are further included:
该数据出口节点服务器判断所接收到的仅仅是该关键值,还是该缺失的关键值和该缺失的关键值所对应的数据分片。The data egress node server determines whether the received key value is only the key value, or the missing key value and the data fragment corresponding to the missing key value.
本发明还提供了一种基于流缓存的透明传输加速网络,包括数据入口节点服务器和数据出口节点服务器,其中,The present invention also provides a transparent transmission acceleration network based on flow cache, including a data entry node server and a data exit node server, wherein,
该数据入口节点服务器,被配置成对接收客户端或者源站发送的数据流进行分片,得到多个数据分片;对每个该数据分片进行处理,得到能表示该数据分片的关键值;将所有关键值发送给数据出口节点服务器;The data entry node server is configured to fragment the data stream sent by the receiving client or the source station to obtain multiple data fragments; process each data fragment to obtain the key that can represent the data fragment value; send all key values to the data export node server;
该数据出口节点服务器,被配置成根据该些关键值还原成对应的数据分片;将该多个数据分片拼接成完整的数据发送给源站或者客户端。The data export node server is configured to restore corresponding data fragments according to the key values; splice the multiple data fragments into complete data and send it to the source station or the client.
在一个实施例中,该数据入口节点服务器包括:In one embodiment, the data entry node server includes:
数据接收模块,被配置成接收客户端或者源站发送的数据流;The data receiving module is configured to receive the data stream sent by the client or the source station;
数据处理模块,被配置成对接收到的数据流按照一定的算法进行分片,得到多个数据分片,并对每个数据分片进行编码以得到能表示该数据分片的关键值;The data processing module is configured to fragment the received data stream according to a certain algorithm to obtain multiple data fragments, and encode each data fragment to obtain a key value that can represent the data fragment;
数据缓存模块,被配置成缓存该数据分片;a data caching module configured to cache the data fragment;
关键值发送模块,被配置成发送所有关键值给该数据出口节点服务器。The key value sending module is configured to send all key values to the data export node server.
缺失关键值接收模块,被配置成判断是否收到该数据出口节点服务器返回的表示缺失该关键值的信息;The missing key value receiving module is configured to determine whether to receive the information indicating that the key value is missing returned by the data export node server;
关键值和对应的数据分片发送模块,被配置成,如果收到该数据出口节点服务器返回的表示缺失该关键值的信息,则把该缺失的关键值和该缺失的关键值所对应的数据分片发送给该数据出口节点服务器。The key value and the corresponding data fragment sending module is configured to, if the information indicating that the key value is missing is received from the data export node server, then the missing key value and the data corresponding to the missing key value The fragments are sent to the data egress node server.
在一个实施例中,该数据出口节点服务器包括:In one embodiment, the data egress node server includes:
数据接收模块,被配置成接收该数据入口节点发送的该关键值,和/或该缺失的关键值与该缺失的关键值所对应的数据分片;A data receiving module configured to receive the key value sent by the data entry node, and/or the missing key value and the data fragment corresponding to the missing key value;
数据区分模块,被配置成区分所接收到的是该关键值,还是该缺失的关键值与该缺失的关键值所对应的数据分片;如果该数据区分模块判断出该接收到的仅仅是该关键值,则将该关键值发送至数据解析模块;如果该数据区分模块判断出所接收到的是该缺失的关键值与该缺失的关键值所对应的数据分片,则将该缺失的关键值所对应的数据分片发送给数据拼接模块,同时,将该缺失的关键值所对应的数据分片缓存至数据缓存模块,以便下次如果传输相同数据则可以不必再由该数据入口节点服务器重新发送数据,而是直接从本地缓存中获取即可;The data distinguishing module is configured to distinguish whether the received key value is the missing key value and the data fragment corresponding to the missing key value; if the data distinguishing module judges that the received key value is only the key value, then send the key value to the data parsing module; if the data distinguishing module judges that what is received is the missing key value and the data fragment corresponding to the missing key value, then the missing key value The corresponding data fragmentation is sent to the data splicing module, and at the same time, the data fragmentation corresponding to the missing key value is cached to the data caching module, so that if the same data is transmitted next time, it is not necessary to be regenerated by the data entry node server. Send data, but get it directly from the local cache;
数据解析和缓存查找模块,被配置成解析该关键值并通过该关键值从该数据缓存模块中查找本地是否缓存有与该关键值对应的数据分片;The data parsing and cache lookup module is configured to parse the key value and use the key value to find out from the data cache module whether there is a data fragment corresponding to the key value cached locally;
数据拼接模块,被配置成,如果本地查找到有该关键值对应的数据分片的缓存,则利用该些缓存的数据分片拼接成完整的数据发送给终端;如果该数据区分模块判断出所接收到的是该缺失的关键值与该缺失的关键值所对应的数据分片,则该数据拼接模块还被配置成利用该缺失的关键值所对应的数据分片拼接完整的数据发送给终端;The data splicing module is configured to, if a cache of data fragments corresponding to the key value is found locally, then use the cached data fragments to concatenate complete data and send it to the terminal; if the data distinguishing module judges that the received If the missing key value and the data fragment corresponding to the missing key value are received, the data splicing module is further configured to use the data fragment corresponding to the missing key value to stitch complete data and send it to the terminal;
缺失关键值发送模块,被配置成,如果该数据解析和缓存查找模块未在本地找到该数据分片的缓存,则发送表示缺失该关键值的信息给该数据入口节点服务器。The missing key value sending module is configured to send information indicating that the key value is missing to the data entry node server if the data parsing and cache lookup module does not find the cache of the data fragment locally.
本发明还提供了一种基于流缓存的透明传输加速网络中的数据入口节点服务器,该数据入口节点服务器被配置成对接收客户端或者源站发送的数据流进行分片,得到多个数据分片;对每个该数据分片进行处理,得到能表示该数据分片的关键值;将所有关键值发送给数据出口节点服务器。The present invention also provides a data entry node server in a flow cache-based transparent transmission acceleration network. The data entry node server is configured to fragment the data flow sent by the receiving client or the source station to obtain multiple data fragments. slice; each data slice is processed to obtain a key value that can represent the data slice; and all key values are sent to the data export node server.
在一个实施例中,该数据入口节点服务器包括:In one embodiment, the data entry node server includes:
数据接收模块,被配置成接收客户端或者源站发送的数据流;The data receiving module is configured to receive the data stream sent by the client or the source station;
数据处理模块,被配置成对接收到的数据流按照一定的算法进行分片,得到多个数据分片,并对每个数据分片进行编码以得到能表示该数据分片的关键值;The data processing module is configured to fragment the received data stream according to a certain algorithm to obtain multiple data fragments, and encode each data fragment to obtain a key value that can represent the data fragment;
数据缓存模块,被配置成缓存该数据分片;a data caching module configured to cache the data fragment;
关键值发送模块,被配置成发送所有关键值给该数据出口节点服务器。The key value sending module is configured to send all key values to the data export node server.
缺失关键值接收模块,被配置成判断是否收到该数据出口节点服务器返回的表示缺失该关键值的信息;The missing key value receiving module is configured to determine whether to receive the information indicating that the key value is missing returned by the data export node server;
关键值和对应的数据分片发送模块,被配置成,如果收到该数据出口节点服务器返回的表示缺失该关键值的信息,则把该缺失的关键值和该缺失的关键值所对应的数据分片发送给该数据出口节点服务器。The key value and the corresponding data fragment sending module is configured to, if the information indicating that the key value is missing is received from the data export node server, then the missing key value and the data corresponding to the missing key value The fragments are sent to the data egress node server.
本发明还提供了一种基于流缓存的透明传输加速网络中的数据出口节点服务器,该数据出口节点服务器被配置成接收上述数据入口节点服务器发送的关键值,并根据该些关键值还原成对应的数据分片;将该多个数据分片拼接成完整的数据发送给源站或者客户端。The present invention also provides a data egress node server in a flow cache-based transparent transmission acceleration network, the data egress node server is configured to receive the key values sent by the above data ingress node server, and restore the corresponding key values according to these key values The data fragmentation; the multiple data fragments are spliced into complete data and sent to the source station or client.
在一个实施例中,该数据出口节点服务器包括:In one embodiment, the data egress node server includes:
1502481CNCN1502481CNCN
数据接收模块,被配置成接收该数据入口节点发送的该关键值,和/或该缺失的关键值与该缺失的关键值所对应的数据分片;A data receiving module configured to receive the key value sent by the data entry node, and/or the missing key value and the data fragment corresponding to the missing key value;
数据区分模块,被配置成区分所接收到的是该关键值,还是该缺失的关键值与该缺失的关键值所对应的数据分片;如果该数据区分模块判断出该接收到的仅仅是该关键值,则将该关键值发送至数据解析模块;如果该数据区分模块判断出所接收到的是该缺失的关键值与该缺失的关键值所对应的数据分片,则将该缺失的关键值所对应的数据分片发送给数据拼接模块,同时,将该缺失的关键值所对应的数据分片缓存至数据缓存模块,以便下次如果传输相同数据则可以不必再由该数据入口节点服务器重新发送数据,而是直接从本地缓存中获取即可;The data distinguishing module is configured to distinguish whether the received key value is the missing key value and the data fragment corresponding to the missing key value; if the data distinguishing module judges that the received key value is only the key value, then send the key value to the data parsing module; if the data distinguishing module judges that what is received is the missing key value and the data fragment corresponding to the missing key value, then the missing key value The corresponding data fragmentation is sent to the data splicing module, and at the same time, the data fragmentation corresponding to the missing key value is cached to the data caching module, so that if the same data is transmitted next time, it is not necessary to be regenerated by the data entry node server. Send data, but get it directly from the local cache;
数据解析和缓存查找模块,被配置成解析该关键值并通过该关键值从该数据缓存模块中查找本地是否缓存有与该关键值对应的数据分片;The data parsing and cache lookup module is configured to parse the key value and use the key value to find out from the data cache module whether there is a data fragment corresponding to the key value cached locally;
数据拼接模块,被配置成,如果本地查找到有该关键值对应的数据分片的缓存,则利用该些缓存的数据分片拼接成完整的数据发送给终端;如果该数据区分模块判断出所接收到的是该缺失的关键值与该缺失的关键值所对应的数据分片,则该数据拼接模块还被配置成利用该缺失的关键值所对应的数据分片拼接完整的数据发送给终端;The data splicing module is configured to, if a cache of data fragments corresponding to the key value is found locally, then use the cached data fragments to concatenate complete data and send it to the terminal; if the data distinguishing module judges that the received If the missing key value and the data fragment corresponding to the missing key value are received, the data splicing module is further configured to use the data fragment corresponding to the missing key value to stitch complete data and send it to the terminal;
缺失关键值发送模块,被配置成,如果该数据解析和缓存查找模块未在本地找到该数据分片的缓存,则发送表示缺失该关键值的信息给该数据入口节点服务器。The missing key value sending module is configured to send information indicating that the key value is missing to the data entry node server if the data parsing and cache lookup module does not find the cache of the data fragment locally.
附图说明Description of drawings
本发明的以上发明内容以及下面的具体实施方式在结合附图阅读时会得到更好的理解。需要说明的是,附图仅作为所请求保护的发明的示例。在附图中,相同的附图标记代表相同或类似的元素。The above summary of the invention and the following specific implementation methods of the present invention will be better understood when read in conjunction with the accompanying drawings. It should be noted that the drawings are merely examples of the claimed invention. In the drawings, the same reference numerals represent the same or similar elements.
图1示出现有技术的透明传输网络框架;Fig. 1 shows the transparent transmission network framework of prior art;
图2示出根据本发明的一实施例的总体示意图;Fig. 2 shows an overall schematic diagram according to an embodiment of the present invention;
图3示出根据本发明的一实施例的总体流程图;Fig. 3 shows an overall flowchart according to an embodiment of the present invention;
图4示出根据本发明的一实施例的数据入口节点服务器的处理流程;Fig. 4 shows the processing flow of the data entry node server according to an embodiment of the present invention;
图5示出根据本发明的一实施例的数据出口节点服务器的处理流程;Fig. 5 shows the processing flow of the data egress node server according to an embodiment of the present invention;
图6示出根据本发明的一实施例的数据入口节点服务器的构架框图;以及FIG. 6 shows a block diagram of a data entry node server according to an embodiment of the present invention; and
图7示出根据本发明的一实施例的数据出口节点服务器的构架框图。Fig. 7 shows a block diagram of a data egress node server according to an embodiment of the present invention.
具体实施方式Detailed ways
以下在具体实施方式中详细叙述本发明的详细特征以及优点,其内容足以使任何本领域技术人员了解本发明的技术内容并据以实施,且根据本说明书所揭露的说明书、权利要求及附图,本领域技术人员可轻易地理解本发明相关的目的及优点。The detailed features and advantages of the present invention are described in detail below in the specific embodiments, the content of which is sufficient to enable any person skilled in the art to understand the technical content of the present invention and implement it accordingly, and according to the specification, claims and drawings disclosed in this specification , those skilled in the art can easily understand the related objects and advantages of the present invention.
本发明提供了一种基于流缓存的透明传输加速方法。其基本的思想是,通过应用加速网络内部对传输业务的处理,达到节省带宽的效果,同时对客户数据的传输起到加速效果。当客户数据经过应用加速网络时,利用应用加速网络多级架构的特点,在数据经过的第一个应用加速节点上对数据进行编码传输,最后一个应用加速节点上进行解码还原原始数据,而不影响客户实际得到的数据。如果客户允许将最后一个应用加速节点架设在客户的网内,则可以节省客户的出口带宽。如果没架设在客户内网,则通过降低中间节点传输的数据量,可以达到加快数据传输速度的效果。The invention provides a method for accelerating transparent transmission based on flow cache. Its basic idea is to achieve the effect of saving bandwidth by accelerating the processing of transmission services inside the network, and at the same time accelerate the transmission of customer data. When customer data passes through the application acceleration network, using the characteristics of the multi-level architecture of the application acceleration network, the data is encoded and transmitted on the first application acceleration node that the data passes through, and the original data is decoded and restored on the last application acceleration node. Affect the data that customers actually get. If the customer allows the last application acceleration node to be set up in the customer's network, the customer's egress bandwidth can be saved. If it is not set up on the customer's intranet, the effect of speeding up the data transmission speed can be achieved by reducing the amount of data transmitted by the intermediate nodes.
图2示出根据本发明的一实施例的总体示意图。如图2所示,除了客户端与边缘节点之间、离源站最近的节点与源站服务器之间传输的是完整数据外,边缘节点与离源站最近的节点之间的应用加速网络所传输的数据是经数据处理后的关键数据。在本发明中,所谓关键数据指的是经过一定算法获得的能代表该数据的表示值(即,Key值)。在一个实施例中,关键数据可以是经编码或压缩后的数据。Fig. 2 shows an overall schematic diagram according to an embodiment of the present invention. As shown in Figure 2, except that the complete data is transmitted between the client and the edge node, and between the node closest to the source station and the server of the source station, the application acceleration network between the edge node and the node closest to the source station The transmitted data is the key data after data processing. In the present invention, the so-called key data refers to a representation value (ie, Key value) obtained through a certain algorithm that can represent the data. In one embodiment, key data may be encoded or compressed data.
透明加速网络对数据的处理方式有以下两种:The transparent acceleration network processes data in the following two ways:
一、客户端向边缘节点上传数据,边缘把数据经过应用加速网络传输到源站。1. The client uploads data to the edge node, and the edge transmits the data to the source station through the application acceleration network.
二、源站发送响应数据给离源最近的应用加速节点,该节点把数据按原路返回给客户端。2. The source station sends the response data to the application acceleration node closest to the source, and the node returns the data to the client through the original route.
因此,对于上传数据而言,边缘节点对应为数据入口节点服务器,离源站最近的节点对应为数据出口节点服务器。对于下载数据而言,边缘节点对应为数据出口节点服务器,离源站最近的节点对应为数据出口节点服务器。Therefore, for uploading data, the edge node corresponds to the data entry node server, and the node closest to the source station corresponds to the data exit node server. For downloading data, the edge node corresponds to the data export node server, and the node closest to the source station corresponds to the data export node server.
图3示出根据本发明的一实施例的总体流程图。该流程图包括以下几个步骤。Fig. 3 shows an overall flow chart according to an embodiment of the present invention. The flowchart includes the following steps.
步骤S301,数据入口节点服务器对接收客户端或者源站发送的数据流进行分片,得到多个数据分片。In step S301, the data entry node server fragments the data flow sent by the receiving client or the source station to obtain multiple data fragments.
步骤S302,该数据入口节点服务器对每个该数据分片进行处理,得到能表示该数据分片的关键值。在一个实施例中,该数据入口节点服务器根据一定的算法对该数据分片进行编码压缩处理。Step S302, the data entry node server processes each data fragment to obtain a key value that can represent the data fragment. In one embodiment, the data entry node server encodes and compresses the data fragments according to a certain algorithm.
步骤S303,该数据入口节点服务器将所有关键值发送给数据出口节点服务器。Step S303, the data entry node server sends all key values to the data exit node server.
步骤S304,该数据出口节点服务器根据该些关键值还原成对应的数据分片。在一个实施例中,该数据出口节点服务器根据一定的算法对该数据分片进行解码处理。Step S304, the data egress node server restores the corresponding data fragments according to the key values. In one embodiment, the data egress node server decodes the data fragments according to a certain algorithm.
步骤S305,该数据出口节点服务器将该多个数据分片拼接成完整的数据发送给源站或者客户端。In step S305, the data export node server splices the multiple data fragments into complete data and sends it to the source station or the client.
该数据出口节点服务器将该多个数据分片拼接成完整的数据发送给终端的步骤还包括以下几个子步骤。The step of the data egress node server splicing the plurality of data fragments into complete data and sending it to the terminal also includes the following several sub-steps.
该数据出口节点服务器判断本地是否缓存有与该关键值对应的数据分片。The data egress node server judges whether there is a data fragment corresponding to the key value cached locally.
若该数据出口节点服务器判断出本地不缓存有与该关键值对应的数据分片,则向该数据入口节点返回表示缺失该关键值的信息,所述数据入口节点服务器在接收到该表示缺失关键值的信息后向所述数据出口节点服务器发送该缺失的关键值和对该缺失的关键值所对应的数据分片;所述数据出口节点服务器利用该缺失的关键值所对应的数据分片以拼接完整的数据发送给终端,并同时缓存该缺失的关键值所对应的数据分片。If the data egress node server determines that there is no data fragment corresponding to the key value in the local cache, it returns information indicating that the key value is missing to the data ingress node, and the data ingress node server receives the information indicating that the key is missing Value information to the data export node server to send the missing key value and the data fragmentation corresponding to the missing key value; the data export node server uses the data fragmentation corresponding to the missing key value to The spliced complete data is sent to the terminal, and the data fragment corresponding to the missing key value is cached at the same time.
若该数据出口节点服务器判断出本地缓存有与该关键值对应的数据分片,则将该多个数据分片拼接成完整的数据发送给终端。If the data egress node server determines that there is a data fragment corresponding to the key value in the local cache, then the multiple data fragments are spliced into complete data and sent to the terminal.
图4示出根据本发明的一实施例的数据入口节点服务器的处理流程。Fig. 4 shows the processing flow of the data entry node server according to an embodiment of the present invention.
步骤S401,数据入口节点接收客户端或者源站发送的数据流。Step S401, the data entry node receives the data flow sent by the client or the source station.
步骤S402,对接受到的数据流按照一定的算法进行分片,得到若干数据分片,并缓存数据分片。在一个实施例中,该数据分片是不定长的。Step S402, fragment the received data stream according to a certain algorithm to obtain several data fragments, and cache the data fragments. In one embodiment, the data segments are of variable length.
步骤S403,对每个数据分片进行处理,得到能表示该数据分片的关键值(Key值)。该关键值指的是经过一定算法获得的能代表该数据分片的关键数据(Key值)。In step S403, each data fragment is processed to obtain a key value (Key value) that can represent the data fragment. The key value refers to the key data (Key value) obtained through a certain algorithm that can represent the data slice.
步骤S404,发送所有key值给数据出口节点。Step S404, sending all key values to the data egress node.
步骤S405,判断是否收到数据出口节点返回的表示缺失该key值的信息。Step S405, judging whether the information indicating that the key value is missing is received from the data export node.
步骤S406,如果收到数据出口节点返回的表示缺失该key值的信息,则把该缺失的key值和该缺失的Key值所对应的数据分片发送给数据出口节点。具体而言,如果数据出口节点发现本地并不存有该Key值,则表示数据出口节点中并不缓存有与该Key值对应的数据,因此,会发送表示缺失Key值的信息给数据入口节点,需要数据入口节点将数据发送给数据出口节点,并且为了下一次不必重复传输该数据,数据入口节点需要将该数据对应的Key值发送给数据出口节点,以便下一次请求数据时可直接从数据输出节点处获得。Step S406, if the information indicating that the key value is missing is received from the data export node, then the missing key value and the data fragment corresponding to the missing Key value are sent to the data export node. Specifically, if the data export node finds that the Key value does not exist locally, it means that the data corresponding to the Key value is not cached in the data export node, so it will send information indicating the missing Key value to the data entry node , the data entry node needs to send the data to the data exit node, and in order not to repeat the transmission of the data next time, the data entry node needs to send the Key value corresponding to the data to the data exit node, so that the next request for data can be directly from the data Obtained at the output node.
步骤S407,如果没有收到数据出口节点返回的表示缺失的key值的信息,则结束此次数据传输。Step S407, if the information indicating the missing key value returned by the data export node is not received, the data transmission is ended.
图5示出根据本发明的一实施例的数据出口节点服务器的处理流程。Fig. 5 shows the processing flow of the data egress node server according to an embodiment of the present invention.
步骤S501,数据出口节点接收数据入口节点发送的key值,或接收缺失的key值与该缺失的key值对应的数据分片的组合。Step S501, the data egress node receives the key value sent by the data ingress node, or receives the combination of the missing key value and the data fragment corresponding to the missing key value.
步骤S502,判断所接收到的是key值还是缺失的key值与该缺失的key值对应的数据分片的组合。Step S502, judging whether the received key value or the combination of the missing key value and the data fragment corresponding to the missing key value.
步骤S503,如果判断出仅仅是key值,则解析该key值并通过该key值查找数据出口节点本地是否缓存有与该key值对应的数据分片。Step S503, if it is determined that it is only the key value, then analyze the key value and use the key value to find out whether the local cache of the data export node has a data fragment corresponding to the key value.
步骤S504,如果本地有数据分片的缓存,则将若干数据分片拼接成完整的数据发送给终端。Step S504, if there is a cache of data fragments locally, splicing several data fragments into complete data and sending it to the terminal.
步骤S505,如果本地没有该数据分片的缓存,则发送表示缺失该key值的信息给数据入口节点。Step S505, if there is no cache of the data fragment locally, send information indicating that the key value is missing to the data entry node.
步骤S506,若步骤S502中判断出所接收到的是缺失的key值与该缺失的key值对应的数据分片的组合,则利用该缺失的key值对应的数据分片拼接完整数据给终端,同时将该缺失的key值对应的数据分片保存在缓存区,以便下次如果传输相同数据则可以不必再由数据入口节点重新发送原始数据,而是直接从本地缓存中获取即可。Step S506, if it is determined in step S502 that the received combination of the missing key value and the data fragment corresponding to the missing key value is used, the data fragment corresponding to the missing key value is used to splice the complete data to the terminal, and at the same time The data fragment corresponding to the missing key value is saved in the cache area, so that if the same data is transmitted next time, the original data does not need to be resent by the data entry node, but can be obtained directly from the local cache.
图6示出根据本发明的一个实施例的数据入口节点服务器的构架图。Fig. 6 shows a structure diagram of a data entry node server according to an embodiment of the present invention.
如图6所述,本发明的数据入口节点服务器包括数据接收模块601、数据处理模块602、数据缓存模块603、关键值发送模块604、缺失关键值接收模块605、以及关键值和对应的数据分片发送模块606。As shown in Figure 6, the data entry node server of the present invention includes a data receiving module 601, a data processing module 602, a data caching module 603, a key value sending module 604, a missing key value receiving module 605, and key values and corresponding data points Slice sending module 606.
该数据接收模块601被配置成接收客户端或者源站发送的数据。The data receiving module 601 is configured to receive data sent by the client or the source station.
该数据处理模块602被配置成对接收到的数据流按照一定的算法进行分片,得到若干数据分片,并对每个数据分片进行编码以得到能表示该数据分片的关键值。The data processing module 602 is configured to segment the received data stream according to a certain algorithm to obtain several data segments, and encode each data segment to obtain a key value that can represent the data segment.
进一步,该数据处理模块602包括数据分片模块和数据编码模块。其中,该数据分片模块被配置成对接受到的数据流按照一定的算法进行分片,得到若干数据分片。在一个实施例中,该数据分片是不定长的。该数据编码模块被配置成对每个数据分片进行处理,得到能表示该数据分片的关键值(Key值)。该关键值指的是经过一定算法获得的能代表该数据分片的关键数据(Key值)。Further, the data processing module 602 includes a data fragmentation module and a data encoding module. Wherein, the data fragmentation module is configured to fragment the received data stream according to a certain algorithm to obtain several data fragments. In one embodiment, the data segments are of variable length. The data encoding module is configured to process each data fragment to obtain a key value (Key value) that can represent the data fragment. The key value refers to the key data (Key value) obtained through a certain algorithm that can represent the data slice.
该数据缓存模块603被配置成对缓存该数据分片。The data cache module 603 is configured to cache the data slices.
该关键值发送模块604被配置成发送所有key值给数据出口节点。The key value sending module 604 is configured to send all key values to data egress nodes.
该缺失关键值接收模块605被配置成判断是否收到数据出口节点返回的表示缺失该key值的信息。The missing key value receiving module 605 is configured to determine whether the information indicating that the key value is missing is received from the data export node.
该关键值和对应的数据分片发送模块606被配置成,如果收到数据出口节点返回的表示缺失该key值的信息,则把该缺失的key值和该缺失的Key值所对应的数据分片发送给数据出口节点。The key value and the corresponding data fragment sending module 606 is configured to, if the information indicating that the key value is missing is received from the data export node, then the missing key value and the data corresponding to the missing Key value are divided into The slice is sent to the data egress node.
图7示出根据本发明的一个实施例的数据出口节点服务器的构架图。Fig. 7 shows a structure diagram of a data egress node server according to an embodiment of the present invention.
如图7所示,本发明的数据出口节点服务器包括数据接收模块701、数据区分模块702、数据解析模块703、数据拼接模块704、缺失关键值发送模块705、以及数据缓存模块706。As shown in FIG. 7 , the data export node server of the present invention includes a data receiving module 701 , a data distinguishing module 702 , a data parsing module 703 , a data splicing module 704 , a missing key value sending module 705 , and a data caching module 706 .
该数据接收模块701被配置成接收数据入口节点发送的key值,或缺失的key值与该缺失的key值所对应的数据分片的组合。The data receiving module 701 is configured to receive the key value sent by the data entry node, or the combination of the missing key value and the data fragment corresponding to the missing key value.
该数据区分模块702被配置成,区分所接收到的是key值还是缺失的key值与该缺失的key值所对应的数据分片的组合。如果数据区分模块702判断出该数据接收模块701所接收到的仅仅是key值,则将该key值发送至该数据解析模块703。如果数据区分模块702判断出数据接收模块701所接收到的是缺失的key值与该缺失的key值所对应的数据分片的组合,则将该缺失的key值所对应的数据分片发送给该数据拼接模块704。同时,还将该缺失的key值所对应的数据分片缓存至数据缓存模块706,以便下次如果传输相同数据则可以不必再由数据入口节点重新发送原始数据,而是直接从本地缓存中获取即可。The data distinguishing module 702 is configured to distinguish whether the received key value is the combination of the missing key value and the data fragment corresponding to the missing key value. If the data distinguishing module 702 determines that what the data receiving module 701 receives is only the key value, it sends the key value to the data parsing module 703 . If the data distinguishing module 702 judges that what the data receiving module 701 receives is a combination of the missing key value and the data fragmentation corresponding to the missing key value, then the data fragmentation corresponding to the missing key value is sent to The data stitching module 704 . At the same time, the data fragment corresponding to the missing key value is also cached in the data cache module 706, so that if the same data is transmitted next time, it is not necessary to resend the original data by the data entry node, but directly obtain it from the local cache That's it.
该数据解析和缓存查找模块703被配置成解析该key值并通过该key值从该数据缓存模块706中查找本地是否缓存有与该key值对应的数据分片。在一个实施例中,该数据解析和缓存查找模块703被配置成按照一定算法获得该key值对应的原始数据分片。The data parsing and cache lookup module 703 is configured to parse the key value and use the key value to find out from the data cache module 706 whether a data fragment corresponding to the key value is locally cached. In one embodiment, the data parsing and cache lookup module 703 is configured to obtain the original data fragment corresponding to the key value according to a certain algorithm.
该数据拼接模块704被配置成,如果本地查找到有该key值对应的数据分片的缓存,则利用该些缓存的数据分片拼接成完整的数据发送给终端。另外,如果数据区分模块702判断出数据接收模块701所接收到的是缺失的key值与该缺失的key值所对应的数据分片的组合,所述该数据拼接模块还被配置成利用该缺失的key值所对应的数据分片拼接完整的数据发送给终端。The data splicing module 704 is configured to, if a cache of data fragments corresponding to the key value is found locally, use these cached data fragments to concatenate complete data and send it to the terminal. In addition, if the data distinguishing module 702 determines that what the data receiving module 701 receives is a combination of the missing key value and the data fragment corresponding to the missing key value, the data splicing module is also configured to use the missing The data fragments corresponding to the key value of the key are spliced and the complete data is sent to the terminal.
该缺失关键值发送模块705被配置成,如果本地没有该数据分片的缓存,则发送表示缺失该key值的信息给数据入口节点。The missing key value sending module 705 is configured to, if there is no cache of the data fragment locally, send information indicating that the key value is missing to the data entry node.
该数据缓存模块706被配置成缓存数据分片。The data caching module 706 is configured to cache data slices.
综上所述,本发明的基于流缓存的透明传输加速方案通过在数据接入节点进行处理,提取数据关键值,到达数据出口节点根据关键值还原完整数据,使传输过程的数据量减少,从而达到降低数据传输时间的效果。本发明的基于流缓存的透明传输加速方案解决了现有技术中的网络数据透明传输技术存在的上述三个弊端。To sum up, the transparent transmission acceleration scheme based on stream cache of the present invention extracts the key value of the data through processing at the data access node, and restores the complete data according to the key value at the data exit node, so that the amount of data in the transmission process is reduced, thereby To achieve the effect of reducing data transmission time. The stream cache-based transparent transmission acceleration solution of the present invention solves the above-mentioned three drawbacks existing in the network data transparent transmission technology in the prior art.
针对上述的第一个弊端,根据本发明的解决方法是,客户第一次下载数据时,在边缘节点上面对数据以块或分片为单位进行缓存,第二次下载同样数据时,离源近的节点只需要把数据块的key值传递给边缘节点,边缘节点从本地缓存中查找到相应的数据块,拼接成完整的数据返回给客户端。Aiming at the above-mentioned first drawback, the solution according to the present invention is that when the client downloads the data for the first time, the data is cached on the edge node in units of blocks or fragments, and when the same data is downloaded for the second time, away from The node near the source only needs to pass the key value of the data block to the edge node, and the edge node finds the corresponding data block from the local cache, splicing it into complete data and returning it to the client.
针对上述的第二个弊端,由于缓存是以数据块为单位的,对于不同的URL,只需要传输不同的数据部分,相同的数据块仍然可以只传输key值,因此传输的数据量也得到削减,提高了响应速度。For the second disadvantage above, since the cache is based on data blocks, for different URLs, only different data parts need to be transmitted, and the same data block can still only transmit key values, so the amount of transmitted data is also reduced , which improves the response speed.
针对上述的第三个弊端,解决方法是在数据入口节点先对数据块进行缓存,并传递key值到数据出口节点,如果数据出口节点已经有该数据块,则可以直接拼接数据响应,如果数据出口节点还没有该数据块,则向数据入口节点反馈,请求传输该数据块的数据。For the above-mentioned third disadvantage, the solution is to cache the data block at the data entry node first, and transfer the key value to the data exit node. If the data exit node already has the data block, you can directly splice the data response. If the data If the exit node does not have the data block yet, it will feed back to the data entry node and request to transmit the data of the data block.
这里采用的术语和表述方式只是用于描述,本发明并不应局限于这些术语和表述。使用这些术语和表述并不意味着排除任何示意和描述(或其中部分)的等效特征,应认识到可能存在的各种修改也应包含在权利要求范围内。其他修改、变化和替换也可能存在。相应的,权利要求应视为覆盖所有这些等效物。The terms and expressions used here are for description only, and the present invention should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features shown and described (or parts thereof), and it should be recognized that various modifications may also be included within the scope of the claims. Other modifications, changes and substitutions are also possible. Accordingly, the claims should be read to cover all such equivalents.
同样,需要指出的是,虽然本发明已参照当前的具体实施例来描述,但是本技术领域中的普通技术人员应当认识到,以上的实施例仅是用来说明本发明,在没有脱离本发明精神的情况下还可做出各种等效的变化或替换,因此,只要在本发明的实质精神范围内对上述实施例的变化、变型都将落在本申请的权利要求书的范围内。Similarly, it should be pointed out that although the present invention has been described with reference to the current specific embodiments, those of ordinary skill in the art should recognize that the above embodiments are only used to illustrate the present invention, without departing from the present invention. Various equivalent changes or substitutions can also be made under the spirit of the present invention. Therefore, as long as the changes and modifications to the above embodiments are within the spirit of the present invention, they will all fall within the scope of the claims of the present application.
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510131557.3A CN104753815A (en) | 2015-03-24 | 2015-03-24 | Flow cache based transparent transmission acceleration network and method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510131557.3A CN104753815A (en) | 2015-03-24 | 2015-03-24 | Flow cache based transparent transmission acceleration network and method |
Publications (1)
Publication Number | Publication Date |
---|---|
CN104753815A true CN104753815A (en) | 2015-07-01 |
Family
ID=53592944
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201510131557.3A Pending CN104753815A (en) | 2015-03-24 | 2015-03-24 | Flow cache based transparent transmission acceleration network and method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN104753815A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108337155A (en) * | 2018-05-10 | 2018-07-27 | 网宿科技股份有限公司 | A kind of e-mail transmission method, server and system |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1859115A (en) * | 2006-01-24 | 2006-11-08 | 华为技术有限公司 | Distributing storage downloading system, device and method for network data |
CN101945103A (en) * | 2010-08-09 | 2011-01-12 | 中国电子科技集团公司第五十四研究所 | IP (Internet Protocol) network application accelerating system |
US8190874B2 (en) * | 2004-12-21 | 2012-05-29 | Nec Corporation | Data transmission system and data transmission method |
CN103944969A (en) * | 2014-03-31 | 2014-07-23 | 中国电子科技集团公司第三十研究所 | Secure transmission optimization method and device for narrow-band network |
CN104270454A (en) * | 2014-10-14 | 2015-01-07 | 无锡云捷科技有限公司 | CDN dynamic application acceleration method based on data transmission optimizing system |
-
2015
- 2015-03-24 CN CN201510131557.3A patent/CN104753815A/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8190874B2 (en) * | 2004-12-21 | 2012-05-29 | Nec Corporation | Data transmission system and data transmission method |
CN1859115A (en) * | 2006-01-24 | 2006-11-08 | 华为技术有限公司 | Distributing storage downloading system, device and method for network data |
CN101945103A (en) * | 2010-08-09 | 2011-01-12 | 中国电子科技集团公司第五十四研究所 | IP (Internet Protocol) network application accelerating system |
CN103944969A (en) * | 2014-03-31 | 2014-07-23 | 中国电子科技集团公司第三十研究所 | Secure transmission optimization method and device for narrow-band network |
CN104270454A (en) * | 2014-10-14 | 2015-01-07 | 无锡云捷科技有限公司 | CDN dynamic application acceleration method based on data transmission optimizing system |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108337155A (en) * | 2018-05-10 | 2018-07-27 | 网宿科技股份有限公司 | A kind of e-mail transmission method, server and system |
WO2019214012A1 (en) * | 2018-05-10 | 2019-11-14 | 网宿科技股份有限公司 | Mail transmission method, server and system |
EP3591907A4 (en) * | 2018-05-10 | 2020-04-22 | Wangsu Science & Technology Co., Ltd. | Mail transmission method, server and system |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN100581257C (en) | Real-time streaming media transmission method and system based on video frame splitting | |
KR102228062B1 (en) | Method and apparatus for content delivery | |
CA2950326C (en) | Protocol stack adaptation method and apparatus | |
US20160006645A1 (en) | Increased data transfer rate method and system for regular internet user | |
CN113422818B (en) | Data cascade transmission method, system and node equipment | |
WO2016197804A1 (en) | Method and device for compressing data packet | |
CN109040786B (en) | Camera data transmission method, device and system and storage medium | |
CN102045772A (en) | Data transmission method and device | |
US8832375B2 (en) | Object type aware byte caching | |
CN111224903B (en) | A data transmission method, device and computer-readable storage medium | |
CN102708192A (en) | Method, system and device for sharing documents | |
CN103731493A (en) | Page transmission method, device and system | |
US20130318300A1 (en) | Byte Caching with Chunk Sizes Based on Data Type | |
CN110519656B (en) | Self-adaptive streaming media playing method, system and server | |
CN103152606B (en) | Video file processing method and device, system | |
CN103209195A (en) | Data acquisition method, terminal and far-end device | |
CN104012055B (en) | Method and apparatus processing data | |
US10033837B1 (en) | System and method for utilizing a data reducing module for dictionary compression of encoded data | |
CN104753815A (en) | Flow cache based transparent transmission acceleration network and method | |
CN117278628B (en) | Data transmission method, device, system, computer equipment and storage medium | |
CN107615810B (en) | Packet header compression system and method for online network code | |
CN106713317A (en) | Method and device for transmitting streaming media file | |
WO2014029081A1 (en) | Compression method and apparatus | |
US20070185971A1 (en) | Method and system for accelerating data communication that is using multipart | |
CN115378855A (en) | Data packet testing method and device, communication device and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150701 |