[go: up one dir, main page]

WO2019114700A1 - 流量分析方法、公共服务流量归属方法及相应的计算机系统 - Google Patents

流量分析方法、公共服务流量归属方法及相应的计算机系统 Download PDF

Info

Publication number
WO2019114700A1
WO2019114700A1 PCT/CN2018/120321 CN2018120321W WO2019114700A1 WO 2019114700 A1 WO2019114700 A1 WO 2019114700A1 CN 2018120321 W CN2018120321 W CN 2018120321W WO 2019114700 A1 WO2019114700 A1 WO 2019114700A1
Authority
WO
WIPO (PCT)
Prior art keywords
service
traffic
application
feature
identification
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/120321
Other languages
English (en)
French (fr)
Inventor
欧阳黜霏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP18887714.6A priority Critical patent/EP3697042B1/en
Publication of WO2019114700A1 publication Critical patent/WO2019114700A1/zh
Priority to US15/930,764 priority patent/US11425047B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/22—Traffic shaping
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/06—Generation of reports
    • H04L43/062—Generation of reports related to network traffic
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/02—Capturing of monitoring data
    • H04L43/026—Capturing of monitoring data using flow identification
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
    • H04L47/2441—Traffic characterised by specific attributes, e.g. priority or QoS relying on flow classification, e.g. using integrated services [IntServ]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/32—Flow control; Congestion control by discarding or delaying data units, e.g. packets or frames
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
    • H04L63/0227—Filtering policies
    • H04L63/0245—Filtering by information in the payload
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
    • H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/02—Capturing of monitoring data
    • H04L43/028—Capturing of monitoring data by filtering

Definitions

  • the present application relates to the field of network traffic analysis, and in particular, to a traffic analysis method, a public service traffic attribution method, and a corresponding computer system.
  • One of the purposes of traffic analysis is to assign messages (or data flows) contained in traffic to different applications. This process is referred to as application identification in this application.
  • application identification For example, an operator may need to charge traffic of a certain mobile application of a certain user, which requires calculating the amount of traffic generated by the user during the certain period of time due to the use of the application, that is, belonging to the application. The size of the traffic. To calculate the traffic size, you need to analyze the traffic during that time period to identify the packets belonging to the application. Based on application identification, operators can perform differentiated billing, provide consumers with richer services, monitor network real-time conditions, and dynamically adjust their network resource allocation.
  • the prior art generally uses the plaintext feature of the message for application identification, and the plaintext feature of the message is a feature composed of characters or numbers that can be directly parsed in the message.
  • the plaintext feature of the message is a feature composed of characters or numbers that can be directly parsed in the message.
  • some of the plaintext features of the original non-encrypted protocol are hidden, and the application recognition uses only the features that are not hidden, so that the accuracy of the application recognition is degraded.
  • an application may invoke multiple services.
  • the existing traffic analysis technology can distinguish traffic belonging to different applications to a certain extent, but rarely involves more fine-grained differentiation, such as distinguishing traffic belonging to different services (referred to in this application).
  • the present application provides a traffic analysis method, which is used to improve the accuracy of traffic analysis in a scenario of network protocol encryption. Specifically, the accuracy of application identification or service identification can be improved.
  • This method can be applied to gateways or other types of network devices.
  • the method includes: acquiring a feature of a packet in a traffic, where the feature includes a ciphertext feature, where the ciphertext feature includes any one or more of a sequence, a length, and a transmission direction of the encrypted packet in the packet.
  • An analysis is performed on the traffic based on the characteristics to identify a service or application to which the traffic belongs.
  • the "traffic” is a generalized one, and may be one or more data streams, and the number of the extracted features may be one or more.
  • the "message” mentioned in this application includes “data packets” and other types of messages, such as messages without data, only headers.
  • the “encrypted packet” may refer to different packets according to different encryption modes, which is not limited in this embodiment.
  • Service refers to a functional component that is smaller or equivalent to an application. The functionality provided by the service is usually invoked by the application through some kind of interface. An application can call one or more services.
  • the order of the packets refers to the location where a single packet appears in a data stream, or the sequence of multiple packets.
  • the length of the packet refers to the packet length.
  • the packet length can usually be obtained from a certain field in the packet. If there is no field indicating the packet length, the packet length can also be calculated in real time.
  • the transmission direction of the packet includes uplink or downlink.
  • the ciphertext feature can be the first A packet in a traffic (the A packet is an encrypted packet), the packet length is m bytes, and the packet length of the second A packet is n.
  • the first direction is that the length of the A message of the uplink is m, and the length of the A message of the downlink (or other type of message) in the second direction is n bytes; or the first The A message in the upstream direction is the nth message in the traffic, and so on.
  • the method provides a method for efficiently identifying a service or an application from a traffic, and solves the problem that the service or the application cannot be identified in the scenario of network protocol encryption, and considers the message in the process of service identification or application identification.
  • the ciphertext feature improves the accuracy of the recognition.
  • ciphertext features can also be used in combination with plaintext features.
  • the plaintext feature includes features of the message that can be directly parsed by characters and/or numbers.
  • the "character and/or number of features" referred to herein includes a single character or a single number, as well as strings and other possible combinations.
  • the character type is, for example, char
  • the string type is, for example, string.
  • the method is applied to a deep packet inspection (DPI) device, and the DPI device may be an independent network device or may be built in a gateway GPRS support node (GGSN). .
  • DPI device After the DPI device applies the method, it can perform charging with a policy and charging rules function (PCRF).
  • PCRF policy and charging rules function
  • the method is applied to other network traffic analysis devices.
  • the message with the extracted plaintext feature includes a part (or part or all of the information) in a transport layer security (TLS) handshake message.
  • TLS transport layer security
  • the message from which the ciphertext feature is extracted includes a data packet such as application data.
  • the performing an analysis on the traffic according to the feature to identify a service or application to which the traffic belongs includes: matching the feature with a recognition rule of the service or an identification rule of the application to identify The service or application to which the traffic belongs.
  • the two recognition rules that may be used here are based on the features obtained by machine learning algorithms.
  • the machine learning method is used to obtain the identification rules of the service or the reference, which makes the whole traffic analysis process more intelligent and improves the accuracy of the traffic analysis.
  • the message used in the learning and learning rule by the machine learning algorithm is not the current message to be analyzed, the message obtained from the historical traffic, or the same or similar feature obtained by other methods.
  • filtering may be performed on the traffic prior to analyzing the traffic.
  • the filtering method may refer to the filtering methods provided in the following aspects.
  • the traffic of various services may continue to be categorized.
  • the traffic attribution method provided by the following aspects may be referred to.
  • the present application provides a public service flow attribution method.
  • a public service is a service that is called by two or more applications. Therefore, the traffic belonging to a certain public service identified in the traffic analysis needs to determine which application is invoked, that is, which part of the traffic belongs to. Application, in order to support subsequent application flow accounting and other operations.
  • the feature of the packet in the traffic is obtained, where the feature includes a ciphertext feature, and the ciphertext feature includes any one or more of a sequence, a length, and a transmission direction of the encrypted packet in the packet.
  • An analysis is then performed on the traffic based on the characteristics to identify the originating service, the exclusive service, and the public service in the traffic.
  • An application is determined based on the exclusive service between the start time of the service A and the start time of the service B based on the recognition time. It is determined that the traffic of the public service between the initial service A and the identification time of the initial service B is attributed to the application.
  • the initial service A and the initial service B are any two adjacent services having a sequence of time (ie, identification time) identified in the traffic, in other words, the initial service B is the identification time.
  • the first start service after the first start service may be the same service or different services.
  • a and B are only for distinguishing between two services, which are easy to understand, and can also be distinguished by "first" and "second".
  • "between" can include an endpoint.
  • the feature further includes a plaintext feature.
  • the services invoked by the application are divided into three types: an initiating service, ie a service running in the application startup phase (not just an application launching service), such as a startup service, a login service, a registration service, etc.; A service, a service that can be called by multiple applications.
  • an exclusive service is a service that is called only by one application, so an exclusive service can uniquely identify an application.
  • public services and exclusive services are mutually exclusive, but the starting service may be an exclusive service or a public service.
  • the "identification time” may be represented by a true time value in a specific implementation as long as the identification order of each service is recorded.
  • the identification time of the service is not the time at which the service was actually identified, and may be represented by other numbers or other types of information that are capable of identifying the order in which the service was identified.
  • a time period can be determined by the identification time of the two initiating services, and the traffic generated in the time period belongs to the application to which the previous initiating service belongs. But this application is simply not determined by the initiating service, because the starting service may also be a public service, so it needs to be determined by an exclusive service in the time period. Because of the characteristics of the exclusive service, it is exclusive during that time period.
  • the application corresponding to the service is the application to which the previous initiating service belongs, and is also the application to which all the traffic generated during the time period belongs, so the traffic generated by the public service in the time period is of course also attributed to the application.
  • the public service is called by multiple applications, the method provided by the present application can determine which application is called by each application for the public service identified, in other words, which application the generated traffic should belong to.
  • traffic may also be filtered prior to performing traffic analysis.
  • the filtering condition is a maximum incoming packet number
  • the maximum incoming packet number is a quantity value of a packet determined according to the identification rule. Since the identification rule is obtained in advance by some methods, the maximum number of messages required for the identification rule to be used can be obtained directly or by some calculation.
  • filtering is performed on the traffic according to network protocol (IP) information of the traffic. Specifically, an autonomous system number (ASN) domain of a certain type of application that needs to perform traffic analysis is obtained by using the IP information, and the traffic is filtered by the ASN domain.
  • IP network protocol
  • ASN autonomous system number
  • the feature performs an analysis on the traffic, identifying an initial service, an exclusive service, and a public service in the traffic, including: combining the feature with a first identification rule, a second identification rule, and a third Identifying rules respectively identifying to identify the originating service, the exclusive service, and the public service in the traffic, wherein the first identification rule, the second identification rule, and the third identification rule are based on
  • the features are obtained by a machine learning algorithm.
  • the first identification rule, the second identification rule, and the third identification rule are identification rules for identifying three services: a start service, an exclusive service, and a public service, respectively, which are based on traffic analysis.
  • the combination of plaintext features and ciphertext features of the message is obtained through a machine learning algorithm.
  • the message here is derived from historical traffic data or data simulated by some mathematical method as described above.
  • each of the above three identification rules may include multiple identification rules for identifying multiple services of the same type.
  • the present application does not limit the first, second or third identification rule to only one identification rule.
  • the characteristics of the applied traffic are extracted (as samples) and the features are input to a machine learning algorithm (typically a supervised machine learning algorithm), and three service identification rules are output through the machine learning process: for identification A first identification rule of the initiating service, a second identification rule for identifying an application exclusive service, and a third identification rule for identifying a public service.
  • the features entered during the machine learning process include the plaintext features and/or ciphertext features of the message.
  • the process of machine learning can be either offline or online.
  • the learning of the service identification rules involved in multiple applications may be performed simultaneously or separately.
  • the association relationship between the application and the service rule is established according to the service identification rule learned in the same application.
  • the three services are distinguished according to the three service identification rules learned before, and the identification result includes the location information of each service (corresponding to the identification time of the service) ). Then segment the traffic under the individual user according to the location where the service is started, determine the application to which the segment belongs according to the exclusive service in each segment, and then attribute the public service located in each segment to each segment. The application to which it belongs.
  • the location information of the service is used to indicate the sequence relationship of the service, and may be represented by the time when the service is identified.
  • the correspondence between the exclusive service and the application that invoked the service may be pre-stored in a memory, and the application is determined based on the corresponding information.
  • the present application provides another public service traffic attribution method, and the method starting service and exclusive service may use only one of them compared with the former method.
  • the feature of the message in the traffic is obtained, the feature includes a ciphertext feature, and the ciphertext feature includes any one of a sequence, a length, and a transmission direction of the encrypted message in the packet. Or multiple.
  • the exclusive service is a service invoked by only one application, the public service being invoked by two or more applications service.
  • An application is determined based on the identified exclusive service A, which is an application that invokes the exclusive service A.
  • the traffic identifying the public service between the exclusive service A and the next identified exclusive service B is attributed to the application. It should be understood that the application calling the exclusive service B may be the same application as before.
  • a and B are only for distinguishing between two services, which are easy to understand, and can also be distinguished by "first" and "second”.
  • "between" can include an endpoint.
  • an initiating service is used in place of the exclusive service in the above process.
  • Any embodiment of the third aspect can be applied to the case where there is no other traffic between the originating service and the exclusive service (referring to the first exclusive service after the originating service) in the same application, because the exclusive service at this time Equivalent to the starting service.
  • the third aspect and the second aspect may be implemented separately in the product, or may be implemented at the same time. At the same time, it may be first determined that there is no other traffic between the initial service and the exclusive service, and if not, the third aspect or the third The method of any of the three aspects; if present, the method of any of the second or second aspect is used.
  • the present application also provides a traffic filtering method, which is generally performed prior to traffic analysis.
  • the identification rule (or other type of analysis rule) of the service or application is used in the traffic analysis.
  • the filtering method determines the traffic analysis by one or more of the identification rules that will be used later before the traffic analysis.
  • the maximum number of incoming packets at that time, more than that number of packets are filtered out.
  • the identification rules all involve the application data packet (here is only an example of the identification rule).
  • the last application data packet involved is the nth, then The application data packet after the nth application data packet can be filtered out, where n can be used as the maximum number of packets.
  • the maximum number of incoming packets may be a value, or may be a plurality of values corresponding to different types of data packets.
  • the present application further provides a traffic filtering method, in which an ASN domain is calculated according to IP information in a traffic, and then traffic filtering is performed through the ASN domain.
  • the present application further provides a traffic analysis apparatus, including one or more units, for implementing any of the methods of the first aspect.
  • the present application further provides a public service flow attribution device, including one or more units, for implementing any one of the second aspect or the third aspect.
  • the present application further provides a flow filtering device comprising one or more units for implementing any of the methods of the fourth aspect or any one of the fifth aspects.
  • the present application provides a computer system comprising a memory and a processor, the memory for storing computer readable instructions, the processor for reading the computer readable instructions and implementing the first aspect to the foregoing Any one or more of the five aspects.
  • the present application also provides a computer readable storage medium, typically non-volatile, for storing computer readable instructions that are read by one or more processors Any one or more of the foregoing first to fifth aspects are then implemented.
  • Multiple or “multiple times” appearing in the present application means “two or more” or “two or more times” unless otherwise specified.
  • the terms “first” and “second” appearing in this application do not have a meaning of order, only to distinguish two subjects in some description contexts for convenience of understanding, but the subject matter indicated is not in all embodiments. Both must be different subjects.
  • “A/B”, “A and/or B” appearing in the present application include A, B, and A and B. In this application Means A is a trademark name, but does not bring The words may also be trademark names.
  • Figure 1 is a hierarchical diagram of traffic
  • FIG. 3 is a schematic diagram of a TLS handshake process
  • FIG. 4 is a schematic diagram of a logical structure of a traffic analysis apparatus according to an embodiment of the present disclosure
  • FIG. 5 is a schematic flowchart of a traffic analysis method according to an embodiment of the present application.
  • FIG. 6 is a schematic diagram of a principle of a traffic attribution method according to an embodiment of the present disclosure.
  • FIG. 7 is a schematic diagram of a logical structure of a traffic analysis apparatus according to an embodiment of the present disclosure.
  • FIG. 8 is a schematic flowchart diagram of a method for constructing a traffic feature according to an embodiment of the present disclosure
  • FIG. 9 is a schematic flowchart of learning a recognition rule of a service or an application according to an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of a traffic filtering method according to an embodiment of the present disclosure.
  • FIG. 11 is a schematic flowchart of three service type identifications according to an embodiment of the present application.
  • FIG. 12 is a schematic flowchart of a method for affixing a public service flow according to an embodiment of the present disclosure
  • FIG. 13 is a schematic diagram of a logical structure of a computer system according to an embodiment of the present application.
  • Traffic Networking packets are generated when devices that are connected through the network interact. These packets are called traffic. Traffic is a general term.
  • Data flow A data packet generated in a complete communication process between the server and the client (from connection establishment to connection end), called the data flow of the connection, usually performs multiple interactions during the application, so it will generate more A stream of data that constitutes application traffic.
  • TCP transmission control protocol
  • finish a transmission control protocol
  • a data flow represents an interaction between two entities, such as an interaction between an application process and a server.
  • Public service An API deployed on a server for multiple applications to call, publicly providing services that perform certain functions, such as map navigation, cloud storage, video transmission, and more.
  • Traffic analysis Obtain network communication messages by means of monitoring, grabbing, copying, etc., and analyze, reorganize, and split them to restore the original communication content to understand the real-time status of both parties.
  • Plaintext feature A feature consisting of characters and/or numbers that can be directly parsed in a message is distinguished from a ciphertext feature.
  • Figure 1 is a hierarchical diagram of traffic.
  • Figure 1 shows the mobile application Facebook
  • the traffic of the application can be divided into three layers from the hierarchy.
  • the first layer is the data flow layer, that is, the traffic generated during the session initiated by the TCP FIN packet to establish the TLS handshake, indicating the application process and An interaction between servers.
  • the second layer is the service layer, that is, the sub-module that interacts with the server in the application, and all the traffic generated by the corresponding process and the server interaction is the traffic of the service module, for example, Cloud storage services, messaging services, and more.
  • the third layer is the application layer, which is the application. It also includes public services such as login services, cloud services, messaging services, and more.
  • the public service in the middle can be called by other applications, which means that the traffic belonging to the public service does not necessarily belong to all
  • the traffic analysis module identifies the public service, it also needs to attribute the traffic of the public service to the application that should belong to it by a certain method, so that the traffic of the application can be accurately calculated.
  • the current traffic identification technology mainly focuses on traffic identification at the application level, and basically does not involve traffic identification at the service level.
  • the public service traffic of the application market has already occupied more than 60% of the total traffic, and the number of applications using the public service module accounts for 95. %the above.
  • the most prominent service identification problem is Class service identification, for example: all use Public service traffic occurs in the application of the map service, for example Map traffic, the identification conflict, seriously affects the operator's business.
  • the application layer traffic identification technology cannot accurately identify the service and generate a high false recognition rate.
  • the commonly used traffic analysis scheme is a plaintext feature recognition method, which uses the plaintext feature of the hypertext transfer protocol (HTTP) packet and the plaintext feature of the TLS handshake packet to identify the traffic.
  • HTTP packets include request packets and response packets.
  • 2 is an example of an HTTP request message (a) and an HTTP response message (b).
  • the HTTP message consists of three parts: the start line, the message header, and the body. Table 1 shows the possible actions of the starting line.
  • the above actions can be used to determine the ongoing interaction between the client and the server.
  • the resource identified by the uniform resource identifier (URI) can determine the content of the interaction
  • the Host field in the header field can be used to determine whether the packet belongs to an application, and so on. Therefore, the plaintext feature analysis technology usually directly uses these character or digital features that can be parsed to predict the state of the network communication.
  • the encryption technology introduces the network communication protocol, only a small portion of the unencrypted traffic can continue to use the technology.
  • the plaintext feature fields of the original HTTP message are all encrypted into a field based on the hypertext transfer protocol secure (HTTPS). More than 90% of the current network traffic is the HTTPS protocol.
  • the structure is to encapsulate a layer of TLS protocol on the original HTTP packet.
  • the handshake process is shown in Figure 3, which is a three-way handshake process of the TCP protocol.
  • the TLS protocol client first sends ClientHello to the server.
  • the server returns ServerHello and the certificate.
  • the client After receiving the certificate, the client generates the public key for encryption, and sends the public key and encryption algorithm to the server.
  • the server confirms the certificate, End the handshake process.
  • Both parties then send encrypted application data packets.
  • the plaintext feature includes the characteristics of the TLS handshake message
  • the ciphertext feature includes the characteristics of the encrypted application data.
  • the prior art only uses plaintext features in the traffic to perform application identification.
  • the basic types of TLS handshake packets mainly include 10 (and other extension types).
  • the feature construction of the TLS handshake message below is mainly based on one or more of the 10 types of messages.
  • These 10 types of messages include (1)-(5) and (7) (equivalent to (9)) shown in FIG. 3, and also include HelloRequest, ServerKeyExchange, CertificateRequest, and CertificateVerify not shown in FIG.
  • the following 10 types of messages are briefly introduced through Table 2. Some of the messages in Table 2 are required according to the requirements of the server or client, and are not required in all scenarios.
  • the ChangeCipherSpec protocol is not part of the handshake protocol. It is sent to indicate that the encryption status of both parties is ready. The next communication uses the ciphertext encrypted communication negotiated by both parties, which will not be described in detail in this application.
  • the Finished package here indicates the end of the handshake process, not the TCP FIN message described above. The communication process between the client and the server is actually to establish a TCP handshake at the TCP layer, then transmit the TLS handshake message shown in Figure 3 by TCP, and then transmit the service packet, and finally end the interaction with the TCP FIN packet.
  • Existing solutions may utilize one or more of the above TLS handshake messages to construct features, translate the features into machine readable rules, such as XML, and store the rules.
  • the rules are configured to filter the traffic according to the corresponding protocol format.
  • the filtering mode can be sequential filtering. From the start of the clientHello packet to the end of the Finish packet, a full matching rule is established. The plaintext fields are all entered). After the filtering is completed, the filtered traffic is sent to the service logic matching module, and the application to which the traffic belongs is identified according to the application ID corresponding to the rule, and the matching result is output.
  • the rule of the above TLS handshake message may be used to establish a rule.
  • the application cannot be distinguished because the same type of application has a high degree of similarity on certain features (such as certificates).
  • the characteristics of the above TLS handshake messages cannot identify the traffic of different services in the same application.
  • public traffic generated by different applications using the same service will be recognized as a single application traffic.
  • a large amount of misidentification will occur.
  • the current plaintext features cannot segment the service traffic, and the public service traffic cannot be identified when the public service traffic occurs. Therefore, after the public service occurs, the statistical traffic tends to count the public traffic of the next or previous application to the current application. This leads to a higher rate of false recognition.
  • the same type of application refers to the same or similar application of the called service, because the server will issue the same type of certificate to the same service. At this time, only the TLS handshake message cannot be identified.
  • the same type of application may be the same type of application, such as two map applications of the same company or different companies, or different types of applications of the same company but calling the same service.
  • FIG. 4 is a schematic diagram of a logical structure of a traffic analysis apparatus 400 according to an embodiment of the present disclosure.
  • the device includes a feature learning module 410, a service identification module 420, and a traffic attribution module 430.
  • the traffic analysis device may be connected to a traffic analysis device 300 for performing analysis on the received traffic, and then inputting the analyzed result to the traffic analysis device 400.
  • the flow rate analysis process is to extract the value range information of the field step by step according to the protocol format (specifically, it is executed by the parsing module in FIG. 4), and the prior art can be used in detail, which is not described in detail in this embodiment.
  • the traffic analysis device 400 may further include a traffic filtering module 440, configured to perform filtering on all or part of the rules obtained by the feature learning module 410, and re-input the filtered traffic to the output of the traffic analysis device 300.
  • the service identification module 420 reduces the processing amount of the service identification module 420 and improves processing efficiency.
  • the parsing process can also be implemented in conjunction with hardware, such as in conjunction with hardware acceleration devices to speed up the parsing process.
  • the multiple modules in Figure 4 can be deployed on the same physical machine or on different physical machines.
  • the flow analysis method provided by the present application is described below, and the flow analysis method belongs to some or all of the functions provided by the flow analysis device 400.
  • FIG. 5 is a schematic flowchart of a method for analyzing a traffic according to an embodiment of the present disclosure.
  • the feature learning module 410 performs machine learning according to the collected historical traffic data or the traffic data obtained by other means, and obtains an application-service rule of each application through machine learning.
  • the message features need to be extracted.
  • the message features here include one or two of the plaintext features of the message and the ciphertext features.
  • the plaintext feature includes features of the message that can be directly parsed by characters and/or numbers.
  • the ciphertext feature includes any one or more of a sequence, a length, and a transmission direction of the encrypted message in the message.
  • An application's application-service rules contain the identification rules for the three services invoked by the application.
  • Three services include start-up services, application-only services, and public services.
  • the application-service rule is used to perform service identification, and because the three rules are associated with a specific application, the rule can also know which application the identified traffic belongs to. Two or more different applications, their initial services and public services may be partially or identical, so the recognition rules obtained by learning may be partially duplicated.
  • the machine learning process can be performed offline, ie not in real time; it can also be executed in real time.
  • some traffic data can be acquired periodically, and application-service rules are generated or updated by a machine learning method.
  • the administrator may manage the rules obtained by the feature learning module 410 through a management configuration module (not shown), for example, the administrator may add, delete, modify, or view the rules themselves.
  • the traffic analysis device 300 reads the packet in the traffic from the memory (for example, the memory), parses the packet according to the protocol format of the packet, and transmits the parsed packet (or traffic) to the traffic. Filter module 440.
  • the parsing process mainly involves a protocol above the transport layer, ie a TCP/IP layer, such as the TLS protocol.
  • Packets based on the TLS protocol can be divided into a TLS handshake portion and a TLS record portion according to the format.
  • the handshake part mainly includes seven types of data packets, including client Hello, ServerHello, and Certificate packets. As mentioned above, 10 types of data packets may not all be used.
  • the traffic filtering module 440 receives the traffic from the traffic analysis device 300, and obtains an application-service rule from the feature learning module 410, performs filtering according to the application-service rule, and sends the filtered packet to the packet.
  • Service identification module 420 receives the traffic from the traffic analysis device 300, and obtains an application-service rule from the feature learning module 410, performs filtering according to the application-service rule, and sends the filtered packet to the packet.
  • the feature learning module 410 stores the application-service rule in a file by using a file or other form
  • the traffic filtering module 440 reads the application-service rule from the memory and performs traffic filtering according to the application-service rule.
  • the traffic filtering module 440 is mainly for pre-processing traffic, such as filtering or splitting, before performing service identification, to reduce system overhead and improve processing efficiency of the service identification module 420.
  • the traffic filtering module 440 can support parsing of different fields in different messages such as HTTP and TLS, and can also support a custom regular filtering mode.
  • the traffic filtering module 440 may not be required.
  • the service identification module 420 receives the filtered traffic from the traffic filtering module 440, and obtains an application-service rule from the feature learning module 410, and performs service identification on the filtered traffic according to the application-service rule to obtain a recognition result.
  • the identification result includes the type of service to which the traffic belongs: the originating service, the application exclusive service, or the public service, and the "location" of each service. Finally, the identification result is sent to the traffic attribution module 430.
  • the location "location” of the service here is not the meaning of the geographical location.
  • the location information of the service can be understood as a mark or indication for indicating the order of time when the service is recognized relative to other services.
  • the location information of the service may be the time point of identifying the service, or may reflect Order numbers, etc.
  • the traffic of the data stream S1 is attributed to the initial service, and then the data stream S1, the apocalyptic service, and the service location are The correspondence is recorded in the memory.
  • the traffic attribution module 430 receives the identification result sent by the service identification module 420, and determines which application the public service traffic belongs to according to the initial service and the exclusive service (or exclusive service only).
  • the service identification module 420 records the recognition result into the memory, which may be a cache, or other types of memory, after which the traffic attribution module 430 reads the recognition result from the memory.
  • the application corresponding to the exclusive service (for example, the application ID) is recorded in the memory, and the time point is backward.
  • the public service that appears is subject to the traffic to the application.
  • the new application is recorded when the next exclusive service is identified (may be the same as the previous application, because the same application may have two or more exclusive services). This method is suitable for the scenario where there is no traffic between the initiating service and the exclusive service, and the exclusive service is equivalent to the initiating service.
  • the initial service is first identified, the application cut time is determined, and the split time is stored in the memory. It should be noted that the "time" here is not necessarily a value of time.
  • the exclusive service is identified, the application corresponding to the exclusive service is recorded in the memory, and the public service that appears from the point in time is attributed to the application. The application recorded in the memory is considered to be updated after the subsequent identification of the next start service.
  • the aging time of the stored content, or the number of entries of the stored content may be set on the premise of implementing the method.
  • the second implementation is taken as an example.
  • the first implementation has only minor differences.
  • the first implementation can be learned by those skilled in the art with reference to the second implementation.
  • the currently received traffic is segmented based on the location information of all the initial services identified. For example, the start service SS a and the start service SS b belong to the first segment; the start service SS b and the start service SS c belong to the second segment.
  • the application corresponding to one segment is determined according to the location information of the exclusive service. For example, if the exclusive service OS b is located in the second segment and the exclusive service OS b is exclusive to the application B, then the second segment is determined to correspond to the application B. It should be understood that the segmentation and the application are not one-to-one correspondence, the second segment corresponds to the application B, but the traffic that does not represent the application B exists only in the second segment, and the application B may be initiated multiple times.
  • the attribution application of the public service is determined according to the location information of the public service and the application corresponding to the segment. For example, the public service PS a is located in the second segment, and the second segment is known to correspond to the application B, then the traffic of the public service PS a is attributed to the application B.
  • S502-S505 is usually a real-time process.
  • FIG. 6 illustrates the traffic attribution process of the public service in a schematic manner.
  • the figure uses an arrow to represent a data stream and also represents a service.
  • the location of the service refers to the starting position of the arrow.
  • the boxes on the arrows represent the upstream message and the downstream message, and multiple blocks combine to form different message characteristics.
  • FIG. 6 it is assumed that three start services SS a , SS b and SS c have been identified by step S504; two exclusive services OS a and OS b ; and two common services PS a and PS b .
  • the start service SS b After the start service SS b , there is an exclusive service OS b before the next start service SS c , and OS b is known to be exclusive to the application B, so it can be determined that the start service SS b is the start service of the application B. , in turn, can determine the application start time of service B is about a starting time of SS b in the indicated location.
  • the exclusive service OS a is exclusive to the application A, so it can be determined that the apocalyptic service SS a is the starting service of the application A.
  • the public service PS a is located in the second segment, which is after application B is started, so its traffic should be attributed to application B. While the other public service PS b has the arrival time of most of the data streams coincides with the second segment, it is seen from the figure that its initial location (the location identifying the public service) is located in the first segment, and At this time, application B has not been started, so the traffic of PS b should not be attributed to application B, but should be attributed to application A.
  • the time when the service is identified (that is, the time indicated by the location of the service) is not the exact time when the application starts running or the service starts running, but the order in which the services are identified is generally consistent with the order in which the services are run. .
  • the goal of the method described below is to determine the attribution of Google public service traffic to improve The accuracy of the traffic identification of the application.
  • the method is similar to that of FIG. 5, and includes: firstly, an application-service rule is obtained by encrypting a traffic feature structure and a feature learning technology, and specifically includes three types of rules, that is, a first identification rule for identifying an initial service, for identifying an exclusive The second identification rule of the service and the third identification rule for identifying the public service (the specific rule learning process is followed); then the application-service rule filtering technology is used to reduce the traffic to be matched, dynamically set the number of incoming packets, etc., to reduce System performance overhead; then use the application-service rules to identify three types of services, and determine which application the public service belongs to based on the location of the different types of services.
  • FIG. 7 is a schematic diagram of the logical structure of the traffic analysis apparatus 700 provided by this embodiment.
  • the traffic analysis device 700 receives the analyzed traffic from the traffic analysis device 800 and performs traffic analysis.
  • the traffic analysis device 700 includes a feature learning module 710, a service identification module 720, a traffic attribution module 730, and a traffic filtering module 740. The device will be described below in conjunction with a detailed method.
  • Fig. 8 shows a method of determining a feature vector.
  • the method is performed by the constructor 711 of the feature learning module 710.
  • the feature matrix is first constructed by the constructor 711 (S801), where each column is a feature.
  • the method of constructing the feature matrix may use one or more of the following three.
  • the first type constructs a feature matrix according to the plaintext of the packet, such as the SNI (server name indication) field in the ClientHello packet as a list of features.
  • the feature matrix is constructed according to the ciphertext feature of the protocol, such as the first packet data length and/or the downlink packet length of the application data, without acquiring the ciphertext content.
  • the third type the feature matrix constructed by combining plaintext and ciphertext. The first time the feature matrix is constructed, it can be artificially constructed, and the subsequent steps can be adjusted according to the range of learned feature values.
  • the feature vector is started to be generated (S802). Specifically, the feature of each data flow in the application traffic is checked. If the data stream has the feature in the corresponding feature column, the flag 1 is marked, and if it does not appear, the flag is 0. This ultimately results in a feature matrix for all data streams, with each row of the matrix representing the feature vector of a data stream.
  • the application traffic of Google Map contains 20 data streams, and the structural features are listed as 30 columns, and then a 20 ⁇ 30 feature matrix composed of 0 and 1 is output.
  • FIG. 9 illustrates a method of obtaining an application-service rule by a machine learning algorithm based on a feature vector.
  • the method is performed by learner 712 of feature learning module 710.
  • the learner 712 acquires the feature vector from the constructor 711, and searches for a feature vector capable of distinguishing the service according to the machine learning algorithm, searches for the feature column and the feature value corresponding to the feature vector of the service, and converts the search result into a rule for identifying the service. (or the identification rule of the service) (S901).
  • three types of identification rules are searched: the first identification rule, the second identification rule, and the third identification rule respectively correspond to the apocalyptic service identification rule, the exclusive service identification rule, and the public service identification rule mentioned in the foregoing embodiments.
  • the learner 712 finds the feature vector for distinguishing the service (S902), the recognition rule corresponding to the feature vector is output, and the service identification rule learned for the same application is merged into the application-service rule of the application ( S903).
  • the learner 712 does not find a feature vector for distinguishing the service (S902), a request to reconstruct the feature matrix is sent to the constructor (S904), requesting reconstruction of the feature matrix.
  • the feature matrix is reconstructed by some predetermined method (S804), for example, the features of the ciphertext (for example, digital features) are equally divided, and then The feature matrix is constructed again based on the segmentation result, and the feature vector is re-outputted.
  • S804 some predetermined method
  • the features of the ciphertext for example, digital features
  • the feature matrix is constructed again based on the segmentation result, and the feature vector is re-outputted.
  • the steps shown in Figures 8 and 9 are iterated until the application-service rules are output.
  • Machine learning algorithms that can be used in this embodiment are, for example, decision trees, artificial neural networks, support vector machines, clustering, Bayesian classification, Markov chain and probability map models, and the like.
  • the rules include three types: a first identification rule, a second identification rule, and a third identification rule.
  • a rule includes one or more fields, as shown in Table 3 - Table 5 below.
  • the "field” in Table 3 - Table 5 refers to the field in the rule and is customized.
  • “Location” is the field in the actual data packet. This field is usually agreed by the Internet Protocol Group. It is visible in the RFC (Request For Comments) document of the corresponding protocol. It is a consensus within the domain; this field can be used to obtain the value to match the rule. The default value of the field in .
  • the value is obtained from the TLS handshake field of the received packet, and the value is obtained from the TLS record length field, and the identification rule is matched to determine whether the two obtained values are www.googleapis.com and 512, respectively. If yes, the match is successful; if not, the match is unsuccessful.
  • the following other rules are used in a similar way, and will not be described again.
  • sAppD[x] represents the length of the xth application data packet sent by the server to the client
  • cAppD[x] represents the length of the application data packet sent by the client to the xth server.
  • the above is the acquisition process of the service identification rule, which is performed offline.
  • the real-time traffic analysis process is described below.
  • the following processes of obtaining traffic, filtering traffic, identifying services, and attribution of public service traffic are real-time, sequential execution processes.
  • FIG. 10 shows a method of traffic filtering, which is not necessary, but can reduce the traffic to be matched and improve the processing efficiency.
  • the method is performed by domain filtering module 741 in traffic filtering module 740.
  • the input of the module 741 has two parts, one part is derived from the message obtained by the traffic analysis device 800 for analyzing the network traffic (ie, the traffic to be filtered), and the other part is derived from the application-service rule outputted by the learning period 712.
  • the output of this module 741 is the filtered flow.
  • the domain (S1001) performs filtering of the traffic according to the determination result and the maximum number of incoming packets (S1002), and the filtered traffic belongs to the ASN domain of Google and satisfies the requirement of the maximum number of incoming packets.
  • the maximum number of incoming packets refers to the maximum number of packets read by the traffic analysis device 700 from a data stream. For example, the maximum number of incoming packets is 5, and the number of read packets is less than or equal to 5, and the number of received packets is less than Will read, that is to say, when the traffic is filtered, more than 5 other packets are filtered out.
  • Figure 11 illustrates a method of performing service identification on filtered traffic.
  • the method is executed by the service identification module 720, and the input is the filtering result of the current network traffic by the domain filtering module 741 and the application-service rule output by the feature learning module 710; the output is the service classification identification result.
  • the single user identification module 721 distinguishes the application traffic of the single user according to the IP, the session ID, the device ID, the user ID, or other identification information in the filtered traffic, and inputs the service traffic into the service classification module 722 (S1101);
  • the module 722 identifies the start service, the exclusive service, and the public service of each of the individual user flows according to the application-service rule (S1102), and sends the recognition result to the traffic attribution module 730.
  • the identification process may be that the message characteristics in the user traffic are extracted, and the application-service rules are matched one by one, and the matching succeeds to end and output the service type and application corresponding to the rule that the matching is successful.
  • the single-user identification module 721 and its execution process are not required.
  • the traffic originates from one user, or the traffic originates from multiple users, but the requirements for the solution are Does not include traffic that distinguishes different users.
  • Figure 12 illustrates a method of assigning traffic to an application.
  • the method is performed by traffic attribution module 730.
  • the input of the module is the service identification result under the single user, and the output is the application attribution of the public service flow.
  • the location of the initial service is obtained (S1201), and the traffic under the single user is segmented by using the location (S1202); and the exclusive service located in the segment (ie, the current segment) is obtained, and the traffic is obtained.
  • the home application (S1203) which is the application that invokes the exclusive service.
  • a cache table is created, and the information recorded in the cache table includes the application ID, the user ID, and the location of the start service corresponding to the segment (S1204).
  • the cache table refers to a table that is stored in the cache in the form of a table.
  • the information may also be stored in other storage spaces by other forms.
  • the location of the identified public service is acquired (S1205). Determining whether it belongs to the current segment according to the location of the public service (S1206), if it belongs to the application to which the public service belongs (S1207); if it does not belong to the current segment, querying the corresponding location in the cache table with its own location information Application information (S1208), and output the application to which the public service belongs. Alternatively, the application information of the corresponding location is directly queried in the cache table according to the location of the public service, and the application to which the public service belongs is output.
  • the ID of an item in this embodiment refers to information used to identify the item, and may be digital, text, code, or other types of information.
  • the location of a certain service in this embodiment refers to the time at which the service is identified, with reference to the starting position of the arrow indicating the service in FIG.
  • any of the methods provided by the foregoing embodiments may be implemented on one or more physical computers, and the devices proposed in the foregoing embodiments may be deployed on one or more physical computers, and the unit module division inside the device is only exemplary. It can be seen that each unit module can be deployed on the same physical computer or on different physical computers.
  • FIG. 13 is a schematic diagram of a logical structure of a computer system according to an embodiment of the present disclosure.
  • the computer system can be any type of computer system such as a network device (eg, a DPI device), a server, a mobile terminal, a personal computer, an onboard computer, or the like.
  • the computer system 1300 includes components such as a processor 1310, a memory 1320, and a network interface 1330 (also referred to as a network card or network adapter, etc.).
  • the computer system can be interconnected with other devices to achieve more functions, such as flow metering.
  • the processor 1310 can be a single core processor or a multi-core processor. When the processor 1310 is a multi-core processor, the method provided by the present application can be run on one core or distributed on different cores.
  • the processor 1310 may be one or multiple, and the types of the plurality of processors may be the same or different.
  • the type of processor is a central processing unit (CPU), a graphics processor, a microprocessor or a coprocessor.
  • the network interface 1330 is used to connect other network devices, including wireless connections and wired connections.
  • the network interface 1330 can be used to obtain traffic from the network to perform traffic analysis or analysis.
  • the memory 1320 includes volatile and non-volatile memory.
  • the non-volatile memory stores computer readable instructions of the traffic analyzing device 1322 and/or the traffic analyzing device 1321 provided by the present application, and may also store other program modules 123.
  • Computer readable instructions eg, an operating system. Any one or more of the methods provided by the foregoing embodiments of the present application may be implemented after the computer readable instructions are read and executed by the processor 1310.
  • Specific implementations of the traffic analysis device 1322 and the traffic analysis device 1321 can be referred to the foregoing embodiments. In other embodiments, traffic analysis device 1322 and traffic resolution device 1321 can be deployed on different physical computers, respectively.
  • the above components are connected by a bus 140.
  • the bus 140 can be one or multiple.
  • the bus 140 includes an advanced microcontroller bus architecture (AMBA) industry standard architecture (ISA) bus, a micro channel architecture (MCA) bus, an extended ISA (extended-ISA) bus, video.
  • ABA advanced microcontroller bus architecture
  • MCA micro channel architecture
  • VESA video electronics standards association
  • PCI peripheral component interconnect
  • the traffic analysis method provided by the present application is different from the TLS handshake scheme identified only for the application in the prior art, and the present application provides a finer-grained service identification.
  • the ciphertext feature of the message is applied in the process of service identification, which improves the accuracy of service identification.
  • the ciphertext feature learning is added in the process of rule learning, and the influence of ciphertext features (such as the length, order, and direction of the application data packet) on service recognition, the structure of the feature matrix, and the learning feature vector are constructed.
  • the application-service rule is generated, and the recognition granularity is increased. Therefore, the problem that the TLS handshake part feature is insufficient to distinguish and identify the traffic is solved.
  • the traffic analysis method provided by the present application combines the characteristics of the encrypted HTTP session part with the TLS handshake plaintext feature, and learns the feature vector by using an adaptive binning method combining the numerical feature and the symbol feature to identify the application or Service traffic to improve recognition accuracy and accuracy.
  • the public traffic attribution method provided by the present application solves the attribution problem through the cooperation of three services, locates the traffic segment by using the initial service, acquires the application tag by using the exclusive service, and uses the segmentation information to belong to the public service traffic, and solves the public The traffic of the service cannot be attributed to the application.
  • the application also provides a filtering method for the ASN domain based on the maximum number of packets and traffic, reducing the size of the traffic to be analyzed, and taking into account the efficiency problem in the rule generation process, merging the redundancy rules, and reducing the number of judgments. Solved the problem that the rule complexity is too high, resulting in serious performance degradation.
  • the TLS handshake rule needs to parse the full process field of the certificate, consume a large amount of memory, and cannot accurately match the individual fields. Therefore, the identification overhead is increased, and the parsed field needs to be optimized, and the complexity of the rule is reduced.
  • the filtering method provided by the present application has the effect of adaptively adjusting the filtering strategy according to the parameters provided by the identification rule, reducing the impact of the redundancy rule on performance, designing a filtering module, reducing the number of readings and performance overhead, and improving the current technical solution.
  • the shortcomings of the field feature establishment rules are adapted to the high-speed real-time traffic identification environment.
  • the number of packets required for traffic identification is greatly limited. Therefore, in the process of the present application, the full traffic feature is not applied, but if the hardware technology advances or any specially constructed environment can To support this feature learning mode, the present application can naturally be extended to such a traffic recognition environment, and the core steps of the identification are still similar to the foregoing embodiments of the present application, and even if there are differences, those skilled in the art can easily think of it.
  • the feature value at the time of identification may be partially changed, and this solution still falls within the protection scope of the present application.
  • the technical solution provided by the present application can be applied to a video key quality indicators (KQI) scenario, such as a content delivery network (CDN) traffic, in addition to an operator's policy and charging control scenario.
  • KQI video key quality indicators
  • CDN content delivery network
  • the reason for the public traffic generated in this scenario is similar to that of the foregoing embodiment.
  • the method provided in the foregoing embodiment can basically identify and distinguish the attribution of the public traffic used by different applications in the CDN, so as to accurately complete the video KQI statistical requirement. More broadly, any scenario in which public traffic generated by a public service needs to be differentiated can be applied to the solution provided by this application.
  • the device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical modules, ie may be located A place, or it can be distributed to multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • the connection relationship between the modules indicates that there is a communication connection between them, and specifically may be implemented as one or more communication buses or signal lines.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

本申请提供一种流量分析方法、装置和计算机系统。该方法包括获取流量中的报文的明文特征和密文特征，所述密文特征包括所述报文中加密后的字段的长度特征；根据所述明文特征和所述密文特征对所述流量执行分析，以识别该流量所归属的服务或应用。该方法可以用于服务识别，也可以应用于应用识别。通过在流量分析中引入密文特征，在报文加密的场景下增加了流量识别的准确度。另一方面，本申请还提供一种公共服务的流量归属方法、装置和计算机系统，通过识别应用的起始服务和/或独占服务，将公共服务的流量归属到相应的应用，提供了应用流量归属的准确度，为后续应用流量计费等服务提供了基础。

Description

流量分析方法、公共服务流量归属方法及相应的计算机系统 技术领域
本申请涉及网络流量分析领域，尤其涉及流量分析方法、公共服务流量归属方法以及相应的计算机系统等。
背景技术
近年来随着移动互联网的发展迅速，网络流量占比逐年上升。为了保障网络用户享受到可靠的服务，方便网络管理方实时管理和监控各个网元间的活动，流量分析技术广泛应用于各种网络设备，包括网关、路由等网络中转或分组设备。当前运营商利用该技术为其开展计费、阻断、策略控制等网络业务提供信息保障。
流量分析的其中一个目的是将流量中包含的报文(或数据流)归属到不同的应用，本申请中将该过程称之为应用识别。例如：运营商可能需要对某个用户的某个手机应用的流量进行计费，这就需要计算出该用户在某个时段内由于使用该应用而产生的流量的大小，亦即属于该应用的流量的大小。要计算这个流量大小，就需要先分析该时段内的流量，从中识别出属于该应用的报文。基于应用识别，运营商就能够进行差异化计费，为消费者提供更丰富的服务，又能够监控网络实时状况，动态调整自己的网络资源分配。
然而，随着网络协议技术和应用程序技术的演进，流量分析技术面临到一些挑战。
一方面，现有技术通常使用报文的明文特征做应用识别，报文的明文特征即报文中能够被直接解析得到的字符或数字组成的特征。但是随着网络协议加密技术的广泛使用，原有非加密协议下报文的部分明文特征被隐藏，而应用识别仅使用未被隐藏的特征，使得应用识别的准确度下降。
另一方面，一个应用可能调用多个服务，现有的流量分析技术能够一定程度上区分属于不同应用的流量，但很少涉及更细粒度的区分，比如区分属于不同服务的流量(本申请称之为服务识别，作为流量分析的另一个目的)，特别是网络协议加密技术引入之后，报文的很多原有的明文特征被隐藏，使得服务识别难上加难。
进一步的，当多个应用调用同一服务时，会产生相似度较高的流量，即公共服务流量，如何将这些相似的公共服务流量识别出来并归属到各自的应用是目前业界没有解决的一个难题。
发明内容
下面介绍本申请提供的流量分析方法、服务识别方法、以及相应的装置等。应理解的是，以下方面未必涵盖本申请提出的所有实现方式，并且不同方面的实现方式和有益效果可互相参考。
第一方面，本申请提供一种流量分析方法，该方法用于在网络协议加密的场景下提高流量分析的精确度，具体的，可以提高应用识别或服务识别的准确度。该方法可以应用在网关或其它类型的网络设备中。该方法包括：获取流量中的报文的特征，所述特征包括密文特征，所述密文特征包括所述报文中加密后的报文的顺序、长度、传输方向中的任意一个或多个。根据所述特征对所述流量执行分析，以识别该流量所归属的服务或 应用。
“流量”是泛指，可以是一条或多条数据流，被提取特征的报文的数量可以是一个或多个。本申请中提到的“报文”包括“数据包”和其它类型的报文，比如不带数据，只有报头的报文。
“加密后的报文”根据加密方式的不同可以指不同的报文，本实施例不做限定。“服务”指的是比应用小或等同于应用的功能组件，服务提供的功能通常通过某种接口被应用调用。一个应用可以调用一个或多个服务。
报文的顺序指的是单个报文出现在一条数据流中的位置，或是多个报文的先后顺序。
报文的长度指的是包长，包长通常可以从报文中的某个字段中获取到；若没有表示包长的字段，也可以实时计算包长。
报文的传输方向包括上行或下行。
举例来说，密文特征可以为一条流量中第一个A报文(A报文为加密后的一种报文)的包长为m字节且第二个A报文的包长为n字节；或者，第一个方向为上行的A报文的长度为m且第二个方向为下行的A报文(或其他类型的报文)的包长为n字节；或者第一个方向为上行的A报文为该流量中的第n个报文，等等，可能的特征组合方式有很多，这里不再一一列举。
可见，该方法提供了一种从流量中高效识别服务或应用的方法，解决了在网络协议加密的场景下服务或应用无法识别的问题，并且通过在服务识别或应用识别的过程中考虑报文的密文特征，提高了识别的准确度。
在一些实施例中，密文特征还可以和明文特征组合使用。所述明文特征包括所述报文中能够被直接解析得到的字符和/或数字组成的特征。这里提到的“字符和/或数字组成的特征”包括单个字符或单个数字，也包括字符串以及其它可能的组合。在具体工程实现中，字符类型例如为char，字符串类型例如为string。
在一些实施例中，该方法应用于深度报文解析(deep packet inspection,DPI)设备，该DPI设备可以是独立的网络设备，也可以内置在网关GRPS支持节点(gateway GPRS support node,GGSN)中。DPI设备应用该方法之后，可以配合策略和计费网关(policy and charging rules function,PCRF)进行计费。在其它一些实施例中，该方法应用于其它网络流量解析设备。
在一些实施例中，被提取明文特征的报文包括安全传输层协议(transport layer security，TLS)握手报文中(部分或全部信息)。
在一些实施例中，被提取密文特征的报文包括application data这种数据包。
在一些实施例中，所述根据所述特征对所述流量执行分析，以识别该流量所归属的服务或应用，包括：将所述特征与服务的识别规则或应用的识别规则匹配，以识别所述流量所归属的服务或应用。这里可能用到的这两种识别规则是基于所述特征，通过机器学习算法获得的。通过机器学习的方法获得服务或引用的识别规则，使得整个流量分析过程更加智能，提高了流量分析的准确度。
应理解的是，通过机器学习算法学习识别规则时用到的报文并非当前待分析的报文，是从历史流量中获得的报文，或者通过其他方法获得的与现实流量具有相同或相似特征的仿真流量，从中获取的报文。
在一些实施例中，在对流量分析之前可以对流量执行过滤，过滤方法可参考下面几个方面所提供的过滤方法。
在一些实施例中，流量分析之后可继续对各种服务的流量进行归属，可参考下面几个方面所提供的流量归属方法。
第二方面，本申请提供一种公共服务流量归属方法。公共服务是被两个或多个应用调用的服务，所以在流量分析中识别出来的属于某一个公共服务的这部分流量需要确定是被哪个应用调用而产生的，也就是说该部分流量属于哪个应用，这样才能支持后续的应用流量计费等操作。
首先，获取流量中的报文的特征，所述特征包括密文特征，所述密文特征包括所述报文中加密后的报文的顺序、长度、传输方向中的任意一个或多个。然后，根据所述特征对所述流量执行分析，识别该流量中的起始服务、独占服务和公共服务。根据识别时间在起始服务A和起始服务B的识别时间之间的独占服务确定一个应用。确定识别时间在所述起始服务A和所述起始服务B的识别时间之间的公共服务的流量归属到所述应用。这的起始服务A和起始服务B是在流量中被识别到的时间(即识别时间)具有先后顺序的任意两个相邻服务，换句话说，所述起始服务B为识别时间在所述第一起始服务之后的第一个起始服务，两者可以是同样的服务，也可以是不同服务。这里的A和B仅是为了区分两个服务，便于理解，也可以用“第一”和“第二”来区分。这里的“在…之间”可以包括端点。
在一些实施例中，所述特征还包括明文特征。
在一些实施例中，应用调用的服务被分为三种类型：起始服务，即运行在应用启动阶段的服务(并非仅指应用启动服务)，例如启动服务、登陆服务、注册服务等；公共服务，即可以被多个应用调用的服务。和公共服务不同，独占服务是仅被一个应用调用的服务，所以通过这样独占服务可以唯一确定一个应用。
应理解的是，公共服务和独占服务是互相排斥的，但是起始服务可能是独占服务，也可能是公共服务。
在一些实施例中，“识别时间”在具体实现时可以不用真的时间值来表示，只要记录各个服务的识别顺序即可。
在一些实施例中，服务的识别时间并非是真实识别到服务的时间，可以用其他一些能够标识服务被识别到的先后顺序的数字或其它类型的信息表示。
由于起始服务是应用启动阶段的服务，所以通过两个起始服务的识别时间可以确定一个时间段，该时间段内产生的流量属于前一个起始服务所归属的应用。但是这个应用是哪个应用单纯通过起始服务无法确定，因为起始服务也可能是公共服务，所以需要通过该时间段内的一个独占服务来确定，由于独占服务的特点，所以该时间段内独占服务对应的应用就是前一个起始服务所归属的应用，也是该时间段内产生的所有流量所归属的应用，所以该时间段内的公共服务产生的流量理所当然也被归属到该应用。公共服务虽然被多个应用调用，但是通过本申请提供的方法就可以确定识别到的每个公共服务到底是被哪个应用调用，换句话说，其产生的流量到底应该归属给哪个应用。
在一些实施例中，在执行流量分析之前还可以对流量进行过滤。具体的，在一些实现方式中，过滤条件为最大进包数量，该最大进包数量为根据识别规则确定的一个报文 的数量值。识别规则由于是预先通过某些方法获得的，所以该识别规则要运用的话需要的最大报文的数量可以直接或通过某种计算方式获得。在另一些实现方式下，根据所述流量的网络协议(internet protocol,IP)信息对所述流量执行过滤。具体的，通过IP信息计算获得需要进行流量分析的某类应用的自治系统号(autonomous system number,ASN)域，通过该ASN域对流量执行过滤。通过在分析之前执行过滤可以减少待分析的报文数量，提高流量分析的效率；而且最大进包数量可根据识别规则适应性调整，提高了过滤的灵活性。
在一些实施例中，所述特征对所述流量执行分析，识别该流量中的起始服务、独占服务和公共服务，包括：将所述特征与第一识别规则、第二识别规则和第三识别规则分别匹配以识别所述流量中的所述起始服务、所述独占服务和所述公共服务，其中所述第一识别规则、所述第二识别规则和所述第三识别规则是基于所述特征，通过机器学习算法获得的。
在一些实施例中，第一识别规则、第二识别规则和第三识别规则为分别识别起始服务、独占服务和公共服务三种服务的识别规则，这三种识别规则是在流量分析之前基于报文的明文特征和密文特征组合，通过机器学习算法获得的。这里的报文如前述所述，来源于历史流量数据或通过某种数学方法仿真的数据。
在一些实施例中，以上三种识别规则中每种识别规则都可以包括多个识别规则，分别用于识别同类型的多个服务。换句话说，本申请并没有限定第一、第二或第三识别规则仅是一个识别规则。
在一些实施例中，提取应用的流量的特征(作为样本)，并将特征输入到机器学习算法(通常为有监督的机器学习算法)，通过机器学习过程输出三种服务识别规则：用于识别起始服务的第一识别规则，用于识别应用独占服务的第二识别规则，以及用于识别公共服务的第三识别规则。机器学习的过程中输入的特征包含报文的明文特征和/或密文特征。
在一些实施例中，机器学习的过程可以是线下的，也可以是线上的。多个应用所涉及的服务识别规则的学习可以同时进行，也可以分别单独进行。可选的，根据同一种应用内学习到的服务识别规则建立应用-服务规则的关联关系。
在一些实施例中，在实时系统中，当待分析的流量到达之后，根据之前学习到的三种服务识别规则区分三种服务，识别结果中包含各个服务的位置信息(相当于服务的识别时间)。然后根据其中起始服务的位置将单个用户下的流量分段，根据每个分段中的独占服务确定该分段所属的应用，然后将位置位于各个分段内的公共服务归属于各个分段所属的应用。其中，服务的位置信息用于指示服务的先后关系，具体可以用识别到服务的时间表示。
在一些实施例中，独占服务和调用该服务的应用的对应关系可以预先存储在存储器中，根据该对应信息确定所述应用。
第三方面，本申请提供另一种公共服务流量归属方法，与前一种方法相比，该方法起始服务和独占服务可以仅使用其中一种。
在一些实施例中，获取流量中的报文的特征，所述特征包括密文特征，所述密文特征包括所述报文中加密后的报文的顺序、长度、传输方向中的任意一个或多个。根据所 述特征对所述流量执行分析，识别该流量中的独占服务和公共服务，其中所述独占服务是仅被一个应用调用的服务，所述公共服务是被两个或多个应用调用的服务。根据识别到的独占服务A确定一个应用，所述应用为调用所述独占服务A的应用。将识别时间在所述独占服务A和下一个识别到的独占服务B之间的公共服务的流量归属到所述应用。应理解的是，调用该独占服务B的应用可能是跟之前一样的同一个应用。这里的A和B仅是为了区分两个服务，便于理解，也可以用“第一”和“第二”来区分。这里的“在…之间”可以包括端点。
在其它一些实施例中，使用起始服务替代上述过程中的独占服务。
第三方面的任意实施例可以应用于同一个应用中起始服务和独占服务(指的是起始服务之后的第一个独占服务)之间不存在任何其它流量的情况，因为此时独占服务相当于起始服务。
第三方面的其它实现方式可参考第二方面提供的实施例，在此不再赘述。第三方面和第二方面在产品中可以单独实现，也可以同时实现，同时实现时可以先判断起始服务和独占服务之间存不存在其它流量，若不存在，则使用第三方面或第三方面任意实施例的方法；若存在，则使用第二方面或第二方面任意实施例提供的方法。
第四方面，本申请还提供一种流量过滤方法，该方法通常在流量分析之前执行。在流量分析中用到服务或应用的识别规则(或其它类型的分析规则)，该过滤方法则是在流量分析之前就先通过后面会用到的这些识别规则中的一个或多个确定流量分析时的最大进包数量，多于该数量的数据包被过滤掉。例如，识别规则均涉及application data数据包(此处仅是一种识别规则的举例)，按照流量中报文的顺序来看，涉及到的最靠后的application data数据包是第n个，那么第n个application data数据包之后的application data数据包就可以被过滤掉了，这里n就可以作为最大进包数量的值。该最大进包数量可以是一个数值，也可以是不同类型的数据包分别对应的多个数值。
第五方面，本申请还提供一种流量过滤方法，该方法中根据流量中的IP信息计算ASN域，然后通过该ASN域进行流量过滤。
第六方面，本申请还提供一种流量分析装置，包括一个或多个单元，用于实现第一方面任意一种方法。另外，本申请还提供一种公共服务流量归属装置，包括一个或多个单元，用于实现第二方面任意一种方法或第三方面任意一种方法。再者，本申请还提供一种流量过滤装置，包括一个或多个单元，用于实现第四方面任意一种方法或第五方面任意一种方法。
第七方面，本申请提供一种计算机系统，包括存储器和处理器，所述存储器用于存储计算机可读指令，所述处理器用于读取所述计算机可读指令并实现前述第一方面到第五方面任意一种或多种方法。
第八方面，本申请还提供一种计算机可读存储介质，该介质通常是非易失性的，该介质用于存储计算机可读指令，该计算机可读指令在被一个或多个处理器读取之后实现前述第一方面到第五方面任意一种或多种方法。
本申请中出现的“多个”或“多次”若无特殊说明则意指“两个或两个以上”，或“两次或两次以上”。本申请中出现的“第一”和“第二”并无限定顺序的意思，仅是为了在某些描述上下文中区分两个主体，以方便理解，但是其所指示的主体并非在所有 实施例中都必须是不同的主体。本申请中出现的“A/B”、“A和/或B”包括A、B以及A和B三种情况。本申请中
Figure PCTCN2018120321-appb-000001
意指A为一个商标名称，但没有带
Figure PCTCN2018120321-appb-000002
的词语也有可能是商标名称。
附图说明
为了更清楚地说明本申请提供的技术方案，下面将对附图作简单地介绍。显而易见地，下面描述的附图仅仅是本申请的一些实施例。
图1为流量的层次示意图；
图2为HTTP请求报文和HTTP响应报文的示例；
图3为TLS握手过程示意图；
图4为本申请一实施例提供的流量分析装置的逻辑结构示意图；
图5为本申请一实施例提供的流量分析方法的流程示意图；
图6为本申请一实施例提供的流量归属方法的原理示意图；
图7为本申请一实施例提供的流量分析装置的逻辑结构示意图；
图8为本申请一实施例提供的流量特征构造方法的流程示意图；
图9为本申请一实施例提供的服务或应用的识别规则学习的流程示意图；
图10为本申请一实施例提供的流量过滤方法的流程示意图；
图11为本申请一实施例提供的三种服务类型识别的流程示意图；
图12为本申请一实施例提供的公共服务流量归属方法的流程示意图；
图13为本申请一实施例提供的计算机系统的逻辑结构示意图。
具体实施方式
为了方便理解本申请提出的技术方案，首先在此介绍本申请描述中会引入的几个要素。应理解的是，以下介绍仅方便理解这些要素，以期理解实施例的内容，并非一定涵盖所有可能的情况。
流量：通过网络连通的设备之间发生交互，就会产生网络通信报文，这些报文被称为流量。流量是一个泛指。
数据流：在服务器与客户端的一次完整通信过程(从连接建立到连接结束)中产生的数据包，称为该次连接的数据流，应用使用过程中通常会执行多次交互，因此会产生多条数据流，组成应用流量。
例如，以建立TLS握手开始，以传输控制协议(transmission control protocol，TCP)FIN(finish)报文为终止的一次会话期间产生的流量。数据流表示两个主体之间的一次交互过程，例如应用进程与服务器间的一次交互。
公共服务：部署在服务器上供多个应用程序调用的API，公开提供完成某些功能的服务，如地图导航、云存储、视频传输等等。
流量分析：通过监听、抓取、拷贝等手段获取网络通信报文，并对其进行解析、重组、切分等还原其原本通信内容的操作，以了解网络通信双方的即时状态。
明文特征：报文中能够被直接解析得到的字符和/或数字组成的特征，区别于密文特征。
请参考图1，为流量的层次结构示意图。图1以移动应用脸书
Figure PCTCN2018120321-appb-000003
为例，该应用的流量从层次结构上可以分为三层，第一层是数据流层，即以建立TLS握手开始，以TCP FIN包为终止的一次会话期间产生的流量，表示应用进程与服务器间的一次交互。第二层是服务层，即该应用中与服务器进行交互的子模块，其对应的进程与服务器交互产生的所有流量即为该服务模块的流量，比如
Figure PCTCN2018120321-appb-000004
云存储服务、消息服务等。第三层是应用层，即该应用程序
Figure PCTCN2018120321-appb-000005
中还包括公共服务，例如登陆服务、云服务、消息推送服务等。
Figure PCTCN2018120321-appb-000006
中的公共服务可以被其它应用程序调用，这意味着属于公共服务的流量不一定全部属于
Figure PCTCN2018120321-appb-000007
当新流量到达之后，流量分析模块在识别出公共服务之后，还需要通过一定方法将公共服务的流量做归属，划分到应该归属的应用程序中，这样才能准确计算应用程序的流量。
当前流量识别技术主要集中在应用层面的流量识别，基本没有涉及服务层面的流量识别，但是目前应用市场公共服务流量已经占据了总体流量的60％以上，使用公共服务模块的应用数量占总体的95％以上。其中最突出的服务识别问题为
Figure PCTCN2018120321-appb-000008
类服务识别，例如：所有使用
Figure PCTCN2018120321-appb-000009
地图服务的应用程序，都会出现公共服务流量，例如
Figure PCTCN2018120321-appb-000010
地图流量，的识别冲突，严重影响运营商的业务。然而在实际应用中，应用层流量识别技术不能精确识别服务，产生较高的误识别率。
现有普遍使用的流量分析方案为明文特征识别方法，利用超文本传输协议(hypertext transfer protocol，HTTP)报文的明文特征和TLS握手报文的明文特征识别流量。HTTP报文包括请求报文和响应报文。图2为HTTP请求报文(a)和HTTP响应报文(b)的示例。HTTP报文由三部分组成，分别是：起始行、消息首部和主体。表1示出了起始行可能的动作。
表1
Figure PCTCN2018120321-appb-000011
在流量分析中，通过上述动作就可以判断客户端和服务器端正在进行的交互行为。例如，利用统一资源标识符(uniform resource identifier,URI)标识的资源可以确定交互的内容，首部字段中的Host字段可以用来判断该报文是否属于某个应用，等等。因此明文特征分析技术通常直接利用这些可以被解析的字符或数字特征去推测网络通信双 方的状态，后续当加密技术引入网络通信协议后，只有少部分未加密的流量能够继续使用该项技术。
由于协议加密技术的应用，原有HTTP报文的明文特征字段，全部被加密变成了基于超文本传输安全协议(hypertext transfer protocol secure,HTTPS)的字段。当前网络流量的90％以上全部为HTTPS协议，其结构是在原来的HTTP报文之上封装了一层TLS协议，其握手过程如图3所示，类此TCP协议的三次握手过程。如图3所示，TLS协议客户端首先发送ClientHello给服务端，服务端返回ServerHello和证书，客户端接受证书后生成加密用的公钥，发送公钥和加密算法给服务端，服务端确认后结束握手过程。之后双方开始发送加密的应用数据(application data)报文。在协议加密的前提下明文特征包括TLS握手报文的特征，而密文特征包括加密的应用数据的特征。现有技术仅使用了流量中明文特征来执行应用识别。
TLS握手报文的基本类型主要包括10种(还有其它扩展类型)。下文中关于TLS握手报文的特征构造主要基于这10种报文中的一个或多个。这10种报文包括图3中示出的(1)-(5)以及(7)(相当于(9))，另外还包括图3中没有示出的HelloRequest、ServerKeyExchange、CertificateRequest以及CertificateVerify。下面通过表2对这10种报文进行简单的介绍。表2中的报文有些是根据服务器或客户端的要求需要的，并非所有场景下都是必须的。
表2
Figure PCTCN2018120321-appb-000012
Figure PCTCN2018120321-appb-000013
需要说明的是，ChangeCipherSpec协议并不属于握手协议的一部分，发送它表明双方的加密状态已经准备好了，接下来的通信使用双方协商好的密文加密通信，在本申请中不再详细介绍。另外，这里的Finished包表示握手过程结束，并非前文所述TCP FIN报文。客户端与服务器的通信过程实际是在TCP层先建立TCP握手，然后以TCP协议传送图3所示的TLS握手报文，然后传送业务报文，最后以TCP FIN报文结束本次交互。
现有方案可利用上述TLS握手报文中的一种或多种来构造特征，将特征转化为机器可读的规则，如XML，并存储这些规则。当网络流量被解析完成后，读入这些规则按对应的协议格式过滤流量，过滤方式可以为顺序过滤，从ClientHello报文开始，到Finish报文结束，建立全量的匹配规则(即报文中所有明文字段全部输入)。当过滤完成后，将过滤流量送入业务逻辑匹配模块，根据规则对应的应用ID识别该流量所归属的应用，将匹配结果输出。
但是，对于某一些同类型的应用，仅靠上述TLS握手报文的特征来建立规则可能存在不能区分应用的问题，因为同类型应用在某些特征(例如证书)上相似度比较高。同时，仅靠上述TLS握手报文的特征也不能识别同一应用下不同服务的流量。尤其是不同应用使用同一个服务产生的公共流量会被识别为单个应用流量，特别是服务内部也存在嵌套服务的情况下，会产生大量的误识别。当前的这些明文特征无法细分服务流量，当出现公共服务流量时也无法完成识别，所以在公共服务出现后，统计流量时往往将下一个或者上一个应用的公共流量统计到当前应用中来，导致误识别率较高。
这里同类型的应用指的是调用的服务相同或相似的应用，因为服务器会给同一种服务颁发同一类证书，此时仅靠上述TLS握手报文不能识别。同类型的应用可能是相同类型的应用，例如两个同公司或不同公司的地图应用，也可能是同一公司的不同类型应用但调用了相同的服务。
请参考图4，为本实施例提供的一种流量分析装置400的逻辑结构示意图。该装置包括特征学习模块410、服务识别模块420、流量归属模块430。
进一步的，该流量分析装置还可以和一个流量解析装置300相连，该流量解析装置300用于对接收到的流量执行解析，然后将解析后的结果输入到流量分析装置400。流 量解析过程就是按照协议格式，一步一步提取字段的值域信息(具体由图4中解析模块执行)，具体可采用现有技术，在本实施例中不做详细介绍。
进一步的，该流量分析装置400中还可以包括一个流量过滤模块440，用于对流量解析装置300的输出结果按照特征学习模块410获得的全部或部分规则执行过滤，将过滤后的流量再输入到服务识别模块420，以减少服务识别模块420的处理量，提高处理效率。解析过程还可以结合硬件实现，例如结合硬件加速装置加速解析过程。
图4中的多个模块可以部署在同一台物理机上，也可以部署在不同的物理机上。
以流量分析装置400装置为例，下面介绍本申请提供的流量分析方法，该流量分析方法属于流量分析装置400提供的部分或全部功能。
请参考图5，为本实施例提供的流量分析方法的方法流程示意图。
S501、特征学习模块410根据采集的历史流量数据或通过其他方式获得的流量数据，执行机器学习，通过机器学习获得每个应用的应用-服务规则。
机器学习的过程中需要提取报文特征，这里的报文特征包括报文的明文特征以及密文特征中的其中一个或两个。所述明文特征包括所述报文中能够被直接解析得到的字符和/或数字组成的特征。所述密文特征包括所述报文中加密后的报文的顺序、长度、传输方向中的任意一个或多个。
一个应用的应用-服务规则包含该应用所调用的三种服务的识别规则。三种服务包括起始服务、应用独占服务、以及公共服务。该应用-服务规则用于执行服务识别，同时因为这三种规则和特定的应用关联，通过该规则也能获知识别的流量属于哪一个应用。两个或更多不同的应用，它们的起始服务和公共服务可能是部分或完全相同的，所以学习获得的识别规则可能是存在部分重复的。
该机器学习过程可以在线下执行，即并非实时的；也可以实时执行。当实时执行时可以周期性地获取一些流量数据，通过机器学习的方法生成或更新应用-服务规则。
在其他一些实施例中，管理者可以通过一个管理配置模块(图中未示出)来管理特征学习模块410获得的规则，例如管理者可以自己增加、删除、修改或查看这些规则。
S502、当流量到达之后，流量解析装置300从存储器(例如内存)中读取流量中的报文，按照报文的协议格式解析报文，将解析后的报文(或称流量)传输给流量过滤模块440。
该解析过程主要涉及传输层以上的协议，即TCP/IP层，例如TLS协议。基于TLS协议的报文按照格式可以分为TLS握手部分和TLS记录(record)部分。在本实施例中，握手部分主要包含有7种类型的数据包，包括Client Hello，ServerHello，Certificate等数据包。前述已经提到过，10种类型的数据包不一定会全部都用到。
S503、流量过滤模块440从流量解析装置300接收流量，并从特征学习模块410获得应用-服务规则，将接收到的报文根据该应用-服务规则执行过滤，并将过滤后的报文发送给服务识别模块420。
具体的，特征学习模块410将应用-服务规则通过文件或其他形式存储在存储器中，流量过滤模块440从该存储器中读取该应用-服务规则后根据该应用-服务规则执行流量过滤。
流量过滤模块440主要是为了在进行服务识别之前对流量进行预处理，例如过滤或 分流等，以减少系统开销，提高服务识别模块420的处理效率。流量过滤模块440能够支持按HTTP、TLS等不同报文中的不同字段进行解析，也能够支持自定义正则过滤模式。
在其他一些实施例中，流量过滤模块440可以不需要。
S504、服务识别模块420从流量过滤模块440接收过滤后的流量，并从特征学习模块410获得应用-服务规则，根据该应用-服务规则对过滤后的流量执行服务识别，获得识别结果。该识别结果中包含流量所归属的服务的类型：起始服务、应用独占服务、或公共服务，以及每种服务的“位置”。最后，将该识别结果发送给流量归属模块430。
这里服务的位置“位置”并非地理位置的意思。服务的位置信息可以理解为一个标记或指示，用于表示该服务相对于其它服务被识别到的时间的先后顺序，例如服务的位置信息可以为识别到该服务的时间点，或是可以反映先后顺序的数字等。
例如，确定某条数据流S1的特征匹配某个应用的起始服务的特征，则该数据流S1的流量均归属到起始服务中，然后将数据流S1、启示类服务、服务位置这种对应关系记录到存储器中。
S505、流量归属模块430接收到服务识别模块420发送的识别结果，根据其中起始服务和独占服务(或仅靠独占服务)来确定公共服务的流量归属到哪个应用。
具体的，服务识别模块420将识别结果记录到存储器中，可以是缓存，也可以是其它类型的存储器，之后流量归属模块430在从该存储器中读取识别结果。
在一种实现方式下，不用考虑应用切分时间(即起始服务的位置)，当识别到独占服务时在存储器中记录该独占服务对应的应用(例如应用ID)，从该时间点往后出现的公共服务，其流量都归属到该应用。后续再识别到下一个独占服务时记录下新的应用(也可能和之前的应用一样，因为同一个应用可能有两个或多个独占服务)。这种方法适宜于起始服务和独占服务之间没有流量，独占服务相当于起始服务的场景。
在另一种实现方式下，先识别起始服务，确定应用切分时间，将切分时间存入存储器。需要说明的是，这里的“时间”并非一定是一个时间的值。当识别到独占服务时在存储器中记录该独占服务对应的应用，从该时间点往后出现的公共服务，其流量都归属到该应用。后续再识别到下一个起始服务之后才考虑更新存储器中记录的应用。
以上两种实现方式中，为了节省存储器的存储空间，可以在实现方法的前提下设置存储的内容的老化时间，或设置存储的内容的条目的数量等。
下面以第二种实现方式为例介绍，第一种实现方式仅存在微小区别，本领域技术人员参考第二种实现方式即可得知第一种如何实现。
首先，根据所识别到的所有起始服务的位置信息将当前接收到的流量分段。举例来说，起始服务SS a和起始服务SS b之间属于第一分段；起始服务SS b和起始服务SS c之间属于第二分段。
然后，根据独占服务的位置信息确定一个分段所对应的应用。例如，独占服务OS b位于第二分段内，而独占服务OS b为应用B所独占，那么确定第二分段对应应用B。应理解的是，分段和应用并非一一对应关系，第二分段对应应用B，但不代表应用B的流量仅存在于第二分段中，应用B可能会启动多次。
最后，根据公共服务的位置信息和分段对应的应用确定公共服务的归属应用。例如， 公共服务PS a位于第二分段，而已知第二分段对应应用B，那么公共服务PS a的流量被归属到应用B。
S502-S505通常为实时处理过程。
为便于理解，图6用示意图的方式示例出公共服务的流量归属过程。图中用一个箭头代表一条数据流，也代表一种服务，服务的位置指的是箭头的起始位置。箭头上的方框代表上行报文和下行报文，多个方框组合形成不同的报文特征。如图6所示，假设通过步骤S504已经识别出三个起始服务SS a、SS b和SS c；两个独占服务OS a和OS b；以及两个公共服务PS a和PS b。
在起始服务SS b之后，下一个起始服务SS c之前，存在一个独占服务OS b,而OS b已知为应用B所独占，所以可确定起始服务SS b为应用B的起始服务，进而可确定应用B的启动时间大约为起始服务SS b的位置所指示的时间。同理，独占服务OS a为应用A所独占，所以可确定启示类服务SS a为应用A的起始服务。
公共服务PS a位于第二分段，是应用B启动之后，所以其流量应该被归属到应用B。而另一个公共服务PS b虽然有大部分数据流的到达时间与第二分段重合，但是从图中看到其初始位置(识别到该公共服务的位置)是位于第一分段的，而此时应用B还未启动，所以PS b的流量不应该被归属到应用B，而应该被归属到应用A。
需要说明的是，识别到服务的时间(即服务的位置所表示的时间)并非是应用启动运行或服务启动运行的确切时间，但是识别到服务的先后顺序与服务运行的先后顺序通常是一致的。
以上对方案进行了整体性地说明，下面将以
Figure PCTCN2018120321-appb-000014
应用(例如Google Map)为例，详细介绍服务识别方法以及服务流量归属的方法，涉及以上各个步骤的具体实现。当前的技术对
Figure PCTCN2018120321-appb-000015
类应用的流量识别准确度较低，无法正确确定公共服务流量的归属，影响正常的运营商流量识别业务的开展，因此，本申请以
Figure PCTCN2018120321-appb-000016
类应用为例介绍流量分析方法。
下面将要介绍的方法的目标是确定Google公共服务流量的归属，以提高
Figure PCTCN2018120321-appb-000017
应用的流量识别准确度。
方法大致过程与图5类似，包括：首先通过加密流量特征构造和特征学习技术，得到应用-服务规则，具体包括三类规则，即用于识别起始服务的第一识别规则，用于识别独占服务的第二识别规则和用于识别公共服务的第三识别规则(具体规则学习过程见后续)；然后利用应用-服务规则过滤技术减少待匹配的流量、动态设定进包数量等，以减少系统性能开销；之后利用应用-服务规则识别三种类型的服务，并根据不同类型服务的位置确定公共服务归属到哪个应用。
请参考图7，为本实施例提供的流量分析装置700的逻辑结构示意图。该流量分析装置700从流量解析装置800中接收被解析后的流量，执行流量分析。具体的，该流量分析装置700包括特征学习模块710、服务识别模块720、流量归属模块730以及流量过滤模块740。下面结合详细的方法介绍该装置。
图8示出了特征向量的确定方法。该方法由特征学习模块710的构造器711执行。首先由构造器711构造特征矩阵(S801)，其中每一列为一种特征。
构造特征矩阵的方法可以使用如下三种中的一种或多种。第一种：根据报文的明文 构造特征矩阵，比如ClientHello报文中的SNI(server name indication)字段作为一列特征。第二种：根据协议的密文特征构造特征矩阵，比如上行应用数据(application data)的第一包数据长度和/或下行数据包长度等，无需获取密文内容。第三种：将明文和密文组合起来构造的特征矩阵。第一次构造特征矩阵，可以为人工构造，后续步骤中可以根据学习到的特征值范围再进行调整。
得到特征矩阵后，开始生成特征向量(S802)。具体的，检查应用流量中每条数据流的特征，如果该数据流在对应特征列出现了该特征，则标记1，没有出现则标记为0。这样最终能够得到全部数据流的特征矩阵，矩阵的每一行表示一条数据流的特征向量。比如，Google Map的应用流量包含20条数据流，构造特征列为30列，则输出20×30的由0和1构成的特征矩阵。
图9示出了基于特征向量，通过机器学习算法获得应用-服务规则的方法。该方法由特征学习模块710的学习器712执行。学习器712从构造器711获取特征向量，并根据机器学习算法寻找能够区分服务的特征向量，搜索该服务的特征向量对应的特征列和特征值，将搜索结果转换为用于识别该服务的规则(或称为服务的识别规则)(S901)。具体的，搜索三种类型的识别规则：第一识别规则，第二识别规则和第三识别规则，分别对应前述实施例提到的启示类服务识别规则、独占服务识别规则和公共服务识别规则。
当学习器712寻找到用于区分服务的特征向量时(S902)，则输出该特征向量对应的识别规则，并将针对同一种应用学到的服务识别规则合并为该应用的应用-服务规则(S903)。当学习器712没有寻找到用于区分服务的特征向量时(S902)，向构造器发送重新构造特征矩阵的请求(S904)，请求重新构造特征矩阵。参考图8，当构造器711确定接收到该请求之后(S803)，采用一些预定的方法重新构造特征矩阵(S804)，例如，将密文的特征(例如数字特征)进行等长的分割，然后根据分割结果再次构造特征矩阵，并重新输出特征向量。迭代图8和图9所示的步骤，直到输出应用-服务规则。
本实施例可以使用的机器学习算法例如决策树、人工神经网络、支持向量机、聚类、贝叶斯分类、马尔科夫链和概率图模型等。
规则包括三种：第一识别规则、第二识别规则和第三识别规则。一种规则包括一个或多个字段，如下表3-表5所示。
需要说明的是，表3-表5中的“字段”指的是规则中的字段，是自定义的。“位置”是实际的数据包中的字段，该字段通常由互联网协议小组约定，在对应协议的RFC(Request For Comments)文档中可见，是领域内共识；通过该字段可以获取到值来匹配规则中的字段的预设值。
表3
Figure PCTCN2018120321-appb-000018
第一识别规则示例：
SNI＝www.googleapis.com&&TLS record＝512
使用该规则的时候，从接收到的数据包的TLS handshake字段获取值，以及从TLS record length字段获取值，与该识别规则匹配，确定获取的两个值是否分别就是www.googleapis.com和512，若是，则匹配成功；若否，则匹配不成功。以下其它规则的使用方法与此类似，不再赘述。
表4
Figure PCTCN2018120321-appb-000019
第二识别规则示例：
Figure PCTCN2018120321-appb-000020
系统：SNI＝clients4.google.com&&sAppD[1]＝＝62&&sAppD[2]＝＝42&&sAppD[3]＝＝38&&sAppD[4]>＝242&&sAppD[4]<＝243&&cAppD[1]＝＝53&&cAppD[2]＝＝50&&cAppD[3]>＝301&&cAppD[3]<＝308
Figure PCTCN2018120321-appb-000021
系统:SNI＝clients4.google.com&&sAppD[1]＝＝376&&nCAppD>＝1&&cAppD[1]>＝848&&cAppD[1]<＝849
其中，sAppD[x]表示server端发送给client端第x个application data数据包的长度；cAppD[x]表示client端发送给server端第x个的application data数据包长度。
表5
Figure PCTCN2018120321-appb-000022
第三识别规则示例：
#SNI_googleadservices.com
#SNI_www.googleapis.com
#CertCommonName_google-analytics.com
以上为服务识别规则的获取过程，该过程是线下执行的。下面介绍实时的流量分析过程。在该实时流量分析过程中，以下获取流量、过滤流量、识别服务、以及公共服务流量归属等过程是实时的、顺序的执行过程。
图10示出了流量过滤的方法，该方法不是必须的，但可以减少待匹配的流量，提高处理效率。该方法由流量过滤模块740中的域过滤模块741执行。该模块741的输入有两部分，一部分来源于流量解析装置800对网络流量的解析得到的报文(即待过滤的流量)，另一部分来源于学习期712输出的应用-服务规则。该模块741的输出为过滤后的流量。
具体的，接收到应用-服务规则和待过滤的流量之后，根据应用-服务规则确定该规则识别服务时需要的最大进包数量(S1001)，另外根据待过滤的流量的IP信息计算Google的ASN域(S1001)，根据该判断结果和前述最大进包数量执行流量的过滤(S1002)，过滤后的流量属于Google的ASN域且满足最大进包数量的要求。
这里的最大进包数量指的是流量分析装置700从一条数据流中读取的报文的最大数量，例如最大进包数量为5，那么读取的报文数量小于或等于5，超过则不会读取了，也就是说，流量过滤的时候多余5个的其他数据包被过滤掉。
图11示出了对过滤后的流量执行服务识别的方法。本方法由服务识别模块720执行，输入为域过滤模块741对当前网络流量的过滤结果以及特征学习模块710输出的应用-服务规则；输出为服务分类识别结果。首先由单用户识别模块721根据过滤后的流量中的IP、Session ID、设备ID、用户ID或者其它身份识别信息，将单个用户的应用流量区分出来并输入服务分类模块722(S1101)；服务分类模块722根据应用-服务规则将单个用户流量中每个应用的起始服务、独占服务和公共服务识别出来(S1102)，将识别结果送入流量归属模块730。识别过程可以是将当个用户流量中的报文特征提取出来，和应用-服务规则一一匹配，匹配成功则结束并输出匹配成功的那个规则所对应的服务类型和应用。
需要说明的是，在其他一些实施例中，单用户识别模块721及其执行过程不是必须的，比如流量本来就来源于一个用户，或者流量虽然来源于多个用户，但是对方案提出的要求中不包括区分不同用户的流量。
图12示出了将流量归属到应用的方法。该方法由流量归属模块730执行。该模块的输入为单用户下的服务识别结果，输出为公共服务流量的应用归属。
具体的，获取起始服务的位置(S1201)，利用该位置将单个用户下的流量分段(S1202)；获取位置位于该分段(即当前分段)内的独占服务，得到该段流量所归属的应用(S1203)，该应用就是调用该独占服务的应用。然后建立缓存表，在缓存表中记录的信息包括该分段对应的应用ID、用户ID和起始服务的位置(S1204)。
为节省存储空间，缓存表中仅存储上一个分段和当前分段对应的应用ID、用户ID和起始服务的位置信息。
应理解的是，缓存表指的是以表格的形式存储在缓存中的一张表，在其它一些实施例中，这些信息也可以通过其它形式存储在其它存储空间中。
若前一个模块识别到公共服务，则获取识别到的公共服务的位置(S1205)。根据公共服务的位置判断是否属于当前分段(S1206)，属于则输出该公共服务所归属的应用(S1207)；如果不属于当前分段，则用自己的位置信息在缓存表中查询对应位置的应用信息(S1208)，并输出该公共服务所归属的应用。或者，根据公共服务的位置直接在缓存表中查询对应位置的应用信息，并输出该公共服务所归属的应用。
需要说明的是，本实施例中某一项的ID指的是用于标识该项的信息，可以是数字、文本、代码或其它类型的信息。本实施例中某一种服务的位置指的是识别到该服务的时间，参考图6中表示服务的箭头的起始位置。
前述实施例提供的任意方法可以实现在一台或多台物理计算机上，前述实施例中提出的装置可以部署在一台或多台物理计算机上，装置内部的单元模块划分仅作为一种示例性的示出，各个单元模块可以部署在同一台物理计算机上，也可以部署在不同的物理计算机上。
请参考图13，为本实施例提供的一种计算机系统的逻辑结构示意图。该计算机系统可以是网络设备(例如DPI设备)、服务器、移动终端、个人计算机、车载计算机等任意一种类型的计算机系统。该计算机系统1300包括处理器1310、存储器1320以及网络接口1330(也被称为网卡或网络适配器等)等组件。该计算机系统可以与其他设备互联实现更多的功能，例如流量计费等。
处理器1310可以为单核处理器，也可以为多核处理器。当处理器1310为多核处理器时，本申请提供的方法可以运行在一个核上，也可以分布运行在不同的核上。处理器1310可以为一个，也可以为多个，多个处理器的类型可以相同或不相同。处理器的类型有中央处理器(central processing unit,CPU)、图形处理器、微处理器或协处理器等。
网络接口1330用于连接其他网络设备，包括无线连接和有线连接。在本实施例中，网络接口1330可以用于从网络上获取流量，以执行流量解析或分析。
存储器1320包括易失性和非易失性存储器，通常非易失性存储器上存储有本申请提供的流量分析装置1322和/或流量解析装置1321的计算机可读指令，还可以存储其他程序模块123(例如操作系统)的计算机可读指令。当这些计算机可读指令被处理器1310读取和运行之后可实现本申请前述实施例提供的任意一种或多种方法。流量分析装置1322和流量解析装置1321的具体实现可参考前述实施例。在其他实施例中，流量分析装置1322和流量解析装置1321可分别部署在不同的物理计算机上。
以上组件通过总线140连接。总线140可以是一条，也可以是多条。总线140包括高级微控制器总线(advance microcontroller bus architecture,AMBA)工业标准结构(industry standard architecture,ISA)总线，微通道结构(micro channel architecture,MCA)总线，扩展ISA(extended-ISA)总线，视频电子标准协会(video electronics standards association,VESA)局域总线，以及外围器件互联(peripheral component interconnect,PCI)总线等。
本申请提供的流量分析方法，不同于现有技术中仅针对应用识别的TLS握手方案，本申请提供了更细粒度的服务识别。在服务识别的过程中应用了报文的密文特征，提高了服务识别的准确度。相应的，在规则学习的过程中加入了密文特征的学习，利用密文 特征(例如application data数据包的长度、顺序、传输方向等)对服务识别的影响，构造特征矩阵，以及学习特征向量，最终生成应用-服务规则，增加了识别粒度，因此，解决了TLS握手部分特征不足以区分和识别流量的问题。进一步的，本申请提供的流量分析方法将加密的HTTP会话部分的特征与TLS握手明文特征结合，通过将数值化特征和符号特征结合的自适应分箱方法，学习特征向量，用于识别应用或服务流量，提高识别准确率和精度。
本申请提供的公共流量归属方法，通过三种服务协同合作的方式解决归属问题，利用起始服务定位流量分段，利用独占服务获取应用标签，利用分段信息归属公共服务的流量，解决了公共服务的流量无法归属到应用的问题。
本申请还提供了基于最大进包数和流量的ASN域的过滤方法，减少需要分析的流量的大小，同时在规则生成过程中就考虑到效率问题，合并冗余规则，减少判断次数，因此，解决了规则复杂度过高，导致性能下降严重的问题。TLS握手规则需要对证书的全流程字段进行解析，消耗大量内存，不能做到针对单独字段进行精确匹配因此增加识别开销，需要优化解析的字段，并降低规则的复杂度。本申请提供的过滤方法效果在于根据识别规则提供的参数自适应的调整过滤策略，减少冗余规则对性能造成的影响，设计了过滤模块，减少读取次数和性能开销，改进了当前技术方案全字段特征建立规则的缺点，适配了高速实时的流量识别环境。
在主干核心网的高速环境中，对流量识别需要的报文数量有较大限制，因此本申请在阐述的过程中，并没有应用全流量特征，但是如果硬件技术进步或者任何特殊构造的环境能够支持这种特征学习方式，本申请能够自然的扩展到这种流量识别环境中，其识别的核心步骤依然与本申请前述实施例类似，即便存在不同之处本领域技术人员也容易想到。另外，由于TLS协议的任意包装，或者层级更低的TCP协议或人为构造的私有协议，可能会部分改变识别时的特征值，此方案仍然属于本申请的保护范围。
本申请提供的技术方案除可应用于运营商的策略和计费控制场景，还可以用于视频关键质量指标(key quality indicators,KQI)场景，例如内容分发网络(content delivery network，CDN)流量的区分，这种场景下公共流量产生的原因与前述实施例类似，基本可以按照前述实施例提供的方法去识别和区分CDN中不同应用使用的公共流量的归属，从而准确的完成视频KQI统计需求。更广泛的，任何由一个公共服务产生的公共流量需要区分的场景都可以适用本申请提供的方案。
需要说明的是，前述实施例中提出模块或单元的划分仅作为一种示例性的示出，所描述的各个模块的功能仅是举例说明，本申请并不以此为限。本领域普通技术人员可以根据需求合并其中两个或更多模块的功能，或者将一个模块的功能拆分从而获得更多更细粒度的模块，以及其他变形方式。
以上描述的各个实施例之间相同或相似的部分可相互参考。
以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的部件可以是或者也可以不是物理模块，即可以位于一个地方，或者也可以分布到多个网络模块上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。另外，本申请提供的装置实施例附图中，模块之间的连接关系表示它们之间具有通信连接，具体可以实现为一条或多条通信总线或信号线。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
以上所述，仅为本申请的一些具体实施方式，但本申请的保护范围并不局限于此。

Claims (20)

  1. 一种流量分析方法，其特征在于，包括：
    获取流量中的报文的特征，所述特征包括密文特征，所述密文特征包括加密后的报文的顺序、长度、传输方向中的任意一个或多个；
    根据所述特征对所述流量执行分析，以识别该流量所归属的服务或应用。
  2. 根据所述权利要求1所述的方法，其特征在于，所述加密后的报文包括以下类型的报文：应用数据application data。
  3. 根据所述权利要求1或2所述的方法，其特征在于，所述根据所述特征对所述流量执行分析，以识别该流量所归属的服务或应用，包括：
    将所述特征与服务的识别规则或应用的识别规则匹配，以识别所述流量所归属的服务或应用，所述服务的识别规则或应用的识别规则是基于所述特征，通过机器学习算法获得的。
  4. 根据所述权利要求1-3任意一项所述的方法，其特征在于，所述特征还包括明文特征，所述明文特征包括所述报文中能够被直接解析得到的字符和/或数字组成的特征。
  5. 根据所述权利要求1-4任意一项所述的方法，其特征在于，根据所述特征对所述流量执行分析，以识别该流量所归属的服务，包括：
    将所述特征与第一识别规则、第二识别规则和第三识别规则分别匹配以识别所述流量中的起始服务、独占服务和公共服务，其中所述第一识别规则、所述第二识别规则和所述第三识别规则是基于所述特征，通过机器学习算法获得的，所述起始服务是一个应用启动阶段所调用的服务，所述独占服务是仅被一个应用调用的服务，所述公共服务是被两个或多个应用调用的服务。
  6. 根据所述权利要求5所述的方法，其特征在于，还包括：
    确定识别时间在起始服务A和起始服务B之间的公共服务的流量归属到一个应用，该应用为调用识别时间在所述起始服务A和所述起始服务B之间的独占服务的应用，所述起始服务A为识别到的任意一个起始服务，所述起始服务B为识别时间在所述起始服务A之后的第一个起始服务。
  7. 一种公共服务流量归属方法，其特征在于，包括：
    获取流量中的报文的特征，所述特征包括密文特征，所述密文特征包括加密后的报文的顺序、长度、传输方向中的任意一个或多个；
    根据所述特征对所述流量执行分析，识别该流量中的起始服务、独占服务和公共服务，其中所述起始服务是一个应用启动阶段所调用的服务，所述独占服务是仅被一个应用调用的服务，所述公共服务是被两个或多个应用调用的服务；
    将识别时间在起始服务A和起始服务B之间的公共服务的流量归属到一个应用，该应用为调用识别时间在所述起始服务A和所述起始服务B之间的独占服务的应用，所述起始服务A为识别到的任意一个起始服务，所述起始服务B为识别时间在所述起始服务A之后的第一个起始服务。
  8. 根据所述权利要求7所述的方法，其特征在于，在获取所述特征之前，还包括：
    根据识别规则确定所述流量分析所需的最大进包数量，所述识别规则为基于所述特 征，通过机器学习算法获得的，所述识别规则用于从所述流量中识别不同的服务；
    基于所述最大进包数量对所述流量执行过滤。
  9. 根据所述权利要求7所述的方法，其特征在于，在获取所述特征之前，还包括：
    根据所述流量的网络协议IP信息对所述流量执行过滤。
  10. 根据所述权利要求7-9任意一项所述的方法，其特征在于，根据所述特征对所述流量执行分析，识别该流量中的起始服务、独占服务和公共服务，包括：
    将所述特征与第一识别规则、第二识别规则和第三识别规则分别匹配以识别所述流量中的所述起始服务、所述独占服务和所述公共服务，其中所述第一识别规则、所述第二识别规则和所述第三识别规则是基于所述特征，通过机器学习算法获得的。
  11. 根据所述权利要求7-10任意一项所述的方法，其特征在于，根据识别时间在第一起始服务和第二起始服务的识别时间之间的独占服务确定一个应用，包括：
    根据所述独占服务和对应信息确定所述应用，所述对应信息包括所述独占服务和调用所述独占服务的应用的对应关系。
  12. 根据所述权利要求7-11任意一项所述的方法，其特征在于，所述特征还包括明文特征，所述明文特征包括所述报文中能够被直接解析得到的字符和/或数字组成的特征。
  13. 一种公共服务流量归属方法，其特征在于，包括：
    获取流量中的报文的特征，所述特征包括密文特征，所述密文特征包括加密后的报文的顺序、长度、传输方向中的任意一个或多个；
    根据所述特征对所述流量执行分析，识别该流量中的独占服务和公共服务，其中所述独占服务是仅被一个应用调用的服务，所述公共服务是被两个或多个应用调用的服务；
    将识别时间在独占服务A和独占服务B之间的公共服务的流量归属到一个应用，该应用为调用所述独占服务A的应用，所述独占服务A为识别到的任意一个独占服务，所述独占服务B包括识别时间在所述独占服务A之后的第一个独占服务。
  14. 根据所述权利要求13所述的方法，其特征在于，在获取所述特征之前，还包括：
    根据识别规则确定执行所述流量分析所需的最大进包数量，所述识别规则为基于所述特征，通过机器学习算法获得的，所述识别规则用于从所述流量中识别不同的服务；
    基于所述最大进包数量对所述流量执行过滤。
  15. 根据所述权利要求13所述的方法，其特征在于，在获取所述特征之前，还包括：
    根据所述流量的网络协议IP信息对所述流量执行过滤。
  16. 根据所述权利要求13-15任意一项所述的方法，其特征在于，根据所述特征对所述流量执行分析，识别该流量中的独占服务和公共服务，包括：
    将所述特征与第二识别规则和第三识别规则分别匹配以识别所述流量中的所述独占服务和所述公共服务，其中所述第二识别规则和所述第三识别规则是基于所述特征组合，通过机器学习算法获得的。
  17. 根据所述权利要求13-16任意一项所述的方法，其特征在于，所述特征还包括明文特征，所述明文特征包括所述报文中能够被直接解析得到的字符和/或数字组成的特征。
  18. 一种计算机系统，其特征在于，包括存储器和处理器，所述存储器用于存储计算机可读指令，所述处理器用于读取所述计算机可读指令并实现如权利要求1-6任意一项所述的方法。
  19. 一种计算机系统，其特征在于，包括存储器和处理器，所述存储器用于存储计算机可读指令，所述处理器用于读取所述计算机可读指令并实现如权利要求7-12任意一项所述的方法。
  20. 一种计算机系统，其特征在于，包括存储器和处理器，所述存储器用于存储计算机可读指令，所述处理器用于读取所述计算机可读指令并实现如权利要求13-17任意一项所述的方法。
PCT/CN2018/120321 2017-12-15 2018-12-11 流量分析方法、公共服务流量归属方法及相应的计算机系统 Ceased WO2019114700A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP18887714.6A EP3697042B1 (en) 2017-12-15 2018-12-11 Traffic analysis method, public service traffic attribution method and corresponding computer system
US15/930,764 US11425047B2 (en) 2017-12-15 2020-05-13 Traffic analysis method, common service traffic attribution method, and corresponding computer system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201711354592.7 2017-12-15
CN201711354592.7A CN109936512B (zh) 2017-12-15 2017-12-15 流量分析方法、公共服务流量归属方法及相应的计算机系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/930,764 Continuation US11425047B2 (en) 2017-12-15 2020-05-13 Traffic analysis method, common service traffic attribution method, and corresponding computer system

Publications (1)

Publication Number Publication Date
WO2019114700A1 true WO2019114700A1 (zh) 2019-06-20

Family

ID=66819996

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/120321 Ceased WO2019114700A1 (zh) 2017-12-15 2018-12-11 流量分析方法、公共服务流量归属方法及相应的计算机系统

Country Status (4)

Country Link
US (1) US11425047B2 (zh)
EP (1) EP3697042B1 (zh)
CN (1) CN109936512B (zh)
WO (1) WO2019114700A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113938400A (zh) * 2021-08-27 2022-01-14 曙光网络科技有限公司 流表管理和维护的方法、设备以及存储介质
CN115174240A (zh) * 2022-07-13 2022-10-11 中国国家铁路集团有限公司 一种铁路加密流量监测系统及方法

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109951347B (zh) * 2017-12-21 2021-11-19 华为技术有限公司 业务识别方法、装置及网络设备
US11455569B2 (en) * 2019-01-09 2022-09-27 International Business Machines Corporation Device discovery and classification from encrypted network traffic
CN110414242B (zh) * 2019-08-02 2021-12-07 中国工商银行股份有限公司 用于检测业务逻辑漏洞的方法、装置、设备及介质
CN111030941A (zh) * 2019-10-29 2020-04-17 武汉瑞盈通网络技术有限公司 一种基于决策树的https加密流量分类方法
US20200326925A1 (en) * 2020-06-26 2020-10-15 Intel Corporation Memory device firmware update and activation with memory access quiescence
CN112003870B (zh) * 2020-08-28 2022-10-14 国家计算机网络与信息安全管理中心 一种基于深度学习的网络加密流量识别方法及装置
CN112153045B (zh) * 2020-09-24 2023-03-28 中国人民解放军战略支援部队信息工程大学 一种私有协议的加密字段的识别方法及系统
CN112929297B (zh) * 2021-01-21 2023-12-29 北京小米移动软件有限公司 一种资源分配方法、资源分配装置及存储介质
US11729154B2 (en) * 2021-02-25 2023-08-15 Comcast Cable Communications, Llc Systems and methods for network privacy
CN115017519B (zh) * 2021-03-04 2025-08-26 中国移动通信集团江苏有限公司 数据加密合规性检测方法及装置
US11934670B2 (en) * 2021-03-31 2024-03-19 Netapp, Inc. Performing various operations at the granularity of a consistency group within a cross-site storage solution
CN113890835A (zh) * 2021-09-29 2022-01-04 杭州迪普科技股份有限公司 Dpi应用测试报文的处理方法及装置
CN114254171B (zh) * 2021-12-20 2024-07-23 湖北天融信网络安全技术有限公司 数据分类方法、模型训练方法、装置、终端及存储介质
CN115766204B (zh) * 2022-11-14 2024-04-26 电子科技大学 一种针对加密流量的动态ip设备标识系统及方法
CN119363393B (zh) * 2024-09-29 2025-11-11 中国联合网络通信集团有限公司 数据识别方法、装置、设备、存储介质及程序产品
CN119762116B (zh) * 2024-12-23 2025-10-03 焦点科技股份有限公司 一种基于多维信息拟合的流量分配方法
CN120675947B (zh) * 2025-08-20 2025-11-18 中移(苏州)软件技术有限公司 流量报文的业务类型识别方法、装置、设备、介质及产品

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101505276A (zh) * 2009-03-23 2009-08-12 杭州华三通信技术有限公司 网络应用流量识别方法和装置及网络应用流量管理设备
CN102201982A (zh) * 2011-04-29 2011-09-28 北京网康科技有限公司 一种应用识别方法及其设备
CN103873320A (zh) * 2013-12-27 2014-06-18 北京天融信科技有限公司 加密流量识别方法及装置
CN105871832A (zh) * 2016-03-29 2016-08-17 北京理工大学 一种基于协议属性的网络应用加密流量识别方法及其装置
WO2016201876A1 (zh) * 2015-06-18 2016-12-22 中兴通讯股份有限公司 一种加密流量的业务识别方法、装置和计算机存储介质

Family Cites Families (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100523486B1 (ko) * 2002-12-13 2005-10-24 한국전자통신연구원 트래픽 측정 시스템 및 그의 트래픽 분석 방법
US7664048B1 (en) * 2003-11-24 2010-02-16 Packeteer, Inc. Heuristic behavior pattern matching of data flows in enhanced network traffic classification
WO2005099214A1 (en) * 2004-03-30 2005-10-20 Telecom Italia S.P.A. Method and system for network intrusion detection, related network and computer program product
US7778194B1 (en) * 2004-08-13 2010-08-17 Packeteer, Inc. Examination of connection handshake to enhance classification of encrypted network traffic
KR100596395B1 (ko) * 2004-12-16 2006-07-04 한국전자통신연구원 ＩＰｖ4망과 ＩＰｖ6망이 공존하는 네트워크 상에서암호화된 유해 트래픽에 대응하는 시스템 및 그 방법
US7447768B2 (en) * 2005-01-19 2008-11-04 Facetime Communications, Inc. Categorizing, classifying, and identifying network flows using network and host components
US7506156B2 (en) * 2005-02-01 2009-03-17 Telefonaktiebolaget L M Ericsson (Publ) Method and apparatus for prioritizing encrypted traffic at an intermediate node in a communications network
US8539064B1 (en) * 2005-09-13 2013-09-17 Aruba Networks, Inc. Analysis of encrypted streaming media traffic
US9349134B1 (en) * 2007-05-31 2016-05-24 Google Inc. Detecting illegitimate network traffic
US8417942B2 (en) * 2007-08-31 2013-04-09 Cisco Technology, Inc. System and method for identifying encrypted conference media traffic
US9304832B2 (en) * 2008-01-09 2016-04-05 Blue Coat Systems, Inc. Methods and systems for filtering encrypted traffic
US20100138910A1 (en) * 2008-12-03 2010-06-03 Check Point Software Technologies, Ltd. Methods for encrypted-traffic url filtering using address-mapping interception
US20100145912A1 (en) * 2008-12-08 2010-06-10 At&T Intellectual Property I, L.P. Detecting peer to peer applications
JP5408608B2 (ja) 2009-03-02 2014-02-05 公立大学法人大阪市立大学 暗号トラヒック識別装置及びそれを備える暗号トラヒック識別システム
CN101668035B (zh) * 2009-09-28 2012-08-22 中国人民解放军理工大学指挥自动化学院 一种实时识别多种p2p-tv应用视频流的方法
US10263899B2 (en) * 2012-04-10 2019-04-16 Seven Networks, Llc Enhanced customer service for mobile carriers using real-time and historical mobile application and traffic or optimization data associated with mobile devices in a mobile network
US9088859B2 (en) * 2012-09-06 2015-07-21 Dell Products, Lp Method and apparatus for connection context aware radio communication management
US9820316B2 (en) * 2013-03-15 2017-11-14 Aerohive Networks, Inc. Preventing asymmetric routing using network tunneling
CN104348741A (zh) 2013-08-06 2015-02-11 南京理工大学常熟研究院有限公司 基于多尺度分析和决策树的p2p流量检测方法和系统
CN103618792B (zh) * 2013-11-29 2017-04-19 华为技术有限公司 数据流的识别方法及设备
WO2015116855A1 (en) * 2014-01-29 2015-08-06 Intertrust Technologies Corporation Secure application processing systems and methods
EP3111613B1 (en) * 2014-02-28 2018-04-11 British Telecommunications public limited company Malicious encrypted traffic inhibitor
WO2015128609A1 (en) * 2014-02-28 2015-09-03 British Telecommunications Public Limited Company Profiling for malicious encrypted network traffic identification
US9760417B2 (en) * 2014-03-10 2017-09-12 Microsoft Technology Licensing, Llc Application dehydration and rehydration during application-to-application calls
CN103916294B (zh) * 2014-04-29 2018-05-04 华为技术有限公司 协议类型的识别方法和装置
CN104038389A (zh) 2014-06-19 2014-09-10 高长喜 多重应用协议识别方法和装置
CN104333461A (zh) * 2014-10-24 2015-02-04 深圳市傲天通信有限公司 互联网应用流量识别方法、系统及识别装置
US9787581B2 (en) * 2015-09-21 2017-10-10 A10 Networks, Inc. Secure data flow open information analytics
US10839077B2 (en) * 2015-12-24 2020-11-17 British Telecommunications Public Limited Company Detecting malicious software
US9984248B2 (en) * 2016-02-12 2018-05-29 Sophos Limited Behavioral-based control of access to encrypted content by a process
GB2551983B (en) * 2016-06-30 2020-03-04 Sophos Ltd Perimeter encryption
US11038906B1 (en) * 2017-02-03 2021-06-15 Level 3 Communications, Llc Network threat validation and monitoring

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101505276A (zh) * 2009-03-23 2009-08-12 杭州华三通信技术有限公司 网络应用流量识别方法和装置及网络应用流量管理设备
CN102201982A (zh) * 2011-04-29 2011-09-28 北京网康科技有限公司 一种应用识别方法及其设备
CN103873320A (zh) * 2013-12-27 2014-06-18 北京天融信科技有限公司 加密流量识别方法及装置
WO2016201876A1 (zh) * 2015-06-18 2016-12-22 中兴通讯股份有限公司 一种加密流量的业务识别方法、装置和计算机存储介质
CN105871832A (zh) * 2016-03-29 2016-08-17 北京理工大学 一种基于协议属性的网络应用加密流量识别方法及其装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3697042A4 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113938400A (zh) * 2021-08-27 2022-01-14 曙光网络科技有限公司 流表管理和维护的方法、设备以及存储介质
CN113938400B (zh) * 2021-08-27 2023-06-27 曙光网络科技有限公司 流表管理和维护的方法、设备以及存储介质
CN115174240A (zh) * 2022-07-13 2022-10-11 中国国家铁路集团有限公司 一种铁路加密流量监测系统及方法

Also Published As

Publication number Publication date
EP3697042A4 (en) 2020-12-16
US11425047B2 (en) 2022-08-23
US20200274812A1 (en) 2020-08-27
EP3697042A1 (en) 2020-08-19
CN109936512B (zh) 2021-10-01
EP3697042B1 (en) 2025-06-25
CN109936512A (zh) 2019-06-25

Similar Documents

Publication Publication Date Title
WO2019114700A1 (zh) 流量分析方法、公共服务流量归属方法及相应的计算机系统
CN109802924B (zh) 一种识别加密数据流的方法及装置
CN108521858B (zh) 基于分组的数据通信的设备标识符相关操作处理
CN106330584B (zh) 一种业务流的识别方法及识别装置
CN110868409A (zh) 一种基于tcp/ip协议栈指纹的操作系统被动识别方法及系统
CN110784383B (zh) Shadowsocks代理网络流量检测方法、存储介质和终端
CN114422211A (zh) 基于图注意力网络的http恶意流量检测方法及装置
CN104283699A (zh) 业务类型确定方法和装置
CN108390860A (zh) 一种数据包的加密、解密方法及装置
Mazhar Rathore et al. Exploiting encrypted and tunneled multimedia calls in high-speed big data environment
CN116599720A (zh) 一种基于GraphSAGE的恶意DoH流量检测方法、系统
WO2020228527A1 (zh) 数据流的分类方法和报文转发设备
CN117062102A (zh) 数据处理方法、装置、计算机可读介质及电子设备
CN113824729B (zh) 一种加密流量检测方法、系统及相关装置
CN105871698B (zh) 一种即时通讯服务的管理方法与系统
CN108206788A (zh) 一种流量的业务识别方法及相关设备
CN117375958A (zh) 一种web应用系统识别方法、装置及可读存储介质
CN109286506B (zh) 一种流量计费的方法、系统及装置
US11233703B2 (en) Extending encrypted traffic analytics with traffic flow data
CN105991465B (zh) 一种应用程序业务处理的方法、设备和系统
CN116910144A (zh) 算力网络资源中心、算力服务系统以及数据处理方法
WO2016201876A1 (zh) 一种加密流量的业务识别方法、装置和计算机存储介质
CN113824644B (zh) Https业务内容识别方法、装置和设备
CN110912904B (zh) 恶意设备识别方法、装置、存储介质和计算机设备
CN107579834A (zh) 一种家庭账号识别方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18887714

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2018887714

Country of ref document: EP

Effective date: 20200511

NENP Non-entry into the national phase

Ref country code: DE

WWG Wipo information: grant in national office

Ref document number: 2018887714

Country of ref document: EP