WO2025190232A1 - Traffic extraction method and apparatus for vulnerability scanner, and electronic device and storage medium - Google Patents

Abstract

The present application relates to the technical field of traffic extraction. Disclosed are a traffic extraction method and apparatus for a vulnerability scanner, and an electronic device and a storage medium. The method comprises: triggering a vulnerability scanner to send request traffic; determining whether the request traffic is redundant traffic, wherein the redundant traffic comprises universal traffic and/or repeated traffic, the repeated traffic being traffic identical to the previously received request traffic, and the universal traffic being request traffic for the vulnerability scanner to verify the survival of a site; and if the request traffic is non-redundant traffic, storing the request traffic. In this way, in the present application, whether request traffic is redundant traffic is determined, i.e., whether the request traffic is invalid traffic is determined, such that the request traffic that is not invalid traffic is stored, thereby reducing invalid traffic among stored request traffic.

Description

Traffic extraction method and device for vulnerability scanner, electronic device, and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Chinese patent application No. 202410300947.8, filed on March 15, 2024, entitled “Traffic extraction method and device, electronic device, and storage medium for vulnerability scanner,” and the entire contents of that application are incorporated herein by reference.

Technical Field

The present application relates to the field of traffic extraction technology, and in particular to a traffic extraction method and device, electronic device, and storage medium for a vulnerability scanner.

Background Art

At present, in order to facilitate researchers to derive the test method of vulnerability verification, the request traffic of the vulnerability scanner is usually obtained and stored first. In related technologies, a vulnerable target program with vulnerability defects is usually deployed first. Among them, vulnerability defects include XSS (cross-site scripting attack) cross-site request forgery vulnerability, SQL (Structured Query Language server database) injection vulnerability, etc. Then, the vulnerable target program is reverse-proxyed through Nginx (a web server with an asynchronous framework), and the IP (Internet Protocol Address) address is published to support access through HTTP (Hypertext Transfer Protocol) requests. At the same time, a storage device that supports HTTP traffic logs is configured. After that, the vulnerability scanning task is issued on the vulnerability scanner, and the target scanning address is set to the address published in Nginx. At this time, the vulnerability scanner will send vulnerability scanning request traffic to the target scanning address. After receiving the request traffic, Nginx will store the HTTP traffic log in the server's data file, and use a traffic parsing tool to parse and store the HTTP traffic log to obtain the request traffic of the vulnerability scanner.

During the implementation of the embodiments of the present application, it was found that at least the following problems exist in the related art:

Currently, there is a lot of invalid traffic in the request traffic of stored vulnerability scanners.

It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field.

Summary of the Invention

In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key/critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

Embodiments of the present application provide a traffic extraction method and device for a vulnerability scanner, an electronic device, and a storage medium to reduce invalid traffic in stored request traffic.

An embodiment of the present application provides a traffic extraction method for a vulnerability scanner, including: triggering a vulnerability scanner to send request traffic; determining whether the request traffic is redundant traffic; the redundant traffic includes: common traffic and/or repeated traffic; the repeated traffic is the same traffic as the request traffic received previously; the common traffic is the request traffic used by the vulnerability scanner to verify the survival of the site; if the request traffic is non-redundant traffic, storing the request traffic.

In the above embodiment, the request traffic sent by the vulnerability scanner may contain some test traffic such as those for verifying whether the page is alive, and the vulnerability scanner may send repeated request traffic when testing different web pages. Moreover, the test traffic such as those for verifying whether the page is alive and the repeated request traffic are both invalid traffic. Therefore, when all the request traffic sent by the vulnerability scanner are directly stored, the stored request traffic will contain a lot of invalid traffic. The present application can reduce some of the invalid traffic in the stored request traffic by determining whether the request traffic is general traffic and then storing the request traffic that is not general traffic. Similarly, the present application can also reduce some of the invalid traffic in the stored request traffic by determining whether the request traffic is repeated traffic and then storing the request traffic that is not repeated traffic. When the present application determines whether the request traffic is redundant traffic and then stores the request traffic that is not redundant traffic, the invalid traffic in the stored request traffic can be better reduced.

Furthermore, the redundant traffic includes general traffic; determining whether the request traffic is redundant traffic includes: calculating the request traffic using a preset hash algorithm to obtain a first hash value corresponding to the request traffic; determining whether the first hash value is in a preset general set; if so, determining that the request traffic corresponding to the first hash value is general traffic.

In the above embodiment, since the hash values of the same request traffic are identical, the hash values corresponding to common traffic can be pre-calculated and stored in a common set. By comparing the hash values of subsequently acquired request traffic with the hash values previously stored in the common set to see if they are identical, it is possible to accurately determine whether the request traffic is common traffic, thereby facilitating accurate storage of non-redundant traffic.

Furthermore, the universal set is obtained in the following manner: triggering a vulnerability scanner to send sample request traffic with a test object; using a preset hash algorithm to calculate each of the sample request traffic and form a hash table; classifying each of the hash tables according to the test object to form N hash sets; the value of N is the same as the number of types of the test objects; filtering out repeated hash tables in each hash set, and forming a universal set based on the repeated hash tables.

In the above embodiment, when testing different test objects, the vulnerability scanner may send multiple request flows for verifying site survival, and these request flows are identical. Therefore, by forming a hash table for the received sample request flows, classifying them by test object to obtain hash sets, and extracting duplicate hash tables from the hash sets, it is possible to determine the common set corresponding to the common traffic. This facilitates subsequent direct determination of whether the received request flows are common traffic based on the common set, thereby facilitating accurate storage of non-redundant traffic.

Furthermore, the test object includes at least one of url, host and directory.

Furthermore, the redundant traffic includes repeated traffic; determining whether the request traffic is redundant traffic includes: replacing the paths corresponding to the URI field in the request traffic and the Referer field in the request traffic with preset paths; using a preset hash algorithm to calculate the request traffic after the replaced path to obtain a second hash value; checking whether the second hash value is recorded in a preset hash value record table, if not, determining that the request traffic is non-redundant traffic, and adding the second hash value to the hash value record table, and storing the request traffic corresponding to the second hash value; if so, determining that the request traffic is repeated traffic.

In the above embodiment, since the vulnerability scanner may send the same request traffic to test different web pages, there may be some duplicate traffic in the received request traffic. By replacing the paths corresponding to the URI field in the request traffic and the Referer field in the request traffic with the preset paths, if the request traffic is the same, the calculated hash values of the request traffic will be the same. By comparing the second hash value corresponding to the request traffic after the replacement path with the second hash value corresponding to the request traffic after the replacement path previously recorded, it is possible to accurately determine whether the received request traffic is the previous duplicate traffic, thereby facilitating the accurate storage of non-redundant traffic.

Furthermore, storing the request traffic includes: obtaining a vulnerability name corresponding to the request traffic; marking the request traffic according to the vulnerability name, and storing the request traffic marked with the vulnerability name.

In the above implementation, by marking the request traffic for vulnerabilities and then storing it, the stored request traffic can be organized and convenient for subsequent use by the user.

Furthermore, storing the request traffic also includes: sending the request traffic marked with the vulnerability name to a preset message queue so that the message consumer can obtain the request traffic from the message queue and process it; obtaining and storing the processing results of the request traffic.

In the above implementation, the request traffic marked with the vulnerability name is sent to a preset message queue, and the request processing method can be changed to an asynchronous processing method to improve the response speed of the system.

Furthermore, after triggering the vulnerability scanner to send request traffic, the method further includes: parsing a URI field in the request traffic; determining a response template based on the URI field; and sending the response template to the vulnerability scanner.

In the above embodiment, after sending request traffic, the vulnerability scanner may stop sending request traffic if it does not receive a corresponding response. Therefore, by constructing a response template corresponding to the URI field in the request traffic in advance, all request traffic in the vulnerability scanner can be correctly responded to, so that the vulnerability scanner will not stop sending request traffic, and can obtain more request traffic.

An embodiment of the present application provides a traffic extraction device for a vulnerability scanner, including: a trigger module for triggering the vulnerability scanner to send request traffic; a determination module for determining whether the request traffic is redundant traffic; the redundant traffic includes: common traffic and/or repeated traffic; the repeated traffic is the same traffic as the request traffic received previously; the common traffic is the request traffic used by the vulnerability scanner to verify the survival of the site; and a storage module for storing the request traffic if the request traffic is non-redundant traffic.

An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the traffic extraction method of the vulnerability scanner mentioned above.

The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments are exemplarily described by corresponding drawings. These exemplary descriptions and drawings do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation. In addition,

FIG1 is a schematic diagram of a traffic extraction method for a vulnerability scanner provided in an embodiment of the present application;

FIG2 is a schematic diagram of another vulnerability scanner traffic extraction method provided in an embodiment of the present application;

FIG3 is a schematic diagram of a traffic extraction device of a vulnerability scanner provided in an embodiment of the present application;

FIG4 is a schematic diagram of an electronic device provided in an embodiment of the present application.

Icon: trigger module 1; determination module 2; storage module 3; processor 4; memory 5; communication interface 6; bus 7.

DETAILED DESCRIPTION

In order to be able to understand the features and technical contents of the embodiments of the present application in more detail, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The accompanying drawings are for reference only and are not used to limit the embodiments of the present application. In the following technical description, for the sake of convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

In the description and claims of the embodiments of the present application and the accompanying drawings, the terms "first," "second," and the like are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the purposes of describing the embodiments of the present application. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions.

Unless otherwise stated, the term "plurality" means two or more.

The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

Example 1

In an embodiment of the present application, a method for extracting traffic from a vulnerability scanner is provided. Referring to FIG. 1 , FIG. 1 is a schematic diagram of the basic flow of the method for extracting traffic from a vulnerability scanner provided in an embodiment of the present application, including:

Step S101: trigger the vulnerability scanner to send request traffic.

In some embodiments, for a vulnerability scanner that provides an API (Application Programming Interface), the vulnerability scanner can be controlled by calling the API interface corresponding to the vulnerability scanner to be triggered.

In other embodiments, for a vulnerability scanner that does not provide an API interface, the vulnerability scanner can be controlled by writing a program to simulate a browser clicking on the vulnerability scanner or simulating a control request to control the vulnerability scanner.

For example, engineers can pre-configure vulnerability scanning templates for the vulnerability scanner based on vulnerability names, test objects, and vulnerability categories. Each vulnerability scanning template corresponds to a preset tag type ID (identity document). When the vulnerability scanner receives a test request, it sends request traffic according to the vulnerability scanning template corresponding to the test request. The sent request traffic will carry the tag type ID corresponding to the vulnerability scanning template.

The format of the tag type ID can be scanner/{vulnerability category}/{test object}/{vulnerability name}. For example, the tag type ID is scanner (scanner)/web (a vulnerability category)/per_url (a test object)/sql_injection (a vulnerability name, the vulnerability name is SQL injection vulnerability). The vulnerability scanning template corresponding to this tag type ID is an SQL injection template that performs an SQL injection test on each URL.

Among them, vulnerability classification, for example: security vulnerabilities of the Web platform, security vulnerabilities of the application itself, etc.

The test object includes at least one of url, host, and directory. It will be understood that when the test object is url, the request traffic is classified or processed based on the request URI address. When the test object is host, the request traffic is classified or processed based on the host name or IP address corresponding to the request. When the test object is directory, the request traffic is classified or processed based on the directory path where the requested resource is located.

Vulnerability names include, for example, SQL injection vulnerability, XSS (Cross Site Scripting), code injection vulnerability, code execution vulnerability, code execution vulnerability, file creation/deletion vulnerability, file inclusion vulnerability, file upload vulnerability, CRLF (Carriage-Return Line-Feed, HTTP response splitting vulnerability) injection vulnerability, directory traversal vulnerability, LDAP (Lightweight Directory Access Protocol) injection vulnerability, SSRF vulnerability (Server-Side Request Forgery vulnerability), URL (Uniform Resource Locator) jump vulnerability, XML (Extensible Markup Language) external entity vulnerability, XPATH (XML Path Language) injection vulnerability, Java deserialization vulnerability, Apache Log4j (remote code execution vulnerability), directory listing vulnerability, and session fixation vulnerability. It is understood that the vulnerability names can also be other existing vulnerability names, which are not limited here.

In the above example, the tag type ID corresponding to the vulnerability scanning template may be carried in the request header of the request traffic.

Step S102: Determine whether the requested traffic is redundant traffic.

The redundant traffic may include: general traffic and/or repeated traffic.

It is understood that the repeated traffic may be the same traffic as the previously received request traffic. The common traffic may be the request traffic used by the vulnerability scanner to verify the survival of the site.

In some embodiments, the redundant traffic includes general traffic. Step S102 may include: calculating the request traffic using a preset hash algorithm to obtain a first hash value corresponding to the request traffic; determining whether the first hash value is within a preset general set; if so, determining that the request traffic corresponding to the first hash value is general traffic. Correspondingly, if not, determining that the request traffic corresponding to the first hash value is not general traffic.

For example, hash algorithms available in the prior art may be used, such as MD5 (Message Digest Algorithm 5), SHA-1 (Secure Hash Algorithm 1), SHA-3 (Secure Hash Algorithm 3), CRC32 (Cyclic Redundancy Check), etc., which are not limited here.

In an alternative embodiment of the above embodiment, the universal set can be obtained by triggering a vulnerability scanner to send sample request traffic containing test objects, using a preset hash algorithm to calculate each sample request traffic and form a hash table, classifying each hash table according to the test object to form N hash sets, filtering out duplicate hash tables in each hash set, and forming a universal set based on the duplicate hash tables. The value of N is the same as the number of test object types.

In the above optional method, the method of using a preset hash algorithm to calculate each sample request flow separately and form a hash table can be as follows: for each request flow: obtaining the vulnerability name and test object corresponding to the request flow, and using the preset hash algorithm to calculate the specific field in the request flow to obtain a third hash value. The third hash values corresponding to the request flows with the same vulnerability name and the same test object are formed into a hash table. Among them, the specific fields can be selected based on the engineer's experience, such as: request path field (Path), request body field (body), request header field (Header), request method field (method), request protocol field (schema), etc.

For example, a vulnerability scanner sends request traffic carrying a tag type ID, where the tag type ID includes a vulnerability name and a test object. Obtaining the vulnerability name corresponding to the request traffic may involve obtaining the vulnerability name from the tag type ID in the request traffic. Similarly, obtaining the test object corresponding to the request traffic may involve obtaining the test object from the tag type ID in the request traffic.

In the above optional manner, each hash table is classified according to the test object to form N hash sets. The test object corresponding to the hash table is obtained, and hash tables of the same test object are grouped into one hash set.

In the above optional method, filtering out duplicate hash tables in each hash set and forming a common set based on the duplicate hash tables can be performed by, for each hash table: sequentially comparing the hash table with the remaining hash tables in the same hash set; if any other hash table is identical or similar to the hash table, the hash table is considered a duplicate hash table; and all duplicate hash tables are formed into a common set.

Whether two hash tables are similar or identical can be determined by screening for duplicate third hash values between the two hash tables and accumulating the number of duplicate third hash values. If the ratio of the number of duplicates to the total number of third hash values is greater than a preset ratio, the two hash tables are determined to be similar. If the ratio of the number of duplicates to the total number of third hash values is equal to 1, the two hash tables are determined to be identical. It will be understood that the total number of third hash values is the total number of third hash values in one of the compared hash tables. The preset ratio can be selected based on the engineer's experience.

It can be understood that after calculating the sample request traffic using a preset hash algorithm to obtain the third hash value, the third hash value can be stored in the cache component (Redis) in a HASH manner to facilitate the subsequent use of the fourth hash value to form a universal set.

In another alternative embodiment of the above embodiment, a universal set can also be obtained by: triggering a vulnerability scanner to send sample request traffic containing a test object; using a preset hash algorithm to calculate each sample request traffic flow to obtain a fourth hash value corresponding to each sample request traffic flow; classifying each fourth hash value according to the test object to form N hash sets; filtering out the fourth hash value that is repeated a preset number of times in each hash set, and forming the fourth hash values that are repeated the preset number of times into a universal set. The preset number of times can be selected based on the engineer's experience, for example, the preset number of times is the same as the number of vulnerability names.

In another optional method of the above embodiment, the universal set can be obtained in the following manner: trigger the vulnerability scanner to send sample request traffic with the test object and vulnerability name; use a preset hash algorithm to calculate each sample request traffic separately to obtain the fourth hash value corresponding to each sample request traffic; classify the fourth hash values according to the test object and vulnerability name to form multiple hash tables; classify each hash table according to the test object to form N hash sets; for each hash set: filter the repeated fourth hash values between different hash tables in the hash set, and form the repeated fourth hash values into a universal set.

For example, a field is added to the request header of the request traffic to indicate whether the task is an initialization task. For example, if it is an initialization task, the field Task-Mode: init-task is inserted. The request traffic with the field Task-Mode: init-task inserted is used as the sample request traffic.

In the above optional manner, classifying the fourth hash value according to the test object and the vulnerability name to form multiple hash tables can be to form a hash table with the fourth hash values of the same vulnerability name and the same test object. In this way, due to the detection of a vulnerability name, multiple sample request flows may be sent, and the vulnerability name and test object corresponding to the multiple sample request flows are the same. By forming a hash table with the fourth hash values corresponding to the sample request flows of the same vulnerability name, a hash table for each vulnerability name can be obtained. Since there will be sample request flows for verifying the survival of the site among the multiple sample request flows sent for each vulnerability name under the same test object. Furthermore, the common flow can be determined by screening the fourth hash values repeated between different hash tables in the hash set.

In the above optional manner, classifying the hash tables according to the test objects to form N hash sets may be performed by obtaining the test objects corresponding to the hash tables and grouping hash tables of the same test objects into one hash set.

Exemplarily, forming a universal set with repeated fourth hash values may involve having a corresponding universal set for each hash set, and marking the universal set with a test object of the hash set corresponding to the universal set. Accordingly, determining whether the first hash value is within a preset universal set may involve obtaining a test object of request traffic corresponding to the first hash value and determining whether the first hash value is within the universal set of test objects corresponding to the first hash value.

Exemplarily, repeated fourth Hash values are grouped into a common set, or repeated third Hash values between different hash tables in each hash set are screened out respectively, and all screened out repeated third Hash values are grouped into a common set.

It is understood that after calculating the sample request traffic using a preset hash algorithm and obtaining a fourth hash value, the fourth hash value can be stored in the cache component in a HASH manner to facilitate subsequent use of the fourth hash value to form a universal set. After obtaining the universal set, the universal set can be stored in the cache component in a MAP storage structure manner, with the test object as the key and the universal set as the value.

Exemplarily, several request flows sent by a vulnerability scanner are obtained, and each request flow is parsed separately to obtain the test object and vulnerability name corresponding to each request flow. For example: the test object corresponding to request flow A is per_url, and the vulnerability name is SQL injection vulnerability. The test object corresponding to request flow B is per_url, and the vulnerability name is SQL injection vulnerability. The test object corresponding to request flow C is per_host, and the vulnerability name is code injection vulnerability. The test object corresponding to request flow D is per_host, and the vulnerability name is code injection vulnerability. The test object corresponding to request flow E is per_url, and the vulnerability name is directory traversal vulnerability. The test object corresponding to request flow F is per_url, and the vulnerability name is directory traversal vulnerability. Each sample request flow is calculated using a preset hash algorithm to obtain a fourth hash value corresponding to each sample request flow. For example: request flow A corresponds to the fourth hash value a. Request flow B corresponds to the fourth hash value b. Request flow C corresponds to the fourth hash value c. Request flow E corresponds to the fourth hash value e. Request flow F corresponds to the fourth hash value f. The fourth hash values a and b corresponding to the sample request flows A and B with the same vulnerability name and the same test object are respectively formed into a hash table G. The fourth hash values c and d corresponding to the sample request flows C and D with the same vulnerability name and the same test object are respectively formed into a hash table H. The fourth hash values e and f corresponding to the sample request flows E and F with the same vulnerability name and the same test object are respectively formed into a hash table I. Then, each hash table is classified according to the test object to form N hash sets. For example, hash table G and hash table I with the same test object are formed into a hash set J. Finally, the fourth hash values that are repeated in the two hash tables G and I in the hash set J are filtered out, and the repeated fourth hash values are included as part of the common set.

In some embodiments, redundant traffic includes duplicate traffic. Determining whether the request traffic is redundant traffic can be: replacing the paths corresponding to the URI field in the request traffic and the Referer field in the request traffic with preset paths. Calculating the request traffic after the path replacement using a preset hash algorithm to obtain a second hash value. Checking whether the second hash value is recorded in the preset hash value record table, if not, determining that the request traffic is non-redundant traffic, adding the second hash value to the hash value record table, and storing the request traffic corresponding to the second hash value. If so, determining that the request traffic is duplicate traffic.

Exemplarily, after receiving the request traffic sent by the vulnerability scanner, the request traffic is parsed into the following fields, such as: method field, host field, URI field, path field, content field, query field, headers field, scheme field, and port field. Then, the paths corresponding to the URI field in the request traffic and the Referer field in the header field in the request traffic are replaced with preset paths. For example, /auth/index/ is replaced with the preset /nTE6Enq1L9. Another example is: /form_data_post is replaced with the preset /nTE6Enq1L9. Another example is: http://xxx.com/form_data_post in the Referer field is replaced with http://xxx.com/nTE6Enq1L9.

Among them, the method field represents the request scheme, the host field represents the target host, the URI field represents the request URI, the path field represents the request path, the content field represents the request body, the query field represents the request parameters, the headers field represents the request header, the scheme field represents the request protocol, the port field represents the target port, and the Referer field represents the HTTP source address.

Step S103: If the requested traffic is non-redundant traffic, store the requested traffic.

In some embodiments, storing the request traffic may include: obtaining a vulnerability name corresponding to the request traffic; marking the request traffic according to the vulnerability name, and storing the request traffic marked with the vulnerability name.

For example, by parsing the tag type ID in the request header of the request traffic, the vulnerability name corresponding to the request traffic is obtained, and the request traffic is marked with the vulnerability name. For example, if the tag type ID of the obtained request traffic is scanner/web/per_url/sql_injection, where sql_injection indicates a SQL injection vulnerability, the vulnerability name of the request traffic is marked as SQL injection vulnerability.

In the above embodiment, storing the request traffic may also include: sending the request traffic marked with the vulnerability name to a preset message queue, so that the message consumer obtains the request traffic from the message queue and processes it; obtaining and storing the processing result of the request traffic.

For example, by parsing the tag type ID in the request header of the request traffic, the vulnerability name corresponding to the request traffic is obtained, and the request traffic is tagged with the vulnerability name. The request traffic tagged with the vulnerability name is sent to a preset message queue. The message consumer obtains the request traffic from the message queue, processes the request traffic, and saves the processed results to a preset storage database or a preset file.

In some embodiments, after triggering the vulnerability scanner to send request traffic, the traffic extraction method of the vulnerability scanner may further include: parsing the URI field in the request traffic; determining a response template based on the URI field; and sending the response template to the vulnerability scanner.

In the above embodiment, determining the response template according to the URI field can be performed by using a preset response database to look up the URI field and obtain the response template corresponding to the URI field. The response database at least stores the correspondence between the URI field and the response template.

In an optional manner of the above embodiment, the response template, for example, a root directory template and a sub-directory template. When the URI field belongs to the root directory, the root directory template is returned, and the root directory template can include the jump condition of the sub-directory template. When the URI field belongs to a specified URI, the corresponding sub-directory template will be directly returned. Among them, the sub-directory template can be the response content. The sub-directory template can also be a page that meets the standard of sending request traffic by the vulnerability scanner by customizing the response content.

For example, the root directory basic response page template represents the response content returned when the vulnerability scanner makes a root directory request. That is, the root directory template is the root directory interface returned to the vulnerability scanner. For example, the root directory template is:

If the sub-directory template is to construct a page that meets the standards of the vulnerability scanner's request traffic by customizing the response content, the sub-directory template can be a dictionary set containing the specified URI, response content, and response status code.

Taking /auth/index/ and /auth/ as examples, /auth/index/ is the designated URI of the response template. When the vulnerability scanner accesses this designated URI, it will return the corresponding sub-directory template. Usually, when constructing the sub-directory template, it is also necessary to insert the trigger tag of the sub-directory template into the root directory template. For example, insert the trigger tag <a href="/auth/index">auth</a> into the root directory template. In this way, when the vulnerability scanner scans the a tag in the root directory template, the vulnerability scanner will request the link corresponding to the a tag, thereby requesting the /auth/index/ directory path. After confirming that the page exists, the vulnerability scanner will generally split the directory and then split the access, thereby triggering the `/auth/` request. The /auth/ page will correspondingly return the WWW-Authenticate field to indicate that the page requires authorization. At this time, the vulnerability scanner will use the AUTH authorization method to perform an authorization blasting test on the page or perform an injection test on the Basic Auth field.

The following table shows an example of the correspondence between the response content corresponding to a specified URI and the content that needs to be inserted into the root directory template when the URI field is a specified URI. For example, when the specified URI is "/query_get/idnex?id=hello", the corresponding response content is "Status Code: 200" and the corresponding root directory template content is "<ahref="/query_get/idnex.php?id=hello">auth</a>".

Table 1

Example 2

Another method for extracting traffic from a vulnerability scanner is provided in an embodiment of the present application. FIG2 is a schematic diagram of a basic flow chart of another method for extracting traffic from a vulnerability scanner provided in an embodiment of the present application, including:

Step S201, triggering the vulnerability scanner to send request traffic, and then executing step S202.

Step S202: After receiving the request traffic, extract the Task-Mode field in the request header of the request traffic through the traffic parsing function, and then execute step S203.

Step S203: Determine whether the task corresponding to the requested traffic is an initialization task according to the Task-Mode field, and then execute step S204.

Step S204: When the task corresponding to the requested traffic is not an initialization task, the requested traffic is calculated using a preset hash algorithm to obtain a first hash value corresponding to the requested traffic, and then step S205 is executed.

Step S205: Determine whether the first Hash value is in a preset universal set. If the first Hash value is not in the preset universal set, execute step S206. If the first Hash value is in the preset universal set, execute step S210.

In step S206, the paths corresponding to the URI field in the request traffic and the Referer field in the request traffic are replaced with preset paths; the request traffic after the replaced paths is calculated using a preset hash algorithm to obtain a second hash value, and then step S207 is executed.

Step S207: Check whether the preset hash value record table records the second hash value. If the preset hash value record table does not record the second hash value, execute step S208. If the preset hash value record table records the second hash value, execute step S210.

Step S208 , determining that the request traffic is non-redundant traffic, adding the second hash value to the hash value record table, and storing the request traffic corresponding to the second hash value, and then executing step S209 .

Step S209, obtaining the vulnerability name corresponding to the request traffic; marking the request traffic according to the vulnerability name, and storing the request traffic marked with the vulnerability name, and then executing step S210.

Step S210, parse the URI field in the request traffic; determine a response template according to the URI field; and send the response template to the vulnerability scanner.

Example 3

Based on the same inventive concept, an embodiment of the present application provides a traffic extraction device for a vulnerability scanner. As shown in FIG3 , the traffic extraction device for the vulnerability scanner includes: a triggering module 1, a determining module 2, and a storage module 3. The triggering module 1 is used to trigger the vulnerability scanner to send request traffic; the determining module 2 is used to determine whether the request traffic is redundant traffic; redundant traffic includes: common traffic and/or duplicate traffic; duplicate traffic is traffic that is identical to previously received request traffic; common traffic is request traffic used by the vulnerability scanner to verify site survival; and the storage module 3 is used to store the request traffic if it is non-redundant traffic.

In some embodiments, the redundant traffic includes general traffic. Determination module 2 is configured to determine whether the request traffic is redundant traffic by: calculating the request traffic using a preset hash algorithm to obtain a first hash value corresponding to the request traffic; determining whether the first hash value is within a preset general set; and if so, determining that the request traffic corresponding to the first hash value is general traffic.

In some embodiments, the traffic extraction device of the vulnerability scanner further includes a universal set acquisition module. The universal set acquisition module is configured to acquire universal sets by: triggering the vulnerability scanner to send sample request traffic containing test objects; using a preset hash algorithm to calculate each sample request traffic and form a hash table; classifying each hash table according to the test object to form N hash sets, where the value of N is the same as the number of test object types; filtering out duplicate hash tables in each hash set, and forming a universal set based on the duplicate hash tables.

In some embodiments, the redundant traffic includes duplicate traffic. Determination module 2 is configured to determine whether the request traffic is redundant traffic by: replacing the path corresponding to the URI field in the request traffic and the Referer field in the request traffic with a preset path; calculating the request traffic after the path replacement using a preset hash algorithm to obtain a second hash value; checking whether the second hash value is recorded in a preset hash value record table; if not, determining that the request traffic is non-redundant traffic, adding the second hash value to the hash value record table, and storing the request traffic corresponding to the second hash value; if so, determining that the request traffic is duplicate traffic.

In some embodiments, the storage module 3 is used to store the request traffic in the following manner: obtaining the vulnerability name corresponding to the request traffic; marking the request traffic according to the vulnerability name, and storing the request traffic marked with the vulnerability name.

In the above embodiment, the storage module 3 is also used to send the request traffic marked with the vulnerability name to a preset message queue so that the message consumer can obtain the request traffic from the message queue and process it; and obtain and store the processing results of the request traffic.

In some embodiments, the traffic extraction device of the vulnerability scanner further includes a response module configured to parse a URI field in the request traffic after triggering the vulnerability scanner to send the request traffic, determine a response template based on the URI field, and send the response template to the vulnerability scanner.

It can be understood that the various embodiments described in the first embodiment are also applicable to the third embodiment if there is no conflict. For the sake of brevity, they will not be repeated here.

Example 4

As shown in Figure 4 , an embodiment of the present application provides an electronic device comprising a processor 4 and a memory 5. Optionally, the device may further comprise a communication interface 6 and a bus 7. The processor 4, the communication interface 6, and the memory 5 may communicate with each other via the bus 7. The communication interface 6 may be used for information transmission. The processor 4 may invoke logic instructions in the memory 5 to execute the vulnerability scanner traffic extraction method of the above embodiment.

In addition, the logic instructions in the memory 5 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

Memory 5, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions/modules corresponding to the methods in the embodiments of the present application. Processor 4 executes the program instructions/modules stored in memory 5 to perform functional applications and data processing, thereby implementing the traffic extraction method of the vulnerability scanner in the above-mentioned embodiments.

The memory 5 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal device. Furthermore, the memory 5 may include a high-speed random access memory and a non-volatile memory.

The electronic device may be a computer or a server.

An embodiment of the present application provides a storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the traffic extraction method of the vulnerability scanner.

An embodiment of the present application provides a computer program product, which includes a computer program stored on a storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the traffic extraction method of the vulnerability scanner.

The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

The technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc., which can store program code, or a transient storage medium.

In the embodiments provided herein, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is merely a logical functional division, and actual implementation may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented.

The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application. Furthermore, the above embodiments may be combined with each other to form new embodiments, unless they conflict.

Claims (10)

A method for extracting traffic from a vulnerability scanner, comprising: Trigger the vulnerability scanner to send request traffic; Determine whether the request traffic is redundant traffic; the redundant traffic includes: common traffic and/or repeated traffic; the repeated traffic is the same traffic as the previously received request traffic; the common traffic is the request traffic used by the vulnerability scanner to verify the survival of the site; If the requested traffic is non-redundant traffic, the requested traffic is stored. The method according to claim 1, wherein the redundant traffic includes general traffic; and determining whether the request traffic is redundant traffic comprises: Calculate the request traffic using a preset hash algorithm to obtain a first hash value corresponding to the request traffic; Determining whether the first hash value is in a preset universal set; If so, determine that the request traffic corresponding to the first hash value is general traffic. The method according to claim 2, wherein the general set is obtained by: Trigger the vulnerability scanner to send sample request traffic with the test object; Calculate the request traffic of each sample using a preset hash algorithm and form a hash table; Classifying each of the hash tables according to the test object to form N hash sets; the value of N is the same as the number of types of the test objects; Duplicate hash tables in each hash set are screened out, and a common set is formed based on the duplicate hash tables. The method according to claim 3, characterized in that the test object includes at least one of url, host and directory. The method according to claim 1, wherein the redundant traffic includes duplicate traffic; and determining whether the request traffic is redundant traffic comprises: Replace the paths corresponding to the URI field in the request traffic and the Referer field in the request traffic with the preset paths; Calculate the request traffic after the path is replaced using a preset hash algorithm to obtain a second hash value; Check whether the second hash value is recorded in a preset hash value record table; if not, determine that the request traffic is non-redundant traffic, add the second hash value to the hash value record table, and store the request traffic corresponding to the second hash value; If so, it is determined that the request traffic is duplicate traffic. The method according to any one of claims 1 to 5, wherein storing the request traffic comprises: Obtain the vulnerability name corresponding to the request traffic; The request traffic is marked according to the vulnerability name, and the request traffic marked with the vulnerability name is stored. The method according to claim 6, wherein storing the request traffic further comprises: Sending the request traffic marked with the vulnerability name to a preset message queue, so that a message consumer can obtain the request traffic from the message queue and process it; Obtain and store the processing result of the request traffic. The method according to any one of claims 1 to 5, characterized in that after triggering the vulnerability scanner to send request traffic, the method further comprises: Parsing the URI field in the request traffic; Determine a response template according to the URI field; Sending the response template to the vulnerability scanner. A traffic extraction device for a vulnerability scanner, characterized by comprising: Trigger module, used to trigger the vulnerability scanner to send request traffic; a determination module, configured to determine whether the request traffic is redundant traffic; the redundant traffic includes: common traffic and/or repeated traffic; the repeated traffic is traffic identical to the previously received request traffic; the common traffic is request traffic used by the vulnerability scanner to verify site survival; A storage module is used to store the requested traffic if the requested traffic is non-redundant traffic. An electronic device, characterized in that it includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, and the processor executing the computer-executable instructions to implement the traffic extraction method of the vulnerability scanner according to any one of claims 1 to 8.