CN116303443A - An Automatic Extraction Method of Network Protocol Design Knowledge Based on RFC Documents - Google Patents

Abstract

The invention discloses a method for automatically extracting network protocol design knowledge based on RFC documents. The method includes acquiring a chapter list of the RFC document and a row list in each chapter; acquiring the chapter list and the row list to generate a list of key lines; parse the structured description in the list of key lines to obtain a list of formatted fields containing partial information and/or a list of undefined names of automata; based on the structure The description of the chapter content corresponding to the key line belongs to, and supplements the formatted field list containing partial information and/or the undefined name list of the automaton; based on the formatted field list containing complete information and the automatic machine migration list to get the network protocol design knowledge extraction results. The present invention can automatically extract relevant network protocol design information.

Description

An Automatic Extraction Method of Network Protocol Design Knowledge Based on RFC Documents

technical field

The invention relates to the field of computer technology, in particular to a network protocol-oriented design comprehension technology, in particular to a method for automatically extracting network protocol design knowledge based on RFC documents.

Background technique

A network protocol is an "agreement" reached in advance between computers when they communicate through the network. As long as they follow the same protocol, they can communicate. As the infrastructure of the Internet, network protocols play a vital role, so the analysis of network protocols is particularly important. The analysis of network protocols is mainly divided into two types: dynamic testing and static analysis. These two types of methods require certain domain knowledge to guide the construction of protocol message structures. RFC (Request For Comments) documents are a series of serialized standards that record Internet norms, protocols, and processes. Basic Internet communication protocols are described in detail in the corresponding RFC documents. The analysis of network protocols requires relevant researchers to read a large number of RFC documents to obtain domain knowledge. The dynamic test method needs to obtain the structure and composition information of the protocol message, and the static detection method needs to obtain specific information such as the length and type of the protocol message field. . However, due to the long time span of the drafting process of RFC documents, many organizations participated in the writing, and there are many types of protocols included, resulting in non-standard descriptions of RFC documents and difficulties in manual reading and understanding, which brings great difficulties to the analysis of network protocols.

Invention content:

Aiming at the above problems, the present invention proposes a method for automatically extracting network protocol design knowledge based on RFC documents, which automatically extracts relevant network protocol design information through analysis and understanding of structured representations and natural language descriptions in RFC documents.

A method for automatically extracting network protocol design knowledge based on RFC documents, said method comprising:

Based on the txt format of the RFC document, obtain the chapter list of the RFC document and the row list in each chapter;

Obtaining the structured description in the chapter list and the line list to generate a key line list; wherein, the structured description includes: message structure and/or automaton;

Parse the structured description in the key line list to obtain a formatted field list containing partial information and/or an undefined name list of the automaton; wherein, the information in the formatted field list includes: chapter name, field name, field type, field length, field value and field description;

In the case where the structured description is a message structure, update the field length, field value and field type based on the chapter content of the structured description corresponding to the key line to obtain a formatted field list containing complete information;

In the case that the structured description is an automaton, obtain the chapter content to which the key row of the structured description belongs, and use the names in the undefined name list to match the chapter content, and the obtained automatic The machine migration information is stored in the automatic machine migration list;

Combining the formatted field list containing complete information and the importing automaton migration list to obtain a network protocol design knowledge extraction result.

Further, the message structure includes: one or more of a graphic format, a pseudo-C code format, and an ASN.1 format.

The acquisition of the structured description in the chapter list and the row list includes:

In the case where the message structure is in the form of graphics and text, use the regular expression r'^\+-+\+$' to match and identify the start line of the structured description, continue to traverse the line list, and when a single When the line is blank, the current line is used as the termination line;

When the message structure is in the form of pseudo-C code, use the regular expression r'^\s*(struct|enum|typedef struct)\s+\w*\s*\{' to match and identify the start line, Use the regular expression r'\}[^;]*;' to match and identify the termination line, and the curly braces between the start line and the termination line are correctly closed;

When the message structure is in the form of ASN.1, use regular expressions r'\w+\s+(OBJECT-GROUP|MODULE-COMPLIANCE|OBJECT IDENTIFIER|OBJECT-TYPE|SEQUENCE)' and r'\w+\ s+::=\s+\w+\s\{'After matching and identifying the start line, continue to traverse the line list. When the current line ends with a right curly brace and the next line is a blank line, the current line is used as the end line.

Further, the structured description in the key row list is parsed to obtain a formatted field list containing partial information, including:

For the message structure in the form of graphics and text, obtain the key line with the "|" symbol or ":" symbol, and use the corresponding "|" symbol or ":" symbol as a separator to cut the key line to obtain several fields and the field name of the field;

Based on the variable-length field symbol contained in the field and the space placeholder in the figure contained in the field name, after obtaining the field length of each field, store it in the formatted field list; wherein, if the field name is The "()" symbol contains field length information, then use the field length information to update the field length and store it in the formatted field list;

After the field name is formatted and updated, it is stored in the formatted field list; the format update includes: removing redundant blank characters in the field name and the characters contained in the "()" symbol and the "[]" symbol Additional information, and reverting the abbreviated symbols contained in the field names to full-word descriptions;

Obtain the chapter name corresponding to the key row and store it in the formatted field list;

and / or,

For the message structure in the form of pseudo-C code, based on the value of the first element after the current line in the key line list, the information type described by the current line is judged; the information type includes: structured information or enumerated information;

If the type of information described in the previous line is structured information, use the regular expression r'[\w\:]+\s+[\*\w\d\[\]]+;' to match the line describing the field information, The field length contained in the "<>" symbol, and the chapter name corresponding to the line describing the field information, respectively obtain the field name and field type, field length and chapter name, and store them in the formatted field list;

If the information type described in the previous line is enumerated information, use the regular expression r'[\w\d]+\([\w\d\.]+\)' to match the line describing the field information to get the field Value information, and by matching the field corresponding to the field value information, after obtaining the field name, field type and chapter name, store it in the formatted field list;

and / or,

For the message structure in ASN.1 form, based on the "SYNTAX" keyword in the key line list, the field type is obtained and stored in the formatted field list;

Based on the information of the first line in the key line list, determine whether the text description of the key line is nested information;

If it is nested information, based on the start line of the key line text starting with the current field name, obtain the field name, and describe the content of UP_FIELD in the form of "::={UP_FIELD}" in the end line of the key line, and get After the superior field name, store the field name and superior field name in the formatted field list;

If it is not nested information, use the information in the first line as the superior field name for subsequent parsing of each field, and when traversing each subsequent key line, use "FIELD_NAME FIELD_TYPE," to get the field name, and combine the field name and The superior field name is stored in the formatted field list;

Obtain the chapter name corresponding to the key row and store it in the formatted field list.

Further, the update field length includes:

Obtaining the chapter corresponding to the chapter name, and re-dividing the chapter content according to the sentences to obtain the updated chapter content;

Generate a matching field description pattern and field name according to the field name, match the updated chapter content, and use the current line and several lines after the current line as the field description after the match is successful;

Match the updated chapter content according to the field name, and if the match is successful, perform part-of-speech analysis on the sentence containing the field name to obtain a sentence containing verb + base;

The word corresponding to the base number is used as the value of the field length, and the unit of the value is obtained based on the last character of the base number;

The field length in the formatted field list is updated according to the value of the field length and the unit of the value.

Further, the update field value includes:

Obtaining the chapter corresponding to the chapter name, and re-dividing the chapter content according to the sentences to obtain the updated chapter content;

Match the updated chapter content according to the field name, and if the match is successful, determine whether there are modal verbs or comparative relative words in the sentence;

In the case of modal verbs or comparative relative words, the part-of-speech analysis is performed on the sentence containing the field name, and the sentence containing MUST (NOT) be (set to) NUM, less than NUM; greater than NUM or is equal to NUM, are not equal to NUM sentences;

Described modal verb or relatively relative word is converted into corresponding symbol, and extract MUST (NOT) be (set to) NUM, less than NUM; Greater than NUM or is equal to NUM, the value in are not equal to NUM;

Based on the symbol and the value, the values of the fields in the formatted field list are updated.

Further, the update field type includes:

Based on the field description information in the structured description including the field name, field type, and field length of the protocol message, deduce the field type and update the formatted field list;

or,

According to the field name and field length in the formatted field list, deduce the field type and update the formatted field list;

or,

According to the field names in the formatted field list, deduce the field type and update the formatted field list;

or,

Based on the field length in the formatted field list, deduce the field type and update the formatted field list.

Further, the automaton includes: one or more of text lists and point-and-line diagrams.

The acquiring the structured description in the chapter list and the line list to generate a key line list includes:

In the case where the automaton form is a literal list, if the current line matches the regular expression [r"State\s+Event\s+Action\s+New State",r"\|\sState\s\|"] , it is recognized as the start line, and the current behavior is a blank line in the subsequent traversal process, and the start line is not None, it is recognized as the end line;

In the case of a point-and-line graph in the form of the automaton, if the current line matches the regular expression [r'^[\|\+]+---->', r'<----[-\|\ +]+$'], it is recognized as the start line, and the current line is blank in the subsequent traversal process, and the start line is not None, it is recognized as the end line.

Further, the structured description in the key line list is parsed to obtain an undefined name list of the automaton, including:

Use spaces to cut key lines to get a list of words;

iterate over said list of words;

If it is possible to obtain all uppercase words or part of speech words that are nouns, then add the all uppercase words and the part of speech words that are nouns to the list of undefined names;

If it is not possible to obtain all capitalized words or words whose part of speech is a noun, use the regular expression [r'(Event\d+):(\w*)',r'(\w*)[sS]tate:' ,r'State\(s\):(\w+)'] matches the state name and event name of the automaton, and adds it to the list of undefined names.

Further, in the case that the structured description is an automaton, obtain the chapter content to which the key line of the structured description belongs, and use the names in the undefined name list to match the chapter content , store the obtained automaton migration information into the automaton migration list, including:

Obtain the content of the chapter to which the key row of the structured description belongs;

Re-divide the chapter content according to the sentence to get the updated chapter content;

If there is a sentence containing a keyword representing a state transition in the updated chapter content, and there is a name in the undefined name list in the first half and the second half of the keyword, then it is determined that the sentence contains a sentence that changes the state. Automata migration information, and after the segmentation of the automaton migration information quaternion, the sentence is stored in the automaton migration list; wherein, the automaton migration information quaternion includes: original state, event, action and purpose state ;

If there is a sentence containing a keyword representing a state transition in the updated chapter content, and there is a name in the undefined name list in the first half or the second half of the keyword, then it is determined that the sentence contains a sentence that does not change the state. The automaton transfer information, and after the sentence is divided into quadruples of the automaton transfer information, it is stored in the automaton transfer list.

Further, the method also includes:

Based on the network protocol design knowledge extraction result, a detection rule and/or fuzzing configuration file is generated.

Compared with the prior art, the network protocol design knowledge automatic extraction method of the present invention, its core is to extract the representative message field name, automatic On this basis, combined with pattern matching and natural language extraction technology, further richer and more comprehensive design knowledge is extracted from the above structure and other texts described in natural language in the document, and finally forms a protocol message The complete design knowledge including field name, field type, field length and automaton information is used to guide the rule generation in the static detection process, the input generation in the fuzzy test process, etc., and provide a basis for code defect detection and vulnerability mining of protocol software. Domain knowledge support improves detection and mining performance.

Description of drawings

Figure 1 is a flow chart of the method for automatically extracting network protocol design knowledge based on RFC documents.

Figure 2 An example diagram of graphic description.

Figure 3 Pseudo-C code description example diagram.

Figure 4 ASN.1 format description example diagram.

Figure 5 is an example diagram of the text list description form.

Figure 6. An example diagram of the description form of the point-line graph automaton.

Figure 7 Fuzzing configuration file representation.

Figure 8 is an example diagram of an automaton test sequence.

Figure 9 RFC 4271 first structured description.

Figure 10 Example of fuzzing configuration file generation.

Detailed ways

In order to make the above features and advantages of the present invention more comprehensible, the technical solutions of the present invention will be further described below through specific examples.

The implementation steps of the present invention are mainly divided into five stages: a preprocessing stage, a structured description recognition stage, a key information extraction stage, a text understanding stage, and a knowledge expression stage.

Preprocessing stage:

1. RFC document preprocessing

The first is to obtain the txt format of the RFC document from the RFC-editor website (RFC-editor is an officially maintained RFC document editing website), and then traverse the document to remove the header and footer of the RFC document. Generally speaking, the RFC document There is a special byte "0x0c" in the lines near the header and footer, which can be matched according to this feature; then through multiple regular expressions (eg: r'^[0-9.]{1,8} .?+') matches the title line in the remaining RFC document, cuts the RFC document into multiple chapters, and obtains a chapter list. Each element in the chapter list is a chapter, and a chapter is a line list. Each element in the line list element is a raw line in the document.

Structured description recognition stage:

Structured description methods can be roughly divided into two categories, namely, message structure and automaton. Among them, message structure can be divided into three forms: graphic, pseudo-C code, and ASN.1. Automaton is divided into text list , point-line graph these two forms.

2. Identify the structural description of the message structure

Traverse the list of chapters obtained in step 1 and the list of lines in the chapters, try to match and recognize all forms of structured description text, recognize according to the characteristics of the summary, and only recognize the first example, if the recognition is successful, set the structured description method of the current RFC document , the recognition failure indicates that the current RFC document has no content to be extracted.

For different description styles of message structures, the specific identification steps are as follows:

a. Graphic and text form: for the chart (as shown in Figure 2) that describes the composition of message fields composed of "+-" symbols, use the regular expression r'^\+-+\+$' to match and identify the starting line, Record the index (index) of the starting line in the current chapter, continue to traverse the list of lines in the chapter, and when encountering a single blank line (that is, only containing a newline character '\n'), record the current behavior termination line . Set the structural description mode of the message structure of the current document to graphic form.

b. Pseudo-C code form: for the key line text (as shown in Figure 3) that describes the composition information of the message field in the form of pseudo-C code, the regular expression r'^\s*(struct|enum|typedef struct)\ s+\w*\s*\{' match to identify the start line, use the regular expression r'\}[^;]*;' match to identify the end line. At the same time, it is necessary to consider whether the "{}" in the key line is closed correctly. Here, the idea of "stack" is used to deal with this situation. Set the structural description method of the message structure of the current document to pseudo-C code form;

c.ASN.1 format: For the key line text that describes the message field composition information in ASN.1 format (as shown in Figure 4), use the regular expression r'\w+\s+(OBJECT-GROUP|MODULE-COMPLIANCE|OBJECT

IDENTIFIER|OBJECT-TYPE|SEQUENCE)' and r'\w+\s+::＝\s+\w+\s\{' match and identify the start line, if the current line ends with a right curly brace "}" and the next line is A blank line is recognized as a terminating line. Set the structured description mode of the packet structure of the current document to ASN.1 format.

3. Identify the structured description of automaton information

For different description styles of automaton information, the specific steps of identification are as follows:

a. Text list form: For the case of text list form (Example Figure 5), if the current line matches the regular expression [r"State\s+Event\s+Action\s+New State",r"\|\ sState\s\|"], it will be recognized as the start line described by the automaton, continue to traverse the next line of the current chapter, if it encounters a blank line in the current line, and the start line is not None, it will be recognized as the end line . Set the structured description method of the automaton of the current document to the form of a text list;

b. Point-line graph form: For the case of point-line graph form (Example Figure 6), if the current line matches the regular expression [r'^[\|\+]+---->',r'<- ---[-\|\+]+$'], it is recognized as the starting line in the description form of the automaton point-line diagram, and continues to traverse the next line of the current chapter. When encountering a blank line in the current line, start If the start line is not None, it is recognized as the end line. Sets the structured description of the automaton of the current document in the form of a dot-line diagram.

4. Extract structured description

Traverse the list of chapters and the list of lines in the chapters obtained in step 1 from the beginning, use the characteristics of the structured description method of the document identified in step 2, match and identify the start line and end line of the structured description, and record the location of the structured description Chapters, the key line description information is added to the corresponding key line list in turn, and finally a key line list information with different description styles is obtained.

Key information extraction stage:

5. Extract message structure information

Traverse the list of key lines of the message structure, analyze and obtain the corresponding network protocol design information according to different description styles, and store the field description information including the field name, field type, and field length of the protocol message in a standardized format in the format Among the fields, a field is stored in the form of <chapter name, field name, field type, field length, field value, field description> six-tuple, and the derivation of some field types, field lengths and field descriptions will be described in the next stage . According to the structured description method of the RFC document obtained in step 2, the key line text information is analyzed, and the specific steps are as follows:

a. Graphic and text form: Traverse the list of key lines obtained in step 3, and parse the content of each line. Parsing is skipped if the current line starts with the sign "+-" or is an empty line, as this is usually a separator line. If there is a "|" or ":" symbol in the current line, use the symbol as a separator to cut the current line to obtain multiple fields described by the current line. Then traverse the multiple fields obtained, first judge whether it is a variable-length field, that is, judge whether there are symbols such as "variable, /, ~" in the current line, if so, set the size of the current field to "-1", representing the The length of the field is uncertain; otherwise, it is a fixed-length field. These field names include their space placeholders in the figure. The actual length of the field can be calculated according to its length, mainly based on "size=(len(field )+3)//BIT_SBL_SIZE", where BIT_SBL_SIZE is the character length in the current image description style with "+-" symbol representing 1bit length (for example: "+-" represents 1bit length, then BIT_SBL_SIZE is 2). Then continue to further confirm the field name, remove the extra blank characters in the field name, and remove the extra information contained in "(), []" in the field name. If there is information describing the length of the field in "()" (for example: 2octets , 4bits), then it will be identified and calculated and the size of the current field will be updated, and the size of the current field will be set as unchangeable (set a flag). Abbreviations that exist in field names are then processed to restore them to full-word descriptions. Finally, the field name of the field is updated, and the chapter where the current key row is located is written into the above six-tuple as the chapter name of the field to obtain a preliminary field, which is added to the formatted field list.

By extracting structured information in the form of graphics and text, the information of "chapter name", "field name" and "field length" can currently be obtained.

b. Pseudo-C code form: traverse the list of key lines obtained in step 3, and parse the content of each line. First, use the information in the first line to judge whether the key line text describes structured information or enumerated information, because structured information contains field information, while enumerated information contains field value information, use spaces to cut the current The value of the first element after the line can be judged. If it describes structured information, traverse each line and use the regular expression r'[\w\:]+\s+[\*\w\d\[\]]+;' to match the line describing the field information, Generally, it has such a feature, the form of "TYPE FIELD_NAME;", so that the type and field_name of the field can be identified. In addition, there may be a situation where the "<>" symbol is used to describe the length of the field, and the size of the field is also obtained at this time. If it describes the enumeration information, use the regular expression r'[\w\d]+\([\w\d\.]+\)' to match the described enumeration information, which is characterized by "ENUM_NAME(VALUE) ", you can also extract the values in it, and then match the fields corresponding to these enumeration values at the end of the key line (for example: "}HandshakeType;"). Finally, set the corresponding field name, field value, and field type, and add this field to the list of formatted fields.

By extracting the structured information in the form of pseudo-C code, the information of "chapter name", "field name", "field length" and "field type" can be obtained at present.

c.ASN.1 format: traverse the list of key lines obtained in step 3, and parse the content of each line. First, use the information in the first line to determine whether the text description of the key line is nested information. If there is an ASN.1 keyword among "OBJECT-GROUP, MODULE-COMPLIANCE, OBJECT IDENTIFIER, OBJECT-TYPE", then Descriptions are nested. If it is nested, it describes a relationship between two fields, and the starting line of the key line text starts with the current field name, and the field name can be obtained through this feature. Traverse each line, if there is a "SYNTAX" keyword, the current line describes the type of the current field, which is easy to extract, and finally there is a description in the form of "::={UP_FIELD}" in the end line of the key line, and the content of UP_FIELD is extracted as The parent field of the current field, set it to the section_name of the current field, and add the current field to the list of formatted fields. If not nested, it describes a field and the field information it contains. Take the information of the starting line as the upper-level field for subsequent parsing of each field, that is, set it as the section_name of each subsequent field, and then traverse each line. The characteristics of the description field are such "FIELD_NAME FIELD_TYPE," which can be obtained by simple separation According to the corresponding information, a field is obtained, and the field is added to the list of formatted fields.

By extracting the structured information in the form of ASN.1, the information of "chapter name", "field name" and "field type" can be obtained at present.

6. Extract automaton information

If the automaton key line list is obtained in step 3, traverse the obtained automaton key line list and parse the content of each line. Identify some names that may be events, states, and actions according to some characteristics in the automaton description, and add them to the undefined names of the automaton. Specifically, use spaces to cut the current line to get a word list, and then traverse the word list. If the word is all capital letters, it may be a state name. If the current word is a noun, it may be an event or action name. , adding these words to the undefined name of the automaton.

If the automaton keyline list is empty, use the regular expression [r'(Event\d+):(\w*)',r'(\w*)[sS]tate:',r'State\(s \):(\w+)'] matches the state name and event name of the automaton, and adds it to the undefined name of the automaton.

Text understanding stage:

Through steps 5 and 6, a series of formatted field lists and undefined name lists of the automata are obtained, but the six-tuple information of each field is not perfect, so this step is to further improve the corresponding fields and the automaton The process of migration information derivation. The input of the present invention at this stage is a list of formatted fields and the contents of the chapters to which the fields belong, and the output is a list of completed formatted fields or a list of automaton migration. For broader matching, section content can be expanded into larger sections or sections of the entire RFC document.

7. Use text description information to derive field length

Traversing the formatted field list obtained in step 5, for the chapters to which each field belongs, the present invention first re-divides the chapters according to sentences. Replace all line breaks with spaces in all the lines in the chapter, and combine the contents of the chapter into complete statement information. Then, use natural language processing tools to split it into individual sentences and store them in chapters.

Traverse each field in the formatted field list, using the field name plus "is", ":\n", "" as a pattern to match the field description, and traverse each line in the entered chapter. If the above pattern exists in the current line or the current line is the current field name, the match is successful. Then use the text content of the current line and the next 5 lines as the text description information of the field and add it to the formatted field list. If there are redundant incomplete sentences, discard them. Field description information is only a part to assist in deriving other field information.

Next, iterate over each sentence in the section where the field is located and in the description of the field. If the sentence contains a field name, use that as the initial parsing location. The length of the field in the statement usually appears in the form of is/are/length of/length/oflength+num. The invention takes the sentences containing these keywords as input, uses NLP technology to analyze the part of speech of the sentences, and recognizes the sentences containing "VBZ CD" (verb+radix) therein. The word corresponding to CD is the length of the packet. At the same time, in order to determine the unit of the extracted length, the present invention judges whether its next character is ['-octet', '-bit', 'octet', 'bit', 'octets', 'bits', ' bytes'] to confirm its unit. The present invention stores the message field length in units of bits, and for octet and byte, the present invention stores the value *8. If there is no unit behind the value, the present invention defaults that it is described in bytes. Therefore, the value *8 will also be stored. In this way, the present invention obtains the length corresponding to the field, and then updates the corresponding value in the formatted field list.

8. Deduce field value

When RFC describes field values in text, it generally uses modal verbs ("MUST", "MUST NOT") or comparative relative words ("is less than", "is equal to"), etc. to describe. Therefore, the present invention traverses each sentence in the chapter, and if the sentence contains a field name, and contains a modal verb or a comparative relative word, it is used as a sentence to be extracted. Next, the present invention utilizes part-of-speech analysis to generate a part-of-speech list of sentences. If it includes the part-of-speech combination as shown in Table 1, then the present invention correspondingly transforms the modal verbs and comparative relative words into corresponding symbols, and then extracts the values in the CD. Finally, a list of values is generated based on the symbol. For example: field<=3 then, values=[0,1,2,3].

Table 1 NLP part-of-speech analysis list

9. Deduce field type

There are three ways to infer field types. One is to use the text description information obtained in step 5 to simply deduce the field type of the current field. If obvious field types such as "octet", "string", "boolean", and "integer" exist in the text description, set the field type to for that type. In addition, after obtaining the field name and field length information relatively well, the type of the current field can be deduced based on these two pieces of information. Therefore, the second is to infer the field type according to the field name. If there are "domain name" and "owner name" in the field name, you can set the field type to "dnsname". If there are "time, ttl" in the field name, "ip , ipv6", you can set the field type to "timestamp", "ip" respectively; the third is to infer the field type according to the field length, if the field length is "8, 16, 32, 64", you can set the field type respectively For "byte, word, dword, qword". The priority of the methods for inferring the field type above is from 1 to 3 in descending order, and the method for inferring the field type through the field length has the lowest priority.

10. Deriving automaton migration information

Through step 5, a series of automaton undefined names are obtained, and these names are used as keywords to match possible automaton migration description sentences in the paragraph description information to identify a piece of automaton migration information (that is, original state, event , action, destination state), and store the migration information in the automaton migration list.

Traverse the list of chapters obtained in step 1, combine each line in each chapter, and use NLTK (a third-party library, natural language toolkit) to segment the paragraph. Then match the obtained normal natural sentences one by one. If there are keywords in the sentence that may indicate state transition such as {"receive", "transfer", "change", "remain"}, further judge in the first half of the keyword Whether there is an automaton undefined name in both the part and the second half, if yes, it is an automaton transition information that changes the state, and if there is only one, it may be an automaton transition information that does not change the state. Then further cut the statement and store each part in the automaton migration information quadruple according to the English grammar, and then add it to the automaton migration list.

Knowledge expression stage:

The formatted field list and automaton migration list obtained through steps 7, 8, 9, and 10 are the final network protocol design information. The formatted field list can be converted into JSON format and stored in a file for subsequent use. Subsequent use scenarios can currently be divided into detection rule generation and fuzz test configuration file generation, specifically as follows:

11. Detection rule generation

Using the field information (field name, field length, field value) in the formatted field list, a series of detection rules in the form of "chk_bf(cond, op)" can be generated, where "cond" represents the condition of the rule, generally Corresponding to the value range or length limit of the field, if the condition is met, the correct operation is defined as "op". For example, "((hold>3&&hold==0), use(hold))" means that between using the hold field, the present invention needs to judge whether its value is 0 or greater than or equal to 3.

12. Fuzz test configuration file generation

Using the information of the fields in the formatted field list (field name, field length, field type, field value), the present invention can automatically generate the test configuration file of BooFuzz (a generalized network protocol fuzzy testing framework) for fuzz testing . According to the field length or field type, determine the mutation primitive to be selected for the current field. For example, s_bytes() represents a field with multiple byte sizes. As shown in Figure 7, the first field is "Marker", and the default value length is not enough "\x00 padding". s_byte() represents a byte-sized field. The field "Optional Parameters Length" in Figure 7 is 1 byte in size. The value of the field is represented by the byte string of its "value" parameter, and the value of the fuzz test is represented by "fuzz_values" parameter representation.

Using the automaton transition information (original state, destination state), the automaton sequence for fuzz testing can be generated. As shown in Figure 8, using "State Machine Record(state='Connect', event='['CollisionDetectEstablishedState',...,'Event 12', 'DelayOpenTimer_Expires']', action='None', new_state='OpenSent' )" This automaton transfers information to obtain the test sequence of the automaton.

Example:

An embodiment is introduced below, taking the extraction of network protocol design knowledge in the BGP protocol RFC 4271 as an example.

In step 1, RFC 4271 is preprocessed to remove unimportant information such as redundant headers and footers, and the preprocessed file "rfc4271.txt.preprocessed" is obtained, and then the chapters of the preprocessed file are matched and divided Sections, get the list of chapters "sections".

In step 2, the chapter list "sections" obtained in step 1 is traversed, and the message structure description in graphic and text form as shown in Figure 9 is recognized in the chapter "4.1.Message Header Format" of RFC4271, so the structure of the RFC4271 message structure Set the description mode to graphic and text form, and end the traversal.

In step 3, if the structured description mode of the automaton information is not recognized, leave it blank.

In step 4, according to the graphic description form obtained in step 2, proceed to step 3.a to extract the structured description in graphic and text form, identify the start line and end line of each structured description, and record the chapter it is in , added to the key line list of the message structure.

In step 5, first select step 5.a according to the RFC structured description method to extract key information from the key line list of the message structure described in graphic form, traverse the key line list, parse the content of each line, and obtain field information in turn, Stored in the list of formatted fields.

In step 6, since the structured description of the automaton information is empty, the more general regular expression [r'(Event\d+):(\w*)',r' is directly used in the process of traversing the key lines (\w*)[sS]tate:',r'State\(s\):(\w+)'] matches the state name and event name of the automaton and adds it to the undefined name of the automaton.

In step 7, the formatted field list obtained in step 4 is traversed, and the field length is further inferred and understood according to the text description in the field information and the chapter where it is located. The following example is the text description and the corresponding field part displayed during the inference process: "[+]Field:Error Code Description:Error Code:This 1-octetunsigned integer indicates the type of NOTIFICATION."

In step 8, the value of the field is inferred, and the numerical value in the text description is extracted by using the part-of-speech analysis of NLP as the candidate value of the field value.

In step 9, the type of the formatted field is inferred according to the priorities of the three inference methods, and the type of the field is updated. For example, "[+]Current fieldname:Withdrawn Routes,field size:-1\n[+]Field:Withdrawn Routes Description:Withdrawn Routes:This is a variable-length fieldthat contains a list of IP address prefixes for the routes that are If there are keywords such as "IP address" in the text description of the field "Withdrawn Routes" in beingwithdrawn from service.", then this field is considered to describe the IP address, and its field type is set to "ip" and its length is " -1", it means that there are multiple IP addresses.

In step 10, use the undefined name of the automaton obtained in step 6 to traverse the chapters, find the description about the automaton, identify the automaton migration information, and add it to the automaton migration list. The following is an example of adding automaton transition information to the automaton transition list: "[+]Added unchanged state machine transition:StateMachine Record(state='Established', event='['CollisionDetectEstablishedState']', action='None' ,new_state='Established')".

In steps 11 and 12, use the formatted field lists obtained in steps 7, 8, and 9 and the automaton migration list obtained in step 10 to generate corresponding JSON format files, and then make detection rules as needed according to the designed application method Generation and generation of fuzzing configuration files. Figure 10 is an example of a fuzzing configuration file according to RFC 4271:

Similarly, the automaton test sequence shown in Figure 10 can also be generated.

Although the present invention has been disclosed as above with the embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent replacements to the technical solutions of the present invention by those of ordinary skill in the art shall fall within the protection scope of the present invention. The scope of protection of the invention is defined by the claims.

Claims (10)

1. An automatic network protocol design knowledge extraction method based on RFC documents is characterized by comprising the following steps:

based on a txt format of an RFC document, acquiring a chapter list of the RFC document and a row list in each chapter;

obtaining structured descriptions in the chapter list and the line list to generate a key line list; wherein the structured description comprises: message structure and/or automaton;

Analyzing the structured description in the key row list to obtain a formatted field list containing partial information and/or an undefined name list of the automaton; wherein the information of the formatted field list includes: chapter name, field type, field length, field value and field description;

under the condition that the structural description is a message structure, updating the field length, the field value and the field type based on the chapter content of the corresponding key row of the structural description so as to obtain a formatted field list containing complete information;

under the condition that the structured description is an automaton, acquiring chapter contents to which a corresponding key row of the structured description belongs, matching the chapter contents by using names in the undefined name list, and storing the acquired automaton migration information into an automaton migration list;

and integrating the formatted field list containing the complete information and the automatic entry migration list to obtain a network protocol design knowledge extraction result.

2. The method of claim 1, wherein the message structure comprises: one or more of a teletext form, a pseudo-C code form, and an asn.1 form.

The obtaining the structured descriptions in the chapter list and the row list includes:

when the message structure is in an image-text form, after the initial line of the structural description is matched and identified by using a regular expression r '- + $';

when the message structure is in a pseudo-C code form, a regular expression r ' +\s is used (struct|enum|typef struct) \s + \w \s \{ ' to match and identify a starting line, and the regular expression r ' \ [); ] x; ' the match identifies the ending line and the curly brackets between the starting line and ending line are correctly closed;

when the message structure is in an ASN.1 form, a regular expression r '\w + \s+ (OBJECT-group|module-complete| OBJECT IDENTIFIER |object-type|sequence)' and r '\w + \s + \w + \s {' are used for matching and identifying the initial line, the line table is continuously traversed, and when the current line ends with a right-hand bracket and the next line is blank, the current line is taken as the termination line.

3. The method of claim 2, wherein parsing the structured description in the critical row list to obtain a formatted field list containing partial information comprises:

Aiming at a message structure in an image-text form, obtaining a symbol with ' sign ' or ' sign: "key row of symbols, and use the corresponding" | "symbol or": the symbol is used as a separator to cut the key line so as to obtain a plurality of fields and field names of the fields;

based on the variable length field symbol contained in the field and the space placeholder of the field contained in the field name in the diagram, obtaining the field length of each field, and storing the field length into a formatted field list; if the "()" symbol in the field name contains field length information, updating the field length by using the field length information, and storing the updated field length into a formatted field list;

after the field names are formatted and updated, storing the field names into a formatted field list; the formatting update includes: removing redundant blank characters in the field names, adding extra information contained in "()" symbols and "[ ]" symbols, and restoring abbreviations contained in the field names into full word descriptions;

acquiring a chapter name corresponding to the key row and storing the chapter name into a formatting field list;

and/or the number of the groups of groups,

aiming at a message structure in a pseudo-C code form, judging the information type described by the current line based on the value of the first element after the current line in the key line list; the information types include: structured information or enumeration information;

If the information type of the forward description is structured information, the information type is expressed by a regular expression r' [ \w\s+ [ \x\w\d [ \ ] ] +; the method comprises the steps of respectively acquiring a field name, a field type, a field length and a chapter name, and storing the field name, the field type, the field length and the chapter name into a formatted field list after' matching the field length condition contained in a description field information, "< >" symbol and the chapter name corresponding to the description field information;

if the information type of the forward description is enumeration information, using a regular expression r '[ \w\d ] + \ ([ \w\d. ] + \)' to match the line of the description field information so as to obtain field value information, and storing a formatted field list after obtaining a field name, a field type and a chapter name by matching the field corresponding to the field value information;

and/or the number of the groups of groups,

aiming at the message structure in the ASN.1 form, obtaining a field type based on a SYNTAX keyword in a key row list, and storing the field type into a formatted field list;

judging whether text description of the key line is nested information or not based on information of a first line in the key line list;

if the information is nested information, the starting line based on the key line text starts with the current field name, the field name is obtained, and the existence of the line is terminated according to the key line ": : after the upper FIELD name is obtained from the content of up_field in the = { up_field } "form description, the FIELD name and the upper FIELD name are stored into a formatted FIELD list;

If the information is not nested information, the information of the first row is used as the next upper-level FIELD name of each FIELD, and the FIELD name is obtained by using 'FIELD_ NAME FIELD _TYPE' when each subsequent key row is traversed, and the FIELD name and the upper-level FIELD name are stored in a formatted FIELD list;

and acquiring the chapter names corresponding to the key rows and storing the chapter names into a formatting field list.

4. The method of claim 2, wherein updating the field length comprises:

acquiring chapters corresponding to the chapter names, and re-dividing the chapter contents according to sentences to obtain updated chapter contents;

generating a mode for matching field description and a field name according to the field name, matching updated chapter contents, and taking the current line and a plurality of lines behind the current line as field descriptions after successful matching;

matching the updated chapter content according to the field names, and performing part-of-speech analysis on sentences containing the field names under the condition that the matching is successful to obtain sentences containing verbs and cardinalities;

taking the word corresponding to the base as the numerical value of the field length, and obtaining a unit of the numerical value based on the next character of the base;

And updating the field length in the formatted field list according to the value of the field length and the unit of the value.

5. The method of claim 2, wherein the updating the field value comprises:

acquiring chapters corresponding to the chapter names, and re-dividing the chapter contents according to sentences to obtain updated chapter contents;

matching the updated chapter content according to the field names, and judging whether a stateful verb or a comparative relationship word exists in the sentence under the condition of successful matching;

in the case of the presence of a stateful verb or a comparison relation word, performing part-of-speech analysis on a sentence containing a field name to obtain a sentence containing MUST (NOT) be (set to) NUM and less than NUM; sentences of greaterthan NUM or is equivalent to NUM, are not equal to NUM;

converting the morbid verbs or the comparison relation words into corresponding symbols, and extracting MUST (NOT) be (set to) NUM and less than NUM; values in greaterthan NUM or is equivalent to NUM, are not equal to NUM;

and updating the field value in the formatted field list based on the symbol and the value.

6. The method of claim 2, wherein the updating the field type comprises:

Deducing the field type and updating a formatted field list based on field description information comprising the field name, the field type and the field length of the protocol message in the structural description;

or alternatively, the first and second heat exchangers may be,

deducing a field type and updating the formatted field list according to the field name and the field length in the formatted field list;

or alternatively, the first and second heat exchangers may be,

deducing the field type and updating the formatted field list according to the field names in the formatted field list;

or alternatively, the first and second heat exchangers may be,

the field type is derived and the formatted field list is updated based on the field length in the formatted field list.

7. The method of claim 1, wherein the automaton comprises: one or more of a text list and a dot line graph.

The obtaining the structured descriptions in the chapter list and the row list to generate a key row list includes:

if the automaton form is a text list, if the current line is matched with one of the regular expressions [ r 'State\s+event\s+action\s+New State', r '\s State\s\I' ], identifying the current line as a starting line, identifying the current line as a blank line in the subsequent traversal process, and identifying the starting line as a terminating line if the starting line is not None;

in the case of the automaton form being a dotted line graph, if the current line matches one of the regular expressions [ r ' < - [/i + ] + - - - >, r ' < - - - [ -/i + ] + $ ' ], then the start line is identified, and in the subsequent traversal process the current line is empty, and the start line is not None, then the end line is identified.

8. The method of claim 7, wherein parsing the structured descriptions in the list of critical rows to obtain a list of undefined names for an automaton comprises:

cutting the key rows by using the spaces to obtain a word list;

traversing the word list;

if it is possible to obtain all capitalized words or words whose part of speech is a noun, adding the all capitalized words and the words whose part of speech is a noun to an undefined name list;

if a fully capitalized word or a word whose part of speech is a noun cannot be obtained, the regular expression [ r '(event\d+): (\w)', r '(\w) [ sS ] State:', r 'state\s (s\w+)' ] is used to match the State name and Event name of the automaton, and is added to the undefined name list.

9. The method of claim 8, wherein, in the case that the structured description is an automaton, acquiring chapter contents to which the corresponding key row of the structured description belongs, and matching the chapter contents with names in the undefined name list, and storing the obtained automaton migration information into an automaton migration list, including:

acquiring chapter contents to which the corresponding key rows of the structured description belong;

Dividing the chapter content again according to sentences to obtain updated chapter content;

if a sentence containing a keyword representing state transition exists in updated chapter content, and the first half part and the second half part of the keyword respectively have a name in the undefined name list, judging that the sentence contains automaton transition information for changing the state, and storing the sentence in the automaton transition list after dividing the sentence into four groups of automaton transition information; the automaton migration information quadruple comprises: original state, event, action, and destination state;

if a sentence containing a keyword representing state transition exists in the updated chapter content, and a first half part or a second half part of the keyword contains a name in the undefined name list, judging that the sentence contains automaton transition information without changing the state, and storing the sentence in the automaton transition list after dividing the sentence into four groups of automaton transition information.

10. The method of any one of claims 1-9, further comprising:

and generating detection rules and/or fuzzy test configuration files based on the network protocol design knowledge extraction result.

Images