CN115174711A - Data processing method, device and medium based on full-stack programmable network - Google Patents

Abstract

This application proposes a data processing method, device and medium based on a full-stack programmable network. In the application scenario of data packet and packet processing, using the characteristics of a software-defined network such as a full-stack programmable network, the The software function modules of each data processing device located on the data plane of the network and the design of the external interface make the data processing device fully programmable and reconfigurable, and can customize the data packet processing logic according to data processing requirements , to achieve rapid deployment of application functions at different business layers. Therefore, the data processing device parses the received data packet and obtains the header information of the data packet, which can be field matched with the predefined matching action table, and the data packet is executed through the pipeline component corresponding to the matched processing action. The processing action meets various processing requirements for data packets, and enriches the processing methods of data packets compared with forwarding network devices such as traditional switches.

Description

Data processing method, device and medium based on full-stack programmable network

technical field

The present application relates to the field of computer network applications, in particular to a data processing method, device and medium based on a full-stack programmable network.

Background technique

With the continuous expansion of the network scale, closed network devices have built-in too many complex protocols, which makes it difficult for operators to customize and optimize the network, and developers cannot deploy new protocols on a large scale in the real environment. At the same time, as users' demands for network traffic continue to expand, various new services continue to emerge, increasing network operation and maintenance costs and management difficulties.

SUMMARY OF THE INVENTION

To achieve the above purpose, the embodiments of the present application provide the following technical solutions:

This application proposes a data processing method based on a full-stack programmable network, the method comprising:

Analyze the received data packets to obtain the corresponding data packet header information;

According to the matching result between the header information of the data packet and the predefined matching action table, the matched processing actions are performed on the data packet through different pipeline components; the different pipeline components include storage pipeline components and/or Computation pipeline components, and forwarding pipeline components;

The data packets processed by the forwarding pipeline component are reorganized and forwarded;

Wherein, the storage pipeline component is used to realize the storage of the data content of the data packet; the computing pipeline component is used to provide required computing services to the applications supported by the network device.

Optionally, according to the matching result between the header information of the data packet and a predefined matching action table, perform the matched processing actions on the data packet through different pipeline components, including:

Field matching is performed between the packet header information and a predefined matching action table;

It is determined that at least one first field of the data packet header information is successfully matched, and the matched processing action is performed on the data corresponding to the first field in the data packet through the pipeline component corresponding to the first field;

It is determined that at least one second field of the header information of the data packet fails to match, and a predefined default action is performed on the data corresponding to the second field in the data packet.

Optionally, performing the matched processing action on the data packet through the corresponding pipeline component includes:

calling the storage pipeline component to store the data content of the data packet; the data content includes metadata in the data packet, intermediate data and/or processed data generated during the processing of the data packet package; and/or,

The computing pipeline component is invoked, the computing capacity of the network device is configured through the virtual machine, and the virtual machine copy corresponding to each application supported by the network device is determined, so as to realize the computing service of the corresponding application through the virtual machine copy.

Optionally, the storing the data content of the data packet includes:

According to the context information of the data packet, continuously store the data content of the data packet that meets the storage conditions;

Wherein, the data content of the data package that meets the storage conditions includes data content belonging to a data type whose requested access times reach a request threshold, and/or data content whose correlation with the data content requested for access last time reaches a relevant threshold.

Optionally, the method further includes:

Regularly update and detect the stored data content;

Delete data content that has not been accessed for a preset period of time stored in accordance with the regularly updated detection results; and/or,

When it is detected that the remaining amount of storage space is less than the preset capacity threshold, when the data content to be stored is obtained, according to the storage time stamp of the stored data content, the pending data packets that have not been requested to be accessed for the longest time are determined;

The to-be-processed data content is replaced by the to-be-stored data content for storage.

Optionally, in the process of invoking the computing pipeline component, the method further includes:

Periodically update or delete the virtual machine copy;

Obtain the computing feature information of the data processing device;

The computing feature information is reported to the controller of the control plane, so that the controller implements unified management and scheduling of a plurality of the data processing devices according to the computing feature information.

Optionally, the received data packets are parsed to obtain corresponding data packet header information,

include:

Get custom packet parsing logic on the control plane;

According to the data packet analysis logic, the received corresponding data packet is analyzed to obtain the corresponding data packet header information;

Performing the matched processing action on the data packet through different pipeline components includes:

obtain data processing logic customized on the control plane suitable for multi-core processors;

Through different pipeline components, according to the corresponding data processing logic, the matched processing actions are performed on the input data packets.

The present application also proposes a data processing device based on a full-stack programmable network, including: a parser, a plurality of data processing devices, and an inverse parser, wherein:

The parser is used to parse each received data packet to obtain corresponding data packet header information;

The data processing apparatus includes a plurality of predefined pipeline components, configured to perform the matched processing on the data packet through the different pipeline components according to the matching result between the data packet header information and the predefined matching action table Action; wherein, the different pipeline components include a storage pipeline component and/or a computing pipeline component, and a forwarding pipeline component; the storage pipeline component is used to realize the storage of the data content of the data packet; the computing pipeline component Used to provide required computing services for applications supported by network equipment;

The inverse parser is configured to reassemble and forward the data packets processed by the forwarding pipeline component.

Optionally, the data processing device further includes:

Update the timer, which is used to record the length of time that any stored data packet is not requested to be accessed, so that when the recorded time length reaches the preset time length, the data packet is not requested to be accessed, and the data packet is deleted; When the duration does not reach the preset duration, the data packet is requested to be accessed, and the record time of the update timer is refreshed;

Wherein, the update timer is configured and generated by invoking an application programming interface of the data processing device.

The present application also provides a computer-readable storage medium on which multiple computer instructions are stored, and the multiple computer instructions are loaded and executed by a processor to implement the above-mentioned data processing method based on a full-stack programmable network.

Based on the above technical solutions, the present application proposes a data processing method, device and medium based on a full-stack programmable network. In the application scenario of data packet message processing, compared with traditional forwarding network devices such as switches, the present application constructs The data processing equipment based on the full-stack programmable network uses the characteristics of the software-defined network of the full-stack programmable network to realize the design of the software function modules and external interfaces of each data processing equipment located on the data plane of the network. The data processing device has complete programmability and reconfigurability, can customize data packet processing logic according to data processing requirements, and realize rapid deployment of application functions of different business layers. Therefore, the data processing device parses the received data packet to obtain the header information of the data packet, which can be field matched with the predefined matching action table, and executes this process on the data packet through the pipeline component corresponding to the matched processing action. The processing action meets various processing requirements for data packets, and enriches the processing methods of data packets compared with forwarding network devices such as traditional switches.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only It is an embodiment of the present application. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without any creative effort.

1 is a schematic structural diagram of an optional example of a data processing device based on a full-stack programmable network proposed by the present application;

2 is a schematic structural diagram of another optional example of a data processing device based on a full-stack programmable network proposed by the present application;

3 is a schematic structural diagram of a P4 compiler located in a control plane in the application of a data processing device based on a full-stack programmable network proposed by the present application;

4 is a schematic flowchart of an optional example of a data processing method based on a full-stack programmable network proposed by the present application;

5 is a schematic flowchart of an optional example of a data processing method based on a full-stack programmable network proposed by the present application;

6 is a schematic flowchart of an optional example of a data processing method based on a full-stack programmable network proposed by the present application;

FIG. 7 is a schematic flowchart of an optional example of a data processing method based on a full-stack programmable network proposed by the present application.

Detailed ways

In view of the description of the background technology, in order to meet the network requirements such as reducing network complexity, virtualization and cloud computing, it is proposed to use a full-stack programmable network, also known as Software Defined Network (Software Defined Network, SDN) technology to build data processing equipment. Using the characteristics of data control decoupling and separation, network virtualization and programmability of full-stack programmable networks, application upgrades and network device upgrades are independent of each other, which speeds up the network deployment of new applications. Differentiated layers improve network control flexibility and reduce hardware footprint and operating costs.

However, at present, as a forwarding network device such as a switch in the network, although it has a certain programmable capability, it can customize the packet header and pipeline processing logic, and does not need to pay attention to the packet definition and specification of the specific protocol, but it can only realize the packet forwarding function. It cannot meet the diverse needs of data processing, such as data storage, computing, etc. In order to further improve the above problems, this application proposes to enhance the network functions supported by the forwarding network equipment, and proposes a full-stack programmable device for multi-core processors. The data processing equipment of the data packet is optimized, and the parser, data processing device (processing pipeline), and inverse parser in the data processing equipment are improved, and the data is forwarded through the internal physical link and hardware of the equipment, and the data is customized as needed. Packet processing logic, realize the data packet forwarding function of different data processing equipment, expand the data packet storage function, capacity calculation function, etc., so that the data processing equipment has the data packet forwarding ability, and also has the data storage ability, computing ability, etc., Better meet application needs.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Referring to FIG. 1, it is a schematic structural diagram of an optional example of a data processing device based on a full-stack programmable network proposed by the present application. The data processing device 100 may be a programmable switching chip, forming a data plane of a full-stack programmable network, After processing and reconstructing the data packets received by a certain port, the data packets are forwarded to another port to realize the forwarding processing of data packets between different ports. During the processing of the data packet, the present application can perform custom storage and/or computing operations on the data packet to meet subsequent application processing requirements. Therefore, as shown in FIG. 1 , the data processing apparatus 100 may include: a parser 110, a data processing apparatus 120 and an inverse parser 130, wherein:

The parser 110 may be configured to parse the input data packet according to the user-defined data packet parsing logic to obtain corresponding data packet header information.

In practical applications, the programmability and reconfigurability of the full-stack programmable network can be used to complete the custom configuration of data processing logic, matching action table and other information on the control plane, that is, to complete the custom configuration of the data plane. As well as the configuration process of the pipeline processing logic, the custom configuration information is sent to the data plane for operation, and the corresponding data packets received by the data plane are processed accordingly to meet the data processing requirements of different services.

Based on this, the present application can use the programmable customization feature of the full-stack programmable network to customize the effective sequence for identifying the packet header and the packet header field in the control plane, and instruct the parser on the data plane to complete the packet. The parsing logic for parsing. In this way, the parser 110 can parse the corresponding data packet input to the parser 110 according to the predefined packet parsing logic to obtain the packet header information, and the implementation process of the packet parsing is not described in detail in this application.

Among them, the full-stack programmable network, compared with the distributed network, separates the control plane from network equipment such as routers and switches, and deploys the control plane on a dedicated control device. For example, the control plane is composed of one or more controllers. , manage the operation of P4 targets through data plane API (Application Programming Interface, such as control plane interface, runtime interface, etc.); and convert network devices into programmable custom functions such as simple control functions and forwarding functions .

The above-mentioned data plane is a basic network composed of data processing equipment (such as forwarding network equipment such as switches) and lines connecting the data processing equipment. The flow table entries (that is, the processing actions in the matching action table) required in the main execution process can be Generated by the control plane, and the controller in the control plane implements the loading and initialization of flow entries.

In order to realize the configuration of each data processing logic of the data plane, as shown in FIG. 2 , taking the data processing device as a P4 switch as an example for illustration, the control plane of the full-stack programmable network may include a P4 compiler 200 and a controller 300 . In practical applications, after loading the P4 compiler tool chain compatible with the basic hardware, the user can input a custom P4 file (ie, program code file), and then, referring to the schematic diagram of the compiler structure shown in Figure 3, it can be compiled in P4 The compiler's front-end compilation environment completes the target-independent conversion of P4 files. The P4 file is a user-defined input based on data processing requirements, and this application does not limit the content of the P4 file.

Among them, the front-end compilation environment may include a checker, an analyzer and an optimizer. The checker can be used to check the lexical, grammatical, semantic and other content of the input P4 file; the analyzer can be used to analyze the P4 file to generate a stateless Intermediate Representation (IR), which usually contains Contains important information such as instruction dependencies and deployment constraints; the optimizer can analyze the context information of the stateless IR to form a context-aware IR for the back-end compilation environment, which can be a JSON file with P4 file information.

After that, the IR configuration instruction and function dependency restriction can be implemented in the mid-end compilation environment, and each network model strategy can be determined, and the configuration implementation process of the strategy is not described in detail in this application. Then, the synthesizer in the back-end compilation environment synthesizes the P4-oriented code implementation and related constraints according to the context-aware IR instructions. The encoder encodes the related constraints to generate a set of conditional constraints, and the converter generates a multi-core set according to the set of conditional constraints. The executable file of the processor, that is, the target compiled file in .cpp (source program format of C++ language) format.

After the P4 compiler is compiled, the executable file generated by the P4 compiler can be run by the running unit, and each network device of the data plane can be initialized, so that the data plane can pass the runtime interface (Run-time API) and control. The plane interacts with each other, obtains information such as data processing logic and matching action table customized by the control plane, and reports the collected network status information of the data plane to the control plane. Based on this, the global network view is determined and provided to each service application.

It can be seen that the control plane of the full-stack programmable network can convert the network policy defined by the management plane into control logic, parse it into a control command, and issue a control flow table to the network device of the underlying data plane. Based on application requirements, the network status information of the data plane can also be obtained, and a global network view can be provided to the network application. Based on the abstract network view of the logic set, it is easier for the software application to interact with the network devices of the underlying data plane, and the network policy is simplified. Execute and network configuration process. This application does not describe the function of the control plane and its implementation process in detail.

Among them, the above-mentioned runtime interface can be used not only to customize the configuration of the matching action table, but also to realize the customized configuration of information such as query counters and default actions performed on packets that fail to match, and can also be used for other control planes. The application invocation realizes the access to the data plane and completes the data processing of the application. The application does not limit the invocation of the runtime interface and its application method, which may be determined according to the circumstances.

The number of data processing apparatuses 120 in the data processing device may be at least one, and each data processing apparatus 120 may include a plurality of predefined pipeline components, which are used for matching results of the data packet header information and the predefined matching action table through Different pipeline components perform matching processing actions on the packets. It can be seen that the present application will implement data processing in a pipeline manner, and the implementation process can be determined in combination with the working principle of the pipeline processor, which will not be described in detail in this application.

In combination with the above description of the technical solutions of the present application, the pipeline in the data processing device 120 can be customized on the control plane according to data processing requirements to obtain a variety of pipeline processing logic, and the network equipment of the data plane can be initialized accordingly. During the process, corresponding predefined pipeline components can be configured, such as storage pipeline components and/or computing pipeline components, and forwarding pipeline components. Wherein, the storage pipeline component can be used to realize the storage of data content of data packets (such as metadata in data packets, intermediate data generated during data packet processing, processed data packets, etc.), regarding the storage of data packets The implementation method is not limited. The computing pipeline component can be used to provide the required computing services for the applications supported by the network device, and realize the computing services of the applications through the complex computing operations that can be defined in detail, accelerate the processing of network data, and reduce the occupancy rate of network resources.

It can be understood that for the processing of data packets, in addition to operations such as storage, calculation, and forwarding, other processing actions, such as modifying and discarding, can also be configured in the matching action table. The corresponding modified pipeline components and discarded pipeline components can be customized. Of course, multiple processing actions can also be integrated into one pipeline component, and the data processing method is not limited in this application.

In the embodiment of the present application, the above-mentioned data processing device will be configured with external interfaces, such as a control interface oriented to the control plane and a programming interface. Through the control interface, various functions in the data plane (such as the embodiments of the present application) can be configured on the control plane. The functions to be implemented by the components of the described data processing equipment) are dynamically configured, of course, the functions of each device located in the control plane can also be dynamically configured. The configuration implementation process will not be described in detail. Descriptions of functional configurations such as logic and data parsing logic. The present application can also obtain network data such as the working status and data of the data plane through the above-mentioned control interface, and accordingly realize the adjustment and dynamic management of the function configuration in the data plane.

The above programming interface can be an application programming interface for network administrators. After the network administrator logs in through the account, after successful authentication, the programming interface can be invoked to realize the development and design of application functions, and dynamically develop new The application functions are distributed, loaded, configured, modified, deleted, etc., and the application function development design and the implementation process of its use are not described in detail in this application.

In still other embodiments proposed in the present application, the data processing apparatus 120 may further include: an update timer for recording the length of time during which any stored data packet is not requested to be accessed, so that when the recorded time length reaches a preset time When the time is long, the data packet is not requested to be accessed, and the data packet is deleted, so as to realize the timely cleaning of expired data packets and reduce the waste of storage resources; when the recorded time length does not reach the preset time length, the data packet is requested. Access, refresh and update the recording time of the timer, so as to continue to record the length of time that the data packet is not requested to be accessed from the current time.

The update timer is configured and generated by invoking the application programming interface of the data processing device. For the configuration implementation process, reference may be made to the relevant description of the configuration process of the data processing logic, matching action table and other information above. This embodiment does not Do detail.

To sum up, compared with the data forwarding device with a single forwarding function in the data plane in the software-defined network architecture, the data processing device in the data plane of the present application is additionally configured with a storage pipeline component and/or a computing pipeline component. When arriving, not only can the original forwarding function be realized, but also storage and/or computing services can be performed on the data plane, which enriches the data processing capability realized by data processing devices such as switches.

The inverse parser 130 can be used to reorganize and forward the data packets processed by the forwarding pipeline components, so as to satisfy the data forwarding function of data processing equipment such as switches, and realize the user-defined network data packet forwarding processing flow. The inverse parser does not describe the implementation process of the reassembly of the packet header information in detail.

In each application scenario based on a full-stack programmable network, for any data processing device in the data plane of the network, such as a switch, the method described above can be used, and the user can complete the data processing on the control plane. The custom configuration of the processing logic of the device, so that during the operation of the data processing device, according to the configured processing logic, the data packets input to the data processing device are processed in a pipeline manner, such as modifying the metadata of the data packets. , store the data packet and/or its processing process, perform complex computing services for the application, etc. After completing the data packet processing, after the reorganization of the data packet header information, it is forwarded to other network devices to realize the data packet in the network. transmission.

The data processing method based on the full-stack programmable network proposed in the present application will be described in detail below from the perspective of any of the above data processing devices, combined with the structure of the data processing device described above, but is not limited to the following embodiments The described data processing method, and for the operations performed by the data processing device according to the embodiment of the present application described in the present application using the flowcharts, it can be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or several steps of operations can be removed from these processes. The following embodiments of the present application are not described in detail, but all belong to the protection scope of the technical solutions of the present application.

Referring to FIG. 4 , it is a schematic flowchart of an optional example of a data processing method based on a full-stack programmable network proposed by the present application. The method is applicable to the data processing device described above. As shown in FIG. 4 , the method may include: But not limited to the following steps:

Step S41, analysing the received data packets to obtain corresponding data packet header information;

In combination with the above description of the technical solutions of the present application, on the control plane of the full-stack programmable network, the custom configuration of each data processing device in the data plane is completed, and the implementation process can refer to the description of the corresponding part of the above embodiment, This embodiment will not be described in detail here. After the data processing device obtains information such as the data processing logic and the matching action table issued by the control plane, the data processing device can run according to the information, so that each module included in the data processing device is in a working state.

In addition, the data processing device can also use the underlying hardware loading unit, through functional abstraction and decoupling of software and hardware, so that the data processing device and its functions do not depend on dedicated hardware, and can also meet application requirements through the software function modules configured as above , through the flexible sharing of resources, to achieve rapid development and deployment of new services.

Therefore, this application can input the user-defined P4 file (degree code file) into the data processing device to complete the underlying configuration, such as inputting the P4 file into the P4 compiler to generate the configuration for the data processing device, such as data processing logic, etc. This configuration is deployed to the data processing device to run, and it satisfies the processing of incoming data packets.

In this way, after completing the configuration of any data processing device in the full-stack programmable network application scenario, such as a switch and other network devices, after it receives a data packet of a data packet, it forwards it to other data processing devices. Or before computer equipment such as terminals and servers, the programmable parser in the data processing equipment can parse the data packets input from the ingress port to realize the separation of the data packet header and the payload part.

In the basic components of the P4 programming language, the data packet header can be divided into two types: the packet header and the metadata. In the case of the packet header used to describe the data packet structure, the separated user-defined packet header can be stored in the data packet structure. , move the pointer backward to complete the parsing of one packet header, continue to parse the next predefined packet header, and repeat until the parsing of each packet header defined in the P4 file is completed. The parsing process can be controlled by The parsing state transition diagram (that is, the packet parsing structure) configured on the plane is deployed on the programmable parser to realize the separation of the header and the payload of each input packet, and the separated header (such as the packet's header) is separated. header, metadata) are stored in the corresponding custom header field, which is used for the matching operation of the matching action table in the pipeline.

Among them, metadata is used to carry data and configuration information. The declaration of metadata is similar to the header, and can include user-defined metadata (such as the operation result of the header field, etc.) and inherent metadata (such as the ingress port of the data packet, the specified data, etc.) The content of the metadata is not limited in this application, which may be determined according to the situation.

Step S42, according to the matching result of the data packet header information and the predefined matching action table, perform the matched processing action on the data packet by different pipeline components;

In practical applications, combined with the above description of data processing equipment, data processing equipment such as switches will be configured with a low-level runtime interface. By calling the runtime interface, matching action tables, query counters, default actions, etc. Configuration, in the configuration process, custom implementation can be performed according to data processing requirements, and this application does not describe the custom implementation method of each information in detail.

Among them, for the above matching action table, matching fields, processing actions and some related attribute parameters (such as table capacity, etc.) can be defined. When the fields defined in the matching action table are successfully matched with the data packets, the corresponding processing actions can be executed; with miss), a predefined default action can be executed. This application does not limit the definition and implementation method of the matching action table.

Optionally, in this embodiment of the present application, a matching action table may be defined in a key-value manner. For example, the header information of the packet header or metadata of the data packet is used as a key value to match the corresponding flow entry (that is, the matching action entry), such as User-defined processing actions such as forwarding, storage, calculation, modification, and discarding (also called processing instructions). Of course, you can also customize the execution order of flow entries (ie, determine the priority), timers, matching fields, etc. information, this application does not limit the configuration and implementation method of the matching action table and its representation.

Exemplarily, in the application of the basic component of the P4 programming language, the following rule can be used to define a matching action table: command_typetable_name[associated_action][match_fields_list][=>][parameters_list], in the matching action table defined by the rule, the command type indicates The types of operations that can be used include adding, deleting, modifying, discarding, storing, calculating, etc. flow entries; the matching field list can include various data packet fields to be matched, and this application does not limit the field types. The attribute parameter list may include related attributes of the above-mentioned processing actions to be executed, such as default actions when matching fails, counters, table sizes, and the like. In practical applications, for the processing action of each flow entry in the matching action table, it can be a user-defined execution sequence (the code fragment that can read the packet header and metadata), and the execution sequence can include several basic Module directives that operate with parameters and header fields or metadata passed to the processing action by the control plane at runtime.

Based on this, after determining the header information of the received data packet, each field of the header of the data packet (such as each header field and each field of the metadata) can be matched with the matching field in the matching action table issued by the controller. Make a comparison to determine whether each field contained in the header information is successfully matched. For any header field that matches successfully, you can determine the pre-defined processing action corresponding to the field in the matching action table, such as the packet processing operator (adding as above). , delete, modify packet headers, etc.), basic arithmetic operators, hash operators, and statistical tracking operators (such as metering, measuring, etc.) Users can customize based on data processing requirements to ensure the support and scalability of P4 language for various protocols.

It should be noted that this application does not limit the matching implementation method between the data packet and the matching action table, and the application can also perform custom configuration and/or dynamic update according to data processing requirements, so as to improve the reliability and performance of the above matching results. Accuracy, the implementation process can be determined with reference to the configuration implementation mode of the matching action table.

Since the pipeline components of the data processing device are editable, the corresponding pipeline components and their pipeline processing logic can be customized according to different data processing requirements. The configuration and implementation method of the pipeline components and their pipeline processing methods are not described in detail in this application. In combination with the above description of the technical solutions of the present application, the present application hopes to complete the forwarding of the received data packets to other network devices, and also store the data packets, data packet processing procedures, etc. for subsequent retrieval. It can also provide computing power, support complex computing services for applications, and meet application processing needs.

Therefore, the different pipeline components predefined in this application may include a storage pipeline component and/or a computing pipeline component, and a forwarding pipeline component; the storage pipeline component is used to realize the storage of the data content of the data packet; The applications supported by the network device provide the required computing services. According to the method described above, after the matching result between the data packet and the matching action table is determined, the corresponding pipeline component can be fetched, and the data packet can be processed in a pipeline manner. Regarding the implementation process of how to call multiple required predefined pipeline components, and input the data packet into the called pipeline component for processing, this embodiment of the present application will not describe in detail here.

Among them, for each pipeline component, it can be a programmable multi-stage pipeline, which is mainly divided into an entry pipeline and an exit pipeline. The entry pipeline can modify the data packets of the input data packets, determine the output port and other operations; the exit pipeline is responsible for the data Modification of grouping. In practical applications, the data plane can sequentially match and process each data packet according to the sequence of the matching action table, and the implementation process is not described in detail in this application.

In step S43, the data packets processed by the forwarding pipeline component are reassembled and then forwarded.

For the data packets output by the above-mentioned pipeline components, a programmable inverse parser can be input to reorganize the data packet header information of the data packets, and the reassembled data packets are processed through the egress port assigned a value in the pipeline. Forward. This application does not describe the working principle of the inverse parser in detail, and during the reorganization process, it is usually implemented based on the inverse parsing structure defined and generated on the control plane, and the application does not limit the self-defined configuration content of the inverse parsing structure.

To sum up, in the application scenario of data packet message processing, compared with traditional forwarding network devices such as switches, this application builds a data processing device based on a full-stack programmable network, using the full-stack programmable network as a software-defined network. It has the characteristics of realizing the design of software function modules and external interfaces of each data processing device located on the data plane of the network, so that the data processing device has complete programmability and reconfigurability, and can be based on data processing requirements. Customize data packet processing logic to achieve rapid deployment of application functions at different business layers. Therefore, the data processing device parses the received data packet to obtain the header information of the data packet, which can be field matched with the predefined matching action table, and executes this process on the data packet through the pipeline component corresponding to the matched processing action. The processing action meets various processing requirements for data packets, and enriches the processing methods of data packets compared with forwarding network devices such as traditional switches.

Referring to FIG. 5 , it is a schematic flowchart of another optional example of a data processing method based on a full-stack programmable network proposed by the present application. The method is applicable to the data processing device described above, and this embodiment may be the above-described data processing device. An optional refinement implementation method of the data processing method, but is not limited to this implementation method, as shown in Figure 5, the method may include but is not limited to the following steps:

Step S51, obtaining the custom data packet analysis logic on the control plane;

Step S52, according to the data packet analysis logic, analyze the received corresponding data packet to obtain the corresponding data packet header information;

Regarding how the parser parses the data packet after it is input to the programmable parser in the data processing device through the ingress port to realize the separation of the packet header and the payload, the parsing implementation process can refer to the above embodiment and the full stack The application principle of the programmable network is determined, which is not described in detail in this embodiment of the present application.

Step S53, carrying out field matching with the packet header information and the predefined matching action table;

Step S54, determine that at least one first field of the packet header information is successfully matched, and obtain a self-defined data processing logic corresponding to the first field on the control plane;

Step S55, running the corresponding data processing logic through the pipeline component corresponding to the first field, and performing the matched processing action on the first field data corresponding to the input data packet;

Combined with the above description of the matching action table, since the matching action table contains a matching field list (ie, matching header field), a processing action list and its parameters, the number of flow tables, etc., and each pipeline is in each matching action table. execution order, etc. In this way, the separated packet header information is matched with the matching action table issued by the controller to determine which header fields exist in the matching action table and which header fields do not exist in the matching action table. It is recorded as the first field, and the first field that fails to match is the second field.

In the matching action table, the processing actions corresponding to different matching fields may be different, that is to say, the processing actions for different field data in the data packet may be different, and the pipeline processing methods of different types of processing actions are different. Therefore, the present application can determine the pipeline processing method for each field data of the data packet according to the above matching results, and call the corresponding pipeline component to perform corresponding processing actions on the field data of the input data packet, so as to satisfy various processing methods for the data packet. need. In the process of pipeline processing, it can be implemented according to the data processing logic customized on the control plane and suitable for multi-core processors. The data processing logic defined for different processing actions can be different. Therefore, this application can use different pipeline components, According to the corresponding data processing logic, the matched processing action is performed on the input data packet.

In still other embodiments, when it is determined that at least one second field of the header information of the data packet fails to match according to the above matching processing method, the data of the second field may be pre-configured according to the default action configured in the matching action table. The defined default action can also be implemented according to the corresponding data processing logic pre-configured on the control plane for the execution process of the default action, and the implementation process is not described in detail in this application.

In step S56, the data packets processed by the forwarding pipeline component are reassembled and then forwarded.

Referring to FIG. 6 , it is a schematic flowchart of another optional example of the data processing method based on the full-stack programmable network proposed by the present application. The operation process of storing the data content of the packet is described, but it is not limited to the storage implementation method described in this embodiment. Regarding the analysis and matching processing process of the data packet, reference can be made to the description of the corresponding part of the above embodiment, This embodiment will not be described in detail here.

As shown in Figure 6, the method may include:

Step S61, calling the storage pipeline component, and continuously storing the data content of the data packet that meets the storage condition according to the context information of the data packet;

After the programmable parser completes the analysis of the data packet, it is necessary to store the data content such as the data packet, the processing process of the data packet, the processed data packet, etc., such as according to the matching action table described above, determine the matching The processing action is a storage operation, and the storage pipeline component can be called to store the data content of the data packet. During the storage process, the application can continuously store the data content that meets the storage conditions according to the context information of the data packet, so as to ensure Data consistency across different storage operations.

The above storage conditions can be determined according to data storage requirements, such as storage according to relevant data such as access hotspots and object access concerns in related fields. Therefore, the data content of the above-mentioned data packets that meet the storage conditions may include requests that belong to the requested access times reached. The data content of the data type of the threshold value, and/or the data content of which the correlation with the data content of the last requested access reaches the relevant threshold value, etc., this application does not limit the request threshold value, the value of the relevant threshold value, and the storage condition content, etc., Subject to availability.

The above-mentioned storage pipeline component may be a storage chip running a predefined data storage logic to realize persistent storage of data content that meets storage conditions. In this way, in the case of an abnormality in the data processing device, the forwarded data packets are lost. In this case, the corresponding data packet can still be directly read from the storage pipeline component, and the data packet does not need to be resent by the data source, which improves the transmission speed of the data packet and reduces the waste of resources caused by repeated transmission of the data packet.

It can be seen that, for the data processing device based on the full-stack coding network proposed in this application, compared with the traditional forwarding network devices such as switches, in the pipeline processing module, on the basis of the forwarding pipeline component, a part of storage space (that is, the storage pipeline is added) is added. The storage space of the component), record the data content that meets the storage conditions by maintaining a form with a content name, and perform various operations on the stored data content according to the form. This application does not limit the form content.

Step S62, regularly update and detect the stored data content;

In order to reduce the waste of storage resources, the data content stored by any data processing node in the data plane can be regularly updated and detected, and the data content that has not been accessed for a long time can be cleaned up in time. In this regard, in this embodiment of the present application, a data content update processing rule may be customized on the control plane in advance according to the service type, data processing scenario, etc., and the present application does not limit the content of the rule. After that, during the operation of the data processing device, the data content stored by itself may be periodically updated and detected according to the predefined data content update processing rules, and the implementation process will not be described in detail.

Based on this, in some embodiments, for each data content stored by the data processing device, the dynamic storage duration of the data content, the non-access duration after each access to the data content, etc. , so that the content of the data that needs to be deleted or updated can be subsequently determined accordingly. This application does not limit the content of the update detection, which may be determined according to the circumstances.

Step S63, according to the regular update detection result, delete the data content that has not been accessed within the stored preset duration;

Combined with the above analysis, after each update detection of the relevant information of the stored data content, if it is determined that some of the stored data content has not been accessed for a long time, in order to reduce the occupation of storage resources, you can delete such types of data from the storage space. data content. Therefore, this application can pre-configure how long the data content has not been accessed (referred to as the preset time length) according to the data processing scenario, business requirements, etc., and the data content can be deleted. This application does not limit the value of the preset time length.

Wherein, in the above data content update processing process, a corresponding update timer (also called an expiration timer) can be configured for each content name form maintained by the data storage device, which is used to record the corresponding data content every time The storage duration after being accessed, if the storage duration does not reach the preset duration, the data content is requested to be accessed, and the update timer can be refreshed. When the requested access starts, the storage time of the data content that has not been accessed is counted again.

Step S64, it is detected that the remaining amount of storage space is less than the preset capacity threshold, and when the data content to be stored is obtained, according to the storage time stamp of the stored data content, determine the to-be-processed data packet that has not been requested to access for the longest time;

In step S65, the data content to be processed is replaced by the data content to be stored for storage.

In still other embodiments, regarding the update processing method of the stored data content by the data processing device, including but not limited to the above-described processing method for cleaning up the data content that has not been accessed for a long time, it can also be based on The dynamic margin of the storage space of the data storage device itself is used to store the newly acquired data content that needs to be stored. Therefore, the present application can perform real-time detection on the remaining amount of storage space of the data processing device, and if it is detected that the remaining amount of storage space is less than the preset capacity threshold, it can be considered that the storage space of the data processing device is insufficient, in order to analyze the currently obtained data to be stored. In the storage of the content, the data packets to be processed that have not been requested to be accessed for the longest time can be replaced according to the storage time stamp of the stored data content, but it is not limited to this processing method.

In some other embodiments proposed in this application, according to the above analysis, in the pipeline processing process of the data processing device, in addition to the data packet forwarding function, a certain computing capacity can also be added, that is, the computing pipeline component can be called to meet the computing requirements of the application. Serve. Therefore, as shown in FIG. 7 , a schematic flowchart of another optional example of the data processing method based on the full-stack programmable network proposed by the present application, after completing the parsing of the data packet, a predefined computing pipeline component can also be called to perform For calculation, regarding the analysis and processing process of the data packet, reference may be made to the description of the corresponding part of the above embodiment, which is not repeated in this embodiment, and the operation process of the calculation pipeline component can be described in this embodiment. As shown in Figure 7, the method may include:

Step S71, invoking the computing pipeline component, configuring the computing capacity of the network device by means of a virtual machine, and determining the virtual machine copy corresponding to each application supported by the network device, so as to realize the computing service of the corresponding application through the virtual machine copy;

In the embodiments of the present application, for some complex calculations of applications, the calculation pipeline component can be called to divide the computing network of the data processing device by means of a virtual machine, and provide a virtual machine with computing capabilities for each application that requires computing services Copy, according to which the computing service corresponding to the application is realized. This application does not describe in detail the computing service type supported by each application and its operation principle, which can be determined according to business processing requirements.

Step S72, regularly update or delete the virtual machine copy;

The computing pipeline component can maintain a computing service form based on the computing services that it can provide, according to which the virtual machine copy can be updated or deleted regularly. The activated computing service may delete the virtual machine copy supporting the computing service; if the function of the computing service is updated, the virtual machine copy supporting the computing service may be updated correspondingly, and the implementation process will not be described in detail.

Step S73, acquiring computing feature information of the data processing device;

Step S74 , reporting the computing feature information to the controller of the control plane, so that the controller implements unified management and scheduling of multiple data processing devices according to the computing feature information.

In order to achieve unified management and scheduling of multiple data processing devices in the network, the computing pipeline component can extract the computing feature information of the data processing device where it is located, such as the remaining computing capacity of the data processing device, and report it to the control plane on the control plane in time. so that the control can perform unified management and scheduling according to the computing feature information, so as to realize the rational utilization of computing resources. This application will not describe in detail the management and scheduling implementation methods of multiple data processing devices.

Embodiments of the present application further provide a computer-readable storage medium on which a computer program is stored, and the computer program is loaded and executed by a processor to implement the steps of the above-mentioned full-stack programmable network-based data processing method, For a specific implementation process, reference may be made to the descriptions in the corresponding parts of the foregoing embodiments, which will not be repeated in this embodiment.

The embodiments of the present application also provide a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned various optional implementations of the full-stack programmable network-based data processing method. For the method and the specific implementation process, reference may be made to the descriptions of the above corresponding embodiments, which will not be repeated.

Finally, it should be noted that the various embodiments in this specification are described in a progressive or juxtaposed manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. . For the data processing device and medium disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, in order to clearly illustrate the possibilities of hardware and software. Interchangeability, the above description has generally described the components and steps of each example in terms of function. Whether these functions are implemented in hardware or software depends on the specific application of the technical solution and design preset conditions. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the core idea or scope of the present application. Therefore, this application is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A data processing method based on a full-stack programmable network, characterized in that the method comprises: Analyze the received data packets to obtain the corresponding data packet header information; According to the matching result between the header information of the data packet and the predefined matching action table, the matched processing actions are performed on the data packet through different pipeline components; the different pipeline components include storage pipeline components and/or Computation pipeline components, and forwarding pipeline components; The data packets processed by the forwarding pipeline component are reorganized and forwarded; Wherein, the storage pipeline component is used to realize the storage of the data content of the data packet; the calculation pipeline component is used to provide required computing services to the application supported by the network device. 2. The method according to claim 1, wherein, according to the matching result of the data packet header information and a predefined matching action table, the data packets are executed by different pipeline components. Processing actions, including: Field matching is performed between the packet header information and a predefined matching action table; It is determined that at least one first field of the data packet header information is successfully matched, and the matched processing action is performed on the data corresponding to the first field in the data packet through the pipeline component corresponding to the first field; It is determined that at least one second field of the header information of the data packet fails to match, and a predefined default action is performed on the data corresponding to the second field in the data packet. 3. The method according to claim 1 or 2, wherein the performing the matched processing action on the data packet through the corresponding pipeline component comprises: calling the storage pipeline component to store the data content of the data packet; the data content includes metadata in the data packet, intermediate data and/or processed data generated during the processing of the data packet package; and/or, The computing pipeline component is invoked, the computing capacity of the network device is configured through the virtual machine, and the virtual machine copy corresponding to each application supported by the network device is determined, so as to realize the computing service of the corresponding application through the virtual machine copy. 4. The method according to claim 3, wherein the storing the data content of the data packet comprises: According to the context information of the data packet, continuously store the data content of the data packet that meets the storage conditions; Wherein, the data content of the data package that meets the storage conditions includes data content belonging to a data type whose requested access times reach a request threshold, and/or data content whose correlation with the data content requested for access last time reaches a relevant threshold. 5. The method according to claim 4, wherein the method further comprises: Regularly update and detect the stored data content; Delete data content that has not been accessed for a preset period of time stored in accordance with the regularly updated detection results; and/or, When it is detected that the remaining amount of storage space is less than the preset capacity threshold, when the data content to be stored is obtained, according to the storage time stamp of the stored data content, the pending data packets that have not been requested to be accessed for the longest time are determined; The to-be-processed data content is replaced by the to-be-stored data content for storage. 6. The method according to claim 3, wherein in the process of invoking the computing pipeline component, the method further comprises: Periodically update or delete the virtual machine copy; Obtain the computing feature information of the data processing device; The computing feature information is reported to the controller of the control plane, so that the controller implements unified management and scheduling of a plurality of the data processing devices according to the computing feature information. 7. The method according to claim 1, wherein the received data packets are analyzed to obtain corresponding data packet header information, comprising: Get custom packet parsing logic on the control plane; According to the data packet analysis logic, the received corresponding data packet is analyzed to obtain the corresponding data packet header information; Performing the matched processing action on the data packet through different pipeline components includes: obtain data processing logic customized on the control plane suitable for multi-core processors; Through different pipeline components, according to the corresponding data processing logic, the matched processing actions are performed on the input data packets. 8. A data processing device based on a full-stack programmable network, comprising: a parser, a plurality of data processing devices and an inverse parser, wherein: The parser is used to parse each received data packet to obtain corresponding data packet header information; The data processing apparatus includes a plurality of predefined pipeline components, configured to perform the matched processing on the data packet through the different pipeline components according to the matching result between the data packet header information and the predefined matching action table Action; wherein, the different pipeline components include a storage pipeline component and/or a computing pipeline component, and a forwarding pipeline component; the storage pipeline component is used to realize the storage of the data content of the data packet; the computing pipeline component Used to provide required computing services for applications supported by network equipment; The inverse parser is configured to reassemble and forward the data packets processed by the forwarding pipeline component. 9. The device according to claim 7, wherein the data processing device further comprises: Update the timer, which is used to record the length of time that any stored data packet is not requested to be accessed, so that when the recorded time length reaches the preset time length, the data packet is not requested to be accessed, and the data packet is deleted; When the duration does not reach the preset duration, the data packet is requested to be accessed, and the record time of the update timer is refreshed; Wherein, the update timer is configured and generated by invoking an application programming interface of the data processing device. 10. A computer-readable storage medium, characterized in that a plurality of computer instructions are stored thereon, and the plurality of computer instructions are loaded and executed by a processor, so as to realize the full-based system according to any one of claims 1-7. A data processing method for a stack programmable network.

Images