RU2854059C1 - Method for obtaining network information regarding a physical network port taken out of the operating system kernel control and managed by an application in user space - Google Patents

Abstract

FIELD: computing.

SUBSTANCE: method includes: creating a virtual network interface by means of the operating system kernel; mirroring incoming network packets from said physical network port to said virtual network interface; processing network packets received on the virtual network interface to obtain network information.

EFFECT: increased flexibility and expanded functional capabilities of the system for obtaining network information by ensuring compatibility of high-performance applications that manage the physical network port bypassing the operating system kernel with standard kernel monitoring and administration tools, without using specialised equipment and without modifying the operating system network stack.

16 cl, 3 dwg

Description

The group of inventions relates to the field of computing technology, and more specifically to technologies for managing and processing network traffic in high-performance systems, such as next-generation firewalls (NGFW), intrusion prevention systems (IPS) and other deep packet inspection (DPI) devices.

In traditional network architectures, packet processing is performed by the operating system kernel, which incurs significant overhead due to context switches and multiple data copies. To overcome these limitations, technologies for direct access to network hardware from user space, such as DPDK (Data Plane Development Kit), have been developed. These technologies allow a user-space application to control a physical network port, removing it from the control of the operating system kernel, thereby improving performance. However, this approach completely isolates the port and the traffic passing through it from standard operating system monitoring and management tools.

Solutions are also known that aim to virtualize network resources in multi-user or multi-tasking environments, such as servers with multiple operating system partitions.

As an example of such a solution, a method for obtaining network information regarding a physical network port taken out from the control of the operating system kernel and controlled by an application in user space is presented, selected as a prototype, during the execution of which logical ports are created for each section of the operating system, which are a projection of the physical ports of the network adapter, while routing traffic between the physical ports connected to the external network and the logical ports associated with the sections of the operating system (OS) is implemented inside the network adapter itself by means of a hardware logical switch, in addition, the direction of traffic is carried out directly between different sections of the OS along an internal path, bypassing the external network, while data flow management in the adapter is carried out by means of a queuing system that allows packets to be directed either directly to applications in user space, or to the kernel of the corresponding section of the OS [US20060221975A1, publication date: 03.01.2012].

A disadvantage of the prototype is the limited flexibility and versatility of the method. The main reason is that all of the method's functions are implemented in specialized hardware, which leads to a number of disadvantages:

– the lack of the ability to modify hardware-implemented logic leads to the need for a complete replacement of equipment or a complex software update when traffic processing requirements change, new protocols or control functions are added;

– the inability to use standard network adapters leads to a significant increase in the cost of specialized equipment and the creation of a critical dependence on a specific supplier, its technological ecosystem and update release schedule;

– the lack of integration with standard system utilities and kernel services makes it impossible to use familiar administration tools and standard routing domains, turning the system into a “black box” for the operating system.

Therefore, a solution is required that combines the high packet processing performance achieved through direct hardware access technologies with flexibility, extensibility, and full integration with standard operating system management and monitoring tools, all while using standard, readily available network equipment.

The technical problem that this group of inventions aims to solve is to eliminate the existing shortcomings of the prototype.

The technical result, which the group of inventions is aimed at achieving, consists in increasing the flexibility of control and expanding the functional capabilities of the system for obtaining network information.

The essence of the first invention from the group of inventions is as follows.

A method for obtaining network information regarding a physical network port that is removed from the control of the operating system kernel and controlled by an application in user space includes:

– creation of a virtual network interface using the operating system kernel;

– performing mirroring of at least incoming network packets from said physical network port to said virtual network interface;

– processing network packets received on a virtual network interface to obtain network information.

The essence of the second invention from the group of inventions is as follows.

A system for obtaining network information by the method of the first invention comprises at least one physical network port, a processor and memory associated therewith, wherein the processor is configured with the ability to:

– control of a physical network port by an application in user space, which takes the port out of the control of the operating system kernel;

– creation of a virtual network interface using the operating system kernel;

– performing mirroring of at least incoming network packets from said physical network port to said virtual network interface;

– processing network packets received on a virtual network interface to obtain network information.

The developed method can be implemented using various systems and devices designed to process network traffic. These systems include general-purpose servers with integrated network interface cards (NICs), specialized network devices (hardware gateways, NGFWs, and DPI systems), software routers and virtual network functions (VNFs) running on NUMA platforms, as well as network accelerators and integrated solutions based on FPGAs or specialized ASICs.

Removing the physical network port from the control of the operating system kernel and managing it through a user-space application improves network packet processing performance by eliminating the operating system kernel's overhead. This can be achieved through specialized technologies such as DPDK, where the management application initializes the runtime environment and gains exclusive direct access to the network adapter hardware via polling mode drivers. This creates the necessary prerequisite for high-performance packet processing, which lays the foundation for subsequent increased management flexibility and expanded system functionality.

Creating a virtual network interface using the operating system kernel restores communication with the kernel that was lost due to the removal of a physical port from its control. This can be achieved by creating a TAP interface, which emulates a data link device and provides a file descriptor to user space for exchanging Ethernet frames. This provides a key mechanism for interaction between the isolated high-performance circuit and the operating system kernel, directly contributing to increased management flexibility by providing the kernel and its utilities with an entry point for monitoring traffic.

Mirroring at least incoming network packets from a physical network port to a virtual network interface enables the creation of copies of network packets and forwarding these copies to the virtual interface without slowing down the primary processing path. Mirroring can be implemented as a process in which a user-space management application reads a packet from the physical port, creates a copy of it, and writes it to the TAP interface file descriptor. This implements a bridge between the high-performance processing loop and the operating system kernel, enabling effective interaction and increasing system management flexibility by providing system services with access to traffic information.

Processing network packets received on a virtual network interface enables analysis and modification of network traffic using both standard operating system kernel tools and specialized algorithms of the control application. Network packet processing can be performed through operations such as analysis, adding, modifying, or deleting virtual local area network (VLAN) tags, as well as other types of network information processing. This enables packet processing simultaneously on the kernel side for management purposes and on the high-performance application side for modification, directly enhancing the functionality and flexibility of the entire system.

A TAP interface can be created as a virtual network interface, enabling data link layer operation with the transmission of complete Ethernet frames, including all headers and VLAN tags, to the operating system kernel. The TAP interface can be created using standard operating system tools for creating TAP devices. This ensures the most complete and accurate interaction, transmitting undistorted packet information to the kernel, which is critical for management flexibility and the correct operation of system traffic analyzers.

Mirroring can be performed for both incoming and outgoing network packets on a physical network port, thereby providing the operating system kernel with full visibility into bidirectional data exchange via the physical port. Mirroring can be implemented by duplicating the mirroring process for both traffic directions. This further increases management flexibility and expands monitoring capabilities, allowing for the tracking of complete data transfer sessions, not just unidirectional traffic.

Mirroring can be implemented by an application running in user space, enabling the mirroring function to be performed most efficiently, since all data is already in the address space of the controlling application. Mirroring can be implemented using an algorithm in which the application itself, having accessed the packets, creates copies of them and forwards them to the virtual interface. This ensures interaction with minimal overhead while maintaining high performance of the primary processing path, a prerequisite for the practical application of this group of inventions and achieving the claimed result without speed degradation.

A physical network port can be removed from the operating system kernel's control using a set of libraries and drivers for fast packet processing, such as DPDK, which implements a specific and preferred method for kernel bypass technology using a ready-made and standardized toolkit. Removing a physical network port can be accomplished by initializing the DPDK environment, linking network ports to user-space drivers, and configuring the appropriate libraries. This ensures the technical feasibility of a high-performance circuit that expands the functionality of the method and increases control flexibility using the inventions.

Network packet processing can be performed using a single operation from a group consisting of VLAN tag information analysis, adding a VLAN tag, changing a VLAN tag, and deleting a VLAN tag. This enables the acquisition and use of network information to implement complex network scenarios without accessing the operating system kernel. This can be represented by a set of algorithms for parsing and modifying Ethernet frame headers in terms of processing VLAN tags. This directly demonstrates the expanded functionality of the system, which gains the ability to perform complex traffic manipulations previously unavailable in an isolated high-performance environment.

Additionally, a data structure can be created and stored in shared memory, establishing a mapping between the physical network port identifier, the identifier of the virtual network interface created for it, and at least one logical interface identifier used for routing. This process enables the mapping of various interface namespaces within the system. Creation of data structures can be accomplished by creating a common data structure in shared memory accessible to various system components. This ensures fast and efficient interaction between various system components, as well as between high-performance processing circuits and the operating system kernel, directly contributing to increased control flexibility by enabling the correct translation of control commands into specific actions with hardware and virtual interfaces.

The group of inventions is characterized by a previously unknown set of essential features from the state of the art, distinguished in that:

- a physical network port, removed from the control of the operating system kernel and controlled by an application in user space, ensures the creation of a high-performance network packet processing circuit that operates in user space through direct access to the network adapter hardware, bypassing the resource-intensive mechanisms of the operating system kernel, thereby creating a technically isolated processing environment that, while possessing high performance, lacks the ability to directly interact with standard operating system services, which makes it possible to create a high-performance processing circuit;

– the creation of a virtual network interface by means of the operating system kernel ensures the presence in the operating system kernel of a standard, controlled network object, which is recognized by all system services and utilities as a normal network interface, thereby creating a logical point of interaction in the kernel space, which makes it possible to create a bridge between the isolated circuit and the kernel, which is necessary for subsequent interaction and increased control flexibility;

– mirroring of at least incoming network packets from a physical network port to a virtual network interface ensures the establishment of an active information flow from the high-performance circuit to the standard circuit of the operating system kernel, in which the control application sends a copy of the packet to the created virtual interface without compromising the processing performance of the original, thereby allowing the operating system kernel to analyze the actual traffic passing through the isolated physical port, which directly ensures interaction between the two circuits and increases management flexibility due to the ability to apply standard monitoring tools to previously inaccessible traffic;

– processing of network packets received on the virtual network interface to obtain network information provides the ability to use information transmitted through the created mirroring mechanism, allowing the kernel network stack and applications working with it to analyze the composition and content of mirrored packets for the purposes of monitoring, collecting statistics and making control decisions, thereby realizing a synergistic effect in which information from one circuit is used to control another, which ensures the achievement of a technical result by increasing control flexibility and expanding the functionality of the entire system.

The combination of the essential features of this group of inventions makes it possible to create a new system architecture in which the high-speed advantages of direct access to hardware resources are combined with the manageability and functionality of a standard kernel network stack, which were previously mutually exclusive approaches.

This ensures the achievement of a technical result consisting of increasing control flexibility and expanding the functional capabilities of the system for obtaining network information.

The group of inventions possesses a set of essential features previously unknown in the state of the art, which indicates its compliance with the patentability criterion of “novelty”.

This group of inventions is unknown in the prior art and does not clearly follow from it. Therefore, this group of inventions meets the patentability criterion of "inventive step."

The inventions from a group of inventions are interconnected and form a single inventive concept, which indicates that the group of inventions meets the patentability criterion of “unity of invention”.

The group of inventions is illustrated by the following figures.

Fig. 1 is a functional diagram of the system for obtaining network information, illustrating its main components and their interrelations.

Fig. 2 - block diagram of the algorithm for initialization and configuration of the system for obtaining network information.

Fig. 3 - block diagram of the algorithm of the working cycle of processing a system packet for obtaining network information.

To illustrate the possibility of implementation and a more complete understanding of the essence of the group of inventions, a variant of its implementation is presented below, which can be changed or supplemented in any way, while the present group of inventions is in no way limited to the variant presented.

To implement the method, a system 100 for obtaining network information is used, implemented as a hardware and software system, such as a next-generation firewall. System 100 includes a processor 110, a control application 120, memory 130, a physical network port 140, and a virtual network interface 150.

Processor 110 is controlled by user-space application 120. Control application 120 is the core component implementing fast packet processing and associated processing modules. Application 120 utilizes kernel bypass technologies, such as a set of fast packet processing libraries, to gain direct control over physical network ports. Control application 120 includes:

– a mirroring execution module 122, which is a logical block within the control application 120, responsible for reading network packets from the physical port and directing copies of them to the virtual network interface 150 associated with it.

– a network information processing module 124, which is a set of components within the control application 120, including a link layer parsing unit, a routing unit and a header generation unit, which perform packet analysis and modification, including operations with virtual local area network tags, based on information obtained from the packet and the system configuration.

Physical network port 140 is a hardware network interface, removed from the control of standard operating system kernel drivers and controlled directly by application 120.

Virtual network interface 150 in the operating system kernel is a virtual network device created by the operating system kernel and, in the preferred embodiment, represented by a virtual data link layer interface. This interface mirrors physical port 140 for the operating system, making traffic visible to standard utilities and the kernel network stack.

The interface mapping data structure 132 is stored in the shared memory 130 and establishes a one-to-one correspondence between various identifiers: the hardware identifier of the physical port 140, its identifier in the fast packet processing environment, the identifier of the mirrored virtual interface 150 in the operating system kernel, and the logical identifier used for routing.

The method is carried out in two main stages: the initialization and configuration stage of the system and the stage of the working cycle of packet processing.

System initialization and configuration stage.

This preparatory stage is performed once when the system 100 is started to set up the environment and create the necessary connections between components.

Step 200. Releasing physical port 140 from kernel control. Control application 120 initializes the environment abstraction. Using specialized drivers, it binds target physical network ports 140 to itself, releasing them from the control of the operating system kernel. Application 120 allocates memory in large pages and creates buffer pools for receiving packets.

Step 202. Creating a virtual mirror interface 150. During the initialization of the fast packet processing environment, the control application 120 issues a command to create a virtual interface 150 of the data link layer for each controlled physical port 140. A new network interface appears in the system, completely controlled by the operating system kernel, but associated with the control application 120.

Step 204. Creating and filling the mapping structure 132. The data structure 132 is initialized and placed in the shared memory 130 and filled with information about the mapping between the physical port 140, its identifier in the fast packet processing environment, and the mirrored virtual interface 150 created for it.

Step 206. Launching processing module 124. All components of control application 120 are launched, which begin accessing the receive queues of physical ports 140 in constant polling mode.

A stage of the batch processing work cycle.

This stage is performed for each incoming packet in real time.

Step 300. Receiving a packet on physical port 140. Control application 120 reads the incoming network packet from the receive queue of physical port 140 into the buffer, bypassing the operating system kernel network stack.

Step 302. Mirroring. Immediately after reading, control application 120 creates a copy of the packet and forwards it to the associated virtual data link layer interface 150. From this point on, the packet copy becomes visible in the operating system kernel and can be analyzed using standard utilities.

Step 304. Network Information Processing on the Primary Path. The original packet continues to be processed on the high-performance user-space path. This step may include virtual local area network (VLAN) tag processing: analysis, in which the data link parsing module determines the presence of a VLAN tag and extracts the identifier from the header, storing it in the packet's metadata; decision, in which the policy-based routing module determines the outgoing interface and the required VLAN identifier for it, storing this decision in the session metadata; modification, in which the metadata-based header generation module generates the final data link header, adding the tag if it does not exist, removing it if it is not needed, or changing the VLAN identifier value if necessary.

Step 306. Sending the packet. The modified packet is placed in the transmit queue on physical port 140 for transmission to the network. After sending the packet, the packet processing cycle begins again and is repeated.

A specific example of the implementation of a group of inventions.

To illustrate the operation of the group of inventions, a specific example of its implementation on a system 100 for obtaining network information, implemented in the form of a next-generation firewall (NGFW), is presented.

Initial conditions:

System 100 has a physical network port 140 with a PCI address of “0000:31:00.0”, which is taken out of the control of the operating system kernel via DPDK technology.

The control application 120 includes tagged traffic processing components:

o Parser-L2 as part of the 124 network information processing module for parsing 802.1q headers

o SmartBalancer for generating FlowMeta metadata

o Router to define the outgoing interface

o Stacker-L2 for generating outgoing packet headers

For physical port 140, a virtual network interface 150 of type TAP with the name “ng_eth0” was created.

In the shared memory 130, a data structure 132 is located for mapping interfaces, containing:

o rte_id = 0 (physical port identifier in the DPDK environment)

o kernel_id = “ng_eth0” (virtual interface identifier in the OS kernel)

o mirror_rte_id = 5 (TAP interface identifier in the DPDK environment)

The processing configuration implements the “Tag → Tag” scenario: incoming traffic with the VlanID_A tag must be redirected with the modified VlanID_B tag through the same physical port 140.

Implementation in Stage 1: Initialization and Configuration.

Removing a physical port from kernel control:

– Control application 120 initializes the DPDK environment with parameters – allow=0000:31:00.0

– Physical port 140 is bound to the vfio-pci driver and receives rte_id = 0

– A mempool is created for mbuf buffers with element sizes according to the specification

Creating a virtual mirror interface:

– Control application 120 creates a TAP interface with the parameter --vdev=net_tap0,iface=ng_eth0,persist

– Virtual interface 150 “ng_eth0” appears in the operating system kernel and receives mirror_rte_id = 5

Initializing the mapping structure:

– In shared memory 130, the iflayout data structure 132 is created

– Structure 132 is filled with the following mapping: physical port (rte_id=0) ↔ mirror interface (mirror_rte_id=5, kernel_id=“ng_eth0”)

Launching processing components:

– The 124 processing module is launched with the Parser-L2, SmartBalancer, Router, Stacker-L2 components

– The mirroring execution module 122 is activated

Implementation in stage 2: work cycle (processing a specific packet)

Scenario: A network packet arrives at system 100.

• Receiving port: physical port 140 (rte_id=0)

• Receipt queue: queue 0

• VLAN tag: VlanID_A

System actions:

Packet reception: Control application 120 reads the packet from the RX queue of physical port 140 into the mbuf buffer. The packet contains an Ethernet header with the EtherType field set to 0x8100 and the VlanID_A value.

Mirroring execution: Mirroring module 122 creates a copy of the incoming packet and forwards it to virtual interface 150 "ng_eth0" (mirror_rte_id=5). From this point on, the packet copy becomes visible in the operating system kernel and can be analyzed using standard utilities.

Processing network information on the main path:

Parser-L2: The component detects the presence of a VLAN tag based on the EtherType=0x8100 value, parses the 802.1q header structure, and writes VlanID_A to the PacketMeta structure.

SmartBalancer: Creates a FlowMeta with a record of the VlanID_A value in the session-ingress/branch/mac-ingress/mac/vlan section and adds a record to the journald system log using the libngj library.

Router: Defines the egress interface as the same physical port 140 with the required VlanID_B value and writes it to the session-egress/branch/mac-ingress/mac/vlan FlowMeta section, adding an entry to journald.

Stacker-L2: Detects the difference between ingress VlanID_A and egress VlanID_B, changes the value of the VlanID field in the Stack PacketMeta structure from VlanID_A to VlanID_B, and generates an 802.1q header with the new value.

Packet transmission: The modified packet with the VlanID_B tag is placed in the TX queue of physical port 140 for transmission to the network. After the packet is transmitted, the system moves on to processing the next packet.

Thus, the packet was routed with a VLAN tag change and minimal latency thanks to high-performance user-space processing. At the same time, thanks to the mirroring mechanism via virtual interface 150, system 100 retained the traffic monitoring function of the operating system kernel and the ability to use standard management utilities.

This example demonstrates how the creation of a virtual mirror interface and the implementation of mirroring can achieve a technical result – increased management flexibility and expanded system functionality for obtaining network information.

Claims (21)

1. A method for obtaining network information regarding a physical network port that is removed from the control of the operating system kernel and controlled by an application in user space, comprising: – creation of a virtual network interface using the operating system kernel; – performing mirroring of at least incoming network packets from said physical network port to said virtual network interface; – processing network packets received on a virtual network interface to obtain network information. 2. The method according to paragraph 1, during the execution of which a TAP interface is created as a virtual network interface. 3. The method according to paragraph 1 or 2, in which mirroring is performed for both incoming and outgoing network packets of the physical network port. 4. The method according to paragraph 1, in which mirroring is carried out by means of the said application running in user space, which processes network packets received from the physical network port and sends copies of them to the virtual network interface. 5. The method according to claim 1, wherein the physical network port is removed from the control of the operating system kernel using a set of libraries and drivers for fast packet processing, such as DPDK (Data Plane Development Kit). 6. The method according to claim 1, wherein the processing of network packets includes at least one of the operations from the group consisting of: analyzing information about a virtual local area network (VLAN) tag, adding a VLAN tag, changing a VLAN tag, and deleting a VLAN tag. 7. The method according to claim 1, which additionally involves creating and storing in shared memory a data structure that establishes a correspondence between the identifier of the physical network port, the identifier of the virtual network interface created for it, and at least one logical interface identifier used for routing. 8. A system for obtaining network information by the method of claim 1, comprising at least one physical network port, a processor and memory associated therewith, wherein the processor is configured to: – control of a physical network port by an application in user space, which takes the port out of the control of the operating system kernel; – creation of a virtual network interface using the operating system kernel; – performing mirroring of at least incoming network packets from said physical network port to said virtual network interface; – processing network packets received on a virtual network interface to obtain network information. 9. The system according to claim 8, characterized in that the processor is configured to create a TAP interface as a virtual network interface. 10. The system of claim 8 or 9, wherein the processor is configured to perform mirroring for both incoming and outgoing network packets of the physical network port. 11. The system of claim 8, wherein the processor is configured to perform mirroring by executing instructions of said application in user space that reads network packets from a physical network port and sends copies of them to a virtual network interface. 12. The system of claim 8, wherein said user-space application uses a set of libraries and drivers for fast packet processing, such as DPDK (Data Plane Development Kit), to remove the physical network port from the control of the operating system kernel. 13. The system of claim 8, wherein the processor is configured to process network packets, including at least one of the operations from the group consisting of: analyzing information about a virtual local area network (VLAN) tag, adding a VLAN tag, changing a VLAN tag, deleting a VLAN tag. 14. The system of claim 8, characterized in that it further comprises shared memory, and the processor is configured to create and store in this shared memory a data structure that establishes a correspondence between the identifier of the physical network port, the identifier of the virtual network interface created for it, and at least one logical identifier of the interface used for routing.