iproute2 and libbpf: vendoring on the small scale

Iproute2 is a collection of network-configuration tools found on almost any Linux system; it includes utilities like arpd, ip (the command old-timers guiltily think they should be using when they type ifconfig), ss, and tc. That last command, in particular, is used to configure the traffic-control subsystem, which allows administrators to manage and prioritize the flow of network traffic through their systems. This subsystem has, for some years, had the capability to load and run BPF programs to both classify packets and make decisions on how to queue them. This mechanism gives administrators a great deal of flexibility in the management of network traffic.

The code for handling BPF programs within iproute2 is old, though, and lacks support for many of the features that have been added to BPF in the last few years. Since that code was written, the BPF community has developed libbpf (which lives in the kernel source tree) as the preferred way to work with BPF programs and load them into the kernel. This is not a small task; libbpf must interpret the instructions encoded as special ELF sections in an executable BPF program and make the necessary calls to the sprawling bpf() system call. This work can include creating maps, "relocating" structure offsets to match the configuration of the running kernel, loading programs, and attaching those programs to the appropriate hooks within the kernel. Libbpf has grown quickly, along with the rest of the BPF ecosystem.

The obvious way to add support for current BPF features to iproute2 is to dump the old code used there in favor of libbpf; patches to that effect have been posted by Hangbin Liu, starting in late October. Shortly thereafter, David Ahern, one of the iproute2 maintainers, pointed out that the posted patches fail to compile on an Ubuntu 20.10 system; Liu responded that Ahern needed to update to a newer version of libbpf, and libbpf developer Andrii Nakryiko suggested including libbpf as a submodule of iproute2. That is where the trouble began; Ahern asserted that libbpf is provided by distributions and that iproute2 needs to work with the version that is available. Nakryiko argued instead that libbpf is "a fast moving target right now" and that packaging it with iproute2 is "pragmatic". Needless to say, this feeling did not find universal agreement.

The arguments against "vendoring" libraries like libbpf in this way have been made many times, and they appeared here as well. Jesper Dangaard Brouer repeated the usual security argument: copies of libraries buried within other packages are difficult to find and update if a problem is found. Jiri Benc added that the end result of this process would be a massive bundling of dependencies, which "would be nightmare for both distros and users". Distributors have long been opposed to bundling dependencies in this way, and the iproute2 developers do not see any reason to go against this policy for libbpf.

The BPF developers see things differently and were not shy about expressing their feelings. Alexei Starovoitov asserted that Ahern is "deaf to technical arguments" and raised the idea of forking iproute2 in response. He later said that using the distribution-provided libbpf would be worse than not using libbpf at all, and that if he could remove the existing BPF support from iproute2, he would do so. Things reached a point where Toke Høiland-Jørgensen asked him to "stop with this 'my way or the highway' extortion" so that the discussion could make some progress.

The argument for bundling libbpf with iproute2 seems to come down to a distrust in the version of libbpf that distributors will ship. There are two aspects to this, one being that iproute2 could end up linking to a version of libbpf that introduces bugs; as Starovoitov put it:

Benc disagreed:

In a separate branch of the discussion, Starovoitov extolled the compatibility of libbpf releases, saying that "users can upgrade and downgrade libbpf version at any time". Others agreed that libbpf releases are managed well and do not create compatibility problems. So it is not really clear what the concern is here.

The related issue, though, is the worry that using the distribution-provided libbpf will lead to inconsistency across systems and an inability for users to know which features will be supported just by looking at the version of iproute2 they are running. Beyond that, the BPF developers fear that distributors will stick with an old libbpf, depriving users of newer features even if they have a current version of iproute2. Nakryiko said that users do want the newer features supported by libbpf, but BPF maintainer Daniel Borkman repeatedly claimed that distributors cannot be trusted to keep up with libbpf releases. That, it is argued, will lead to a poor experience for iproute2 users. Tying a specific version of libbpf to each iproute2 release, instead, would make life more predictable.

Ahern didn't entirely buy that line of reasoning, though:

That brought out a few other developers who complained about the need to keep multiple components all at the latest versions for things to work properly. Edward Cree, for example, complained about the lack of attention to interoperability in general:

Starovoitov actually agreed that the situation needs to improve — someday. "BPF ecosystem eventually will get to a point of fixed file format, linker specification and 1000 page psABI document." For now, though, things are evolving too quickly for that, he added.

Toward the end, Nakryiko suggested that, if distributors make a point of updating libbpf and setting the dependencies for iproute2 to require the updated versions, things might work well enough. Benc replied that Red Hat, at least, works that way now, leading Nakryiko to reply that this "would be sufficient in practice". Starovoitov still pushed for iproute2 to require a minimum libbpf version that increases with each release so that there would be at least some correlation between the iproute2 and libbpf versions. Stephen Hemminger, who also maintains iproute2, seems unwilling to impose that sort of requirement, though.

In the end, it appears that iproute2 will be packaged like almost all other Linux utilities; it will use the version of libbpf shipped by the distribution it is running on rather than supplying its own. The distributors will just have to be sure that they keep libbpf current with other kernel-derived utilities; this is something they have long since learned to do.

The tensions around the quickly evolving BPF subsystem seem unlikely to go away, though; one group of users is already relying on BPF-based tools while the developers continue to evolve the whole ecosystem in a hurry. Ahern described the situation this way: "The trend that related s/w is outdated 2-3 months after a release can be taken as a sign that bpf is not stable and ready for distributions to take on and support". But that, too, should be a tractable problem; Linux as a whole has been supporting users while undergoing constant modification for nearly 30 years, after all.

label NAME --- Each address may be tagged with a label string.

In order to preserve compatibility with Linux-2.0 net aliases, this string must coincide
with the name of the device or must be prefixed with device name followed by a colon. (eth0:duh)

2: wlp59s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 04:ed:33:4c:b4:4a brd ff:ff:ff:ff:ff:ff
    inet 192.168.86.185/24 brd 192.168.86.255 scope global dynamic noprefixroute wlp59s0
       valid_lft 71064sec preferred_lft 71064sec
    inet6 2604:4080:1321:8090:308c:1c2:a8b0:a941/64 scope global dynamic noprefixroute 
       valid_lft 3075sec preferred_lft 1275sec
    inet6 fe80::ddec:2fe4:a8b6:842f/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    3125287182 3018779  0       246     0       0       
    TX: bytes  packets  errors  dropped carrier collsns 
    403122417  957242   0       0       0       0       

wlp59s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.86.185  netmask 255.255.255.0  broadcast 192.168.86.255
        inet6 fe80::ddec:2fe4:a8b6:842f  prefixlen 64  scopeid 0x20<link>
        inet6 2604:4080:1321:8090:308c:1c2:a8b0:a941  prefixlen 64  scopeid 0x0<global>
        ether 04:ed:33:4c:b4:4a  txqueuelen 1000  (Ethernet)
        RX packets 3018891  bytes 3125321368 (3.1 GB)
        RX errors 0  dropped 246  overruns 0  frame 0
        TX packets 957317  bytes 403128107 (403.1 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0