Standardizing BPF

The extended BPF (eBPF) virtual machine allows programs to be loaded into and executed with the kernel — and, increasingly, other environments. As the use of BPF grows, so does interest in defining what the BPF virtual machine actually is. In an effort to ensure a consistent and fair environment for defining what constitutes the official BPF language and run-time environment, and to encourage NVMe vendors to support BPF offloading, a recent effort has been undertaken to standardize BPF.

BPF programs are written in C, and compiled into BPF bytecode. Like other bytecode instruction sets, BPF programs are platform-independent and just-in-time (JIT) compiled. For a long time, "platform-independent" for BPF simply meant the ability to run BPF programs on multiple different architectures on Linux. That definition has expanded in recent years, with Microsoft implementing a version of BPF for Windows, and network-interface vendors, such as Netronome, providing the ability to offload BPF networking programs. NVMe vendors are also looking into supporting offloading functionality to BPF for storage devices with a new framework called eXpress Resubmission Path (XRP), though this effort is currently stalled due to BPF not being standardized.

What's in scope for standardization?

BPF is not simply an instruction set, but rather a combination of an instruction set and a run-time environment. The latter must, at a minimum, include the necessary logic to execute the BPF program; either through an interpreter, or by JIT-compiling the program directly into native instructions. Additionally, it may include features such as static verification of the program, performing type checking using information provided via BTF, built-in data structures, and more. While the BPF instruction set architecture (ISA) is in scope for standardization, it's less clear which other aspects of BPF are appropriate.

This question was posed by Christoph Hellwig in his 2022 LSFMM/BPF presentation. While all of the participants in the discussion agreed that standardizing the ISA is the highest priority, there was also discussion about whether to standardize certain run-time semantics such as what happens when a program divides by zero. In the discussion, Alexei Starovoitov explained that, initially, BPF would simply exit the program if a divide by zero was encountered. After realizing that abruptly exiting a program can be dangerous (it may, for example, have needed to clean up some state), the semantics were changed to instead simply return zero and produce no exception; matching the behavior of aarch64.

The discussion concluded with a general agreement that the first order of business was to fully document and decide on a versioning system for the ISA.

In a follow-up email, Hellwig suggested that the ISA should be versioned according to Clang CPU levels — the "processor version" used by Clang when compiling BPF programs. Starovoitov pointed out in response that multiple instructions have, in the past, been added to the BPF ISA without a bump in the Clang CPU level, so the CPU levels weren't a clean match with the BPF ISA versions. Starovoitov suggested a number of other approaches, such as versioning with an upstream kernel commit hash, or simply declaring the current ISA as 1.0 and bumping it for every new instruction. Hellwig was unenthusiastic about the idea of using kernel commit hashes, but was amenable to the idea of considering the current ISA as version 1.0.

Following these discussions, the BPF ISA documentation has been improved significantly, with all of the current instructions being fully documented. The documentation page lists the instruction set as v1.0, so it would seem that Starovoitov's idea of treating the current ISA as v1.0 was chosen as the way forward.

Yet, while the current ISA is fully documented, there are still new instructions being added that will presumably be included in the official v1.0 BPF ISA. Yonghong Song proposed a set of six such instructions to be included in the new -mcpu=v4 Clang CPU level. These will surely not be the last instructions added to the ISA, but for now they appear to be the last instructions that will be added to v1.0.

Choosing a standards organization

In addition to finalizing the ISA and deciding what else is in scope for standardization, there is another important question to resolve before standardization can begin in earnest: with which organization will the standard be ratified?

The natural choice is the eBPF Foundation, which was founded as a subsidiary of the Linux Foundation in December 2021; the foundation is responsible for managing both the finances and the technical direction of the BPF project. For technical matters, the foundation has a steering committee composed of engineers from various companies throughout the tech industry. Were BPF to be standardized through the eBPF Foundation, the steering committee would presumably be the responsible party.

Standardizing through the eBPF Foundation would likely be the most straightforward option, incurring the smallest amount of latency in achieving consensus. It does, however, have a major drawback: the eBPF Foundation has never published a standard. This, on its own, isn't necessarily a hard blocker for publishing (every organization had to publish their first standard at some point), but it does mean that the eBPF Foundation would have to go through the standardization process without the benefit of prior experience. In this regard, while the bureaucracy of a more recognized organization could be considered a pain point, it could also be considered a feature if that organization's processes and experience help to ensure that the standard is well considered and of the highest quality. On the other hand, some members of the steering committee, such as Dave Thaler, have experience from working with other standards organizations such as the Internet Engineering Task Force (IETF).

One alternative to publishing with the eBPF Foundation is publishing directly through ISO, an international standards organization that is home to the C programming language standard, among others, that we all know and love. Standardizing with ISO would likely guarantee the strongest possible worldwide consensus, as it is an international standards body with a rigorous and widely reviewed ratification process. For that reason, it is also likely to be the most difficult and time-consuming option. While I am by no means an expert in the domain of standardizing with ISO, it appears that, in order to even consider standardizing BPF with the ISO, the standard would first have to be brought before the American National Standards Institute (ANSI), (the ISO member representing the US), which would then propose the idea to the larger international ISO community. Ratifying the BPF standard with the ISO may be a desirable long-term goal, but seems unlikely to be the approach taken for the initial standardization effort.

IETF discussions

Yet another alternative is standardizing with the IETF, which is best known for creating the standards that comprise the "Internet Protocol Suite", more commonly known as TCP/IP, though it also publishes standards for non-networking topics such as file formats. The IETF is also an international standards body, though its process for standardization is less onerous than the ISO. As such, it may represent an ideal middle ground between the eBPF Foundation and the ISO.

Discussions have been ongoing between members of the BPF and IETF communities, including on an IETF mailing list, following a BPF standardization meeting at the IETF 115 conference in 2022 as to whether IETF is an appropriate venue for BPF standardization. The topic was recently revisited at IETF 116 in 2023. Despite there being some vocal opponents among the attendees, the overall consensus in favor of standardizing BPF was apparently quite strong relative to the norm for IETF. Jari Arkko, an IETF Area Director (AD), posted this summary on the IETF BPF mailing list:

The chairs asked if the room felt the problem was well defined and scoped. The meeting was almost unanimous that it was. Same for recommending to start the work. The community seems to want the work to go ahead by a larger level of consensus than we're normally used to in IETF BOFs.

Yet, while IETF as an organization seems enthusiastic to move forward (pending some legal matters as discussed below), Arkko also pointed out that more work needs to be done in terms of formally defining what is in scope for standardization:

I do have one concern however. I think the meeting discussed the issues only in the abstract, and spent almost no time discussing the actual list of work items. There's a draft list of work items in the charter (https://datatracker.ietf.org/doc/charter-ietf-bpf/), and the room hums seemed to say that the charter is acceptable. However, to what extent has this been discussed on list or somewhere else? I personally thought some items were quite clearly feasible while I wasn't so sure of others.

Arkko certainly has a point. As discussed above, the scope of standardization for BPF could be broad. If BPF proceeds with the IETF, achieving consensus on what will be in scope for the first publication of the standard will certainly be one of the major work items.

Now that some sort of consensus has been achieved in the IETF community, it seems likely that BPF standardization will proceed through the IETF. Before work can formally begin in earnest, however, there are still a few legal matters to finalize. The co-chairs of the IETF 116 BPF BoF meeting, Suresh Krishnan and Lorenzo Colitti, mentioned that the IETF legal counsel was still doing due diligence on some questions related to licensing and copyright. Though these legal matters are expected to be resolved without issue, final approval has yet to be given. Assuming there are no legal hiccups, the next step would be to formally create an IETF working group, which would likely be co-chaired by Krishnan and myself.

Worth noting as well is that BPF is not the first major subsystem in the kernel that is undergoing a standardization effort. Virtio was first standardized through the Organization for the Advancement of Structured Information Standards (OASIS) back in March 2016, and there are lessons to be learned from that effort. For instance, Rusty Russell, who led this effort, also made it a point to shop around for different standards organizations. According to the LWN article linked above, he was warned that, "some standards groups exist primarily to slow things down", which did not suit his goals of finding an organization that was "interested in the creation of useful standards without a lot of unnecessary hoops to jump through." The BPF community will have similar goals of its own and, so far, it seems that it is following virtio's example of putting in the legwork to find an organization whose processes match the project's needs.

It will be interesting to see where the standardization effort goes from here. Some interested parties, such as the aforementioned NVMe vendors, seem to be deferring substantial investment until BPF is fully standardized. There is thus a significant incentive for the effort to proceed. In the meantime, we can at least enjoy the steady stream of high-quality BPF documentation inspired by this standardization effort.