Exceptions in BPF

The BPF virtual machine in the kernel has been steadily gaining new features for years, many of which add capabilities that C programmers do not ordinarily have. So, from one point of view, it was only a matter of time before BPF gained support for exceptions. As it turns out, though, this "exceptions" feature is aimed at a specific use case, and its use in most programs will be truly exceptional.

Kumar Kartikeya Dwivedi posted the BPF exceptions patch set on July 13. The API presented to BPF programs is simple, taking the form of two kfuncs. To raise an exception, a BPF program can call:

    void bpf_throw(u64 cookie);

A call to bpf_throw() will cause the program's entire call stack to be unwound, and the program will return to its caller with (by default) a return status of zero; the cookie value is ignored. There is no way for a program to catch an exception called further down the call stack. It is, however, possible to define a function to be called after the call stack has been unwound, but before control is returned to the caller:

    void bpf_set_exception_callback(int (*callback)(u64));

The given callback() will be called once unwinding is complete, and will be passed the cookie value given to bpf_throw(); its return value will then be returned to the original caller of the BPF program. There can be only one bpf_set_exception_callback() call in a program; once the callback is set, it cannot be changed.

One might thus be forgiven for thinking that this exception mechanism does not look like it does in other languages supporting the feature, and that bpf_throw() might better be spelled exit(). It clearly is not meant to allow BPF programs to catch and respond to unusual situations. The use case for exceptions, as it turns out, is different and unique to BPF.

BPF programs must, famously, convince the kernel's verifier that they are safe to run before they can be successfully loaded. Doing so requires handling every possible case — even cases that the programmer knows can never happen, but which the verifier is less certain about. So, for example, a BPF function far down the call stack might have to check that an integer value is within a given range, even though the developer knows that it must be, because the verifier does not know that. The check must do something reasonable in response to an out-of-bounds value and, perhaps, return a failure status all the way back up the call chain, all for a case that can never happen.

And, as we all know, developers are never wrong about cases that can never happen.

As Dwivedi described, exceptions are intended to address this problem:

The primary requirement was for implementing assertions within a program, which when untrue still ensure that the program terminates safely. Typically this would require the user to handle the other case, freeing any resources, and returning from a possibly deep callchain back to the kernel. Testing a condition can be used to update the verifier's knowledge about a particular register.

So, in other words, the real reason for exceptions is to provide a mechanism by which the verifier can be informed of invariants that the developer knows about while having an emergency exit mechanism for those times when the developer is wrong. There is a set of assertion macros provided to make this feature easily available in BPF programs. So, for example, a developer will be able to write:

    bpf_assert_lt(foo, 256);

This assertion will perform the indicated test and, should it fail, make a call to bpf_throw(). Meanwhile, the verifier will be able to use the knowledge that foo is, indeed, less than 256 as it evaluates the subsequent code.

There is one notable problem still, as described in the changelog to this patch in the series:

For now, bpf_throw invocation fails when lingering resources or locks exist in that path of the program. In a future followup, bpf_throw will be extended to perform frame-by-frame unwinding to release lingering resources for each stack frame, removing this limitation.

Given that the verifier is now counting on bpf_throw() to prevent execution from proceeding past a failed assertion, this seems like a significant limitation indeed. It could probably be used by a sufficiently malicious developer to convince the verifier to accept a program that does something unpleasant. That suggests that implementing the frame-by-frame unwinding will be a prerequisite to getting this work merged.

Both BPF and Rust are intended to make kernel programming safer, but they take a different approach to the problem. A Rust program will, by default, panic if any of a large number of things goes wrong. BPF programs, instead, are intended to be verified as simply lacking that sort of wrong behavior before they are ever allowed to execute. BPF exceptions can be seen as an admission that the "prove correctness before loading" approach has its limits, and that sometimes it is necessary to just throw up your hands and bail out.

Index entries for this article
Kernel	BPF
Kernel	Releases/6.7

---

to post comments

Exceptions in BPF

Posted Jul 21, 2023 16:25 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

Guys, really? You are now adding freaking exceptions.

This is not just "feature creep", it's a "feature runaway train at 100mph".

Exceptions in BPF

Posted Jul 21, 2023 16:55 UTC (Fri) by adobriyan (subscriber, #30858) [Link]

Superficially kuint devs did exceptions first:

struct kunit_try_catch {
/* private: internal use only. */
struct kunit *test;
struct completion *try_completion;
int try_result;
kunit_try_catch_func_t try;
kunit_try_catch_func_t catch;
void *context;
};

Exceptions in BPF

Posted Jul 21, 2023 20:00 UTC (Fri) by Smon (guest, #104795) [Link] (2 responses)

Couldn't a compiler simulate this already with existing functionality?

Exceptions in BPF

Posted Jul 22, 2023 16:00 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (1 responses)

Maybe. Sort of. With a lot of effort.

The problem is that you need to return all the way up the stack, and therefore you need to return some kind of "we're bailing out" status code to indicate the problem to the caller. But you probably already have return values at many of those call sites, so now you need to transform those return values in some way, probably into an option type or tagged union (or some equivalent). In principle that should be possible, but I don't know if BPF makes it straightforward or performant.

Exceptions in BPF

Posted Jul 24, 2023 4:13 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

One of the bragging points of BPF was "but it always returns a value!", and that's why it's apparently better than WASM.

With the addition of exceptions, this guarantee is lost.

Not that it mattered either way in practice, but still. BPF is now just adding features without even considering their impact on the overall BPF model.

Exceptions in BPF

Posted Jul 22, 2023 8:42 UTC (Sat) by jezuch (subscriber, #52988) [Link] (6 responses)

Maybe this thould be renamed as "panic", because it's doing the same thing as panic in Rust. These are no generic exceptions.

Exceptions in BPF

Posted Jul 22, 2023 9:15 UTC (Sat) by softball (subscriber, #160655) [Link] (3 responses)

A core difference is: BPF looks to treat throws as success-by-default (rc 0), Rust panics are failure-by-default. Rust panics have to be caught explicitly. My experience with Rust is limited so far, but catching panics seems rare and unidiomatic: there's already featureful error handling available (Result type), so any panics are usually serious enough to not be handlable.

Exceptions in BPF

Posted Jul 23, 2023 7:35 UTC (Sun) by xi0n (guest, #138144) [Link] (2 responses)

Panics in idiomatic Rust are only caught to prevent them from crossing an FFI boundary, i.e. escaping a Rust callback that’s been called from C. Otherwise, programs can set a global panic handler if the standard one is inadequate, but that is only really used for embedded and other no_std environments.

This saying, the mechanism proposed here is so close to Rust panics (and Go panics) that NOT calling it such will only lead to confusion, esp. when “exception” is such an overloaded term already.

Exceptions in BPF

Posted Jul 25, 2023 6:59 UTC (Tue) by taladar (subscriber, #68407) [Link]

Setting a custom panic handler is also frequently used in situations where the panic should be logged before exiting.

Exceptions in BPF

Posted Nov 5, 2023 13:03 UTC (Sun) by ibukanov (subscriber, #3942) [Link]

Go panics are exactly the exceptions. They can be caught at arbitrary points and can pass arbitrary values across the stack.

Exceptions in BPF

Posted Jul 22, 2023 12:07 UTC (Sat) by dezgeg (guest, #92243) [Link] (1 responses)

Sounds it would be too confusing when 'Kernel panic' is already a thing

Exceptions in BPF

Posted Jul 23, 2023 18:48 UTC (Sun) by rqosa (subscriber, #24136) [Link]

Maybe naming it "error" (or something like that) would be a reasonable choice that isn't too similar to those 2 other terms?

(Java uses the word "error" in a similar sense in java.lang.Error's name — i.e. for "serious problems that a reasonable application should not try to catch".)

Exceptions in BPF

Posted Jul 21, 2023 23:07 UTC (Fri) by randomguy3 (subscriber, #71063) [Link] (1 responses)

I'd argue that rust borrow checks and panics are conceptually very similar to bpf verification and exceptions. In both cases, you are attempting to eliminate certain classes of failures before running the program, and providing a mechanism to assert invariants and cleanly handle bugs.

The difference is primarily scope: as a general purpose language, rust's borrow checker is weaker than bpf's verification (with a few escape hatches provided as well), and its use of asserts/panics more extensive.

Exceptions in BPF

Posted Jul 22, 2023 9:07 UTC (Sat) by softball (subscriber, #160655) [Link]

A core aspect they have in common is to err on the side of false-positives. False-negatives would be disastrous, and perfect, 1:1 deduction isn't possible (halting problem), so the only reasonable thing to do is disallow certain known-good (to the developer) scenarios.

Exceptions in BPF

Posted Jul 22, 2023 1:30 UTC (Sat) by walters (subscriber, #7396) [Link] (2 responses)

> A Rust program will, by default, panic if any of a large number of things goes wrong

I don’t think so. If you’re talking about Rust code using std certainty there can be a lot of implicit OOM panics. But kernel Rust doesn’t use std.

Idiomatic Rust avoids gratuitous unwrap invocations and array accesses, etc. Iterators can often remove implicit bounds checks too. Now, writing probably panic-free code is an active topic.

But I don’t think “if any of a large number of things” is really accurate, on balance.

Exceptions in BPF

Posted Jul 22, 2023 4:24 UTC (Sat) by wahern (subscriber, #37304) [Link] (1 responses)

> If you’re talking about Rust code using std certainty there can be a lot of implicit OOM panics. But kernel Rust doesn’t use std.

True, kernel Rust has its own standard library, but most of it seems to be littered with '#[cfg(not(no_global_oom_handling))]' just like the userspace std library.

Exceptions in BPF

Posted Jul 22, 2023 18:28 UTC (Sat) by walters (subscriber, #7396) [Link]

> True, kernel Rust has its own standard library, but most of it seems to be littered with '#[cfg(not(no_global_oom_handling))]' just like the userspace std library.

(We're now a bit past my relatively superficial knowledge of Linux kernel Rust but...)

I'm pretty sure that's because they don't want a long term fork of alloc.rs and vec.rs etc. The upstream Rust project has this config option I'm pretty sure *precisely* to help enable usage of the upstream battle-tested collections while disabling the APIs that will implicitly panic on OOM.

And the Linux build system does pass "--cfg no_global_oom_handling". And the example Rust code does use try_push() not push(), etc.

Or to say this simply: kernel Rust does not have implicit panics on OOM.

What would certainly be interesting to try to evaluate is how many possible panics there are in any nontrivial kernel Rust code. The classic example is somevec[offset]. I'd expect the number is greater than zero. But are there "a lot"? Are there enough where it *actually* feels like "panic if any of a large number of things goes wrong" is true? I'm doubtful.

Rust and static safety

Posted Jul 22, 2023 1:32 UTC (Sat) by geofft (subscriber, #59789) [Link] (1 responses)

> A Rust program will, by default, panic if any of a large number of things goes wrong. BPF programs, instead, are intended to be verified as simply lacking that sort of wrong behavior before they are ever allowed to execute.

Well... kind of. There are indeed a large number of things that cause a Rust panic (which, for kernelspace Rust code, turns into an oops, not a kernel panic), such as indexing an array out of bounds. But there are also a large number of things that are verified by the Rust compiler, preventing wrong behavior before it can ever execute, and I think that's sort of the selling point of Rust!

A good example is the venerable null pointer. It's not actually a pointer - it's someone using a pointer type to convey there is nothing to point to. In C, you can attempt to "dereference" a null pointer, which is a fundamentally meaningless operation that will lead to incorrect behavior. In Rust, pointer types are defined as non-null, and the standard Optional data type when wrapping a pointer ends up in memory just like a nullable C pointer, but the language prevents you from using it if the value is equal to zero. An Optional type cannot be used directly; you have to use an if or match statement that breaks down the two possibilities, Some actual value or None. (Or you can call a function that does so - such as the standard .unwrap() function which will generate a Rust panic if it's None.)

In other words, at compile time, a Rust program can be verified as never dereferencing a null reference.

This can be generalized in a few ways. Rust also ensures that all references are to valid data, which is what the feared "borrow checker" does: if the compiler can't be convinced that the pointed-to data is still around when you're using the pointer, it will fail to compile. You can also imagine data types that have more than just null as special cases, such as the kernel's ERR_PTR scheme, where small negative values are actually errnos. A function that returns a char * might actually return (char *)-ENOMEM, aka ERR_PTR(-ENOMEM), and expect callers to check IS_ERR on the pointer before using it, with bad consequences if they forget. In Rust this is better defined as a data type that can be either a pointer or an error code (and indeed rust/kernel/error.rs defines it this way): you can't misinterpret an error code as a pointer or vice versa. Currently this isn't stored all in one pointer-sized data type the way it is in C, but it will be soon.

The big difference between Rust and BPF in this context is that Rust treats all of these tools as aids to the programmer, which can be bypassed if needed if you're doing something complex, and BPF treats these as hard requirements and simply refuses to let you do complex things. Rust is trying to eliminate common types of mistakes, but it's not intended to be used in a way where the output is more highly privileged than the input. So it is possible (and quite common for interop with C) to "unsafely" produce a reference from somewhere, effectively telling Rust, hey trust me on this one, this is a valid pointer even though I can't prove it to you. BPF, on the other hand, is all about allowing userspace to load programs into the kernel without the security risk of loading a real kernel module. So it can't have any bypass mechanisms. (And so it needs something like the mechanism in this article to say, I can't prove this assumption to you, so you can just evaluate whether it's true at runtime and bail out of executing the program if it isn't.)

Rust and static safety

Posted Jul 22, 2023 16:19 UTC (Sat) by randomguy3 (subscriber, #71063) [Link]

this is kind of what I was trying to get at with my comment, but you did a much better job!

Exceptions in BPF

Posted Jul 23, 2023 2:44 UTC (Sun) by iteratedlateralus (guest, #102183) [Link]

I think this is the natural progression of things technological. It becomes easier and easier to build abstractions on top of something that used to be a lot of hard work to accomplish even the most basic of tasks. I think the BPF feature set should be purposefully limited in scope due to the valuable attack surface that features create.