[go: up one dir, main page]

|
|
Log in / Subscribe / Register

A firewall for device drivers

By Jonathan Corbet
August 13, 2021
Device drivers, along with the hardware they control, have long been considered to be a trusted part of the system. This faith has been under assault for some time, though, and it fails entirely in some situations, including virtual machines that do not trust the host system they are running under. The recently covered virtio-hardening work is one response to this situation, but that only addresses a small portion of the drivers built into a typical kernel. What is to be done about the rest? The driver-filter patch from Kuppuswamy Sathyanarayanan demonstrates one possible approach: disable them altogether.

Virtual machines typically have direct access to little or no physical hardware; instead, they interact with the world by way of emulated devices provided by the host. That puts the host in a position of power, since it is in total control over how those virtual devices work. If a driver has not been written with the idea that the devices it manages could be hostile, chances are good that said driver can be exploited to compromise the guest and exfiltrate data — even when the guest is running with encrypted memory that is normally inaccessible to the host.

The virtio work hardens a handful of virtio drivers to prevent them from misbehaving if the host decides to not play by the rules. Getting there was a lot of work (which still has not reached the point of being merged), and there is a decidedly non-zero chance that vulnerabilities remain. Even if the virtio work is perfect, though, the kernel contains thousands of other drivers, most of which have not received anything close to the same amount of attention; few of them can be expected to be sufficiently robust to stand up to a malicious device. If the host can convince a guest to load the driver for such a device, the security game may well be over.

One possible solution to this problem is to methodically go through and harden all those thousands of until-now neglected drivers. The result would surely be a better kernel, but holding one's breath for this outcome would be ill-advised. Even if the developer effort for such a project can be found, there is a lot of code that would have to be tested with a large array of devices, a significant number of which stopped being widely available many years ago. Any realistic plan must accept that many drivers will never be hardened in this way.

The alternative is to simply make those drivers unavailable; a driver that cannot run at all is unlikely to compromise the system. Most virtual machines only need a handful of drivers; the rest are just dangerous dead weight. The obvious thing to do is to build a kernel with only the needed drivers, yielding a result that is not only safer, it will also be much smaller. The problem with this idea is that distributors hate the idea of shipping multiple kernels with different configurations. Each one adds to the build, test, and support loads, and it only takes a few configuration options to create a large array of kernel images. Distributors are thus highly motivated to ship a single kernel image if possible.

This is where Sathyanarayanan's patch set comes in; it provides a way for the system administrator to control which drivers are allowed to run. It adds two new command-line options — filter_allow_drivers and filter_deny_drivers — for that purpose; specific drivers can be added to either list using a "bus:driver" notation. The string "ALL" matches anything. So, for example, booting a system with:

    filter_allow_drivers=ALL:ALL

will allow all drivers to run — the default situation. The allow list is applied first and overrides the deny list, so a configuration like this:

    filter_allow_drivers=pci:e1000 filter_deny_drivers=ALL:ALL

will allow the e1000 network adapter driver to run, but will block everything else. There is also a new driver attribute in sysfs (called allowed) that can be used to change a driver's status at run time.

Driver subsystem maintainer Greg Kroah-Hartman was not impressed with this submission; he suggested either building a special kernel image or using the existing mechanisms to block unwanted device drivers instead. These could include denying them in the system's modprobe configuration or using the knobs in sysfs to unbind drivers from their devices. As Andi Kleen explained, though, these mechanisms do not quite satisfy the requirements. Configuring modprobe does not help with built-in drivers and, in any case, the intent is to prevent untrusted drivers from running at all. By the time user space can manually unbind a driver, it has already set itself up in the kernel and may already be trying to drive a malicious device.

Another way of looking at the situation, Kleen added, is to see a guest running on a potentially hostile host as being like a server on the Internet. The server almost certainly runs a firewall to restrict access to ports that are known (or at least hoped) to be safe; the driver filter is the equivalent of the firewall for the guest. That simplifies the hardening problem to the point that it might be feasible.

Whether these arguments will convince Kroah-Hartman remains to be seen; the conversation went quiet without reaching any sort of definitive conclusion. The problem that is driving this work seems real, though; if the current solution does not make the cut, we are likely to see other attempts to do something similar in the future. Devices have gone from hiding behind the kernel to being a part of the kernel's attack surface; security-focused developers will naturally want to reduce that surface as much as possible.

Index entries for this article
KernelDevice drivers
KernelSecurity/Kernel hardening


to post comments

A firewall for device drivers

Posted Aug 13, 2021 18:08 UTC (Fri) by josh (subscriber, #17465) [Link] (2 responses)

This seems awkward to handle via a full list on the kernel command line. I wonder if this could use the existing driver signing infrastructure with a slight extension: Have two (or more) different signatures on each module, with different keys, and on the command-line pass which key "slot" to use to verify modules. That effectively allows selecting among "module sets" determined at compile time.

A firewall for device drivers

Posted Aug 13, 2021 18:43 UTC (Fri) by derobert (subscriber, #89569) [Link] (1 responses)

That has a similar problem as building different kernel versions. If on a VM I administer, what if I need to enable one extra driver? I might be using PCI pass through, for example. I'd have to either switch to the unrestricted set, decreasing security of the VM, or build a new kernel with a third set of signatures.

A firewall for device drivers

Posted Aug 16, 2021 23:23 UTC (Mon) by ssmith32 (subscriber, #72404) [Link]

Or toggle the sysfs knob:

> There is also a new driver attribute in sysfs (called allowed) that can be used to change a driver's status at run time.

A firewall for device drivers

Posted Aug 13, 2021 21:16 UTC (Fri) by bferrell (subscriber, #624) [Link]

We could just abandon the concept of modular kernels, but then we're back to 1995... It's secure. The kernel will have an exactly known set of drivers, but a LOT of modern admins will NOT know how to recompile the kernel to change that set.

I'm all for it!

A firewall for device drivers

Posted Aug 13, 2021 22:05 UTC (Fri) by Paf (subscriber, #91811) [Link] (1 responses)

This seems like a sound and simple solution except that I don’t think the kernel command line can take enough arguments.

A firewall for device drivers

Posted Aug 14, 2021 9:28 UTC (Sat) by amboar (subscriber, #55307) [Link]

Perhaps the patch could exploit bootconfig

https://www.kernel.org/doc/html/latest/admin-guide/bootco...

A firewall for device drivers

Posted Aug 14, 2021 0:10 UTC (Sat) by dullfire (guest, #111432) [Link] (1 responses)

This sounds like a solution in search of a problem

I believe all mainstream distros let you select which drivers should go in the initrd (pretty sure most even have tooling to find a reduced, if not minimal set). As long drivers are built as modules, you can just not include banned drivers in the initrd, and stick modprobe blacklists.

The only situation that doesn't cover is built-in modules (which should both be rare, and usually/always nonsensical to blacklist).

Of course this doesn't include a solution to bootstrap it, but neither does the proposed solution.

A firewall for device drivers

Posted Aug 24, 2021 2:07 UTC (Tue) by ras (subscriber, #33059) [Link]

> As long drivers are built as modules, you can just not include banned drivers in the initrd, and stick modprobe blacklists.

Unfortunately the VM itself can change the blacklists, and obtain the modules from somewhere and insmod them. When your goal is to ensure someone who has taken over the VM can't escape from it, that's not a solution.

However, a simple sysfs "blown fuse flag" (ie, one you can not change back) that turns off module loading would work. You just run set it in the initrd, after it's loaded all the modules. There already is a corresponding capability.

A firewall for device drivers

Posted Aug 14, 2021 0:37 UTC (Sat) by pmulholland (guest, #124686) [Link] (1 responses)

I thought this article was going to be about hardware level firewalls, so I had prepared a comment about PCIe message signalled interrupts and the requirement to write directly to the interrupt controller from an external device... having RTFA, I see I’ll need to wait for another time to see how hardware firewalls are done. I can definitely see how a rough bus master or interrupt source could wreak havoc on a system.

A firewall for device drivers

Posted Aug 14, 2021 21:31 UTC (Sat) by ttuttle (subscriber, #51118) [Link]

I would read that if you wanted to write it and link to it here.

A firewall for device drivers

Posted Aug 14, 2021 19:08 UTC (Sat) by malor (guest, #2973) [Link] (9 responses)

Back in the old days, I used to compile static kernels that had only the exact device drivers I wanted, on the theory that module loading was not secure.

I remain convinced that this is the case, and that static kernels would be safer, completely removing a giant attack space.

This was a long time ago, so I could be misremembering, but I believe I was forced to stop doing this because the kernel stopped supporting static compilation; it was loadable modules or nothing.

A firewall for device drivers

Posted Aug 14, 2021 20:56 UTC (Sat) by pebolle (guest, #35204) [Link] (3 responses)

> kernels that had only the exact device drivers I wanted

A nice idea. I sort of tried to achieve that goal until I realized it would take an insane amount of time to reach it and to stay at it at every release of a new kernel.

> This was a long time ago, so I could be misremembering, but I believe I was forced to stop doing this because the kernel stopped supporting static compilation; it was loadable modules or nothing.

That would mean CONFIG_MODULES (or its equivalent) was broken. I would be surprised if it still is.

A firewall for device drivers

Posted Aug 15, 2021 18:42 UTC (Sun) by jayalane (guest, #133964) [Link]

I used to do this also. Usually once you get a config that builds a kernel that boots your system you use it for the base of new kernels’ config, so the work is mostly one time (this as they mess with new config options etc. you might have to edit it a little, but not so frightful as the first one.

A firewall for device drivers

Posted Aug 20, 2021 13:49 UTC (Fri) by bustervill (guest, #85383) [Link]

> A nice idea. I sort of tried to achieve that goal until I realized it would take an insane amount of time to reach it and to stay at it at every release of a new kernel.

I used to do it as well, spending large amounts of time and attention. I think this is exactly the problem and the reason people (including me) prefer not to have do it anymore.

An XKCD comic is worth a thousand words: https://xkcd.com/1671/

A firewall for device drivers

Posted Aug 21, 2021 0:52 UTC (Sat) by pabs (subscriber, #43278) [Link]

ISTR Linux has a localmodconfig make target that could be used for this; build a full kernel, boot it on the hardware you want, note the loaded modules, and using localmodconfig build a kernel config with those modules and then turn CONFIG_* with =m into =y and build the result.

A firewall for device drivers

Posted Aug 14, 2021 22:49 UTC (Sat) by Wol (subscriber, #4433) [Link] (1 responses)

I still run a (almost) only static kernel. I think you're misremembering when kernels grew too big to fit in the space available for 32-bit. So a lot of stuff had to be compiled as modules to prevent there be loads of unused code that stopped the kernel from loading.

You'd need to do a bit of work, but not that much - gentoo users do it all the time :-) Just download the distro source kernel, do a lsmod to see what modules are loaded, then alter the config so those modules are compiled in. Leave out the "make modules" and "make modules_install" steps, and you should have a module-free kernel.

If you learn there are modules you missed, you can go down one of two routes. Either make a note of them to compile them into your kernel, or manually install them into /lib/modules or wherever they go.

Cheers,
Wol

A firewall for device drivers

Posted Aug 15, 2021 3:59 UTC (Sun) by dullfire (guest, #111432) [Link]

There are even nifty options like "make localmodconfig" which gives you an excellent starting point for selecting which modules you need.

Of course that only copies the currently loaded modules, but the thoery is if you currently running system supports most (or even all) the functionality you need, then those are a good base.

A firewall for device drivers

Posted Aug 15, 2021 20:47 UTC (Sun) by iabervon (subscriber, #722) [Link] (2 responses)

The problem I've had with not having modules is with built-in wifi drivers detecting hardware before the root partition with the firmware is mounted, and not retrying later. Since everybody builds the drivers as modules, the assumption is that the firmware will be available as soon as the driver code is available. I could work around it (unbind and rebind the device once userspace is ready), but I don't really want to use an arrangement that clearly nobody else is testing.

A firewall for device drivers

Posted Aug 15, 2021 22:12 UTC (Sun) by Wol (subscriber, #4433) [Link] (1 responses)

In that case (and I have the same problem with video card firmware), can't you include them in the kernel?

Cheers,
Wol

A firewall for device drivers

Posted Aug 16, 2021 11:45 UTC (Mon) by dullfire (guest, #111432) [Link]

While I have never gotten it to consistently work for GPUs (in theory there is a dependencies list you could jump through... I've never management to find it), for well written Wifi drivers, you should be able to unbind them (via sysfs), and then rebind them. Which should result in a new firmware request.

I believe buildin modules can also pull firmware from the initrd (since firmware loading has been move into the kernel, for reasons).

A firewall for device drivers

Posted Aug 15, 2021 8:02 UTC (Sun) by mokki (subscriber, #33200) [Link]

Could we instead tag each driver as "hardened". Then in virtualized (and other) environments we set a flag to allow only hardened drivers, built-in or not. Plus of course some whitelist, which should be hopefully smaller. At least the effort to review and harden drivers can be shared by community instead of each admin doing it separately.
Maybe the hardening flag could be a bit mask to tell what kind of hardenings have been done on it. Aka input validation, iommu support etc


Copyright © 2021, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds