System calls and rootkits

A patch to add some security checks before making system calls would seem like a reasonable addition to the kernel, but because it is, at best, a half-measure, it received a less than enthusiastic response. Preventing rootkits—malware that alters the kernel to hide its presence and function—from altering the system call table was the rationale behind the patch, but it would only work for the current crop of rootkits. Once that change was made, rootkit authors would just change their modus operandi in response.

There are many possible ways that a root user—or malware running as root—can modify a Linux system to run rootkit code. Some currently "popular" rootkits modify the system call table, though it is ostensibly read-only. Some commercial malware scanners that run on Linux have also been known to use this technique. In both cases, certain system calls are re-routed from the standard kernel code to code that lives elsewhere. That code, running in kernel mode, can then do just about anything it wants with the system.

Arjan van de Ven proposed a patch that hooked into the system call entry code to check the address of the call to ensure that it was within the addresses occupied by kernel code. He describes the change and its impact this way:

Various kernel hackers pointed out the flaws inherent in that scheme. As Andi Kleen succinctly puts it:

One of the more interesting ideas to come out of the discussion was Alan Cox's thoughts on using a hypervisor to enforce protections:

Ingo Molnar described a rather complicated scheme that might increase the likelihood of a rootkit being detected, but with a fairly high cost—in build complexity as well as the ability to debug the resulting kernel. The compiler would be changed to insert calls to rootkit checks randomly throughout the kernel binary in ways that would be difficult or impossible for a rootkit to detect and evade. In the end, though, a rootkit could simply install a new kernel that does exactly what it wants, then cause, or wait for, a reboot.

Without some kind of hardware enforcement (e.g. Trusted Platform Module) or locked-down virtualization, Linux is defenseless against attacks that run as root. The kernel could change to thwart a particular kind of attack, such as van de Ven's patch, but other kinds of attacks will still succeed. It is clearly a situation where "the only way to win is not to play this game", as Pavel Machek—amongst others—noted in the thread.

In the end, van de Ven wrote off the patch as an exercise in measuring the cost of this kind of runtime checking. It was fairly low cost solution, but without any major upside. The real upside was getting kernel hackers thinking about the problem, which could lead to some better solutions down the road.