A rootkit dissected

A recently discovered Linux rootkit has a number of interesting attributes that make it worth a look. While it demonstrates the power that a rootkit has (to perform its "job" as well as hide itself from detection) this particular rootkit also has some fairly serious bugs—some that are almost comical. What isn't known, at least yet, is how the system where it was discovered became infected; there is no exploit used by the rootkit to propagate itself.

The rootkit was reported to the full-disclosure mailing list on November 13 by "stack trace". Customers had noticed that they were being redirected to malicious sites by means of an <iframe> in the HTTP responses from Stack Trace's site. Stack Trace eventually found that the Nginx web server on the system was not delivering the <iframe> and tracked it to a loadable kernel module, which was attached to the posting. Since then, both CrowdStrike and Kaspersky Lab's Threatpost have analyzed the module's behavior.

The first step for a rootkit is to get itself running in the kernel. That can be accomplished by means of a loadable kernel module. In this case, the module is targeted at the most recent 64-bit kernel version used by Debian 6.0 ("Squeeze"):

    /lib/modules/2.6.32-5-amd64/kernel/sound/module_init.ko 

The rootkit targets HTTP traffic, so that it can inject an <iframe> containing an attack: either a malicious URL or some kind of JavaScript-based attack. In order to do that in a relatively undetectable way, it must impose itself into the kernel's TCP send path. It does so by hooking tcp_sendmsg().

Of course, that function and other symbols that the rootkit wants to access are not exported symbols that would be directly accessible to a kernel module. So the rootkit uses /proc/kallsyms to get the addresses it needs. Amusingly, there is code to fall back to looking for the proper System.map to parse for the addresses, but it is never used due to a bug. Even though the kernel version is hardcoded in several places in the rootkit, the System.map helper function actually uses uname -r to get the version. The inability to fall back to checking System.map, along with this version-getting oddity make it seem like multiple people—with little or no inter-communication—worked on the code. Other odd bugs in the rootkit only add to that feeling.

For example, when hooking various functions, the rootkit carefully saves away the five bytes it needs to overwrite with a jmp instruction, but then proceeds to write 19 bytes at the start of the function. That obliterates 14 bytes of code, which eliminates any possibility of unhooking the function. Beyond that, it can't call the unhooked version of the function either, so the rootkit contains private copies of all the functions it hooks.

Beyond hooking tcp_sendmsg(), the rootkit also attempts to hide its presence. There is code to hide the files that it installs, as well as its threads. The file hiding works well enough by hooking vfs_readdir() and using a list of directories and files that should not be returned. Fortunately (or unfortunately, depending on one's perspective), the thread hiding doesn't work at all. It uses the same file-hiding code, but doesn't look in /proc nor convert the names into PIDs, so ps and other tools show the threads. In the original report, Stack Trace noted two threads named get_http_inj_fr and write_startup_c; those names are fairly descriptive given the behavior being seen. The presence of one or both of those names in the process list would mean that the system has the rootkit loaded.

The rootkit does successfully remove itself from the list of loaded modules. It directly iterates down the kernel's module list and deletes the entry for itself. That way lsmod will not list the module, but it also means that it cannot be unloaded, obviating the "careful" preparations in the hooked functions for that eventuality.

As with other malware (botnets in particular), the rootkit has a "command and control" client. That client contacts a particular server (at a hosting service in Germany) for information about what to inject in the web pages. There is some simple, weak encryption used on the link for both authentication and obfuscation of the message.

Beyond just missing a way to propagate to other systems, the rootkit is also rather likely to fail to persist after a reboot. It has code to continuously monitor and alter /etc/rc.local to add an insmod for the rootkit module. It also hooks vfs_read() to look for the exact insmod line and adjusts the buffer to hide that line from anyone looking at the file. But it just appends the command to rc.local, which means that on a default installation of Debian Squeeze it ends up just after an exit 0 line.

Like much of the rest of the rootkit, the HTTP injection handling shows an odd mix of reasonably sensible choices along with some bugs. It looks at the first buffer to be sent to the remote side, verifies that its source port is 80 and that it is not being sent to the loopback address. It also compares the destination IP address with a list of 1708 search engine IP addresses, and does no further processing if it is on the list.

One of the bugs that allowed Stack Trace to diagnose the problem is the handling of status codes. Instead of looking for the 200 HTTP success code, the rootkit looks for three strings on a blacklist that correspond to HTTP failures. That list is not exhaustive, so Stack Trace was able to see the injection in a 400 HTTP error response. Beyond that, the rootkit cleverly handles chunked Transfer-Encodings and gzip Content-Encodings, though the latter does an in-kernel decompress-inject-compress cycle that could lead to noticeable server performance problems.

None of the abilities of the rootkit are particularly novel, though it is interesting to see them laid bare like this. As should be obvious, a rootkit can do an awful lot in a Linux system, and has plenty of ways to hide its tracks. While this rootkit only hid some of its tracks, some of that may have happened after the initial development. The CrowdStrike conclusion is instructive here: "Rather, it seems that this is contract work of an intermediate programmer with no extensive kernel experience, later customized beyond repair by the buyer."

The question of how the rootkit was installed to begin with is still open. Given the overall code quality, CrowdStrike is skeptical that some "custom privilege escalation exploit" was used. That implies that some known but unpatched vulnerability (perhaps in a web application) or some kind of credential leak (e.g. the root password or an SSH key) was the culprit. Until and unless some mass exploit is used to propagate an upgraded version of the rootkit, it is really only of academic interest—except, of course, to anyone whose system is already infected.