How the XZ backdoor works

Versions 5.6.0 and 5.6.1 of the XZ compression utility and library were shipped with a backdoor that targeted OpenSSH. Andres Freund discovered the backdoor by noticing that failed SSH logins were taking a lot of CPU time while doing some micro-benchmarking, and tracking down the backdoor from there. It was introduced by XZ co-maintainer "Jia Tan" — a probable alias for person or persons unknown. The backdoor is a sophisticated attack with multiple parts, from the build system, to link time, to run time.

The community response to the attack is just as interesting as the technical aspects. For more information on that, refer to this companion article.

Build time

The backdoor consists of several distinct phases, starting when the package is being built. Gynvael Coldwind wrote an in-depth investigation of the build-time parts of the backdoor. Releases of XZ were provided via GitHub, which has since disabled the maintainers' accounts and taken the releases offline. Like many projects that use GNU Autoconf, XZ made releases that provided several versions of the source for download — an automatically generated tarball containing the source and related files in the repository, along with versions containing the generated build files. Those extra files include the configure script and makefiles for the project. Releasing versions that contain the generated files allows downstream users of the software to build without needing to install Autoconf.

In this case, however, the scripts in the maintainer-provided source tarballs were not generated by Autoconf. Instead, one of the build scripts contained the first stage of the exploit in m4/build-to-host.m4. This script is originally from the Gnulib library; it provides a macro that converts between the style of pathname used by the build environment and the run-time environment of the program. The version in these XZ releases was modified to extract the next stage of the exploit, which is contained in tests/files/bad-3-1corrupt_lzma2.xz.

This file is included in the repository, ostensibly as part of XZ's test suite, though it was never used by those tests. It was committed well before the release of version 5.6.0. The file, supposedly a corrupted XZ file, is actually a valid XZ stream with some bytes swapped — for example, 0x20 is swapped with occurrences of 0x09 and vice versa. When decoded, it yields a shell script that unpacks and executes the next stage of the backdoor.

The next stage of the backdoor is located in tests/files/good-large_compressed.lzma. This is the injected.txt file attached to Freund's message. That file contains more than just the next stage of the script — it also contains additional binary data that forms the actual backdoor itself. The final script skips over the header of the file from which it was extracted, and then uses awk to decrypt the remainder of the file. Finally, that decrypted stream is decompressed using the XZ command-line program, in order to extract a pre-compiled file called liblzma_la-crc64-fast.o, which is also attached to Freund's message.

Link time

The extracted file is a 64-bit relocatable ELF library. The remainder of the build process links it into the final liblzma library which ends up being loaded into OpenSSH on some distributions. Those distributions patch OpenSSH to use systemd for daemon-readiness notifications; libsystemd in turn depends on liblzma for compressing journal files. Lennart Poettering has since posted some example code (written by Luca Boccassi) showing how to let applications use systemd readiness notifications without pulling in the entire library. When the malicious liblzma is used by a dynamically linked process, it uses the indirect function mechanism to involve itself in the linking process.

Indirect functions are a feature of the GNU C library (glibc) that permits a developer to include several versions of a function and select which version to use at dynamic linking time. Indirect functions are useful for including optimized versions of a function that rely on specific hardware features, for example. In this case, the backdoor provides its own version of the indirect function resolvers crc32_resolve() and crc64_resolve() that select versions of crc32() and crc64() to use, respectively. liblzma does not usually use indirect functions, but using faster functions to calculate checksums does sound like a plausible use of the feature. This plausible deniability is probably why the exploit itself lives in a file called liblzma_la-crc64-fast.o.

When the dynamic linker finalizes the locations of those functions, it calls the backdoor's resolver functions. At this point, dynamic linking is still in progress, so many of the linker's internal data structures have not yet been made read-only. This would let the backdoor manipulate libraries that had already been loaded by overwriting entries in the procedure linkage table (PLT) or global offset table (GOT). However, liblzma is loaded fairly early in the link order of OpenSSH, which means that the OpenSSL cryptography functions that are the backdoor's ultimate target may not have been loaded yet.

To deal with that, the backdoor adds an audit hook. The dynamic linker calls all the registered audit hooks when it is resolving a symbol. The backdoor uses this to wait until it sees the RSA_public_decrypt@got.plt symbol being resolved. Despite the name, this function is actually part of handling an RSA signature (which is a decryption operation) — OpenSSH calls it while validating an RSA certificate provided by the client during a connection.

Run time

Once the backdoor detects this function being linked, it replaces the function with its own version. What the altered version does is still being investigated, but at least one of its functions is to attempt to extract a command from the public-key field of the provided RSA certificate (which means that certificates that are used in this attack cannot actually be used to authenticate normally). The backdoor checks whether the command is signed by the attacker's private key and has valid formatting. If it does, then the backdoor directly runs the given command as the user running sshd, which is usually root.

Anthony Weems has put together an explanation of the run-time portion of the exploit, including a honeypot to detect attempts to use the exploit, and code to generate command payloads. Using the backdoor involves signing the command to be executed with a private key, but the attacker's is not available, so the backdoored server needs to be patched to use another private key. This also means that detecting backdoored servers remotely is nearly impossible, since they will not react any differently to connections that don't use the attacker's private key.

Ultimately, the effect of the backdoor appears to be that a compromised SSH server which receives a connection with a hand-crafted RSA certificate for authentication can be made to run attacker-controlled code.

Anti-analysis

The design of the backdoor makes it difficult to notice without directly inspecting liblzma. For example, the choice to enable remote code execution rather than an authentication bypass means that use of the exploit does not detect a login session that could be noticed by traditional administration tools. The backdoor's code also uses several techniques to make discovery more difficult. For example, the string "RSA_public_decrypt@got.plt", which is used by the audit hook, never appears in the binary of the exploit. Instead, it uses a trie to hold various strings. Serge Bazanski posted a list of strings in the malicious liblzma encoded this way.

Examining that list shows that RSA_public_decrypt is likely not the only function interfered with; several other cryptography routines are listed. It also shows various functions and strings that are used to interfere with OpenSSH's logging. This is not yet confirmed, but it seems likely that a compromised SSH server would not actually log any connection attempts that use the exploit.

The backdoor also includes many checks to ensure it is running in the expected environment — a standard precaution for modern malware that is intended to make reverse-engineering more difficult. The backdoor is only active under specific circumstances, including: running in a non-graphical environment, as root (see this comment from Freund), in a binary located at /usr/sbin/sshd, with sshd having the expected ELF header, and where none of its functions have had a breakpoint inserted by a debugger. Despite these obstacles, community efforts to reverse-engineer and explain the remainder of the backdoor's code remain underway.

The backdoor also includes code that patches the binary of sshd itself to disable seccomp() and prevent the program from creating a chroot sandbox for its children (see this comment). In total, the code of the backdoor is 87KB, which is plenty of space for additional unpleasant surprises. Many people have put together their own summaries of the exploit, including this comprehensive FAQ by Sam James, which links to other resources.

Being safe

The exploit was caught promptly, so almost no users were affected. Debian sid, Fedora Rawhide, the Fedora 40 beta, openSUSE Tumbleweed, and Kali Linux all briefly shipped the compromised package. NixOS unstable also shipped the compromised version, but was not vulnerable because it does not patch OpenSSH to link libsystemd. Tan also included some other changes to the XZ code to make detecting and mitigating the backdoor more difficult, such as sabotaging sandboxing measures and making preemptive efforts to redirect security reports. Even though the exploit did not reach their stable versions, several distributions are nonetheless taking steps to move to a version of XZ that does not contain any commits from Tan, so users should expect to see security updates related to that soon. Readers may also wish to refer to the security notice for their distribution for more specific information.