Kernel.org's road to recovery
On October 3, a basic kernel.org returned to the net. Git hosting is back, but only for a very small number of trees: mainline, stable, and linux-next. The return of the other trees is waiting for the relevant developers to reestablish their access to the site - a process that involves developers verifying the integrity of their own systems, then generating a new PGP/GPG key, integrating it into the web of trust, and forwarding the public key to the kernel.org maintainers. This procedure could take a while; it is not clear how many developers will be able to regain their access to kernel.org before the 3.2 merge window opens.
The front-page web interface is back though, as of this writing, it is not being updated to reflect the state of the git trees. Most other kernel.org services remain down; some could stay that way for some time. It is worth remembering that kernel.org only has one full-time system administrator, a position that has been funded by the Linux Foundation since 2008. That administrator, along with a number of volunteers, is likely to be quite busy; some of the less-important services may not return anytime soon.
A full understanding of what happened is also likely to take some time. Even in the absence of a report on this intrusion, though, there are some conclusions that can be made. The first is obvious: the threat is real. There are attackers out there with time, resources, motivation, and skills. Given the potential value of either putting a back door into the kernel or adding a trojan that would run on developers' machines, we have to assume that there will be more attacks in the future. If the restored kernel.org is not run in a more secure manner, it will be compromised again in short order.
The site's administrators have already announced that shell accounts will not be returning to the systems where git trees are hosted. Prior to the breakin, there were on the order of 450 of those accounts; that is a lot of keys to the front door to have handed out. No matter how careful all those developers may be - and some are more careful than others - the chances of one of them having a compromised machine approach 100%. Keeping all those shell accounts off the system is clearly an important step toward a higher level of security.
Kernel.org has its roots in the community and was run the way kernel developers often run their machines. So, for example, kernel.org tended to run mainline -rc kernels - a good exercise in dogfooding, perhaps, but it also exposed the system to bleeding-edge bugs, and, perhaps more importantly, obscured the real cause of kernel panics experienced last August, delaying the realization that the system had been compromised. The kernel currently running on the new systems has not been announced; one assumes it is something a little better tested, better supported, and stable. (No criticism is intended by pointing this out, incidentally. Kernel.org has been run very well for a long time; the point here is that the environment has changed, so practices need to change too.)
At this point it seems clear that a single administrator for such a high-profile site is not an adequate level of staffing. Given the resources available in our community, it seems like it should be possible to increase the amount of support available to kernel.org. There are rumors that this is being worked on, but nothing has been announced.
Developers are going to have to learn to pay more attention to the security of their systems. There are scattered reports of kernel developers turning up compromised systems; in some cases, they may have been infected as the result of excessive trust in kernel.org. Certain practices will have to change; for that reason, the Fedora project's announcement of a zero-tolerance policy toward private keys on Fedora systems is welcome. Developers are on the front line here: everybody is depending on them to keep their code - and the infrastructure that distributes that code - secure.
There is an interesting question related to that: will kernel developers move back to kernel.org? These developers have had to find new homes for their git repositories during the outage; some of them are likely to decide that leaving those repositories in their new location is easier than establishing identities in the web of trust and getting back into kernel.org. Linus has said in the past that he sees the presence of a kernel.org-hosted tree in a pull request as a sign that the request is more likely to be genuine. Requiring that repositories be hosted at kernel.org seems like an unlikely step for this community, though. It is not entirely clear whether trees distributed around the net increase the security risk to the kernel, or whether putting all the eggs into the kernel.org basket would be worse.
One other conclusion would seem to jump out at this point: kernel.org got hit
this time, but there are a lot of other important projects and hosting
sites out there. Any of those projects is just as likely to be a target as
the kernel. If we are not to have a long series of embarrassing compromises,
some with seriously unfortunate consequences, we're going to have to take
security more seriously everywhere. Doing so without ruining our
community's openness is going to be a challenge, to say the least, but it
is one we need to take on. Security is a pain, but being broken into and
used to attack your users and developers is even more so.
| Index entries for this article | |
|---|---|
| Kernel | Development tools/Infrastructure |
| Kernel | Kernel.org |
| Security | Free software infrastructure |
| Security | Kernel.org |