Fedora, Red Hat, and distributor security

Fortunately for the Fedora Project (and its users), an audit has determined that nobody made use of the key while the intruder was present. So, even if some means for capturing something as transient as the passphrase were in place, that passphrase was not exposed, and, thus, cannot have been compromised.

Needless to say, the project is changing its package signing key anyway.

Interestingly, the Fedora Project was not the only target in this attack: Red Hat, too, was compromised. Unlike Fedora, Red Hat did not issue a statement specifically about this intrusion; instead, the information was included in an openssh security update. In this case, the attacker was more successful, to the point of being able to sign "...a small number of OpenSSH packages relating only to Red Hat Enterprise Linux 4 (i386 and x86_64 architectures only) and Red Hat Enterprise Linux 5 (x86_64 architecture only)." This language deserves to be questioned: it is only necessary to sign a single openssh package (certainly qualifying as a "small number") to compromise thousands of RHEL hosts, and the "only" terms describe what must be a large majority of deployed RHEL systems. Seriously scary, but Red Hat has been able to convince itself that none of the compromised packages were fed out to RHEL subscribers. So this attack, too, failed - but not by much.

Needless to say, disclosures like this raise more questions than they answer. The one question that Fedora and Red Hat will have to answer at some point is this: how did the intruder get in? One assumes that Fedora and Red Hat are running their own distributions on their internal systems; it thus stands to reason that, if those distributions contained a vulnerability that allowed the attacker to get in, many other systems will also be vulnerable. If, instead, this compromise is the result of administrator or developer error (or, say, of a lost laptop), administrators responsible for Fedora and RHEL systems can breathe a little easier. Either way, they deserve to know how this series of events came to pass.

Some people would like to have that information immediately. Beyond that, there has, predictably, been a fair amount of grumbling (also predictably, from a relatively small number of people) on the Fedora lists on how this incident was handled. Your editor, too, has argued that some information took too long to emerge. He will now argue that, while Fedora still has more to disclose, the project has said enough to give itself some breathing room while it struggles to put its infrastructure back together and figure out what really happened. There's all kinds of good reasons why more information may not be immediately forthcoming, including the obvious possibility that nobody really knows yet how the intruder gained access. There is little to be gained from hammering on Fedora at this point at this time.

That said, anything the project can say to tell its users whether they should be worried about an undisclosed vulnerability in their systems would be most welcome, and sooner would be better than later.

Meanwhile, what can be done, and what Fedora board member Jeff Spaleta, in particular, has been pushing for, is to think about how things should be handled the next time. Says Jeff:

The Fedora Project should certainly write up a policy for situations like this. It would be good for Fedora's users, but it could have an effect far beyond that: such a policy could serve as an example for other distributors as well. It is, after all, probably safe to say that Fedora is not the only distributor which has, thus far, neglected to put plans in place for this sort of disaster.

We all need such plans. For better or for worse, distributors have come to occupy an important position with regard to the security of much of the net. Millions of systems run packages signed by Linux distributors; they depend, implicitly, on the security of the process used to create those packages. That process is not a small one; it can involve hundreds of developers, before even counting all of those involved in upstream development projects. The consequences of a failure anywhere in that chain of trust can be severe. It is not surprising that the distribution system was attacked; perhaps the only real surprise is that it has not happened more often - that we know of. These attacks will happen again; distributors need to have a firm idea of how they will respond.

A related subject is worth a quick mention: as of this writing, the Fedora Project has issued no security updates since August 12, almost a full two weeks. A number of significant vulnerabilities, including the postfix symbolic link vulnerability, remain unpatched for Fedora users. Red Hat has done better, but not by much. Linux users depend heavily on the distributor security update process, and, for these two distributions, that process has been severely disrupted. If there had been a truly serious vulnerability disclosed during this time, people charged with keeping Fedora and RHEL systems secure might have found themselves in a difficult position. One need not be overly paranoid to envision this type of disruption being done intentionally as part of a zero-day attack on the net.

This incident should serve as a sort of wake-up call for both distributors and their users. Distributors wanting to retain their users' trust should be thinking about documenting things like:

• How the packaging chain is kept secure. It would be good to know how many people are able to sign packages, and how they gain access to the systems where this signing is done.

• What sort of plans the distributor has in place for dealing with security problems. One can only assume that Red Hat dedicated (and continues to dedicate) a large amount of staff time to understanding and recovering from this incident. Would other distributors be willing and able to do the same?

• What are the plans for dealing with a major security breach? How might a critical security update be propagated during a time when the integrity of the packaging system has been compromised?

• Should something go wrong, when and how will information be communicated to the wider community?

Conversely, anybody who is deploying an important Linux-based system should be asking such questions when choosing a distribution for that system. If the system requires high-assurance guarantees in this regard, it probably makes sense to look at the vendors who are willing to provide such guarantees for a fee. But, again, the lesson we have learned from recent events is that the time to ask these questions is now, and not when something has gone wrong and people are running around in circles.

As a whole, Linux users have been very well served by distributors since the very beginning. The distributors pull together thousands of software releases and integrate them into a coherent whole; they then make the results available, often for free. They provide fixes when things break, and most of them pay particular attention to fixing security-related problems. And they have done a top-quality job of not being used as a conduit for hostile software. It's a great system, and that has not changed. But we have learned something about how heavily we depend on that system, and how it can fail. Proper application of the lessons from this episode should help us all to be more secure in the future.