Security in an error-prone world

In 1955, the Journal of the American Medical Association said that the interiors of contemporary cars were so poorly designed that it was amazing when anybody escaped an accident without serious injury. Ten years later, Ralph Nader's seminal Unsafe at Any Speed was published. In response, automotive engineers said that they designed their cars to operate safely — they were designed to drive, not to crash. Crashes were the result of bad driving, so the proper response was better driver education. The addition of safety features would come at a cost in style and comfort; it would also cost more. Customers, they said, did not want those safety features.

Fifty years later, though, cars are designed around crumple zones and crumple-resistant passenger areas. They have airbags, seat belts with pre-tensioners, collision sensors, and more. Modern cars, Konstantin said, are designed with driver errors in mind; as a result, automotive fatalities are a fraction of their peak nearly 50 years ago.

Computers and their software, though, are still designed like 1960s cars. They are all about power and comfort. Engineers will say that these systems have been designed to run safely, that things fail when humans make mistakes. Protecting users from their own mistakes is expensive, safety features can hurt the usability of software, customers are not asking for more safety features, and so on. The problem is best solved, they say, with more user education.

Konstantin faced this problem full-on in 2011, when he was hired in the aftermath of the kernel.org compromise. The approach he found was to design security infrastructure like a medieval fortress — or like Helm's Deep from the Lord of the Rings. There is a big wall with archers to defend it, a second line of defense (consisting solely of Gimli the dwarf), an inner keep, and a final line made up of two old white guys. Plus a disaster-recovery plan called "Gandalf."

The thing is, we design systems like this, but then somebody gets in anyway. The forensics consultants are called in; they find out that the back door used was always there — the administrators used it to get their work done. Or the attacker used an internal PHP application that should have never been there; it has a comment saying "temporary for Bob," but nobody even remembers who Bob is. People make mistakes, and they always will; we need, he said, to think more about how we can prevent these mistakes from becoming security problems. We need, in other words, to equip our systems with airbags to prevent serious problems when things do go wrong.

Airbags for systems

Konstantin then went through the various levels of a system to talk about what those airbags might look like.

At the networking level, we are already deploying firewalls, using virtual LANs, zoning, and virtual private networks, and performing routine nmap scans. But there are a number of things we are not doing, starting with payload inspection to see what is happening on our network; that is hard, especially when encrypted traffic is involved. There are mechanisms for tracking the reputation of IP addresses for spam blocking, but reputations are not used for other kinds of traffic. We are not, in general, actually bothering with DNSSEC or bothering to check TLS certificates. Or even to use TLS certificates in many cases.

For servers, we are using virtualization for isolation, applying security updates, doing centralized logging, and using SSH keys for logging in. But we should be able to do far better than that. We should stop disabling SELinux (or AppArmor or whatever); they are there for when something goes wrong. SELinux can keep an eye on that PHP application that has no business connecting to other sites, digging through /proc, looking at /etc/passwd, scanning the network, or sending email. Running a system with SELinux enabled can be learned, Konstantin said; we need to stop turning it off.

We should also be using two-factor authentication on all of our servers. Hardware tokens (he favors YubiKeys) are better than software tokens running on phones, but either is better than nothing at all. SSH keys should be stored on smart cards (or a YubiKey NEO) rather than on laptops. The Linux Foundation team has put up a set of documents on how to make this combination work well with Linux.

Containers, he said, will not, contrary to some claims, make system administrators obsolete. But they can help isolate software stacks from each other. Think of a container, he said, as a sort of whole-OS static linking mechanism. They are a type of airbag: they allow a complex software stack to be bundled away and hidden from the world; the whole thing can then be further confined with SELinux. Containers also make good crash-test dummies — they can be used to test software in production-like environments. In such a setting, it's easy to check for open ports, improper log files, etc. This, he said, "is what DevOps is about."

On workstations things are a bit more difficult; confining applications is not an easy task. SELinux is nearly useless on a desktop. The X Window System is the essence of pre-safety design; there is only security between users, and applications have full access to everything. So a single vulnerable X application means the compromise of the entire desktop. X, Konstantin said, must die. Unfortunately, that won't happen for a long time.

Then there is the issue of web browsers. They run code from untrusted outside users, they have a huge attack surface, and they have proprietary plugins. And we can't live without them. So we end up with issues like CVE-2015-4495, which was actively exploited to search systems for SSH keys and passwords.

As a rule, the most unprotected system on the entire net is the system administrator's desktop. It sits on the VPN with full access to the net, it has access to the password vault and is full of privileged SSH keys. There is also often a handy list of other systems that those keys will grant access to. The system is full of dangerous keystrokes, disclosing passwords to any attacker that happens by.

How does one address this threat? Requiring the use of one-time passwords — preferably not supplied by a phone app — is the first basic step. SSH keys should be stored on smart cards, and never in home directories. Proper security policies need to be written and administrators educated, forcefully if need be, to follow them. Konstantin also suggested looking into Qubes, which, he said, is the only serious attempt at workstation security out there. Qubes sidesteps most X vulnerabilities and can minimize the impact of things going wrong. Its safety orientation makes it "the Volvo of distributions."

When it comes to the functioning of administrative teams, there is no alternative to relying on education, so it is necessary to be prepared for failures. A team should establish secure communications so that its members can talk to each other when the network cannot be trusted. Email communications should employ PGP signatures, and instant messaging should be via a trusted mechanism as well. There need to be firm policies about what can be sent in clear text; important stuff should always be encrypted. Sites need workstation security policies, hardening checklists, and examples of secure workflows. Separate, dedicated browsers should be used for work and play. The system administrative team should use code review like any other development project, and changes should require signoffs.

Checklists should be created and used for everything: deployment of a new system, code review, staff onboarding, staff departure, etc. There should be a procedure to quickly lock out an administrator — a tricky task. Checklists can be the most powerful tool available to avoid bad experiences.

In closing, Konstantin reiterated that mistakes are going to happen; the important thing is to make sure that these accidents are not fatal. Our current systems are great to drive, but they do not forgive mistakes; we are at our "unsafe at any speed" moment. We have the technology to make things safer, but we're not using it; that needs to change. Konstantin and his team are putting together a set of recommended policies and are looking for help to improve them.

[Your editor would like to thank the Linux Foundation for supporting his travel to KLF].