OpenSSH bug falls through the cracks

Linux distributions often patch the software they distribute, to fix bugs or add features. Anything they add is pushed upstream to the project responsible for the package—at least in theory. When that theory is not borne out in practice, it can lead to the kind of unhappiness and finger pointing that went along with a recent OpenSSH release. The release notes point at Debian for failing to report it upstream, but the bug was actually fixed much earlier, in Red Hat Enterprise Linux 4 (RHEL4).

The bug in question is rather nasty, allowing a local attacker to hijack X Windows programs of a user who logged in using ssh with X forwarding enabled. Under those circumstances, the ssh client and server arrange that any X programs started on the logged-in machine actually display on the client machine's desktop. This is very useful for running X programs across the internet as the X traffic is encrypted as part of the ssh session.

Due to a broken interaction with Internet Protocol version 6 (IPv6)—the next generation protocol for internet traffic—ssh can get confused about the port number of the X server. If a particular port (which maps to the X DISPLAY environment variable) is not available to be used under IPv4—the protocol in use today—but is available under IPv6, the ssh server will incorrectly set DISPLAY. If it is an attacker's program that is listening on the IPv4 port, it will be able to hijack X programs that get run.

Up until sometime in the last several years, this would not have happened for most Linux boxes because IPv6 was generally not enabled. In that case, the ssh server would recognize that it could not get the port it wanted and try another, eventually setting DISPLAY correctly. Because IPv6 is much newer, these kinds of bugs may exist in other network programs. This bug should serve as a reminder to developers to closely check their IPv6 support.

Clearly, though, the bug fell through the cracks. The OpenSSH team shows its annoyance in the release notes:

It was reported in January to the Debian bug tracking system, but not fixed and released until late March. OpenSSH does releases every six months or so, with 4.9 being released on March 30. Having to turn around another release four days later to fix a problem that was known for a few months could certainly make for annoyed developers. So how did the bug get fixed in Debian, with a Common Vulnerabilities and Exposures (CVE) number being assigned, but without notifying the OpenSSH team?

The Debian bug entry is instructive, because it documents some of the steps that led to the hurried release. In particular, Phil Miller thought he had done the right thing to report the problem in February:

That email must have gotten lost or been eaten by a spam filter as de Raadt would presumably have gotten it to the right people had he seen it. The bug description clearly puts it in the realm of a security problem, but the bug was not classified that way in the Debian system. Had it been, it would have been handled differently, possibly triggering an email to the proper place. But the bug report also shows that Red Hat fixed it in 2005.

It was reported to Red Hat by a customer and got entered into their bugzilla as bug #163732. Unfortunately, that bug report is confidential because it contains potentially sensitive customer information. This makes it difficult to track further. Indications are that it was not seen as a security problem and that it was believed to have been already known as an OpenSSH bug. Apparently no one checked to make sure the OpenSSH folks knew of it though.

Closer cooperation between the OpenSSH maintainers for Red Hat and the upstream team would probably have helped. Red Hat has been carrying the patch along for quite some time. Because the security implications were not clear and the patch is quite simple, it may not have seemed to be all that necessary to get it upstream. Though, there are more than twenty patches listed in the fedora OpenSSH CVS repository for rawhide, which will become Fedora 9.

The OpenSSH team would be well served by paying closer attention to various distribution patches to their code as well. It is certainly plausible that those interested in finding security holes to exploit might start by seeing if any patches floating around for critical services like OpenSSH were useful. By being more proactive, OpenSSH might have found and fixed this bug much earlier. The way this particular bug avoided notice seems to be mostly happenstance; if there is blame to be placed, there is plenty to go around.

RHEL and other "enterprise" distributions have long support cycles which means that the versions of various packages being maintained are well behind the upstream project. It doesn't take very many bug reports getting shot down because they have already been fixed in a more recent version before distribution maintainers lose enthusiasm for making those reports. But it is an essential part of the process. The OpenSSH team has the reputation of being somewhat difficult to work with, which may have helped this particular problem get overlooked.

It is a difficult problem to solve fully. Distributions have their own set of requirements which may be in opposition to those of the upstream project. Those projects may also have policies and procedures that distributions are not up to speed on. The Linux kernel often sees the same kind of conflicts, which is why distributions often maintain their own set of kernel patches for features their customers need. But it is in everyone's best interest to work those problems out so that distributions carry along as few patches as possible while upstream projects do not miss out on bug fixes and features.

> That email must have gotten lost or been eaten by a spam filter as de Raadt > would
presumably have gotten it to the right people had he seen it.

Presumably.. although it seems that there's mostly a non-story here: the belief and intention
of the parties involved was that the bug had been forwarded upstream, so besides the actual
unintentional miscommunication at some point along the line, everyone tried to do the right
thing.

> The bug description clearly puts it in the realm of a security problem, but > the bug was
not classified that way in the Debian system.

The original submitter gave it "Tags: security", so I think it was classified correctly.

> The OpenSSH team would be well served by paying closer attention
> to various distribution patches to their code as well
Unfortunately tracking distribution patches is not very easy. There is no common interface
which has a list of patch files for a package.

Upstream authors must either look through the different bug report interfaces, or download the
package source and look for applied patches. Additionally there are lots of different patch
systems (eg. Debian has dpatch, quilt, dbs, simple-patchsys, or just no system at all ;).

So in most cases, I believe upstream authors trust distribution maintainers to do the right
thing and send a copy of patches to the author or the mailing list.

> Unfortunately tracking distribution patches is not very easy. There is no 
> common interface which has a list of patch files for a package.

http://oss-security.openwall.org/wiki/distro-patches

In this case, debian is clearly not to blame, and the last time I remember Debian working with
OpenSSH on a serious security issue (circa 2003, I believe), OpenSSH gave them kudos for being
a very helpful and cooperative vendor.

Does Canonical's Launchpad offer a solution to this?

If it was available as Free software instead of a proprietary service, it will be much more
acceptable as a solution. 

Launchpad is proprietary.  The best it could offer is a temporary workaround.

And, alas, it doesn't appear to offer even that.

The flow of patches should not necessarily be a push from downstream -> upstream.

If I was a maintainer of an upstream open source package, I think I would try to also
regularly compare the various changes downstream apply. Just to see if something is
interesting.

If you were maintainer of a significant upstream package, you'd likely realize that it's
easier said than done.

<em>If you were maintainer of a significant upstream package, you'd likely realize that it's
easier said than done.</em>

Because the tools are not in place. That's why I said "try".

I don't expect someone to be able to track all downstreams packages. One developer cannot
track all distributions especially because each distribution track changes in a different way.

But if people work together it should be possible to come up with a set of tools to help all
those upstreams maintainers.

One could then do something like:

list-patches ssh ubuntu 8.04

Most distributions work in the open. This information doesn't change overnight and could be
centralized on a server. Maybe distrowatch or something similar.

The tools would help both ways, as distributors would be able to also look at what other
distributors are doing.

It's not a lot of work to send an email, but it's easy to forget.

We (OpenSSH maintainers) do check and merge downstream patches from time to time. It is
something of a pain to trawl through the various (completely different) vendor systems for
maintaining packages and I don't think it is at all sensible to have to depend on this to pick
up security fixes.

Is it worth having a central list of security contact details and bug submission procedures
for open source projects? Such information doesn't change often, and it might avoid the "I
emailed the project lead" problem.

"hijack ten programs?"  "hijack X programs, where X is an integer from 4 to 30 inclusive?"
Calling something just plain "X" is usually too ambiguous.

The phrase "X Window System programs" is just tedious.  It's like saying "Do you have Band Aid
Brand adhesive strips?"  Nobody uses that sort of language unless they're forced to.

What is your motivation?  Are you trying to get people to change how they refer to the
Windowing System Known as X?

"allowing a local attacker to hijack X11 programs"

How about that? :)

I've seen this bug too. I have to say that I didn't particularly realise the security
implications, although I understand them in hindsight. I was glad when it got fixed by Red Hat
and then I forgot all about it.

The OpenBSD community has become very inward looking. I couldn't find any evidence that they'd
even looked at the DF bug for example. Does it affect OpenBSD? Apparently no-one cared enough
to even ask, or they simply don't notice any news from outside.

We may end up with the Free Software people proving to have been right, years after the fact -
as happened with 'git'. At the time OpenSSH took off, there were some smaller GPL'd SSH clones
with less obnoxious maintainers. Those projects lost traction with the success of OpenSSH but
of course the source code still exists. If it's going to become a problem to maintain OpenSSH,
one of those clones might be the replacement. Certainly if OpenSSH continues to fall down on
security it has lost its most obvious advantage in that space.

Of course we looked at the Debian bug, the problem is that we only learned of  it after they
made a public release. There are good mechanisms to avoid this sort of problem occurring
(nominated security contacts, vendor-sec, etc.) but none of them were used.

I'm not sure how any of this leads to us being "inward looking", which is frankly insulting
given how much time some of us spend ensuring OpenSSH continues to run on platforms we don't
use frequently or at all.

> we only learned of it after they made a public release. 

A what now? Debian's BTS is public, throughout the whole cycle, there was no 'public release'
here.

Unless you mean to say that after a CVE was requested for an issue that had been reported in a
public bug tracking system for months. Which by the way is one of the "good mechanisms" that
are already out there.

When you made a scene about this originally, you were mistaken and I corrected you then, but
clearly your anger has kept you from hearing the facts. Claiming that "Debian" failed to avoid
this sort of problem just goes to show you do not know what you are talking about. The
original submitter of the bug was not a Debian developer and their posting to the Debian BTS
does not constitute Debian failing to use good mechanisms. If you dont know why, I'll tell
you: Debian don't control random people posting bugs to the public system, and once its been
posted, there is nothing that Debian can do to make it go away.

If your idea of good security practices are that Debian should have taken a many month old bug
that had been sitting on a public web site, that has been indexed by search engines, reposted
to many mailing lists, gated to NNTP and forums and wasted time trying to cover that up by
making the bug go away and chasing around google, yahoo, etc. to remove their cache'd
searches, scrubbing our public mailing list archives, asking GMANE to remove from their
archives the posting, etc. and then gone to vendor-sec to ask that a coordinated release was
undertaken... then you have to be out of your mind, or are just slandering Debian because
thats a convenient way to draw attention away from the fact that OpenSSH had a security hole. 

Just take the hit, you had a security bug, and it sucked that it got a CVE assigned four days
after you released and were forced to release again. I know it makes you look bad, but don't
blame that on Debian, that makes you look worse.

In my book, the design of IPv6 shares a lot of the blame for this security issue.  If IPv6
didn't require such significant code changes to implement, and was a lot more backward
compatible with IPv4, we wouldn't be discovering so many wonderful and interesting new bugs
with it.

Alas.

it's not the wonderful and interesting new bugs that annoy me about IPv6, it's watching the
IPv6 people discover the same things that were discovered and fixed in IPv4 that they missed
becouse they were so busy telling people how important IPv6 is

Yeah, that source routing issue last year made my jaw drop.  It indicated that IPv6 is
probably somewhere around 10 years less mature than IPv4.  Or maybe that the v6 guys are bound
and determined to learn a few well-known and fairly obvious security lessons for themselves.

I keep hoping someone will figure out how to just extend ipv4 so that both protocols can move
over the same wire and apps can be updated with only tiny changes.

too many of them are also wedded to the idea that all systems with access to the Internet are
equal, and therefor NAT and firewalls are evil.

becouse of these beliefs, I expect that we will run into many interesting interactions as
people try to use IPv6 for real-world projects and find that things don't work as expected

while I firmly believe that everyone should be able to choose to get onto the Internet at this
level, I am equally strong in my belief that this level of access is not appropriate for all
machines.

a couple of examples:
computers in a company that need access to the Internet for limited business functions should
not need to be addressable directly from the outside.

consumers should be able to choose an ISP that acts to protect them (both inbound and outbound
filtering)

I don't believe that many networking people believe "Firewalls are evil."  Now, *stupid*
firewalls are evil: I'm tired of running into routes that blindly drop ECN or TCP windows
because they are mangling packets for "security."

NAT *is* evil and should die.  It's a hack.  The security you like about "NAT" is the
anonymity and rejecting unrequested incoming packets.  The semi-random anonymous IPv6 IPs that
Microsoft and others have adopted provide the same anonymity benefits and the "security"
provided by NAT is nothing more than stateful firewalling.  NAT did force home
gateway/firewall vendors to provide stateful firewall because many-to-one NAT cannot be done
without it, but in NAT itself there's no security there.

Using "NAT" for security is also a misuse of terms.  NAT does include many to one, but it also
includes one to many, many to many, and even IP to IP, port to port translation so that
192.168.1.1:25 connects to 192.168.2.1:25 without blocking any packets at all.  There's no
security in such a mapping even though it is NAT all the way.

True, there's no additional security in NAT.  In theory.

In practice, NAT completely changed the way most Windows machines are broken.  Back in the
late 90s, Windows machines were usually taken out by smashing some part of the network stack.
Nowadays that pretty much never happens.  Almost all attacks go in through the browser, Flash,
or just as trojans via email.

Why?  NAT.  True, MS did spend a fair amount of time cleaning up their network stack but
that's moot since it's all hidden behind NAT anyway.

I agree, NAT is a horrible horrible thing to do.  But it's real-world benefits are
unassailable.  It's a non-negotiable one-button firewall deployable worldwide.  It
single-handedly cleaned up broken firewalls all over the country, from mountain shacks to
multi-thousand-node office networks.  Buy a box, plug it in, and you're safer.  Period.  No
other network security innovation has brought about such a profound positive change for so
many people, not even SSH.

So, when you say NAT should die (and I'm all for it), you're actually saying that NAT should
be replaced by something even better right?  Something that carefully studies the networking
lessons from the last ten years and improves on it?  Because simply abandoning NAT would be a
big step backward for most people and would reopen a lot of attack vectors which are currently
nicely shut.

The IPv6 team doesn't seem to understand this.  Yes, I'm worried.

What you appear to have just said is what I said, NAT was good because it forced home user
networking gear to adopt stateful firewalls.

NAT does not need to be replaced by anything.  We need home user network gear to abandon it
for IPv6 but keep the stateful firewall features.

I guess I'm not explaining myself very well.

"Stateful Firewall" is so ambiguous as to be virtually meaningless.  Most stateful firewalls
have thousands of knobs, dials, and levers, all sorts of different policies and ACLs...  Set
one lever to a wrong position and you get pwned.

Every stateful firewall I've ever seen takes fairly deep networking knowledge to set up and
maintain.  And they're all different!  Cisco experience doesn't help much with Sonicwall or
Linksys or Netscaler or Barracuda.

NAT, on the other hand, is either on or off.  On, you're presumably safe.  Off, you're not
safe.  Terminology is consistent across all manufacturers.  Go ahead and explain how to set up
a NAT to your boss over the phone.  He'll catch on pretty quick.  And *that* is why NAT has
taken over the world.

So, no.  Replacing NATs with stateful firewalls as they exist today would be a big step
backward for network security worldwide.

Do the IPv6 devs realize that they need to consider usability too?

You must be talking about serious enterprise level firewalls.  Yes, they have a lot of knobs
and settings.

A home/consumer level IPv6 firewall would look just like a NAT firewall does today.  Plug it
in and it allows outgoing traffic and blocks incoming traffic.  Nothing hard about that!

And as a point of fact: Many home firewall/router systems also have knobs and switches.  My
Linksys wireless router has an Advanced tab with all sorts of complicated things on it.

Also I have no idea why you consider stateful firewall to be an ambiguous term.  It's a system
that tracks connection state (or network "flows") and allows firewall block/allow decisions to
be made based on that state.

> A home/consumer level IPv6 firewall would look just like a NAT firewall does today.

Show me one and I'll happily evaluate it.  Until it I actually see one, I'll continue to say
that it needs to be invented.  :)

> My Linksys wireless router has an Advanced tab

I'm quite happy with the existence of knobs and levers as long as users don't have to see them
in normal use.  Unfortunately, on every firewall I've seen so far, blocking inbound traffic
without using NAT requires the Advanced tab or at least some pretty advanced knowledge.

Picture explaining to a non-technical person how to block all inbound traffic on your Linksys
without using NAT.  It will probably turn into a networking lesson.  Yes, in theory this is an
easy problem to solve -- it's just UI.  In practice, nobody has solved it yet.

> It's a system that tracks connection state (or network "flows") and allows firewall
block/allow decisions to be made based on that state.

That phrase probably describes 99.9% of the firewalls sold today.  That's why I consider it
ambiguous.

> That phrase probably describes 99.9% of the firewalls sold today.  That's why I consider it
ambiguous.

It isn't ambiguous at all.  99.9% of firewalls today track the connection state.  Some old
stuff still in the field is simple packet filter but everything new *is* stateful.

OK, %s/ambiguous/irrelevant/g   :)

So then to sum up:
 - most every router/firewall already contains stateful firewall
 - stateful firewall is what provides security
 - NAT and stateful firewall are separate things
 - NAT is irrelevant to security
 - NAT is irrelevant to anonymity

Why then are you upset that IPv6 discourages NAT?
Why do you want IPv6 routers to use NAT for security?
What benefit do you expect future customers of IPv6 home network gear to get from NAT?

> NAT and stateful firewall are separate things

NAT is just one policy a stateful firewall can implement.  I wouldn't call that separate.

> NAT is irrelevant to security

NAT is the single easiest to use policy on firewalls shipping today.  And it's disturbingly
effective.  That makes it quite relevant to security doesn't it?

As I've said on this very thread, I loathe NAT.  I really hope IPv6 will do away with it.
And, again, here's the point: before it can, IPv6 needs to provide something better.  Something
even more secure and even easier to administer. [1]

In the last 15 years of watching IPv6 gestate, I haven't seen any work on this front (I don't
follow v6 very closely anymore so it's entirely possible I've missed it; tell me if I have).
Maybe papers have been written, specs hammered out, names and policies standardized, and
Cisco/Linksys, F5, BI, Foundry, NS, etc are all in agreement.  Maybe working software even
exists.  If not, though, I'm afraid IPv6 has a lot of catching up to do.

It doesn't matter how advanced something is, it's worthless if it's not usable by the people
deploying it.  That's why NAT is so popular.  And *that* is where IPv6 needs to do better.
Just dismissing NAT as teh sux is to miss why it's been so successful.  (Hint: the IPv4
shortage is not even an issue yet).

At this point, I feel like I've repeated myself again and am well on my way to looping back
for fourths.  If my point still not clear, I apologize.

    - Scott

[1] NAT is pretty much optimal as far as ease of administration: on / off.  Things go bad if
you need to transit weird protocols like SIP or non-PASV FTP of course.  That's where IPv6
will really shine...  if and when the industry starts making easy to use IPv6 firewalls.

OK, you win, I give up.  Everything I said earlier you never bothered to once read.

So sure, someone like you needs NAT.  Enjoy your IPv4 NAT.

I'm afraid you didn't bother reading even worse...

bronson, +1 for nice wrap-up.  It's a pity v6 crowd seems like determined to learn it the hard
way, again.

NAT is a kluge but *not* an egg-head one.  v6 is both a kluge *and* an egg-head one.  This
kind of stuff is usually horrific on deployment.

-- 
M.Sc.,
just in case

bronson's "wrapup" ignored everything I said about stateful firewall being the solution.

I'd love to see his reaction if I were to take whatever router he uses and configure NAT on it
such that every incoming packet maps back to his internal IP address and then tell the
firewall to allow incoming packets.  That is a valid NAT configuration.  Some home routers
call it "DMZ" or "Server".

bronson just won't accept that NAT isn't the security, the firewall is the security.

NAT without security can be had (in Linux terms) by pairing SNAT and DNAT rules or using the
NETMAP target.

Here is IPv6 security without NAT in Linux iptables firewall terms:
ip6tables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
ip6tables -A FORWARD -i eth0 -j ACCEPT
ip6tables -A FORWARD -j DROP

Three rules.  No NAT.  Same security.
What would a hypothetical IPv6 home router call this?  Nothing!  It would be the default!  No
complicated knobs and switches.  It cannot get easier!

Explain what I didn't read.

As for bronson not reading me:
I explained how NAT is irrelevant to security.  Then in his last response he repeated how NAT
is an effective security policy.  It's not.  It has nothing to do with security.  As I
explained several times!

Then he repeats that he wants IPv6 to provide something better than NAT before getting rid of
NAT.  It doesn't need to!  It has security through stateful firewall just like current
systems!  As I explained several times!