OpenBSD and the latest OpenSSL bugs

By Jake Edge
June 11, 2014

The latest round of OpenSSL bugs was disclosed to the public on June 5, but it is clear that some organizations and distributions had earlier knowledge of the flaws. That is fairly typical for security holes of this sort; distributions get some time to fix the flaws before they are made public (typically simultaneously with the release of the updates). But OpenBSD was not one of the organizations notified in advance; why that is, and whose fault it is, are much in dispute since then.

OpenBSD project leader Theo de Raadt complained about the lack of early notice in a message to the OpenBSD misc and tech mailing lists. OpenBSD has famously forked the OpenSSL code post-Heartbleed into a new library called LibReSSL (or LibreSSL). Both its OpenSSL and LibReSSL packages were affected by the bugs, though, so it is unsurprising that de Raadt is unhappy first hearing about the bugs several days after others had been informed.

According to a timeline published by OpenSSL project member Mark J. Cox, who handled the issues for the project, the distros mailing list was notified of the problem on June 2. That allowed members of that private list—restricted to security representatives from Linux distributions and the BSDs—to request the patches and a copy of the draft advisory. OpenBSD is conspicuously absent from the list of those participating in that list.

As it turns out, de Raadt had been asked if he wanted to join the distros list back in early May. A different OpenSSL problem led Red Hat security response team member Kurt Seifried to CC de Raadt on the report and ask if he or some other OpenBSD member would like to join the list. The distros mailing list is meant to disclose and discuss security problems that affect the entire Unix ecosystem (rather than those that just affect Linux, for which there is a linux-distros mailing list). In characteristic fashion, de Raadt replied:

I'm sure you've all got your "processes" for handling these things. But then you get paid for handling these things in some way, don't you?

We don't get paid. And therefore, I don't know where I should find the time to be on another mailing list. It is not like I would have sent a mail to anyone. In general our processes are simply commit & publish. So I'll decline.

Once Cox's timeline made it clear that most other distributions (both Linux and BSD) had been given an advance heads-up about the issue, de Raadt and other OpenBSD developers accused OpenSSL of knowingly keeping the knowledge of the bugs from the project: "Unfortunately I find myself believing reports that the OpenSSL people intentionally asked others for quarantine, and went out of their way to ensure this information would not come to OpenBSD and LibreSSL."

For his part, Cox states that OpenSSL chose the distros mailing list as its means of disclosing the bug early to the various affected operating systems. Because OpenBSD was not on the list, it didn't find out, he said in a comment on his timeline post. Furthermore, "OpenBSD have approached us to be notified about future issues and we've asked them to join the list as they certainly would qualify and would find it beneficial not just for any future OpenSSL issues."

The timeline shows that there were some organizations that received early warning of the bugs, including a few well in advance of the distros posting. Others were notified at around the same time as the posting, but without any details. Whether OpenSSL considered notifying OpenBSD separately from the mailing list is not clear. The project is certainly aware of the LibReSSL effort (and likely unhappy with how its code has been characterized by the OpenBSD crowd), and that it would likely be affected by these problems. But it is entirely possible that notifying OpenBSD just slipped through the cracks as well.

The conversation fairly quickly degenerated. It is clear that de Raadt and others do not see the distros list as the appropriate venue for early disclosure of vulnerabilities. They believe that affected organizations and projects should be contacted individually, it seems. Regardless of whether anyone at OpenBSD gets paid to read security mailing lists, it is undeniable that having a representative on the list would have gotten the project the early disclosure it is looking for, however.

The conversation is also a bit hard to follow since various participants, including Seifried and distros/linux-distros administrator Solar Designer (Alexander Peslyak), sent private mail to de Raadt that he responds to publicly. In addition, de Raadt's emails don't seem to thread correctly for some reason. But he makes it abundantly clear that he is livid about the issue and he lashes out at Peslyak, Seifried, and Cox.

But, ultimately, it is de Raadt's opposition to embargoes (which typically come with early disclosure) that is part of the reason no one from OpenBSD is on the relevant list. Peslyak said that de Raadt had been invited to join the list in 2012, but declined not just for himself but for the entire OpenBSD project. Peslyak, who has been a voice of reason throughout (for example, he has encouraged OpenSSL to contact LibreSSL directly in the future), also said that de Raadt's anti-embargo stance contributed to the current situation:

[...] but given our past discussion I (personally) thought you wouldn't want to receive it under an embargo. So frankly even if I were aware, I would probably not be alarmed if I heard that you were not being notified. I think this will be different now that you seem to have expressed a different preference.

It is most unfortunate for their users that OpenBSD and LibReSSL did not get the extra few days to fix the problems found in OpenSSL. It is not exactly clear who is most "to blame" for that, but it is clear that things could be done better (by both OpenBSD and OpenSSL) in the future. For some on the OpenBSD/LibReSSL "side", this episode is evidence of why those projects cannot work with OpenSSL. That may be, but the tone and contents of the emails from de Raadt and others may have also made it obvious (again) why it is hard for anyone outside of the OpenBSD clique to work with that project. It is a project that does a lot of good work, but it is not one that is known for getting along with others.

Index entries for this article
Security	Bug reporting
Security	OpenSSL

to post comments

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 3:24 UTC (Thu) by flewellyn (subscriber, #5047) [Link] (7 responses)

I'm torn on this. On the one hand, it's really not okay for OpenSSL to have one special mailing list that they use to disclose vulnerabilities, especially when that mailing list is not obvious by its title and membership. I seem to recall the OpenSSL project having some other issues with communication in that regard, along the lines of "No, you silly, the listed email address for contacting us may be contact@openssl.org, but everybody knows that the REAL one that we read is actually someotheraddress@openssl.org!"

Or something like that. That kind of ad hoc organization obviously infects their codebase as much as their org itself, and that's not a good thing at all, especially for a security-focused project as critical as this one. OpenSSL doesn't have a good track record for communicating, or for that matter knowing what they are doing, if the LibreSSL commit logs are anything to go by.

On the other hand, wow, Theo does not merely burn his bridges, he nukes them from orbit. And sure, meritocracy, sure, show me the code first, sure, he still does great work...there's just a point at which behaving in such a hostile and unpredictable fashion towards others is going to start hurting your project, if you act like this. Especially if you are the project leader.

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 6:55 UTC (Thu) by nim-nim (subscriber, #34454) [Link] (3 responses)

The list does not seem to be a special openssl list, nor is it hosted @openssl, they seem to have done the right thing and use a common general-purpose security list instead of sprouting their own.

Unless I'm missing something.

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 0:39 UTC (Fri) by misc (subscriber, #73730) [Link] (2 responses)

No it is not only for openssl. That's for various vendors, to organize big updates. The idea of embargo is a often debated one, but so far, it seems to work fine in practice. The list access is controlled, and you kinda have to show you are responsible of a product/distro security to be there.

And having a few weeks to work on the patch bring some benefit in term of testing, etc. For example, the first patches for openssl were indeed wrong and were corrected by Canonical and Red Hat, so we can see the power of collaboration. In turn, it avoid the rush of having to update twice in a row ( openssh 3.6 for example ). And funnily enough, openssh 5.0 release note do ask for responsible disclosure ( http://www.openssh.com/txt/release-5.0 ) due to a CVE found just after the 4.9 release. As they request the use of a private mailing list, I guess the team behind openssh have a more nuanced view on what is exactly "responsible disclosure" than Theo who seems to want to commit and release.

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 1:20 UTC (Fri) by viro (subscriber, #7872) [Link] (1 responses)

You seem to use some strange definition of "to work fine". vendor-sec embargo rules are disgrace and quite a few of us flat-out refuse to have anything to do with that mess due to a _long_ history of deliberate abuse of said rules by PR-seeking wankstains^W^Wconscious security researchers. More often than not, v-s embargo is nothing but press release coordination, with the only goal being to drum up the publicity for something very overblown.

I unsubscribed from that FPOS after particularly egregious example of that sort (too long to retell and I doubt that I could stay anywhere near printable if I tried). No idea when Linus had decided that he had enough, but he's not there either, AFAIK - due to the same embargo policy. And no, it didn't get better since then - see e.g. the clusterfuck with enormous spew of trivial "fixes" in X libraries, oversold to hell and back _and_ actually broken and insufficiently reviewed exactly due to the embargo. Security by press release has a lot in common with science by press release; for more details ask the "arsenic DNA" crowd, or "Darwinius masillae" one, or...

OpenBSD and the latest OpenSSL bugs

Posted Jun 21, 2014 23:59 UTC (Sat) by BenHutchings (subscriber, #37955) [Link]

vendor-sec has been dead for years. I think the newer mailing lists (distros@ and linux-distros@vs.openwall.org) have more limited embargo periods (max. 19 days), though I'm not sure what the limit was for vendor-sec.

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 13:22 UTC (Thu) by khim (subscriber, #9252) [Link] (2 responses)

The whole thing is just ridiculous. As Theo admitted himself when they find a security problem they just “commit and publish” which means that the whole world is forced to rush and apply patches in hurry without any advance warning. But when OpenBSD is treated in the same vein it's somehow a problem? Why? Is it because they treated differently? Sure, they are treated differently, but that's because the rules they apply to everyone are applied to OpenBSD! Payback is bitch, sure, but why complain about the situation you've created in the first place? Why anyone should offer you “early notice” when you categorically refuse to do the same for others?

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 22:46 UTC (Thu) by rahvin (guest, #16953) [Link] (1 responses)

His annoyance is easy to identify IMO if you've followed the whole LibreSSL thing. He's been trying to use libressl as a fundraising drive for OBSD. This ties into his comment about being paid to read the security list.

Now because he didn't want to participate in the openSSL community and wanted to fork it for the financial benefits he's caught off guard by a security issue everyone else new about and he's concerned it could impact his fundraising via LibreSSL so he throws his typical hissyfit and blames everyone else.

This is the work he created for himself when he forked it. If he wasn't willing to make that work commitment he made a mistake. Don't think for minute this won't happen again. And the beauty of it is at some point down the road when he's fully forked the project there will be some issue found that's still in his code base that will bite him on the ass then as well and he'll still probably try to blame everyone else.

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 0:28 UTC (Fri) by misc (subscriber, #73730) [Link]

Well, they indeed doesn't hide it :
http://www.openbsd.org/papers/bsdcan14-libressl/mgp00033....

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 3:33 UTC (Thu) by JdGordy (subscriber, #70103) [Link] (13 responses)

Theo's logic for not being on the security list seems a bit absurd to me. Instead of spending (presumably not much) time watching a list and working with it he now has to scramble to apply patches and rebuild while everyone else had time to coordinate??

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 6:24 UTC (Thu) by salimma (subscriber, #34460) [Link]

Not to mention his stance on embargoes. He really wants every security bug to be zero-day vulnerabilities?

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 7:07 UTC (Thu) by smurf (subscriber, #17840) [Link]

He obviously wants to have it both ways.

Kindof reminds me of my daughter. However, she's in the midst of puberty, so she has an excuse (albeit a lame one). Theo does not.

Anyway. By now, anybody working with the OpenBSD people should know perfectly well that ultimately, you will get the flak for their bad decisions, so this article is hardly any news.

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 12:50 UTC (Thu) by niner (guest, #26151) [Link] (10 responses)

Not only that. He claims to have no time to even skim a security related mailing list but expects others to spend time to cater to his wishes and send a personal email. Spreading emails to a list of interested individuals is exactly what mailing lists are for.

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 12:56 UTC (Thu) by mstone_ (subscriber, #66309) [Link] (8 responses)

bingo. if it's too much work for him to follow the list, it's not clear why it's not too much work for other people to be responsible for keeping track of what he wants to know and making sure he knows it.

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 17:18 UTC (Thu) by Lennie (subscriber, #49641) [Link] (7 responses)

It all depends on the kind of traffic that mailinglist gets.

I think someone in the comments mentioned that it isn't OpenSSL only.

If that is true it would mean you'll get lots of updates on all kinds of software that is used on Linux/BSD, not just OpenSSL.

Does anyone know how OpenBSD is organized ?:
Specifically how is Theo employed ?

I expected he was the only employee or the OpenBSD Foundation or something like that.

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 14:42 UTC (Fri) by mstone_ (subscriber, #66309) [Link] (5 responses)

The linux vendors also see traffic about software they don't care about. You look at the subject and ignore what you're not interested in. (Unless you're Theo, then you expect someone else to read the traffic and make sure you are told about exactly that set of things which interest you. In most situations, that person is called a "secretary" and gets paid for the service.)

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 18:07 UTC (Fri) by viro (subscriber, #7872) [Link] (4 responses)

It doesn't work that way. The volume is a secondary problem - compared to e.g. l-k it's _very_ low-traffic. The real bitch is that a fair number of postings there carry (OK, carried, but I don't believe that anything has changed in that respect) utterly ridiculous embargo terms.

It works like that: $TWIT_WITH_GREP goes over some bunch of libraries and finds a lot of dubious places. Of the "sure, it's badly written and needs to be fixed on general principles" variety. One or two might be exploitable in real-world setups. Good for them; such things need to be hunted down and fixed. Unfortunately, said twit decides to get more PR mileage from that. Easy - let's request a CVE for each instance (exploitable or not) *and* arrange for coordinated PR offensive. Which is where vendor-sec really comes into the picture - twit reports it there and demands a month (or two) worth of embargo. By the end of which they get (Number of Distros * Number of CVEs) announcements, all at once, each referring to them.

If you receive such mail via vendor-sec, you *can't* do anything about the bloody thing until the embargo runs out. Even if the proposed fix is crap and there's a better solution. And yes, sometimes embargoes are needed, but not on such terms.

So I certainly can understand somebody refusing to subscribe to vendor-sec. If something relevant comes up there without embargo terms from hell, I'd appreciate having the report forwarded my way, but that's it - I refuse to honour overblown embargoes. The same goes for Linus, AFAIK.

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 18:17 UTC (Fri) by mstone_ (subscriber, #66309) [Link]

There's nothing wrong with not wanting to participate, as long as you then accept that there's going to be stuff you're not privy to and don't complain that it's other peoples' job to keep you informed in the way you prefer. Right? In this case, the issue was embargoed and so what you're suggesting is that not only should someone figure out what's important, they also need to decide whether the embargo terms are "reasonable enough" to forward (maybe with some kind of back and forth to determine the reasonableness?). That seems like a high bar for one volunteer to demand of another volunteer.

Note also that you're arguing for your reasons, not the reasons Theo actually stated. "I don't know where I should find the time to be on another mailing list"

OpenBSD and the latest OpenSSL bugs

Posted Jun 14, 2014 22:33 UTC (Sat) by spender (guest, #23067) [Link] (2 responses)

You know vendor-sec doesn't exist anymore right? As in, not for over three years. I guess you also know there aren't month-long embargoes on the new list. Here's the new list: http://oss-security.openwall.org/wiki/mailing-lists/distros . The *maximum* embargo is 14-19 days.

I'm not a fan of embargoes either, so I don't subscribe to such lists. That said, we generally can respond quicker than a distro and are rarely ever affected by public exploits (unlike upstream).

So your arguments are a little stale, and I don't much care for your alternative of "just fixing the bug" with your not-so-cute one-liner commit messages, sometimes introducing new bugs in the process (revealed by subsequent one-liner commit messages).

Seems to me like you want a distraction from the massive problems with upstream's own security handling, as if it's the existence of these cherry-picked security "researchers" from years ago that are continuing to prevent you today from taking security seriously.

A researcher who reports to security@kernel.org today will end up seeing their issue fixed without any mention of security with one of Linus' signature commit messages. It's getting a little old and I'm surprised people are still buying your stale excuses.

-Brad

OpenBSD and the latest OpenSSL bugs

Posted Jun 15, 2014 6:16 UTC (Sun) by viro (subscriber, #7872) [Link] (1 responses)

I'm glad to hear about v-s demise; it's been long overdue. As for the embargo policy on replacement... How long was the embargo on CVE-2013-1981 and friends? That's a bit more than a year ago.

Brad, care to give your opinion about the quality of those patches and severity of the vulnerabilities covered by those? My reading of the situation (and I'm not on the list(s) where it had been reported, so I hadn't seen the threads that had led to that) is that

a) authors went after low-hanging fruit - AFAICS, all places they'd found could be found by simple grep and quick look through the hits. Nothing wrong with that, of course.

b) severity had been overblown - that crap needed fixing, all right, but embargo had been unwarranted. Compromise of X server could lead to compromise of clients talking to it (as if the ability of that server to feed an arbitrary input to client *and* hide the reaction from user hadn't been enough) and suid-root X client that could be convinced to talk to attacker's X server could lead to escalation. The practical impact of the first part is nil and the second one needs much more exotic setup than the announcement implied.

c) all that pile didn't get anywhere near enough review - it introduced at least one genuine stack corruption, triggered by real-world programs talking to non-compromised server (NULL written at some location in caller's stack frame). Again, I'm not blaming the patch authors. Shit happens - which is why code review is needed.

d) the lack of code review had almost certainly been due to embargo cutting down on the amount of potential reviewers. Worse, classifying that as "important security fix" had rushed the deployment after the embargo had been lifted.

e) the only reason I can conjecture for the whole sorry mess is that the authors wanted to maximize the PR mileage.

Look, I'm not saying that embargoes are never needed. Or that 14--19 days is horribly long. The problem starts when patch authors get to *demand* the embargo and when it gets tangled into the whole "how severe the bugs in question are, the worse is more impressive" mess...

Anyway, I'm really glad to hear that vendor-sec got replaced by something that might end up being saner. Said that, the bits and pieces of stories I've seen don't leave the impression that much has improved in the last few years... ;-/

OpenBSD and the latest OpenSSL bugs

Posted Jun 16, 2014 16:49 UTC (Mon) by spender (guest, #23067) [Link]

I've read your other comments on the huge pile of Xorg vulns (http://lwn.net/Articles/551860/ etc), you seem to like to bring it up often though it doesn't seem you actually have any first-hand experience with what happened behind the scenes.

Specifically, I haven't been able to find any public evidence that the reporter (Ilja) demanded some ridiculously long embargo, nor does it seem that he reported it to the distros list. He seemed to have reported it directly to the Xorg security team and worked directly with Alan Coopersmith. You can see some mention of details in one of the presentations he gave on the subject:
http://events.ccc.de/congress/2013/Fahrplan/system/attach...
Note slides 5, 29, and 33. On slide 33 it mentions a two week embargo (which would suggest the Xorg team then sent it to the distros list), presumably after all the fixes were created, which took nearly three months.

You seem to be the only one taking huge offense at the handling of all the bugs -- I certainly don't see it coming from the Xorg security team. From the presentation and Alan Coopersmith's tweets: https://twitter.com/alanc/status/417884288466444288 onward, there seems to be mutual respect and appreciation of efforts.

Given that at least 200 bugs were reported and fixed, it makes sense for this to be bundled up into a single update instead of numerous updates. Alan Coopersmith explained the reasoning behind the large number of CVEs already here: http://lwn.net/Articles/552035/ and here: http://lwn.net/Articles/552033/ They weren't demanded by the reporter as part of some PR conspiracy as you previously claimed.

Making a big deal out of two of those 200 fixes puts you ironically in a similar position as these security people you despise ;)

Still, I don't see how cherry-picking specific researchers/reports has anything to do with the new distros list, which wasn't even involved in the creation of fixes in the case you mentioned. Not to mention how this "security circus" that constantly gets referred to prevents kernel upstream from taking security seriously. Frankly, in this instance, given that many of these vulnerabilities were ~20 years old and trivial to spot according to you, we should just be glad someone (not you) finally bothered to look and got them fixed. "Many eyes" and all that.

-Brad

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 16:27 UTC (Fri) by dag- (guest, #30207) [Link]

Say hello to procmail !

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 9:18 UTC (Fri) by jezuch (subscriber, #52988) [Link]

> He claims to have no time to even skim a security related mailing list but expects others to spend time to cater to his wishes and send a personal email.

He claims that he expects them to do that because they're paid for it while he is not.

BUT.

He claims that he has no time to be on "another list". But he wants to be notified anyway - hence be on some other another list. I found it a little funny :)

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 10:47 UTC (Thu) by ncm (guest, #165) [Link] (1 responses)

It brings me no pleasure to note that this story would have been better if it were half as long.

OpenBSD and the latest OpenSSL bugs

Posted Jun 12, 2014 17:56 UTC (Thu) by Karellen (subscriber, #67644) [Link]

Maybe Jake didn't have time for that.

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 7:20 UTC (Fri) by malor (guest, #2973) [Link] (3 responses)

It strikes me that, given what we now know about the NSA's massive surveillance and active cracking of encryption whenever possible, it's downright irresponsible to discuss private, security-related information on any unencrypted channel.

Email is no longer a good method for spreading news of security problems, because the spooks are going to read that mail, and exploit the bugs before they can be fixed.

OpenBSD and the latest OpenSSL bugs

Posted Jun 13, 2014 7:32 UTC (Fri) by dlang (guest, #313) [Link] (2 responses)

so what encrypted channel do you think exists that has no chance of the spooks listening in on (either by intercepting the channel and breaking the encryption, by having control of one of the endpoints to read the message as it's decrypted for viewing, or by simply subscribing like anyone else?)

and how long do you think the channel is going to remain 'spook proof'?

Also, keep in mind that anyone you send the info about a security problem to is going to have to pass the information to others, so all those organizations internal communications needs to be equally secure, and people had better not talk about it in a meeting, because it's really hard to have face-to-face discussions encrypted....

OpenBSD and the latest OpenSSL bugs

Posted Jun 14, 2014 8:10 UTC (Sat) by malor (guest, #2973) [Link] (1 responses)

I didn't say the solution would be easy, just that it's necessary.

In a world of ubiquitous surveillance, people could die because of unencrypted discussions of this type. Nice people, ones you'd like to have over for beer and movie night, who happen to hold opinions that are deemed threatening by their government.

If a solution doesn't exist to let a group of security researchers discuss things in a fairly secure way, then we really need to write something. Perhaps something vaguely like SilentCircle might be appropriate.

OpenBSD and the latest OpenSSL bugs

Posted Jun 17, 2014 14:53 UTC (Tue) by dlang (guest, #313) [Link]

> I didn't say the solution would be easy, just that it's necessary.

you can claim that it's necessary as much as you want, but I'm saying that you need to first establish that it's even possible.

I'm claiming that in an environment where you don't know who the people are, and where trying to establish who everyone is would be detrimental to your mission, keeping 'spooks' off of the list isn't going to be possible.

Adding value.

Posted Jun 16, 2014 10:21 UTC (Mon) by mjcox@redhat.com (guest, #31775) [Link]

After Heartbleed we figured we ought to formally define a disclosure process, and the team voted on a proposal of how to deal with the next serious security issue.

We based our proposal on our experiences dealing with OpenSSL issues as well as how they are handled by the rest of the industry. It was based on these assumptions:

- the more people you tell in advance the more likely is that a leak will occur. This has been proven with previous notifications both from OpenSSL and other projects.

- OpenSSL is used in a lot of products. Seriously, a mindblowing number of products from an equally large number of companies use OpenSSL.

- OpenSSL strongly believe that the right to advance patches/info should not be based in any way on paid membership to some forum. You must not be able to pay cash to get security patches in advance.

- OpenSSL saw some companies that were given advance notice of the Heartbleed issue use it in marketing statements about how they protected their customers before their competitors. "if you bought our product/service then you would have been protected". This is egregious.

- We don't get enough serious vulnerabilities in OpenSSL to warrant spending signficiant time keeping our own list of vendors we trust, or signing framework agreements, or dealing with changes, and policing the policy. (Even dealing with the disclosure around this CCS issue was several person-weeks of effort.)

- OpenSSL has attempted using third parties to handle notification on our behalf in the past, like CPNI, oCERT, or CERT/CC, but we have had issues with all of these.

- It's in the best interests of the internet as a whole to get fixes for security issues out quickly. OpenSSL embargoes should be measured in days and weeks, not months or years.

- Many public sites affected by OpenSSL issues will be running a version of OpenSSL they got from some vendor (like Red Hat or Ubuntu). The quickest way for them to be protected is to get a new vendor version of the package to deploy. Sites who use their own OpenSSL compilations should be able to handle a quick patch and recompile once the issue is public.

- We can benefit from peer review of the patches and advisory. Keeping security issues private is a pain because they can't get the level of testing or scrutiny that they otherwise would.

For our June OpenSSL advisory we therefore decided

- to give a couple of companies we know to have experience with in depth SSL protocol issues a few weeks headsup to help investigate and ensure our fixes and advice was valid, and to probe if this issue was being exploited in the wild (there were no signs of this)

- to let the companies that have a general purpose OS that uses OpenSSL have a few days notice in order to prepare packages for their users. We used the http://oss-security.openwall.org/wiki/mailing-lists/distros for this purpose. Two of the companies we told found issues with the patches that would otherwise have required additional releases.

- to give companies that handle critical infrastructure or have widely used sites some notice that an update was coming with the date and time, but no other details about the issues. We used the ops-trust forum for this purpose. https://openid.ops-trust.net/about

- We'd like to have given the whole internet notice that an update was coming out at a certain date and time, but given the level of press around Heartbleed this would just have led to a lot of stress and panic. Next time we might try this (or set up some regular cadence for updates so users always know when the next update is coming).

Things were a little more complex this time because the reporter of the most serious issue first contacted JPCERT who notified CERT who in turn gave a headsup to all their contacts. This meant we had to turn down requests for patches from companies who didn't qualify for disclosure this time. Can't hug every cat. Some companies were quite insistant that they were more important than others and that we should just allow them as an exception. Even though in many cases they just wanted the details so they can patch their products in advance. If you are one of those companies you should consider how you can add value to the open source projects you rely on.

What is a "general purpose OS" anyway? How about a provider that doesn't provide you an OS but provides you an OS as a platform/service. One of the reasons we used the distros list was so they could make those determinations in a wider context and not have arbitrary made up rules from OpenSSL. It would certainly help other OSS projects that want help giving advance notifications if there was a standard way that works. (As stated in this article OpenBSD had said not to be told about issues under embargo, but they've now asked to find out about future OpenSSL issues).

OpenSSL have said we'll evaluate how disclosure went this time and use it to adjust our process for future serious issues (and publish the policy so it's clear and obvious). Some of the companies told in advance certainly added value to the process, finding issues in the advisories and patches, but that was unfortunately the minority of those who were told.

OpenBSD and the latest OpenSSL bugs

Posted Jun 16, 2014 22:23 UTC (Mon) by RogerOdle (guest, #60791) [Link] (3 responses)

I have had my own problems with OpenSSL. I was working on a remote embedded system that needed to communicate back to the office with two requirements: 1) authenticate the source of the information and 2) verify that the contents were not modified. The system would frequently core-dump in the OpenSSL code.

The code base does not meet the requirements for embedded, in particular, it uses dynamic memory allocation. No critical systems that I know of allow dynamic memory control at run time. Many do not even allow garbage collection for good reasons. I see a need for a TSL solution that meets the needs of embedded systems and what embedded systems need meets or exceeds whatever a server system needs since they are unattended and often installed at remote places. Service may be days or weeks away.

You may think that I am bashing OpenSSL. I looked at other libraries like NSS and GnuTSL. At a glance, I did not see superior code practice in any of them. I saw more effort in making things work faster than in making things work correctly. Given the nature of these things, correct operation must be given top priority. And given how much is at stake, some form of formal review and acceptance testing by someone other than the developers themselves must be used. I do not know how TSL vendors are qualified but some method for assurance needs to be made. OpenSSL is not the only critical core service it is only the most visible now.

OpenBSD and the latest OpenSSL bugs

Posted Jun 17, 2014 13:36 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

s/TSL/TLS/g? Or is there something TSL stands for (seems too consistent to be a typo, but I don't know of a derogatory term for TLS analogous to "M$" it might represent)?

Anyways, I certainly agree about code quality in NSS at least. It's not nice and tidy in there. I think my memory leak patches on the bugzilla just had their fourth birthday recently too.

OpenBSD and the latest OpenSSL bugs

Posted Jun 17, 2014 13:48 UTC (Tue) by ortalo (guest, #4654) [Link]

Well, do you remember that OpenSSL is a volunteer-based project?

In an ideal world, these volunteers would be able to cover all the use cases only with their good will enerygy. But I guess we do not live in an ideal world and we should first and foremost be grateful for their contribution and try to imitate them to bring our own contributions. (Yep, I'm evangelizing.)

IMHO, the real shame is that sometimes, it seems only volunteers actually produce decently secure libraries... (Yep, I'm bitter too - more and more frequently btw.)

Embedded SSL/TLS lib? What about polarssl?

Posted Jun 23, 2014 14:07 UTC (Mon) by Duncan (guest, #6647) [Link]

I'm not a dev and know little about the inner workings of encryption libs, but from what I've read, polarssl should better fit your needs than nss/openssl/gnutls any of the three, because polar is designed for embedded while the others aren't.

FWIW, I know about polarssl due to long time pan (news/nntp client) list involvement. Pan is gplv2 (only, and as a long existing project without copyright assignment that would be difficult to change) and uses gnutls, which naturally put us in a bind when gnutls tried to go lgplv3, which isn't backward compatible with gplv2. I guess gnutls eventually realized how many projects like pan they were leaving in a bind as they ultimately reverted back to (IIRC) lgplv2.1+, but meanwhile, pan was having to look at alternatives, and polarssl came up as one possibility. That's my only relationship with it. I've never actually used it, just came close when it looked like pan would have to switch due to the gnutls licensing thing, so don't have a clue what it's like to work with as a dev or user.

I just looked at the site ( http://polarssl.org , courtesy of gentoo's package tree database ) and polarssl is dual-licensed gplv2 with FOSS license exception (allowed to ship with popular FOSS licensed software, see site for details) and commercial, for those who refuse to free their software, so that shouldn't be an issue in most cases.

FWIW... Hope that's helpful. =:^)

Duncan

OpenBSD and the latest OpenSSL bugs

Posted Jun 22, 2014 20:30 UTC (Sun) by clemensg (guest, #94377) [Link] (1 responses)

De raadt just can't stop being a dick, he was asked to join and he declined in typical arrogant-de-raadt-fashion. End of story.
I don't blame anybody who stops working with him.

Hopefully one day a new fork will be announced: OpenBSD - deRaadt = LibreBSD

OpenBSD and the latest OpenSSL bugs

Posted Jun 23, 2014 8:09 UTC (Mon) by smurf (subscriber, #17840) [Link]

http://en.wikipedia.org/wiki/Theo_de_Raadt tells us that Theo has forked OpenBSD from NetBSD in 1995 precisely because of that sort of thing, so presumably OpenBSD - deRaadt = NetBSD.

No new fork necessary.