Smuggling email inside of email

Normally, when a new vulnerability is discovered and releases are coordinated with those affected, the announcement is done at a convenient time—not generally right before the end-of-year holidays, for example. The SMTP Smuggling vulnerability has taken a different path, however, with its announcement landing on December 18. That may well have been unpleasant for some administrators that had not yet updated, but it was particularly problematic for some projects that had not been made aware of the vulnerability at all—though it was known to affect several open-source mailers.

Discovery and disclosure

The vulnerability was discovered by Timo Longin of SEC Consult back in June; the company contacted three affected vendors, GMX, Microsoft, and Cisco in July. GMX fixed the issue in August, and Microsoft did so in October, but Cisco responded that the "identified vulnerability is just a feature of the software and not a bug/vulnerability". That led SEC Consult to contact the CERT Coordination Center (CERT/CC) for further assistance, using the Vulnerability Information and Coordination Environment (VINCE) tool:

There, we submitted all our research details and explained the case to the vendors involved. We received feedback from Cisco that our identified research is not a vulnerability, but a feature and that they will not change the default configuration nor inform their customers. Other vendors did not respond in VINCE but were contacted by CERT/CC.

Based on this feedback and as multiple other vendors were included in this discussion through the CERT/CC VINCE platform without objecting, we wrongly assessed the broader impact of the SMTP smuggling research. Because of this assumption, we asked CERT/CC end of November regarding publication of the details and received confirmation to proceed.

A talk about the vulnerability was accepted for the 37th Chaos Communication Congress (37C3), which was held at the end of December. So the vulnerability needed to be announced before that happened. The talk acceptance occurred on December 3 and the announcement came out roughly two weeks later—just before the holidays. The "wrongly assessed" wording in the quote above seems to indicate that SEC Consult recognizes that it made a mistake here. In addition, the talk is said to have contained "a decent apology".

One of the other "vendors" mentioned in the (admirably detailed) blog post is Sendmail, but Postfix is mentioned elsewhere as well. It is clear that SEC Consult was fully aware that those two mailers were vulnerable, but the company apparently relied on CERT/CC to involve additional affected projects, which just as clearly did not happen. For example, Postfix creator and maintainer Wietse Venema called the vulnerability announcement "part of a non-responsible disclosure process"; he softened that stance somewhat in a Postfix SMTP Smuggling page, but still noted that "[critical] information provided by the researcher was not passed on to Postfix maintainers before publication of the attack". The result was "a presumably unintended zero-day attack", he said.

Vulnerability

The Simple Mail Transfer Protocol (SMTP), which is described in RFC 5321, provides a text-based protocol for submitting and exchanging email on the net. At some level, this flaw is another indication that the "robustness principle" (also known as "Postel's law") was not actually all that sound from a security standpoint; being liberal in what is accepted over-the-wire has often led to problems. In this case, the handling of line endings in conjunction with the SMTP end-of-data indication can lead to situations where email can be successfully spoofed—leading to rogue email that passes various checks for authenticity.

The SMTP DATA command is used for the actual text that will appear in an email, including headers and the like; the so-called "envelope", which describes the sender and receiver, is another set of SMTP commands (EHLO, MAIL FROM, RCPT TO) that precede the DATA. The blog post announcing the vulnerability has lots of diagrams and explanations for those who want all the gory details. The DATA command is ended with a line that is blank except for a single period ("."); the line endings for SMTP are defined to be carriage-return (CR or "\r") followed by line-feed (LF or "\n"), so end-of-data should be signaled with "\r\n.\r\n".

It turns out that some mailers will accept just a line-feed as the line terminator, but others will not; there is a difference in interpretation of "\n.\n" that can be used to smuggle an email message inside another:

    EHLO ...
    MAIL FROM: ...
    RCPT TO: ...
    DATA
    From: ...
    To: ...
    Subject: ...

    innocuous email text
    \n.\n
    MAIL FROM: <admin@...>
    RCPT TO: <victim@...>
    DATA
    From: Administrator <admin@...>
    To: J. Random Victim <victim@...>
    Subject: Beware of phishing scams

    Like this one
    \r\n.\r\n

There are also variations on the line endings (e.g. "\n.\r\n") that can be used to fool various mailers. The core of the idea is to have the outbound mail server ignore the "extra stuff" as part of the initial email message and send the mail on to a server that sees the single message as more than one—and acts accordingly. SPF checks can be made to pass and even DKIM can be spoofed by using an attacker-controlled DKIM key in the smuggled headers. DMARC can be used to thwart the smuggling, but common configurations of it are still vulnerable. In addition, because mail servers, especially those of the larger email providers, often handle multiple domains, there is an opportunity to smuggle email that purports to come from different domains inside innocuous-seeming messages to users of those services.

The blog post mostly concentrates on the three vendors identified, but clearly notes that "Postfix and Sendmail [fulfill] the requirements, are affected and can be smuggled to". It provides several lists of domains that can be spoofed via GMX, Microsoft Exchange Online, or Cisco Secure Email Cloud Gateway. In fact, SEC Consult uses the Cisco product itself, so the company has changed its settings away from the default to avoid the problem, which Cisco does not acknowledge as any kind of bug.

Postfix and beyond

Meanwhile, over in open-source land, folks seemed rather astonished that the vulnerability dropped with no warning. Marcus Meissner posted about the vulnerability to the oss-security mailing list: "As if we did not have sufficient protocol vulnerability work short[ly] before Christmas break this year, here is one more". He is likely referring to the Terrapin vulnerability in the SSH protocol, which was announced on December 18—well after coordinating with many different SSH implementation projects. The SMTP Smuggling vulnerability followed a rather different path, as Stuart Henderson pointed out:

I'm a little confused by sec-consult's process here. They identify a problem affecting various pieces of software including some very widely deployed open source software, go to the trouble of doing a coordinated disclosure, but only do that with...looking at their timeline... gmx, microsoft and cisco?

Meissner noted that SUSE was not alerted to the problem via VINCE and that the Postfix timeline (in Venema's post) only started after the announcement. Erik Auerswald speculated that SEC Consult expected CERT/CC to alert other affected projects; instead it would seem that CERT/CC gave the go-ahead to release the findings. After Rodrigo Freire wondered why the problem was considered a vulnerability, Auerswald explained:

Any user of an affected outbound server can spoof email from any user of the same outbound server despite SPF and DKIM (DMARC+DKIM can prevent this in some cases, also more senders can be spoofed in specific cases, for details see the blog post). But for this to work, the inbound server must act as a confused deputy. Both outbound and inbound servers need to be differently vulnerable to enable the attack. This specific attack can be prevented unilaterally on either the outbound or the inbound server.

[...] For email server open source projects, relevant for the oss-security list, the primary vulnerability is to act as a confused deputy inbound server, because users of such email servers usually have a much smaller number of accounts than the big freemail providers. But, in general, they could also possibly act as a vulnerable outbound server, e.g., after a [legitimate] user account has been compromised.

CVEs for Sendmail, Postfix, and Exim have been assigned. Postfix has released updates with a new smtpd_forbid_bare_newline option (that defaults to off, but will default to on in the upcoming Postfix 3.9 release); those who do not want to upgrade can work around most of the problems using existing options. Exim has an advisory, bug report, and bug-fix release (4.97.1) for the problem. Sendmail has a snapshot release (8.18.0.2) with a new option to address the flaw as well. However, other than Postfix updates from SUSE and Slackware, most distributions have not yet issued updates for this problem, which leaves a lot of Linux users vulnerable.

So the major open-source mailer projects scrambled to fix the vulnerability over the holidays, but the process has obviously failed here. We have not (yet?) heard from CERT/CC for its side of the story, but either it or SEC Consult should definitely have contacted those projects in the multiple months that have gone by since the discovery. Given that SEC Consult knew about Sendmail and Postfix, though, it is a little hard to understand how a simple heads-up message to the security contacts for those projects was not sent.

It seems likely that there are smaller mailers and other tools that are still affected by the flaw—though they may only have just heard of it—a month or two of advance notice, especially for small projects, could have made a lot of difference. One can only hope that lessons are being learned here and that any coordination will be better ... coordinated (and communicated) down the road.