SPF on vger

June 14, 2006

This article was contributed by Jake Edge.

A recent announcement about adding Sender Policy Framework (SPF) capabilities to the machine hosting the linux-kernel mailing list (lkml) has sparked a lively debate. The first step, it seems, is to add an SPF record for vger.kernel.org and later this summer to enable SPF checking on incoming email. Both steps are controversial and the majority of posters seem to be against the change, but Matti Aarnio, one of the postmasters for vger, plans to go ahead with the changes.

SPF is a technique that allows a domain to specify which hosts are allowed to send email that have an envelope sender (i.e. SMTP MAIL FROM) using that domain. A domain administrator adds a TXT record to the DNS entry for the domain that describes all hosts allowed to send mail. This allows receiving Mail Transfer Agents (MTAs) to look up the SPF record and determine whether the domain in the envelope has been forged -- at least in theory.

Unfortunately, there are a number of problems with this scheme, most having to do with email forwarding. Consider the case where a user has a yahoo.com email account that they are forwarding to their ISP. When yahoo forwards email that it receives, it uses the original envelope sender, but that domain has almost certainly not listed yahoo.com as an authorized sender. The same issue occurs if a user is trying to use their yahoo.com email as the sender, but are required to use their ISP's SMTP server. In that case, Yahoo will rightly not have the ISP listed as a legitimate sender for their domain.

The SPF folks have suggested solutions for these problems, but many of them require fundamental changes in how MTAs operate. The Sender Rewriting Scheme (SRS) proposal in particular breaks longstanding email tradition by having forwarding MTAs change the envelope sender as they forward email. Opponents of SPF not only argue that changing this tradition is a bad idea, but also that it is very unlikely to be widely implemented any time soon. Additionally, Mail User Agents (MUAs) would need to learn about SRS encoding in order to parse sender addresses for filtering email at the user end.

SPF does provide a way to definitively determine that an email is coming from an authorized host, but failing the SPF check does not in any way imply that the email is invalid, as the mail could have been forwarded by a non-SRS compliant MTA. The main benefit for domains that publish SPF records may be a reduction in the blowback from a 'joe job' (a spammer uses a victim domain as the sender on a large amount of spam, some of which bounces, leaving the victim to deal with all the bounce messages).

Opponents point out that because of the forwarding problems, publishing an SPF record for your domain essentially asks other MTAs to mark perfectly valid mail as suspicious at best and forged at worst. Worse yet, some mail administrators are configuring their MTAs to reject mail that fails SPF checking.

For the lkml, the immediate impact will be minimal, but still annoying to some. People who have subscribed using addresses that are forwarded to SPF-checking ISPs may no longer receive emails from the list. Some list archiving software may also be affected. Once SPF checking is enabled, some users may find their mail getting rejected depending on how strictly the SPF policy is enforced. Expect another hue and cry on the lkml when and if that happens.

Index entries for this article
GuestArticles	Edge, Jake

SPF is dead. Long live CSV !

Posted Jun 15, 2006 7:06 UTC (Thu) by copsewood (subscriber, #199) [Link]

SPF is a good idea, but in my understanding a badly flawed implementation. This only protects the envelope Mail From (and maybe the HELO) header and (probably rightly) makes no attempt to protect the mail body From: . Trying to make the internal headers meaningful to the human mail recipient is Microsoft's take on this (Patented SenderID which they call SPF version 2) but this makes this concept even more complex and unworkable (as the XML SenderID format is too slow and unreliable in connection with DNS use). Mail authentication technologies which don't protect the body From: can still carry phishes which fool human mail recipients but which appear to authenticate at the MTA level. Technologies which do attempt to protect mail internal headers are more likely to break existing practice. For those wanting the complexity of full message as opposed to just envelope authentication, DomainKey signed mail makes more sense, but still has weaknesses in connection with misleading domain names.

Other than for helping with whitelisting known acceptable mail senders, which you will only be able to do for high volume senders you know and trust, SPF is little used in practice despite the very many domains which have published SPF records. Because of the forwarding problem mentioned in the article, SPF does not give clear enough automated filter choices for MTAs to know whether a domain that publishes a SPF record stating what its legitimate mail sending addresses are, should not appear on envelopes of email coming from elsewhere.

Having successfully implemented this, in my view CSV http://mipassoc.org/csv/ makes a lot more sense for a robust and straightforward MTA rejection-capable technology which ensures the domain of the MTA sending mail across administrative boundaries accepts responsibility for doing this. It solves many of the problems associated with SPF because CSV only concerns the domain in the HELO envelope header.

1. Any domain owner capable of publishing DNS for their domain can state an origin IP address for any subdomain claimed in the HELO header. Suppose someone is sending mail directly to your MTA from a dynamic IP host. Currently you would have good reasons to reject this based on an dynamic IP DNSBL. The collateral damage from this penalises hobbyists running their own mail setups from any dynamic IPs and even static IPs which PTR to script-generated domain names. However, if the HELO domain publishes a CSV record for this IP address and has a good reputation, with CSV you can safely accept this mail. If the HELO domain:

a. doesn't have an acceptable reputation based on an RHS DNSBL or
b. corresponds to a PTR subdomain of the IP block owner who uses CSV to state that mail from this domain subtree should not carry their domain name on HELO envelopes,

In either case your MTA can safely reject such messages.

2. When it becomes possible to establish the domain responsible for directly sending or relaying a message, collaboration determining domain reputations for automated filtering decisions will become much more reliable than trying to do the same job based upon IP addresses. Currently only the IP address on a spam says anything reliable about its origin, and this is usually a compromised Windows zombie. Trying to reject or filter on this basis is analogous to determining the origin of a single sand particle on a moving beach.

SPF: yes, ma'am

Posted Jun 15, 2006 10:11 UTC (Thu) by zmi (guest, #4829) [Link] (3 responses)

SPF is great in helping identify if an incoming e-mail is forged or not.
Imagine a server connects to yours saying "I send mail from
Bill.Gates@microsoft.com to your x@y.z user". Until now, you had to
deliver those, trusting ALL servers on the bad-bad Internet. With SPF, you
can look @microsoft.com if it allows that specific server to send mail
from microsoft.com, and if not, you can just drop the connection.

This way, lots of forged e-mails are prevented, resulting in fewer
problems (here on our servers, YMMV). I've had the problem once that
somebody sent in my name a bad e-mail to our customers - but since SPF
that problem is gone.

The "only" downside is e-mail forwarding. I've had some itches, and it's
not really nice. SRS is too complicated. But until somebody gives me a
better/easier/nicer way to prevent forged e-mails, SPF is my protection of
choice. I'd recommend you turn on checks to SPF on your mailserver, and
see lots of forgeries being dropped.

mfg zmi

SPF: yes, ma'am

Posted Jun 15, 2006 11:50 UTC (Thu) by nim-nim (subscriber, #34454) [Link] (2 responses)

Given the spam/ham ratio nowadays, *any* filter even a random filter will drop spam/forgeries.

The question is not if it's capable of dropping bad messages, but if it can reasonably distingish spam from ham.

So far the evidence in SPF doesn't.

SPF: it's NOT AT ALL about SPAMfiltering...

Posted Jun 15, 2006 16:42 UTC (Thu) by zmi (guest, #4829) [Link]

People keep believing that SPF is to filter SPAM. That's not the target.
SPF can *only* say if an e-mail is forged or not.

If you setup in SPF that from lwn.net only host 1.1.1.1 is allowed to
send, and some server checks SPF and gets a mail from lwn.net from host
2.2.2.2, then it can discard it as forged. That's it.

The fact that SPAM very often sends with forged addresses is just a side
effect. There are other tools for SPAM.

SPF: yes, ma'am

Posted Jun 15, 2006 16:53 UTC (Thu) by iabervon (subscriber, #722) [Link]

It's not surprising that it can't distinguish spam from ham, because it wasn't designed to distinguish spam from ham, contains no support for doing that, and the specification states that it does not do that. It is intended exclusively to distinguish forgeries, and spam is relatively rarely forged. It isn't intended to provide any benefit to the recipient; it's intended to benefit the innocent third party whose address the forger is using.

Of course, it's too soon to use it; the rest of the email system is still such that legitimate operations are standardly done by essentially forging email. At present, there is no standardized and reliable mechanism for a desktop MUA to submit email to the system in a way that authenticates the sender (as controlling the reception of email to the address). And there's the mess with transparent forwarding. (I think SMTP is essentially just a big mess, like a lot of its contemporary protocols, which were done before people had a good understanding of how to design application-layer network protocols effectively.)

SPF on vger

Posted Jun 15, 2006 12:51 UTC (Thu) by job (guest, #670) [Link] (12 responses)

SPF is a bad idea, with a terrible implementation. I can not understand
why anybody would want it. It accomplishes nothing since users care about
mail headers, not smtp headers. It breaks forwarding. It is impossible to
implement for all domains who actually use their TXT records for
something. And it requires most of the world to participate in their
scheme before they it's effective doing nothing.

All this to combat joe-jobs, which aren't a problem to anyone. Spam is
the problem. Which got many SPF proponents to start lying and talk about
it as a spam solution. That was dishonest. I know you want to see your
pet project adopted around the world, but sorry, it is a braindead idea.

But don't take my word for it, read what Allman, Bernstein, Venema and
all the other e-mail luminaries who actually know this stuff wrote. No
one thought it was a good idea, and with very good reasons too.

SPF on vger

Posted Jun 15, 2006 13:34 UTC (Thu) by cate (subscriber, #1359) [Link]

As the open relay. It took a lot of time to close the mail servers, but now I think all major mail servers will reject the mail from open relays.

SPF on vger

Posted Jun 15, 2006 14:04 UTC (Thu) by pizza (subscriber, #46) [Link] (2 responses)

First, don't use the term 'spam', as it's so ambiguious ("unwanted e-mail") as to have no real meaning. Please use a more specific term; you want to deal with trojans or phishing mails differently than the latest Victoria's Secret catalog.

Oddly enough, the former two tend to rely heavily on forged SMTP envelopes, which is precisely what SPF is intended to deal with, and it accomplishes that fairly well. Does it break certian practices? Well, yes. But what its detractors fail to understand is that this is a trade-off that many, many willingly make, especially when it is their reputation and/or money on the line.

Don't forget that these problems exist because of the deficencies of the original SMTP (and yes, DNS) systems.

"Requiring most of the world to participate" is actually a feature of the Internet -- the network is dumb; the end-points are smart. But it also makes change very hard to implement.

As such, the disruption from replacing the whole schebang will be far greater, even though everyone agrees that it's what really needs to be done. And that will certianly break many things that work now.

Incidentally, is there an "official" use for TXT records? "Arbitrary Binary Data up to 255 characters" sounds like there isn't, and a domain owner choosing to use that "arbitrary data" for purposes of reducing forged mail being sent under their domain certianly sounds like an appropriate use.

Using DNS TXT records is a cool idea because it doesn't require any new infrastructure, unlike, for example, using PGP signatures, which works well on an individual basis but otherwise scales terribly due to the necessity of establishing trust anonymously.

SPAM == Unsolicited bulk email. It's simple

Posted Jun 15, 2006 14:26 UTC (Thu) by dwheeler (guest, #1216) [Link]

I completely disagree. Spam has a simple and clear meaning: "Unsolicited bulk email"; it's sometimes called "UBE" instead. "Bulk" perhaps is a little ambiguous, but if you're sending out more than 1000 copies of an email, and the receivers didn't request it (e.g., by joining a mailing list), you are a spammer sending spam.

Some spam has trojans, or other nasty stuff. It's a good idea to protect against other kinds of malicious email. But usually the malicious email is ALSO spam. If we could get rid of spam, we'd greatly reduce any other kind of problem as well on email.

SPF on vger

Posted Jun 22, 2006 15:54 UTC (Thu) by forthy (guest, #1525) [Link]

I agree that SPF is not a good idea, but I support it (not just lip service), for two purposes:

it gets me rid of all those "security updates from Microsoft" before my virus checker gets to see them (since microsoft.com has a SPF record).
it adds up to the pain for using SMTP that it might help to overthrow it - especially when it is adopted.

I view SPF not as a solution to a specific problem, but as a nail on the coffin of SMTP, and I'm ready to adopt the next nail if I can find one.

SPF, joe jobs, and phishing

Posted Jun 15, 2006 17:12 UTC (Thu) by rfunk (subscriber, #4054) [Link] (7 responses)

I agree that SPF is a bad idea, or at least one that demonstrates
insufficient understanding of the way SMTP is used in the real world.

However, i can tell you from firsthand experience that joe jobs are a
real problem, and I can certainly understand why the victims of them
would be tempted to use anything that claims to be a solution. Not only
are joe jobs a problem due to blowback bounces, but also because they
harm the reputation of the victim domain, leading the domain to land on
private blacklists with no recourse for getting off.

Of course, anyone tempted to publish an SPF record as a solution to joe
jobs needs to realize that it's only a solution to the extent that people
check the SPF records, and considering the type of blowback I've been
seeing, many places are still doing little or no spam filtering in the
first place, let alone checking something as questionable as SPF.

It's also worth noting that forged mail is not limited to phishing
attempts; I consider phishing to be a separate problem.

SPF, joe jobs, and phishing

Posted Jun 15, 2006 18:13 UTC (Thu) by dwmw2 (subscriber, #2063) [Link] (5 responses)

There are much better solutions to the problem of bounces to joe-jobs. Solutions which don't require wholesale changes to the way that email works.

SPF, joe jobs, and phishing

Posted Jun 15, 2006 19:32 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

like what?

SPF, joe jobs, and phishing

Posted Jun 15, 2006 21:32 UTC (Thu) by dwmw2 (subscriber, #2063) [Link] (1 responses)

You didn't actually read the why not SPF page linked above, did you?

In particuar, I was thinking of BATV. Not only does it instantly stop the bounces to mail you didn't actually send, but it also allows others to detect fake mail. Try faking MAIL FROM:<dwmw2@infradead.org> to any site which bothers with sender verification callouts to avoid mail from invalid addresses (like sourceforge, amongst many others).

550-Verification failed for <dwmw2@infradead.org>
550-Called:   2001:4bd0:203e::1
550-Sent:     RCPT TO:<dwmw2@infradead.org>
550-Response: 550-This address never sends messages directly, and should not accept bounces.
550-550-Please see http://www.infradead.org/rpr.html or contact
550-550 postmaster@infradead.org for further information.
550 Sender verify failed

SPF, joe jobs, and phishing

Posted Jun 22, 2006 23:51 UTC (Thu) by kitterma (guest, #4448) [Link]

How many of those solutions are accessible to someone who doesn't run their own dedicated mail server?

SPF, joe jobs, and phishing

Posted Jun 15, 2006 19:35 UTC (Thu) by rfunk (subscriber, #4054) [Link]

Note that I mentioned bounces being only part of the problem with
joe-jobs.

SPF, joe jobs, and phishing

Posted Jun 22, 2006 23:48 UTC (Thu) by kitterma (guest, #4448) [Link]

None of which are implementable by someone who isn't running their own mail server and using custom software. With SPF, with all it's flaws, any domain owner that can publish a TXT record in their DNS can gain some protection.

SPF checking may be relatively rare, but in my experience it is enough that within a month of publishing a -all SPF record, bounce messages due to forged sending using my domains ended. There is enough SPF checking going on to provide deterrence.

SPF is a horrible idea in theory. In practice, unless your user base sends to peope who do a lot of forwarding, it works pretty well for many domains. Eventually, it will be obsolete because something better will come along. In the meantime, it does the job for me and lots of others.

SPF and vger

Posted Jun 23, 2006 3:24 UTC (Fri) by SDGathman (guest, #38604) [Link]

SPF is not for everyone - that is why it is optional. However, it is a good fit for vger. The forwarding problems are caused by incorrect *checking* of SPF (neglecting to compile or check a list of non-SRS forwarders), and doesn't affect publishers unless the receiver is *really* braindead and, for example, checks SPF from behind their MX (a common first time user error).

The proposed vger application involves publishing an SPF record. The only potential forwarding issue for publishers is web greeting card type sites that don't use their own domain for the return path. This is not a problem for vger.

I have been publishing and checking SPF in production for 2 years. Most of the complaints from anti-SPF people are based on misunderstandings. For instance, the "forwarding problem" is a "doctor it hurts when I do this" problem. If you have no idea who you get forwarded mail from (even though you set them all up), then don't check SPF. Problem solved.

SRS is actually not a good solution to handle forwarders. Simply listing the forwarders is much cleaner. Listing them is easier if the forwarders publish SPF - cause then you know their IPs automatically. SRS is useful as a BATV alternative that also handles relaying.

The roaming sender problem is elegantly solved using SMTP AUTH - which has been widely available for at least 10 years.

If you publish CSV, you might as well publish SPF for your HELO names and pick up SPF checkers as well (caveat, only if HELO name is distinct from MFROM domain - an SPF namespace collision wart).

SPF on vger

Posted Jun 24, 2006 0:52 UTC (Sat) by neilbrown (subscriber, #359) [Link]

I think there is one potential usage of SPF that is very simple and
should cut down a fair bit of noise in SMTP-space with no real loss of
value.

That is: if an incoming message doesn't get a clear SPF-PASS, then
don't every send an automatic reply to it. This means no
delivery-failure reports, and no vacation messages.

I think it is much better that these don't get sent at all than that
they get sent to the wrong place.

Providing you use the standard 'guess' for domains that don't publish
SPF, you still send most of the report that you need to. And if this
was widely adopted, it might even act as a gentle stick to encourage
more people to regularise the mail sending, and always send through
the right MTA...