Some actual data presented anecdotally

Posted Jul 8, 2011 21:49 UTC (Fri) by geuder (subscriber, #62854)
In reply to: Some actual data presented anecdotally by felixfix
Parent article: A decline in email spam?

Mentioning PayPal. This week I received a genuine mail from PayPal, the first for years, because I don't use them regurarly. Gmail put it into the spam folder, obviously the word PayPal is nearly equivalent to spam.

Generally Gmail spam folder receives several false positives each week, maybe some 5% of all detected spam.

Even more worrying is that both gmail and IBM hosted mailservice reject legitimitate mail before SMTP is complete. (I have seen this in logs often enogh, although I don't follow them systematically)

Obviously they have overlooked in their spam filters that mail can be forwarded automatically. So if a Paypal message is forwarded and arrives from a non-paypal server, it must be spam. If your forwarding server forwards a couple of spam messages, subsequent legitimite message might be rejected, because this server sends too much spam.

Yes, spam is down from 20 a day to maybe 5 a day for me. But how much much is caused by rejection during SMTP, I really don't know.

to post comments

Some actual data presented anecdotally

Posted Jul 9, 2011 19:50 UTC (Sat) by anselm (subscriber, #2796) [Link] (16 responses)

It's really a lot better to reject spam during the SMTP dialog rather than after having accepted the message. Once you have accepted the message and decided it's spam, you can't bounce it back to the purported sender since the sender address is likely to be forged. So you write it into the recipient's »spam« folder, which they usually never check, and the message is practically lost.

On the other hand, if you reject spam while the SMTP message is ongoing, the actual sender sees an SMTP error, so if the mail is really legitimate, at least they know that their message can't be delivered. Since, with spam, a large proportion of false positives results from misconfigured DNS at the sender's and things like that, this puts pressure on the senders to fix their systems. You can also try to annoy apparent spammers by »tarpitting«, i.e., putting in deliberate long delays before rejecting their messages, so their servers can't send as much spam out as they would if everyone just accepted it.

Some actual data presented anecdotally

Posted Jul 9, 2011 20:39 UTC (Sat) by geuder (subscriber, #62854) [Link] (15 responses)

> It's really a lot better to reject spam during the SMTP

If you could trust your spam detection algorithms, yes. In practice you have false positives and then rejection is fatal.

> recipient's »spam« folder, which they usually never check,

Many users might not. I always do, and I find false positives every week (Gmail). Sometimes just mailing list messages of little value, but I've also found personal messages from a friend or a job offer (and it was not a pharmacist's job)

Some actual data presented anecdotally

Posted Jul 9, 2011 21:02 UTC (Sat) by felixfix (subscriber, #242) [Link] (10 responses)

It is safe enough to reject mail during the envelope phase. If it is spam, the sender will just discard it and may even blacklist you so as to not waste more time. If it is a valid sender, they can resend. In my case, I run no spam filter, I only drop messages sent to bogus accounts. If someone misspells my account, they can correct it easily enough, if they care.

Even if your spam filter mistakenly rejects a message during the envelope phase, a legitimate sender can obfuscate the buzzwords if they care enough. If it is a bank or such, they shouldn't be sending likely spam words anyway.

Some actual data presented anecdotally

Posted Jul 11, 2011 21:24 UTC (Mon) by lindahl (guest, #15266) [Link] (9 responses)

Most people ignore bounces these days, which means that rejecting apparent spam is the same as blackholing it.

Some actual data presented anecdotally

Posted Jul 11, 2011 22:22 UTC (Mon) by anselm (subscriber, #2796) [Link] (8 responses)

Well, if the senders don't care whether I actually got their mail, it won't matter either way. They will have wasted their time writing it in the first place, not mine.

Those people who do care, however, will at least have the SMTP error message to tell them that their mail couldn't be delivered and why (and pretty much instantly, at that), which is a whole lot better than if their mail had gone to ferment in my spam folder for a few days or even been deleted outright.

Some actual data presented anecdotally

Posted Jul 11, 2011 22:26 UTC (Mon) by lindahl (guest, #15266) [Link] (7 responses)

There is a lot of spam designed to look like bounces. It is difficult for the average user to figure out how to ignore them without ignoring real bounces.

Some actual data presented anecdotally

Posted Jul 11, 2011 22:31 UTC (Mon) by dlang (guest, #313) [Link]

not to mention all the backscatter from when the spammers are using your e-mail address as a source for their spam.

bounce messages are pretty worthless nowdays thanks to this abuse. many mail servers don't even generate them by default any more.

Some actual data presented anecdotally

Posted Jul 11, 2011 22:39 UTC (Mon) by anselm (subscriber, #2796) [Link] (5 responses)

Which is why you don't bounce spam. What you want to do is do your filtering during the SMTP dialog, so you don't even need to accept spam (and bounce it later, file it into the recipient's spam folder, or throw it away). You reject apparent spam before your MTA has accepted responsibility for it, and if there is a false positive, the sender gets a nice SMTP error message, from their own MTA, saying their message couldn't be delivered.

(People who won't look at error messages from their own MTA are beyond help.)

Some actual data presented anecdotally

Posted Jul 11, 2011 22:46 UTC (Mon) by lindahl (guest, #15266) [Link] (4 responses)

I get a lot of fake bounces that look like they come from my own MTA.

Some actual data presented anecdotally

Posted Jul 12, 2011 6:56 UTC (Tue) by anselm (subscriber, #2796) [Link]

So what alternative do you suggest?

Here in Germany, if you're handling mail on behalf of other people it is in many cases illegal to just throw stuff away that looks like spam. So you need to either shove suspicious mail into a »spam« folder which the recipients usually look at only sporadically or not at all , or not accept it at all in the first place, by sending error codes during the SMTP dialogue. This applies not only if you are an ISP, but also to many companies, universities, etc.

People will need to figure out how to read their MTAs' error messages. Otherwise, how are they going to deal with mistyped addresses, full recipient mailboxes, etc. all of which can prevent mail from being delivered even in the absence of spam filtering and backscatter. Also, I personally have been a mail admin for 20 years or so, and I have never to my recollection seen a convincing fake error message from my own MTA.

Some actual data presented anecdotally

Posted Jul 12, 2011 14:04 UTC (Tue) by nye (subscriber, #51576) [Link] (2 responses)

>I get a lot of fake bounces that look like they come from my own MTA

Presumably though if you send a message, and shortly afterwards the same message comes back to you with a note saying that it couldn't be delivered to the address you specified, it's not too hard to differentiate that from fake bounces to messages you clearly never sent.

Anyway in practice people obviously do notice bounces, as they generally manage to do just fine in cases where they've typed the address in wrong, for example.

Some actual data presented anecdotally

Posted Jul 12, 2011 23:09 UTC (Tue) by dmarti (subscriber, #11625) [Link] (1 responses)

This is a good use case for SPF. If your mail server has an SPF record, you can reliably filter out real bounces from forgeries. (I read all my bounces, and would much rather have you bounce my mail than stick it in a spam folder.)

Some actual data presented anecdotally

Posted Jul 13, 2011 0:04 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

You can do that without SPF - BATV solves this problem without breaking things like forwarding.

Some actual data presented anecdotally

Posted Jul 14, 2011 5:17 UTC (Thu) by slashdot (guest, #22014) [Link] (3 responses)

You could also pretend you rejected the e-mail to the SMTP client, but actually still deliver to the Junk mail folder.

Some actual data presented anecdotally

Posted Jul 14, 2011 6:26 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

not easily, because when you reject the mail to the SMTP sender, it's almost always done before the message itself gets delivered to you.

Some actual data presented anecdotally

Posted Jul 14, 2011 8:13 UTC (Thu) by anselm (subscriber, #2796) [Link] (1 responses)

You can still reject a message in the SMTP dialog after having seen the body.

However, I don't see the point of rejecting a message and filing it to the spam folder at the same time. The point of the exercise is to not need a spam folder in the first place, by simply refusing outright to accept mail that would otherwise go in the spam folder.

Some actual data presented anecdotally

Posted Jul 14, 2011 14:39 UTC (Thu) by bronson (guest, #4806) [Link]

In case a filter wants to reject it but an admin or user wants to double check.