Mast Kalandar

bandar's colander of random jamun aur aam

Thu, 30 Jul 2009

< The National Policy on Open Standards | · | Thanks IMSc (and Chennai too) >

Mail and Spam handling

Tags: , , , [link] [comments (5)] [raw]

The following text can be classified as a rant so before plunging in I must underline that I have great sympathy for those who administer mail servers and have to deal with complaints about excessive spam on the one hand and about lost mail on the other.

The current fulminations are the result out of spending half a morning and a good bit of an afternoon debugging the problems arising from the following sequence of events:

  1. A couple of days ago one of our users had his webmail session hijacked. This hijacked session has been used to generate a mountain of spam over the last few days.
  2. As a consequence of this activity our site was (justifiably) put on a blacklist by some DNSRBL services.
  3. Some of the domains with which we exchange a lot of e-mail (in both directions) stopped accepting SMTP traffic from us since they "subscribe" to these DNSRBL services. So outgoing mail from us to these domains started bouncing.
  4. We use callout verification of senders. Such verification started failing for incoming mail from domains that refused our SMTP connections (once the cached verification data had expired). So incoming mail from these domains to us also started bouncing.

Of course, there is an all round "blame game" going on about who is responsible.

  1. The big bad bunch of spammers who are making everyone miserable.
  2. The user who used a buggy browser.
  3. The incorrect defaults of the webmail program that does not impose timeouts for session keys.
  4. The lack of system log monitoring that let this go on (for more than a day!) without anyone noticing.
  5. The blind use of DNSRBL's by some mail configurations.
  6. The low cache timeouts for sender verification data by other mail configurations.

However, here is a basic lesson for mail administrators everywhere who are tempted to give the knee-jerk response "Go fix your spamming mail servers!" to mail administrators whose domains they have blacklisted.

It is illogical to reject out-of-hand incoming mail from the same mail exchangers that you are trying to send mail to.

A second lesson is that the net gain in all of this lies with the large Web 2.0 mail sites (like Google and Hotmail). Few dare to blacklist them and they employ an army of people to monitor logs and configurations. It is then only natural that users gravitate to use these sites for all their mail needs.

In the long run that is not a Good Thing(TM).

comment bubble Russell Coker, Fri Jul 31 02:40:53 2009 permanent link

If you used a different IP address for callout verification than you use for sending outbound mail then you would have had a lot less trouble.  Also if you had not used callout verification you would also have had less trouble.

Most mail servers use DNSBL services, it's something you have to get used to.  They can't do otherwise, there's too much spam.

comment bubble Kapil Hari Paranjape, Fri Jul 31 06:38:05 2009 permanent link

Using a different IP address for callout verification looks like a neat idea.

However, callout verification and strict adherence to RFC's are responsible for 50% of our spam filtering so we are loth to drop either!

We use DNSBL services too! My objection is to their use as boolean spam/ham determinators. For example, if they are used as part of a scoring mechanism then "genuine" mail which has very negative spamicity can get through.

comment bubble Simon, Sat Aug 1 01:49:49 2009 permanent link

"We use callout verification of senders."

My sympathy evaporated. You use an abusive spam filtering scheme and it bit you.

"It is illogical to reject out-of-hand incoming mail from the same mail exchangers that you are trying to send mail to."

The RFCs were written such that you can send email when you can't receive it. It is your choice of spam filtering that added an additional requirement. It was widely observed when people started doing this that it would reduce reliability of email delivery, you've just discovered one of the cases where this happens.

Indeed some folks baulked at insisting of the envelope sender having a valid domain (the RFCs didn't insist on that, indeed they were rather too liberal on these kinds of points).

I believe the binary nature of DNSRBLs is good. The problem with statistical measures is the would be genuine sender may end up confused. If I say "you are on blacklist X" he can fix the issue, if I say 'your HELO looks a little suspicious, and you are on RFC ignorant for not accepting postmaster@, and the reverse DNS is a little shakey, and you run on Windows, and....' what are they to do and in which order? For Postfix policy-weightd does a statistical combination of these kinds of checks, which does effective spam filtering but without looking at the content of email, which I'm happily testing even if the output can make getting through rather hard at times.

"... and they employ an army of people to monitor logs and configurations."

I've yet to see any evidence of this from Hotmail. The lights are on but no one is home.

comment bubble Kapil Hari Paranjape, Sat Aug 1 06:25:52 2009 permanent link

Consequent to the negative remarks by Russell and Simon I had another look at available information on "callout verification of senders". I must admit that if I were to (re-)configure our mail server today, it seems likely that I would not use sender callout verification.

Since this is the end of the month, I do have actual statistics about how effective the spam filtering at this site has been.

The following data is for my account (which is one out of about 300 accounts here).

6896 mails have been rejected at SMTP time based on sender verifications.

3244 mails have been sent by procmail to "/dev/null" because their spamassassin scores were stratospheric.

1085 mails have been classified by procmail as "probably-junk" because spamassassin thought they were spam.

2406 mails have been delivered.

Since the post above talks about some cases of false positives so it is impossible to assert that the system was 100% correct. However, based on out-of-band information, the number of false positives was limited to the 10 or so mails already mentioned. The number of false negatives was in the region of 50 --- or 1-2 a day (out of the 2406 mails delivered).

So one argument that is difficult to accept is that sender verification "does not work at all". There are alternative techniques available today (like greylisting and smtp-time-scanning) that would probably work just as well and create less "backscatter". Now, if only someone at IMSc will spend time to configure, check and deploy one of these more modern systems, it would be nice!

comment bubble Kapil Hari Paranjape, Wed Sep 9 07:38:53 2009 permanent link

I believe the binary nature of DNSRBLs is good.
If I say "you are on blacklist X" he can fix the issue

In this context the post Fun with Blacklists by Michal Cihar is probably relevant.

E-mail (will not be displayed)
OpenID (required)
Simple HTML and wiki markup are allowed.


< July 2009 >
    1 2 3 4
5 6 7 8 91011

2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005, 2004, 2003, 2002, 2001, 2000, 1999, 1997, 1995,