Impact of latest Sober virus

Extract of mail traffic summary

Per-Day Traffic Summary                                                         
 date       received  delivered   deferred    bounced   rejected
----------------------------------------------------------------------       
  Nov 20 2005       83         22         0        0         420           
  Nov 21 2005      398        181         0        0         690           
  Nov 22 2005      150         59         0        0        5169           
  Nov 23 2005      433        231         0        0        7517      

delivered email includes verp bounces. but realistically the main list only receives 2-3 messages a day, jamaat lists a few more. So assume that only around 10 “real” messages flow through our server/day.

So what happened, Sober struck again and one of our main mechanisms to survive was our plain-text only policy.

Now would be a good time to tell your friends and family to

Upgrade to Firefox 1.5!

Plain(text) Simple

When you have an important message to convey (or perhaps a not so important one), it’s all too easy to fire up your text editor and send out email with fancy fonts and lots of bright colours. Why not attach an illustrative image or PowerPoint presentation for good measure? It just makes the email as interactive as possible.

These type of thoughts are more common than one might think, and it comes as no surprise to us that people often complain bitterly about our plaintext only email policy.

HTML email may be appropiate for one-to-one correspondence but it really has no place when sending email to a mailing list which has subscribers of varying technical savvyness and of which a majority connect via dial-up telephone lines. Let us describe to you some of the reasons behind our absolute adherence to a no HTML and no attachments policy.

First and foremost, plaintext emails are universally accessible. No matter what email client (email software) the recipient is using, it is almost guaranteed that the message will render (display) correctly on their machine. The message you send is the message that will be received. With HTML email, you are betting that the recipient will have a client capable of rendering your emails correctly. More often that not, this bet pays off the person on the other end will see what you intended. However, with thousands of list subscribers we want to make sure that 100% of our users will be able to read all mail sent over our servers without difficulty.

Another problem with HTML is that it is too frequently used inappropriately. On another mailing list that many of us read, there was a post where every single letter of an email was posted in a different colour from the one preceding it and the one after it. Presumably the author thought that this spiced up their email, whereas in actual fact the email was close to seizure-inducing and very difficult to read. Our eyes are used to reading black and white text. When text is in too many colours in such a short amount of space, our eyes just switch off. We don’t want to get in the business of deciding how much HTML is too much, and what usage of HTML is appropriate for two reasons: first that we feel that plaintext email can sufficiently convey any message that needs to be mass distributed, and second, that it is simply less resource intensive not to have to make these decisions.

Our mail server is configured to bounce HTML email with a pointer to a website which explains how to configure various mail user agents (MUA) and webmail clients to send plain text email. Most posters are able to understand the bounce message and make the appropiate changes but for those who find bounce messages intimidating (can’t blame you), we recommend that you visit this site .

Another can of worms, as we mentioned before, are attachments. Users seem to be conditioned due to their usage of attachments in a corporate/personal environment that sending attachments to mailing lists are okay. Each list adminstrator has their own policy guided by what’s appropiate for the majority of their subscribers and whether a viable alternative exists. Given our user demographics we disallow all attachments on our list. A simple alternative, and one more often than not overlooked, is to upload the file to a website and then provide a link to it within the email. This way only people who want to download the file must use their resources to download it.

The mass sending of attachments over email is a colossal waste of resources. Not only for our servers, but also for the thousands of subscribers who are connecting over dial-up modems and have to spend time waiting for mail that they may or may not be interested in to download. Other mailing lists which cater to a similar audience to ours allow attachments, and we know of many people who complain about spending up to four minutes downloading a single email from these lists (and sometimes there are four or five such messages). We don’t want our lists to be a burden for our subscribers. We want them to be able to receive concise information that is quick to download, and then be able to choose if they want to download further materials.

Combine this with the cache-friendly headers we send out for various file-types including PDFs, GIFs, and JPEGs, and this makes for an efficient information delivery system for jamaats who host their mailing lists and websites with us.

We hope that this helps to explain why we allow only plaintext/attachment-free emails. Our motivation for doing so is to force message posters to send mail in a format which makes it easiest for the recipients to view the message. When a link is provided to further content, the reader is able to choose if they want to read it. The most important thing to remember is that if a message is worth reading, people will read it even in plaintext; and if a message is junk, no amount of colouring and formatting will convince readers that it is anything but that.

Knock-Knock, Come Again

One of the ideas we had implemented for a while on our edge mail infrastructure was the concept of Greylisting.

The goal of greylisting is to reduce the amount of spam that reaches user mailboxes by taking advantage of the fact that spammers do not operate fully standards-compliant mail servers.

The premise behind greylisting is that spammers use a “fire-and-forget” method in sending spam out. If an address doesn’t work, spammers don’t care and don’t bother to retry sending it.

How greylisting takes advantage of this is that the first time mail delivery is attempted, the mail server responds with a “Temporary failure, try again later” message. Legitimate mail servers will try again later, and the second time the message is let through. Spammers won’t try again and hence the message will never get through.

Greylisting enhanced mail servers are usually set up so that once your mail passes through once, your “triplet”–that is the combination of From/To/Mail Server IP–is cached, so that next time it sees the same triplet, it lets you through the first time.

We didn’t move our greylisting system over in the migration right now but may do so in the future.

Building a stronger edge infrastructure for email

qmail has a lot of quirks due to its design of small compartmentalized daemons. One of the issues which impacts us a lot is its delayed-bounce behavior. An out of the box qmail installation only rejects mail for recipient domains not in ‘rcpthosts’ and from senders listed in ‘badmailfrom’.

That means if someone sends a mail to you for a non-existent user at your site, qmail will happily take the mail in, say “250 Ok” to the sender, then figure it has no user to deliver to, generate a bounce and send it back. Now assume someone sends a 5 MB mail: qmail accepts 5 MB in, and tries to send a 5 MB bounce out: 10 MB wasted.

Even worse, if the sender address is forged, qmail may annoy the wrong site with the 5 MB bounce, or it may not be able to deliver the bounce at all, and it may happen that the bounce sticks in your queue until its lifetime is expired.

There have been many patches to qmail (which are not endorsed by the author Daniel Bernstein or djb as he known in the MTA community) but it takes a lot of effort to layer various patches to qmail to give it features which other MTA’s such as Postfix or Exim have out of the box.

As such a few years after we started our site and the spam problem reached an intolerable level, volunteers setup Postfix as our edge-MTA. Postfix has a lot of features particularly in anti-spam capabilities and is actively developed.

In Postfix parlance, we configured Postfix to accept email to a bunch of domains as ‘relay_domains’ with ‘relay_transport’ set to our qmail/ezmlm box (this box had been configured to only accept email from our Postfix box). We also setup a list of valid recipients (after stripping out verp encoded suffixes) so that invalid recipients were rejected at the SMTP transaction phase. We also reject those senders who we would rejected as unknown recipients (reject_unlisted_sender).
We are also grateful to a large mail provider who gives us access to their DNS block-list which blocks a lot of IP’s connecting to us. We have a list of bad sender domains, we reject email from domains which have no DNS A or MX records (you can’t send them a bounce) and we do a decent amount of HELO/EHLO checks

Postfix also has the ability to reject/discard email based on header and body of the email via the use of perl compatible regular expressions and we use this capability to reject pretty much everything except plain-text email. This cuts down a lot of viruses and spam to us and eases the tasks of the moderators as well as the mail admins

Advice to our readers. Don’t use ‘catch-all’ email addresses. If you have your own domain name – they are typically configured to allow ‘anyone’@your-domain-name.com. This is convenient ‘initially’ you may not know which email addresses you want and it also has the benefit of catching mis-spelt email addresses.

But spammers love it – they can send to anything-they-like@your-domain-name.com and you will receive it!. So we would recommend you do it the other way around – instead of receiving everything – specify just the email addresses you want to receive and reject everything else.

Email at the beginning

When we first started out, our focus was to combine two existing mailing lists which were run using majordomo and Listproc. Whilst both were fine mailing list servers, they were a but unwiedly to setup and Listproc at that time required a paid license which made it expensive.

Around the same time (circa 1996-97), qmail was gaining ground as a small/fast/customisable MTA and of particular interest to us was ezmlm a mailing list manager (MLM) which was easy to setup and had allmost all the features we wanted.

Thus our first mail system consisted of qmail as our incoming/outgoing MTA (mail transfer agent) and ezmlm driving all our mailing lists. We were able to work with ezmlm-idx authors to get some of our specific changes back into upstream code

ezmlm is a no-frills MLM with an email-driven command interface. Bounce management via it’s use of VERP encoded sender addresses eases a lot of admin tasks