There's no doubt we could all use better spam detection tools. I've tried many on the server and client side and have been disappointed, although they all made life a little easier.

We'll blog some of our experiments and observations for the community, as we build better spam filtering for XMS.

One of the challenges in building effective spam filtering is the sheer volume. It consumes bandwidth and hardware resources at an alarming rate. I was prompted to begin this research for faster, more efficient and scalable spam classification tools when a very good and very popular spam filter recently started crashing regualarly. It happens when new viruses are introduced and can last for a month or two. The spam filter doesn't support asynchronous I/O and simply crashes under very high volume. One of our solutions was to use the asyncronous I/O features in XMS to set the threshold for the spam filter. We passed messages that couldn't be handled when volume peaked. The spam filter works very well during off-peak periods, removing about 98% of spam with few false positives. We could also build a spam filter cluster, but that seems expensive and other inherent problems in today's filters prompt us to explore new design possibilities.

This past week, we ran an experiment on a tranactional gateway that should have only received a few thousand messages for the week. We added a Blacklist filter to the gateway server, XMS. The filter checked the sending ip of all incoming messages for a listing in a single blacklist (cbl.abuseat.org was used for the test). We sent delivery status messages to those that were listed, so they could take action if warranted. 80,000 messages were rejected over the week, which was about 90% of all incoming messages. We weren't able to identify any false positives in this method over the test period, which is a big positive, since filtering with regular expressions on message content often creates false positives. Of the spam that passed the filter, we observed that most were new zombies that hadn't been added to the blacklist yet and some originated from email service providers and other marketing gateways that were designed to bypass spam filtering and even some from bonded senders. I think most of them are reputable senders and this number just represented the small percentage of messages that these gateways regularly send that could be considered spam. That's about 3% for most gateways designed to be reputable.

Check back here for other observations made in the ActivSoftware Spam Lab.