Rspamd bayes engine benchmark

2016-10-14 00:00:00 +0200

I have recently decided to compare Bayes classifier in Rspamd with the closest analogues. I have tried 3 competitors:

  1. Rspamd(version 1.4 git master)
  2. Bogofilter - classical bayesian filter
  3. Dspam - the most advanced bayesian filter used by many projects and people

For Dspam, I have tested both chain and osb tokenization modes. I have tried to test chi-square probabilities combiner (since the same algorithm is used in Rspamd), however, I could not make it working somehow.

Testing methodology

First of all, I have collected some corpus of messages with about 1k of spam messages and 1k of ham messages. All messages were carefully selected and manually checked. Then, I have written a small script that performs the following steps:

  1. Split corpus randomly into two equal parts with about 500 messages of Ham and Spam correspondingly.
  2. Learn bayes classifier using the desired spam filtering engine (-d for Dspam, -b for Bogofilter).
  3. Use the rest of messages to test classifier after learning procedure.
  4. Use 95% confidence factor for Rspamd and Dspam (e.g. when probability of spam is less than 95% then consider that a classifier is in undefined state, Bogofilter, in turn, automatically provides 3 results: spam, ham, undefined).

This script collects 6 main values for each classifier:

  1. Spam/Ham detection rate - number of messages that are correctly recognized as spam and ham
  2. Spam FP rate - number of false positives for Spam: HAM messages that are recognized as SPAM
  3. Ham FP rate - number of false positives for Ham: SPAM messages that are recognized as HAM
  4. Ham and Spam FN rate - number of messages that are not recognized as Ham or Spam (but not classified as the opposite class, meaning uncertainty for a classifier)

The worse error for a classifier is Spam False Positive, since it detects an innocent message as Spam. Ham FP and false negatives are more permissive: they just mean that you receive more spam than you want.

Results

The raw results are pasted at the following gist.

Here are the corresponding graphs for detection rate and errors for the competitors.

Conclusions

Rspamd Bayes performs very well comparing to the competitors. It provides higher spam detection rate comparing to both Dspam and Bogofilter. All competitors demonstrated the common spam false positives rate. However, Dspam is more aggressive in marking messages as Ham (which is not bad because Bayes is the only check Dspam provides).

Rspamd is also much faster in learning and testing. With Redis backend, it learns 1k messages in less than 5 seconds. Dspam and Bogofilter both require about 30 seconds to learn.

I have not included SpamAssassin into the comparison since it uses naive Bayes classifier similar to Bogofilter. Hence, it’s quality is very close to Bogofilter's one.

Furthermore, unlike competitors, Rspamd provides a lot of other checks and features. The goal of this particular benchmark was to compare merely Bayesian engines of different spam filters. To summarise, I can conclude that quality of Bayes classifier in Rspamd is high enough to recommend it for using in the production environments or to replace Dspam or Bogofilter in your email system.

Rspamd 1.3.5 has been released

2016-09-01 00:00:00 +0200

The next stable version of Rspamd is now available to download. This release contains a couple of bugfixes and minor improvements.

Termination handlers

Rspamd can now perform some actions on termination of worker processes. For example, it is useful for neural network plugin to save training data on exit. It was also essential for RRD statistics to synchronize RRD on controller’s termination to avoid negative message rates on graphs.

Minimum learns has been fixed

This option was improperly configured previously so it didn’t work as desired. However, it is indeed useful to stop statistical classification before there is enough training for the Bayes classifier. With 1.3.5 release, this option has been fixed.

Rspamd on OpenBSD

There were a couple of bug fixes that allowed Rspamd to run on OpenBSD again. These bugs were cloaked by other systems, however, they were potentially dangerous for those systems as well.

DMARC and DKIM improvements

Andrew Lewis has added various improvements for DKIM, DMARC and SPF plugins to handle cases when the corresponding policies are not listed by senders: e.g. when there is no SPF record or DKIM key for some domain.

Ratelimits improvements

It is now possible to disable ratelimits for specific users.

Mailbox messages and rspamc

Rspamd command line client rspamc can now work with messages in UNIX mailbox format which is sometimes used to store messages on the disk.

Spamhaus DROP Support

Rspamd now supports Spamhaus DROP dns block list that is used to block large botnets over the world.

DKIM verification improvements

Some bugs related to canonicanization of empty messages are fixed in the DKIM plugin.

Fix critical issue with line endings finding

There was a critical bug in Rspamd related to parsing of newlines offsets in a message. In some certain cases it could lead to serious malfunction in URLs detector and some other crucial parts of Rspamd.

Minor bugfixes

There are a couple of minor bugfixes in this release, for example, parsing of \0 symbol in lua_tcp module. HFILTER_URL_ONLY is fixed not to produce overly high scores. All invocations of table.maxn have been removed from Lua plugins as this function is deprecated in Lua.

Rspamd 1.3.4 has been released

2016-08-22 00:00:00 +0200

The new stable versions of Rspamd and Rmilter have been released: 1.3.4 and 1.9.2 accordingly. There are a couple of improvements and some important bugfixes. Please note that in the unlikely case you have used regexp rules in Rmilter then you SHOULD NOT upgrade Rmilter and file a bug report (however, I’m pretty sure that it’s not used by anybody since it hasn’t ever been documented). Here is a list of notable changes in Rmilter and Rspamd.

Rspamd reload command has been fixed

It is now possible to gracefully reload Rspamd configuration by sending HUP signal or by using reload subcommand for the init scripts. Graceful reload is useful when it’s required to update configuration without stopping email processing. During this process, Rspamd starts a new worker processes with the new configuration whilst the existing ones process the pending messages.

Better ASN/country support

ASN/country detection module has been split from the ip_score module allowing use of this data in other modules, for example, in the multimap module to match maps based on country or ASN.

Variable maps in the multimap module

It’s now possible to create maps based on the results of other Lua or internal Rspamd modules. This is particularly useful to link different modules with mulitmap.

DNNSEC stub resolver support

It’s now possible to enable DNSSEC checks in Rspamd through use of a DNSSEC compatible recursive resolver (e.g. Unbound) and check for DNSSEC authentication results in Lua DNS module.

DMARC and DKIM module fixes

There are some important fixes for DMARC and DKIM modules in this version of Rspamd that are related to canonicalization in DKIM and subdomains policies in DMARC.

Redis backend configuration

Now Redis backend in the statistical module can use the global redis settings similar to other modules.

Tasks checksums

Each task and each MIME part now has its own checksum that could be used to detect the same message or the same attachment.

DKIM signature header is now folded by Rspamd

Since DKIM signature header might be quite long, Rspamd now folds it to fit 80 characters wide common for MIME messages.

Ratelimit module fixed

This release of Rspamd fixes a regression introduced in 1.3.3 which prevented the ratelimit module from working properly.

Fixed X-Forwarded-For header processing

Processing of X-Forwarded-For header in the controller has been fixed.

Rmilter configuration improvements

It is now possible to use += operator to append elements to Rmilter lists (e.g. whitelists) and = to redefine the parameter completely. Hosts lists now can contain hostnames along with IP addresses. List parameters can now be empty to clear lists that are non-empty by default. DKIM signing can be completely disabled in the configuration.

Rmilter regexp rules are removed

Support for regexp rules has been removed from Rmilter. This is an old feature which has never been documented nor used by any users. It was likely broken so I have decided to remove it from Rmilter completely to simplify configuration parser and the overall processing logic. If you are using it then do not update Rmilter and please file a bug report in the github issue tracker.

Rmilter bugfixes

Unconditional greylisting support is now restored in Rmilter. Headers added or removed by Rspamd are now treated by Rmilter correctly.