Rspamd 1.4 has been released

21 Nov 2016

Today, after 4 months of development, we’ve released major updates for both Rspamd and Rmilter: Rspamd is updated to version 1.4 and Rmilter is updated to version 1.10. These updates include many new features, including Redis pool support, new modules, improved neural networks support, zstd compression for protocol and many other important improvements.

Redis pool support

Rspamd now connects to Redis using a pool of persistent connections. This feature does not require any special setup and allows reuse of existing connections improving load profile for Redis instances. Enabling this feature allowed Rspamd to use Redis more extensively for different tasks.

New neural nets plugin

Neural nets plugin has been reworked to store both training vectors and neural nets in Redis. This change allows to use a single neural network for the whole cluster of Rspamd scanners improving thus both the quality of classification and the speed of training.

Bayes improvements

Some work has been performed to improve the Bayesian statistical classifier. Rspamd now uses more metadata to estimate ham/spam probability. You can read more about Bayes classifier in Rspamd compared to other spam filters here: https://rspamd.com/misc/2016/10/14/bayes-performance.html.

New Antivirus plugin

Rspamd can now check messages for viruses using Antivirus plugin. This module provides multiple features including:

  • different antivirus types support: ClamAV, Sophos and F-Prot
  • support of custom patterns (e.g. experimental databases for ClamAV)
  • support of caching for checks result
  • support of attachments only mode to save AV resources
  • whitelists, size limits and custom condition scripts

New MX check plugin

Rspamd can now verify MX validity for scanned messages using the new MX check plugin. This plugin is useful for protecting from messages with invalid return paths.

Compression support in the protocol

Rmilter and Rspamd now support zstd compression. This algorithm is fast and efficient for reducing of network and CPU load when transferring data over the network. Zstd is also used to store large chunks of data in Redis (e.g. serialized neural nets).

Reworked model for DNS failures in SPF, DKIM and DMARC

Rspamd now has better understanding of temporary failures when performing DNS related checks, e.g. DKIM, DMARC or SPF. There are special symbols to represent both temporary and permanent errors for these plugins.

Adaptive & user-defined ratelimits

Ratelimit module now supports adaptive ratelimits meaning that limits can be made stricter for new and/or bad reputation senders & more lenient for good reputation senders. Furthermore, ratelimits are now composable from keywords providing greater flexibility & user-defined keywords can be created with Lua functions to support custom requirements.

Monitored objects

There is a new concept in Rspamd: monitored resources. This means that Rspamd periodically check if some resource is still available and healthy. For example, this feature is enabled for RBLs and URIBLs. In this mode, Rspamd checks that the DNSBL is available and that it does not blacklist the world. If these checks fail, then a monitored resource is ignored for further checks.

Redis backend for fuzzy storage

It is now possible to store fuzzy hashes in Redis. This storage is more fast, scalable and more featureful than SQLite. rspamadm utility can convert fuzzy hashes from SQLite storage to Redis using fuzzy_convert tool.

Delhash support for fuzzy storage

You can now remove a specific hash from fuzzy storage without a message, you just need to find it in the logs and call rspamc fuzzy_delhash <hex>. Multiple hashes can be specified for this command.

Metric exporter and metadata exporter

Metric exporter allows for periodically pushing Rspamd’s internal statistics to an external monitoring system (currently just Graphite is supported). Metadata exporter is a flexible mechanism for conditionally pushing user-defined message metadata to an external system (current backends are Redis Pub/Sub & HTTP).

Dynamic configuration in Redis

This feature is useful when you want to manage multiple instances of Rspamd centrally. Currently, dynamic configuration is limited to scores of symbols, actions and global enable/disable definitions for symbols only. In future, these functionality is planned to be extended.

Users settings in Redis

Users settings module now supports loading for users settings from Redis server. This is useful feature for dynamic configuration of users’ preferences without reloading of the whole bunch of settings.

Errors ring buffer

Rspamd logger now stores errors in a central ring buffer that contains information about the most recent errors occurred in all Rspamd processes. Controller worker can return this buffer as JSON when asking for /errors path (this requires enable_password).

Messages rework

It is now possible to have multiple messages when returning Rspamd reply, e.g.

{"messages": {"smtp_message": "Try again later"}}

Rmilter 1.10 also supports this to tell MTA some specific error message (e.g. ratelimit or greylisting).

Multiple updates to Rspamd Lua API

There are many new features in Rspamd Lua API:

  • periodic events:
rspamd_config:add_periodic(ev_base, 1.0, function(cfg, ev_base)
  local logger = require "rspamd_logger"
  i = i + 1
  logger.infox(cfg, "periodic function, %s", i)
  return false -- if return false, then the periodic event is removed
end, true)
  • on_load and on_terminate scripts
rspamd_config:add_on_load(function(cfg, ev_base, worker)
  if worker:get_name() == 'normal' then
    -- Do something
  end
end)
  • multiple hashes support
local hash = require "rspamd_cryptobox_hash"
hash.create_specific('md5', 'string'):hex()
-- b45cffe084dd3d20d928bee85e7b0f21
  • HTTPS support in lua_http
  • many improvements in ANN module, including batch training and threaded training
  • zstd compression and decompression support has been added to rspamd_util

Rules improvements

Various new rules to detect suspicious patterns; fixes to improve accuracy. Better HTML rules, fixed various bugs in DNS related services, namely, removed couple of untrusted DNSBLs (SORBS and UCEPROTECT).

WebUI improvements

There are many major improvements to the Rspamd Web Interface including the following:

  • new symbols scores configuration tab:
  • new last errors table in the history tab
  • WebUI is now loaded on demand for each tab
  • updated d3 graphs scripts
  • the default passwords are now BANNED from using in WebUI
  • read-only mode has been added to the interface

Conclusions

Rspamd 1.4 and Rmilter 1.10 are the current stable branches and all users are recommended to update their Rspamd versions. Please read the migration guide if you are unsure about the upgrade process.