Preface

Rspamd has always been oriented on the performance but it was always quite hard to measure how fast it was as normally it runs just fast enough.

However, I was recently offered to process Abusix Intelligence feeds using Rspamd. These feeds are used to improve Rspamd fuzzy storage quality, to feed URLs and Emails to the DNS black lists provided by Rspamd project and used in SURBL module.

Problem statement

The amount of data that required to be processing is huge - it is about 100 millions of messages per day.

Here is an example to calculate connections count when processing these messages using Rspamd:

$ rspamc stat | \
  grep 'Connections count' | \
  cut -d' ' -f3 ; \
  sleep 10 ; \
  rspamc stat | \
  grep 'Connections count' | \
  cut -d' ' -f3
23548811
23564384

It means that over 10 seconds Rspamd has to process around 15 thousands of messages which gives us a rate of 1500 messages per second.

Rspamd setup

The settings used to process this amount of messages are pretty similar to those that are provided by default.

There is also some significant amount of home-crafted scripts written in Lua to provide the following functionality:

• Provides deduplication to save time on processing of duplicates
• Performs conditional checks for url and emails blacklisting:
  • checks if an url is in whitelists (around 5 whitelists stored in Redis are used)
  • check if an url is already listed
  • check if it matches any suspicious patterns
• Checks if a message should be learned on fuzzy storage (various conditions)
• Stores messages in IMAP folders providing sorting, partitioning and sampling logic
• Doing various HTTP and Redis queries for servicing purposes

Hardware

Now some words about hardware being used.

Previously we have set the same setup on a small instance of AX-60 and it was loaded for around 80%. We have decided to move to a more powerful server to have some margin for processing more emails and doing some experiments.

Hence, we now have an AX-160 AMD server rented in Hetzner. This is quite a powerful machine and the current load pictures look like this one:

top - 14:36:26 up 23:26,  1 user,  load average: 15.76, 13.22, 12.46
Tasks: 511 total,   3 running, 508 sleeping,   0 stopped,   0 zombie
%Cpu(s): 14.1 us,  4.6 sy,  0.0 ni, 78.9 id,  0.0 wa,  0.0 hi,  2.4 si,  0.0 st
MiB Mem : 128802.5 total,  56985.7 free,  27897.5 used,  43919.3 buff/cache
MiB Swap:   4092.0 total,   3925.5 free,    166.5 used. 100018.6 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 14085 unbound   20   0 2058412   1.6g   6852 S 131.2   1.3   1478:04 unbound
 66509 _rspamd   20   0  806976 733336  23592 S  68.8   0.6 169:52.21 rspamd
 66498 _rspamd   20   0  780144 699540  23852 S  62.5   0.5 156:19.14 rspamd
 66502 _rspamd   20   0  816152 744352  23796 S  56.2   0.6 164:26.39 rspamd
 66468 _rspamd   20   0  773532 697084  23736 S  50.0   0.5 117:36.32 rspamd
 66491 _rspamd   20   0  806652 722340  23728 S  50.0   0.5 148:04.54 rspamd
 66476 _rspamd   20   0  767300 705996  23596 S  43.8   0.5 129:04.30 rspamd
 66481 _rspamd   20   0  797944 730528  23896 S  43.8   0.6 139:34.35 rspamd
 66443 _rspamd   20   0  727632 657104  23372 S  37.5   0.5  88:39.26 rspamd
 66451 _rspamd   20   0  742192 665196  23632 S  37.5   0.5  94:49.75 rspamd
 66456 _rspamd   20   0  790908 725784  23488 S  37.5   0.6 101:32.06 rspamd
 66463 _rspamd   20   0  771540 696064  23692 S  37.5   0.5 108:08.65 rspamd
 66487 _rspamd   20   0  780220 713024  23428 S  37.5   0.5 144:51.79 rspamd
 66447 _rspamd   20   0  762440 689592  23736 S  31.2   0.5  90:23.93 rspamd
 66455 _rspamd   20   0  763520 696108  23580 S  31.2   0.5  97:57.57 rspamd
 66464 _rspamd   20   0  764644 688724  23696 S  31.2   0.5 111:32.74 rspamd
 66469 _rspamd   20   0  756952 678704  23612 S  31.2   0.5 127:55.02 rspamd
127011 rbldns    20   0  358824 307700   2244 R  31.2   0.2  10:26.14 rbldnsd
 10767 redis     20   0 9912104   7.7g   2532 S  25.0   6.1 236:29.63 redis-server
 66438 _rspamd   20   0  746772 680624  23424 R  25.0   0.5  82:18.04 rspamd
 66433 _rspamd   20   0  751180 687244  23472 S  18.8   0.5  80:12.21 rspamd
 66437 _rspamd   20   0  737200 669428  23796 S  18.8   0.5  81:37.81 rspamd
 10671 stunnel4  20   0   24.0g  77252   3644 S  12.5   0.1 269:06.53 stunnel4
 26994 root      20   0   11900   3984   3072 R  12.5   0.0   0:00.02 top
 66442 _rspamd   20   0  808808 707020  23608 S  12.5   0.5  85:11.64 rspamd
 17821 clickho+  20   0   21.8g   3.9g  18964 S   6.2   3.1 116:13.04 clickhouse-serv

Rspamd is also being fed via proxy worker that runs on another host and performs initial data collection and emitting messages via the Internet providing transport encryption using HTTPCrypt. However, its CPU usage is quite negligible - it uses only a single CPU core by around 40% in average.

Results analytics

As you can see, this machine runs also Clickhouse, Redis, own recursive resolver (Unbound), and it still has ~80% idle processing these 1500 messages per second.

If we look at the performance counters by attaching to some of the worker processes, we would see the following picture:

# timeout 30 perf record -p 66481
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.171 MB perf.data (29833 samples) ]
# perf report

# Overhead  Command  Shared Object            Symbol
# ........  .......  .......................  .......................................................................
#
     5.23%  rspamd   rspamd                   [.] lj_alloc_free
     3.35%  rspamd   rspamd                   [.] lj_str_new
     3.03%  rspamd   librspamd-server.so      [.] gc_sweep
     2.20%  rspamd   rspamd                   [.] lj_alloc_malloc
     1.94%  rspamd   rspamd                   [.] gc_sweep
     1.50%  rspamd   libc-2.28.so             [.] __strlen_avx2
     1.32%  rspamd   rspamd                   [.] release_unused_segments
     1.24%  rspamd   rspamd                   [.] lj_BC_TGETS
     1.17%  rspamd   libjemalloc.so.2         [.] free
     1.04%  rspamd   librspamd-server.so      [.] lj_BC_JLOOP
     1.03%  rspamd   librspamd-server.so      [.] propagatemark
     1.01%  rspamd   libpthread-2.28.so       [.] __pthread_mutex_lock
     1.01%  rspamd   libglib-2.0.so.0.5800.3  [.] g_hash_table_lookup
     0.94%  rspamd   libjemalloc.so.2         [.] malloc
     0.77%  rspamd   rspamd                   [.] lj_func_newL_gc
     0.76%  rspamd   rspamd                   [.] propagatemark
     0.75%  rspamd   rspamd                   [.] lj_tab_get
     0.69%  rspamd   libpthread-2.28.so       [.] __pthread_mutex_unlock_usercnt
     0.65%  rspamd   librspamd-server.so      [.] t1ha2_atonce
     0.61%  rspamd   librspamd-server.so      [.] newtab
     0.60%  rspamd   libicui18n.so.63.1       [.] icu_63::NGramParser::search
     0.59%  rspamd   [kernel.kallsyms]        [k] copy_user_generic_string
     0.58%  rspamd   librspamd-server.so      [.] match
     0.58%  rspamd   librspamd-server.so      [.] lj_tab_new1
     0.56%  rspamd   librspamd-server.so      [.] rspamd_task_find_symbol_result
     0.52%  rspamd   [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
     0.48%  rspamd   librspamd-server.so      [.] rspamd_vprintf_common
     0.46%  rspamd   librspamd-server.so      [.] lj_str_new
     0.42%  rspamd   rspamd                   [.] index2adr
     0.42%  rspamd   rspamd                   [.] lj_BC_CALL
     0.42%  rspamd   libc-2.28.so             [.] __strcmp_avx2
     0.42%  rspamd   libc-2.28.so             [.] __memmove_avx_unaligned_erms

The top consumers are Lua allocator and garbage collector. Since we are using Rspamd experimental package on Debian Buster, then it is built with bundled LuaJIT 2.1 beta3 and Jemalloc allocator, however, it seems that there is some issue with this allocator in Debian Buster, so I had to load it manually via the following command:

# systemctl edit rspamd.service

[Service]
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"

Followed by restarting of Rspamd.

It is interesting that this Rspamd setup accepts all connections encrypted using HTTPCrypt but chacha_blocks_avx2 takes less than 0.16% of CPU according to perf report.

This particular instance of Rspamd is slightly tuned to use more memory to save some CPU cycles:

# local.d/options.inc

lua_gc_step = 100;
lua_gc_pause = 400;
full_gc_iters = 10000;

These options tell Rspamd to preserve Lua objects in memory for longer time, at the same time in this mode, we can also observe GC stats on workers that performs full GC loop each 10k messages being scanned:

$ tail -f /var/log/rspamd/rspamd.log | fgrep 'full gc'

perform full gc cycle; memory stats: 58.66MiB allocated, 62.01MiB active, 6.08MiB metadata, 84.71MiB resident, 90.64MiB mapped; lua memory: 107377 kb -> 38015 kb; 308.0022420035675 ms for gc iter

As you can see, full GC iter takes quite a significant time. However, it still keeps Lua memory usage sane. The ideas behind this GC mode have been taken from the generational GC idea in LuaJIT Wiki.

Resulting graphs

Here are some UI captures taken from a previous machine:

As you can observe, there was some HAM portion increase over the recent days, however, it was caused by adding new sampling logic and duplicates filtering to save CPU resources (these messages are marked as ham and excepted from scan).

There is also a Clickhouse based dashboard that’s created using Redash:

Since we have Clickhouse on board, we can do various analytics. Here is an average scan time for messages:

:) select avg(ScanTimeVirtual) from rspamd where Date=today();

SELECT avg(ScanTimeVirtual)
FROM rspamd
WHERE Date = today()

┌─avg(ScanTimeVirtual)─┐
│    95.62269064131341 │
└──────────────────────┘

… and average size of messages:

:) select median(ScanTimeVirtual) from rspamd where Date=today();

:) select avg(Size) from rspamd where Date=today();

SELECT avg(Size)
FROM rspamd
WHERE Date = today()

┌──────────avg(Size)─┐
│ 1778.31            │
└────────────────────┘

Conclusions

So with this load rate (1500 messages per second) and with the average size of messages around 2Kb, Rspamd processes each message in around 100ms in average. I hope these numbers could give one some impression about Rspamd performance in general.

I would like to give the main kudos to Abusix who are constantly supporting Rspamd project and have generously provided their amazing spam feeds to improve Rspamd quality!

We have released Rspamd 1.9.3 today.

This release contains some new features and many bug fixes. There are no incompatible changes introduced with this release to our best knowledge.

This release includes the following features and important changes.

Hashicorp Vault support

From version 1.9.3, Rspamd can use Hashicorp Vault to store and manage DKIM keys. Vault usage provides secure and flexible storage of the private keys that can scale and use various backends to store sensible data (secrets).

There is a new subcommand for rspamadm utility called vault that is intended to create/remove and securely rotate DKIM private keys using vault.

You can read more about it using the following link: https://rspamd.com/doc/modules/dkim_signing.html#dkim-signing-using-vault

Added least passthrough result

Some modules should set metric result as least possible result. For example, DMARC policy failure should at least mark failed messages as spam but it should not prevent messages from being rejected. From this release, such modules use least policy to set actions allowing to apply a more strict policy if needed.

Tunable memory management

From this version, Rspamd allows to manage memory policies for Lua garbage collection allowing to fit memory/cpu constraints more flexible. When Rspamd is built with jemalloc (e.g. in the default packages provided by the project), it can also print detailed memory statistics on full gc loops.

Here is an example for tuning GC in Rspamd when there are lots of free memory available (around 1Gb per scanner process):

# local.d/options.inc
# http://pgl.yoyo.org/luai/i/2.10+Garbage+Collection
lua_gc_step = 100;
lua_gc_pause = 400;
# number of scanned messages to perform full GC iteration 
full_gc_iters = 10000;

Improved oversigning logic

It is now possible to oversign existing only headers and ignore it if a header is missing. It is done by changing (o) to (x).

The default list of headers signed is changed accordingly:

Important bugs fixes

Here is the list of the most important bugs fixes:

• HTML: Fix size attribute processing - this issue caused rule MANY_INVISIBLE_PARTS to be improperly triggered on many HTML messages
• Do not blacklist mail by SPF/DMARC for local/authed users
• Lots of Clickhouse plugin fixes
• Fix buffer overflow when printing small floats - this issue caused random crashes in WebUI reported by many users
• Fix DoS caused by bug in glib - details in https://gitlab.gnome.org/GNOME/glib/issues/1775

Full list of the meaningful changes

• [Conf] Add IP_SCORE_FREEMAIL composite rule
• [Feature] Add cryptobox method to generate dkim keypairs
• [Feature] Add fast hashes to lua cryptobox hash
• [Feature] Add least passthrough results
• [Feature] Allow oversign if exists mode
• [Feature] Clickhouse: Modernise table initial schema
• [Feature] Implement IUF interface for specific fast hashes
• [Feature] Lua_util: Allow to obfuscate different fields
• [Feature] Tune memory management in Rspamd and Lua
• [Fix] Avoid buffer overflow when printing long lua strings
• [Fix] Change the default oversigning headers to a more sane list
• [Fix] Clickhouse: Do not store digest as it is not needed now
• [Fix] Clickhouse: Fix lots of storage issues
• [Fix] Clickhouse: Support custom actions
• [Fix] Deny URLs where hostname is bogus
• [Fix] Do not blacklist mail by SPF/DMARC for local/authed users
• [Fix] Fix DoS caused by bug in glib
• [Fix] Fix UCL parsing of the multiline strings
• [Fix] Fix buffer overflow when printing small floats
• [Fix] Fix init code for servers keypairs cache
• [Fix] Fix issue with urls with no tld (e.g. IP)
• [Fix] Fix memory in arc signing logic
• [Fix] Fix memory leak in language detector during reloads
• [Fix] Fix mixed case content type processing
• [Fix] Fix processing of the ip urls in file
• [Fix] Fix use after free
• [Fix] HTML: Fix size attribute processing
• [Fix] Hum, it seems that 99ff1c8 was not correct
• [Fix] Lua_task: Fix task:get_from method
• [Fix] Preserve fd when mapping file to scan
• [Fix] Re-use milter_headers settings when doing arc signing
• [Fix] Set dmarc force action as least action
• [Fix] Switch to GMT
• [Fix] allow PKCS7 signatures to be text/plain, too
• [Project] Add initial version of the vault management tool
• [Project] Add vault support for DKIM and ARC signing
• [Project] Implement keys rotation in the vault
• [Project] Improve dkim keys generation for vault
• [Project] Improve keys creation in rspamadm vault
• [Rework] Move lua_worker to a dedicated unit
• [WebUI] Add URL fragments (#) support
• [WebUI] Fix AJAX request URL

We have released Rspamd 1.9.2 today.

This release contains some new features and bug fixes. The only potentially slashing changes are the changes in Clickhouse module:

• Times are now stored in GMT timezone so you can use Clickhouse for analytics that crosses time zones. The potential drawback is the mess with the currently stored data. This should be resolved automatically once new data arrives.

• Clickhouse schema has been updated to the version 4 with new fields and some minor changes. The existing database should be converted automatically and there are no incompatible changes in columns.

This release includes the following features.

Improvements in Clickhouse plugin

Rspamd now stores more data in Clickhouse:

• Mime recipients
• Message IDs
• Scan time for a message, both normal and virtual
• SPF checks results
• Some new calculated columns, such as MIMERcpt, MIMEFrom, SMTPFrom and SMTPRcpt

These columns are intended to improve analytical capabilities of Clickhouse plugin.

OpenDKIM compatible DKIM signing setup

This version now includes a simplified DKIM signing setup option inspired with OpenDKIM.

You can read more about it here: https://rspamd.com/doc/modules/dkim_signing.html#use-of-signing_table

This mode is intended to simplify migration from the existing setups based on OpenDKIM to Rspamd.

Better encrypted archives support

Rspamd can now properly detect encryption in ZIP archives. Mime types plugin now also tries to resolve hex encoding hack used by some spammers to send malware to users (see PR 2582).

Calendar files parser

From the version 1.9.2, Rspamd can extract meaningful data from Calendar files in iCal format (.ics files). These files are sometimes used by spammers so Rspamd can now extract hyperlinks and emails from calendar attachments to improve filtering quality.

New rspamadm dns_tool utility

It is now possible to do some DNS checks with the new tool. For example, it is now possible to verify SPF records as they are observed by Rspamd, including elements extraction, for example a or mx and verification of the IP addresses. Here is how it looks like:

Better bitcoin addresses detection

We have improved bitcoin addresses detection by fixing some issues in the BTC wallet validation code. It now allows to catch Pay-To-Script addresses.

Full list of the meaningful changes

• [Conf] Allow to load users plugins from plugins.d
• [Conf] oversign openpgp and autocrypt headers
• [Feature] Add SPF FFI library for Lua
• [Feature] Add more verbosity for SPF caching
• [Feature] Antivirus: Handle encrypted files specially
• [Feature] Clickhouse: Slashing - add new fields to CH
• [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table
• [Feature] Dkim_signing: Allow to use new options as maps
• [Feature] Import fpconv library
• [Feature] Lua_maps: Allow static regexp and glob maps
• [Feature] Parse ical files
• [Feature] Rspamadm: Add dns_tool utility
• [Feature] Store SPF records digests
• [Feature] Use fpconv girsu2 implementation for printing floats
• [Fix] Clickhouse: Use integer seconds when inserting rows
• [Fix] Fix floating point printing
• [Fix] Fix processing of embedded urls
• [Fix] Lua_clickhouse: Fix CH errors processing
• [Fix] Make spf digest stable
• [Fix] Properly detect encrypted files in zip archives
• [Fix] Slashing: Store times in GMT timezone in ClickHouse
• [Rules] Add additional conditions to perform BTC checks
• [Rules] Fix pay-to-hash addresses validation

We have released Rspamd 1.9.1 today.

This release includes one potentially dangerous change: all configuration files are now preprocessed using Jinja templates.

Hence, if you have sequences like {=/=}, or {%/%}, or {#/#} anywhere in the configuration files including even comments then you need to take extra care when moving these configuration to the new version! There are workarounds described above to do that.

Here is the list of the most important changes in this version.

Jinja templates in the configuration

From version 1.9.1, Rspamd supports Jinja2 templates provided by Lupa Lua library. You can read the basic syntax documnentation and the abilities provided by these templating engines using the links above. Rspamd itself uses a specific syntax for variable tags: {= and =} instead of the traditional {{ and }} as these tags could mean, e.g. a table in table in Lua.

Templating might be useful to hide some secrets from config files and places them in environment. Rspamd automatically reads environment variables that start from RSPAMD_ prefix and pushes it to the env variable, e.g. RSPAMD_foo=bar comes to env.foo="bar" in templates.

New template subcommand in Rspamadm

Rspamadm has now template subcommand to apply templates engine to the input file or files:

Options supported:

Changes in URLs extraction for HTML parts

Rspamd now tries to extract URLs from plain text of HTML parts. Unfortunately, despite of being contraversal, some Email clients do that as well. One of the notable example is Outlook. Hence, from this release Rspamd also looks for URLs in plain HTML text.

Per user settings for mime_types plugin

Mime types plugin now supports per user settings to allow individual black and white lists of extensions. Here is an example to increase score for exe extensions for some specific user:

test {
  from = "user@example.com";

  apply {
    plugins {
      mime_types = {
        bad_extensions = {
          exe = 100500,
        }
      }
    }
  }
}

Mime types plugin now also supports reverse mapping of content type to extension to allow processing of attachments where an exact file name is not specified.

Better greylisting conditioning

It is now possible to disable or enable greylisting in Rspamd based on the presence of some specific symbols. This feature allows more fine grained greylisting control.

Bitcoin addresses validation

It is not a secret that the wave of spam and scam related to crypto currencies has been flooding the email flows in the recent time. Rspamd has a special rule called LEAKED_PASSWORD_SPAM to block some of the scam types. In this version, Rspamd also checks bitcoin wallets to distinguish them from random long strings to reduce false positives rate significantly. It also allows to build a database of wallets used for scam and spam.

Replies plugin validation

Replies plugin now stores the from/reply-to address when tracking outbound messages and whitelists merely replies that come that address. It helps to avoid replies abusing where spammers were able to catch some legit message ids somewhere in public lists and used them in In-Reply-To headers to dodge spam filtering in Rspamd.

List of major bug fixes

This version includes some important fixes:

• Add crash safety for HTTP async routines
• Clickhouse: Fix table schema upload
• Core: Fix squeezed dependencies handling for virtual symbols
• Finally fix default parameters parsing in actions section
• Fix ES sending logic (restore from coroutines mess)
• Fix finishing script for Clickhouse collection
• Fix priority for regexp symbols registration
• Neural: Fix training
• Rework cached Redis logic to avoid sentinels breaking
• SURBL: Fix regression in surbl module
• Fix double signing in the milter

Full list of the meaningful changes

• [Conf] Add vendor groups for symbols
• [Feature] Add rspamadm template command
• [Feature] Allow to add messages from settings
• [Feature] Allow unconnected DNS servers operations
• [Feature] Check limits after being set, migrate to uint64
• [Feature] Greylist: Allow to disable greylisting depending on symbols
• [Feature] Improve lua binary strings output
• [Feature] Mime_types: Implement user configurable extension filters
• [Feature] Mime_types: When no extension defined, detect it by content
• [Feature] Preprocess config files using jinja templates
• [Feature] Replies: Filter replies sender to limit whitelisting to direct messages
• [Feature] Treat all tags with HREF as a potential hyperlinks
• [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM
• [Fix] Add crash safety for HTTP async routines
• [Fix] Another fix for Redis sentinel
• [Fix] Clickhouse: Fix table schema upload
• [Fix] Core: Fix squeezed dependencies handling for virtual symbols
• [Fix] Finally fix default parameters parsing in actions section
• [Fix] Fix ES sending logic (restore from coroutines mess)
• [Fix] Fix finishing script for clickhouse collection
• [Fix] Fix priority for regexp symbols registriation
• [Fix] Fix various issues found by PVS Studio
• [Fix] Initialize lua debugging earlier
• [Fix] Neural: Fix training
• [Fix] Rework cached Redis logic to avoid sentinels breaking
• [Fix] SURBL: Fix regression in surbl module
• [Fix] Fix double signing in the milter
• [Project] Add support of HTTP proxy in requests
• [Rework] Change lua global variables registration
• [Rework] Rework HTML content urls extraction
• [Rework] Start rework of aliasing in Rspamd
• [WebUI] Combine Scan and Learning into one tab
• [WebUI] Fix symbol score input type
• [WebUI] Show grayed out pie
• [WebUI] Update Throughput summary values dynamically

We have released Rspamd 1.9.0 today.

There are various important features in this release. The vast majority of those should not have any impact on the existing systems. However, you are recommended to read the Upgrade Notes.

This release contains lots of improvements, reworks and bugs being fixed. Here is a list of the most important changes in this release:

External services

Rspamd is now shipped with the external services module contributed by Carsten Rosenberg. This module provides a generic integration with the following services:

• Generic ICAP protocol:
  • ClamAV (using c-icap server and squidclamav)
  • Sophos (via SAVDI)
  • Symantec Protection Engine for Cloud Services
  • Kaspersky Web Traffic Security 6.0
• OleTools
• DCC
• VadeSecure

This plugin is a part of a more generic lua_scanners framework that allows more flexible integration with different Antivirus and AntiSpam OEM services.

New mime modify tool in Rspamadm

Rspamadm mime subcommand now allows to modify messages. This tool allows to add or remove headers in a message, add footers in HTML and text parts and do some subject rewriting. For example, to add a footer and rewrite subject in a message, one can use the following command: rspamadm mime modify --text-footer=footer.txt --html-footer=footer.html --rewrite-header="Subject=TEST: %s".

This tool has full MIME support (including multipart messages), supports various messages encoding and convert those to UTF8, allows to modify both plain old messages and multipart messages with attachments. It also properly detects and excludes GPG signed/encrypted messages.

Offline DKIM signing tool in Rspamadm

Another tool that has been added to rspamadm is sign subcommand. This command allows to perform DKIM signing using a specific private key for some message or messages. It can either return isolated header or modify the message itself.

In conjunction with the previous tooling, it could be used to modify and sign messages produced by mailing lists or some local forwarding scripts.

Please bear in mind that this tool is available when using LuaJIT only as it requires FFI interfaces.

HTTP Keep-Alive support

From this version, Rspamd supports keep-alive in its HTTP Lua client. This could be used to implement high frequent requests to some external services and to reduce load by keeping the pool of HTTP connections instead of opening a connection per each request.

New Lua UDP client

Rspamd now support sending of generic UDP requests as well as TCP ones. There are various modes supported, both one-way and two-ways with optional retransmits and timeouts. This module comes with the documentation available.

Better unicode normalisation

This version comes with further improvements towards unicode normalisation and detecting anomalies. Words now are sanitized from any combining characters after translating to NFKC form.

Configuration graph utility

You can now visualise your configuration by building the include graph. There is a new tool called rspamadm configgraph that takes configuration and convert it to the graphviz DOT graph.

Flexible actions support

Rspamd now support any actions definitions in addition to normal ones. You can set custom actions with thresholds or without (e.g. to set action equal to phishing or to social). All actions should be explicitly defined in the configuration.

New Received headers parser

We have migrated from a strict RFC compatible state machine to a custom parser for the Received headers. This change allows to extract more data from non-conforming Received headers used by some MTA (Exim is one of the notable examples).

Telephone URLs support

Rspamd now parses and processes telephone URLs. That allows to build blacklist for spam/scam/phishing phones as well as plain URLs.

Support for ED25519 signatures

Rspamd now supports dual signing and ed25519 DKIM signatures. New ed25519 keys could be generated using rspamadm dkim_keygen tool:

rspamadm dkim_keygen -t ed25519
CuHc4MOZYXEVH0M4WFQHL5UC2NbVJO8aq2CjGNznxm36mJPlu9GVMfq0lQI1dkeoHByqfsJgMgnCX0vFeMkjoA==
selector._domainkey IN TXT ( "v=DKIM1; k=ed25519; "
	"p=+piT5bvRlTH6tJUCNXZHqBwcqn7CYDIJwl9LxXjJI6A=" ) ;

Ed25519 keys are much shorter than RSA providing RSA2048 security margin in just 32 bytes for public keys. Unfortunately, these signatures are not widely supported so dual signing is still required. We believe that support of the modern algorithms in Rspamd would those algorithms to spread.

Custom functions in Regexp module

Regexp module is now extended with custom Lua functions support. This feature allows to mix fast regexp rules and custom logic of Lua rules without explicit composite rules.

Added support of gzip archives

Rspamd now support reading filenames from GZip archives that are surprisingly often used by some spam senders. With this feature, Rspamd can filter some tricky scam emails that are targeting to install backdoors or malware on users’ machines (e.g. cryptolockers).

Additional detection of types for attachments

To filter malware and bad attachments that are somehow hidden by malicious Content-Type header, Rspamd also performs libmagic scan on attachments to detect the real type by its content. This is useful to detect and filter some tricky malware that utilizes bugs in popular email clients.

Lots of major fixes

This version includes many major fixes that required massive rework to improve stability and performance:

• Race conditions in the maps reading code
• Support RFC2231 encoding in headers
• Better zero characters handling
• Better HTML parsing and handling of the URLs
• Coroutines are now explicitly separated from the async code to prevent tricky race conditions that caused crashes on certain load
• Allow to disable/enable composite symbols
• Lots of fixes in 7z processor
• Detect encrypted rarv5 archives
• Fix ETags support
• Fix processing of NDNs of certain type
• Improved Content-Type parsing
• Fix deletion of the duplicate headers
• Fix parsing of mime parts without closing boundary

We recommend to update Rspamd to this version to apply all these features and fixes.

Full list of the meaningful changes

• [Conf] Add missing includes
• [Conf] Move to options
• [Conf] Rbl: DWL is actually special whitelist
• [Conf] Relax some uribl rules
• [Conf] Remove abuse.ch
• [CritFix] Html: Entities are not valid within tag params values
• [Feature] Add rspamadm mime sign tool
• [Feature] Add configgraph utility
• [Feature] Add dedicated ZW spaces detection for URLs
• [Feature] Add flag to url object when visible part is url_like
• [Feature] Add method task:lookup_words
• [Feature] Add pyzor support (by crosenberg)
• [Feature] Allow to add upstream watchers to Lua API
• [Feature] Allow to set rewrite subject pattern from settings
• [Feature] Better escaping of unicode
• [Feature] Clickhouse: Allow to store subject in Clickhouse
• [Feature] Core: Add QP encoding utility
• [Feature] Core: Add libmagic detection for all parts
• [Feature] Core: Add support for gzip archives
• [Feature] Core: Allow to construct scan tasks from raw data
• [Feature] Core: Detect charset in archived files
• [Feature] Core: Ignore and mark invisible spaces
• [Feature] Core: Normalise zero-width spaces in urls
• [Feature] Core: Process data urls for images
• [Feature] Core: Relax quoted-printable encoding
• [Feature] Core: Support RFC2231 encoding in headers
• [Feature] Core: Support telephone URLs
• [Feature] Core: allow to emit soft reject on task timeout
• [Feature] DCC: Add bulkness and reputation checks to dcc
• [Feature] Elastic: Modernize plugin
• [Feature] Export visible part of url to lua
• [Feature] Fuzzy_storage: add preliminary support of rate limits
• [Feature] HTML: Specially treat data urls in HTML
• [Feature] Implement event watchers for upstreams
• [Feature] Implement includes tracing in Lua
• [Feature] Improve dkim part in configwizard
• [Feature] Lua_scanners: Add VadeSecure engine support
• [Feature] Lua_task: Add flexible method to get specific urls
• [Feature] Mime_types: Add MIME_BAD_UNICODE rule
• [Feature] Mime_types: Use detected content type as well
• [Feature] Plugins: Add preliminary version of the external services plugin
• [Feature] Query sentinel on master errors
• [Feature] Regexp: Allow local lua functions in Rspamd regexp module
• [Feature] Rspamadm: Allow to append footers to plain messages
• [Feature] Rspamadm: Allow to rewrite headers in messages
• [Feature] Selectors: Add ipmask processor
• [Feature] Settings: Allow hostname match
• [Feature] Settings: Allow local when selecting settings
• [Feature] Settings: Allow multiple selectors
• [Feature] Settings: Allow to inverse conditions
• [Feature] Support User-Agent in HTTP requests
• [Feature] Support ed25519 dkim keys generation
• [Feature] Try to filter bad unicode types during normalisation
• [Feature] external_services - oletools (olefy) support
• [Feature] lua_scanners - icap protocol support
• [Feature] lua_scanners - spamassassin spam scanner
• [Fix] Add filter for absurdic URLs
• [Fix] Add some more cases for Received header
• [Fix] Allow to disable/enable composite symbols
• [Fix] Arc: Use a separated list of headers for arc signing
• [Fix] Archive: Final fixes for 7z archives
• [Fix] Clickhouse: Fix database usage
• [Fix] Controller: Make save stats timer persistent
• [Fix] Core: Detect encrypted rarv5 archives
• [Fix] Core: Don’t detect language twice
• [Fix] Core: Fix address rotation bug
• [Fix] Core: Fix content calculations for message parts
• [Fix] Core: Fix emails comments parsing and other issues
• [Fix] Core: Fix etags support
• [Fix] Core: Fix headers folding on the last token
• [Fix] Core: Fix iso-8859-16 encoding
• [Fix] Core: Fix log_urls flag (and encrypted logging)
• [Fix] Core: Fix part length when dealing with boundaries
• [Fix] Core: Fix parts distance calculations
• [Fix] Core: Fix processing of NDNs of certain type
• [Fix] Core: Implement logic to find some bad characters in URLs
• [Fix] Core: treat nodes with ttl properly in lru cache
• [Fix] Fix Content-Type parsing
• [Fix] Fix HTTP headers signing case
• [Fix] Fix control interface
• [Fix] Fix deletion of the duplicate headers
• [Fix] Fix emails filtering in emails module
• [Fix] Fix greylisting log message and logic
• [Fix] Fix issues with storing of the accepted addr in rspamd control
• [Fix] Fix maps object update race condition
• [Fix] Fix memor leaks and whitespace processing
• [Fix] Fix processing of null bytes in headers
• [Fix] Fix rcpt_mime and from_mime in user settings
• [Fix] Fix rfc2047 decoding for CD headers
• [Fix] Fix rfc2231 for Content-Disposition header
• [Fix] Fix setting of the subject pattern in config
• [Fix] Greylist: fix records checking
• [Fix] HTML: Another HTML comments exception fix
• [Fix] HTML: Another entities decoding logic fix
• [Fix] HTML: Fix HTML comments with many dashes
• [Fix] HTML: Fix entities in HTML attributes
• [Fix] HTML: Fix some more SGML tags issues
• [Fix] Ignore whitespaces at the end of value in DKIM records
• [Fix] MID module: Fix DKIM domain matching
• [Fix] Milter_headers: Fix remove_upstream_spam_flag and modernise config
• [Fix] Mime_parser: Fix issue with parsing of the trailing garbadge
• [Fix] Mime_parser: Fix parsing of mime parts without closing boundary
• [Fix] Multimap: Fix operating with userdata
• [Fix] Process orphaned symbols section
• [Fix] Rdns: Fix multiple replies in fake replies
• [Fix] Rework groups scores definitions
• [Fix] Set proper element when reading data from Sentinel
• [Fix] Set rspamd user to initialise supplementary groups on reload
• [Fix] Settings: Fix selectors usage
• [Fix] Sort data received from Sentinel to avoid constant replacing
• [Fix] groups.conf - filename typo
• [Fix] lua_scanner - oletools typos, logging
• [Fix] lua_scanners - actions and symbol_fail
• [Fix] lua_scanners - fix luacheck
• [Fix] lua_scanners - kaspersky - response with fname
• [Fix] lua_scanners - savapi redis prefix
• [Fix] tests - antivirus - fprot symbols
• [Project] Add concept of flexible actions
• [Project] Add heuristical from parser to received parser
• [Project] Add new flags to clickhouse, redis and elastic exporters
• [Project] Attach new received parser
• [Project] Fallback to callbacks from coroutines
• [Project] Implement keep-alive support in lua_http
• [Project] Lua_udp: Implement fully functional client
• [Project] Plug keepalive knobs into http connection handling
• [Project] Rspamadm: Add modify tool
• [Rework] Convert rspamd-server to a shared library
• [Rework] Dcc: Rework DCC plugin
• [Rework] Enable explicit coroutines symbols
• [Rework] Rework telephone urls parsing logic
• [Rework] Rewrite RBL module
• [Rework] Settings: Rework settings check
• [Rework] Slashing: Distinguish lualibdir, pluginsdir and sharedir
• [Rework] Unify task_timeout
• [Rework] Use VEX instructions in assembly, relocate
• [WebUI] Notify user if uploaded data was not learned
• [WebUI] Remove redundant condition