Rspamd 1.9.1 has been released

We have released Rspamd 1.9.1 today.

This release includes one potentially dangerous change: all configuration files are now preprocessed using Jinja templates.

Hence, if you have sequences like {=/=}, or {%/%}, or {#/#} anywhere in the configuration files including even comments then you need to take extra care when moving these configuration to the new version! There are workarounds described above to do that.

Here is the list of the most important changes in this version.

Jinja templates in the configuration

From version 1.9.1, Rspamd supports Jinja2 templates provided by Lupa Lua library. You can read the basic syntax documnentation and the abilities provided by these templating engines using the links above. Rspamd itself uses a specific syntax for variable tags: {= and =} instead of the traditional {{ and }} as these tags could mean, e.g. a table in table in Lua.

Templating might be useful to hide some secrets from config files and places them in environment. Rspamd automatically reads environment variables that start from RSPAMD_ prefix and pushes it to the env variable, e.g. RSPAMD_foo=bar comes to env.foo="bar" in templates.

New template subcommand in Rspamadm

Rspamadm has now template subcommand to apply templates engine to the input file or files:

Options supported:

-n, –no-vars	Don’t add Rspamd internal variables
-e , --env	Load additional environment var from specific file (name=value)
-l , --lua-env	Load additional environment vars from specific file (lua source)
-s , --suffix	Store files with the new suffix
-i, –inplace	Replace input file(s)

Changes in URLs extraction for HTML parts

Rspamd now tries to extract URLs from plain text of HTML parts. Unfortunately, despite of being contraversal, some Email clients do that as well. One of the notable example is Outlook. Hence, from this release Rspamd also looks for URLs in plain HTML text.

Per user settings for `mime_types` plugin

Mime types plugin now supports per user settings to allow individual black and white lists of extensions. Here is an example to increase score for exe extensions for some specific user:

test {
  from = "user@example.com";

  apply {
    plugins {
      mime_types = {
        bad_extensions = {
          exe = 100500,
        }
      }
    }
  }
}

Mime types plugin now also supports reverse mapping of content type to extension to allow processing of attachments where an exact file name is not specified.

Better greylisting conditioning

It is now possible to disable or enable greylisting in Rspamd based on the presence of some specific symbols. This feature allows more fine grained greylisting control.

Bitcoin addresses validation

It is not a secret that the wave of spam and scam related to crypto currencies has been flooding the email flows in the recent time. Rspamd has a special rule called LEAKED_PASSWORD_SPAM to block some of the scam types. In this version, Rspamd also checks bitcoin wallets to distinguish them from random long strings to reduce false positives rate significantly. It also allows to build a database of wallets used for scam and spam.

Replies plugin validation

Replies plugin now stores the from/reply-to address when tracking outbound messages and whitelists merely replies that come that address. It helps to avoid replies abusing where spammers were able to catch some legit message ids somewhere in public lists and used them in In-Reply-To headers to dodge spam filtering in Rspamd.

List of major bug fixes

This version includes some important fixes:

Add crash safety for HTTP async routines
Clickhouse: Fix table schema upload
Core: Fix squeezed dependencies handling for virtual symbols
Finally fix default parameters parsing in actions section
Fix ES sending logic (restore from coroutines mess)
Fix finishing script for Clickhouse collection
Fix priority for regexp symbols registration
Neural: Fix training
Rework cached Redis logic to avoid sentinels breaking
SURBL: Fix regression in surbl module
Fix double signing in the milter

Full list of the meaningful changes

[Conf] Add vendor groups for symbols
[Feature] Add rspamadm template command
[Feature] Allow to add messages from settings
[Feature] Allow unconnected DNS servers operations
[Feature] Check limits after being set, migrate to uint64
[Feature] Greylist: Allow to disable greylisting depending on symbols
[Feature] Improve lua binary strings output
[Feature] Mime_types: Implement user configurable extension filters
[Feature] Mime_types: When no extension defined, detect it by content
[Feature] Preprocess config files using jinja templates
[Feature] Replies: Filter replies sender to limit whitelisting to direct messages
[Feature] Treat all tags with HREF as a potential hyperlinks
[Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM
[Fix] Add crash safety for HTTP async routines
[Fix] Another fix for Redis sentinel
[Fix] Clickhouse: Fix table schema upload
[Fix] Core: Fix squeezed dependencies handling for virtual symbols
[Fix] Finally fix default parameters parsing in actions section
[Fix] Fix ES sending logic (restore from coroutines mess)
[Fix] Fix finishing script for clickhouse collection
[Fix] Fix priority for regexp symbols registriation
[Fix] Fix various issues found by PVS Studio
[Fix] Initialize lua debugging earlier
[Fix] Neural: Fix training
[Fix] Rework cached Redis logic to avoid sentinels breaking
[Fix] SURBL: Fix regression in surbl module
[Fix] Fix double signing in the milter
[Project] Add support of HTTP proxy in requests
[Rework] Change lua global variables registration
[Rework] Rework HTML content urls extraction
[Rework] Start rework of aliasing in Rspamd
[WebUI] Combine Scan and Learning into one tab
[WebUI] Fix symbol score input type
[WebUI] Show grayed out pie
[WebUI] Update Throughput summary values dynamically