Rspamd 1.9.3 has been released

2019-05-13 00:00:00 +0200

We have released Rspamd 1.9.3 today.

This release contains some new features and many bug fixes. There are no incompatible changes introduced with this release to our best knowledge.

This release includes the following features and important changes.

Hashicorp Vault support

From version 1.9.3, Rspamd can use Hashicorp Vault to store and manage DKIM keys. Vault usage provides secure and flexible storage of the private keys that can scale and use various backends to store sensible data (secrets).

There is a new subcommand for rspamadm utility called vault that is intended to create/remove and securely rotate DKIM private keys using vault.

You can read more about it using the following link: https://rspamd.com/doc/modules/dkim_signing.html#dkim-signing-using-vault

Added least passthrough result

Some modules should set metric result as least possible result. For example, DMARC policy failure should at least mark failed messages as spam but it should not prevent messages from being rejected. From this release, such modules use least policy to set actions allowing to apply a more strict policy if needed.

Tunable memory management

From this version, Rspamd allows to manage memory policies for Lua garbage collection allowing to fit memory/cpu constraints more flexible. When Rspamd is built with jemalloc (e.g. in the default packages provided by the project), it can also print detailed memory statistics on full gc loops.

Here is an example for tuning GC in Rspamd when there are lots of free memory available (around 1Gb per scanner process):

# local.d/options.inc
# http://pgl.yoyo.org/luai/i/2.10+Garbage+Collection
lua_gc_step = 100;
lua_gc_pause = 400;
# number of scanned messages to perform full GC iteration 
full_gc_iters = 10000;

Improved oversigning logic

It is now possible to oversign existing only headers and ignore it if a header is missing. It is done by changing (o) to (x).

The default list of headers signed is changed accordingly:

Header Sign type
From Strictly oversign
Sender Conditionally oversign
Reply-To Strictly oversign
Subject Strictly oversign
Date Conditionally oversign
Message-Id Conditionally oversign
To Strictly oversign
Cc Strictly oversign
Mime-Version Conditionally oversign
Content-Type Conditionally oversign
Content-Transfer-Encoding Conditionally oversign
Resent-To Do not oversign
Resent-Cc Do not oversign
Resent-From Do not oversign
Resent-Sender Do not oversign
Resent-Message-Id Do not oversign
In-Reply-To Conditionally oversign
References Conditionally oversign
List-Id Do not oversign
List-Help Do not oversign
List-Owner Do not oversign
List-Unsubscribe Do not oversign
List-Subscribe Do not oversign
List-Post Do not oversign
Openpgp Conditionally oversign
Autocrypt Conditionally oversign

Important bugs fixes

Here is the list of the most important bugs fixes:

  • HTML: Fix size attribute processing - this issue caused rule MANY_INVISIBLE_PARTS to be improperly triggered on many HTML messages
  • Do not blacklist mail by SPF/DMARC for local/authed users
  • Lots of Clickhouse plugin fixes
  • Fix buffer overflow when printing small floats - this issue caused random crashes in WebUI reported by many users
  • Fix DoS caused by bug in glib - details in https://gitlab.gnome.org/GNOME/glib/issues/1775

Full list of the meaningful changes

  • [Conf] Add IP_SCORE_FREEMAIL composite rule
  • [Feature] Add cryptobox method to generate dkim keypairs
  • [Feature] Add fast hashes to lua cryptobox hash
  • [Feature] Add least passthrough results
  • [Feature] Allow oversign if exists mode
  • [Feature] Clickhouse: Modernise table initial schema
  • [Feature] Implement IUF interface for specific fast hashes
  • [Feature] Lua_util: Allow to obfuscate different fields
  • [Feature] Tune memory management in Rspamd and Lua
  • [Fix] Avoid buffer overflow when printing long lua strings
  • [Fix] Change the default oversigning headers to a more sane list
  • [Fix] Clickhouse: Do not store digest as it is not needed now
  • [Fix] Clickhouse: Fix lots of storage issues
  • [Fix] Clickhouse: Support custom actions
  • [Fix] Deny URLs where hostname is bogus
  • [Fix] Do not blacklist mail by SPF/DMARC for local/authed users
  • [Fix] Fix DoS caused by bug in glib
  • [Fix] Fix UCL parsing of the multiline strings
  • [Fix] Fix buffer overflow when printing small floats
  • [Fix] Fix init code for servers keypairs cache
  • [Fix] Fix issue with urls with no tld (e.g. IP)
  • [Fix] Fix memory in arc signing logic
  • [Fix] Fix memory leak in language detector during reloads
  • [Fix] Fix mixed case content type processing
  • [Fix] Fix processing of the ip urls in file
  • [Fix] Fix use after free
  • [Fix] HTML: Fix size attribute processing
  • [Fix] Hum, it seems that 99ff1c8 was not correct
  • [Fix] Lua_task: Fix task:get_from method
  • [Fix] Preserve fd when mapping file to scan
  • [Fix] Re-use milter_headers settings when doing arc signing
  • [Fix] Set dmarc force action as least action
  • [Fix] Switch to GMT
  • [Fix] allow PKCS7 signatures to be text/plain, too
  • [Project] Add initial version of the vault management tool
  • [Project] Add vault support for DKIM and ARC signing
  • [Project] Implement keys rotation in the vault
  • [Project] Improve dkim keys generation for vault
  • [Project] Improve keys creation in rspamadm vault
  • [Rework] Move lua_worker to a dedicated unit
  • [WebUI] Add URL fragments (#) support
  • [WebUI] Fix AJAX request URL

Rspamd 1.9.2 has been released

2019-04-16 00:00:00 +0200

We have released Rspamd 1.9.2 today.

This release contains some new features and bug fixes. The only potentially slashing changes are the changes in Clickhouse module:

  • Times are now stored in GMT timezone so you can use Clickhouse for analytics that crosses time zones. The potential drawback is the mess with the currently stored data. This should be resolved automatically once new data arrives.

  • Clickhouse schema has been updated to the version 4 with new fields and some minor changes. The existing database should be converted automatically and there are no incompatible changes in columns.

This release includes the following features.

Improvements in Clickhouse plugin

Rspamd now stores more data in Clickhouse:

  • Mime recipients
  • Message IDs
  • Scan time for a message, both normal and virtual
  • SPF checks results
  • Some new calculated columns, such as MIMERcpt, MIMEFrom, SMTPFrom and SMTPRcpt

These columns are intended to improve analytical capabilities of Clickhouse plugin.

OpenDKIM compatible DKIM signing setup

This version now includes a simplified DKIM signing setup option inspired with OpenDKIM.

You can read more about it here: https://rspamd.com/doc/modules/dkim_signing.html#use-of-signing_table

This mode is intended to simplify migration from the existing setups based on OpenDKIM to Rspamd.

Better encrypted archives support

Rspamd can now properly detect encryption in ZIP archives. Mime types plugin now also tries to resolve hex encoding hack used by some spammers to send malware to users (see PR 2582).

Calendar files parser

From the version 1.9.2, Rspamd can extract meaningful data from Calendar files in iCal format (.ics files). These files are sometimes used by spammers so Rspamd can now extract hyperlinks and emails from calendar attachments to improve filtering quality.

New rspamadm dns_tool utility

It is now possible to do some DNS checks with the new tool. For example, it is now possible to verify SPF records as they are observed by Rspamd, including elements extraction, for example a or mx and verification of the IP addresses. Here is how it looks like:

Better bitcoin addresses detection

We have improved bitcoin addresses detection by fixing some issues in the BTC wallet validation code. It now allows to catch Pay-To-Script addresses.

Full list of the meaningful changes

  • [Conf] Allow to load users plugins from plugins.d
  • [Conf] oversign openpgp and autocrypt headers
  • [Feature] Add SPF FFI library for Lua
  • [Feature] Add more verbosity for SPF caching
  • [Feature] Antivirus: Handle encrypted files specially
  • [Feature] Clickhouse: Slashing - add new fields to CH
  • [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table
  • [Feature] Dkim_signing: Allow to use new options as maps
  • [Feature] Import fpconv library
  • [Feature] Lua_maps: Allow static regexp and glob maps
  • [Feature] Parse ical files
  • [Feature] Rspamadm: Add dns_tool utility
  • [Feature] Store SPF records digests
  • [Feature] Use fpconv girsu2 implementation for printing floats
  • [Fix] Clickhouse: Use integer seconds when inserting rows
  • [Fix] Fix floating point printing
  • [Fix] Fix processing of embedded urls
  • [Fix] Lua_clickhouse: Fix CH errors processing
  • [Fix] Make spf digest stable
  • [Fix] Properly detect encrypted files in zip archives
  • [Fix] Slashing: Store times in GMT timezone in ClickHouse
  • [Rules] Add additional conditions to perform BTC checks
  • [Rules] Fix pay-to-hash addresses validation

Rspamd 1.9.1 has been released

2019-04-05 00:00:00 +0200

We have released Rspamd 1.9.1 today.

This release includes one potentially dangerous change: all configuration files are now preprocessed using Jinja templates.

Hence, if you have sequences like {=/=}, or {%/%}, or {#/#} anywhere in the configuration files including even comments then you need to take extra care when moving these configuration to the new version! There are workarounds described above to do that.

Here is the list of the most important changes in this version.

Jinja templates in the configuration

From version 1.9.1, Rspamd supports Jinja2 templates provided by Lupa Lua library. You can read the basic syntax documnentation and the abilities provided by these templating engines using the links above. Rspamd itself uses a specific syntax for variable tags: {= and =} instead of the traditional {{ and }} as these tags could mean, e.g. a table in table in Lua.

Templating might be useful to hide some secrets from config files and places them in environment. Rspamd automatically reads environment variables that start from RSPAMD_ prefix and pushes it to the env variable, e.g. RSPAMD_foo=bar comes to env.foo="bar" in templates.

New template subcommand in Rspamadm

Rspamadm has now template subcommand to apply templates engine to the input file or files:

Options supported:

-n, –no-vars Don’t add Rspamd internal variables
-e , --env Load additional environment var from specific file (name=value)
-l , --lua-env Load additional environment vars from specific file (lua source)
-s , --suffix Store files with the new suffix
-i, –inplace Replace input file(s)

Changes in URLs extraction for HTML parts

Rspamd now tries to extract URLs from plain text of HTML parts. Unfortunately, despite of being contraversal, some Email clients do that as well. One of the notable example is Outlook. Hence, from this release Rspamd also looks for URLs in plain HTML text.

Per user settings for mime_types plugin

Mime types plugin now supports per user settings to allow individual black and white lists of extensions. Here is an example to increase score for exe extensions for some specific user:

test {
  from = "user@example.com";

  apply {
    plugins {
      mime_types = {
        bad_extensions = {
          exe = 100500,
        }
      }
    }
  }
}

Mime types plugin now also supports reverse mapping of content type to extension to allow processing of attachments where an exact file name is not specified.

Better greylisting conditioning

It is now possible to disable or enable greylisting in Rspamd based on the presence of some specific symbols. This feature allows more fine grained greylisting control.

Bitcoin addresses validation

It is not a secret that the wave of spam and scam related to crypto currencies has been flooding the email flows in the recent time. Rspamd has a special rule called LEAKED_PASSWORD_SPAM to block some of the scam types. In this version, Rspamd also checks bitcoin wallets to distinguish them from random long strings to reduce false positives rate significantly. It also allows to build a database of wallets used for scam and spam.

Replies plugin validation

Replies plugin now stores the from/reply-to address when tracking outbound messages and whitelists merely replies that come that address. It helps to avoid replies abusing where spammers were able to catch some legit message ids somewhere in public lists and used them in In-Reply-To headers to dodge spam filtering in Rspamd.

List of major bug fixes

This version includes some important fixes:

  • Add crash safety for HTTP async routines
  • Clickhouse: Fix table schema upload
  • Core: Fix squeezed dependencies handling for virtual symbols
  • Finally fix default parameters parsing in actions section
  • Fix ES sending logic (restore from coroutines mess)
  • Fix finishing script for Clickhouse collection
  • Fix priority for regexp symbols registration
  • Neural: Fix training
  • Rework cached Redis logic to avoid sentinels breaking
  • SURBL: Fix regression in surbl module
  • Fix double signing in the milter

Full list of the meaningful changes

  • [Conf] Add vendor groups for symbols
  • [Feature] Add rspamadm template command
  • [Feature] Allow to add messages from settings
  • [Feature] Allow unconnected DNS servers operations
  • [Feature] Check limits after being set, migrate to uint64
  • [Feature] Greylist: Allow to disable greylisting depending on symbols
  • [Feature] Improve lua binary strings output
  • [Feature] Mime_types: Implement user configurable extension filters
  • [Feature] Mime_types: When no extension defined, detect it by content
  • [Feature] Preprocess config files using jinja templates
  • [Feature] Replies: Filter replies sender to limit whitelisting to direct messages
  • [Feature] Treat all tags with HREF as a potential hyperlinks
  • [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM
  • [Fix] Add crash safety for HTTP async routines
  • [Fix] Another fix for Redis sentinel
  • [Fix] Clickhouse: Fix table schema upload
  • [Fix] Core: Fix squeezed dependencies handling for virtual symbols
  • [Fix] Finally fix default parameters parsing in actions section
  • [Fix] Fix ES sending logic (restore from coroutines mess)
  • [Fix] Fix finishing script for clickhouse collection
  • [Fix] Fix priority for regexp symbols registriation
  • [Fix] Fix various issues found by PVS Studio
  • [Fix] Initialize lua debugging earlier
  • [Fix] Neural: Fix training
  • [Fix] Rework cached Redis logic to avoid sentinels breaking
  • [Fix] SURBL: Fix regression in surbl module
  • [Fix] Fix double signing in the milter
  • [Project] Add support of HTTP proxy in requests
  • [Rework] Change lua global variables registration
  • [Rework] Rework HTML content urls extraction
  • [Rework] Start rework of aliasing in Rspamd
  • [WebUI] Combine Scan and Learning into one tab
  • [WebUI] Fix symbol score input type
  • [WebUI] Show grayed out pie
  • [WebUI] Update Throughput summary values dynamically