We have released Rspamd 1.9.0 today.

There are various important features in this release. The vast majority of those should not have any impact on the existing systems. However, you are recommended to read the Upgrade Notes.

This release contains lots of improvements, reworks and bugs being fixed. Here is a list of the most important changes in this release:

External services

Rspamd is now shipped with the external services module contributed by Carsten Rosenberg. This module provides a generic integration with the following services:

• Generic ICAP protocol:
  • ClamAV (using c-icap server and squidclamav)
  • Sophos (via SAVDI)
  • Symantec Protection Engine for Cloud Services
  • Kaspersky Web Traffic Security 6.0
• OleTools
• DCC
• VadeSecure

This plugin is a part of a more generic lua_scanners framework that allows more flexible integration with different Antivirus and AntiSpam OEM services.

New mime modify tool in Rspamadm

Rspamadm mime subcommand now allows to modify messages. This tool allows to add or remove headers in a message, add footers in HTML and text parts and do some subject rewriting. For example, to add a footer and rewrite subject in a message, one can use the following command: rspamadm mime modify --text-footer=footer.txt --html-footer=footer.html --rewrite-header="Subject=TEST: %s".

This tool has full MIME support (including multipart messages), supports various messages encoding and convert those to UTF8, allows to modify both plain old messages and multipart messages with attachments. It also properly detects and excludes GPG signed/encrypted messages.

Offline DKIM signing tool in Rspamadm

Another tool that has been added to rspamadm is sign subcommand. This command allows to perform DKIM signing using a specific private key for some message or messages. It can either return isolated header or modify the message itself.

In conjunction with the previous tooling, it could be used to modify and sign messages produced by mailing lists or some local forwarding scripts.

Please bear in mind that this tool is available when using LuaJIT only as it requires FFI interfaces.

HTTP Keep-Alive support

From this version, Rspamd supports keep-alive in its HTTP Lua client. This could be used to implement high frequent requests to some external services and to reduce load by keeping the pool of HTTP connections instead of opening a connection per each request.

New Lua UDP client

Rspamd now support sending of generic UDP requests as well as TCP ones. There are various modes supported, both one-way and two-ways with optional retransmits and timeouts. This module comes with the documentation available.

Better unicode normalisation

This version comes with further improvements towards unicode normalisation and detecting anomalies. Words now are sanitized from any combining characters after translating to NFKC form.

Configuration graph utility

You can now visualise your configuration by building the include graph. There is a new tool called rspamadm configgraph that takes configuration and convert it to the graphviz DOT graph.

Flexible actions support

Rspamd now support any actions definitions in addition to normal ones. You can set custom actions with thresholds or without (e.g. to set action equal to phishing or to social). All actions should be explicitly defined in the configuration.

New Received headers parser

We have migrated from a strict RFC compatible state machine to a custom parser for the Received headers. This change allows to extract more data from non-conforming Received headers used by some MTA (Exim is one of the notable examples).

Telephone URLs support

Rspamd now parses and processes telephone URLs. That allows to build blacklist for spam/scam/phishing phones as well as plain URLs.

Support for ED25519 signatures

Rspamd now supports dual signing and ed25519 DKIM signatures. New ed25519 keys could be generated using rspamadm dkim_keygen tool:

rspamadm dkim_keygen -t ed25519
CuHc4MOZYXEVH0M4WFQHL5UC2NbVJO8aq2CjGNznxm36mJPlu9GVMfq0lQI1dkeoHByqfsJgMgnCX0vFeMkjoA==
selector._domainkey IN TXT ( "v=DKIM1; k=ed25519; "
	"p=+piT5bvRlTH6tJUCNXZHqBwcqn7CYDIJwl9LxXjJI6A=" ) ;

Ed25519 keys are much shorter than RSA providing RSA2048 security margin in just 32 bytes for public keys. Unfortunately, these signatures are not widely supported so dual signing is still required. We believe that support of the modern algorithms in Rspamd would those algorithms to spread.

Custom functions in Regexp module

Regexp module is now extended with custom Lua functions support. This feature allows to mix fast regexp rules and custom logic of Lua rules without explicit composite rules.

Added support of gzip archives

Rspamd now support reading filenames from GZip archives that are surprisingly often used by some spam senders. With this feature, Rspamd can filter some tricky scam emails that are targeting to install backdoors or malware on users’ machines (e.g. cryptolockers).

Additional detection of types for attachments

To filter malware and bad attachments that are somehow hidden by malicious Content-Type header, Rspamd also performs libmagic scan on attachments to detect the real type by its content. This is useful to detect and filter some tricky malware that utilizes bugs in popular email clients.

Lots of major fixes

This version includes many major fixes that required massive rework to improve stability and performance:

• Race conditions in the maps reading code
• Support RFC2231 encoding in headers
• Better zero characters handling
• Better HTML parsing and handling of the URLs
• Coroutines are now explicitly separated from the async code to prevent tricky race conditions that caused crashes on certain load
• Allow to disable/enable composite symbols
• Lots of fixes in 7z processor
• Detect encrypted rarv5 archives
• Fix ETags support
• Fix processing of NDNs of certain type
• Improved Content-Type parsing
• Fix deletion of the duplicate headers
• Fix parsing of mime parts without closing boundary

We recommend to update Rspamd to this version to apply all these features and fixes.

Full list of the meaningful changes

• [Conf] Add missing includes
• [Conf] Move to options
• [Conf] Rbl: DWL is actually special whitelist
• [Conf] Relax some uribl rules
• [Conf] Remove abuse.ch
• [CritFix] Html: Entities are not valid within tag params values
• [Feature] Add rspamadm mime sign tool
• [Feature] Add configgraph utility
• [Feature] Add dedicated ZW spaces detection for URLs
• [Feature] Add flag to url object when visible part is url_like
• [Feature] Add method task:lookup_words
• [Feature] Add pyzor support (by crosenberg)
• [Feature] Allow to add upstream watchers to Lua API
• [Feature] Allow to set rewrite subject pattern from settings
• [Feature] Better escaping of unicode
• [Feature] Clickhouse: Allow to store subject in Clickhouse
• [Feature] Core: Add QP encoding utility
• [Feature] Core: Add libmagic detection for all parts
• [Feature] Core: Add support for gzip archives
• [Feature] Core: Allow to construct scan tasks from raw data
• [Feature] Core: Detect charset in archived files
• [Feature] Core: Ignore and mark invisible spaces
• [Feature] Core: Normalise zero-width spaces in urls
• [Feature] Core: Process data urls for images
• [Feature] Core: Relax quoted-printable encoding
• [Feature] Core: Support RFC2231 encoding in headers
• [Feature] Core: Support telephone URLs
• [Feature] Core: allow to emit soft reject on task timeout
• [Feature] DCC: Add bulkness and reputation checks to dcc
• [Feature] Elastic: Modernize plugin
• [Feature] Export visible part of url to lua
• [Feature] Fuzzy_storage: add preliminary support of rate limits
• [Feature] HTML: Specially treat data urls in HTML
• [Feature] Implement event watchers for upstreams
• [Feature] Implement includes tracing in Lua
• [Feature] Improve dkim part in configwizard
• [Feature] Lua_scanners: Add VadeSecure engine support
• [Feature] Lua_task: Add flexible method to get specific urls
• [Feature] Mime_types: Add MIME_BAD_UNICODE rule
• [Feature] Mime_types: Use detected content type as well
• [Feature] Plugins: Add preliminary version of the external services plugin
• [Feature] Query sentinel on master errors
• [Feature] Regexp: Allow local lua functions in Rspamd regexp module
• [Feature] Rspamadm: Allow to append footers to plain messages
• [Feature] Rspamadm: Allow to rewrite headers in messages
• [Feature] Selectors: Add ipmask processor
• [Feature] Settings: Allow hostname match
• [Feature] Settings: Allow local when selecting settings
• [Feature] Settings: Allow multiple selectors
• [Feature] Settings: Allow to inverse conditions
• [Feature] Support User-Agent in HTTP requests
• [Feature] Support ed25519 dkim keys generation
• [Feature] Try to filter bad unicode types during normalisation
• [Feature] external_services - oletools (olefy) support
• [Feature] lua_scanners - icap protocol support
• [Feature] lua_scanners - spamassassin spam scanner
• [Fix] Add filter for absurdic URLs
• [Fix] Add some more cases for Received header
• [Fix] Allow to disable/enable composite symbols
• [Fix] Arc: Use a separated list of headers for arc signing
• [Fix] Archive: Final fixes for 7z archives
• [Fix] Clickhouse: Fix database usage
• [Fix] Controller: Make save stats timer persistent
• [Fix] Core: Detect encrypted rarv5 archives
• [Fix] Core: Don’t detect language twice
• [Fix] Core: Fix address rotation bug
• [Fix] Core: Fix content calculations for message parts
• [Fix] Core: Fix emails comments parsing and other issues
• [Fix] Core: Fix etags support
• [Fix] Core: Fix headers folding on the last token
• [Fix] Core: Fix iso-8859-16 encoding
• [Fix] Core: Fix log_urls flag (and encrypted logging)
• [Fix] Core: Fix part length when dealing with boundaries
• [Fix] Core: Fix parts distance calculations
• [Fix] Core: Fix processing of NDNs of certain type
• [Fix] Core: Implement logic to find some bad characters in URLs
• [Fix] Core: treat nodes with ttl properly in lru cache
• [Fix] Fix Content-Type parsing
• [Fix] Fix HTTP headers signing case
• [Fix] Fix control interface
• [Fix] Fix deletion of the duplicate headers
• [Fix] Fix emails filtering in emails module
• [Fix] Fix greylisting log message and logic
• [Fix] Fix issues with storing of the accepted addr in rspamd control
• [Fix] Fix maps object update race condition
• [Fix] Fix memor leaks and whitespace processing
• [Fix] Fix processing of null bytes in headers
• [Fix] Fix rcpt_mime and from_mime in user settings
• [Fix] Fix rfc2047 decoding for CD headers
• [Fix] Fix rfc2231 for Content-Disposition header
• [Fix] Fix setting of the subject pattern in config
• [Fix] Greylist: fix records checking
• [Fix] HTML: Another HTML comments exception fix
• [Fix] HTML: Another entities decoding logic fix
• [Fix] HTML: Fix HTML comments with many dashes
• [Fix] HTML: Fix entities in HTML attributes
• [Fix] HTML: Fix some more SGML tags issues
• [Fix] Ignore whitespaces at the end of value in DKIM records
• [Fix] MID module: Fix DKIM domain matching
• [Fix] Milter_headers: Fix remove_upstream_spam_flag and modernise config
• [Fix] Mime_parser: Fix issue with parsing of the trailing garbadge
• [Fix] Mime_parser: Fix parsing of mime parts without closing boundary
• [Fix] Multimap: Fix operating with userdata
• [Fix] Process orphaned symbols section
• [Fix] Rdns: Fix multiple replies in fake replies
• [Fix] Rework groups scores definitions
• [Fix] Set proper element when reading data from Sentinel
• [Fix] Set rspamd user to initialise supplementary groups on reload
• [Fix] Settings: Fix selectors usage
• [Fix] Sort data received from Sentinel to avoid constant replacing
• [Fix] groups.conf - filename typo
• [Fix] lua_scanner - oletools typos, logging
• [Fix] lua_scanners - actions and symbol_fail
• [Fix] lua_scanners - fix luacheck
• [Fix] lua_scanners - kaspersky - response with fname
• [Fix] lua_scanners - savapi redis prefix
• [Fix] tests - antivirus - fprot symbols
• [Project] Add concept of flexible actions
• [Project] Add heuristical from parser to received parser
• [Project] Add new flags to clickhouse, redis and elastic exporters
• [Project] Attach new received parser
• [Project] Fallback to callbacks from coroutines
• [Project] Implement keep-alive support in lua_http
• [Project] Lua_udp: Implement fully functional client
• [Project] Plug keepalive knobs into http connection handling
• [Project] Rspamadm: Add modify tool
• [Rework] Convert rspamd-server to a shared library
• [Rework] Dcc: Rework DCC plugin
• [Rework] Enable explicit coroutines symbols
• [Rework] Rework telephone urls parsing logic
• [Rework] Rewrite RBL module
• [Rework] Settings: Rework settings check
• [Rework] Slashing: Distinguish lualibdir, pluginsdir and sharedir
• [Rework] Unify task_timeout
• [Rework] Use VEX instructions in assembly, relocate
• [WebUI] Notify user if uploaded data was not learned
• [WebUI] Remove redundant condition

We have released Rspamd 1.8.3 today.

There are no incompatible changes introduced with this version to our best knowledge. However, due to Bayes changes, you might see slightly different Bayes outcomes (not likely).

The most important features and fixes

• Large Unicode handling rework

  Rspamd now provides better access to normalized/unnormalized unicode. There are various of changes in the internal routines that work with Unicode, including statistics, language detector and regular expressions. The outcome of this project is the ability to fight against certain techniques applied by spammers to obfuscate messages.

• Critical fixes related to Unicode

  We have added some fail-safety checks against certain types of messages that could cause Rspamd crashes due to invalid unicode in headers. This has been also backported to 1.8.2.

• Redis sentinel support

  Rspamd 1.8.3 includes limited support for Redis Sentinel to allow high-availability for access to the non-volatile data. You can get more details on the Redis documentation page

• Kaspersky antivirus support

  Rspamd now supports Kaspersky antivirus using its clamav compatible interface via antivirus module

• Improvements of the Authentication-Results header

  Now, Rspamd produces more readable Authentication-Results header used by milter_headers and arc modules. It now can deal with multiple DKIM signatures, provides selector information and more extended (and traditional) data about SPF checks.

• Better statistical tokens

  We continue to improve Bayes filtering quality by adding more tokens produced from URLs found in the text. In some rare cases this might cause slightly changed results with pre-trained statistical corpus.

• Better and RFC conformant support of pct in DMARC module

Full list of the meaningful changes

• [CritFix] Make flags mutually exclusive for mime parts
• [CritFix] Strictly deny unencoded bad utf8 sequences in headers
• [Feature] Add Kaspersky antivirus support
• [Feature] Add method to get dkim results
• [Feature] Add more words regexp classes
• [Feature] Allow to choose words format in rspamadm mime
• [Feature] Allow to get all types of words from Lua
• [Feature] Allow to get task flags in C expressions
• [Feature] Allow to require encryption when accepting connections
• [Feature] Ignore bogus whitespaces in the words
• [Feature] Implement more strict configuration tests
• [Feature] Improve SPF results in Authentication-Results
• [Feature] Support ClickHouse database
• [Fix] Add fail-safety for utf8 regexps
• [Fix] Do not trigger BROKEN_CONTENT_TYPE on innocent text parts
• [Fix] Emit error if connection has been terminated with no stop pattern
• [Fix] Fix boundaries checks in embedded messages
• [Fix] Fix double free
• [Fix] Perform policy downgrade on sample out, add tests
• [Fix] Properly escape utf8 regexps in hyperscan mode
• [Fix] Selectors - attachments args condition
• [Fix] Some fixes for raw parts
• [Fix] Treat learning errors as non-fatal
• [Fix] Use tld when looking for DKIM domains
• [Project] Words unicode structure rework
• [Project] Add preliminary Redis Sentinel support
• [Project] Improve Authentication-Results header
• [Project] Rework DKIM checks results
• [Project] Use more generalised API to produce meta words

We have released Rspamd 1.8.2 today.

There are no incompatible changes introduced with this version to our best knowledge. However, due to Bayes changes, you might see slightly different Bayes outcomes. rspamd_update module is now disabled by default as we are working on a better and safer solution that will be available in the next Rspamd version.

The most important features and fixes

• Critical fixes in SURBL module

  SURBL module could insert absurd scores and skip group symbol limit in some cases.

• Critical fixes in Bayes statistics

  Bayes algorithm has been changed significantly in this release:

  • If chi-squared method fails due to exp underflow, Rspamd will switch to a naive method
  • Tokens weighting has been reworked to match chi-squared model
  • OSB weights have been adjusted to discriminate more far tokens and increase weight for closer tokens
  • Reworked meta-tokens algorithm, it is now moved to Lua
  • Skip stop words in statistics

• Fuzzy check rework

  In this version, we have moved selection algorithm from C to Lua with more flexibility about parts and data selection.

• Critical fixes in Ratelimit module

  We have found and fixed various issues in Ratelimit plugin and restored both traditional and modern syntax of ratelimits. Presumably, all version prior this one are broken when dealing with ratelimits.

• Major rework of all asynchronous rules and events

  In Rspamd 1.8.2, we have introduced new model of asynchronous rules processing with the following features:

  • More straightforward in terms of the control flow
  • It is possible to get better debugging info in case of issues (including the concrete line of code)
  • It is possible now to see the real execution time of rules and show slow rules including network waiting
  • New model is less complex than the previous one

• Critical issue in ARC module

  We have fixed the issue in ARC module that could lead to around 11% of wrong signatures being produced due to an incorrect \0 handling in raw RSA signature.

• Add support for encrypted cookies in Message-ID

  It is now possible to generate and check message-ids that require no storage and Rspamd even don’t need to observe outbound messages. However, it will be able to detect replies to our messages and get some more information if needed:

  We generate message id using nonce||aesencrypt(nonce, k, known_cookie) then we can just take a look into In-Reply-To header and apply read_nonce() + aesdecrypt(nonce, k) to compare result with known_cookie.

  Nonce length will be 16 bytes (32 hex encoded) and another 16 bytes could be used to encrypt secret cookie + some random thing derived from the message itself.

• Added DWL support in the default configuration
• Add support of words regexps
• Allow to set custom limits for upstream lists
• Detect orphaned parts and attach them to message
• Support query arguments in controller’s custom commands
• Write DKIM selector in dkim allow/reject symbols

Full list of the meaningful changes

• [Conf] Add DWL support in the default configuration
• [Conf] Disable rspamd_update by default (again)
• [Conf] Fix configuration sample for ratelimit
• [CritFix] Disable broken url tags by default
• [CritFix] Fix \0 processing when doing RSA sign
• [CritFix] Fix adding symbols to their primary groups
• [Feature] Add rspamadm cookie utility
• [Feature] Add specialised functions for generating encrypted cookies
• [Feature] Add support of cookies in replies module
• [Feature] Add support of words regexps
• [Feature] Allow to add 3rd party clang plugins
• [Feature] Allow to create lua regexps from glob or plain patterns
• [Feature] Allow to set custom limits for upstream lists
• [Feature] Detect orphaned parts and attach them to message
• [Feature] Filter tokens in bayes
• [Feature] Fold b= value when doing arc sealing
• [Feature] Ignore cookies in the future and too old in the past
• [Feature] Skip stop words in statistics
• [Feature] Store stop words and allow to query them
• [Feature] Support query arguments in controller’s custom commands
• [Feature] Tune upstream limits in Rspamd proxy
• [Feature] Use different callback symbols for different uribls
• [Feature] Write DKIM selector in dkim allow/reject symbols
• [Fix] Add obs_fws state support to eoh state machine
• [Fix] Add sanity check when applying mime boundaries heuristic
• [Fix] Antivirus - virus names with 0 were recognized as tables
• [Fix] Disable headernames in bayes temporarily
• [Fix] Do not allow syntax errors in include files…
• [Fix] Do not allow to merge an object with an array (or vice versa)
• [Fix] Don’t perform forged recipients check for missing recipients
• [Fix] Fix DKIM based RBLs
• [Fix] Fix actrie implementation (sync from upstream), fixed OOB read
• [Fix] Fix explicit methods call in selectors
• [Fix] Fix extraction of additional parts
• [Fix] Fix finalization for internal plugins
• [Fix] Fix override_defaults function
• [Fix] Fix squeezed symbols when using settings
• [Fix] Fix urls insertion in Clickhouse module
• [Fix] Furhter fixes to ratelimits logic
• [Fix] Ignore signatures when looking for boundaries
• [Fix] Properly set learned count
• [Fix] Really fix ratelimits configuration and work
• [Fix] Remove ambigious format flag from printf
• [Fix] Restore URLs exporting in ClickHouse plugin
• [Fix] Rework bayes calculations…
• [Fix] Switch from chi-square to naive for large Fisher value
• [Fix] Treat normal password as enable password if there is no enable password
• [Fix] Use proper syntax for making DNS requests
• [Fix] Various fixes in embedded plugins
• [Project] Change fuzzy check selection logic to lua_fuzzy library
• [Project] Rework async events and symbols
• [Project] Move all metatokens in Bayes to lua_stat from C
• [WebUI] Add history rows per page control

We have released Rspamd 1.8.1 today.

There are couple of the incompatibilities introduced, hence, please check the migration notes. These incompatibilities could affect almost any Rspamd installation with custom configuration, so please read the upgrade notes carefully.

The most important features and fixes

• Critical fixes in DMARC module

  DMARC module could insert temporary failure symbol instead of strong rejection in certain cases. This has been fixed in 1.8.1. There are many other bugs fixed in this module since the whole check logic has been rewritten in accordance to RFC.

• Fuzzy check

  Due to bug introduced in 1.8.0, there algorithm used to deterministically skip words in large text parts was not deterministic. It means that the exact words pipelines produced by different Rspamd instances might be different. It could affect if your words_limit was reached (default: words_decay = 200 words). Hence, for large text parts it was expected to have misses in fuzzy and in Bayes classification. Whilst bayes missing should not be significant, the fuzzy misses might be very severe and they might break fuzzy detection for large text parts.

  In 1.8.1, we have fixed this issue and, since we have already broken the compatibility with 1.7.9, we have decided to increase words_decay to 600. Please ensure that you don’t override this parameter anywhere (e.g. in local.d/options.inc, override.d/options.inc or any other override or local file) or your compatibility with Rspamd fuzzy storage would be lost for messages with more than words_decay threshold words.

• Various language detection issues have been fixed
• Experimental clustering plugin
• Important fixes for the dynamic ratelimits
• Fix options insertion
• Plug memory leak in redis pool
• Add check_violation feature to DKIM/ARC signing to avoid signing messages when there is an existing invalid DKIM signature
• Add only unique elements to Clickhouse url arrays
• Allow g+: and g-: composite atoms to include symbols with positive score and negative score
• Allow dkim domains check in surbl module
• Allow maps with HTTP auth using standard URL syntax
• Allow to disable actions by users settings by setting them to null
• Extend whitelisting options:
  • Allow per element whitelist/blacklist only behaviour
  • Introduce three types of lists:
    • Blacklists (bl)
    • Whitelists (wl)
    • Bidirectional lists (both)
• Use heuristical verdict instead of the plain action in plugins to detect if a message is clearly spam, junk, ham or uncertain when there is not enough confidence despite of the action being set
• Various Web Interface improvements and fixes driven by Alexander Moisseev

Full list of the meaningful changes

• [CritFix] Fix options insertion
• [CritFix] Fix words decay one more time (affects long messages)
• [CritFix] Increase default words_decay
• [CritFix] Plug memory leak in redis pool
• [Feature] Add check_violation feature to DKIM/ARC signing
• [Feature] Add only unique elements to Clickhouse url arrays
• [Feature] Allow g+: and g-: composite atoms
• [Feature] Allow dkim domains check in surbl
• [Feature] Allow maps with HTTP auth
• [Feature] Allow to disable actions by users settings
• [Feature] Extend whitelisting options
• [Feature] Store url object in images
• [Feature] Use verdict instead of the plain action in plugins
• [Fix] Allow to call fstring append with NULL string
• [Fix] DCC - luacheck
• [Fix] Do not load torch on each rspamadm invocation
• [Fix] Fix boundaries detection and rework stop words algorithm
• [Fix] Fix dependencies for DNS_SIGNED symbol
• [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit
• [Fix] Fix groups mess
• [Fix] Fix groups mess
• [Fix] Fix parsing address with comments
• [Fix] Fix resolving in DMARC reports
• [Fix] Fix various issues with parsing of the received headers
• [Fix] Fix watchers issue in lua_tcp when doing no resolving
• [Fix] Plug memory leak in language detector (affects reloads)
• [Fix] Remove one letter stop words
• [Fix] Slashing: backport chunk logic from libucl
• [Fix] Stop libevent from using cached time in rspamadm
• [Fix] Try to fix watchers chaining
• [Fix] Various fixes in redis sync interface
• [Fix] ip_score - respect check_authed and check_local settings from config
• [Project] Rework passthrough actions
• [Project] Clustering module
• [Rework] Always create result for a task
• [Rework] Completely rewrite DMARC checks logic
• [Rework] Rework and fix whitelist plugin
• [WebUI] Add symbols sorting buttons
• [WebUI] Change symbols order without updating history
• [WebUI] Colorize symbols
• [WebUI] Do not display password form when secure_ip is set
• [WebUI] Fix symbol description tooltips display
• [WebUI] History: add sorting by symbol score value

We have released Rspamd 1.8.0 today.

There are couple of the incompatibilities introduced, hence, please check the migration notes, especially if you use clickhouse module or users settings.

The most important features and fixes

• New selectors framework

  This framework allows to combine and process different data extracted from messages and use that in different plugins, such as multimap, reputation or ratelimits. It is also possible to use data extracted in Rspamd regular expressions.

• Coroutines API support in Lua

  Now you can write code in a usual imperative manner but you still will not block any other tasks. Each potentially blocking operation creates a yielding-point. In turn, this means the code is suspended until the operation is done (just like blocking) and resumes only when there is some result. Meanwhile, other tasks are processed as usual.

• Clickhouse optimization

  Rspamd now uses a flat table to optimize ClickHouse SQL requests. In fact, joins are not recommended by the ClickHouse developers as multiple joins have proven to be slow. Hence, Rspamd has moved all data to a single table. Schema migration is done automatically, however, please read the migration notes in case of any doubts. Old data is not migrated nor deleted automatically.

  There is now optional data retention support in the ClickHouse module. You can set retention policies for the data stored in Clickhouse to conform different regulations (e.g. GDPR).

• Unicode processing improvements

  Rspamd now normalizes all unicode data using NFKC schema prior to processing. This helps to prevent “glyph” attacks used by some spammers nowadays. Unicode conversion has also been improved to continue on bad symbols instead of giving up and working with raw data.

• Language detection improvements

  We have reworked the language detector to use stop-words and rely on unicode glyphs more extensively. As the result of this work, the speed of language detection has been increased significantly (by 10 times in some cases). The preciseness of the detection has also been improved.

• Fixed various bugs in sesssions handling

  We have located and fixed various hidden issues caused by async rules chaining. It might cause inconsistencies in the dependencies processing, crashes in rare cases and other “bad things”.

• Various Web Interface improvements and fixes

  There are multiple improvements and fixes in the Web Interface. In particular, the issues with cluster support and aggregation have been addressed.

• New mailing lists for the project

  • Rspamd-Users: general purpose mailing list that will replace this group
  • Rspamd-Announce: read only list with low traffic that is intended for project announcements only, for example, new versions, or vulnerabilities disclosures

  Unfortunately, there is no automatic conversion from the Google groups to the new mailing lists, hence, you need to subscribe to those lists manually. We are sorry about the potential inconveniences caused by this transition.

  To subscribe to a list, click the list name at https://lists.rspamd.com/mailman/listinfo . The page that is displayed should contain all of the necessary subscription instructions for that list. You can always find all possible support channels here.

Full list of the meaningful changes

• [Feature] Add arguments schemas to processors and extractors
• [Feature] Add functional selectors library
• [Feature] Add generic selector to reputation module
• [Feature] Add more ratelimits: by digest, by attachments data, by filenames
• [Feature] Add preliminary stop words detection support
• [Feature] Add pure Lua debugm function
• [Feature] Add schema validation for Redis settings
• [Feature] Add selectors combine function
• [Feature] Add some recursion protection to lua logger
• [Feature] Add support for Lua API tracing
• [Feature] Allow to apply schema to arguments
• [Feature] Allow to get dkim signing data directly from HTTP headers
• [Feature] Allow to reuse existing authentication results
• [Feature] Cache selectors results in re runtime
• [Feature] Implement new text tokenizer based on libicu
• [Feature] Integrate selectors framework to multimap
• [Feature] Relax FORGED_RECIPIENTS
• [Feature] Support (almost) all html entities
• [Feature] Support adding and deletion of recipients in the milter block
• [Feature] Support gathering HTTP body from fragments in lua_http
• [Feature] Support multi flag in regexp and glob maps
• [Feature] Support selectors in ratelimit module
• [Feature] Support selectors in settings
• [Feature] Use khash in HTML parser
• [Feature] Use pure Lua debugm function
• [Fix] Add fail-safety for destroying sessions
• [Fix] Allow to add result-less fake DNS records
• [Fix] Another try to fix race conditions on config unload
• [Fix] Call Lua callback on DNS timeouts
• [Fix] Deprecate task:inc_dns_req as it is redundant
• [Fix] Do not allow events deletions on cleanup
• [Fix] Do not try to process skipped messages
• [Fix] Fix HTTP requests with no body
• [Fix] Fix another cleanup race condition
• [Fix] Fix bug in processing of pcre regexps
• [Fix] Fix byte array allocation in the pool
• [Fix] Fix crashes on task cleanup
• [Fix] Fix dynamic buckets in ratelimits
• [Fix] Fix endless loop when waiting for Rspamd to stop
• [Fix] Fix lua_util.str_split in case of delimiters set
• [Fix] Fix more issues with watching of async events
• [Fix] Fix stop words detection and loading logic
• [Fix] Fix various corner cases for language detection
• [Fix] Fix watchers in lua_tcp
• [Fix] Fix words decay algorithm
• [Fix] Implement watchers replacement to handle nested calls
• [Fix] Save faked code into fake dns record
• [Fix] Show the proper frame when using lua_util.debugm
• [Fix] Use fake dns records in tests
• [Fix] Use unicode replacements for HTML entities
• [Fix] fixed “cannot find dependency on symbol 1” issue when using replaced symbols in spamassassin rules
• [Fix] partition_id is not available in old versions of CH
• [Project] Add implicit conversion logic to selectors
• [Project] Add initial support for selectors in regexps
• [Project] Add method concept
• [Project] Further changes in unicode operations
• [Project] Implement Clickhouse migrations
• [Project] Implement implicit conversions to userdata
• [Project] Implement insert method
• [Project] Implement selectors registration for regular expressions
• [Project] Implement selectors support in re_cache
• [Project] Improve language detector: cleanup unused files, categorize
• [Project] Migrate CH data to a fat table
• [Project] Rework selectors logic
• [Project] Start Clickhouse utilities library
• [Project] Start unicode rework
• [Project] coroutine threaded model for API calls: thread pool
• [Rework] Move phishtank to a DNS based service
• [Rework] Rework Clickhouse plugin to use the new API
• [Rework] Rework language detector
• [Rework] Rework utf content processing in text parts
• [WebUI] Add progress bar for AJAX requests
• [WebUI] Avoid errors table reinitialization
• [WebUI] Avoid history table reinitialization
• [WebUI] Avoid throughput summary table reinitialization
• [WebUI] Destroy summary table on disconnect
• [WebUI] Fix “auth” request URL
• [WebUI] Fix disabling and hiding controls on page reload
• [WebUI] Fix maps loading from neighbours
• [WebUI] Fix symbols sorting by score
• [WebUI] Fix tables destroying
• [WebUI] Fix throughput data consolidation
• [WebUI] Fix upload buttons disabling
• [WebUI] Notify user on module loading failure
• [WebUI] Update FooTable 3.1.4 -> 3.1.6