Rspamd

Fast, free and open-source spam filtering system.

Rspamd 1.8.3 has been released

2018-12-03 00:00:00 +0100

We have released Rspamd 1.8.3 today.

There are no incompatible changes introduced with this version to our best knowledge. However, due to Bayes changes, you might see slightly different Bayes outcomes (not likely).

The most important features and fixes

  • Large Unicode handling rework

    Rspamd now provides better access to normalized/unnormalized unicode. There are various of changes in the internal routines that work with Unicode, including statistics, language detector and regular expressions. The outcome of this project is the ability to fight against certain techniques applied by spammers to obfuscate messages.

  • Critical fixes related to Unicode

    We have added some fail-safety checks against certain types of messages that could cause Rspamd crashes due to invalid unicode in headers. This has been also backported to 1.8.2.

  • Redis sentinel support

    Rspamd 1.8.3 includes limited support for Redis Sentinel to allow high-availability for access to the non-volatile data. You can get more details on the Redis documentation page

  • Kaspersky antivirus support

    Rspamd now supports Kaspersky antivirus using its clamav compatible interface via antivirus module

  • Improvements of the Authentication-Results header

    Now, Rspamd produces more readable Authentication-Results header used by milter_headers and arc modules. It now can deal with multiple DKIM signatures, provides selector information and more extended (and traditional) data about SPF checks.

  • Better statistical tokens

    We continue to improve Bayes filtering quality by adding more tokens produced from URLs found in the text. In some rare cases this might cause slightly changed results with pre-trained statistical corpus.

  • Better and RFC conformant support of pct in DMARC module

Full list of the meaningful changes

  • [CritFix] Make flags mutually exclusive for mime parts
  • [CritFix] Strictly deny unencoded bad utf8 sequences in headers
  • [Feature] Add Kaspersky antivirus support
  • [Feature] Add method to get dkim results
  • [Feature] Add more words regexp classes
  • [Feature] Allow to choose words format in rspamadm mime
  • [Feature] Allow to get all types of words from Lua
  • [Feature] Allow to get task flags in C expressions
  • [Feature] Allow to require encryption when accepting connections
  • [Feature] Ignore bogus whitespaces in the words
  • [Feature] Implement more strict configuration tests
  • [Feature] Improve SPF results in Authentication-Results
  • [Feature] Support ClickHouse database
  • [Fix] Add fail-safety for utf8 regexps
  • [Fix] Do not trigger BROKEN_CONTENT_TYPE on innocent text parts
  • [Fix] Emit error if connection has been terminated with no stop pattern
  • [Fix] Fix boundaries checks in embedded messages
  • [Fix] Fix double free
  • [Fix] Perform policy downgrade on sample out, add tests
  • [Fix] Properly escape utf8 regexps in hyperscan mode
  • [Fix] Selectors - attachments args condition
  • [Fix] Some fixes for raw parts
  • [Fix] Treat learning errors as non-fatal
  • [Fix] Use tld when looking for DKIM domains
  • [Project] Words unicode structure rework
  • [Project] Add preliminary Redis Sentinel support
  • [Project] Improve Authentication-Results header
  • [Project] Rework DKIM checks results
  • [Project] Use more generalised API to produce meta words

Rspamd 1.8.2 has been released

2018-11-19 00:00:00 +0100

We have released Rspamd 1.8.2 today.

There are no incompatible changes introduced with this version to our best knowledge. However, due to Bayes changes, you might see slightly different Bayes outcomes. rspamd_update module is now disabled by default as we are working on a better and safer solution that will be available in the next Rspamd version.

The most important features and fixes

  • Critical fixes in SURBL module

    SURBL module could insert absurd scores and skip group symbol limit in some cases.

  • Critical fixes in Bayes statistics

    Bayes algorithm has been changed significantly in this release:

    • If chi-squared method fails due to exp underflow, Rspamd will switch to a naive method
    • Tokens weighting has been reworked to match chi-squared model
    • OSB weights have been adjusted to discriminate more far tokens and increase weight for closer tokens
    • Reworked meta-tokens algorithm, it is now moved to Lua
    • Skip stop words in statistics
  • Fuzzy check rework

    In this version, we have moved selection algorithm from C to Lua with more flexibility about parts and data selection.

  • Critical fixes in Ratelimit module

    We have found and fixed various issues in Ratelimit plugin and restored both traditional and modern syntax of ratelimits. Presumably, all version prior this one are broken when dealing with ratelimits.

  • Major rework of all asynchronous rules and events

    In Rspamd 1.8.2, we have introduced new model of asynchronous rules processing with the following features:

    • More straightforward in terms of the control flow
    • It is possible to get better debugging info in case of issues (including the concrete line of code)
    • It is possible now to see the real execution time of rules and show slow rules including network waiting
    • New model is less complex than the previous one
  • Critical issue in ARC module

    We have fixed the issue in ARC module that could lead to around 11% of wrong signatures being produced due to an incorrect \0 handling in raw RSA signature.

  • Add support for encrypted cookies in Message-ID

    It is now possible to generate and check message-ids that require no storage and Rspamd even don’t need to observe outbound messages. However, it will be able to detect replies to our messages and get some more information if needed:

    We generate message id using nonce||aesencrypt(nonce, k, known_cookie) then we can just take a look into In-Reply-To header and apply read_nonce() + aesdecrypt(nonce, k) to compare result with known_cookie.

    Nonce length will be 16 bytes (32 hex encoded) and another 16 bytes could be used to encrypt secret cookie + some random thing derived from the message itself.

  • Added DWL support in the default configuration
  • Add support of words regexps
  • Allow to set custom limits for upstream lists
  • Detect orphaned parts and attach them to message
  • Support query arguments in controller’s custom commands
  • Write DKIM selector in dkim allow/reject symbols

Full list of the meaningful changes

  • [Conf] Add DWL support in the default configuration
  • [Conf] Disable rspamd_update by default (again)
  • [Conf] Fix configuration sample for ratelimit
  • [CritFix] Disable broken url tags by default
  • [CritFix] Fix \0 processing when doing RSA sign
  • [CritFix] Fix adding symbols to their primary groups
  • [Feature] Add rspamadm cookie utility
  • [Feature] Add specialised functions for generating encrypted cookies
  • [Feature] Add support of cookies in replies module
  • [Feature] Add support of words regexps
  • [Feature] Allow to add 3rd party clang plugins
  • [Feature] Allow to create lua regexps from glob or plain patterns
  • [Feature] Allow to set custom limits for upstream lists
  • [Feature] Detect orphaned parts and attach them to message
  • [Feature] Filter tokens in bayes
  • [Feature] Fold b= value when doing arc sealing
  • [Feature] Ignore cookies in the future and too old in the past
  • [Feature] Skip stop words in statistics
  • [Feature] Store stop words and allow to query them
  • [Feature] Support query arguments in controller’s custom commands
  • [Feature] Tune upstream limits in Rspamd proxy
  • [Feature] Use different callback symbols for different uribls
  • [Feature] Write DKIM selector in dkim allow/reject symbols
  • [Fix] Add obs_fws state support to eoh state machine
  • [Fix] Add sanity check when applying mime boundaries heuristic
  • [Fix] Antivirus - virus names with 0 were recognized as tables
  • [Fix] Disable headernames in bayes temporarily
  • [Fix] Do not allow syntax errors in include files…
  • [Fix] Do not allow to merge an object with an array (or vice versa)
  • [Fix] Don’t perform forged recipients check for missing recipients
  • [Fix] Fix DKIM based RBLs
  • [Fix] Fix actrie implementation (sync from upstream), fixed OOB read
  • [Fix] Fix explicit methods call in selectors
  • [Fix] Fix extraction of additional parts
  • [Fix] Fix finalization for internal plugins
  • [Fix] Fix override_defaults function
  • [Fix] Fix squeezed symbols when using settings
  • [Fix] Fix urls insertion in Clickhouse module
  • [Fix] Furhter fixes to ratelimits logic
  • [Fix] Ignore signatures when looking for boundaries
  • [Fix] Properly set learned count
  • [Fix] Really fix ratelimits configuration and work
  • [Fix] Remove ambigious format flag from printf
  • [Fix] Restore URLs exporting in ClickHouse plugin
  • [Fix] Rework bayes calculations…
  • [Fix] Switch from chi-square to naive for large Fisher value
  • [Fix] Treat normal password as enable password if there is no enable password
  • [Fix] Use proper syntax for making DNS requests
  • [Fix] Various fixes in embedded plugins
  • [Project] Change fuzzy check selection logic to lua_fuzzy library
  • [Project] Rework async events and symbols
  • [Project] Move all metatokens in Bayes to lua_stat from C
  • [WebUI] Add history rows per page control

Rspamd 1.8.1 has been released

2018-10-16 00:00:00 +0200

We have released Rspamd 1.8.1 today.

There are couple of the incompatibilities introduced, hence, please check the migration notes. These incompatibilities could affect almost any Rspamd installation with custom configuration, so please read the upgrade notes carefully.

The most important features and fixes

  • Critical fixes in DMARC module

    DMARC module could insert temporary failure symbol instead of strong rejection in certain cases. This has been fixed in 1.8.1. There are many other bugs fixed in this module since the whole check logic has been rewritten in accordance to RFC.

  • Fuzzy check

    Due to bug introduced in 1.8.0, there algorithm used to deterministically skip words in large text parts was not deterministic. It means that the exact words pipelines produced by different Rspamd instances might be different. It could affect if your words_limit was reached (default: words_decay = 200 words). Hence, for large text parts it was expected to have misses in fuzzy and in Bayes classification. Whilst bayes missing should not be significant, the fuzzy misses might be very severe and they might break fuzzy detection for large text parts.

    In 1.8.1, we have fixed this issue and, since we have already broken the compatibility with 1.7.9, we have decided to increase words_decay to 600. Please ensure that you don’t override this parameter anywhere (e.g. in local.d/options.inc, override.d/options.inc or any other override or local file) or your compatibility with Rspamd fuzzy storage would be lost for messages with more than words_decay threshold words.

  • Various language detection issues have been fixed
  • Experimental clustering plugin
  • Important fixes for the dynamic ratelimits
  • Fix options insertion
  • Plug memory leak in redis pool
  • Add check_violation feature to DKIM/ARC signing to avoid signing messages when there is an existing invalid DKIM signature
  • Add only unique elements to Clickhouse url arrays
  • Allow g+: and g-: composite atoms to include symbols with positive score and negative score
  • Allow dkim domains check in surbl module
  • Allow maps with HTTP auth using standard URL syntax
  • Allow to disable actions by users settings by setting them to null
  • Extend whitelisting options:
    • Allow per element whitelist/blacklist only behaviour
    • Introduce three types of lists:
      • Blacklists (bl)
      • Whitelists (wl)
      • Bidirectional lists (both)
  • Use heuristical verdict instead of the plain action in plugins to detect if a message is clearly spam, junk, ham or uncertain when there is not enough confidence despite of the action being set
  • Various Web Interface improvements and fixes driven by Alexander Moisseev

Full list of the meaningful changes

  • [CritFix] Fix options insertion
  • [CritFix] Fix words decay one more time (affects long messages)
  • [CritFix] Increase default words_decay
  • [CritFix] Plug memory leak in redis pool
  • [Feature] Add check_violation feature to DKIM/ARC signing
  • [Feature] Add only unique elements to Clickhouse url arrays
  • [Feature] Allow g+: and g-: composite atoms
  • [Feature] Allow dkim domains check in surbl
  • [Feature] Allow maps with HTTP auth
  • [Feature] Allow to disable actions by users settings
  • [Feature] Extend whitelisting options
  • [Feature] Store url object in images
  • [Feature] Use verdict instead of the plain action in plugins
  • [Fix] Allow to call fstring append with NULL string
  • [Fix] DCC - luacheck
  • [Fix] Do not load torch on each rspamadm invocation
  • [Fix] Fix boundaries detection and rework stop words algorithm
  • [Fix] Fix dependencies for DNS_SIGNED symbol
  • [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit
  • [Fix] Fix groups mess
  • [Fix] Fix groups mess
  • [Fix] Fix parsing address with comments
  • [Fix] Fix resolving in DMARC reports
  • [Fix] Fix various issues with parsing of the received headers
  • [Fix] Fix watchers issue in lua_tcp when doing no resolving
  • [Fix] Plug memory leak in language detector (affects reloads)
  • [Fix] Remove one letter stop words
  • [Fix] Slashing: backport chunk logic from libucl
  • [Fix] Stop libevent from using cached time in rspamadm
  • [Fix] Try to fix watchers chaining
  • [Fix] Various fixes in redis sync interface
  • [Fix] ip_score - respect check_authed and check_local settings from config
  • [Project] Rework passthrough actions
  • [Project] Clustering module
  • [Rework] Always create result for a task
  • [Rework] Completely rewrite DMARC checks logic
  • [Rework] Rework and fix whitelist plugin
  • [WebUI] Add symbols sorting buttons
  • [WebUI] Change symbols order without updating history
  • [WebUI] Colorize symbols
  • [WebUI] Do not display password form when secure_ip is set
  • [WebUI] Fix symbol description tooltips display
  • [WebUI] History: add sorting by symbol score value