Rspamd 1.8.1 has been released

16 Oct 2018

We have released Rspamd 1.8.1 today.

There are couple of the incompatibilities introduced, hence, please check the migration notes. These incompatibilities could affect almost any Rspamd installation with custom configuration, so please read the upgrade notes carefully.

The most important features and fixes

  • Critical fixes in DMARC module

    DMARC module could insert temporary failure symbol instead of strong rejection in certain cases. This has been fixed in 1.8.1. There are many other bugs fixed in this module since the whole check logic has been rewritten in accordance to RFC.

  • Fuzzy check

    Due to bug introduced in 1.8.0, there algorithm used to deterministically skip words in large text parts was not deterministic. It means that the exact words pipelines produced by different Rspamd instances might be different. It could affect if your words_limit was reached (default: words_decay = 200 words). Hence, for large text parts it was expected to have misses in fuzzy and in Bayes classification. Whilst bayes missing should not be significant, the fuzzy misses might be very severe and they might break fuzzy detection for large text parts.

    In 1.8.1, we have fixed this issue and, since we have already broken the compatibility with 1.7.9, we have decided to increase words_decay to 600. Please ensure that you don’t override this parameter anywhere (e.g. in local.d/options.inc, override.d/options.inc or any other override or local file) or your compatibility with Rspamd fuzzy storage would be lost for messages with more than words_decay threshold words.

  • Various language detection issues have been fixed
  • Experimental clustering plugin
  • Important fixes for the dynamic ratelimits
  • Fix options insertion
  • Plug memory leak in redis pool
  • Add check_violation feature to DKIM/ARC signing to avoid signing messages when there is an existing invalid DKIM signature
  • Add only unique elements to Clickhouse url arrays
  • Allow g+: and g-: composite atoms to include symbols with positive score and negative score
  • Allow dkim domains check in surbl module
  • Allow maps with HTTP auth using standard URL syntax
  • Allow to disable actions by users settings by setting them to null
  • Extend whitelisting options:
    • Allow per element whitelist/blacklist only behaviour
    • Introduce three types of lists:
      • Blacklists (bl)
      • Whitelists (wl)
      • Bidirectional lists (both)
  • Use heuristical verdict instead of the plain action in plugins to detect if a message is clearly spam, junk, ham or uncertain when there is not enough confidence despite of the action being set
  • Various Web Interface improvements and fixes driven by Alexander Moisseev

Full list of the meaningful changes

  • [CritFix] Fix options insertion
  • [CritFix] Fix words decay one more time (affects long messages)
  • [CritFix] Increase default words_decay
  • [CritFix] Plug memory leak in redis pool
  • [Feature] Add check_violation feature to DKIM/ARC signing
  • [Feature] Add only unique elements to Clickhouse url arrays
  • [Feature] Allow g+: and g-: composite atoms
  • [Feature] Allow dkim domains check in surbl
  • [Feature] Allow maps with HTTP auth
  • [Feature] Allow to disable actions by users settings
  • [Feature] Extend whitelisting options
  • [Feature] Store url object in images
  • [Feature] Use verdict instead of the plain action in plugins
  • [Fix] Allow to call fstring append with NULL string
  • [Fix] DCC - luacheck
  • [Fix] Do not load torch on each rspamadm invocation
  • [Fix] Fix boundaries detection and rework stop words algorithm
  • [Fix] Fix dependencies for DNS_SIGNED symbol
  • [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit
  • [Fix] Fix groups mess
  • [Fix] Fix groups mess
  • [Fix] Fix parsing address with comments
  • [Fix] Fix resolving in DMARC reports
  • [Fix] Fix various issues with parsing of the received headers
  • [Fix] Fix watchers issue in lua_tcp when doing no resolving
  • [Fix] Plug memory leak in language detector (affects reloads)
  • [Fix] Remove one letter stop words
  • [Fix] Slashing: backport chunk logic from libucl
  • [Fix] Stop libevent from using cached time in rspamadm
  • [Fix] Try to fix watchers chaining
  • [Fix] Various fixes in redis sync interface
  • [Fix] ip_score - respect check_authed and check_local settings from config
  • [Project] Rework passthrough actions
  • [Project] Clustering module
  • [Rework] Always create result for a task
  • [Rework] Completely rewrite DMARC checks logic
  • [Rework] Rework and fix whitelist plugin
  • [WebUI] Add symbols sorting buttons
  • [WebUI] Change symbols order without updating history
  • [WebUI] Colorize symbols
  • [WebUI] Do not display password form when secure_ip is set
  • [WebUI] Fix symbol description tooltips display
  • [WebUI] History: add sorting by symbol score value