We are excited to present the latest version of Rspamd - version 3.6. This release brings a host of new features, enhancements, and fixes to improve further the performance, flexibility, and security of the Rspamd email scanning system. With additions like language detection configuration, dynamic multimap symbol registration, and enhanced fuzzy storage options, Rspamd continues to evolve as a powerful and reliable solution for filtering and classifying messages.

New Features:

• Added one_shot option to specific multimap rules for improved rule behavior
• Introduced language detection configuration and associated attributes
• Added sentinel_password option to enhance Redis sentinel password protection
• Enabled denial of specific fuzzy flags by default for better control over fuzzy storage
• Implemented a controller endpoint to retrieve fuzzy hashes from messages
• Added extra symbol for URL redirector when reaching nested limit for easier identification
• Included a function to transliterate utf8 to ascii with normalization for text processing
• Added html parsing limit and set order to urls structure for improved handling
• Expanded functionality of lua_rsa library with additional functions
• Enabled fuzzy workers to exchange blocked information
• Allowed weak flags in fuzzy storage for more versatile fuzzy matching
• Enabled reading options from maps in the multimap plugin for dynamic configuration
• Provided alternative methods when fasttext detection is enabled
• Enabled counting stats per key per flag for better statistics tracking
• Completed implementation of dynamic composites for more flexible rule composition
• Improved processing of HTML parts before text ones for better text extraction
• Reorganized struct rspamd_url for reduced memory footprint
• Implemented saving fuzzy ratelimit buckets for rate control
• Added ip_map strategy to external_relay plugin for more versatile IP handling
• Implemented on_load support for maps to perform actions on map loading

Fixes:

• Addressed race condition between config new/free using a counter to ensure stability
• Enhanced fasttext language model with pre-tokenized words for improved detection
• Fixed issues with rspamd_has_only_html_part for accurate HTML detection
• Resolved order of destruction race between Redis pool and Lua for stable behavior
• Addressed parsing of invalid mask values for proper configuration handling
• Adjusted header parsing to include the last character when no value is present
• Addressed various issues with fuzzystat for accurate fuzzy storage behavior
• Corrected counter usage for more accurate counting
• Implemented measures to clean pending bucket and remove bad hyperscan files
• Updated stats before encryption to ensure accurate data representation
• Improved DMARC grammar by allowing spaces before ;
• Fixed registration issue in RBL plugin when using symbols_prefixes
• Removed obsolete files related to rspamd-redirector

Project Enhancements:

• Enabled dynamic registration of multimap symbols for flexible rule management
• Implemented fasttext language detection for efficient language classification
• Refactored default max shots to avoid interfering with options
• Rewrote dkim keygen tool in Lua for better performance and functionality
• Added thread hijacking composite rule for improved rule handling

Please note that this is not an exhaustive list of changes and other minor improvements, bug fixes and optimizations have also been included in this release.

We are excited to announce the release of Rspamd 3.5, packed with new features, improvements, and fixes. This version brings enhancements to configuration, critical fixes, and added functionalities to the Rspamd project. Here’s an overview of what you can expect in this release:

New Features:

• Added SURBL hashbl support
• Introduced the thresholds field to the scan result
• Added the ability to execute Lua scripts for blocked fuzzy clients
• Added preliminary support for external maps in the multimap plugin
• Enabled the building of maps by combining tuples of selectors
• Added query support for external maps for settings
• Introduced selector_alias in map definitions
• Enabled MIME part filters on the antivirus module
• Improved rate limit Redis scripts
• Added the specific_urls_filter_map extractor in Selectors
• Reworked the selectors framework

Critical Fixes:

• Deserialized Hyperscan to page-aligned space to prevent alignment issues
• Filled path field in Hyperscan notice command

Fixes:

• Multiple fixes related to Hyperscan, Redis configuration, Ratelimit, RBL, and URL reputation plugin
• Fixed off-by-one error in CSS tokenizer and issues with boundaries containing only dashes
• Restored strict_domains support and replaced broken strict_domains with phishing_exceptions
• Reworked list applications and added external maps support
• Improved handling of hostnames with no dots

Rework:

• Stopped reporting soft reject in history
• Converted the chartable plugin to C++ for convenience
• Changed the approach for customization of settings

Rules:

• Added the MID_END_EQ_FROM_USER_PART rule to the Mid section

Upgrade notes

In addition to the numerous improvements in Rspamd 3.5, this release introduces some notable changes to the supported platforms. We are excited to announce the provision of arm64 packages, extending Rspamd’s compatibility to a wider range of devices. However, as part of our commitment to providing up-to-date and secure software, we have removed support for outdated and end-of-life (EOL) Debian distributions, specifically Ubuntu Bionic and Debian Buster. This decision ensures that our users are running Rspamd on well-maintained platforms with active security updates. For more context on this change and guidance on upgrading your distribution, please refer to the following [document]((/packages_support_policy.html)

It is essential to carefully review the upgrading implications to ensure a smooth transition to Rspamd 3.5. These changes allow us to focus on delivering the best possible email filtering solution while promoting the use of secure and up-to-date platforms.

We have released Rspamd 3.4 today. This is a bugfix release with no incompatible changes. Several new features have also been added. Here are the most important changes in this version explained.

Main changes

Sharing hyperscan database among Rspamd processes

Hyperscan databases are now shared between all Rspamd processes reducing memory footprint, especially when multiple worker processes are running.

Critical fix in the compatibility with the integrations and headers alterations

There was a critical compatibility issue, caused by the change in the milter_headers reply block that prevents some Rspamd integrations to be functional. In this release that issue has been fixed, and the compatibility with the previous output format has been restored.

Fix additional fields in the Redis schema

Some fields were no longer accepted in Redis settingsissue. Now it works correctly.

All significant changes

Here is the list of the important changes:

• [Feature] Milter_headers: Add x-rspamd-action routine
• [Feature] Share hyperscan database among processes
• [Fix] Another corner case in url parsing
• [Fix] Another fix for the enable password
• [Fix] Another try to fix close method in lua_tcp
• [Fix] Fix emoji joiner FP
• [Fix] Fix favicon.ico Content-Type header
• [Fix] Fix hang when close is used
• [Fix] Lua_tcp: Sigh, another try to fix close invocation
• [Fix] Mx_check: Cache the fact of a missing MX record
• [Fix] Try to fix parsing of the unencoded > characters in html attributes
• [Fix] Try to fix the case where password == enable_password
• [Project] (Re)implement hyperscan caching
• [Project] Rework cleanup
• [Project] Synchronize hyperscan caches via the main process
• [Rework] Convert multipattern to use hyperscan tools
• [Rework] Make http normalize path function a generic function
• [Rework] Split locked and unlocked files, as mmap does not need flock normally
• [Rework] Start movement of the hyperscan related routines into a single unit
• [Rework] Store the current worker, so other libraries could use this information
• [Rework] Use blocking socket for IPC between main and workers
• [Rework] Use more predictable size for commands buffers
• [Rules] Do not insert ONCE_RECEIVED_STRICT on RDNS missing
• [Rules] Reduce score of HTTP_TO_HTTPS - subject to remove completely

We have released Rspamd 3.3 today. There are incompatible changes in this release, so please get familiar with the upgrade guide.

Here are the most important changes in this version explained.

Main changes

Reworked and redesigned symbols cache

Symbols cache is responsible for rules exectuion and planning. In this release, there was a major rework of it’s logic and functionality. For example, it can now keep track of timeouts, plan fast events before slow and implement real passthrough for the rules that define such a behaviour. It is useful, when you want some rule to be executed as quick as possible to block or pass evident spam/ham without wasting network/cpu resources. The major drawback of such a rework is that passthrough rules are now really passthrough and can prevent other rules from being executed (that is expected from the design, but it could be not the case before).

Critical fix in the neural network module

There was a regression introduced in the version 3.2 that prevented old keys in Redis to be cleaned that caused infinite Redis database growth. This is fixed in the release 3.3 and the mitigation of this bug are described in the upgrade guide.

DKIM parser now ignores unknown tags

By standard, DKIM checker must ignore unknown tags for forward compatibility. Rspamd will now behave properly and ignore unknown tags as specified in RFC.

Upstreams support in lua_http and lua_tcp modules

It is possible now to use the functionality of the upstreams directly in Lua modules that use lua_http and lua_tcp libraries. It allows better support of the names resolution, IPv6 support for resolving the hostnames and internal handling of the upstreams logic by C code automatically.

CNAME records support in the DNS resolver

Rspamd DNS resolver now supports querying and parsing of the CNAME records. This technique might be useful for fighting some specific spam patterns.

Various memory leaks detected and plugged

In this release, we have found and fixed a good bunch of memory leaks and memory corruptions in the code.

All significant changes

Here is the list of the important changes:

• [Conf] Add missing groups for whitelist module symbols
• [CritFix] Neural: Fix keys regression after #3968
• [Feature] Accept upstream in lua_tcp
• [Feature] Add ability to statically maintain disabled/enabled patterns
• [Feature] Add function to store upstreams for HTTP urls
• [Feature] Allow augmentations set in Lua API
• [Feature] Allow lua_http module to accept upstreams
• [Feature] Allow to limit write access to fuzzy storage by key
• [Feature] Allow to sort symbols output
• [Feature] Check content for binary stuff before dumping it to Lua
• [Feature] Implement symbols augmentations
• [Fix] Add missing flags
• [Fix] Add more sanity checks for rua in dmarc_report
• [Fix] Adjust length of the fuzzy checks for short text parts
• [Fix] Another try to fix add headers compatibility logic
• [Fix] Another try to fix race condition in the runtime destruction
• [Fix] Avoid cyclic references in symcache and fix memory leaks
• [Fix] Avoid overriding IP with Sender IP
• [Fix] BAD_REP_POLICIES did not trigger when message was classified as spam by Bayes
• [Fix] Bind AF_UNIX DGRAM client connection to annonymous address
• [Fix] Disable IPv6 lookups for Blocklist.de RBL
• [Fix] Distinguish dynamic and static items
• [Fix] Dkim: Ignore unknown DKIM kv pairs as stated in RFC
• [Fix] Dmarc report: Use local timezone instead of GMT
• [Fix] Do not exclude authenticated users from URIBL lookups
• [Fix] Empty envelopes should not be emitted as arrays (json+messagepack) when populated envelopes are objects. This greatly complicates decoding in strictly typed languages.
• [Fix] External_relay: Restore the originating hostname check
• [Fix] Fix DKIM keys with spaces still allowing errors on invalid base64
• [Fix] Fix copying of sockaddr_un addresses
• [Fix] Fix crash with cname replies
• [Fix] Fix dependencies propagation
• [Fix] Fix iteration over milter headers
• [Fix] Fix ordering when sorting symcache
• [Fix] Fix reading of the cached maps
• [Fix] Fix several issues with the HTTP keepalive parsing
• [Fix] Fix stack smashing
• [Fix] Fix synchronous auth/select in lua_redis
• [Fix] Fix various symcache issues
• [Fix] Ignore all (I hope) unknown DKIM signature KV pairs
• [Fix] Ignore directories in RarV5 archives
• [Fix] Libucl: avoid memory leak on objects merging
• [Fix] Lua_tcp: Another try to fix closing logic
• [Fix] Mempool: Fix alloc_array function to actually multiply nmembers by size
• [Fix] Only check allowed fuzzy worker update ips for non-unix sockets
• [Fix] Plug memory leak in regexp destruction with pcre2
• [Fix] Properly check the original email flag
• [Fix] Properly deal with get_symbol/get_metric_symbol ambiguity
• [Fix] Properly parse expressions atoms
• [Fix] Properly set Host in rspamd_proxy
• [Fix] Rbl: Fix received positioned checks
• [Fix] Remove check for a score with no symbol being registered
• [Fix] Same fix for lua_tcp
• [Fix] Skip cname records when processing SPF records
• [Fix] Skip sending dmarc reports in no-opt mode fixes https://github.com/rspamd/rspamd/issues/4241
• [Fix] Stop slow timer on task destruction
• [Fix] Symcache: Do not use C style comparators in C++ sorts
• [Fix] Try to avoid a corner case for @ pattern
• [Fix] Try to fix dkim reputation adjustements
• [Fix] Try to fix passthrough results processing logic
• [Fix] Try to fix the mess with read only flag
• [Fix] Upstreams: Don’t ignore revive_time config option
• [Fix] Use proper format string, sigh…
• [Fix] Use space category in ragel automata to resolve space characters
• [Fix] Zstd: Fix compression with the new Zstd API
• [Fix] milter_headers: Header fields may be inserted at wrong position.
• [Project] Rework symbols cache
• [Rework] Rewrite rspamc in C++

We have released Rspamd 3.2 today. This version is mostly bugfix release with several new features implemented.

Here are the most important changes in this version explained.

Main changes

DNS over TCP support

For a long time, Rspamd was unable to switch to TCP when processing DNS replies that are too large to be transferred over UDP. The portion of such a messages was never high, but there are some notable examples of the records that cannot fit into a UDP packet even with EDNS0 extension enabled. That are mostly poorly maintained TXT records that contains lot’s of legacy Google verification junk. However, it affected the SPF authentication, so I have decided to implement TCP fallback after all.

BIMI support

Rspamd can now download and verify logotypes from a validated certificates using a dedicated helper written in Rust. With this plugin, Rspamd can enrich your emails, appending a header with the logotype image (in SVG format) if all BIMI validation steps were successful.

Average scan time

It is now possible to fetch an average messages scan time via Rspamd HTTP API, Prometheus endpoint and even via ps command (supported on some OS only).

Monitor helper tool

You can now attach a CLI helper to Rspamd to get some real time performance graphs directly from your terminal:

Cloudmark support

You can now use Cloudmark via the external services module.

Other fixes and improvements

Core and API

• Fixed SSL support in many places
• Switched to XXHash3 as a fast hash source
• Fix upstreams name resolution when there is also a port
• Allow hyperscan for ppc64, as vectorscan now suports it
• Lua_magic: Add a sane CSV heuristic
• Allow to restore SSL handlers after keepalive pooling

Plugins

• Neural: dd ROC feature to neural network plugin
• Fixed retention settings in Clickhouse plugin
• Fixed important issues in the reputation plugin

Rules

• Added some sanity limits for symbol groups
• Fix symbol for DKIM temporary failure
• Remove ancient and inefficient rules

All changes

Here is the list of the important changes:

• [Conf] Score MIME_OBFUSCATED_ARCHIVE to 8 points
• [Conf] Set one_shot for URIBL rules by default
• [CritFix] Fix upstreams name resolution when there is also a port
• [Feature] Add ROC feature to neural network plugin
• [Feature] Add public suffic compilation utility
• [Feature] Add support of Cloudmark
• [Feature] Allow hyperscan for ppc64, as vectorscan now suports it.
• [Feature] Allow to skip DNS resolution for keep-alive connections
• [Feature] Aws_s3: Allow to store large parts separately
• [Feature] BIMI: Add preliminary version of the BIMI plugin
• [Feature] JSON endpoint for querying maps
• [Feature] Lua_magic: Add a sane CSV heuristic
• [Feature] Lua_mime: Add schema for message transfer
• [Feature] Output average scan time in /stat endpoint
• [Feature] Show average scan time in rspamc stat output
• [Fix] Add guards to avoid race condition on TCP connection
• [Fix] Allow spaces in DKIM key records
• [Fix] Apply the similar fix to the url_reputation
• [Fix] Avoid overwriting whitelisted_signers_map
• [Fix] Backport PR from libucl
• [Fix] Clear SSL errors
• [Fix] ClickHouse cleanup of old partitions
• [Fix] Do not double call error handler on ssl errors in the timeout path
• [Fix] Do not forget to clear pointers on IOC reset
• [Fix] External_relay: Remove useless check of the map value
• [Fix] Find suspicious url encodings that could break url extraction
• [Fix] Fix HTTP(s) client timeout
• [Fix] Fix exclude flags setting
• [Fix] Fix expanding of the variables
• [Fix] Fix host header usage in lua_http
• [Fix] Fix http maps shared memory cache cleanup
• [Fix] Fix logic in HTML processing FSM
• [Fix] Fix parsing of the compound mailto urls
• [Fix] Fix processing captures from pcre2
• [Fix] Fix removing from khash
• [Fix] Fix stuctured headers pushing
• [Fix] Further fix for i386 compilation
• [Fix] Improve duplicate settings error reporting
• [Fix] Lua: task:remove_result didn’t work in some cases
• [Fix] Output service parts as well
• [Fix] Phishing: Deal with phishing + redirected URL
• [Fix] Phishing: Fix finding domains in the phishing map
• [Fix] Plug memory leak by using mempool for a copied address
• [Fix] Properly find the request and the number of requested entries
• [Fix] Rbl: Fix inversed logic of the url_full_hostname
• [Fix] Read file maps if they were not pre-read during preload
• [Fix] Restrict x86_64 assembly to x86_64
• [Fix] Return a real number of recipients when dealing with aliases
• [Fix] Rework unshedule DNS request function
• [Fix] Support definition of ungrouped symbol in conf file, use group info from lua or other conf file
• [Fix] Unschedule DNS request when clearing IO channel
• [Fix] When checking for phishing, we need to convert punicode -> UTF8, not vice versa
• [Fix] lua_cfg_transform - actions without score (discard)
• [Fix] lua_cfg_transform - silly break break actions
• [Fix] ratelimit - symbol per bucket
• [Project] BIMI: Fix helper integration issues
• [Project] Further DNS over TCP architecturing
• [Project] Rdns: Add more functions for TCP based requests
• [Project] Rdns: Add preliminary reading logic for TCP channels
• [Project] Rdns: Add reaper for inactive TCP connections
• [Project] Rdns: Add timeout logic for TCP requests
• [Project] Rdns: Do not treat TCP channels failure as fatal
• [Project] Rdns: Fix TCP connection mess
• [Project] Rdns: Fix TCP stuff cleanup
• [Project] Rdns: Fix various ownership issues
• [Project] Rdns: Implement TCP writing logic
• [Project] Rdns: Initial support of TCP IO channels
• [Project] Rdns: More fixes in TCP handling
• [Project] Rdns: Restore the previous EDNS0 size
• [Project] Rdns: Send truncated replies via TCP
• [Project] Rdns: Unregister TCP requests
• [Rework] Allow to restore SSL handlers after keepalive pooling
• [Rework] Allow to set a different behaviour for actions from settings
• [Rework] Include SSL flag into keepalive hash
• [Rework] Make rspamadm dmarc_report default behaviour more sane
• [Rework] Mempool: Use explicit alignment
• [Rework] Rdns: Use faster and more compact hash table for DNS requests
• [Rework] Rework SSL flag operations
• [Rework] Take disabled flag into account
• [Rework] Timeouts are now global per event and not reseted by IO activity
• [Rework] Use xxh3 as a default hash and fix memory/alignment issues
• [Rules] Fix old rules to stop global functions usage
• [Rules] Fix symbol for DKIM temporary failure
• [Rules] Remove ancient and inefficient rules
• [Rules] Slightly reduce MULTIPLE_FROM score