We have released Rspamd 1.7.7 today. There are no incompatible changes introduced with this version to our best knowledge.

The most important features and fixes

• Add rspamadm mime tool to do various email operations:
  • extract text/HTML content
  • extract statistical tokens
  • exctact URLs
• Fixed encryption mode in Rspamd proxy
• Fixed various crashes in maps during reload
• Preload maps data before starting of the worker processes when possible
• Better HTML styles processing: add ZeroFont exploit filtering rules
• Fix ED25519 DKIM signatures as described by the latest RFC draft
• Added crash reporting system via libunwind

Full list of the meaningful changes

• [CritFix] Check NM part of pubkey to match it with rotating keypairs
• [CritFix] Do not overwrite PID of the main process
• [CritFix] Fix maps after reload
• [CritFix] Fix maps race conditions on reload
• [CritFix] Fix shmem leak in encrypting proxy mode
• [Feature] Add a concept of ignored symbols to avoid race conditions
• [Feature] Add ability to print bayes tokens in rspamadm mime
• [Feature] Add method to get statistical tokens in Lua API
• [Feature] Add preliminary mime stat command
• [Feature] Add rspamadm mime tool
• [Feature] Add urls extraction tool
• [Feature] Address ZeroFont exploit
• [Feature] Allow rspamadm mime to process multiple files
• [Feature] Allow to extract words in rspamadm mime
• [Feature] Allow to print mime part data
• [Feature] Allow to show HTML structure on extraction
• [Feature] Distinguish IP failures from connection failures
• [Feature] Improve output for mime command
• [Feature] Improve styles propagation
• [Feature] Main process crash will now cleanup all children
• [Feature] Preload file and static maps in main process
• [Feature] Print stack trace on crash
• [Feature] Process font size in HTML parser
• [Feature] Propagate content length of invisible tags
• [Feature] Read ordinary file maps in chunks to be more safe on rewrites
• [Feature] Support base tag in HTML
• [Feature] Support more size suffixes when parsing HTML styles
• [Feature] Support opacity style
• [Fix] Another fix for nested composites
• [Fix] Fill nm id in keypairs cache code
• [Fix] Fix colors alpha channel handling
• [Fix] Fix destruction logic
• [Fix] Fix double free
• [Fix] Fix maps preload logic
• [Fix] Fix nested composites process
• [Fix] Fix proxying of Exim connections
• [Fix] Fix reload crash
• [Fix] Fix rspamadm -l command
• [Fix] Update ed25519 signing schema
• [WebUI] Stop using “const” declaration
• [WebUI] Update RequireJS to 2.3.5

We have released Rspamd 1.7.6 today. There are no incompatible changes introduced with this version to our best knowledge.

The most important features and fixes

• Fix multiple neural networks support: it is now possible to learn multiple neural networks with different settings as documented
• Rework rspamadm to use mostly Lua for subcommands for better documentation and extensions support
• Add pubkey checks for dkim_signing module (#2277)
• DMARC reports are now compressed using gzip as suggested by RFC
• Settings module can now skip message processing to improve performance
• Bayes classifier now consider more metatokens from the headers
• ED25519 DKIM signatures are now supported
• Fixed serious issues with composites, maps and other components
• Major memory leak hunting and eliminating (especially those that occurs during reload)
• Add more tests and allow to create fake DNS records to make certain tests self-contained (e.g. DKIM or DMARC)

Full list of the meaningful changes

• [CritFix] Fix multiple neural networks support
• [Feature] Add decryption function to keypair command
• [Feature] Add gzip compression for HTTP requests in elastic module
• [Feature] Add gzip methods to lua util
• [Feature] Add maps based on Top Level Domains
• [Feature] Add pubkey checks for dkim_signing
• [Feature] Add support of fake DNS records
• [Feature] Add tool to encrypt files
• [Feature] Allow to add symbols using settings directly
• [Feature] Allow to match private and public keys for DKIM signatures
• [Feature] Allow to set task flags via settings
• [Feature] Allow to specify fake DNS address from the config
• [Feature] Implement signatures verification using rspamadm keypair
• [Feature] Implement signing using rspamadm keypair
• [Feature] Improve error reporting for DKIM key access issues
• [Feature] Provide $HOSTNAME variable in UCL
• [Feature] Rework levenshtein distance computation
• [Feature] Split message parsing and processing
• [Feature] Support ED25519 DKIM signatures
• [Feature] Support encrypted configs in UCL
• [Feature] Suppress duplicate warning on very large radix tries
• [Feature] Use OSB to combine header names
• [Fix] Cleanup maps data on shutdown
• [Fix] Fix ‘~’ behaviour in composites
• [Fix] Fix HTTP maps updates
• [Fix] Fix NIST signatures
• [Fix] Fix RFC822 comments when processing a mime address
• [Fix] Fix double free
• [Fix] Fix dynamic settings application
• [Fix] Fix for CommuniGate Pro maillist
• [Fix] Fix keypair creation method to actually create keypair…
• [Fix] Fix matching patterns with no paths
• [Fix] Fix memory leak in parsing comments
• [Fix] Fix parsing of urls with numeric password
• [Fix] Fix plugins intialisation in configwizard
• [Fix] Fix potential crash on reload
• [Fix] Fix potential race condition for a finished HTTP connections
• [Fix] Fix race-condition leak on processes reload
• [Fix] Fix signing in openssl mode
• [Fix] Free language detector structures
• [Fix] Relax alignment requirements
• [Fix] Send DMARC reports compressed
• [Fix] Try to fix leak in dmarc module
• [Fix] Try to plug memory leak in metric exporter
• [Project] Convert rspamadm subcommands to Lua
• [WebUI] Display smtp sender/recipient in history
• [WebUI] Fix elements disabling in “Symbols” tab
• [WebUI] Limit recipients list in history column to 3
• [WebUI] Match envelope and mime addresses following in arbitrary order
• [WebUI] Update column header
• [WebUI] Wrap addresses in history

We are proud to announce the next major version of Rspamd today: 1.7.0. This version includes more than 1000 changes since Rspamd 1.6 branch. The most significant changes are:

• Better machine learning support and embedding of Lua Torch
• Language detection support based on ngramms and heuristics
• New rspamadm configwizard command for a simple configuration setup
• New statistics model for Redis backend allowing expiration and better analytics
• Improved wizards for statistics conversion and management
• Added automatic corpus test and rescoring utility based on Google Summer of Code 2017 project completed by @cpragadeesh
• New Elasticsearch plugin
• New experimental Reputation plugin
• Various other important improvements and fixes

Release cycle model change

From this version onwards, Rspamd will no longer have stable and experimental branches. All development will be concentrated in the main branch with more frequent releases of both minor and major versions. Major version change (e.g. from 1.7 to 1.8) will mean some important change with backward compatility or a clear conversion path. Minor releases (e.g. 1.7.0 -> 1.7.1) will be released as soon as there are enough important changes. Any critical fix will cause a new version to be released.

We have decided to eliminate the concept of stable branches since it makes the processes of development and migration more complicated for both developers and Rspamd users. We will concentrate on stability and backward compatibility of the main branch instead.

Migration notes

Please read migration notes. We believe that no explicit configuration changes are required to upgrade from Rspamd 1.6.6.

New features in Rspamd 1.7

Here is a brief description of the new features appeared in Rspamd 1.7.

Better machine learning support

Rspamd has bundled torch support which is enabled by default. This framework is one of the principal Machine Learning frameworks implemented in Lua. We have decided to bundle it directly in Rspamd due to poor support of packaged version of Torch in the vast majority of Linux and Unix distributions. It is currently available for Intel x86_64 architecture only. Rspamd includes the following components of this framework:

• basic Tensors and math framework (torch7)
• neural networks support (nn)
• optimization algorithms package (optim)
• random forests support

So far, Torch is used in neural network module to improve neural network model and the speed of processing. This module has been improved to support clustered configurations, mail flows separation, and multiple neural networks.

Torch also empowers a new rescore utility described later in this article.

You can use Torch to build your own Machine Learning models to improve the quality of your spam filtering rules.

Language detection support

Rspamd 1.7 includes a new language detection support. It uses NGramms model to support more than 50 languages. Rspamd implements a fast and sophisticated algorithm to detect texts languages using unicode properties, ngramms and statistical methods to provide a more precise language detection. This information could be used for training models (e.g. word2vec embedding), better bayes classification (e.g. by removing of stop words), or by individual rules.

New rspamadm configwizard command

This command is intended to simplify Rspamd configuration and migration. It provides an interactive console UI to setup the most commonly used Rspamd functions not necessarily configured out-of-the-box:

It can also be used on any stage to adjust configuration:

• DKIM signing setup: rspamadm configwizard dkim
• Controller password: rspamadm configwizard controller
• Statistics tools: rspamadm configwizard statistic
• Redis setup: rspamadm configwizard redis

New model for Redis backend of statistical data

Rspamd has switched from the old layout of Redis storage where tokens were stored in two large hash tables: RSBAYES_SPAM and RSBAYES_HAM to a model where each token is stored separately: RS_<token_id> and 2 buckets S for spam hits and H for ham hits. The new model requires more space in Redis, however, it allows to expire meaningless or not frequently used tokens efficiently reducing storage requirements. A new explicitly enabled plugin called bayes_expire provides inteligent renewal and eviction of statistical tokens.

It is also possible to store token values inside statistical buckets for debugging and analytics purposes.

You can convert an old statistics in Redis (or sqlite) to a new one using rspamadm configwizard statistic command.

Corpus test and rescore tool

This project has been started as a Google Summer Of Code project and was completed by @cpragadeesh in 2017. It allows Rspamd to run against some pre-labeled corpus of spam and ham (rspamadm corpus_test tool) and then analyze anonymous logs produced by this command to adjust the best possible scores for Rspamd rules (rspamadm rescore). Here is a sample of this command run:

$ rspamadm rescore -l ./rescore_logs -o scores.ucl -d -i 10 --penalty-weight=0 --momentum=0.1 --learning-rate=0.01 --l1=0.0001

...

We plan to improve our scores using this tool and distribute updates using rspamd_update plugin.

Elasticsearch plugin

Rspamd now provides elastic search plugin that provides integration with Elasticsearch engine and Kibana board to visualise Rspamd data. It has such features as symbols heat map, geographical distribution of spam/ham sources, proportions of traffic scanned and so on. Elastic search plugin and clickhouse plugin provide a decent platform to collect analytics about your mail flows and spam filter efficiency.

Reputation plugin

Rspamd 1.7 includes an experimental reputation plugin. It is currently in work-in-progress state, however, we have a working prototype with the following features:

• DKIM domain, URL domains and IP reputation support
• Flexible backends supports: both Redis for private usage and DNS for public services
• Flexible time buckets: long term buckets and short term buckets
• Flexible aggregation tool (weighted aggregation, decision trees and so on)

This plugin is intended to replace the original ip score plugin and will provide much more reputation types (e.g. URL and DKIM reputation). It is also possible to build systems with both public reputation data that could be provided via DNS and internal reputation data stored in Redis buckets. This could be particularly beneficial for large email service providers.

In this version, the plugin is in still experimental stage but it is close to production testing so far.

7Zip support

Rspamd can now detect and process data from 7zip files. This functionality lives within mime types module and allows to filter malicious files in 7zip attachments.

Various improvements and changes

In conclusion, we can add that this version of Rspamd includes a lot of improvements in stability, performance and quality of filtering areas.

Today, we release the new major version 1.6.0 of Rspamd. The most significant change in this version is the addition of Milter protocol support in Rspamd. Therefore, Rmilter project is finally turned to the abandoned state and should not be used in new installations. All Rmilter users should consider migration to Rspamd milter support. This release has some incompatible changes so please check the migration guide.

Here is the list of most noticeable changes.

Milter protocol support

From Rspamd 1.6, rspamd proxy worker supports milter protocol which is supported by some of the popular MTAs, such as Postfix or Sendmail. The introduction of this feature also finally obsoletes the Rmilter project in honor of the new integration method. Milter support is presented in rspamd_proxy only, however, there are two possibilities to use milter protocol:

• Proxy mode (for large instances) with a dedicated scan layer
• Self-scan mode (for small instances)

Here, we describe the simplest self-scan option:

In this mode, rspamd_proxy scans messages itself and talk to MTA directly using Milter protocol. The advantage of this mode is its simplicity. Here is a sample configuration for this mode:

# local.d/worker-proxy.inc
milter = yes; # Enable milter mode
timeout = 120s; # Needed for Milter usually
upstream "local" {
  default = yes; # Self-scan upstreams are always default
  self_scan = yes; # Enable self-scan
}

For more advanced proxy usage, please see the corresponding documentation.

ARC support added

There is full support of ARC signatures and seals for emails scanned in Rspamd 1.6.0. ARC signatures can establish that a specific message has been signed and then forwarded by a number of trusted relays. There is a good overview of the ARC standard here: https://dmarc.org/presentations/ARC-Overview-2016Q2-v03.pdf.

Rspamd ARC module supports both verification and signing for outbound messages. Its configuration is very similar to dkim_signing module.

New statistics model for Redis storage

Rspamd 1.6 includes experimental support for new tokens storage in Redis. In this scheme, it is easier to get data about specific tokens and perform tokens expiration. However, this support is not enabled by default in this release but you can try it as well as Bayes expiration plugin by yourself. In future releases, this model will be the default and you will be able to convert the existing storage to the new scheme without data loss.

New expiration algorithm for internal caches

Rspamd now has an implementation of Least Frequently Used (LFU) algorithm instead of the classic Least Recently Used (LRU) used before. The idea comes from the Redis server where it has been used for a long time. With this algorithm, Rspamd will cache frequently used items for longer time and the overall performance of the caches is expected to be better.

DMARC reports support

DMARC module now supports reports sending (using SMTP smarthost) for specific domains and policies. There are many options available for fine tuning of these reports content, frequency, domains and many other. DMARC reports are intended to provide better feedback for the resources that are using DMARC (e.g. paypal.com) from their recipients. Namely, they can detect some phishing trends and react to them somehow.

Spam trap plugin

New plugin spam trap has been added to Rspamd to simplify spamtraps organization. This plugin allows to learn fuzzy storages or/and Bayes from some honeypots.

URL redirector improvements

There are various changes in url redirector module. Namely, it now expires processing items more aggressively to avoid leftovers. Some of dependency issues have also been resolved. Furthermore, this plugin now has a list of top redirection destinations allowing to deal with some bad URLs exploited by spammers.

Multiple metrics support has been removed from Rspamd

From version 1.6, multiple metrics support is now completely removed from Rspamd. The only valid metric is now default. This feature has never ever been used since version 0.2 of Rspamd, however, it consumed some resources and introduced extra complicatinons to the protocol and configuration.

Hence, this feature has been removed and the new endpoint /checkv2 has been added to the protocol. The legacy /check and /symbols endpoints are still using old protocol definition and they will be used for backward compatibility in future.

Compression support in proxy

Rspamd proxy now supports transport compression when sending messages to the scanning layer.

Various performance improvements

Here is a list of significant performance related changes:

• Hfilter regular expressions now can use hyperscan
• DKIM body hash is now cached to improve multiple signatures support
• Snowball stemmers are also cached for better performance

Miscellaneous

Here is a list of other changes made in this release:

• Various rules fixes (FORWARDED, URI_COUNT_ODD and others)
• Bugfixes and other improvements
• New Lua API functions

Synopsis

We have migrated hardware that served https://rspamd.com site and related services including fuzzy storage.

Problem description

All Rspamd users who are using rspamd.com fuzzy storage might see the following messages in the log:

fuzzy_check_timer_callback: got IO timeout with server rspamd.com(5.9.155.182), after 3 retransmits

Normally, Rspamd re-resolves hostnames in this case. However, if there is a single server specified (as enabled by default) there is no resolving on errors. Unfortunately, this bug has been fixed merely in the master branch and is not released in the stable versions yet.

Potential outcome

The quality of filtering might be temporary reduced as fuzzy storage helps to filter certain spam types.

Workaround

You just need to restart Rspamd and it will use the new IP address as intended. We do apologise for any inconveniences caused.