Rspamd performance measures

2019-05-16 00:00:00 +0200

Preface

Rspamd has always been oriented on the performance but it was always quite hard to measure how fast it was as normally it runs just fast enough.

However, I was recently offered to process Abusix Intelligence feeds using Rspamd. These feeds are used to improve Rspamd fuzzy storage quality, to feed URLs and Emails to the DNS black lists provided by Rspamd project and used in SURBL module.

Problem statement

The amount of data that required to be processing is huge - it is about 100 millions of messages per day.

Here is an example to calculate connections count when processing these messages using Rspamd:

$ rspamc stat | \ grep 'Connections count' | \ cut -d' ' -f3 ; \ sleep 10 ; \ rspamc stat | \ grep 'Connections count' | \ cut -d' ' -f3 23548811 23564384

It means that over 10 seconds Rspamd has to process around 15 thousands of messages which gives us a rate of 1500 messages per second.

Rspamd setup

The settings used to process this amount of messages are pretty similar to those that are provided by default.

There is also some significant amount of home-crafted scripts written in Lua to provide the following functionality:

  • Provides deduplication to save time on processing of duplicates
  • Performs conditional checks for url and emails blacklisting:
    • checks if an url is in whitelists (around 5 whitelists stored in Redis are used)
    • check if an url is already listed
    • check if it matches any suspicious patterns
  • Checks if a message should be learned on fuzzy storage (various conditions)
  • Stores messages in IMAP folders providing sorting, partitioning and sampling logic
  • Doing various HTTP and Redis queries for servicing purposes

Hardware

Now some words about hardware being used.

Previously we have set the same setup on a small instance of AX-60 and it was loaded for around 80%. We have decided to move to a more powerful server to have some margin for processing more emails and doing some experiments.

Hence, we now have an AX-160 AMD server rented in Hetzner. This is quite a powerful machine and the current load pictures look like this one:

top - 14:36:26 up 23:26, 1 user, load average: 15.76, 13.22, 12.46 Tasks: 511 total, 3 running, 508 sleeping, 0 stopped, 0 zombie %Cpu(s): 14.1 us, 4.6 sy, 0.0 ni, 78.9 id, 0.0 wa, 0.0 hi, 2.4 si, 0.0 st MiB Mem : 128802.5 total, 56985.7 free, 27897.5 used, 43919.3 buff/cache MiB Swap: 4092.0 total, 3925.5 free, 166.5 used. 100018.6 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14085 unbound 20 0 2058412 1.6g 6852 S 131.2 1.3 1478:04 unbound 66509 _rspamd 20 0 806976 733336 23592 S 68.8 0.6 169:52.21 rspamd 66498 _rspamd 20 0 780144 699540 23852 S 62.5 0.5 156:19.14 rspamd 66502 _rspamd 20 0 816152 744352 23796 S 56.2 0.6 164:26.39 rspamd 66468 _rspamd 20 0 773532 697084 23736 S 50.0 0.5 117:36.32 rspamd 66491 _rspamd 20 0 806652 722340 23728 S 50.0 0.5 148:04.54 rspamd 66476 _rspamd 20 0 767300 705996 23596 S 43.8 0.5 129:04.30 rspamd 66481 _rspamd 20 0 797944 730528 23896 S 43.8 0.6 139:34.35 rspamd 66443 _rspamd 20 0 727632 657104 23372 S 37.5 0.5 88:39.26 rspamd 66451 _rspamd 20 0 742192 665196 23632 S 37.5 0.5 94:49.75 rspamd 66456 _rspamd 20 0 790908 725784 23488 S 37.5 0.6 101:32.06 rspamd 66463 _rspamd 20 0 771540 696064 23692 S 37.5 0.5 108:08.65 rspamd 66487 _rspamd 20 0 780220 713024 23428 S 37.5 0.5 144:51.79 rspamd 66447 _rspamd 20 0 762440 689592 23736 S 31.2 0.5 90:23.93 rspamd 66455 _rspamd 20 0 763520 696108 23580 S 31.2 0.5 97:57.57 rspamd 66464 _rspamd 20 0 764644 688724 23696 S 31.2 0.5 111:32.74 rspamd 66469 _rspamd 20 0 756952 678704 23612 S 31.2 0.5 127:55.02 rspamd 127011 rbldns 20 0 358824 307700 2244 R 31.2 0.2 10:26.14 rbldnsd 10767 redis 20 0 9912104 7.7g 2532 S 25.0 6.1 236:29.63 redis-server 66438 _rspamd 20 0 746772 680624 23424 R 25.0 0.5 82:18.04 rspamd 66433 _rspamd 20 0 751180 687244 23472 S 18.8 0.5 80:12.21 rspamd 66437 _rspamd 20 0 737200 669428 23796 S 18.8 0.5 81:37.81 rspamd 10671 stunnel4 20 0 24.0g 77252 3644 S 12.5 0.1 269:06.53 stunnel4 26994 root 20 0 11900 3984 3072 R 12.5 0.0 0:00.02 top 66442 _rspamd 20 0 808808 707020 23608 S 12.5 0.5 85:11.64 rspamd 17821 clickho+ 20 0 21.8g 3.9g 18964 S 6.2 3.1 116:13.04 clickhouse-serv

Rspamd is also being fed via proxy worker that runs on another host and performs initial data collection and emitting messages via the Internet providing transport encryption using HTTPCrypt. However, its CPU usage is quite negligible - it uses only a single CPU core by around 40% in average.

Results analytics

As you can see, this machine runs also Clickhouse, Redis, own recursive resolver (Unbound), and it still has ~80% idle processing these 1500 messages per second.

If we look at the performance counters by attaching to some of the worker processes, we would see the following picture:

# timeout 30 perf record -p 66481 [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 1.171 MB perf.data (29833 samples) ] # perf report # Overhead Command Shared Object Symbol # ........ ....... ....................... ....................................................................... # 5.23% rspamd rspamd [.] lj_alloc_free 3.35% rspamd rspamd [.] lj_str_new 3.03% rspamd librspamd-server.so [.] gc_sweep 2.20% rspamd rspamd [.] lj_alloc_malloc 1.94% rspamd rspamd [.] gc_sweep 1.50% rspamd libc-2.28.so [.] __strlen_avx2 1.32% rspamd rspamd [.] release_unused_segments 1.24% rspamd rspamd [.] lj_BC_TGETS 1.17% rspamd libjemalloc.so.2 [.] free 1.04% rspamd librspamd-server.so [.] lj_BC_JLOOP 1.03% rspamd librspamd-server.so [.] propagatemark 1.01% rspamd libpthread-2.28.so [.] __pthread_mutex_lock 1.01% rspamd libglib-2.0.so.0.5800.3 [.] g_hash_table_lookup 0.94% rspamd libjemalloc.so.2 [.] malloc 0.77% rspamd rspamd [.] lj_func_newL_gc 0.76% rspamd rspamd [.] propagatemark 0.75% rspamd rspamd [.] lj_tab_get 0.69% rspamd libpthread-2.28.so [.] __pthread_mutex_unlock_usercnt 0.65% rspamd librspamd-server.so [.] t1ha2_atonce 0.61% rspamd librspamd-server.so [.] newtab 0.60% rspamd libicui18n.so.63.1 [.] icu_63::NGramParser::search 0.59% rspamd [kernel.kallsyms] [k] copy_user_generic_string 0.58% rspamd librspamd-server.so [.] match 0.58% rspamd librspamd-server.so [.] lj_tab_new1 0.56% rspamd librspamd-server.so [.] rspamd_task_find_symbol_result 0.52% rspamd [kernel.kallsyms] [k] _raw_spin_lock_irqsave 0.48% rspamd librspamd-server.so [.] rspamd_vprintf_common 0.46% rspamd librspamd-server.so [.] lj_str_new 0.42% rspamd rspamd [.] index2adr 0.42% rspamd rspamd [.] lj_BC_CALL 0.42% rspamd libc-2.28.so [.] __strcmp_avx2 0.42% rspamd libc-2.28.so [.] __memmove_avx_unaligned_erms

The top consumers are Lua allocator and garbage collector. Since we are using Rspamd experimental package on Debian Buster, then it is built with bundled LuaJIT 2.1 beta3 and Jemalloc allocator, however, it seems that there is some issue with this allocator in Debian Buster, so I had to load it manually via the following command:

# systemctl edit rspamd.service [Service] Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"

Followed by restarting of Rspamd.

It is interesting that this Rspamd setup accepts all connections encrypted using HTTPCrypt but chacha_blocks_avx2 takes less than 0.16% of CPU according to perf report.

This particular instance of Rspamd is slightly tuned to use more memory to save some CPU cycles:

# local.d/options.inc

lua_gc_step = 100;
lua_gc_pause = 400;
full_gc_iters = 10000;

These options tell Rspamd to preserve Lua objects in memory for longer time, at the same time in this mode, we can also observe GC stats on workers that performs full GC loop each 10k messages being scanned:

$ tail -f /var/log/rspamd/rspamd.log | fgrep 'full gc' perform full gc cycle; memory stats: 58.66MiB allocated, 62.01MiB active, 6.08MiB metadata, 84.71MiB resident, 90.64MiB mapped; lua memory: 107377 kb -> 38015 kb; 308.0022420035675 ms for gc iter

As you can see, full GC iter takes quite a significant time. However, it still keeps Lua memory usage sane. The ideas behind this GC mode have been taken from the generational GC idea in LuaJIT Wiki.

Resulting graphs

Here are some UI captures taken from a previous machine:

As you can observe, there was some HAM portion increase over the recent days, however, it was caused by adding new sampling logic and duplicates filtering to save CPU resources (these messages are marked as ham and excepted from scan).

There is also a Clickhouse based dashboard that’s created using Redash:

Since we have Clickhouse on board, we can do various analytics. Here is an average scan time for messages:

:) select avg(ScanTimeVirtual) from rspamd where Date=today(); SELECT avg(ScanTimeVirtual) FROM rspamd WHERE Date = today() ┌─avg(ScanTimeVirtual)─┐ │ 95.62269064131341 │ └──────────────────────┘

… and average size of messages:

:) select median(ScanTimeVirtual) from rspamd where Date=today(); :) select avg(Size) from rspamd where Date=today(); SELECT avg(Size) FROM rspamd WHERE Date = today() ┌──────────avg(Size)─┐ │ 1778.31 │ └────────────────────┘

Conclusions

So with this load rate (1500 messages per second) and with the average size of messages around 2Kb, Rspamd processes each message in around 100ms in average. I hope these numbers could give one some impression about Rspamd performance in general.

I would like to give the main kudos to Abusix who are constantly supporting Rspamd project and have generously provided their amazing spam feeds to improve Rspamd quality!

Rspamd 1.9.3 has been released

2019-05-13 00:00:00 +0200

We have released Rspamd 1.9.3 today.

This release contains some new features and many bug fixes. There are no incompatible changes introduced with this release to our best knowledge.

This release includes the following features and important changes.

Hashicorp Vault support

From version 1.9.3, Rspamd can use Hashicorp Vault to store and manage DKIM keys. Vault usage provides secure and flexible storage of the private keys that can scale and use various backends to store sensible data (secrets).

There is a new subcommand for rspamadm utility called vault that is intended to create/remove and securely rotate DKIM private keys using vault.

You can read more about it using the following link: https://rspamd.com/doc/modules/dkim_signing.html#dkim-signing-using-vault

Added least passthrough result

Some modules should set metric result as least possible result. For example, DMARC policy failure should at least mark failed messages as spam but it should not prevent messages from being rejected. From this release, such modules use least policy to set actions allowing to apply a more strict policy if needed.

Tunable memory management

From this version, Rspamd allows to manage memory policies for Lua garbage collection allowing to fit memory/cpu constraints more flexible. When Rspamd is built with jemalloc (e.g. in the default packages provided by the project), it can also print detailed memory statistics on full gc loops.

Here is an example for tuning GC in Rspamd when there are lots of free memory available (around 1Gb per scanner process):

# local.d/options.inc
# http://pgl.yoyo.org/luai/i/2.10+Garbage+Collection
lua_gc_step = 100;
lua_gc_pause = 400;
# number of scanned messages to perform full GC iteration 
full_gc_iters = 10000;

Improved oversigning logic

It is now possible to oversign existing only headers and ignore it if a header is missing. It is done by changing (o) to (x).

The default list of headers signed is changed accordingly:

Header Sign type
From Strictly oversign
Sender Conditionally oversign
Reply-To Strictly oversign
Subject Strictly oversign
Date Conditionally oversign
Message-Id Conditionally oversign
To Strictly oversign
Cc Strictly oversign
Mime-Version Conditionally oversign
Content-Type Conditionally oversign
Content-Transfer-Encoding Conditionally oversign
Resent-To Do not oversign
Resent-Cc Do not oversign
Resent-From Do not oversign
Resent-Sender Do not oversign
Resent-Message-Id Do not oversign
In-Reply-To Conditionally oversign
References Conditionally oversign
List-Id Do not oversign
List-Help Do not oversign
List-Owner Do not oversign
List-Unsubscribe Do not oversign
List-Subscribe Do not oversign
List-Post Do not oversign
Openpgp Conditionally oversign
Autocrypt Conditionally oversign

Important bugs fixes

Here is the list of the most important bugs fixes:

  • HTML: Fix size attribute processing - this issue caused rule MANY_INVISIBLE_PARTS to be improperly triggered on many HTML messages
  • Do not blacklist mail by SPF/DMARC for local/authed users
  • Lots of Clickhouse plugin fixes
  • Fix buffer overflow when printing small floats - this issue caused random crashes in WebUI reported by many users
  • Fix DoS caused by bug in glib - details in https://gitlab.gnome.org/GNOME/glib/issues/1775

Full list of the meaningful changes

  • [Conf] Add IP_SCORE_FREEMAIL composite rule
  • [Feature] Add cryptobox method to generate dkim keypairs
  • [Feature] Add fast hashes to lua cryptobox hash
  • [Feature] Add least passthrough results
  • [Feature] Allow oversign if exists mode
  • [Feature] Clickhouse: Modernise table initial schema
  • [Feature] Implement IUF interface for specific fast hashes
  • [Feature] Lua_util: Allow to obfuscate different fields
  • [Feature] Tune memory management in Rspamd and Lua
  • [Fix] Avoid buffer overflow when printing long lua strings
  • [Fix] Change the default oversigning headers to a more sane list
  • [Fix] Clickhouse: Do not store digest as it is not needed now
  • [Fix] Clickhouse: Fix lots of storage issues
  • [Fix] Clickhouse: Support custom actions
  • [Fix] Deny URLs where hostname is bogus
  • [Fix] Do not blacklist mail by SPF/DMARC for local/authed users
  • [Fix] Fix DoS caused by bug in glib
  • [Fix] Fix UCL parsing of the multiline strings
  • [Fix] Fix buffer overflow when printing small floats
  • [Fix] Fix init code for servers keypairs cache
  • [Fix] Fix issue with urls with no tld (e.g. IP)
  • [Fix] Fix memory in arc signing logic
  • [Fix] Fix memory leak in language detector during reloads
  • [Fix] Fix mixed case content type processing
  • [Fix] Fix processing of the ip urls in file
  • [Fix] Fix use after free
  • [Fix] HTML: Fix size attribute processing
  • [Fix] Hum, it seems that 99ff1c8 was not correct
  • [Fix] Lua_task: Fix task:get_from method
  • [Fix] Preserve fd when mapping file to scan
  • [Fix] Re-use milter_headers settings when doing arc signing
  • [Fix] Set dmarc force action as least action
  • [Fix] Switch to GMT
  • [Fix] allow PKCS7 signatures to be text/plain, too
  • [Project] Add initial version of the vault management tool
  • [Project] Add vault support for DKIM and ARC signing
  • [Project] Implement keys rotation in the vault
  • [Project] Improve dkim keys generation for vault
  • [Project] Improve keys creation in rspamadm vault
  • [Rework] Move lua_worker to a dedicated unit
  • [WebUI] Add URL fragments (#) support
  • [WebUI] Fix AJAX request URL

Rspamd 1.9.2 has been released

2019-04-16 00:00:00 +0200

We have released Rspamd 1.9.2 today.

This release contains some new features and bug fixes. The only potentially slashing changes are the changes in Clickhouse module:

  • Times are now stored in GMT timezone so you can use Clickhouse for analytics that crosses time zones. The potential drawback is the mess with the currently stored data. This should be resolved automatically once new data arrives.

  • Clickhouse schema has been updated to the version 4 with new fields and some minor changes. The existing database should be converted automatically and there are no incompatible changes in columns.

This release includes the following features.

Improvements in Clickhouse plugin

Rspamd now stores more data in Clickhouse:

  • Mime recipients
  • Message IDs
  • Scan time for a message, both normal and virtual
  • SPF checks results
  • Some new calculated columns, such as MIMERcpt, MIMEFrom, SMTPFrom and SMTPRcpt

These columns are intended to improve analytical capabilities of Clickhouse plugin.

OpenDKIM compatible DKIM signing setup

This version now includes a simplified DKIM signing setup option inspired with OpenDKIM.

You can read more about it here: https://rspamd.com/doc/modules/dkim_signing.html#use-of-signing_table

This mode is intended to simplify migration from the existing setups based on OpenDKIM to Rspamd.

Better encrypted archives support

Rspamd can now properly detect encryption in ZIP archives. Mime types plugin now also tries to resolve hex encoding hack used by some spammers to send malware to users (see PR 2582).

Calendar files parser

From the version 1.9.2, Rspamd can extract meaningful data from Calendar files in iCal format (.ics files). These files are sometimes used by spammers so Rspamd can now extract hyperlinks and emails from calendar attachments to improve filtering quality.

New rspamadm dns_tool utility

It is now possible to do some DNS checks with the new tool. For example, it is now possible to verify SPF records as they are observed by Rspamd, including elements extraction, for example a or mx and verification of the IP addresses. Here is how it looks like:

Better bitcoin addresses detection

We have improved bitcoin addresses detection by fixing some issues in the BTC wallet validation code. It now allows to catch Pay-To-Script addresses.

Full list of the meaningful changes

  • [Conf] Allow to load users plugins from plugins.d
  • [Conf] oversign openpgp and autocrypt headers
  • [Feature] Add SPF FFI library for Lua
  • [Feature] Add more verbosity for SPF caching
  • [Feature] Antivirus: Handle encrypted files specially
  • [Feature] Clickhouse: Slashing - add new fields to CH
  • [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table
  • [Feature] Dkim_signing: Allow to use new options as maps
  • [Feature] Import fpconv library
  • [Feature] Lua_maps: Allow static regexp and glob maps
  • [Feature] Parse ical files
  • [Feature] Rspamadm: Add dns_tool utility
  • [Feature] Store SPF records digests
  • [Feature] Use fpconv girsu2 implementation for printing floats
  • [Fix] Clickhouse: Use integer seconds when inserting rows
  • [Fix] Fix floating point printing
  • [Fix] Fix processing of embedded urls
  • [Fix] Lua_clickhouse: Fix CH errors processing
  • [Fix] Make spf digest stable
  • [Fix] Properly detect encrypted files in zip archives
  • [Fix] Slashing: Store times in GMT timezone in ClickHouse
  • [Rules] Add additional conditions to perform BTC checks
  • [Rules] Fix pay-to-hash addresses validation