List of changes

Remove Rambler email bl for now

Switch RAMBLER_URIBL to a locally managed source

Switch from ragel to C for Content-Type parsing

Add `-e` option for lua_repl

Add per-domain emails normalisation rules

Add sessions cache to debug dangling sessions

Add short_text_direct_hash for fuzzy check module

Add text_part:get_stats function

Allow to add custom processing script for surbl

Allow to check reply-to email

Allow to customize spam header, remove existing spam headers

Allow to disable specific workers in the config

Allow to discard messages instead of rejection

Allow to specify custom delimiter in emails plugin

Allow to specify custom User-Agent for rspamc

Allow to store symbols data in Clickhouse

Allow to use HTTPS when connecting to Clickhouse

Enable sessions cache tracking for milter connections

Implement per-line mode in lua_repl (like `perl -p`)

Implement rdns-curve plugin based on rspamd cryptobox

Improve maps cached data lifetime

Improve maps checking frequency

Improve monitored timeouts logic

milter_headers: add `extended_headers_rcpt` option

Milter headers: Add X-Spam-Flag to rmilter-compatibility headers

Milter headers: remove-header routine

Multimap: received filters for extracting TLDs from hostnames

Normalize email aliases in emails module

Re-add rambler email bl (as hashed list)

Reload file maps more frequently

Rework newlines strip parser one more time

Skip updates for messages scanned via controller

Split long DKIM public keys

Store more data when stripping newlines

Support SPF macros transformations

Support suppressing DMARC reports for some domains

Add missing `break` statement

Allow modifiers in SPF macros

DKIM sign tools: edge-cases around use_esld

Do not cache SPF records with macros

Do not overwrite score when setting pre-action

Fix comparision logic

Fix DKIM base64 folding for milter flagged messages

Fix emails module configuration

Fix folding for arc headers when milter interface is used

Fix gmail dots removal

Fix rspamc detection in greylist module

Fix some more issues with HTTP maps

Milter sessions can live forever

Normalize fuzzy probability better

Plug memory leak

RBL: Fixed hashed email address lookups

Try to deal with brain-damaged milter behaviour

Use `\n` to fold headers for milter

Allow to use custom callback for monitored checks

Further steps towards one process monitoring

Send health checks from a single worker

Allow to init resolver without rspamd_config

Do not crash when resolver failed to initialize

Fix abstract context layout

Fix CGP helper reply parsing

Fix crashes when socket write errors occur

Fix parsing IPv6 nameservers in resolv.conf

Milter: Don't defer on "greylist" action

Add rspamd_proxy to the default configuration set

Add sample arc module config

Do away with systemd specifics completely

Increase min_bytes to avoid FP

Remove ratelimits from default configuration

Fix accepting on IPv6 sockets

Fix corruption when multiple fuzzy are defined

Fix learn condition in fuzzy check

Fix memory leak in fuzzy check

Fix memory leak in maps scheduling

Paese the last character in DKIM signature correctly

Zero fill sockaddr_un

Add ability to add doc strings by example

Add API to verify DKIM (and ARC) signatures

Add compression/decompression to proxy

Add count to url structure

Add initial support of the new protocol reply

Add Lua plugin spamtrap

Add `monitored_address` for rbls

Add new schema for bayes tokens

Add preliminary ARC support to dkim code

Add preliminary support of ARC signing

Add rules to detect bad 8bit characters in From and To

Add scanning support for milter protocol

Add support for bidirectional symbols in rspamd_stats

Add support for static maps

Add support of maps with multiple regexps matches

Add `text_multiplier` param

Add the preliminary ARC plugin

Add top redirector targets rank

Allow async events to be registered from LUA rules

Allow storing bayes tokens in Redis

Allow to exclude specific domains from mx check

Allow to have a stack of watcher finalisers

Allow to pass hostname to `-i` flag in Rspamc

Allow to set custom user agent in url redirector

Allow to use custom callback when parsing resolv.conf

Allow to use domain from authenticated user

Bayes expiry plugin

Check dkim sign keys for modifications

DKIM signing: sign_networks/local address specific use_domain settings

DMARC: Support excluding domains from sampling

Expire processing items for URL redirector aggressively

Fix surbl monitored for IP lists, add `monitored_domain` option

Implement caching for dkim body hashes

Implement milter protocol scan reply

Improve omograph phishing detection

Initial support of self-scan in Rspamd proxy

Keep track of headers in milter interface

Milter headers: better controls for local/authenticated

Multimap: email:domain:tld filter

Preliminary DMARC reporting implementation

Reuse stemmers in the cache

Rework confighelp to load Lua plugins

Rework hfilter to use hyperscan if possible

Rework lua RSA API

Rmilter_headers: approximate rmilter's extended_spam_headers

Start integration of milter support in proxy

Store average words length and short words count

Store hash of headers order and names

Support MTA name header

Support multiple types of dkim signing in Lua

Support numeric arguments for Redis requests

Use headers hash in bayes metatokens

Use normal resolv.conf rules of rotation in Rspamd

Use version 2 proto for checking messages

Allow to follow symlinks when safe

Append MX name for authentication results as required

Change default text multiplier from 0.5 to 2.0

Check min_bytes for images as well

Deal with 7bit charsets properly

Deal with 8bit characters in email addresses

Deal with unpaired <a> tags

Detect confighelp in plugins initialisation

Disable certain checks for utf spoof detection

DKIM Signing: avoid nil index when From header is missing

Do not add exact hashes from different parts

Do not check DMARC if SPF or DKIM were not checked

Do not check URLs that are resolved to be redirected

Do not set bayes probability if we don't use it

Do not stop on illegal unicode points - replace them

Fix another race condition in arc checks

Fix arc count logic

Fix ARC signing

Fix brain-damaged spamc protocol for now

Fix calling for peak functions

Fix couple of issues in FORWARDED rule

Fix CTE propagation from parent containers to children parts

Fix errors processing in the controller

Fix format string in milter

Fix issues in SPF macros parsing

Fix logging format string

Fix logic of cached passwords check

Fix lowercasing of stemmed words

Fix LRU elements removal

Fix memory leak when accepting from unix sockets

Fix milter connections persistence

Fix objects merging in UCL

Fix order of operations to avoid race condition

Fix parsing of long regexp types

Fix passing data to log helper when many symbols defined

Fix pools management for milter session

Fix processing of the watchers

Fix queue id macro in milter

Fix R_BAD_CTE_7BIT rule

Fix Redis timeout set

Fix REPLYTO_UNPARSEABLE rule

Fix setting of email address

Fix some more issues about duplicated fuzzy requests

Fix spamc support in rspamd proxy

Fix syntax error in spamtrap plugin

Fix url counts for href urls

Fix url handling in the protocol

Multimap: Received IP filters with Redis

Oops, fix d9d0fa5e86db2f4470d34395a233b450478b2f60

Parse rgb[a](x,x,x[,x]) css colors

Phishing: strict_domains

Reduce maps aggressiveness

Reresolve upstreams even if there is a single server there

Rspamadm grep: Disable Lua patterns in string search by default

Skip text parts when checking binary parts in fuzzy check

Support v2 checks in controller

Treat empty address as valid

Try harder to detect CTE

Try to deal with v4 mapped to v6 addresses on accept

Use dkim signing callback properly

Use non-volatile memory for storing data

Use static maps instead of ugly hack for radix_from_config

Use the same pool for related sessions

Continue modularisation for lua library

Initial milter protocol support

Make log pipes worker agnostic, add scanners API

Move authentication results generation to a separate routine

Move common DKIM functions to a separate lua module

Move global functions to a separate directory

Prepare dkim module for ARC checks

Propagate ucl variables from the command line

Remove multiple metrics support from Rspamd

Stop using name 'rmilter' for the modern protocol

Use LFU algorithm in LRU cache

Fix received TLS rules

Improve URL_COUNT_ODD rule

Increase min_bytes to avoid FP

Remove ratelimits from default configuration

Fix accepting on IPv6 sockets

Zero fill sockaddr_un

Add `text_multiplier` param

Check min_bytes for images as well

Do not add exact hashes from different parts

Fix memory leak when accepting from unix sockets

Fix some more issues about duplicated fuzzy requests

Phishing: strict_domains

Skip text parts when checking binary parts in fuzzy check

Fix memory leak in fuzzy check

Fix memory leak in maps scheduling

Multimap: email:domain:tld filter

DKIM Signing: avoid nil index when From header is missing

Do not set bayes probability if we don't use it

Do not stop on illegal unicode points - replace them

Fix brain-damaged spamc protocol for now

Fix Redis timeout set

Fix spamc support in rspamd proxy

Multimap: Received IP filters with Redis

Parse rgb[a](x,x,x[,x]) css colors

Reresolve upstreams even if there is a single server there

Treat empty address as valid

Try harder to detect CTE

Try to deal with v4 mapped to v6 addresses on accept

Fix corruption when multiple fuzzy are defined

Fix learn condition in fuzzy check

Add rules to detect bad 8bit characters in From and To

DKIM signing: sign_networks/local address specific use_domain settings

Support numeric arguments for Redis requests

Deal with 8bit characters in email addresses

Fix couple of issues in FORWARDED rule

Fix passing data to log helper when many symbols defined

Fix R_BAD_CTE_7BIT rule

Fix REPLYTO_UNPARSEABLE rule

Fix setting of email address

Rspamadm grep: Disable Lua patterns in string search by default

Add unigramms support in bayes

Allow configurable sign headers for DKIM

Allow to add unigramm metatokens from Lua

DKIM Signing: envelope match exception for local IPs

UCL: register parser variables from Lua

Always try to adjust filename

Do extra copy to ensure that original content is never touched

Fix SPOOF_REPLYTO rule

Ignore Rmilter added Received

More fixes for hashed email dnsbls

Plug memory leak in chartable module

Fix classifier learning with Redis backend

Fix issue when parsing encoded rfc822/messages

Add escaped version of lua_ucl import

Add task:headers_foreach function

Allow to process filenames from content type

Allow to query hashed emails

Ignore bayes with mostly metatokens or with too few text

Probabilistically skip metatokens

Retrieve all virus names from SAVAPI

Rework classifiers lua metatokens

Store headers order

Store text tokens inside bayes tokens

Use cached shingles keys

Add missing score normalisation for HFILTER_URL_ONLY

Avoid lookup in absent hash

Check return values from Lua functions called from C

Do not count sending and loading time in rspamc

Escape json strings for controller rejplies from Lua

Fix archive scans for savapi

Fix domain_only emails RBL

Fix ip_score map configuration

Fix JSON output for history_redis

Fix one character length substrings search

Fix parsing of non-RFC compatible Exim received

Fix parsing of options for workers with the same type

Fix processing of small tokens vectors

Fix rfc2047 tokenization

Fix typo

More fixes for inplace decoding

Try to avoid modifications of the original data

URL redirector: Fix call to is_redirector

Set token data as uint64_t instead of chars array

Add history_redis default configuration

Add spoofed rules

Add URL_IN_SUBJECT rule

Allow to get task's subject

Allow to specify maximum number of shots for symbols

Distinguish URLs found in Subject

Memoize LPEG grammars

Parse else parts in SA rules

Process subject for mixed characters

Resolve url chains in url_redirector module

Stat greylisted messages as greylisted not soft-rejected

Support checking for redirector in Lua SURBL

Support tag_exists SA function

Work with broken rfc2047 tokens

Check all watcher's dependencies

Do not compile hyperscan with no SSSE3 support

Do not crash if cannot decode qp encoded part

Fix dependencies of DKIM when multiple signatures are found

Fix lists in whitelist plugin

Fix one-shot symbols weight calculations

Fix options and shots match

Fix order of symbol options

Fix parsing of dot at the end of the address

Fix parsing of lua table arguments

Fix processing of subject words

Fix string split memoization

Fix templates grammar usage

Fix various issues related to Lua stack manipulation

Force actions: Use postfilter if we have honor_action / require_action

Further fixes to avoid PHISHING FP

Preserve order of options in symbols

Rspamadm grep: deal with unusually-formatted logs

Use hostname suffix when dealing with history

Remove outdated SA rules

Add composite for hacked wordpress phishing

Fix base64 decoding when there are unparseable characters

Additional symbol metadata in metadata exporter

Add method to get protocol reply from Lua

Add symbols when tagged rcpt/sender are normalised

Add task:get_symbols_all() function

Allow multiple formats of DKIM signing key

Allow to cache and use flexible protocol reply

Allow to set one_shot flag from register_symbol

Allow to skip certain types of hashes when learning fuzzy

Cache and insert scan time into the protocol

Detect newlines in rspamc --mime

DKIM signing: support use of maps

Greylist: Support excluding low-scoring messages from greylisting

Implement lua history in controller

Implement redis history querying

Preliminary implementation of redis history plugin

Support using request headers in settings

Change default template to deal with non-ASCII characters

Deal with lists of maps in whitelist module

DKIM signing: use domain-specific signing key

Do not reallocate completed zstd buffer

Do not use local_addrs in proxy

Fix crash when resolver is undefined

Fix double free when closing lua_tcp connections

Fix for lua 5.3

Fix freeing of arrays iterators

Fix issue with task:get_symbol and symbols with no metric

Fix log line duplication in `rspamadm grep`

Fix memory corruption on termination

Fix out-of-bound access in base64 decode

Fix ratelimit + greylisting

Fix subject rewriting

Fix task:set_recipients function

Fix URI_COUNT_ODD rule

Follow the traditional symbols conventions in RCPT_COUNT rule

Greylist: Suppress greylist action for whitelisted hosts too

Metadata exporter: use rule-specific settings for emails

Properly set missing fields in exporter

Proxy: max_retries option

RCPT_COUNT fixes

Rework HAS_X_PRIO rule to match symbols conventions

Update issues in ac-trie

Use optimised base64 decoding in DKIM

Add default config for spamassasssin plugin

Add default configuration for antivirus module

Add dkim signing docs

Add mx_check default config

Add replies config

Add trie default config

Add heuristic to find text parts in files

Add rule to detect broken content type

Allow to extract CTE in Lua API

Allow to set from address for a lua_task

Allow to set recipients of a task from Lua

Enchance text_part:get_content method

Remove + aliases from emails

Support rmilter block and dkim signature in CGP helper

Support running event loop from Lua

Antivirus: use scanner-specific redis prefix

Couple of fixes for DKIM signing module

Distinguish missing and broken mandatory headers

Do more heuristical detection for missing CTE

Do not resort cache on each check

Fix CGP escaping

Fix MISSING_MIME_VERSION rule for plain messages

Fix parsing of cte in expressions

Fix partial matches in rspamadm grep

Fix setting class on style field

Fix processing of stop_patterns with `\0` character

Fix setting of raw key for signing

Fix lua exports from plugins during reload

Fix prefilters action scores

Fix symbols processing order

Add configurations for asn, clickhouse and dcc

Add default config for url redirector plugin

Add the default config for greylist module

Allow to edit all local maps from WebUI by default

Deal with absent headers in DKIM

Do not trust remote shingles count

Fix bad memory leak in TLS certificates validation

Fix critical memory issues with radix maps

Fix descriptors leak on reload

Fix headers selection in DKIM verification

Fix parsing of boundaries that end with `--`

Repair PTR_ARRAY_FOREACH macro

Add CORS support to the controller

Add FROM_NAME_EXCESS_SPACE rule

Add REPLYTO_EMAIL_HAS_TITLE rule

Add `caseless_hash` method to `lua_util`

Add `rip` keyword to ratelimit module

Add a simple benchmark for content type parsing

Add boundaries parsing in content type

Add charset detection for text parts

Add content disposition parser

Add fallback if too many updates are failing

Add function to convert struct tm to time using timezone

Add function to normalize HTTP paths

Add fuzzy collection plugin

Add fuzzy logic for images

Add gmime parser to mime_tool

Add heuristic to detect broken messages

Add heuristic to find displayed URLs

Add heuristic to process broken email addresses

Add images normalization

Add mechanism for disabling composites (Fixes #1270)

Add method to create regexp from a glob pattern

Add mime encoding manipulation routines

Add mime tool to explore messages

Add more meta tokens from received headers

Add neighbours option to support Rspamd cluster in WebUI

Add new function to parse mime addresses

Add new methods for lua_tcp

Add own headers decoding routine

Add own routine to generate a message id

Add parser for SMTP date

Add per-task lua cache to reuse 'heavy' objects

Add plugins list path in WebUI

Add preliminary multipart support

Add preliminary version of DKIM signing module

Add profiling support in client output

Add rfc2047 grammar

Add rfc2047 variant for QP decoding

Add rmilter_headers module (Fixes #1227)

Add sse42 version of base64 decoding

Add ssse3 and avx2 base64 decoders

Add support of libgd

Add the preliminary version of redirects resolver in Lua

Add ucl_object_iterate_full function

Add url encoding function

Allow SOA requests in lua dns

Allow custom parse types in lua ucl

Allow plugins to register webui handlers

Allow to add options explicitly to symbols

Allow to call a callback when symbol frequency is on peak

Allow to call redirector script from SURBL

Allow to create variable length dkim keys

Allow to have module specific options for Redis in plugins

Allow to pass sign key directly from Lua

Allow to register configuration docs from Lua API

Allow to return options as a table

Allow to set peak callbacks from Lua

Allow to specify custom method for a message

Allow to store dkim keys in Redis

Allow to store messages in files

Apply DCT using AAN for fuzzy signature

Avira SAVAPI support

Cache and simplify DCT and jpeg decode

Cache libicu converters

Detect URLs with suspicious omographs

Do not increase score for duplicate options

Do not trust CTE, check base64 and qp strictly

Dynamic reputation in URL reputation plugin

Extend redis lock when learning spawned

Filter non-utf chars from all decoded headers

Fix phishing detection for IDNA urls

Ignore bad symbols on base64 decoding

Ignore too wide elements in SPF

Implement fuzzy collection mode

Implement helo maps in multimap

Implement human readable buckets configuration

Implement min-hash shingles for DCT data from images

Implement new algorithm for fuzzy hashes of images

Implement new unicode normalizer

Implement quoted printable decoding

Implement received headers flags

Implement rspamdgrep tool

Implement sane checksum for config file

Implement url tags concept

Improve detection of omographs using libicu

Improve url redirector module

Multimap: Received header processing

Multiple improvements in the maps

New URL filters in multimap

Plugin to force actions on selected symbols

RBL module: support hashing for emails and helo RBL

Reuse URL tags in SURBL module

Rework RRD ds count, add conversion path

Rework surbl module to avoid extra redirector calls

Send config id to the WebUI

Simplify HTTPCrypt client support

Skip processing for large images

Start collection only mode implementation for fuzzy storage

Start import of the optimized base64 decode

Store all received headers in lua

Store relational order of all headers in a message

Support DKIM signing in Lua plugins

Support HTTPCrypt client in lua_http

Support setting SMTP message in multimap

Support setting metric subject from Lua

Support setting subject in force actions module

Treat v6 mapped addresses as v4 addresses

URL reputation plugin

Use Redis instead of memcached in URLs redirector

Use Rspamd rfc2047 decoder instead of gmime one

Use a different normalization for fuzzy images

Use normalized images in fuzzy hashes

Use own code for parsing of date

Use shingles for images fuzzying

Use t1ha for hashes, allow inlining

Use t1ha instead of metrohash and xxhash32

Various new features in metadata exporter module

rmilter_headers: authentication-results (#78)

Add additional check to mark redis connection inactive

Add packed attribute for protocol structure

Adopt OMOGRAPH_URL rule

Allow static maps

Allow to disable classifiers checks using settings and conditions

Another try to fix 0 length maps

Another try to fix corruption during maps reload

Another try to fix descriptors leak

Another try to fix reload and logger

Antivirus module: register virtual symbols for patterns

Avoid extensive reallocs

Avoid mempool leak in SA plugin on reload

Avoid race condition on saving cache and reload

Avoid reusing g_error (Fixes #1262)

Break pool connection on fatal redis errors

Check for NaN properly

Couple of fixes for date parsing

Date header timezone adjustments (#1279)

Deal with EOF properly

Decode filename in content disposition

Disable fuzzy images by default

Disable zero-copy mode for text parts to avoid crashes

Do not destroy session when not all finish scripts are done

Do not greyscale images

Do not leave parent-less workers processes on fatal errors

Do not lowercase Content-Disposition to perform decoding

Do not penalize characters just after numeric prefix

Do not refork workers that are intended to die

Do not set pre-result and update records for no `Queue-ID` messages

Do not skip post-filters when pre-filters have set some results

Do not stop symbols planning if async events are pending

Do not try to set keys for unencrypted requests in proxy

Encode URLs according to rfc3986

Encode URLs before sending them to the protocol

Filter bad characters from message id

Fix CTE detection heuristic

Fix Content-Type in HTTP requests

Fix IDN eslds phishing checks

Fix adding maps from config in Lua

Fix another reload memory issue

Fix argument returned on redis backend errors

Fix assertion in graph handling

Fix body trie matching

Fix build

Fix byte array expansion during toutf8 conversion

Fix charset normalisation

Fix checking of DKIM bodies that needs just `\n` to be added

Fix couple of cornercases with email addresses

Fix couple of issues

Fix dependencies tracking for callback symbols

Fix detection of jpeg size

Fix errors handling in fuzzy backend initialization

Fix fuzzy hashes count

Fix globbing and convert lists to arrays in fuzzy_check

Fix heuristical CTE detection for QP encoding

Fix ignoring of bad text parts

Fix indexes in array access, interleave loop

Fix int64 -> double conversion

Fix invalid memory access on reload

Fix issues with empty updates

Fix issues with quoted-printable encoding

Fix keys names

Fix lots of issues in mime parser code

Fix lua maps load

Fix macro name

Fix mas group score calculations

Fix matching of the same patterns from different tries

Fix memory corruprtion and leak

Fix memory leak in HTTP maps

Fix memory leak in expression destroying

Fix memory leak in parsing of mime names

Fix memory leak in safe ucl iterators

Fix memory leak on reload in plugins

Fix modules reconfigure on reload

Fix monitored setup fro URLBLs with IP addresses

Fix name of var

Fix new rrd updates

Fix out of bounds access

Fix parsing messages with no body

Fix parsing of '=' character in headers

Fix parsing of messages with no content type

Fix plugins callbacks in webui

Fix possible memory corruption in redis pool

Fix probability calculations for fuzzy redis backend

Fix processing errors in lua_tcp

Fix processing of emails with name only

Fix processing of non-multipart messages

Fix processing of parts with no valid content type

Fix race condition in SIGUSR2 handler

Fix redis options parsing when no redis servers are defined

Fix reload and hyperscan ready event

Fix reload memory issue

Fix rra_ptr conversion

Fix rrd file conversion

Fix setting of content-type attributes

Fix signing headers creation in DKIM

Fix stddev calculations

Fix surbl plugin to work with composite maps

Fix timezones parsing

Fix tokens usage

Fix urls and emails hashes

Fix usage of unsafe ucl iterators

Fix work with broken utf8 tokens

Fix writing of user to roll history

Forgotten worker

Further memory leaks fixes

Ignore lua metatokens in bayes for now

Improve OMOGRAPH_URL rule

Lua IP from string should be invalid if parsing failed

Miltiple fixes to new lua_tcp, add debugging

More fixes for iterators cleanup

More fixes to logger initialization

More heuristic fixes for phishing detection

More leaks eliminated

More leaks...

More random fixes for reload...

Multimap: Fixes for email filters

Multiple fixes for fann module

Multiple memory corruption fixes

Normalize path in HTTP router

Plug memory leak

Plug memory leak in adding radix trees

Plug memory leak in configuration parser

Plug memory leak in expressions parsing during reload

Plug memory leak in learning fuzzy storage

Plug memory leak in lua_tcp

Plug reload leaks

Plug termination memory leaks

Really increase lock lifetime

Replies module: fix symbol weight

Restore content type params related functions

Set task's subject from mime subject

Sigh, one more reload leak

Simplify images shingles

Some more memory issues are fixed

Stop hardcoding of lua in C

Stop processing of bad parts as text parts

Strictly filter bad characters when emittin json

Strings returned from lua are ephemeral

Support unix sockets for lua redis

Try to fix issues with reloading config

Try to fix race condition in redis_pool

Use checksum to avoid intersection between different ANNs

Use rspamd hashes in embedded ucl

Use sane default rewrite subject (*** SPAM *** %s)

Various collection mode fixes

Various fixes to mime parser

Various reload leak fixing

Whitelist certain extensions from archive checks

Add preliminary implementation of the mime parser

Adopt code for the new options

Change logger setup interface

Composite configuration (#1270)

Finally remove gmime dependency from Rspamd

Further fixes to symbols frequencies

Implement content type parser for mime

Kill all InternetAddressList usages

Multiple fixes for symbols cache statistics

Refactor struct names

Rework images fuzzy hashes algorithm

Rework lua_tcp to allow TCP dialog

Start massive rework to get rid of gmime

Start new approach for multiparts parsing

Start rework of mime addresses

Start rework of symbols cache updates

Start switching to libicu

Use a special structure for stats tokens

Use hash tables for symbols options

Use libicu instead of iconv for conversions

Use new scheme to parse mime parts

Deal with absent headers in DKIM

Do not trust remote shingles count

Fix headers selection in DKIM verification

Add EXT_CSS rule

Add toggle for disabling SURBLs

Extend redis lock when learning spawned

Parse <link> HTML tags

Avoid reusing g_error (Fixes #1262)

Do not reset loaded ANN when learning is requested

Fix another issue with external deps in SA

Fix body trie matching

Fix checking of DKIM bodies that needs just `\n` to be added

Fix fuzzy hashes count

Fix keys names

Fix length calculations for url encoded urls

Fix matching of the same patterns from different tries

Fix name of var

Fix parsing of URLs with spaces and other bad chars

Fix probability calculations for fuzzy redis backend

Fix signing headers creation in DKIM

Plug memory leak

Really fix chained SA dependencies

Really increase lock lifetime

Use checksum to avoid intersection between different ANNs

Use rspamd hashes in embedded ucl

Yet another change for testing external deps

ASN support in Clickhouse module

Add clickhouse plugin

Add generic tool to add universal maps for lua modules

Add logger.debugm to debug lua modules

Allow to register metrics symbols using register_symbol

Allow to specify prefix for fann_redis

Clickhouse: support different masks for IPv4/IPv6

Support forcing action in antivirus plugin

Add handling of regexp maps

Allow backslashes in http urls

Avoid mapping of empty files

Do not load tld file to speed up rspamadm

Do not resolve numeric IP addresses due to ipv6 insanity

Filter incorrect training data

Fix Fuzzyconvert tool when password or DB is given

Fix build with custom glib/gmime

Fix converting of learn count from sqlite to redis

Fix crashes with invalid received and task:set_from_ip

Fix external dependencies for SA module

Fix fann_redis when number of scores has been changed

Fix hyperscan usage for non compatible platforms

Fix loading of maps from UCL objects

Fix memory leak for task-less redis requests

Fix mid module with new maps syntax

Fix parsing of URLs with username

Fix re cache initialisation

Fix replacements to sanitize '%' character

Fix set and regexp like static maps

Fix some issues in redis settings

Fix static IP maps

Fix total learns counter for redis stats

Fix usage of config during reload

Fix various warnings and issues

Invalidate ANN if training data is incorrect

Miltiple fixes to fann_redis module

More fixes for URLs with backslashes

Properly get options for ip_score module

Relax requirements for Received as gmail cannot RFC

Remove or fix hyperscan incompatible regexps

Settings: correctly read redis config

Rework lua logger interface slightly

Use new maps add function

Add guards for inactive redis connections

Another fix for proxying files using rspamd_proxy

Cleanup inactive redis connections

Do not sometimes try to exec posfilters before classification

Fix application of IPv6 mask

Fix chunked encoding when reading messages

Fix file mode for rspamd_proxy

Fix hyperscan compilation on regexp change

Fix issue with finding of end of lines pointers

Fix iteration over headers array (introduced in 1.4)

Fix processing of learned tokens count for redis backend

Fix race condition in checking of cached maps

Fix workers scripts by sharing workers configs

Introduce raw content to text parts

Plug memory leak and potential memory corruption

Adaptive ratelimits

Add ASN -> rbldnsd script for asn.rspamd.com

Add DMARC_NA symbol

Add F-Prot support to antivirus module

Add HTTP backend to metadata exporter

Add Lua API module for monitored objects

Add R_DKIM_NA / R_SPF_NA / AUTH_NA symbols

Add R_DKIM_PERMFAIL symbol

Add R_SPF_PERMFAIL symbol

Add Sophos antivirus support

Add ZSTD compression to Lua API

Add `mid` Lua module

Add `one_param` flag for metric symbols

Add a generic lua classifier

Add a very basic interface to access workers data from on_load

Add ability to delete a hash by its data to fuzzy_check plugin

Add ability to enable/disable symbols via dynamic_conf

Add ability to lookup settings by key

Add common way to disable Lua modules

Add compression support to rspamd client

Add condition to do antiviral check

Add configuration for lua classifiers

Add configuration knobs for the errors circular buffer

Add decompression support in rspamd client

Add errors exporter to the controller

Add expected value for monitored DNS resources

Add exporter from error ringbuf to ucl

Add extended version for fann creation function

Add ffi friendly version of process_regexp function

Add frequency and time display to webui

Add fuzzy_delhash command to rspamc client

Add implementation of redis connections pool

Add latency and offline time monitoring

Add learning support for lua classifiers

Add max-size and timeout options to CGP helper

Add method to enable/disable symbols in config

Add methods to get metric's actions and symbols from Lua

Add mmap support to lua_text

Add monitored object for surbl plugin

Add more exceptions to surbl whitelist

Add more meta-tokens to bayes

Add neural net classifier to fann_scores module

Add neural net serialization/deserialization

Add new dynamic conf module

Add periodic events support for lua_config

Add plugin to check MX'es for the sender's domain

Add preliminary monitored module

Add preliminary support of dynamic conf updates in Redis

Add preliminary version of clamav plugin

Add redis cache to asn module

Add replies compression

Add spamhaus DROP dnsbl

Add support for dictionary in client compression

Add support for fuzzy learn and unlearn from lua

Add support for input encryption

Add support of min_learns to neural net classifier

Add termination callbacks for workers

Add user-agent for rspamc

Add utility to perform classifier tests

Add zstd compression library

Allow HTTPS requests in lua_http

Allow conditions for pre and postfilters

Allow custom functions for ratelimits

Allow for excluding messages from AV scanning based on size

Allow for getting worker stats from Lua

Allow getting task UID from Lua

Allow parsing of mailbox messages from the commandline

Allow plugins to publish their lua API via rspamd_plugins

Allow to compare other systems with Rspamd

Allow to execute Lua scripts by controller

Allow to have a function to set custom greylist message

Allow to iterate over multiple tags

Allow to pass extra data from plugins to log helper

Allow to plan new periodics at different time

Allow to reset hashes

Allow to run rspamadm lua just as a lua interpreter

Allow to store settings in redis

Allow to update dynamic conf in Redis

Allow to use dictionaries for compression

Allow to use md5, sha1, sha256, sha384 and sha512 hashes in Lua

Allow whitelisting by IP for greylisting plugin

Antivirus: Support whitelists & pattern-matching sig names

Backport pack/unpack routines from Lua 5.3

Check settings with equal priopities in alphabetical order

Compress neural net in redis

Consider more tags when doing WHITE_ON_WHITE rule

Descriptive options for DMARC failure symbols

Descriptive options for RBL symbols

Enable configuration for monitored objects

Execute on_load scripts with ev_base ready

Fann scores now uses metadata from a message

Implement FANN threaded learning

Implement classifying for lua classifiers

Implement finish scripts for worker processes

Implement monitoring for DNS resources

Implement real priorities for pre and post filters

Insert two symbols: FANN_HAM and FANN_SPAM instead of one

Module to push metadata/messages to redis pubsub

Monitor RBL records

Move fann_classifier to a separate plugin

Normalize all ANN inputs

Preliminary version of metric exporter module

Preserve decompression context between tasks

Ratelimit: Support dynamic bucket size/leak rate

Relax FORGED_RECIPIENTS: allow senders to BCC themselves

Remove symbols weights on composites processing

Return symbol scores when getting resulting symbols

Rework lua tcp module

Rule to detect some obvious X-PHP-Originating-Script forgeries

Rule to identify some X-PHP-Script forgeries

Rules for scoring Google Message-ID fixes

Send hashes values to reply

Set expire for dmarc reports

Stop using cymru zone as it is unstable

Stop using of GLists for headers, improve performance

Store `for` in task:get_received_headers

Store `for` part in received headers

Store enabled flag for webui session

Store error messages in ring buffer

Support compressed maps

Support excluding selected users from ratelimits

Support looking up NS records in lua_dns

Support modern style SURBL configuration

Support multiple hashes in delhash path

Support new messages in rspamc

Support requests without reads in lua_tcp

Support setting task message from Lua

Track visibility of HTML elements

Try to add CRLF when checking DKIM

Try to guess line endings when folding headers

Try to improve normalization function for bayes

Use FFI to optimize SA module

Use length based arguments for redis, allow lua_text as arg

Use more layers for fann and another normalization

User-defined ratelimits

Utility to convert fuzzy storage from sqlite to redis

Yield DMARC_DNSFAIL on lookup failure

Adopt fuzzy storage for flexible backends

Allow plain IP addresses in Rspamd maps

Another fix for brain-damaged hiredis

Another fix for rdns write errors

Another fix for rdns_make_request_full invocation

Another fix in DKIM canonicalization

Another memory leak plugged

Another try to deal with posix idiotizm

Another try to fix RDNS events processing logic

Avoid double frees in HEAD requests

Avoid extra symbols for RBLs

Banish table.maxn from Lua parts

Check for socket error before connection in lua_tcp

Correctly propagate redis timeouts to Lua

Do not add extra newline in MIME mode

Do not be cheated by system hiredis

Do not classify when a message has not enough tokens

Do not crash on redis errors

Do not distinguish NXDOMAIN and NOREC for monitored

Do not replan retransmits if merely one server is defined

Do not use headers to calculate messages digests

Don't force action in replies module for authenticated users/local networks

Explicitly ban default passwords in webui

Finally fix ambiguity between parsed and resolved spf elts

Fix 'decoded' value in task:get_header_full()

Fix DKIM calculations

Fix DKIM signing for messages with no newline at the end

Fix DNS request in monitored

Fix DNS write errors processing

Fix HTTP methods other than GET and POST

Fix PERMFAIL for v6/v4 ambiguities

Fix absurdic scores for HFILTER_URL_ONLY

Fix actions in rolling history

Fix actrie patterns

Fix applying of lua dynamic confg

Fix autolearning errors and redis cache

Fix bayes learn_condition

Fix build with the recent OpenSSL

Fix caching and compressed maps

Fix check plain text part

Fix crash on OpenBSD in `url_email_start`

Fix double free in SPF

Fix extraction of shingles from redis fuzzy storage

Fix false sharing for symbols in the cache

Fix float usage in util:get_time

Fix folding algorithm to deal with empty tokens

Fix format string

Fix format string usage in controller errors handling

Fix handling of '\0' in lua_tcp

Fix handling of HTTP HEAD methods

Fix hash creation

Fix hiredis stupidity

Fix implicit settings module settingsup

Fix interaction with lua GC to avoid craches

Fix ip_score module registration

Fix issue with empty messages and dkim

Fix issues with CGP helper

Fix issues with the recent SPF changes

Fix key name to load ANN correctly

Fix lua tcp module by saving `do_read` in callback data

Fix memory leak in client when using compression

Fix min_learns option

Fix on_finish scripts and async handlers

Fix options for SPF dnsfail symbol

Fix parsing includes and redirects in SPF

Fix parsing of lua comments with empty lines

Fix parsing of unquoted HTML attributes

Fix periodic events and redis

Fix processing of fuzzy learns from Lua

Fix processing of redirect in SPF includes

Fix processing of symbols when reject limit is reached

Fix refcounts when map is specified by IP

Fix rspamd{session} class in Lua API

Fix setting ratelimit key for 'ip' bucket

Fix some cases of TLD urls detector

Fix statconvert tool

Fix stats for backend-less classifiers

Fix training script for fann_redis

Fix variable in ann module

Fix various errors in lua dynamic conf plugin

Forget old ANN when max_usages is reached to avoid overtrain

Further canonicalization fixes

Further fixes for fann_redis prefixes

Handle failures for inactive pooled connections

Improve multimap info message

More fixes in ANN loading

More fixes to fann_redis

More issues in fann_redis

More spaces fix in DKIM signature

Multiple fixes to asn script, add IPv6 support

Multiple issues in fann_redis

No greylist rejected messages

One more attempt to fix lua_redis

One more check for readdir...

Params should be treated as a hash

Plug memory leak in regexp desctructor

Process headers only once

Properly handle nil values in ratelimit plugin

Really fix redis shingles check

Remove fann with incorrect layers count

Remove mentions of deleted include

Remove some incompatible functions

Settings: fix `authenticated` parameter (#886)

Skip MX check for authenticated users and local networks

Slightly fix ANN routines

Stop caching records with DNS failures

Treat all errors in redis_pool as fatal errors for a connection

Try avoid false-positives in HEADER_FORGED_MDN rule

Try to avoid race condition when using rrd

Try to reload redis scripts if they are missing

Unbreak once_received skipping for local networks

Unlock ANN on error

Use memmove for overlapping regions

Use real size instead of displayed for core limits

Use the correct macro to get the size of control

Various fixes for errors ringbuffer

Yield R_SPF_DNSFAIL if lookup of included record fails

mid: fix map initialization

mid: handle incorrect rgexps in the map

Add extract training data function to fann_redis

Add preliminary train tests

Add redis storage feature to fann_redis

Adopt fuzzy storage for abstract backend

Adopt plugins

First reiteration on fann scores

Implement loading/invalidating

Make lua_redis task agnostic

Make rspamd protocol messages useful

Massive removal of legacy code

More cleanup actions

Remove legacy code never used for classifiers

Remove outdated and unused lua_session module

Reorganize fuzzy backend structure

Reorganize the internal backend structure

Restore old fann_scores, move common parts

Rework and simplify rbl plugin

Rework parsing of DMARC records

ASN module; support matching ASN/country in multimap

Add SPF method in spf return result

Add Yandex and Mail.ru forwarding rules

Add mempool maps in multimap

Add rule for identifying mail sent by eval()'d PHP code

Add support of stub DNSSEC resolver to rdns

Add task:get_digest method

Allow for more fine-grained scoring for ip_score

Allow to get digest of a mime part from lua

Allow to print message digest in logs

Fold DKIM-Signature header

Implement encrypted logs

Log URLs encrypted if we have log encryption pubkey

Pass authenticated bit to lua

Read redis backend statistics configuration from global section

Show the exact value matched for multima symbols

Store task checksum

Avoid setting limits when required elements are missing

DMARC: Fix alignment checking for subdomains

DMARC: deal with missing and spurious spaces

Defer insertion of results in ip_score to avoid skewing stats

Disable DMARC for local/authorized mail

Fix handling of proxied headers in controller

Fix hex printing of strings

Fix issue with spaces in maps

Fix parsing of forwarded IP

Fix reload in some plugins and workers

Fix reloading on SIGHUP

Fix some border cases for DKIM canonicalization

Fix url maps

Make dnssec configurable option disabled by default for now

rspamadm statconvert: force db to be a string

rspamadm statconvert: use db/password for learn cache

Rework flags in rspamd logger

Check hyperscan cache sanity before loading

Fix setting of fuzzy keys (completely breaks fuzzy storage)

Add SARBL (sarbl.org) uribl

Add `--search-pattern` option to rspamd_stats

Add some sanity check for very long from/to log elements

Allow to create hashes from string in a single step

Fix order of pre and postfilters

Improve lua URLs API

Improve message about fuzzy rules

Pre-calculate blake2 digest for all parts

Print radix duplicate keys as IP addresses

Simple mechanism for disabling RBLs in local.d/rbl.conf

Use faster hash function for fuzzy storage

rspamd_stats: support log directory reading

Add sanity check for url filters

Do not show rmilter section as a fake metric in rspamc

Fix URL filters

Fix a stupid mistake in util.strequal_caseless

Fix blake2b hash of the string "rspamd"

Fix filename maps filter

Fix finding tld in util.get_tld

Fix multimap content filters

Fix returning boolean from Lua

Fix returning of REDIS_NIL

Try to deal with multiple workers terminated

Use forced DNS request when calling for lua_http

Rework multimap filters, add redis maps

Add a special symbol for SPF DNS errors: R_SPF_DNSFAIL

Add correlations report in fuzzy stats

Add experimental CGP integration

Add method to get urls length in a text part

Add new methods to lua_html to access HTML tags

Allow all types of symbols to be added via __newindex method

Allow to create settings for authenticated users

Allow to get block content for HTML tags

Improve DNS failures when dealing with SPF

Properly implement R_WHITE_ON_WHITE rule

Remove old ugly rules

Rspamc can now add dkim signature in mime mode

Store content length for HTML tags

Support reacher set of HTML colors

Try to avoid FP for low contrast fonts detection

Add missing HTML colors

Add spaces to dkim signature to allow folding

Avoid returning NaN as score on scan

Decode entitles in href parts

Do not cache SPF records with DNS errors

Do not crash on cyclic depends

Do not insert HELO/HOSTNAME unknown when they are not passed

Do not set absent hostname to "unknown"

Do not stress redis with KEYS command (#791)

Fix DMARC_BAD_POLICY symbol

Fix HFILTER_URL module

Fix HFILTER_URL_ONELINE rule

Fix buffering in CGP integration

Fix colors propagation from parent nodes

Fix confusing OpenSSL API usage of i2d_RSAPublicKey

Fix dependencies id sanity check

Fix folding for semicolon separated tokens

Fix largest possible TLD behaviour

Fix last token folding

Fix length calculations in white on white rule

Fix multiple request headers structure

Fix multiple values headers freeing

Fix parsing of background color

Fix printing from field in log_urls

Fix processing of last element of DMARC policies

Further fixes for HTML colors

Further fixes for multiple values headers

Further fixes for white on white rule

Further fixes in HTML tags parsing

Ignore content type/subtype case

Increase score of R_WHITE_ON_WHITE

Parse CGP envelope data

Propagate colors in HTML

Restore multiple values headers in protocol

Restore multiple values in headers processing

Some more changes to tag's content length calculations

Some more fixes for low contrast fonts detector

SpamAssassin plugin: support check_freemail_header('EnvelopeFrom', [..])

Trigger HTML_SHORT_LINK_IMG on any external image

rspamd_stats: remove deprecated defined(@array)

Fix catena passwords validation

Fix crash when the first received is faked

Add DMARC_BAD_POLICY symbol when DMARC policy was invalid

Allow for matching hostnames in multimap (#773)

Allow for setting action based on DMARC disposition

Allow limiting of the inbound message size

Allow maps with multiple symbols and scores

Allow regexps in the emails maps

Allow to register metric symbols from multimap

Allow to reset redis tokens instead of appendig values

Allow to store strings in radix maps

Check UTF validity when there are utf regexps in a map

Correctly work when there is no hard reject action

Implement dependencies for maps

Another effort to unbreak sqlite locking

Avoid crash when closing mmapped file

Do not break history on NaN in required score

Ensure that hyperscan cache written is written properly

Filter NaN from scores in history

Fix DNSBL maps

Fix another locking issue in sqlite

Fix another locking issue with mapped files

Fix deadlock in mmaped file stats

Fix dependencies in multimap plugin

Fix emails module configuration

Fix greylist plugin (#755)

Fix greylisting plugin variable usage

Fix installed permissions for rspamd_stats

Fix locking in mmapped statistics

Fix paths in tests

Fix prefilter mode for multimap

Forgot to commit leftover changes

Really fix local.d includes

Restore selective greylisting behaviour

Set max size on per connection basis

Use temporary storage for hyperscan cache

Remove systemd socket activation

Fix SA rawbody processing - exclude top part

Fix decoding of UTF HTML entitles

Fix encrypted fuzzy requests

Fix leak of shared memory fds and files

Fix levenshtein distance calculations

Fix mime headers processing

Fix parsing of URLs in texts

Fix parsing of missing classes

Fix redis structure by adding {NULL, NULL} member

Fix some more URL detector issues

Fix systemd sockets activation

Fix unencrypted passwords processing in the controller

Fix writing CDPs to the database

Fix writing of encrypted HTTP requests

Plug memory leak in dkim module

Plug memory leak in headers getting code

Pre-filters and post-filters were completely broken

Properly support SA body regexps

Really skip filters in case of pre-result set

Restore the intended pre-filters behaviour

Adopt new maps code

Compile ragel sources when building rspamd

Finish rework for the rest of places that use HTTP

Fix DKIM headers canonicalization

Fix lua maps API

Import linenoise for line editing

Include config structure to all rcl handlers

Make chartable module useful

Move http internal structures to a private header

Partly fix controller

Remove dedicated images list

Rename http proxy to rspamd proxy

Rename mime parts structures

Rework HTTP code

Rework exceptions and newlines processing

Rework pre and postfilters system

Separate method to close backend connections

Start the complete maps rework

Use dynamically generated ragel C sources

Add 'blacklist' and 'strict' modes for whitelists

Add 'symbols_enabled' and 'groups_enabled' to settings

Add ESMTPSA received type

Add a simple script to evaluate rspamd rules in the logs

Add a simple tool to generate DKIM keys

Add a trivial heuristic for codepages

Add and use mumhash for non-crypto hashing

Add better method to check lua userdata types

Add better zip files search algorithm

Add catena PBKDF function

Add configuration knobs for in and out parser scripts

Add content filtering support to multimap

Add different timeouts for proxy connections

Add dot commands for lua REPL:

Add execution of lua global functions script

Add function for pretty printing of inet addresses

Add function to convert fstring_t to c string

Add function to create temporary shared memory mapping

Add function to generate random hex string

Add generic fucnction to parse IP maps

Add initial version of HTTP lua repl

Add learn condition to the default configuration

Add learn conditions for classifiers

Add limit for dkim signatures to be checked

Add locking routines for lua_util

Add lua API for getting info from archives

Add lua utility to decode URL encoding

Add method to copy message from http connection

Add mirrors feature

Add more algorithms for shingles generation

Add more domains to redirectors list

Add more encodingsto cryptobox hash API

Add more file utilities to lua_util

Add more functions to extract data from text parts

Add more methods to get headers from a task

Add more methods to init http message body

Add more options for redis config parsing function

Add new representation of email address

Add new symbols to filter bad extensions in messages

Add new utility methods to mimepart object

Add openphish support to rspamd phishing module

Add parsers for SMTP address in ragel

Add parsing of mirror hosts for fuzzy storage

Add parsing scripts for master connection as well

Add preliminary greylist plugin

Add preliminary phishtank support

Add preliminary rarv5 support

Add preliminary version of ssl toolbox

Add protection against open files limit and accepting sockets

Add rar v4 support

Add reading scripts for master connection

Add replies plugin

Add results parsing code

Add routines to compare and check pubkeys

Add simpler versions of refcounts

Add some time manipulation functions for lua APi

Add support for non-standard BATV signatures

Add support of address with at-domain list

Add support to search archives by magic

Add task:get_rawbody method

Add test to check shared memory support sanity

Add the initial version of LUA repl to rspamadm

Add throughput graph for RRD backend to WebUI

Add universal function to make a proper redis request

Add universal function to parse redis servers for plugins

Add util.unlink function

Add utility function to return random number from 0 to 1

Add utility method to convert ftok to C string

Add utility to map shared memory segments

Add versions to fuzzy storage

Add workaround for legacy clients in rspamd proxy

Add workaround for systems without sane shmem support

Add xoroshiro+ fast rng for non-crypto purposes

Adopt plugins for new maps API

Allow SPF to be checked for empty tasks

Allow binary patterns in lua_trie

Allow catena encrypted passwords in controller

Allow client ip match in the settings

Allow easy adding and overriding of fuzzy rules

Allow empty tasks to be processed

Allow hostnames in IP maps

Allow https maps

Allow multiple PBKDF types in `rspamadm pw`

Allow named fuzzy rules

Allow non zero terminated patterns in multipattern

Allow partial hash updates

Allow pipelining for redis.make_request

Allow sending empty requests using client

Allow setting fuzzy flag by symbol not by value

Allow setting scores and actions from lua

Allow shared memory simple http client

Allow static lua files in any parts of rspamd sources

Allow to change flag from fuzzy learn condition

Allow to check rspamd_text using maps

Allow to disable composite rules from settings

Allow to disable some modules from common redis setup

Allow to extract ucl_object from lua using common API

Allow to get settings and settings id hash from lua_task

Allow to have specific settings for mirrored traffic

Allow to open message from a shared memory segment

Allow to parse pubkeys from the rcl config

Allow to pass extradata from rspamd to rmilter

Allow to query storage about number of fuzzy hashes stored

Allow to read logs without symbols scores

Allow to read password from console for rspamc

Allow to set ciphers and CA paths in config

Allow to skip some initialization phases to speed up rspamadm

Allow underscore separated names in settings

Allow versioning for sqlite databases

Always allow to terminate rspamd

Better deal with backend errors

Better lua_redis logging

Configure CA path and ciphers

Create a dedicated parser to strip newlines

Deduplicate the same urls in multimap module

Distinguish luajit from lua

Do not print garbadge in --compact output

Dynamically detect if a CPU is incompatible with hyperscan

Enable forced resolving for some lua plugins

Enable rrd by default

Enable workaround for exim

Fix task functions to work without rspamd_config

Further improvements to chartable module

Further micro-optimizations for hashing and shingles

Further relax parser

Humanize numbers in stats widgets

Implement HTTPS client

Implement SSL support in http client

Implement body rules for the trie plugin

Implement braced regexp quantifiers

Implement compare scripts for mirrors results

Implement compare scripts setup

Implement composites policies

Implement conditional learning for classifiers

Implement constructing of map from UCL

Implement dkim signing in dkim check plugin

Implement fuzzy storage updates

Implement fuzzy updates push protocol

Implement https maps

Implement inter-process maps caching

Implement mapping of remote flags to local flags

Implement mirroring in rspamd proxy

Implement multi-flags fuzzy replies

Implement multiple-sources fuzzy storage

Implement order of pre/post filters

Implement partial deleting for multi-flags

Implement pipelining for redis async interface

Implement ragel parser for received headers

Implement reading of messages to shared memory

Implement refcount for messages

Implement retransmits for master connection

Implement zero-copy mode for HTTP reading

Improve SPF domain detection logic

Improve config:register_symbol function

Improve error report for type mismatch in lua

Improve fstrings API

Improve getting SMTP data from lua_task

Improve levenshtein distance function

Improve logging in proxy and add refcounts

Improve logging lua types

Improve master/slave logging

Improve phishing plugin

Improve phishtank and openphish support

Improve ragel build target

Improve statistics script

Initialize ssl library to use SSL connections

Interpolate points sent to webui

Limit logging of elements that could have too many items

Lock ANN file when loading

New abstract hashing API in cryptobox

Normalize quoted addresses

Now cryptobox lua API accepts rspamd text as input

Optimize alignment to speed up hashing

Parse received date and ESMTPA proto

Parse received timestamp

Pass settings id to log helper

Pass settings id to lua script from log helper

Perform files expansion on proxying

Preliminary implementation of fuzzy master/slave updates

Print userdata using tostring if possible

Propagate HTTP result string when using proxy

Properly implement unweighted round-robin algorithm

Reduce number of timers queries

Rework and improve fuzzy storage

Rework dns resolving API for lua, add 'forced' option

Rework fann module to understand settings

Rework listening system to allow multiple socket types per worker

Rework ratelimit module to set expiration

Save bayes probability in memory pool var

Save settings id hash for convenience

Search for SSL_set_tlsext_host_name support

Send DKIM signature to protocol reply

Show DKIM signature in rspamc client

Show symbols description in scan output

Sign message merely after DKIM check

Simplify machines by assuming that headers are unfolded

Sort symbols when displaying them in log

Split main connection from mirrored connections

Start moving to the new email address structure

Store HTTP headers in a hash table

Store more information about compressed files

Store raw headers value to use them in DKIM

Store text parts content with newlines stripped

Support DKIM signing

Support EXIF jpeg images

Support archive files list extraction

Support archives when matching patterns in multimap

Support premium/academic feed for openphish

Support rspamd_updates via https

Supprort FQDNs in phishing module maps

Try to read on fuzzy timeout to avoid fake timeouts

Try to select the optimal possible function for input

Unescape and unquote smtp addresses

Update fuzzy timestamp when adding value

Update mumhash

Use -flto if possible when optimizations are enabled

Use extended map types in lua map, unify code

Use file lock in logger to avoid deadlocks

Use generic global string split function

Use metrohash as well

Use mumhash by default for incremental hashing

Use mumhash for words hashing

Use new ip parsing API

Use new maps API for local addrs

Use new ragel parser in message parsing code

Use new received parser instead of old one

Use new redis API in DMARC plugin

Use new redis API in greylist plugin

Use new redis API in ip_score plugin

Use new redis API in ratelimit plugin

Use new redis API in replies plugin

Use new version of register_symbol in rspamd plugins

Use offset when passing shm to deal with encrypted requests

Use one pass to remove newlines and store their positions

Use rspamd specific type checks for userdata

Use shared memory storage for http maps

Use universal redis definitions in rspamd plugins

Various improvements in greylist module

Wait for sqlite if locked when switching to WAL mode

Add filenames sanity filtering for mime types

Add guards for empty parts

Add missing types

Add more priority for config file symbols registered from UCL

Add sanity checks when compiling regexp maps

Add spaces instead of newlines to the normalized content

Add workaround for ancient openssl

Add workaround for gmime CTE stupidity

Add workaround for hex digits

Adjust MISSING_MIMEOLE score

Adjust body/body_buf when stealing encrypted message

Adopt lua task API for the new email addresses structure

Allow for disabling DMARC reporting when Redis is configured

Always register openphish and phishtank virtual symbols

Always use shmem on linux

Another change of newlines policy

Another d3evolution update

Another fix for exim workaround

Another fix for legacy clients

Another fix for maps scheduling

Another fix for marking upstreams inactive

Another fix for postfilters

Another fix for redis timeouts

Avoid `table.getn` method as it has been removed in lua 5.3

Avoid double hashing for images

Avoid linking with actrie if hyperscan is enabled

Check copy result when sending message to mirrors

Cleanup message when assiging body

Cleanup stack from global vars

Correctly parse query type

Disable all symbols if enable_groups is found in settings

Disable fts as it is completely broken in bloody linux

Disable multiple autolearn checks

Disallow updates by default

Do not abort when cannot load a map

Do not check recursion for non-DNS SPF record types

Do not delete uninitialized events

Do not die if shmem_mkstemp fails

Do not die when no metrics defined

Do not even try pcre in case of regexp maps

Do not greylist messages if redis has failed somehow

Do not greylist on rejection

Do not leave temporary maps cached

Do not output meaningless errors

Do not send NaN in json

Don't mix hyperscan and pcre processing within a same task

Finally rework and simplify redis backend for statistics

Fix Exim shutdown patch

Fix JIT compilation for PCRE2 expressions

Fix JIT usage for PCRE2

Fix REPL output

Fix SMTP address parsing machine

Fix UTF8 mode in PCRE2

Fix a stupid misprint in word 'phishing'

Fix accepting the first update when local idx is -1

Fix adding maps from ucl

Fix adding upstream to an active queue

Fix and rescore R_PARTS_DIFFER logic

Fix body rules in SA plugin

Fix body start position

Fix border case in urls detector

Fix border cases for incremental hashing

Fix caseless uthash application

Fix chartable issue with starting digits

Fix client_ip in users settings

Fix compilation issue

Fix conditional learning

Fix crash on empty maps

Fix creating of URLs from LUA

Fix creating of temporary shmem segment

Fix creation of mmapped statfiles

Fix descriptors leak on shmem detaching

Fix detaching of shared memory segments

Fix detection of URLs in text parts

Fix directories processing for rspamc

Fix displaying of rewrite subject in WebUI

Fix dkim private keys operations

Fix dkim signing

Fix dynamic scoring of subject symbols

Fix email address build

Fix encrypted proxy requests

Fix errors counting in upstreams

Fix errors handling in the proxy

Fix event bases for IO events

Fix events handling when scheduling map wacth

Fix fann rewrite

Fix files fallback for shmem transfer

Fix fuzzy adding in webui

Fix fuzzy storage encrypted mirroring

Fix fuzzy storage sync replies

Fix handling of the same words

Fix inserting values to the sources list

Fix ipv6 mask application

Fix issue with missing recipients

Fix issues with multiple returns from lua

Fix learning for non-existent backend

Fix legacy clients support in proxy

Fix length calculations for shared memory segments

Fix listening on UDP sockets

Fix loading of file maps

Fix long regexp flags (e.g. {sa_body})

Fix lua compare function init

Fix maps descriptions

Fix maps locking

Fix max_train setup in ANN module

Fix memory corruption

Fix memory leak in unsigned maps reading

Fix misprints for lto usage

Fix more issues with scripts processing

Fix next-to-last extension length check

Fix openssl initialization

Fix order of arguments

Fix order of initialization

Fix parser

Fix parsing of binary tries

Fix parsing of braced IPv6 addresses

Fix parsing of nested braces in SMTP comments

Fix parsing of rarv5 archives

Fix periodic scheduling when a map is not modified

Fix possible FP in TRACKER_ID rule

Fix post-filters processing

Fix potential NULL dereference

Fix processing of <br> and <hr> tags

Fix processing of addresses in protocol

Fix processing of messages without received headers

Fix proxying issue for unconnected sessions

Fix proxying of the encrypted messages

Fix race condition with shared memory by refcounts

Fix ratelimit initialization

Fix redis set request in replies plugin

Fix redis timeout events handling

Fix redis timeouts processing logic

Fix refcounts in lua_redis

Fix results checking if no master connection is active

Fix return value for couple of lua functions

Fix round-robin selection when upstreams have no weight

Fix rows calculation in graph

Fix rspamd_redis_make_request syntax in replies plugin

Fix scheduling of locked map events

Fix scores detection

Fix searching for newline positions

Fix secure_ip setting in controller

Fix sending data to graph command

Fix setting of score for parts differ

Fix setting of the lua top

Fix setting path for lua

Fix setting path for phishtank

Fix settings application

Fix shm_open call as described in POSIX

Fix size of length in fuzzy mirror wire protocol

Fix smtp grammar issues

Fix some issues with redis API

Fix some issues with retries in the proxy

Fix stack growing

Fix start of body detection in DKIM

Fix state on timeout

Fix stats script

Fix substring search when there are '\0' in strings

Fix symbol name for spf soft fail

Fix symbol type's check

Fix symbols registration and execution

Fix the case of multiple values keys

Fix the default symbol names according to metric

Fix timeout setup on learning

Fix timeouts in redis cache processing

Fix timeouts processing in lua_redis

Fix upstreams interaction for rspamd proxy

Fix usage of rdns reply structure

Fix varargs loop

Fix whitelists and blacklists in SA rules

Fix write servers setup for redis

Fix writing of HTTP messages

Force rspamd to upgrade fuzzy db on opening

Free the correct pointer

Further fixes for lto and static linking

Further fixes for surbl extensions map

Further fixes in maps code

Further improvements to error messages in fuzzy check

Further tweaks to redis garbadge collection

Groups are now case insensitive

Handle log pipe read errors

Handle nested dependencies in SpamAssassin plugin

Implement new automata to skip empty lines for dkim signing

Improve error messages on fuzzy add

Improve lua redis handling

Improve phishing module logging

Improve printing of fuzzy errors

Improve rrd diagnostic errors

Improve strcase hash used in uthash

Include fuzzy key to distinguish storages with different keys

Include slave cluster name into http request

Include some more information about archives

Indicate upstream error on timeout

Initialize hash tables array to avoid crashes

Initialize parser scripts properly

Initialize vars to avoid warnings

Inverse logic for saving ANN

Link lpeg to rspamd lua library

Make extension checks case-insensitive

Mark expired hashes as not found and not as zero flag

Match archive name as well

More and more fixes to redis states

More fixes about shared memory in proxy

More fixes for redis refcounts

More fixes to end of headers detection

More fixes to events logic

More fixes to multi-flag fuzzy storage

More fixes to parts distance calculations

More guards for redis free

One more fix in redis destructor

One more try to fix redis

PIE is required for static build

Partial fix for mmap'd statistics tests

Plug memory leak in proxy

Properly detect end of headers position

Properly init and free session structures

Reduce PRECEDENCE_BULK rule weight

Reduce the default thresholds for learning

Remove Type=forking from systemd unit file (#709)

Remove bad FANN file to save computational resources

Remove event before closing of fd to avoid race conditions

Remove parsing of 'from' variable in redis backend

Remove some bad domains from whitelists

Repair optional dependencies

Reset master connection when retransmitting scan request

Restore ONCE_RECEIVED symbol

Restore compatibility with old lua API behaviour

Restore redis runtime state

Reverse options when received

Send updates to mirrors only if we have some changes

Set host attribute properly when making HTTP request from lua

Set terminated state before calling of async free

Simplify MISSING_MIMEOLE rule

Simplify state machine by ignoring multiple spaces

Skip setting RPATH for static builds

Slightly reduce weights of rules with high FP rate

Some fixes to libmagic initialization

Some more fixes to ratelimit plugin

Strip '\r\n' properly

Switch hashes to mumhash

Treat NaN values properly in graph command

Try to avoid FP when checking for phished URLs

Try to avoid recursive events deletions

Try to fix false positive URL detections in text parts

Try to fix issue in redis stats backend when task is closed

Try to fix proxying of stupid spamc protocol to HTTP mirrors

Try to fix redis crashes

Try to fix upstreams with one element

Try to handle multiline history in a more sane way

Unbreak build on gcc < 4.9

Update RPM spec/sources (#700)

Update d3evolution version

Update mumhash implementation

Use custom error function for pre and post filters

Use new postfilters and prefilters API in the plugins

Use non-blocking mode for systemd sockets

Use shared memory merely for local backends in the proxy

Use watchers for spf plugin

Varioud fixes to the maps code