Maps are one of the most important and flexible features in Rspamd, allowing for dynamic configuration of various elements without requiring service restarts. This article provides a comprehensive overview of Rspamd maps, their types, configuration options, and best practices.
Maps in Rspamd are dynamic data sources that contain lists of keys or key-value pairs that can be reloaded at runtime without restarting the service. Maps can be defined using various backends including:
The significant distinction between maps and static configuration elements is that maps can be updated “live” without the costly restart procedure. Rspamd automatically monitors maps for changes:
inotify where available (meaning you must save maps files atomically using rename as otherwise you might end up with an incomplete load)Rspamd supports several map types, each with specific functionality:
Maps can specify their type using prefixes in URI definitions, like:
regexp;/path/to/file
hash;http://example.com/map
radix;/path/to/ip_list
Performance consideration: regular expression maps are slower than other type of maps especially if your Rspamd is built without Hyperscan support.
Maps can be defined in four main ways:
map = "http://example.com/file.txt"
# or
map = "/path/to/local/file"
map = [
"http://example.com/file.txt",
"/path/to/local/file"
]
When using composite paths, data from all sources is concatenated.
map = [
"foo bar",
"baz qux"
]
# For IP maps:
ip_list = [
"192.168.1.1/24",
"10.0.0.0/8"
]
map = {
name = "My important map";
description = "Contains important data";
url = "http://example.com/map.txt";
timeout = 10.0; # in seconds
}
Or for multiple URLs:
map = {
name = "Multi-source map";
urls = [
"http://example.com/map.txt",
"/local/backup/map.txt"
];
}
Maps can contain various elements depending on their type:
key1 # Single key with comment
# Full line comment is ignored
# Empty line ignored
key2 1 # Key and value (for hash maps)
"key3 with space"
"key with \" escaped" value with spaces
/regexp/i
/regexp/is some other value
Options after the final slash:
i - ignore caseu - use UTF-8 regexpm - multi-line regular expression - this flag causes the string to be treated as multiple lines. This means that the ^ and $ symbols match the start and end of each line within the string, rather than just the start and end of the first and last lines.x - extended regular expression - this flag instructs the regular expression parser to ignore most white-space that is not escaped (\) or within a bracketed character class. This makes it possible to break up the regular expression into more readable parts. Additionally, the # character is treated as a meta-character that introduces a comment which runs up to the pattern’s closing delimiter or to the end of the current line if the pattern extends onto the next line.s - dot-all regular expression - this flag causes the string to be treated as a single line. This means that the . symbol matches any character whatsoever, including a newline, which it would not normally match. When used together as /ms, they allow the . to match any character while still allowing ^ and $ to respectively match just after and just before newlines within the stringO - do not optimize regexp (rspamd optimizes regexps by default)r - use non-UTF-8 regular expressions (raw bytes). Defaults to true if raw_mode is set to true in the options section.A - return and process all matches (useful for Lua prefilters)L - match left part of regexp (useful for Lua prefilters in conjunction with Hyperscan)Please note that you must use /u modifier if you want to match UTF8 characters or classes.
192.168.0.1 # Mask is /32 (single IP)
[::1] # IPv6, mask is /128
[::1]/64 # IPv6 with CIDR
192.168.0.1/19 # IPv4 with CIDR
You can mix both IPv4 and IPv6 addresses or networks in a single map.
Rspamd implements sophisticated caching for HTTP maps:
There are two main startup scenarios:
To provide resilience, you can configure fallback options:
map = [
"https://maps.rspamd.com/rspamd/whitelist.inc.zst",
"${DBDIR}/local_whitelist.inc",
"fallback+file://${CONFDIR}/maps.d/whitelist.inc"
];
In this example, the fallback+file:// option is used only when the HTTP source is unavailable during cold starts.
When Rspamd is running, maps follow a specific lifecycle:
The default poll interval can be adjusted in the configuration:
options {
map_timeout = 60s; # Default HTTP map poll interval
map_file_watch_multiplier = 0.1; # Local files are checked more frequently
}
From Rspamd version 1.2 onwards, each map can have a digital signature using the EdDSA algorithm. To sign a map, you can use rspamadm signtool, and to generate a signing keypair, use rspamadm keypair -s -u:
keypair {
pubkey = "zo4sejrs9e5idqjp8rn6r3ow3x38o8hi5pyngnz6ktdzgmamy48y";
privkey = "pwq38sby3yi68xyeeuup788z6suqk3fugrbrxieri637bypqejnqbipt1ec9tsm8h14qerhj1bju91xyxamz5yrcrq7in8qpsozywxy";
id = "bs4zx9tcf1cs5ed5mt4ox8za54984frudpzzny3jwdp8mkt3feh7nz795erfhij16b66piupje4wooa5dmpdzxeh5mi68u688ixu3yd";
encoding = "base32";
algorithm = "curve25519";
type = "sign";
}
Then you can use signtool to edit the map file:
rspamadm signtool -e --editor=vim -k <keypair_file> <map_file>
To enforce signing policies, you should add a sign+ string to your map definition:
map = "sign+http://example.com/map"
To specify the trusted key you could either put the public key from the keypair in the local.d/options.inc file as following:
trusted_keys = ["<public key string>"];
or add it as a key definition in the map string:
map = "sign+key=<key_string>+http://example.com/map"
Rspamd supports “external maps” for dynamic lookups to external services. These maps query an external service for each key lookup rather than loading the entire map into memory.
External maps are configured with the external = true parameter:
external_map = {
external = true;
backend = "http://lookup-service.local/api";
method = "query"; # Can be "query", "header", or "body"
timeout = 1.0; # Timeout in seconds
}
The external map API has three methods for passing lookup keys:
For complex data structures, you can specify an encoding:
external_map = {
external = true;
backend = "http://lookup-service.local/api";
method = "body";
encode = "json"; # Can be "json" or "messagepack"
}
HTTP maps support two authentication methods:
map = "http://user:password@example.com/map.txt"
options {
http_auth {
example.com {
user = "username";
password = "secret";
}
}
}
This approach is more secure as credentials aren’t stored in map URLs.
Rspamd supports compressed maps with Zstandard (.zst or .zstd extension):
map = "https://maps.rspamd.com/rspamd/whitelist.inc.zst"
Compressed maps:
For resilience, Rspamd supports fallback options for maps:
map = [
"https://maps.rspamd.com/rspamd/whitelist.inc.zst",
"fallback+file://${CONFDIR}/maps.d/whitelist.inc"
];
The fallback+ prefix indicates this source should only be used if other sources fail:
Rspamd supports Constant Database (CDB) maps, which provide extremely fast, read-only key-value lookups. CDB maps are especially useful for large datasets with frequent lookups.
cdb_map = {
external = true;
cdb = "/var/lib/rspamd/domain_settings.cdb";
}
CDB maps can be used in regular maps or as external maps for per-key lookups.
CDB files can be created using the Rspamd Lua API:
local rspamd_cdb = require "rspamd_cdb"
local builder = rspamd_cdb.build('/path/to/map.cdb')
builder:add('key1', 'value1')
builder:add('key2', 'value2')
builder:finalize()
Benefits of CDB over other map types:
Rspamd provides several Lua functions for working with maps programmatically:
local lua_maps = require "lua_maps"
-- Add a map from configuration
local my_map = lua_maps.map_add('module_name', 'option_name', 'map_type', 'description')
-- Add a map from UCL object
local my_map = lua_maps.map_add_from_ucl(ucl_object, 'map_type', 'description')
-- Bulk map configuration
local map_defs = {
my_map = {
type = 'set',
description = 'My important list',
optional = true
},
ip_map = {
type = 'radix',
description = 'IP blocklist'
}
}
lua_maps.fill_config_maps('module_name', options, map_defs)
Once a map is loaded, you can use it for lookups:
-- Simple lookup
local result = my_map:get_key('test_key')
-- With callback
my_map:get_key('test_key', function(is_found, value, code, task)
if is_found then
-- Use the value
end
end, task)
-- Iterate through all entries
my_map:foreach(function(key, value)
-- Process each entry
return true -- Continue iteration
end)
After analyzing the lua_maps_expressions.lua file, I’ll add information about map expressions to your article. This is an important feature that allows combining multiple maps with selectors in powerful expressions.
Map expressions provide a powerful way to combine multiple maps using boolean logic. This feature allows you to create complex filtering rules by combining simpler components using logical operators.
Map expressions allow you to:
This creates a flexible framework for defining complex conditions without writing custom Lua code.
Map expressions are defined using a structured format:
whitelist_ip_from = {
rules {
ip {
selector = "ip";
map = "/path/to/whitelist_ip.map";
type = "radix"; # Optional, can be automatically inferred
}
from {
selector = "from(smtp)";
map = "/path/to/whitelist_from.map";
}
}
expression = "ip & from";
}
The key components are:
The expression syntax supports the following operators:
& - logical AND| - logical OR! - logical NOT+ - arithmetic plus (can be used for weighted combinations)() for groupingExamples:
ip & from # Both IP and From must match
ip | from # Either IP or From must match
ip & !from # IP matches but From does not match
(ip & from) | asn # Either both IP and From match, or ASN matches
Selectors are functions that extract specific values from a message. Common selectors include:
ip - IP address of the message senderfrom(smtp) - SMTP From addressfrom(mime) - From header in the messagercpt(smtp) - SMTP recipientheader(name) - Value of a specific headerurl - URLs in the messageasn - Autonomous System Number of the senderYou can create custom selectors or use any of Rspamd’s built-in selectors.
When a task is processed against a map expression:
When a match occurs, the function returns:
Here’s a complete example of creating and using a map expression:
local lua_maps_expressions = require "lua_maps_expressions"
local whitelist_config = {
rules = {
ip = {
selector = "ip",
map = "/path/to/whitelist_ip.map",
type = "radix"
},
from = {
selector = "from(smtp)",
map = "/path/to/whitelist_from.map",
type = "set"
},
domain = {
selector = "from(smtp).domain",
map = "/path/to/whitelist_domains.map",
type = "set"
}
},
expression = "(ip & from) | domain",
symbol = "WHITELIST_EXPRESSION" -- Optional symbol to register
}
local whitelist = lua_maps_expressions.create(rspamd_config, whitelist_config, "whitelist_module")
-- Later in the code:
local function symbol_callback(task)
local result, matched = whitelist:process(task)
if result then
-- Matched! We can access details via the 'matched' table
if matched.ip then
task:insert_result("IP_WHITELISTED", 1.0, matched.ip.matched)
end
if matched.from then
task:insert_result("FROM_WHITELISTED", 1.0, matched.from.matched)
end
if matched.domain then
task:insert_result("DOMAIN_WHITELISTED", 1.0, matched.domain.matched)
end
end
end
rspamd_config:register_symbol({
name = "CHECK_WHITELIST_EXPRESSION",
callback = symbol_callback
})
Map expressions offer several advantages:
The main API functions for map expressions include:
cfg: Rspamd configuration objectobj: Configuration table with rules and expressionmodule_name: Optional module name for loggingprocess(task) methodnil if no match, or two values if matched:
Maps are one of the most powerful features in Rspamd, allowing for dynamic configuration and real-time updates without service restarts. By understanding the different map types, configuration options, and best practices, you can build flexible and efficient filtering rules that adapt to changing conditions.
When designing your Rspamd setup, consider: