Today we have released Rspamd 3.10.2, which is being maintained in the stable rspamd-3.10 branch.
The following fixes are included in this release:
- Fix for ARC signing contributed by
@jscissr - Added EOF to Prometheus metrics by
@henry-spanka
Today we have released Rspamd 3.10.2, which is being maintained in the stable rspamd-3.10 branch.
The following fixes are included in this release:
@jscissr@henry-spankaToday we have released Rspamd 3.10.1 featuring several important bug fixes.
@arkamar@smarschingToday we have released Rspamd 3.10.0; the most important new features and fixes are highlighted below.
Rspamd now correctly processes MIME UTF8 messages and doesn’t penalize them
Added the ability to set a negative group score limit using the min_score parameter.
It is now possible to set specific ratelimits and expiration for individual fuzzy encryption keys
More portable way in some specific configurations
Added SenderScore Reputation RBL and RPBL return codes to improve sender reputation checks.
Introduced full support for OpenSSL 3.0, including integration for DKIM (DomainKeys Identified Mail).
@JasonStephenson and @spacefreak86rep handling@oliykslua_redisFixed and updated old rules to enhance overall system performance.
We extend our gratitude to everyone who contributed to this release.
Today we have released Rspamd 3.9.0, featuring many new features and fixes. The most important ones are highlighted below. Refer to the migration notes for an overview of potentially-breaking changes.
Rspamd now uses a reduced window size of 2 words by default. This change does not require retraining of statistics. In our tests, this reduced window size has produced the equal or better results with better performance and lower storage requirements - around 4 times less than with the previous default window size of 5 words. The new rspamadm classifier_test utility can be used for your own experiments.
This release provides a module for using LLMs for text classification and unsupervised learning. You can read more about it in a dedicated blog post.
known_senders and replies modulesThis release introduces enhancements to the known_senders and replies modules, enabling them to work together to flag verified user contacts. With these improvements, senders to whom a user has previously replied will automatically receive negative scores. For more details, please refer to the documentation of these modules.
To avoid potential confusion, dynamic ratelimits are now disabled by default and must be configured explicitly. Refer to the migration notes for details on how to do this.
rspamc client uses it now by defaultgrow_factor to work in an orderly fashionSUBJ_ALL_CAPS for unicase scripts by @ikedas@aduernberger@PxPert@twesterhever and @ishisoraWe extend our gratitude to everyone who contributed to this release.
Historically, our only text classification method has been Bayes, a powerful statistical method that performs well with sufficient training. However, Bayes has its limitations:
Large Language Models (LLMs) offer promising solutions to these challenges. These models can perform deep introspection with some sort of contextual “understanding”. However, their high computational demands (typically requiring GPUs) make scanning all emails impractical. Separating LLM execution from the scanning engine mitigates resource competition.
In Rspamd 3.9, I have tried to integrate the OpenAI GPT API for spam filtering and assess its usefulness. Here are the basic ideas behind this plugin:
BAYES_SPAM, FUZZY_DENIED, REPLY, etc.) are excluded from the GPT scanThe former two points reduce the GPT workload for something that is already known, where GPT cannot add any value in the evaluation. We also use GPT as one of the classifiers, meaning that we do not rely solely on GPT evaluation.
To evaluate the performance of the GPT-based classifier, we developed the rspamadm classifier_test utility, capable of evaluating both supervised and unsupervised classifiers:
For example, the Bayes engine, trained on a robust corpus, demonstrates the following results:
$ rspamadm classifier_test --ham /ham --spam /spam --cv-fraction 0.3
Spam: 348 train files, 815 cv files; ham: 754 train files, 1762 cv files
Start learn spam, 348 messages, 10 connections
Start learn ham, 754 messages, 10 connections
Learning done: 348 spam messages in 1.61 seconds, 754 ham messages in 3.88 seconds
Start cross validation, 2577 messages, 10 connections
Metric Value
------------------------------
True Positives 735
False Positives 22
True Negatives 1717
False Negatives 49
Accuracy 0.97
Precision 0.97
Recall 0.94
F1 Score 0.95
Classified (%) 97.90
Elapsed time (seconds) 12.71
These results are impressive but assume the classifier is properly and decently trained. In scenarios involving a fresh system or high variability in emails, gathering reliable statistics might be challenging.
In contrast, the GPT engine operates as an unsupervised learning algorithm. We assume that LLM models have enough “understanding” of the language to distinguish spam and ham without direct training on emails. Moreover, we provide only text data, not raw email content.
Below are the results from different GPT models:
Metric Value
------------------------------
True Positives 129
False Positives 35
True Negatives 263
False Negatives 69
Accuracy 0.79
Precision 0.79
Recall 0.65
F1 Score 0.71
Classified (%) 95.20
Elapsed time (seconds) 318.91
This model is cost-effective and can be used as a baseline. The results were obtained from a low-quality sample corpus, resulting in high false positives and negatives.
Metric Value
------------------------------
True Positives 178
False Positives 25
True Negatives 257
False Negatives 9
Accuracy 0.93
Precision 0.88
Recall 0.95
F1 Score 0.91
Classified (%) 90.02
Elapsed time (seconds) 279.08
Despite its high cost, this advanced model is suitable, for example, for low-traffic personal email. It demonstrates significantly lower error rates compared to GPT-3.5, even with a similar low-quality sample corpus.
Another interesting approach involves using GPT to supervise Bayes engine training. In this case, we benefit from the best of both worlds: GPT can operate without training, while Bayes can catch up afterward and perform instead of GPT (or at least serve as a cost-saving alternative).
So we tested GPT training Bayes and compared efficiency using the same methodologies.
GPT results:
Metric Value
------------------------------
True Positives 128
False Positives 13
True Negatives 301
False Negatives 68
Accuracy 0.84
Precision 0.91
Recall 0.65
F1 Score 0.76
Classified (%) 97.89
Elapsed time (seconds) 341.77
Bayes classifier results (trained by GPT in the previous test iteration):
Metric Value
------------------------------
True Positives 19
False Positives 43
True Negatives 269
False Negatives 9
Accuracy 0.85
Precision 0.31
Recall 0.68
F1 Score 0.42
Classified (%) 65.26
Elapsed time (seconds) 29.18
Bayes still exhibits uncertainty in classification, with more false positives than GPT. Improvement could be achieved through autolearning and by refining the corpus used for testing (our corpus contains many ham emails that look like spam even for human evaluators).
The GPT plugin operates as follows:
excluded set (e.g. Fuzzy/Bayes spam/Whitelists)OpenAI provides an API for these requests, incurring costs (currently no free tier available). However, for personal email usage or automated Bayes training without manual intervention, GPT presents a viable option. For instance, processing a substantial volume of personal emails with GPT-3.5 costs approximately $0.05 daily (for about 100k tokens).
For large-scale email systems, it may be preferable to use another LLM (e.g. llama) internally on a GPU-powered platform. The current plugin is designed to integrate with different LLM types without significant modifications. This approach also enhances data privacy by avoiding sending email content to a third-party service (though OpenAI claims their models do not learn from API requests).
Despite not achieving 100% accuracy, the GPT plugin demonstrates efficiency comparable to human-filtered email. Future enhancements will focus on improving accuracy through additional metadata integration into the GPT engine, while optimizing token usage efficiency. There are also plans to better utilize LLM knowledge in Rspamd, particularly for better fine-grained classification.
The GPT plugin will be available starting from Rspamd 3.9, requiring an OpenAI API key and financial commitment for accessing ChatGPT services.