# Self-hosting email the hard way from your own routable IPv4 block up
*2026-06-06 — note*
Fresh from [rewilding the web](https://anil.recoil.org/notes/rewilding-the-web-report), I updated the Recoil self-hosting
infrastructure that [Nick Ludlam](https://nick.recoil.org) and I have run since 1997. Most exciting, this is now
'email the hard way' that includes getting our very own dedicated IPv4 allocation routed thanks
to my buddy [Thomas Gazagnaire](https://github.com/samoht) helping out from France\!
This post will be quite technical, aimed at those interested in building their own email stack. We'll talk about [why bother](#why-run-your-own-email) running your own email; [receiving email](#email-receipt) (covering [IP reputation](#ip-reputation-and-denylists), [getting our own IPv4 allocation](#getting-our-very-own-ipv4-address-block), [stopping bots](#stopping-the-zombie-spam-horde), and [Sieve delivery](#delivering-email-with-lmtp-and-sieve)); [sending email](#sending-email-to-other-people) reliably with [SPF](#spf-describes-who-can-send-email-for-a-domain), [DKIM](#dkim-cryptographic-signing-of-outbound-mail), [DMARC](#dmarc-ties-it-together-and-provides-reporting-stats) and [SRS](#srs-to-keep-email-forwarding-working); [user access](#accessing-email-for-users) via Dovecot IMAP and Roundcube webmail; and finally [what's left to do](#what-else-is-left-to-do) and a [reflection](#is-this-a-negative-result-for-self-hosting) on future research ideas.
## Why run your own email?
For someone just getting into systems and networking, it's a hugely educational
experience. Running my own servers has been how I've learnt how the Internet
works, and how I got into open-source back in the day, by [installing OpenBSD](https://anil.recoil.org/notes/openbsd-developer) and [fixing bugs in PHP](https://anil.recoil.org/notes/commit-access-to-php)\!
More broadly, self-hosting is important for sovereign access to your own data
as the [web steadily consolidates](https://doi.org/10.1145/3503158) among a few
players. A [2023 analysis](https://www.netmeister.org/blog/mx-diversity.html)
showed that two companies can mostly read all of your email traffic:
> So all in all, the answer to the question of who can read your email pretty
> much boils down to -- yep -- "Google and Microsoft". Even if your domain
> doesn't use one of their mail servers, chances are that [whoever you are sending mail to does](https://mako.cc/copyrighteous/google-has-most-of-my-email-because-it-has-all-of-yours).
> \-- [MX diversity](https://www.netmeister.org/blog/mx-diversity.html), [Jan Schaumann](https://mstdn.social/@jschauma), 2023
Email is right at the centre of our digital lives; consider just how many online
accounts you could reset if your email got [hijacked or phished](https://dl.acm.org/doi/10.1145/3716489.3728437). In many ways, it's *the* online service that connects up all the other ones.
### Should you run your own email?
Hosting your email is a fair bit of work, but that's spread over a long period of time as you keep an eye on it. [Nick Ludlam](https://nick.recoil.org) and I started our hosting adventure back in 1998 or so when we [worked at NASA](https://anil.recoil.org/notes/netapp-tr-3071-1) for the summer. The first time we did a major server mode back in 2002, here's Nick and [Chris Luke](https://nanog.org/events/nanog-48/content/2951/)
loading up our second server into a dodgy white minivan to host in Easynet, right outside [London's first Internet cafe](https://en.wikipedia.org/wiki/Cyberia,_London).
From Cyberia outside Fitzrovia over to Brick Lane and Easynet hosting! (2002)
If you do decide to have a go for yourself, you'll need to find stable Internet hosting (your own home network isn't the best choice,
for reasons we'll see later in this post), and also build up a reputation. Luckily, finding stable Internet is *much* easier these days than it was in the late 90s, and we use [Mythic Beasts](https://mythic-beasts.com) who are excellent, reliable and local.
I'll explain in this post how we obtained our own dedicated IPv4 address block
to help build up a high deliverability index for our personal email that's
entirely independent! While email looks like one service to most users, it
is actually three separate activities: [email receipt](#email-receipt), [email
submission](#sending-email-to-other-people), and [email access](#accessing-email-for-users). I'll dive into how each of
these work on Recoil now, in case it's useful for your own setup.
## Email receipt
Each domain (like `recoil.org`) on the Internet runs an SMTP server that allows
it to receive email from other servers. You can query this server for any
domain by looking up its 'MX' DNS record:
```bash
$ host -t mx recoil.org
recoil.org mail is handled by 10 pork.recoil.org.
```
This indicates that the Internet host `pork.recoil.org` must accept connections from any server
on the internet that wishes to deliver email to `@recoil.org`.
The core difficulty is that [SMTP](https://www.rfc-editor.org/rfc/rfc5321) (as
designed in the more trusting 1980s) doesn't mandate a built-in proof of trust.
Anyone can claim to be anyone else, and the 'sender' in the email we receive
can be trivially forged.
The IETF's [response](https://datatracker.ietf.org/doc/html/rfc2505) was to accrete a stack of checks over the years that an
email sender must pass, or be filtered by the recipient. If we mess these
identity checks up, then our email won't get delivered reliably across the
Internet and the service won't be very useful.
### IP reputation and denylists
Spam was cheap to send from anywhere on the Internet, and so naturally grew as
the wider network gained adoption. Paul Vixie came up with a [DNS-based
blocklist](https://en.wikipedia.org/wiki/Domain_Name_System_blocklist) back in
1997\. Since then, many more have sprung up, operated by organisations like
[Spamhaus](https://www.spamhaus.org/), [Spamcop](https://www.spamcop.net/) and
[Barracuda](https://www.barracudacentral.org/rbl) that aggregate reports about
botnets, compromised hosts and spammers into lists that any email server can
consult.
The DNS blacklist/whitelist protocol ([RFC 5782](https://www.rfc-editor.org/rfc/rfc5782)) is super simple and can be queried right from your command line:
```bash
$ dig 2.0.0.127.zen.spamhaus.org +short # testing address
127.0.0.2
127.0.0.10
127.0.0.4
```
That's a localhost testing address, but the presence of a DNS record in the RBL
means that that server is suspect. This is where having control of your own
IPv4 address really pays off. Reputation for email via these RBLs accrues
against the address and not the domain. In theory, the IP address you use for
your own self-hosted email [might have been re-used](https://doi.org/10.1145/3419394.3423657) from a cloud provider
by someone else, and therefore be tainted by other people's bad behaviour.
*(Update: [Ryan Gibb](https://ryan.freumh.org) observes that as these lists often work on IP blocks, your IP neighbour can also effect your reputation in a multitenant cloud. He reports that he uses [Hetzner](https://www.hetzner.com/) for his use who [manually allowlist](http://bef.no/DitchingWindowsAndAWS/) SMTP servers to minimise abuse).*
### Getting our very own IPv4 address block
In contrast, a fresh IPv4 starts neutral and earns its reputation through
consistent, well-formed email sending. The only way to fully control this is
to control the address space itself, which is why Team Recoil now have our very
own IPv4 address block: `185.33.27.0/24`\!
Getting our very own address allocation involved joining a very long queue due to IPv4
exhaustion. The [RIPE NCC](https://www.ripe.net/) is the regional registrar for
Europe and ran out of unallocated IPv4 space in [November
2019](https://www.ripe.net/publications/news/the-ripe-ncc-has-run-out-of-ipv4-addresses/),
and since then the only way to get an allocation directly from RIPE is via a
[waiting list](https://www.ripe.net/manage-ips-and-asns/ipv4/) for small allocations.
> While we have run out of IPv4 addresses, RIPE NCC members can still request a
> single /24 allocation (256 addresses). \[...\] Requests are added to a waiting
> list and processed when we recover IPv4 addresses in the future. \[...\] This
> is only available to LIRs that have never received an IPv4 allocation from
> the RIPE NCC before (of any size).
> \-- [RIPE NCC, /24 Allocation via the Waiting List](https://www.ripe.net/manage-ips-and-asns/ipv4/)
I got into that queue from the UK, and my buddy [Thomas Gazagnaire](https://github.com/samoht) did the same from France.
He got to the head of his queue well ahead of me, and we got allocated our
own `/24` block after about a six month wait.
#### Signing up to RIPE for yourself
If you want to do this yourself and are based in Europe, then pay the annual RIPE NCC membership fee to open a
[Local Internet Registry](https://www.ripe.net/membership/become-a-member/)
(LIR) account.
After that, confirm you've never previously received an IPv4 allocation,
join the waiting list, and wait until enough addresses are recovered (e.g. from
defunct LIRs, returned space, or revoked allocations) for a slot to come
up. The wait is currently a year or two I think and seems to depend which country you're in.
RIPE LIR waiting list statistics
#### Setting up IPv4 routes for an autonomous system
The next step was to route this allocation to an actual machine hooked up
to the public Internet. While it is possible to [advertise our own routes](https://en.wikipedia.org/wiki/Border_Gateway_Protocol), in the
interests of expediency we decided to request [Mythic Beasts](http://mythic-beasts.com) to take care of it for us and handle the peering.
The procedure to do this via RIPE is straightforward. The IPv4 block is given
an assignment by creating a '[RPKI ROA](https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastructure)' in
their database. This is a PKI chain-of-trust used to connect up IP routing blocks on the
Internet.
An [Autonomous System](https://en.wikipedia.org/wiki/Autonomous_system_\(Internet\))
(AS) is a unit of independent routing policy on the Internet and announced to the rest of the world via
[BGP](https://www.rfc-editor.org/rfc/rfc4271).
Working out who actually owns a given ASN turns out to be surprisingly hard as the WHOIS
databases are inconsistently updated, but [third-party databases exist](https://doi.org/10.1145/3487552.3487853)
to help.
In our case [our IP block](https://ipinfo.io/AS44684/185.33.27.0/24) is connected to the [Mythic Beasts AS44684](https://ipinfo.io/AS44684).
Once that was sorted, RIPE once again uses DNS to announce the connection to Mythic:
```bash
$ dig soa -x 185.33.27.0 @pri.authdns.ripe.net +noall +authority
27.33.185.in-addr.arpa. 86400 IN NS ns1.mythic-beasts.com.
27.33.185.in-addr.arpa. 86400 IN NS ns2.mythic-beasts.com.
```
Another nice aspect is being able to control our own reverse DNS to this IP block, which is another
important signal for email reputation:
```bash
$ host pork.recoil.org
pork.recoil.org has address 185.33.27.128
pork.recoil.org has IPv6 address 2a00:1098:39c::3
$ host 185.33.27.128
128.27.33.185.in-addr.arpa domain name pointer pork.recoil.org.
```
### Stopping the Zombie spam horde
We've so far gone to an enormous amount of hassle to get a clean IPv4 block,
but this is necessary but not sufficient to protect ourselves! We also need to
configure our server to defend itself against the zombie botnet hordes.
The overwhelming majority of incoming TCP connections on port 25
are botnets attempting to deliver spam, probe for open relays, guess
credentials (or more recently) harvest data for AI training.
Spending CPU cycles parsing the body of every one of these requests is both
wasteful and dangerous, so a good setup will try to filter these out as early as
possible.
We first deploy Postfix's [postscreen](https://www.postfix.org/POSTSCREEN_README.html), which
listens on port 25 as the first port of call. Architecturally it's a protocol proxy that accepts the
TCP connection, runs a battery of cheap checks:
- DNSBL lookups in parallel from multiple providers
- adds a deliberate pre-greet pause from [RFC 5321 §3.1](https://www.rfc-editor.org/rfc/rfc5321) that catches
bots which start talking before the server's banner appears
- a couple of pipelining and non-SMTP-command tests to check for compliance
It only hands the connection off to the real `smtpd` process if the client looks legitimate after
these. Bad clients are dropped during the pre-greet pause with a temporary failure, which
will encourage false positives to retry in the future.
Interestingly this proxy is exactly what [we do in Docker for Desktop](https://anil.recoil.org/notes/cacm-docker-cover), where a userspace [OCaml VPNKit proxy](https://anil.recoil.org/notes/icfp25-ocaml5-js-docker) mediates between containers and the host network without exposing the host stack directly. I'm going to reimplement postscreen in OxCaml soon...
For those configuring your own server, the relevant Postfix knobs are in `main.cf`:
```ini
postscreen_dnsbl_sites = zen.spamhaus.org*3 bl.spamcop.net*2 b.barracudacentral.org*2
postscreen_dnsbl_action = enforce # reject when DNSBL score crosses threshold
postscreen_greet_action = enforce # reject pre-greet slammers
```
The `*3` and `*2` weights let us combine blocklists rather than trust any single
source; in the above we trust Spamhaus alone and the two weaker lists need to both agree to
reject.
Just postscreen by itself seems to reject over 90% of incoming spam requests, but there's
another trick we can apply.
One minor wrinkle that [Nick Ludlam](https://nick.recoil.org) noticed is that Apple's iCloud email service interacts
badly with postscreen. Apple's outbound MX pool that delivers email doesn't
retry from the same IP, which means postscreen's allowlist for that connection
never matches and mail can stay in limbo for hours. This isn't really a
misconfiguration on our end, but it affect users badly.
The fix (since Apple still owns the whole of `17.0.0.0/8`!) is to allowlist that whole range up front in
`postscreen_access.cidr` so it bypasses the protocol tests entirely. The
[mailcow allowlist guide](https://docs.mailcow.email/manual-guides/Postfix/u_e-postfix-postscreen_whitelist/)
walked me through the syntax in case you hit the same problem:
```bash
# /etc/postfix/postscreen_access.cidr
17.0.0.0/8 permit # Apple iCloud MX pool is somewhere in here
```
#### Greylisting
Once a connection survives postscreen, our next defensive layer is
greylisting ([RFC 6647](https://www.rfc-editor.org/rfc/rfc6647)),
implemented for us by [rspamd](https://rspamd.com/). The idea is that
the first time we ever see a particular source, our server returns a temporary failure rather
than accepting the message, and records the source for future reference.
Legitimate MTAs are required by [RFC 5321](https://www.rfc-editor.org/rfc/rfc5321) to queue/retry after a
few minutes. When the retry comes in, we'll have seen it from the first attempt
and then let the message through. A large fraction of botnets are
"single-shot" delivery engines that just move on to the next victim when they
get a failure, since maintaining a retry queue is expensive when you're
shotgunning millions of messages. The cost to a *real* sender is a one-time
delay of a few minutes, which is amortised across all users from that source
(the greylisting only happens to the first sender).
All these mechanics are implemented in rspamd, which itself has a nice local connection: its author
[Vsevolod Stakhov](https://github.com/vstakhov) did his PhD in the CL with [Jon Crowcroft](mailto:jon.crowcroft@cl.cam.ac.uk) and me, and I spent
much time back in 2015 working with him with [HTTPCrypt](https://www.highsecure.ru/httpcrypt.pdf), a scheme for
opportunistic HTTP encryption that uses NaCl-style cryptography to skip the full TLS
handshake by passing the server public key out-of-band (typically via DNS).
The paper got soundly [rejected from USENIX Security 2015](https://www.usenix.org/conference/usenixsecurity15)
for reasons I can't remember but weren't very important, but the
protocol lives on as the [encryption layer between rspamd and its clients](https://docs.rspamd.com/developers/encryption/)\!
Ok, so now by the time a message has survived postscreen and greylisting, well over 99%
of the original bot traffic has been turned away at the door for a tiny
fraction of the CPU cost of actually reading it. After this, we still need
to do a bit of work scanning the content itself.
#### Milter, ClamAV and Bayesian filtering
Everything that makes it past postscreen and greylisting gets handed to rspamd
over the [milter](https://www.postfix.org/MILTER_README.html) protocol.
Our postfix is configured to consult rspamd for every message,
which allows rspamd to either hard reject or defer a delivery while the
sending server is still 'on the line'
This allows dodgy mail to be refused at the source rather than accepted for delivery
and then bounced afterwards (which is unreliable, as the bouncing address may also
be fake and cause [backscatter spam](https://en.wikipedia.org/wiki/Backscatter_\(email\)) to be generated by us!).
The Postfix configuration for this milter is as easy as running the rspamd
daemon configured to listen on localhost.
```ini
# /etc/postfix/main.cf
smtpd_milters = inet:localhost:11332
```
The first port of call is to filter out messages with known virus attachments.
[ClamAV](https://www.clamav.net/) is what we use for this, and it maintains
its own [virus signature](https://docs.clamav.net/manual/Signatures.html) database.
rspamd hands the message body to the local `clamd` daemon over a Unix
socket, and rejects outright on a virus hit so the message is never delivered.
```ini
# /etc/rspamd/local.d/antivirus.conf
clamav { type = "clamav"; servers = "/run/clamav/clamd.ctl"; action = "reject"; }
```
The other main filter (among many) is rspamd's "[Bayesian
classifier](https://docs.rspamd.com/configuration/statistic/)" that scores each
incoming message against a dynamic corpus of known [spam and
ham](https://en.wikipedia.org/wiki/Anti-spam_techniques) messages stored in
[Redis](https://redis.io/). The classifier auto-learns from messages that score
extremely high or low, but it can also be personalised by the Recoil users by
keeping an eye on each user's Junk folder and adding those to the classifier.
These messages are learnt by being piped to the `rspamc` command, which can
learn both ham and spam on stdin. Over a few weeks of doing this on every
false-positive (or false-negative), the classifier gets pretty good
at matching what our users want to see without having to maintain a static database.
```bash
$ rspamc stat
Results for command: stat (0.028 seconds)
Messages scanned: 6206
Messages with action reject: 135, 2.18%
Messages with action soft reject: 0, 0.00%
Messages with action rewrite subject: 0, 0.00%
Messages with action add header: 137, 2.21%
Messages with action greylist: 658, 10.60%
Messages with action no action: 5276, 85.01%
Messages treated as spam: 272, 4.38%
Messages treated as ham: 5934, 95.62%
```
### Delivering email with LMTP and Sieve
Once our intrepid email message has run the gauntlet of postscreen,
greylisting, rspamd, ClamAV and Bayesian classifiers, we *finally* get to
actually send it to the right user.
By default Postfix would just write the file straight into the user's home
directory, but that's not much use in the modern world where the volume of
email most people receive means that we'd like to file them into folders.
We therefore hand the message over to [Dovecot](https://www.dovecot.org/) via
[LMTP](https://www.rfc-editor.org/rfc/rfc2033), which is basically SMTP (the protocol
we used to receive the email from the outside world), but without the queueing complexity.
This handoff happens over a normal Unix domain socket that's inside Postfix's
queue directory:
```ini
# /etc/postfix/main.cf
virtual_alias_domains = recoil.org
virtual_alias_maps = hash:/etc/postfix/recoil.org
mailbox_transport = lmtp:unix:private/dovecot-lmtp
```
`virtual_alias_maps` turns an arbitrary `anything@recoil.org` address into an
address that can be delivered locally (e.g. `anil@recoil.org` goes to `avsm@pork.recoil.org`).
The reason for handing off to Dovecot rather than letting Postfix write the maildir directly
is that Dovecot then owns the local user operations of indexing, quotas, full-text search and Sieve
filtering. We'll come back to this in the [email retrieval](#accessing-email-for-users) section.
#### Durable on-disk storage with Maildir
Our storage format on disk is the reliable old [Maildir](https://en.wikipedia.org/wiki/Maildir)
format, which stores each email as a single file under each user's `~/Maildir`. It's a format
that we've been using on Recoil since 1998, when we first used [qmail](http://qmail.org/man/man5/maildir.html) as our mail server.
The reason I like it so much is that [processing libraries](https://github.com/avsm/maildir-eio) are
trivial to write, so email is never locked up in a proprietary format or database over the march
of decades.
The format itself is minimal. Each `~/Maildir` is just three
subdirectories `tmp/`, `new/` and `cur/`.
```bash
/home/avsm/Maildir/.archive.2018
/home/avsm/Maildir/.archive.2018/tmp
/home/avsm/Maildir/.archive.2018/cur
/home/avsm/Maildir/.archive.2018/new
/home/avsm/Maildir/.github.mention
/home/avsm/Maildir/.github.mention/cur
/home/avsm/Maildir/.github.mention/new
/home/avsm/Maildir/.github.mention/tmp
...
```
The concurrency story is much simpler as [POSIX guarantees local atomicity](https://en.wikipedia.org/wiki/Rename_\(computing\))
of file rename operations. An incoming message
is first written to a unique file in `tmp/`, then once fully written is (atomically) renamed
into `new/`.
A client reading the Maildir then moves files from `new/` into `cur/` once
they've been seen. The message file itself is immutable, and so clients use the
filename to store message information by appending a flag suffix (e.g. `:2,S`
for Seen, `:2,SR` for Seen and Replied). With this setup, no user level mailbox-wide locking is needed and so
Postfix can deliver a new message without synchronisation while normal email
reading is going on.
#### Indexing the email with Flatcurve
I did evaluate some much more modern email servers like [Stalwart](https://stalw.art/)
which have tons of cool features, but ultimately decided against switching
because they don't support Maildir. They instead require stashing email in a
custom database format (e.g. in RocksDB) which (I think) mixes up the durability
of email with having fast search.
Instead, I took advantage of Dovecot support for full text indexing via a
*separate* index, which is the
[Flatcurve](https://github.com/slusarz/dovecot-fts-flatcurve) full-text index.
Flatcurve is actually a wrapper around the venerable [Xapian](https://xapian.org/),
which (like rspamd) is yet another locally developed Cambridge technology\!
[Martin Porter](https://en.wikipedia.org/wiki/Martin_Porter) (inventor of the famous
[stemming algorithm](https://en.wikipedia.org/wiki/Stemming)), did the Computer Science
Diploma in the CL in 1967 and released the first version of Xapian in 2002. Note that
there's no connection to our [OCaml Xen XAPI toolstack](https://anil.recoil.org/papers/2010-icfp-xen) which was
developed by us independently in 2004\!
In our setup, Flatcurve keeps a separate Xapian index per mailbox under
`~/Maildir/fts-flatcurve` and updates it automatically as new mail arrives.
The Dovecot side of the config is just:
```ini
# /etc/dovecot/conf.d/90-fts-flatcurve.conf
mail_plugins { fts = yes; fts_flatcurve = yes }
fts_autoindex = yes
```
Flatcurve also ships CLI tools as part of the Dovecot plugin to mess around
with the index or do CLI searches (handy for agentic search!):
```bash
$ doveadm fts flatcurve stats -u avsm INBOX
INBOX guid=436177290d20ee4e664a00007b9d9320 last_uid=1018743
messages=83485 shards=2 version=1
```
#### The Sieve language for server-side filtering
Before the message lands in the maildir, we also need to decide which folder
to put it into, or do other preprocessing like labeling. Since this can get
arbitrarily complicated based on the user's needs, we use a
[Pigeonhole Sieve](https://pigeonhole.dovecot.org/) plugin that runs user-defined
delivery-time filters.
Sieve ([RFC 5228](https://www.rfc-editor.org/rfc/rfc5228)) is a declarative language designed specifically for mail
filtering. There's a system-wide script that runs first and
files anything rspamd has flagged into `Junk`, and then each user's personal
script runs:
```ini
# /etc/dovecot/conf.d/90-sieve.conf
sieve_script spam-to-junk { type = before; path = /etc/dovecot/sieve/spam-to-junk.sieve }
sieve_script personal { path = ~/sieve; active_path = ~/.dovecot.sieve }
```
```sieve
# /etc/dovecot/sieve/spam-to-junk.sieve
require ["fileinto", "mailbox"];
if header :contains "X-Spam" "Yes" {
fileinto :create "Junk";
stop;
}
```
Running these rules at delivery time on the server
means the same rules apply whether I'm eventually reading mail on my laptop, phone, or
via webmail. I can also write Sieve rules for very custom vacation rules, email
priorities, and coding agents like Claude can easily figure out the DSL intricacies.
I found the [Sieve cheatsheet](https://gist.github.com/Hotrod369/6b7a24e1ea060e48e0c02459cbb950a0) very
useful here: you can do things like [modify messages](https://gist.github.com/Hotrod369/6b7a24e1ea060e48e0c02459cbb950a0#complex-sieve-script-examples), create dynamic folders and [edit headers](https://doc.dovecot.org/2.3/configuration_manual/sieve/extensions/editheader/).
There's also a "ManageSieve" ([RFC 5804](https://www.rfc-editor.org/rfc/rfc5804)) daemon running,
which lets a mail client edit a user's Sieve script remotely without needing shell access.
I got this working with both [Thunderbird](https://addons.thunderbird.net/en-US/thunderbird/addon/sieve/)
and [Roundcube](https://plugins.roundcube.net/packages/kolab/managesieve) which bundles a plugin natively.
My email filter's massive, but I generate it from OCaml code that outputs something like:
```sieve
if header :contains "List-Id" "caml-list.inria.fr"
{
fileinto "dev.caml-list";
stop;
}
if header :contains "List-Id" "types-list.LISTS.SEAS.UPENN.EDU"
{
fileinto "lists.types";
stop;
}
<...>
```
The Junk Bayesian training loop from earlier piggybacks on this too, as a Sieve
IMAP event
([RFC 6785](https://www.rfc-editor.org/rfc/rfc6785)) script fires on each
folder move, which then pipes the message to `rspamc learn_spam` or `learn_ham`.
This might all feel like a Rube Goldberg machine, but each component does have
its own specialised role.
## Sending email to other people
We've so far put a stupid amount of effort into *receiving* email safely, but
this wouldn't be much use if we also can't reliably *send* email that won't
get rejected.
Failing any one of the gauntlet of checks by the hyperscalers will send our mail to
someone's spam folder or be quietly dropped. There are three separate protocols
that work together to avoid this unfortunate outcome: SPF, DKIM and DMARC.
### SPF describes who can send email for a domain
The [SPF](https://www.rfc-editor.org/rfc/rfc7208) (Sender Policy Framework) protocol
is a DNS TXT record at the apex of our email domain which declares which IP addresses are
allowed to *originate* mail claiming to be from `@recoil.org`. As before, we can
query the live record for Recoil from the CLI:
```bash
$ dig +short TXT recoil.org
"v=spf1 a mx -all"
"Llamaz United"
$ dig mx recoil.org +short
10 pork.recoil.org.
```
The first one is the SPF record, and the second is a random record I created in
1999 to test TXT records. The SPF entry says that the only valid senders for
`recoil.org` are whatever hosts the `MX` records of `recoil.org` point to. In
our case this is just `pork.recoil.org`, and everything else is expected
to be illegitimate email not authorized by us.
It's possible to also declare a softer `~all` softfail that lets receivers accept dubious mail with a
warning. This setup is safe for Recoil because we never send legitimate mail from anywhere except our
own mail server.
One little footgun is that we have lots of IP addresses bound to the
pork.recoil.org host (because of the aforementioned /24 IPv4 block that we were
allocated), and so the Postfix daemon needs to be bound specifically to one
address to ensure that all of the outbound TCP connections it makes are indeed
from the MX entry and not another address from our pool.
```ini
# /etc/postfix/main.cf
smtp_bind_address = 185.33.27.128
smtp_bind_address6 = 2a00:1098:39c::3
```
Without these, Postfix would happily send out from whatever address the kernel
decides to use, which might be the IPv4 we use for the webmail or some other
service entirely, and the receiver's SPF check would fail...
### DKIM: cryptographic signing of outbound mail
SPF only authenticates the connecting IP of the email server to other people, but
says nothing about the contents of the message itself. We could still use a way
to authenticate that a full message has been certified by Recoil as originating
from us, even if that message has been through several other email relays (e.g. by
being forwarded).
The [DKIM](https://www.rfc-editor.org/rfc/rfc6376) (DomainKeys Identified Mail)
protocol adds a per-message cryptographic signature, in a `DKIM-Signature:`
email header. This covers a canonicalised form of the body and selected headers,
to permit some flexibility in rearranging the email but still be robust against
tampering. The public verification keys for DKIM live in the public DNS and so
are available for any receiver to easily check.
The job of signing every single message leaving Recoil is [rspamd's job](https://docs.rspamd.com/modules/dkim_signing/). Our
DKIM private key lives on disk and never leaves the server. rspamd
adds the signature to every outbound message during the same milter pass that
[scores inbound mail](#milter-clamav-and-bayesian-filtering):
```
# /etc/rspamd/local.d/dkim_signing.conf
domain {
recoil.org { selector = "mail"; path = "/var/lib/rspamd/dkim/recoil.org.mail.key" }
}
```
Since you don't want to have the same private key in use forever, DKIM
supports rotation via "selectors" (`mail` in our case). This lets us rotate keys by
publishing a new public key under a new selector while keeping the old one
live, so signatures on already-sent mail still verify. The public side
lives in DNS at `._domainkey.`:
```
$ dig +short TXT mail._domainkey.recoil.org
"v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA..."
```
One gotcha if you're setting this up yourself is that DKIM TXT records
frequently exceed the [255-byte string
limit](https://www.rfc-editor.org/rfc/rfc7208) and have to be split into
multiple quoted strings inside one record. Most authoritative DNS providers
will do that for you, but you may need to mess around in their various web UIs to figure out how.
Another oddity about DKIM is that it's really intended for live verification. If you want
to go back and re-verify some emails that are (say) a few years old, then the DNS keys
will have expired and so you won't be able to do so. If anyone knows of any DKIM
[transparency logs](https://certificate.transparency.dev/) like exist for TLS, I'd love
to try it to go back over my historical email and do some data mining.
### DMARC ties it together and provides reporting stats
Neither SPF nor DKIM actually check the `From:` header that
decides what our users actually see in their mail client:
- SPF only authenticates the envelope sender (the `MAIL FROM` SMTP command)
- DKIM only authenticates the sending domain in its own `d=` tag.
Without some glue, a spammer could pass the SPF check for their own `evil.example.com`,
then sign a message with a valid DKIM key for `evil.example.com`, and
*still* write `From: anil@recoil.org` in the message that the user eventually reads.
The protocol glue to prevent this is [DMARC](https://www.rfc-editor.org/rfc/rfc7489), which checks
that the domains authenticated by SPF and DKIM actually match the visible
`From:` in the email message, and also tells receivers what to do when the check fails.
As you might have guessed by now, this involved yet another DNS record:
```
$ dig +short TXT _dmarc.recoil.org
"v=DMARC1; p=quarantine; rua=mailto:postmaster@pork.recoil.org"
```
The strongest policy is `p=reject`, but we're going for a softer 'quarantine'
until I'm comfortable with the setup for a few more months.
A _really_ useful part for actually debugging deliverability (given how many
third parties are involved here) is `rua=`, which is the email address for
a regular aggregate report.
Once a day or so, every major receiver who gets email from Recoil (including Google,
Microsoft, Yahoo, Fastmail, and some smaller ones) sends an XML report to
this address summarising the messages they saw claiming to be from
`recoil.org`.
Some of these actually look like valid fails; for example this one from Yahoo
seems to indicate that we've had some email sent *not* from our servers:
```xml
$ gzcat yahoo.co.uk!recoil.org!1780617600!1780703999.xml.gz
Yahoodmarchelp@yahooinc.com1780732274.74281717806176001780703999recoil.orgrr
