Own mail server: Postfix + Dovecot + SPF/DKIM/DMARC on Debian 12

Running your own mail server is rare these days. Google Workspace and Microsoft 365 are cheap and secure – for 90% of people they are a better choice. But there are situations where a self-hosted server makes sense: GDPR regulations, custom integrations, large scale. This post is a complete tutorial that lets you set up the server in 4-6h.

Prerequisites

• Debian 12 (or Ubuntu 24.04) server with public IPv4
• Domain with configured NS (your own nameserver or managed)
• Reverse DNS (PTR) – you must be able to set it (DigitalOcean, Hetzner, OVH – yes; cheap hosting – often no)
• Open ports: 25 (SMTP), 587 (submission), 993 (IMAPS)

PTR record is critical. Without it your mails go to spam. Check before you start:

# Check reverse DNS
dig -x 1.2.3.4 +short
# Should return e.g. "mail.example.com"

If it returns something else (e.g. “static.1-2-3-4.provider.com”) – write to provider support and ask to change PTR.

Installing packages

sudo apt update
sudo apt install postfix postfix-policyd-spf-python \
  dovecot-core dovecot-imapd dovecot-lmtpd \
  opendkim opendkim-tools \
  certbot

During Postfix install you will be asked:

• General type: Internet Site
• System mail name: example.com (your domain)

Postfix configuration

main.cf

Edit /etc/postfix/main.cf:

smtpd_banner = $myhostname ESMTP $mail_name
biff = no
append_dot_mydomain = no
readme_directory = no

# Hostname and domain
myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain

# Network
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
inet_interfaces = all
inet_protocols = ipv4

# SASL (for mail clients)
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain = $myhostname

# TLS (encryption)
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_tls_security_level = encrypt
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_ciphers = high
smtpd_tls_exclude_ciphers = aNULL, eNULL, EXPORT, DES, RC4, MD5, PSK, SRP, DSS, AECDH, ADH

# Antispam restrictions
smtpd_helo_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname
smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_non_fqdn_sender, reject_unknown_sender_domain
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_non_fqdn_recipient, reject_unknown_recipient_domain, reject_unauth_destination, check_policy_service unix:private/policyd-spf

# Relay restrictions (critical - protects against open relay)
smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination

# Alias
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
virtual_mailbox_maps = hash:/etc/postfix/virtual
virtual_mailbox_domains = example.com
virtual_transport = lmtp:unix:private/dovecot-lmtp

Key settings that matter:

• smtpd_relay_restrictions with defer_unauth_destination – protects against open relay. NEVER remove it. Test the configuration at https://www.mail-tester.com.

• smtpd_tls_protocols excluding old TLS – without it the server accepts SSLv3 (vulnerable to POODLE). Excluding !TLSv1.1 and older = security.

• check_policy_service unix:private/policyd-spf – enables SPF checking (more on that below).

master.cf

Add to /etc/postfix/master.cf (at the end):

# Dovecot SASL
submission inet n - y - - smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject

# PolicyD SPF
policyd-spf unix - n n - 0 spawn
  user=policyd-spf argv=/usr/bin/policyd-spf

Verify Postfix

# Check syntax
sudo postfix check

# Reload config
sudo systemctl reload postfix

# Check status
sudo systemctl status postfix

Dovecot configuration

/etc/dovecot/dovecot.conf

protocols = imap lmtp
listen = *, ::

/etc/dovecot/conf.d/10-mail.conf

mail_location = maildir:~/Maildir
mail_privileged_group = mail

/etc/dovecot/conf.d/10-auth.conf

disable_plaintext_auth = yes
auth_mechanisms = plain login

/etc/dovecot/conf.d/10-master.conf

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0600
    user = postfix
    group = postfix
  }
}

Creating a user

# System user (no /bin/sh for security)
sudo useradd -m -s /usr/sbin/nologin jan
sudo passwd jan

The password is also the IMAP/SMTP password. For higher security use virtual users (see: further extension), but that is out of this tutorial’s scope.

OpenDKIM configuration

DKIM signs each message cryptographically. Without it Gmail and others treat you suspiciously.

Generating keys

sudo mkdir -p /etc/opendkim/keys/example.com
cd /etc/opendkim/keys/example.com
sudo opendkim-genkey -s default -d example.com
sudo chown opendkim:opendkim default.private default.txt

/etc/opendkim.conf

Domain                  example.com
KeyFile                 /etc/opendkim/keys/example.com/default.private
Selector                default

# Standard
Canonicalization        relaxed/simple
Mode                    sv
SubDomains              no
AutoRestart             yes
AutoRestartRate         10/1M
Background              yes
DNSTimeout              5
UserID                  opendkim:opendkim

# Socket
Socket                  inet:12301@localhost

/etc/default/opendkim

SOCKET="inet:12301@localhost"

Add to Postfix

In /etc/postfix/main.cf add:

milter_protocol = 2
milter_default_action = accept
smtpd_milters = inet:localhost:12301
non_smtpd_milters = inet:localhost:12301

Restart and test

sudo systemctl restart opendkim
sudo systemctl restart postfix

DNS records

Add the following records in your domain DNS:

A record for mail.example.com

mail.example.com.  3600  A  1.2.3.4

MX record

example.com.  3600  MX  10  mail.example.com.

SPF record (TXT)

example.com.  3600  TXT  "v=spf1 mx ~all"

~all means soft fail – mails from other servers are suspicious, but not rejected. Safer than -all at the start.

DKIM record (TXT)

Get the public key:

sudo cat /etc/opendkim/keys/example.com/default.txt

Add to DNS:

default._domainkey.example.com.  3600  TXT  "v=DKIM1; k=rsa; p=MIGfMA0GCSq..."

DMARC record (TXT)

_dmarc.example.com.  3600  TXT  "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com; pct=100"

p=quarantine means: if SPF+DKIM fail, mail goes to spam. rua= is the address for reports (who tries to impersonate your domain).

SSL/TLS with Let’s Encrypt

# Stop Postfix/Dovecot temporarily (needs port 80)
sudo systemctl stop postfix dovecot

# Get cert
sudo certbot certonly --standalone -d mail.example.com

# Start services
sudo systemctl start postfix dovecot

# Test auto-renewal
sudo certbot renew --dry-run

Certbot auto-renews the certificate (script in /etc/cron.d/certbot). Postfix and Dovecot automatically use the new certificate after restart – worth adding to cron:

0 3 1 * * /usr/bin/systemctl reload postfix dovecot

Testing

1. mail-tester.com

Send a mail from jan@example.com to the address shown on mail-tester.com. It should score 9+/10. If <7 – something is wrong.

2. Send to Gmail

Send a mail to your Gmail account. Check Show original – it should have:

• SPF: PASS
• DKIM: PASS
• DMARC: PASS

3. Send to your own address

# Check if mail arrives locally
echo "Test" | mail -s "Test" jan@example.com
mailq  # queue
sudo tail -f /var/log/mail.log  # logs

4. Test TLS

# Check if submission uses TLS
openssl s_client -connect mail.example.com:587 -starttls smtp

Troubleshooting

Mail does not arrive, logs: “relay denied”

smtpd_relay_restrictions does not allow it. Check whether the client authenticates (SASL).

SPF=neutral despite a record

Common mistake: PTR points to 1-2-3-4.provider.com instead of mail.example.com. Without correct reverse DNS SPF will not pass.

DKIM=FAIL

The public key in DNS does not match the private one. Check:

# Local key
sudo cat /etc/opendkim/keys/example.com/default.private | head -1

# DNS
dig TXT default._domainkey.example.com +short

Mails go to spam despite 10/10 on mail-tester

Two reasons:

1. New domain – Gmail/Outlook treat new domains suspiciously for the first 2-4 weeks. Solution: send regular valid mails, “warm up” the domain.
2. Message content – too many links, missing alt text, capital letters in subject. Fix the content.

Production hardening

1. Fail2ban – automatic IP ban after 3 failed login attempts:

   sudo apt install fail2ban
   sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
   # Enable jail for postfix and dovecot
   sudo systemctl enable --now fail2ban

2. Postscreen – Postfix has built-in anti-spam. Add to master.cf:

   smtpd     pass  - - - - - smtpd
     -o smtpd_proxy_filter=127.0.0.1:10025

3. Monitoring – Prometheus + Grafana + node_exporter for system metrics, checking Postfix queue with cron.

4. Config backup – /etc/postfix/, /etc/dovecot/, /etc/opendkim/. Without it after a failure recovery takes 6h.

What is next

• Cloudflare for WordPress
• BTRFS + snapshots on Arch/CachyOS
• WordPress REST API in practice

If you need a mail server set up in Polish cloud (DigitalOcean, Hetzner) with full SPF/DKIM/DMARC configuration and monitoring – get in touch. 1-2 days of work, starting from 1500 PLN.